Foreman (with katello) seems to be a dream alternative to redhat satellite and now more or less defunct spacewalk. It has Eratta support: https://theforeman.org/static/images/blog_images/2016-03-02-addressing-cves-with-katello/all_errata.png

Also appears to be able to patch CentOS (which has no official eratta though but good enough).

Katello seems only to be used for Yum based distributions.

Would there be a way to build a similar system for Ubuntu? Maybe input Ubuntu USN and use as eratta or simply fetch all available patches and somehow present them in the foreman user interface? Preferably with checkboxes for each package I want to update.

I'm not really sure how to accomplish this, sounds much like I need to learn how to write plugin to accomplish this in a more fancy manner.

Maybe this is rocket science, and I am doomed to bluntly run apt-get upgrade and such through remote execution/ansible jobs in foreman..?

Anyone of you have experience with a more controlled and granular ubuntu patching (or debian for that matter) using at least foreman?

So I took this tutorial and followed it to the tee: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-openvpn-server-on-ubuntu-16-04

I have completed everything but I do not know that much about networking and I have no idea what I have done. After all this setup I have two new Logins; OpenVPN and OpenVPN-ca. Did I create a VPN server on my original login so that I can relay my other accounts through it or is this just by-product of having installed OpenVPN. How can I import an .OPVN file and use it with openVPN, Can I do it with my original Login that I installed it in or do I need to login into one of the OpenVPN logins that have appeared.

I understand I have configured my Internet traffic to go through a specialized connection but I don't know how or where to start it and I do not know how to import my VPN services .OPVN files. The one I have imported shows up under the network manager under VPN service but every time I click it, it show a connection failure immediately.

I realize to you guys this is probably a stupid question but I would appreciate any help or education you could give me.

I would like to run a virtual lab to show a piece of software - imagine 10 students, each using 3 VMs (somehow students shouldn't be able to touch each other's machines). It's going to be a small factor server that has fake raid. I'm going to need mdadm - Intel says use mdadm for Intel RSTe fake raids. I am in the process of choosing what hypervisor (if that's correctly termed) to install. I'd like a decent UI, preferable web-based, I won't shy away from ssh. mdadm support too.

My options so far:

I have some experience with ProxMox. I like it, the stuff missing from the UI i can do myself - create fake adapters to simulate NAT and to segregate machines with multiple network adapters. mdadm might be difficult to install - there are tutorials, but it's not officially supported.

I have tried ESXi, but I don't find it easy to use. It's not a normal linux. The UI is pretty, but I couldn't manage to import a Windows machine - neither from OVA/OVF, nor even a .vmware image - it needs to convert them, which makes no sense. It also cannot use mdadm, or if it can it will be very, very difficult.

XEN i've heard doesn't have a control panel, instead I have to install its windows client on one of the machines.

OpenStack is another software I'm considering. I'm trying to run DevStack to evaluate, but it's not going well. I've heard good things about it, and from what I've heard it's like AWS and suits my requirements fine. But I've been told it's quite complex and better suits a company with multiple servers, not a single machine. It contains multiple pieces (nova, keystone, glance, neutron ...) and it might not be easy to handle. I don't know about mdadm, but since it runs on multiple distributions, it might be doable.

I don't know about other options. Any comments?

When I was in a Windows shop with tons of RDP, Citrix, some VNC and yes.. some SSH accessible boxes, an application like MRemoteNG was incredibly valuable. I could, in a tree-heirarchy, create groups of connections. MremoteNG mostly was a wrapper/container for other 3rd party applications, like Putty, or Windows native RDP client.

Once you created your connections and groups, you could right click a group and connect to all. The application would then create a tab named after the group, and then sub-tabs, one for each connection. It was a nifty way to pop into all e-mail servers, or say.. pop into all the Windows/Citrix instances for a particular customer.

In Linux, for years I've managed everything locally with openssh, GNU Screen, and xfce4-terminal (easier terminal to set tab names too). But running GNU screen locally, isn't really where you get the true benefits of GNU screen. I'd like to leave a session running on every single sever we have and just attach to that session when i reconnect to that server.

Gnome Connection Manager is a python application which...sort of does what I want, except for the fact that the terminal emulator in it is kinda poor and lacking options (no right click menu.. feels too much like Putty). Furthermore, it doesn't see my ~/.ssh/config and is reliant on me re-creating all my connections. Again. And while it'll allow me to connect to all my hosts in a group, there's no tab hierarchy, like MRemoteNG.

Remmina is even worse.

I guess what I'm after is

• a wrapper around xfce4-terminal, Gnome-Terminal or Konsole (or something similar).
• I only need SSH access and local access
• I'd like it to be ~/.ssh/config aware in some fashion. Having to maintain my connections in two places is absurd.
• tree-hierarchy of groups/hosts
• ability to right click and connect to all hosts in a group
• preferably tab/sub-tab relationship in the connection window
• tabs should auto-name themselves based on group and/or host name. (i already have this working in screen, it's amazing how many terminals don't support this).

Honestly - if it wasn't such a pain in the ass, I'd use Gnu Screen locally to manage Gnu Screen remotely..but the keybindings get a little out of control at that point.

Any pointers?

I wrote a simple, interactive tool in Python to push out additional SSH keys to remote systems in places where I couldn't use Ansible or another means for this.

Sharing this here in case anyone else can use it, I told myself that I'd never again use an ssh for loop for this.

So I'm trying to make a Local DNS using Bind in CentOS7. everything was fine and it loaded all is good and it reslove URLs BUT when I'm trying to resolve IPs it doesn't wok! and I got this error in /var/log/messages

192.168.100.9 is the IP of the local machine that's have the DNS itself

Nov 4 07:59:22 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:500:2d::d#53 Nov 4 07:59:25 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:500:1::803f:235#53 Nov 4 07:59:25 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:500:1::803f:235#53 Nov 4 07:59:26 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:503:ba3e::2:30#53 Nov 4 07:59:26 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:500:2f::f#53 Nov 4 07:59:26 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:503:ba3e::2:30#53 Nov 4 07:59:26 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:500:2f::f#53 Nov 4 07:59:28 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:503:c27::2:30#53 Nov 4 07:59:29 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:503:c27::2:30#53 Nov 4 07:59:30 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:7fe::53#53 Nov 4 07:59:30 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:500:3::42#53 Nov 4 07:59:30 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:7fe::53#53 Nov 4 07:59:30 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:500:3::42#53 Nov 4 07:59:30 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:dc3::35#53 Nov 4 07:59:30 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:dc3::35#53 Nov 4 07:59:32 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:7fd::1#53 Nov 4 07:59:32 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:7fd::1#53 Nov 4 07:59:32 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:500:2d::d#53 Nov 4 07:59:33 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:503:ba3e::2:30#53 Nov 4 07:59:33 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:503:ba3e::2:30#53 Nov 4 07:59:34 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:500:1::803f:235#53 Nov 4 07:59:34 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:500:1::803f:235#53 Nov 4 07:59:38 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:500:2f::f#53 Nov 4 07:59:38 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:500:2f::f#53 Nov 4 07:59:40 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:503:c27::2:30#53 Nov 4 07:59:40 localhost named[9780]: error (network unreachable) resolving './NS/IN': 2001:503:c27::2:30#53 Nov 4 07:59:42 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:7fe::53#53 Nov 4 07:59:42 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:500:3::42#53 Nov 4 07:59:42 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:dc3::35#53 Nov 4 07:59:42 localhost named[9780]: error (network unreachable) resolving '192.168.100.9/A/IN': 2001:500:2d::d#53 

EDIT:Found the solution Thanks for your help guys.

It was pretty much stupidity from me. I should have used -x with dig.

Besides Amanda, Unitrends, and Bacula, are there other Linux-based backup solutions you can install on your own hardware?

At minimum, it needs to support VMware image backups. Bonus if it can backup MySQL with a plugin.

I've encountered the following error messages when dealing with CIFS shares on a variety of RHEL machines:

CIFS VFS: Send error in SessSetup = -5 CIFS VFS: cifs_mount failed w/return code = -5

I've looked high and low for the meaning of these error codes and come up with nothing. Anyone know what they mean?

I'm starting my RHCSA prep and just wondering which distro would be closer to RHEL to use on a spare machine. CentOS or Fedora.

All I am doing is to copy a directory recursively to a destination directory. I rpmbuild -ba the .spec.

 %define __spec_install_post %{nil} %define debug_package %{nil} #%define __os_install_post %{_dbpath}/brp-compress Summary: be3 Name: be3 Version: 1.0 Release: 21 License: GPL+ Group: Development/Tools SOURCE0 : %{name}-%{version}.tar.gz BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root %description %{summary} %prep %setup -q %build # Empty section. %install rm -rf %{buildroot} mkdir -p %{buildroot}/home/testeng/be3 # in builddir rsync -a home/testeng/be3* %{buildroot}/home/testeng/be3 %clean rm -rf %{buildroot}/* %files %attr(-,testeng,test) /home/testeng/be3* 

However, when I try to yum install the rpm I build, it gives me dependencies issues.

--> Processing Dependency: libssl.so.4()(64bit) for package: be3-1.0-21.x86_64 ---> Package libxcb.x86_64 0:1.11-4.el7 will be installed ---> Package libxshmfence.x86_64 0:1.2-1.el7 will be installed ---> Package mesa-libglapi.x86_64 0:10.6.5-3.20150824.el7 will be installed --> Finished Dependency Resolution Error: Package: be3-1.0-21.x86_64 (/be3-1.0-21.x86_64) Requires: libssl.so.4()(64bit) Error: Package: be3-1.0-21.x86_64 (/be3-1.0-21.x86_64) Requires: libcrypto.so.4()(64bit) 

What baffles me is, why is there even dependencies that I need to install? It's just a mkdir -p then rsync the content to that directory.

That said, there are libssl.so.6 and libcrypto.so.6 in the source. Can anyone give me some pointers? I'm quite confused.

Apologies in advance for the long post. If I could find a way to shorten it, I would.

So I have posted here before about the difficulty I have had getting traction moving into professional Linux admin. Over the last two months, thanks to a combination of additional study, heavy revision of my CV and seasonal change (end of summer) interest has really picked up but I am bewildered as to how to demonstrate the 'in depth knowledge' that employers seem to expect in technical tests given the apparent fondness for testing on stuff that isn't on the spec or on my resume and the truly vast range of software out there. Given the breadth and the rate of innovation and churn I don't really see how anyone could. It is this that I am seeking advice on.

I am completely candid about my actual levels of skill and experience on my CV and at interview. A number of people have advised me to 'embellish' my skills but I have declined to do so. Everything I present is something that I have really used and I am clear about how, where and for what I have used it. I am getting a lot of phone calls from recruiters and employers and at the moment averaging one face-to-face interview a week with about half inviting me back for a second interview/technical test. This tells me that the skill set and presentation on my CV is reasonably attractive and that my interview technique is 'sufficient'. The thing is that the employers then present a technical test on stuff that I have never used nor claimed expertise on and typically has not even been discussed or advised in advance, it just seems to be expected that as a potential admin/engineer, candidates know about 'all the software', have some sort of telepathy that can understand exactly what has been set up and how and what the person who did that was trying to achieve. Everyone's stack is different, even for the same type of role and I am at a loss trying to identify where to focus my efforts to sharpen my skills.

Skill set given on my resume:

Candidate Summary

An IT trouble shooter, consultant, administrator and user trainer with experience gained from conducting assignments for private clients. After transitioning from a largely non-IT related role to the IT support industry in 2015, I am now looking for new opportunities in the commercial sector to offer infrastructure support services with a focus on Linux based environments.

Skills / Technical Summary

• Ubuntu Linux (desktop & server) 11 years experience
• Red Hat Linux (desktop & server) – 5 year experience. Familiar with all parts of RHCSA syllabus (RHEL7) Scored 170/300 (pass mark 210) in RHCSA exam May 2016
• Debian Linux (desktop & server) 7 years experience
• Strong interpersonal and communication skills – see ‘<$Previous employer>’ below
• SSH/SCP/SFTP server and client setup inc. cross-platform
• LAMP stack local build and in cloud with FTP/SSH access
• Virtualisation: KVM (including virtualised networks); VirtualBox; VMWare; Amazon EC2
• WordPress consultation, design; setup; integration; implementation for public facing client sites– e.g. see <$Client website> ; <$Client website> (independent subdomain)
• Apache – set up and configuation lcocally and with various cloud services in support of http file server; WordPress; Webmin; PHPMyAdmin etc
• nGinx – as reverse proxy for Apache
• selinux – see paper published on my LinkedIn
• lvm
• LDAP client configuration
• CUPS print server set up and administration
• SAMBA server and client setup and administration
• YUM repository setup and configuration
• User documentation and dissemination (see ‘eBook creation’ immediately below)
• ebook creation (ePub/Mobi/azw) – see paper published on my LinkedIn
• Comfortable using terminal interface (‘Shell’)
• Basic BASH scripting- see example on my GitHub linked below
• Basic Python – currently studying to improve
• Basic familiarity with Puppet
• Git/Github - https://github.com/<Github Profile>
• Basic familiarity with Docker
• MySQL/MariaDB setup and administration
• Basic familiarity with AWS, especially EC2
• Linux custom kernel compilation and packaging to incorporate ‘problem’ drivers
• Debian packaging and conversion with rpm
• Regular attendee at ‘Linuxing in London’ Meetups – see https://www.meetup.com/Linuxing-In-London/
• Microsoft Windows XP; 7; 8; 8.1; 10
• Microsoft Office 2010; 2013; 365 to advanced level and trainer in 1:1 and corporate environments – see below under both ‘Freelance’ and ‘<$Previous employer>’
• Bespoke applications trainer in corporate environment to staff at all levels – junior admin to director– see below under ‘<$Previous employer>’
• Software systems integration pilot and facilitation
• Microsoft Internet Explorer; Mozilla Firefox; Google Chrome
• RSA token administration (IronKey) and training
• LibreOffice
• iPhone/iOS user and profile administration
• User accountability monitoring setup using Mobicip (iOS) and Refog (Windows)
• HP / Xerox personal and shared printers setup; installation; administration

Employer example 1:

From Job spec:

Systems Administrator

Duties The systems administrator/developer will be responsible for various administration and support tasks on the WordPress and Ruby on Rails websites we host, as well on their supporting infrastructure.

The systems administrator/developer must have the following skills:

1. Ability to discuss and describe technical problems in simple, lay terms
2. Deployment and management of virtualised infrastructure; for example, Amazon AWS
3. Familiarity with unix/linux systems
4. Familiarity with popular open source software such as Apache, Varnish, MySQL and Nagios
5. Familiarity with fundamental internet protocols (HTTP, TCP/IP, DNS, etc)
6. Sufficient development experience to be able to diagnose problems in any programming language and make simple fixes
7. Experience using source control systems; in particular, git
8. Experience using Chef, Puppet or similar configuration management systems
9. Excellent written and spoken English
10. Ability to approach problems methodically Bonus

The following skills would be useful:

1. Familiarity with Ubuntu/Debian Linux in particular
2. Experience with container­based approaches, eg Docker
3. Experience working on projects involving online payments and PCI compliance

Technical test

Debug a completely banjaxed server stack with Varnish in front of nginx as a reverse proxy to Wordpress on Apache. I wrote about this here. Test machine was an ubuntu VM in VirtualBox. No use of many of the skills listed in the spec. At the time of applying to this employer (with an earlier version of my CV) I had never claimed any experience whatsoever with either Varnish or Nginx (the latter which wasn't even mentioned in the job spec). Sure I can google them etc but yeeah... I have subsequently familiarised myself with nginx.

Employer Example 2:

From job spec:

DevOps engineer to join our existing team and help us roll out, monitor, scale and optimise our systems.

• Design, set up and maintain deployment systems
• Design, set up and interrogate system monitoring
• Improve and optimise existing system infrastructure
• Implement best­practice access control and system security
• Work with development teams to identify and resolve uptime, performance and scalability issues
• Communicate with multi­disciplinary internal and external teams
• Take ownership of problems and get stuff done
• Stay up to date with the latest technology developments and be prepared to present them internally and to clients

Requirements

• Demonstrable investigative and problem solving skills
• Amazon Web Services (especially EC2 & RDS)
• At least one scripting or system orchestration language
• Linux (preferably Ubuntu)
• Relational database admin (preferably PostgreSQL)
• Some knowledge of TCP/IP Networking
• Working iteratively with regular delivery (especially continuous delivery)
• Git (or another DVCS)

Would be great if you knew about these:

• Python
• nginx
• NewRelic
• memcached
• Vagrant (or other virtual machine tools)
• Jenkins (or other continuous integration tools)
• Sentry

Technical test:

Here's our preproduction server farm on AWS, the ssh login for all of them, a pen and paper. I'll be back in an hour, tell me what you can find out about all of these machines.

I identify that the machines are serving a custom python app via nginx supported by gunicorn and coordinated using rabbitmq and celery. There were 16 machines in the farm (8 or so unique) so obviously limited time to explore the custom configuration on each one. They had told me in advance that their whole operation was currently all managed via bash scripts so there were no playbooks or recipes to refer to, these are all 'pet' machines. They all had plymouth (ubuntu graphical login manager) installed on them for instance, on AWS... I have recently used nginx a little and state basic python on my resume but I have never used or claimed any knowledge of Rabbitmq, Celery or Gunicorn. Yes I googled them on that occasion but I could not tell you offhand much about them. I'm not a dev and have not claimed to be. Again, not listed in the job spec and not discussed previously. Recruiting manager seemed disappointed that I had not gone in depth with their manually installed/served custom applications run by a company with 150 staff. Yes, I had asked after the previous employer test 'anything I can prepare?' and the recruiting manager just said 'oh, just how to ssh into an Ubuntu machine'.

How on earth do I prepare for my next technical test?

I am one person, not a whole company. I can only prepare for specifics that aren't listed on my resume or the job spec if I am told what these are and with the resources of a single individual. Obviously if I fancy myself to any level on a given software then I list it on my resume. I do not have a 'test-buddy' to set up a server farm for me to play detective with or set up a borked stack for me to debug. Even if I did you could bet that the next employer would then want to test me on something else that was neither in my CV, their job spec or discussed in advance. I am not pretending expertise that I don't have but I cannot believe that every candidate automatically knows how to do everything. How do I deal with this?

Thanks

I'm working throught the famous list, and I opted for using Foreman instead of Spacewalk.

When I create and start a new host, Foreman creates for instance:

/var/lib/tftpboot/pxelinux.cfg/01-52-54-00-d1-b1-4c 

I have to copy said file to:

/var/lib/tftpboot/pxelinux.cfg/52-54-00-d1-b1-4c 

and reboot the new machine. Afterwards, everything works as expected. This is an inelegant ""solution"", and I'd like to solve it for good.

On the internet, I have found this: http://projects.theforeman.org/issues/15985

I have to add the missing "01-" to the provisioning template used by the OS in front of the machine MAC address. I can not find where in the template/s the url to request the configuration is defined.

Does anyone know where to add the "01-" ? I should be able to find something like:

tftp connect <%= foreman_url %> tftp get pxelinux.cfg/<%= pxe_config_file %> 

right? Maybe, not exactly that. But close. Then, I could add the "01-" to the definition of the pxe_config_file string.

[Centos7.2]I'm following this guide. when I attempt to create my own virtual machine, it gives me the following error:

# virt-install \ > --name test1.test-env.com \ > --ram 2048 \ > --disk path=/data/vm_storage/test1.qcow2,size=12 \ > --vcpus 1 \ > --os-type linux \ > --os-variant centos7 \ > --network bridge=virbr0 \ > --graphics none \ > --console pty,target_type=serial \ > --location '/data/boot_images/CentOS-7-x86_64-DVD-1511.iso' \ > --extra-args 'console=ttyS0,115200n8 serial' ** (process:15425): WARNING **: Failed to load shared library 'libosinfo-1.0.so.0' referenced by the typelib: libxslt.so.1: cannot open shared object file: No such file or directory /usr/share/virt-manager/virtinst/osdict.py:197: Warning: cannot retrieve class for invalid (unclassed) type 'void' loader = libosinfo.Loader() ERROR could not get a reference to type class 

Did some google, but none have that error, there's something similar but a different library but it's about ubuntu on another thread that didn't help much for me.

just need a page that shows real quick the status of the applications. We are using nagios and I just need to show a quick status page for management. Showing them nagios gets them confused. I prefer free and open source :)

This may be a better question for /r/sysadmin, but I figured I would start here. Anyone know if its possible to automate the installation of an Oracle DB from within a Kickstart script? Or is this something better handled by a configuration management tool or done in some form of post-os installation capacity?

I have googled and googled. In my.cnf, skip networking, bind-address 127.0.0.1, and skip-external-locking are all commented out. I have also tried with a bind address of 0.0.0.0.

The firewall is turned off, and iptables -L shows ACCEPT as the only rule.

'root'@'%' has been created.

netstat -atn shows 3306 listening to 0.0.0.0

netstat -l shows *:mysql listening on * : *

If i telnet myhost 3306 the connection times out.

Any ideas?

Hi!

I've been finding a problem the last month (and other added the last days)

I'm kinda noob on elasticsearch and all I know is from internet howto's and youtube videos.

There are always X unassigned shards (it's a standalone node with 8vcpu and 32GB RAM) and I don't know how to reassign them. I've followed some tutorial but it gives me an error when trying to force the shard allocation.

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ "commands" : [ { "allocate" : { "index" : "indexNAME-20161024", "shard" : 3, "node" : "Viper", "allow_primary" : true } } ] }'

Answers:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"[allocate] allocation of [INDEXNAME][3] on node {Viper}{IG88cQOjQoSexClMjUPN7w}{172.31.11.109} {172.31.11.109:9300} is not allowed, reason: [YES(no allocation awareness enabled)][YES(allocation disabling is ignored)][YES(shard not primary or relocation disabled)] [YES(allocation disabling is ignored)][YES(primary is already active)][YES(below shard recovery limit of [2])][YES(target node version [2.1.0] is same or newer than source node version [2.1.0])][NO(shard cannot be allocated on same node [IG88cQOjQoSexClMjUPN7w] it already exists on)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)] [YES(only a single data node is present)][YES(node passes include/exclude/require filters)]"}],"type":"illegal_argument_exception","reason":"[allocate] allocation of [INDEXNAME][3] on node {Viper}{IG88cQOjQoSexClMjUPN7w}{172.31.11.109}{172.31.11.109:9300} is not allowed, reason: [YES(no allocation awareness enabled)][YES(allocation disabling is ignored)][YES(shard not primary or relocation disabled)][YES(allocation disabling is ignored)][YES(primary is already active)][YES(below shard recovery limit of [2])][YES(target node version [2.1.0] is same or newer than source node version [2.1.0])][NO(shard cannot be allocated on same node [IG88cQOjQoSexClMjUPN7w] it already exists on)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(only a single data node is present)][YES(node passes include/exclude/require filters)]"},"status":400 

I assume that it's impossible to reassign on the same node and I'm scared of data loss (there is a daily snapshot of the ec2 instance). The other problem is that every day, around 6:30/6:45 the machine hangs. Looks that it's because there is not enough memory and kills java process (extract from var/log/kern.log). I've googled a bit and maybe it's a garbage collector but eating 15 free GB of ram for that looks weird:

Out of memory: Kill process 25966 (java) score 339 or sacrifice child Oct 9 06:51:29 localhost kernel: [107439.225008] Killed process 25966 (java) total-vm:18860224kB, anon-rss:13947056kB, file-rss:13936kB Oct 9 06:51:35 localhost kernel: [107446.879240] init invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 Oct 9 06:51:35 localhost kernel: [107446.879244] init cpuset=/ mems_allowed=0 Oct 9 06:51:35 localhost kernel: [107446.879247] CPU: 5 PID: 1 Comm: init Not tainted 3.13.0-77-generic #121-Ubuntu 

Any thought?