Quantcast
Channel: linuxadmin: Expanding Linux SysAdmin knowledge
Viewing all articles
Browse latest Browse all 17905

Moving /var on its own partition without entering single user mode

$
0
0

Hello everyone! I need to move the /var on its own partition on a server where I have only SSH access. I’ve searched around the web and everyone seems to suggest to enter single user mode first, so no process will write on /var during the mirroring process, but of course this is not possible to do over an SSH connection.

I have set up the partition and everything is ready from that part, so I’m not asking for help with that. The only thing that remains to be done is to copy the contents of the current /var to the new partition and replace the old /var directory with it. So, my question is what is the proper way to ensure that all the contents of /var are going to be copied to the new partition with no issue, without entering single user mode first.

Also, as far as I understand there is really no way to do this without at least some minutes of downtime depending the size of /var, if I’m wrong about that please let me know what is the proper way to do it in order to have no or at least very little downtime.

Thank you everyone for your time.

Update and solution:

In the following section I am describing step by step the procedure that I ended up following for moving the /var directory on its own partition, while trying the same time to limit the downtime as much as possible.

  • First thing I did was to mount the new partition on /mnt and do an initial pass with rsync, in order to copy the majority of the data before attempting to stop any processes that where using /var. The idea behind this, was that when the time comes to stop things like apache or mysql and the downtime starts, most of the data will have already been copied so the final rsync pass will finish very fast.

    sudo rsync -aAXv /var/ /mnt/ 
  • After rsync was done I got the following warning,

    rsync warning: some files vanished before they could be transferred (code 24) at main.c(1183) [sender=3.1.1] 

    so I decided to immediately run rsync once again and this time got no error.

  • I ran the following command in order to get a list of all the processes that were currently using /var,

    sudo lsof -n | grep /var | grep -v /var/run | cut -f1 -d ' ' | sort | uniq 
  • Then I stopped all the services that were not absolutely essential in order for the web server to function normally, I did a third pass with rsync and this time it took about a couple of minutes.

  • I ran the lsof command once again and stopped all the services that were using /var, that was the point where the downtime started.

  • I did then a fourth pass with rsync, this time it took only a few seconds.

  • Then I umounted /mnt, moved /var to /var_old, created a new /var and mounted the new partition on /var.

  • Finally I verified that the contents of /var were as they should be, edited /etc/fstab so the new /var will mount on boot and finally rebooted the machine.

At this point I have checked everything and all seem to function as they should. I haven't deleted /var_old yet just in case I missed something, but I will probably in a day after I will be 100% sure that everything functions the same as before.

Notes:

A mistake I made was that I tried to swap the old /var with a new partition using sudo instead of logging in as root. The problem with that, was that the moment I ran the sudo mkdir /var command in order to recreate an empty /var, the sudo program then also created a /var/lib/sudo directory with temporary files. So, at that point I logged in as root using the su command, deleted /var/lib and from there I did the rest.

This procedure was done on a VPS, which means that in case everything went horribly wrong and the SSH access was lost in the middle of the procedure, I had the option to reboot the server from the control panel. In case thought this was a dedicated server instead of a VPS, /u/zaffle gave a nice tip which may come in handy. Before attempting to stop any services, first make sure to run shutdown -r +30, so in case something goes wrong and you lose SSH access the server will autocratically reboot after 30 minutes.

Probably the reboot at the end wasn't absolutely required, as it is possible to simply restart all the services that where stopped before the final rsync pass. But since my server typicly takes less than a couple of minutes to reboot then it was no big deal. Also, I wanted to make sure that the changes I made in /etc/fstab was correct and indeed the new partition of /var mounts on correctly on boot. Finally, in case something went wrong and after that the server had issues with booting better to know now so I can fix it, instead of the next time that I have to reboot the server for some other reason and get a nasty surprise.

Just for the record, the total downtime I managed to achieve was less than 5 minutes.

submitted by /u/magkopian
[link] [comments]

Viewing all articles
Browse latest Browse all 17905

Trending Articles