So, around two months ago my two bigger Cloud Linux KVM servers started to act weird.
With weird I mean something I haven't seen on Debian before. Somehow the server gets into a read / write disk lock and can't preform anything...
At first I thought it was a problem from my side where I misconfigured one server. So my boss decided to reconfigure that server and set up a second one... Same thing happens on that one too.
When looking at the logs and monitoring utility, a few minutes before the lock the CPU disk I/O skyrockets, and afterwards the load gets around 100-900 as the server cannot resolve the threads (nothing can be written to the disks).
I asked at the CL support forum as we're paying them for support, but till now even they don't know anything.
So I invoke the Hive mind to help me (I cast maximised quickened Hivemind with a 9th spell level slot)
The servers are set on a proxmox 3.4 hypervisor and both have a 6TB qcow2 disk image (for migration purposes, and due to the fact that I'm a n00b at virtualisation and just a month into the problem found about the lve disk capability for the kvm).
Both servers have 32 real cores, and 30 of these are dedicated to the kvm.
But nada.
Now it bugged me and I did a few tests.
I made a *sh script that acts like a myql data read/write. Basically it dds multiple 1-5mb files and then reads them randomly.
I did the same thing on debian, and had no problems, but I locked the cloudlinux test server I set up in a free proxmox node I had.
Now I did fix one server's problem by migrating all databases to an offsite percona cluster (finally, I was trying to push that for ages).
The only problem is the second server. We can't have offsite databases due to it being a cPanel shared hosting server.
What do? Anyone encountered simmilar problems or?
[link] [comments]