I have just taken over over hundreds of Virtual servers in a vSphere environment and been asked to clean house an consolidate. It's safe to say that at least 30% of these have not been used and are not needed. How did things get this way?
They have never had any sort of centralized... anything. They had several admins who could create, clone, and move servers around for the last 2 years. Some of these were developers on various experiments and projects (some which may be expired), some of them contractors, and former employees. So what I got was a ton of poorly labeled servers in various states in development, testing, and production. Almost nothing is documented, although from their IPs I have been able to find various tickets where former admins were asked to create something for some group.
Things I already removed: machines powered off, machines with no network capability, machines in a state of degradation where they could not be used (bad disk sectors, kernel panics on boot, in a VLAN no longer in existence, etc).
Now I am ready to tackle some of the less obvious orphans. Many have uptimes in the hundreds of days. Most are CentOS 5, some Debian 6.
- Those with web servers, I am going to check their traffic logs and network connections.
- Those with database servers, I am going to check their network connections.
- Check lastlog and compare it to sudoers and /etc/passwd
- Check date stamps of various /home directories, and see what is in them. Sadly, a majority of these people never created more than a root user.
- I'd check secure and messages, but most have been long since lost with logrotate, so my decision would have to be based on 5 weeks of data and no further back.
I have been authorized that if I have a serious doubt to "shut it off and see who complains" especially in the dev and testing space. Then, after 30 days, if no one says anything, I can delete them.
I am just soliciting other clever, creative, or funny ideas from Reddit Linux admins and their experiences.
[link] [25 comments]