I work as a sys admin, but I mostly do operational support for various applications that we have running. Most of this home built and is fucking ancient. The problem is that with constant system maintenance or failovers there are a lot of manual steps that have to be completed. System management is pretty compartmentalized as well and I have little power to change it. I don't have root, so things like Puppet are out of the question for now.
Here is an example of the problem, there is OS patching or DB patching (frequent because of PCI) and we have to stop a bunch of systems and services, cron out various tasks, etc... Then after reboots we have to bring everything back online.
Most of these systems are distributed, so order matters and it's impossible to have a shell script locally that does everything you need or start everything at system startup.
I've been thinking of a good way to automate this. The latest idea is to create a python based REST API service on each machine that would inventory certain things like cron job, running processes, etc... REST API would provide status on these things, but also let me control them remotely.
I would for example be able to run a script from one host that would ping REST APIs on various nodes and shut down stuff in order and cron out various jobs. I know I can do this with a bunch of shell scripts and ssh, but this just seems flimsy to me. Naturally security would be baked in and all that good stuff.
The question is, do you know of anything like this already?
P.S. I know this would not be a problem if the systems were designed correctly, but there is not much I can do about this fucking old tech stack here.
[link][12 comments]