So here's my deal. Support guy at a company with a small IT team. We have a mix of Windows and Linux servers, and no official sysadmin. Currently Linux admin duties are being split between two developers, both of whom would rather focus on coding. Since I'm the only one there who is keen on Linux, they've been delegating more and more of that to me, and want to give me full admin duties over the Linux part of the operation soon.
Here's what we've got. The only Windows servers are at HQ. There are Linux servers at a dozen branch sites on aging IBM hardware, mostly running Fedora Core 4 (EoL 2006, seriously) and one or two with Centos 4 (EoL 2012). Roles:
Application server for a severely outdated telnet-based ERP app, a separate instance of which handles ERP functions at most branch sites. Right now it's the company's most widely-used ERP system (there are a total of 5 in service, three of which should have been phased out 10 years ago). Server specs for this are minimal, and it could probably be run off a VM (or a Raspberrry Pi) easily. Bandwidth requirements for this app are also nil, but our WAN consists of horrid high-latency site-site internet VPN's. This thing also apparently only runs on RPM distros with severely outdated kernels. It's not going to be phased out any time soon, and I can't migrate it onto a Centos 7 server.
Web server for a web reporting interface used by the aforementioned ERP app.
CUPS server used for printing from said app.
SMB file server used for a lot of office files, for importing office docs into the web reporting frontend. Not LDAP integrated, users have to be configured individually or use common accounts. Some pretty important production equipment also run Java-based apps off these shares.
My first priority on these is a better disaster recovery procedure. When one went down it was a complete clusterfuck that I was fortunately not on the hook for. I noticed that the data and OS were all on a single partition set up as a striped logical volume across 3 HDD's. No RAID. Databases for web server and application servers are backed up daily but the shares are not, so those were lost. There were no spares on-site, so it had to be shipped back to HQ, configured from scratch, and shipped back. Each one is a time bomb waiting to lose the company tens of thousands.
Some other ideas:
Get backup VM's set up for the outdated app servers. They will be usable over WAN temporarily when another on-site server burns out. Switching all the users over currently consists of the rather labor intensive process of replacing config files and shortcuts for each of them, and I'm thinking a virtual interface on the site router with a static NAT mapping to the VM's could be brought online instead, so the users are connecting to the same IP's. If the network ever gets upgraded, I'll use VM's for production, and inside a DMZ back at HQ (or within a NAT-ed KVM VM with a more up to date distro as the main host OS on each site server).
Rig together some kind of DIY NAS on the cheap (e.g. an old desktop with a SAS/RAID controller card and FreeNAS), use it as a backup server for the SMB shares, have it as a warm spare SMB server when the original burns out.
Any other thoughts on how I could get this mess in order on a shoestring budget? A second server on-site isn't an option. Main problem is a shitty WAN connection that isn't getting upgraded anytime soon, ERP app that only runs on outdated kernel, and aging hardware with no on-site redundancy and backups for most of what it does.
Also open to any other tips about tools I should start using immediately as part of general admin duties. I'd like to deploy Nagios and Webmin for instance, and start redoing our network (my CCNA is getting rusty and the network is as much of a cluster as everything else).
[link][3 comments]