<TL;DR> I got hacked- what do?
Early Jan, I noticed that there were entries in /var/log/auth.log that indicated that I (my non-root user account 'M') was logging in via SSH from IP's in Europe(Romania, Italy, Spain, and a few others). I took this as a sign that I had been hacked. I changed the password on the 'M' account and removed it from /etc/sudoers and all other security groups.
Since then, I've monitored my logs MUCH more closely and have seen (and continue to see) many -now failed attempts to keep using the 'M' account. Also, nation by nation by continent, I've started blocking the IP ranges from all but N.America and specific exceptions on my firewall appliance (separate device). While this cut down on observed attempts to near 0, I still see the odd attempt to login(SSH) the user 'M' from within the US.
Further I've found two abuse complaints that indicate that my static IP has been used to originate spam/phishing emails. So this seems to be the reason I was hacked --to set up a phising mail zombie server. However, I can find NO EVIDENCE of a mail server on the hacked box nor any other on my internal/home network. So- I've now blocked SMTP, SMTP_Secure, IMAP, IMAP_Secure from originating within my network and reaching outside.
Question Time:
1) How is the bad guy successfully originating mail from my system with no MTA installed? --What do I look for to find/classify/remove this?
2) While SSH'd, there doesn't seem to be a well maintained history of commands run by the logged in user. --Is there simple/direct method to log ALL command line activity?
More things I've done:
[] No additional user accounts or groups have been created. (as compared with a generic/fresh Debian(7.3) install)
[] There's always a certain noise level of failed login attempts from root, admin, etc.. I consider these low level threat/noise and Fail2Ban doesn't let them much further than 3 failed attempts in 90minutes.
[] I don't know how to answer: How did they get my PW in the first place? I've scanned every box in the house (tablets, servers, desktops, laptops, phones(android)) for key loggers and have found nothing. Plus no attempts to connect with the new account I've created to replace 'M'.
[] I don't think that this is 'script kitty' activity since the amount of login attempts went UP after I changed the password. This feels coordinated. But I don't have knowledge/proof to back that up.
Thanks for reading.. thanks for any insight/advice you might have for me as I learn from this mistake.
EDIT: Spelling, formatting
[link][19 comments]