Geoff Holden

Well, I’ve spent the last couple of hours trying to investigate a break-in on my home server.

I had noticed that all of a sudden I was getting a lot (like 20 a minute) of ‘undeliverable mail’ errors in my inbox. I looked at one and saw that it was an eBay phishing scam, and the mail was originating from my server!

So I logged in to the server and saw an account active that would never normally be (the account was created only for Samba usage). After killing the processes spawned by the account in question, the user logged back in, and started taunting me with the fact that he/she had gained root access. At this point I shut down the server such that it couldn’t cause any more harm. Before the server managed to shut down, the malicious user did manage to delete all of the files belonging to the account and attempted to make the server unusable–luckily I do nightly backups of all users’ files, and the server had powered off before any other damage was done. (Thank you Gentoo for suggesting that /boot be a seperate partition, and left unmounted)

After disconnecting the network from the computer and starting it back up, I started to sift through the log files. Turns out the account was comprimised a few days ago after trying several unsuccessful attempts to break the root password they started to go through common first names trying for an account having the password the same as the account name. After going through the webserver logs it looks like I had managed to shut everything down before anyone had fallen for the phishing scam. This user also didn’t get to delete their .bash_history file before the server shut down, so I was able to find out some more information about what the user was trying to do. I found evidence of them trying to install some IRC bots, and shutting down a kernel exploit (turns out I was running a 2.4 series kernel circa December ‘04). I also saw that he/she was poking around in some of the files I had left in /tmp (using pico of all things… I’m almost ashamed to admit that I even had that editor installed, vi is the only sensible text editor). After looking at the mail log around 1,000 messages were sent in 8 minutes before my ISP cut off my outgoing mail account for the day (which is good… if only all service providers were trying to be part of the solution, rather than the problem). But even if I hadn’t been cut off, I was lucky enough to have caught this one and had the server shut down around 2 minutes after that, which would’ve meant another 200 emails would’ve gotten out.

So, after finding all of that, and cleaning up the other remnants of the intrusion, the server is almost ready to go back online–but with a few enhanced security measures:

• All passwords are now of ‘acceptable’ strength
• The firewall limits SSH & FTP connections to a select few subnets
• Accounts that only use Samba can no longer get a shell on the server
• The kernel’s been upgraded to a more recent version

Update (June 23, 10:12am): A good article about this sort of thing in the NY Times just popped up on digg.

“A successful phishing operation might bring in thousands of fresh account numbers, along with other identifying details: names, addresses, phone numbers, passwords, PIN’s, and mothers’ maiden names. The richer the detail (and the higher the account balance), the better the asking price.”

Update (June 23, 11:55pm): I’ve also since noticed that the intruder managed to wipe out most of my MP3 collection, but once again those files are subject to the same backup procedures and can be safely restored from DVD.