A few lessons learned from the MetaFilter hack
Jun 7, 2001 | Q

This is just a list of little things I learned while cleaning up after the MetaFilter hack. I don’t pretend that this list is comprehensive; instead, it’s a simple few things that you should think about before and after someone hacks into your network.

1. If a machine is hacked, pull it off the network immediately, and only put it back on the network once you can be sure it’s isolated from the rest of your internal network and from the Internet.

Any time you discover evidence of unauthorized access to a machine, you have to assume the worst — that the machine is not only hacked, but that it is collecting information from the rest of your network, that it is serving as a launching point for attacks on others, and that it will do actual damage to other computers if left to its own devices. Thus, as soon as you discover that it’s been hacked, yank the network cable until you can collect your senses and decide what to do.

In the case of the MetaFilter server, though, we had to get it back onto the network in order to do any diagnosis; it runs without a monitor, and in addition, Matt is in California, and could only examine it over the Internet. Because of this, the best option seemed to be throwing a few extra access lists onto the router that would limit access to the machine to Matt’s IP address and my own, and that would prevent the machine from being able to talk back to any other machine but those two. After those were installed, I was able to plug the machine back into the network and we could start sleuthing.

2. Always subscribe to the mailing list for security notices related to your operating system, and always read the security notices that you receive.

No matter which operating system you run (and, accordingly, how much you hate other operating systems), there will be security issues that come up. If you have a machine connected to the Internet, it’s your job to make sure that that machine is secure, both for your own benefit and for the benefit of those who won’t have to worry about your machine being used to launch an attack on theirs.

Here’s a group of pages which explain how to sign onto the lists covering most operating systems:

When security advisories are released, read them and make sure you understand whether or not they apply to your setup, what the risk is if you don’t apply the patch, and how to install them completely. If the advisory describes a threat to your setup, then act to neutralize the threat at the soonest opportunity available to you. (Remember, once you know about the bug, so do thousands of other people, and it’s only a matter of time before someone tries to use it against your machines.)

3. Use a firewall or router to limit access to your machines.

Just because a machine needs to provide services to the Internet at large doesn’t mean that it has to have every since service available for people to use and abuse. If you’re providing web services, then limit people’s ability to access your machine to just the port on which the webserver is running. If you’re running a mail server, limit people to just the ports necessary to send and receive mail.

Put access lists on your routers or firewalls that prevent people from spoofing into your network from the outside. This means that you should deny access through your router to any machine claiming to have an IP address that could only exist within your own network. Similarly, don’t allow machines into your network which carry private network IP addresses — they’re inevitably trying to spoof their way in, and avoid being traceable. (For example, right now, my access lists show, from the last 12 hours, 136 attempts from the 10.0.0.0 address space, 13 attempts from the 172.16.0.0 address space, and 131 attempts from the 192.168.0.0 address space.)

Remember, limiting access not only helps prevent hackers from installing trojans and other malicious software, it also helps prevent them from accessing their trojans and backdoors if they manage to get them installed. Most of these remote control apps set themselves up on a port and await contact or instruction from the person who set them up. If the hacker doesn’t have the ability to get through the firewall and contact that port, then you’re halfway to neutralizing the threat posed by that trojan.

4. Keep logs on your publicly-accessible machines, and archive the logs regularly.

One of the best ways to sleuth after-the-fact is to have reliable logs files. Frequently, you can get IP addresses from the logs; in addition, they may contain information that can help you figure out how the hacker broke into your machine. (Remember that the hacker could have very well modified the logs, though, either to erase traces of his visits, or to divert blame elsewhere.)

For example, if the person hacked into a webserver, the webserver logs may contain the URLs which they sent to gain access to the machine. (For the MetaFilter hack, the URLs contained a file which wasn’t originally part of the website.) If the person hacked into the machine through a mail server, the mail logs may have the headers that show the exploit used. Ad infinitum.

Likewise, though, you should regularly dump log files out to a removable medium or other storage system which isn’t accessible to the hackers during their uninvited stay on your machine. Otherwise, they can modify the logs to erase evidence of their visits, and you can lose valuable clues as to who they were, and how to both clean up after them and prevent their return.

5. Make a backup of any files you suspect were affected, before you go about cleaning the server up.

Having information like timestamps, access lists, and audit chains on changed files is invaluable. You want to make sure that your cleanup process doesn’t affect any of this before you have a chance to see it and figure out how it fits into the puzzle.

Likewise, you’ll want to pay attention to any possible applications which could modify the files you care aboutwithout your explicit acknowledgement. For example, a web-based mail app that I use deletes and restarts its log file every time that you start the service, so if I care to see the logs that it creates when it’s running, it’s important for me to know that I can’t stop and start the service without backing those log files up.

6. Know the security resources available to you on the Internet.

If you’re running a hackable server, then you should be aware that there are a lot of resources out there that can help you track down the answers to questions that come up when you’re trying to figure out what happened after the fact.

  • Sam Spade: possibly the best single resource out there. When you come across an IP address in your logs, plug it in here, and you’ll find out what network it’s on, the contact information for those who run the network, and the upstream Internet service providers. If you know how to read the reports, you can also figure out if the machine is still up, or if it’s protected behind a firewall or proxy.
  • RIPE Whois: unfortunately, many of the machines used as launching points for attacks are in Europe; RIPE is the organization that assigns IP addresses for the region, and its lookup facility will help you figure out who to contact about a break-in attempt.
  • Again, each operating system vendor maintains a database of technical information that’s generally both available to the public and searchable. Microsoft’s Support Knowledge Base and Security website are both pretty good, as is Apple’s Tech Info Library. I’ve never used Sun’s SunSolve Online, so I can’t comment on it. For Linux, you’re better off going to each vendor’s site (although Red Hat’s support knowledge base isn’t a bad place to start). Lastly, Cisco has the Technical Assistance Center, which is a good place to start if someone’s hacked into your Cisco router.
  • Google Groups: an archive of all the postings to Usenet, the public messaging network. When people get hacked, or when new hacks are released, they tend to be mentioned multiple times on Usenet; searching for information related to data you find on your hacked server can usually help you track down both descriptions of the problem and possible solutions.