We've Been Hacked... or Have We? 65
hidden_fire asks: "I recently got a job as a Web Programmer at a web company that hosts many sites. The company had many badly firewalled Windows and Linux servers without any security patches, and a shared administrator password. I warned them that they needed to improve their security, but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised, and the Sasser Worm took us offline. Now, I've been allowed to rebuild the compromised box and tighten our firewalling, but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, a Linux kernel lockup, strange ports being open, and performance issues. I think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security. Has anybody else been in this situation? How can I detect these guys if their tools don't show in virus scans?"
Some tips for seeing what's going on... (Score:4, Informative)
Linux: nmap the box from a trusted PC on the same network, and then build a copy of netstat on the trusted PC for the server in question... copy the binary over, and run "netstat -pultw" as root... it'll list all ports that are listening for connections, and, the processes that are opening them (by PID, and usually by name). The reason for the clean copy is that a lot of root kits replace netstat on infection...
Hope this helps...
tcpdump/snort/ethereal/any sniffer you can use (Score:4, Informative)
Ultimately you're going to need to build up some pretty decent filters or you'll just overwhelm the machine doing the sniffing, but if you know what traffic should look like going across your firewall, you can look for the anomalies quite easily.
Probably the easiest way for you to do this would be to use something like Snort with some pretty decent rules. The downside to IDS is that it takes a lot of effort to get the rules setup properly for your network, but that could easily be done in your spare time, off the clock as it were.
Virus scanners aren't always the best solution to finding back doors and such. On Windows there sure to be malware out there that just hasn't been noticed yet because it doesn't do anything overtly "virus-like". Trojans and malware like NetBus or BackOrifice (to use two very well known examples) are easily altered to hide from many AV apps (just that no one bothers anymore).
Because of this, any machine you think is compromised should be rebuilt and patched up from scratch. Once you've got it rebuilt, and before you put it back on the network, use Tripwire (or even a shell script) to take a baseline MD5 hash of every single windows and application file on your harddrive. This has two advantages. 1) You build up a list of known good files on your machine. 2) It becomes easy to spot new/added files and investigate them. You can do this on any platform, *nix or Windows.
To get the evidence you want though, you're going to need to use tcpdump/snort/ethereal (or any other sniffer you are comfortable with) to prove there is traffic going across your [firewall|egress routers] that shouldn't be.
Re:Some tips for seeing what's going on... (Score:4, Informative)
Under Linux, boot from a read-only media (rescue cd) and verify the md5sums of the files installed against the installation cds. Any binaries that have changed will stick out like a sore thumb. Debian (and other
I don't know if there's anything similar for Windows, but if there is, it probably isn't free
mystery worm out there (Score:5, Informative)
These compromised IIS servers often have the server attached to the explorer.exe process and are therefore not detectable by virus scans. Using netstat or filemon you can find the open ports. The only solution is a bare-metal rebuild.
Have fun if that is what you are dealing with.
As an aside, if this company is unconcerned about the compromise of credit card information you might want to find a new place to hang your hat. The civil and criminal liabilities are pretty steep for the compromise of financial transaction information (if you are in the U.S.) and they extend to individuals inside the company, not just the board and officers.
Re:Some tips for seeing what's going on... (Score:4, Informative)
"netstat -n -a | more" will tell you what ports are in use. This is a simple preliminary check. It will give you an idea of what to look for.
Fport [slashdot.org] is a great little tool that will tell you what processes are listening on what ports. It's many times better than netstat. This will likely give most trojans. Look for ports you don't recognize, and programs you don't recognize. Keep in mind that fport lists outgoing connections as well as incoming ones, and doesn't differentiate. Any ports or programs you don't recognize, google it and figure out what it is, how it got there, etc.
Next, check the standard startup locations. HKLM/Software/Microsoft/Windows/CurrentVersion/Ru
and the same under HCU/Software/Microsoft/Windows/CurrentVersion/Run
Also, look for hidden directories, and large files. You may be hosting an FTP Warez dump. Look for *.nfo;*.rar;*.ace;*.0*;*gamez*;*appz* and anything else you can think of.
Unfortunately, I don't know of any Md5 sum tools or anything for windows.
Finally, rebuild, if you can. Rebuild from current data, and known good code. Don't trust code on the compromised machine. Best practice for recovering from a compromise type stuff. That really should be your first, last, and only step, but I doubt you'll be allowed, considering that your boss isn't taking security seriously.
Re:Some tips for seeing what's going on... (Score:3, Informative)
The only real way to bypass it is from outside the compromised kernel, which means either a clean boot off of a boot CD or such (which would be pointless for netstat monitoring because the system should be clean at that point anyway) or a view of the net traffic from an external clean machine.
I would recommend the external sniffer method. I would use ethereal and/or tcpdump on a clean machine to monitor traffic to the suspected compromised machine, and filter out "normal" traffic. (of course this will not detect a backdoor that is piggybacked ontop of another valid protocol such as HTTP if you are filtering out "normal" traffic, but you gotta start somewhere...)
Of course, if the hacker were really slick, he might even flash some malicious code into the BIOS so it could survive reboots and even reinstalls, but in reality, how many really good hackers are out there? (Just kidding, but not about the BIOS idea!) Of the rootkits I've seen so far, adore is one of the best.
Re:Some tips for seeing what's going on... (Score:3, Informative)
http://www.blisstonia.com/shareware/Win
WinMD5 [blisstonia.com]
This works very well and it's simple.
Re:How to spot what is happening (Score:4, Informative)
A few other comments:
Virus scanners won't help on jot against a custom hack (as Valve found out, for instance). They can be helpful, but don't put full reliance on them.
Running an Intrustion Detection/Prevention System such as Snort, Samhain, Prelude, etc. will help you manage the monitoring side of things; more than a few machines becomes a pain without additional help. Also take a look centralising all your logs on a syslogng server or something similar, if you don't already (note that there are various solutions out there to get Windows boxes to log to a syslog server).
A honeypot may distract the hacker from your production servers for long enough for you to identify that there's a problem.
Also take a look at "HoneyTokens": specifically created database records that trigger alarms if they're accessed - usually high profile fictious targets that would make excellent trophy hacks - there's more info on this over at SecurityFocus [securityfocus.com].
If you suspect that a machine has been compromised, as other have said, the ONLY WAY TO BE SURE is to rebuild the box from scratch. While this may be a real pain, hopefully it'll help you get the procedures in place to make this as painless as possible, so it's not all bad.
Perform security audits/pentests every now and again. Tools like Nessus [nessus.org] help: here's a good series on using Nessus [securityfocus.com] (part 2 [securityfocus.com], part 3 [securityfocus.com]).
Get familiar with security tools such as the top 75 recommendations at Insecure.org [insecure.org] (home of Nmap).
Remember that security is a PROCESS, so be thorough; get an entire plan together and cover all the bases that you can, taking special care to identify and cover the weak points. Your company's security is only as good as its weakest link; for instance, priviledge escalation of weak user account passwords is a good one.
Read SecurityFocus [securityfocus.com], PacketStorm [packetstormsecurity.org], CERT [cert.org] and the like, and try to get involved in their communities; they can be invaluable! They're also got a lot of good tutorials, such as how to lock down Apache, IIS; securing PHP, ASP; etc.