We've Been Hacked... or Have We? 65
hidden_fire asks: "I recently got a job as a Web Programmer at a web company that hosts many sites. The company had many badly firewalled Windows and Linux servers without any security patches, and a shared administrator password. I warned them that they needed to improve their security, but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised, and the Sasser Worm took us offline. Now, I've been allowed to rebuild the compromised box and tighten our firewalling, but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, a Linux kernel lockup, strange ports being open, and performance issues. I think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security. Has anybody else been in this situation? How can I detect these guys if their tools don't show in virus scans?"
One of the first rules... (Score:5, Insightful)
Sounds like (Score:3, Insightful)
Lesson to learn is secure that stuff, what they don't know will hurt them.
http://www.programming-reviews.com/Cuckoos_Egg_Tr
Re:Sounds like (Score:5, Funny)
Re:Sounds like (Score:2, Funny)
Re:Sounds like (Score:1)
Some tips for seeing what's going on... (Score:4, Informative)
Linux: nmap the box from a trusted PC on the same network, and then build a copy of netstat on the trusted PC for the server in question... copy the binary over, and run "netstat -pultw" as root... it'll list all ports that are listening for connections, and, the processes that are opening them (by PID, and usually by name). The reason for the clean copy is that a lot of root kits replace netstat on infection...
Hope this helps...
Re:Some tips for seeing what's going on... (Score:4, Informative)
Under Linux, boot from a read-only media (rescue cd) and verify the md5sums of the files installed against the installation cds. Any binaries that have changed will stick out like a sore thumb. Debian (and other
I don't know if there's anything similar for Windows, but if there is, it probably isn't free
Re:Some tips for seeing what's going on... (Score:4, Informative)
"netstat -n -a | more" will tell you what ports are in use. This is a simple preliminary check. It will give you an idea of what to look for.
Fport [slashdot.org] is a great little tool that will tell you what processes are listening on what ports. It's many times better than netstat. This will likely give most trojans. Look for ports you don't recognize, and programs you don't recognize. Keep in mind that fport lists outgoing connections as well as incoming ones, and doesn't differentiate. Any ports or programs you don't recognize, google it and figure out what it is, how it got there, etc.
Next, check the standard startup locations. HKLM/Software/Microsoft/Windows/CurrentVersion/Ru
and the same under HCU/Software/Microsoft/Windows/CurrentVersion/Run
Also, look for hidden directories, and large files. You may be hosting an FTP Warez dump. Look for *.nfo;*.rar;*.ace;*.0*;*gamez*;*appz* and anything else you can think of.
Unfortunately, I don't know of any Md5 sum tools or anything for windows.
Finally, rebuild, if you can. Rebuild from current data, and known good code. Don't trust code on the compromised machine. Best practice for recovering from a compromise type stuff. That really should be your first, last, and only step, but I doubt you'll be allowed, considering that your boss isn't taking security seriously.
Re:Some tips for seeing what's going on... (Score:3, Informative)
http://www.blisstonia.com/shareware/Win
WinMD5 [blisstonia.com]
This works very well and it's simple.
Re:Some tips for seeing what's going on... (Score:2)
Re:Some tips for seeing what's going on... (Score:3, Informative)
The only real way to bypass it is from outside the compromised kernel, which means either a clean boot off of a boot CD or such (which would be p
How to spot what is happening (Score:3, Interesting)
Here's a couple of things you could do:
Download and build chkrootkit [chkrootkit.org]. This will detect a lot (most?) stealthed kits on Linux systems, and it is always my first port of call when I'm invited in to clean up after a breakin.
Plug in a hub (so all traffic can be seen by multiple machines - a switch ain't as good, unless it has a monitoring port) in front of the machine(s) and run tcpdump or ethereal on another system to watch traffic from the machine. This will let you watch exactly what traffic is happening on those weird ports, or watch outbound SMTP traffic for spammer activity.
We don't put Windows-based systems on the internet, partly for security reasons, and partly because we don't have any Windows specialists, so I can't help for on-the-box detection there, although I would expect a commercial virus scanner should find everything.
Re:How to spot what is happening (Score:4, Informative)
A few other comments:
Virus scanners won't help on jot against a custom hack (as Valve found out, for instance). They can be helpful, but don't put full reliance on them.
Running an Intrustion Detection/Prevention System such as Snort, Samhain, Prelude, etc. will help you manage the monitoring side of things; more than a few machines becomes a pain without additional help. Also take a look centralising all your logs on a syslogng server or something similar, if you don't already (note that there are various solutions out there to get Windows boxes to log to a syslog server).
A honeypot may distract the hacker from your production servers for long enough for you to identify that there's a problem.
Also take a look at "HoneyTokens": specifically created database records that trigger alarms if they're accessed - usually high profile fictious targets that would make excellent trophy hacks - there's more info on this over at SecurityFocus [securityfocus.com].
If you suspect that a machine has been compromised, as other have said, the ONLY WAY TO BE SURE is to rebuild the box from scratch. While this may be a real pain, hopefully it'll help you get the procedures in place to make this as painless as possible, so it's not all bad.
Perform security audits/pentests every now and again. Tools like Nessus [nessus.org] help: here's a good series on using Nessus [securityfocus.com] (part 2 [securityfocus.com], part 3 [securityfocus.com]).
Get familiar with security tools such as the top 75 recommendations at Insecure.org [insecure.org] (home of Nmap).
Remember that security is a PROCESS, so be thorough; get an entire plan together and cover all the bases that you can, taking special care to identify and cover the weak points. Your company's security is only as good as its weakest link; for instance, priviledge escalation of weak user account passwords is a good one.
Read SecurityFocus [securityfocus.com], PacketStorm [packetstormsecurity.org], CERT [cert.org] and the like, and try to get involved in their communities; they can be invaluable! They're also got a lot of good tutorials, such as how to lock down Apache, IIS; securing PHP, ASP; etc.
Would Firewall Analytics Help? (Score:1)
tcpdump/snort/ethereal/any sniffer you can use (Score:4, Informative)
Ultimately you're going to need to build up some pretty decent filters or you'll just overwhelm the machine doing the sniffing, but if you know what traffic should look like going across your firewall, you can look for the anomalies quite easily.
Probably the easiest way for you to do this would be to use something like Snort with some pretty decent rules. The downside to IDS is that it takes a lot of effort to get the rules setup properly for your network, but that could easily be done in your spare time, off the clock as it were.
Virus scanners aren't always the best solution to finding back doors and such. On Windows there sure to be malware out there that just hasn't been noticed yet because it doesn't do anything overtly "virus-like". Trojans and malware like NetBus or BackOrifice (to use two very well known examples) are easily altered to hide from many AV apps (just that no one bothers anymore).
Because of this, any machine you think is compromised should be rebuilt and patched up from scratch. Once you've got it rebuilt, and before you put it back on the network, use Tripwire (or even a shell script) to take a baseline MD5 hash of every single windows and application file on your harddrive. This has two advantages. 1) You build up a list of known good files on your machine. 2) It becomes easy to spot new/added files and investigate them. You can do this on any platform, *nix or Windows.
To get the evidence you want though, you're going to need to use tcpdump/snort/ethereal (or any other sniffer you are comfortable with) to prove there is traffic going across your [firewall|egress routers] that shouldn't be.
OT: Your Sig (Score:1)
Re:OT: Your Sig (Score:2)
----8----
Hall herself claimed later that she had been paraphrasing Voltaire's words in his Essay on Tolerance:
"Think for yourselves and let others enjoy the privilege to do so too."
Hall died in 1919.
In his A Book of French Quotations (1963), Norbert Guterman suggested that the probable source for the quotation was a line in a 6 February 1770 letter to M. le Riche: "Monsieur l'abbé, I detest what you write, but I would give my life to make it possible f
Re:OT: Your Sig (Score:2)
Er, what do you mean "strange"? (Score:2)
but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, ... strange ports being open...
Perhaps I'm being naive, but what's preventing you from using 'netstat', 'nmap -sV', and plain old 'netcat' to figure out what the strange ports are doing?
Re:Er, what do you mean "strange"? (Score:2)
but nmap and netcat would be fine. from elsewhere. maybe nessus too, while you're at the old security console.
Re:Er, what do you mean "strange"? (Score:2)
In theory, once a root-level compromise has occurred, you can trust nothing a computer says anymore. Crackers have, through steady effort, made that almost 100% true, and if they aren't there already, they don't have far to go.
No binary can be trusted on a hacked system.
For the curious, I recomend Googling "rootkit"; there are a lot of helpful resources out there.
mystery worm out there (Score:5, Informative)
These compromised IIS servers often have the server attached to the explorer.exe process and are therefore not detectable by virus scans. Using netstat or filemon you can find the open ports. The only solution is a bare-metal rebuild.
Have fun if that is what you are dealing with.
As an aside, if this company is unconcerned about the compromise of credit card information you might want to find a new place to hang your hat. The civil and criminal liabilities are pretty steep for the compromise of financial transaction information (if you are in the U.S.) and they extend to individuals inside the company, not just the board and officers.
Re:mystery worm out there (Score:1)
But by posting this question to /., and by being able to show a record of his attempts at fixing the problem, he should be off the hook. He can point to this story as a time-stamped description of his stopped efforts. The the blame would shift to his boss.
But I could be wrong.
Re:mystery worm out there (Score:2)
Re:mystery worm out there (Score:1)
I am not a lawyer, but I play one on Slashdot!
But I could be wrong.
Yup.
Re:mystery worm out there (Score:2)
If they're in Russia, and making money illegally, and not owned by organized crime -- they will be soon.
Tell ya what... (Score:5, Funny)
Re:Tell ya what... (Score:4, Funny)
Re:Tell ya what... (Score:2)
Hire an expert. (Score:4, Insightful)
My company has an outside security company run quarterly checks against our network, and they sometimes catch stuff that I miss. Just don't let them talk you into buying a over-priced checkpoint firewall when all you need is a Linux box and Iptables.
--Ajay
Re:Hire an expert. (Score:2)
Just make sure that if he does nothing it is clear that he was warned and that you tried to do the right thing. Then make a copy of the memo for yourself before giving it to him.
Finally, follow your boss's orders and get back to web progr
Do your job? (Score:5, Insightful)
Re:Do your job? (Score:4, Insightful)
Re:Do your job? (Score:3, Insightful)
Welcome to the wonderful world of corporate webdev (Score:5, Insightful)
You're already doing your job, and not being listened to. Since I'm not a sysadmin, I've got no direct advice for you regarding the tracking of such activity. However, it seems to me that this is the smaller of two problems. The first is being able to do your job from a technical perspective. The second (and it seems, more immediate) problem is being able to do your job from a political perspective.
Your boss has already watched his public facing site(s) and servers go down due to his failure to listen to you. Now it sounds like he's about to make the same fatal mistake. This, of course, places you in the lovely position of having to remind him that he's about to make another major tactical error... but you also have to do so in as subtle manner as possible, so it doesn't sound like a recrimination. All I can suggest is to try to make it blatantly obvious to him without coming out and saying it, thus giving him the opportunity to "discover" his error and correct it on his own.
One other suggestion: document, document, DOCUMENT! Make sure that you can prove later (should it be necessary) that you did everything you could. This is another area of vital importance for your job security that also must be done very carefully. Simply CCing the higher-ups will likely piss off your boss (and possibly the folks you're CCing as well), and may look like unnecessary whistle-blowing or complaining. Do it as unobtrusively as possible, but make sure that you're covered, in case there's any question later.
One possible solution to both problems is to communicate all of your concerns in an e-mail. Write it during your lunch hour, so he can't get upset at you for "wasting" more company time on it. Make it clear exactly why you think there is an issue, and mention (if you can do so gently and without provoking him) the past incident. Remember that if you want someone to do something, you shouldn't tell them why *you* want them to do it. They don't care about what you want. Tell them why *they* want to do it. Best-case scenario, he listens. Worst-case scenario, he gets a little more annoyed at you, but you've got your documentation.
I don't envy your position, and wish you luck.
Run, don't walk. (Score:5, Insightful)
On the other hand, if they continue as they are, they may not survive, and you are screwed again.
Re:Run, don't walk. (Score:2)
Watch out (Score:4, Interesting)
The worst case scenario here is that you detect a problem, attempt to fix it yourself, and trigger Something Bad[tm] in the process: the cracker retaliates, or you break a working app because you upgraded something out-of-sync with glibc (or whatever), or you otherwise become the catalyst for noticeable downtime that will piss off your boss and get you fired, or worse - they turn you into a scapegoat (see the Intel case against that security chap.)
Just make sure you cover your ass. You've notified your boss, copy those emails to a nice safe place (headers and all), and don't do anything stupid.
Best scenario is to build a fresh box, backup the old box's data, restore it to the new box (clean! no code! only data!) Don't bother trying to salvage a compromised O/S installation. Too many things to miss. And, when you're building fresh, don't ssh via one of the infected boxes! Don't inadvertently give *any* info to the crackers that you're setting up a new machine. Better yet, build it with the ethernet cable unplugged, if possible. Do it from CDs.
Sounds like a breakin... (Score:2, Insightful)
Also, one good thing to do is place a clean box in between the comprimised server and its internet connect and run tcpdump/tethereal on the brided connection. The first thing you need to do is be able to identify _all_ traffic go
If youre boss is not a techie (Score:4, Insightful)
You'll be given all the time and budget to fix it. FUD doesnt always require proof, unless someone calls in some consultant.
Resume time (Score:2)
think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security.
You've got a money hemorrage on your servers. Don't bother trying to talk sense to your boss, just strengthen up your arms and be read to jump ship and swim when the next boat comes along. Start shopping your resume now, and remember: when everyone is out to get you, paranoia is just good sense.
Sue them. (Score:1)
Re:Sue them. (Score:2)
And therein lies the problem. As a consumer, there are thousands of companies I could buy from online, but as long as security is given such low priority, how can I trust any of them?
Let the boss know... (Score:2)
Tell him that keeping your servers secure IS paying work. As a customer, if I know that a company is not keeping my credit cards, etc. secure, I will not do business with them. Ask him what would happen if there was a big 'scandal' (for lack of better terms), and thousands of customers credit was exploited. How much of an opp
Sounds Kinda like my job.....LOL (Score:2, Interesting)
At my job, I am one of two web developers. Besides us, there are the two owners and our systems admin. The owners want to become a viable commercial hosting service with secure storefronts, etc. Fine says me.
The problem lies in that one of the two owners (The husband) is a pig-headed idiot. Recently he asked us to implement a RAID solution for the webserver (notice the lack of an 's' at the end of webserver).
Re:Sounds Kinda like my job.....LOL (Score:1)
What I'm actually surprised at is that three established guys (for which you say have 40+ years of experience between you.. hope you're not counting high school computer classes
Re:Sounds Kinda like my job.....LOL (Score:3, Insightful)
Maybe ask for an IT budget whereby you and the others get a fixed amount of cash to spend on hardware, since he's obviously paranoid about spending (forgetting that he pays you tons more to deal with the crap he buys) and wants to avoid getting something more expensive than it needs to be, and you want to avo
Re:data managers (Score:3, Insightful)
What I said to one guy like that is "Sir, we respect you and will do whatever you want us to do here, because you're the boss and it's your company, but you hired us to take the load off you so you'd be able to do less work and make more money. Trust us to do a good job and we will. We might not do everything exactly the same as you w
Re:Sounds Kinda like my job.....LOL (Score:1)
The longer you stay under these conditions the worse your reputation will be. The money you are making had better be enough to cover personal attacks on you by your boss that may haunt you in the future.
Nuke/Pave (Score:2)
simple, ntop (Score:2)
Forget running ethereal or other packet sniffers, they're far too fine-grained for what you're trying to do: prove they're being abused.
Connect a small box running your favorite *nix running ntop [ntop.org] to the service port (or whatever they call it, I'm half asleep) of the switch/router to which the box(es) in question are connected.
That's it.
Ntop will give you very nice graphs to print out for the guy who needs a clue, showing not only the IPs involved, but the ports, percentages of traffic per protocal/port/w
Security is Paying Work (Score:2)
Security is paying work, unless your company has a reputation without worth. You can get that sentiment out of any of the good security books out there, but saddly management can only see the spending cycle, not the averted crash-burn-patch-fix cycle. Remind him that if we were all "paranoid" to begin with, we wouldn't have had a hacker tell us to patch our stuff. What if the hacker didn't like you so much?
Security really isn't a game or a buzz word. Folks have to own that concept, not just parrot it.
You are a web programmer. (Score:1)
So my best advise to you is that if it's not in your job description, leave it the fuc
Re:You are a web programmer. (Score:1)
If you find a problem, whether it's security or something else, notify the appropriate people (CC your boss). If they chose not to fix it, that's their problem. At least your a** will be clean if/when there's legal action resulting from the issue.
This sounds like a job for ... (Score:2)
But seriously, get a Snort box installed and be more active(*) in your intrusion detection. Surely your boss can't object if you slap down a printed snort long on his desk, and show him proof of intrusion.
dave
(*) I absolutely refuse to use the word 'proactive'. I'm not playing buzzword bingo here.
Nuke the site from orbit. (Score:1)
Also have a look at EtherApe (Score:2)
The main reason for that is that etherape actually maps out live traffic patterns (and volume) on a network. I personally am quite surprised it's not a default part of more security bootdisks. You'll spot an infection straightaway as they tend to 'broadcast' on the network.
As for protection, host checksumming has one disadvantage: it's AFT
warezzzzzzzzz (Score:1)