We've Been Hacked... or Have We? 65
hidden_fire asks: "I recently got a job as a Web Programmer at a web company that hosts many sites. The company had many badly firewalled Windows and Linux servers without any security patches, and a shared administrator password. I warned them that they needed to improve their security, but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised, and the Sasser Worm took us offline. Now, I've been allowed to rebuild the compromised box and tighten our firewalling, but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, a Linux kernel lockup, strange ports being open, and performance issues. I think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security. Has anybody else been in this situation? How can I detect these guys if their tools don't show in virus scans?"
One of the first rules... (Score:5, Insightful)
firost psot (Score:-1, Offtopic)
Re:firost psot (Score:-1, Troll)
Sounds like (Score:3, Insightful)
Lesson to learn is secure that stuff, what they don't know will hurt them.
http://www.programming-reviews.com/Cuckoos_Egg_Tr
Re:Sounds like (Score:5, Funny)
Re:Sounds like (Score:2, Funny)
Re:Sounds like (Score:1)
Some tips for seeing what's going on... (Score:4, Informative)
Linux: nmap the box from a trusted PC on the same network, and then build a copy of netstat on the trusted PC for the server in question... copy the binary over, and run "netstat -pultw" as root... it'll list all ports that are listening for connections, and, the processes that are opening them (by PID, and usually by name). The reason for the clean copy is that a lot of root kits replace netstat on infection...
Hope this helps...
Re:Some tips for seeing what's going on... (Score:4, Informative)
Under Linux, boot from a read-only media (rescue cd) and verify the md5sums of the files installed against the installation cds. Any binaries that have changed will stick out like a sore thumb. Debian (and other
I don't know if there's anything similar for Windows, but if there is, it probably isn't free
Re:Some tips for seeing what's going on... (Score:4, Informative)
"netstat -n -a | more" will tell you what ports are in use. This is a simple preliminary check. It will give you an idea of what to look for.
Fport [slashdot.org] is a great little tool that will tell you what processes are listening on what ports. It's many times better than netstat. This will likely give most trojans. Look for ports you don't recognize, and programs you don't recognize. Keep in mind that fport lists outgoing connections as well as incoming ones, and doesn't differentiate. Any ports or programs you don't recognize, google it and figure out what it is, how it got there, etc.
Next, check the standard startup locations. HKLM/Software/Microsoft/Windows/CurrentVersion/Ru
and the same under HCU/Software/Microsoft/Windows/CurrentVersion/Run
Also, look for hidden directories, and large files. You may be hosting an FTP Warez dump. Look for *.nfo;*.rar;*.ace;*.0*;*gamez*;*appz* and anything else you can think of.
Unfortunately, I don't know of any Md5 sum tools or anything for windows.
Finally, rebuild, if you can. Rebuild from current data, and known good code. Don't trust code on the compromised machine. Best practice for recovering from a compromise type stuff. That really should be your first, last, and only step, but I doubt you'll be allowed, considering that your boss isn't taking security seriously.
Re:Some tips for seeing what's going on... (Score:3, Informative)
http://www.blisstonia.com/shareware/Win
WinMD5 [blisstonia.com]
This works very well and it's simple.
Re:Some tips for seeing what's going on... (Score:2)
Re:Some tips for seeing what's going on... (Score:3, Informative)
The only real way to bypass it is from outside the compromised kernel, which means either a clean boot off of a boot CD or such (which would be pointless for netstat monitoring because the system should be clean at that point anyway) or a view of the net traffic from an external clean machine.
I would recommend the external sniffer method. I would use ethereal and/or tcpdump on a clean machine to monitor traffic to the suspected compromised machine, and filter out "normal" traffic. (of course this will not detect a backdoor that is piggybacked ontop of another valid protocol such as HTTP if you are filtering out "normal" traffic, but you gotta start somewhere...)
Of course, if the hacker were really slick, he might even flash some malicious code into the BIOS so it could survive reboots and even reinstalls, but in reality, how many really good hackers are out there? (Just kidding, but not about the BIOS idea!) Of the rootkits I've seen so far, adore is one of the best.
How to spot what is happening (Score:3, Interesting)
Here's a couple of things you could do:
Download and build chkrootkit [chkrootkit.org]. This will detect a lot (most?) stealthed kits on Linux systems, and it is always my first port of call when I'm invited in to clean up after a breakin.
Plug in a hub (so all traffic can be seen by multiple machines - a switch ain't as good, unless it has a monitoring port) in front of the machine(s) and run tcpdump or ethereal on another system to watch traffic from the machine. This will let you watch exactly what traffic is happening on those weird ports, or watch outbound SMTP traffic for spammer activity.
We don't put Windows-based systems on the internet, partly for security reasons, and partly because we don't have any Windows specialists, so I can't help for on-the-box detection there, although I would expect a commercial virus scanner should find everything.
Re:How to spot what is happening (Score:4, Informative)
A few other comments:
Virus scanners won't help on jot against a custom hack (as Valve found out, for instance). They can be helpful, but don't put full reliance on them.
Running an Intrustion Detection/Prevention System such as Snort, Samhain, Prelude, etc. will help you manage the monitoring side of things; more than a few machines becomes a pain without additional help. Also take a look centralising all your logs on a syslogng server or something similar, if you don't already (note that there are various solutions out there to get Windows boxes to log to a syslog server).
A honeypot may distract the hacker from your production servers for long enough for you to identify that there's a problem.
Also take a look at "HoneyTokens": specifically created database records that trigger alarms if they're accessed - usually high profile fictious targets that would make excellent trophy hacks - there's more info on this over at SecurityFocus [securityfocus.com].
If you suspect that a machine has been compromised, as other have said, the ONLY WAY TO BE SURE is to rebuild the box from scratch. While this may be a real pain, hopefully it'll help you get the procedures in place to make this as painless as possible, so it's not all bad.
Perform security audits/pentests every now and again. Tools like Nessus [nessus.org] help: here's a good series on using Nessus [securityfocus.com] (part 2 [securityfocus.com], part 3 [securityfocus.com]).
Get familiar with security tools such as the top 75 recommendations at Insecure.org [insecure.org] (home of Nmap).
Remember that security is a PROCESS, so be thorough; get an entire plan together and cover all the bases that you can, taking special care to identify and cover the weak points. Your company's security is only as good as its weakest link; for instance, priviledge escalation of weak user account passwords is a good one.
Read SecurityFocus [securityfocus.com], PacketStorm [packetstormsecurity.org], CERT [cert.org] and the like, and try to get involved in their communities; they can be invaluable! They're also got a lot of good tutorials, such as how to lock down Apache, IIS; securing PHP, ASP; etc.
Would Firewall Analytics Help? (Score:1)
tcpdump/snort/ethereal/any sniffer you can use (Score:4, Informative)
Ultimately you're going to need to build up some pretty decent filters or you'll just overwhelm the machine doing the sniffing, but if you know what traffic should look like going across your firewall, you can look for the anomalies quite easily.
Probably the easiest way for you to do this would be to use something like Snort with some pretty decent rules. The downside to IDS is that it takes a lot of effort to get the rules setup properly for your network, but that could easily be done in your spare time, off the clock as it were.
Virus scanners aren't always the best solution to finding back doors and such. On Windows there sure to be malware out there that just hasn't been noticed yet because it doesn't do anything overtly "virus-like". Trojans and malware like NetBus or BackOrifice (to use two very well known examples) are easily altered to hide from many AV apps (just that no one bothers anymore).
Because of this, any machine you think is compromised should be rebuilt and patched up from scratch. Once you've got it rebuilt, and before you put it back on the network, use Tripwire (or even a shell script) to take a baseline MD5 hash of every single windows and application file on your harddrive. This has two advantages. 1) You build up a list of known good files on your machine. 2) It becomes easy to spot new/added files and investigate them. You can do this on any platform, *nix or Windows.
To get the evidence you want though, you're going to need to use tcpdump/snort/ethereal (or any other sniffer you are comfortable with) to prove there is traffic going across your [firewall|egress routers] that shouldn't be.
OT: Your Sig (Score:1)
Re:OT: Your Sig (Score:0)
Hmm (Score: -1, Too Obvious).
Re:OT: Your Sig (Score:2)
----8----
Hall herself claimed later that she had been paraphrasing Voltaire's words in his Essay on Tolerance:
"Think for yourselves and let others enjoy the privilege to do so too."
Hall died in 1919.
In his A Book of French Quotations (1963), Norbert Guterman suggested that the probable source for the quotation was a line in a 6 February 1770 letter to M. le Riche: "Monsieur l'abbé, I detest what you write, but I would give my life to make it possible for you to continue to write."
----8----
http://www.classroomtools.com
Thanks for pointing this out. I had never even heard of Hall before this. However, in my defense, from that paper:
"In any case, the phrase was too eloquent, so it became quoted, and famous names attach themselves to quotes, to the detriment of the less well-known originators."
To be honest, I had completely forgotten I even had a sig setup as I've configured / to not show them.
Re:OT: Your Sig (Score:2)
Er, what do you mean "strange"? (Score:2)
but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, ... strange ports being open...
Perhaps I'm being naive, but what's preventing you from using 'netstat', 'nmap -sV', and plain old 'netcat' to figure out what the strange ports are doing?
Re:Er, what do you mean "strange"? (Score:2)
but nmap and netcat would be fine. from elsewhere. maybe nessus too, while you're at the old security console.
Re:Er, what do you mean "strange"? (Score:2)
In theory, once a root-level compromise has occurred, you can trust nothing a computer says anymore. Crackers have, through steady effort, made that almost 100% true, and if they aren't there already, they don't have far to go.
No binary can be trusted on a hacked system.
For the curious, I recomend Googling "rootkit"; there are a lot of helpful resources out there.
mystery worm out there (Score:5, Informative)
These compromised IIS servers often have the server attached to the explorer.exe process and are therefore not detectable by virus scans. Using netstat or filemon you can find the open ports. The only solution is a bare-metal rebuild.
Have fun if that is what you are dealing with.
As an aside, if this company is unconcerned about the compromise of credit card information you might want to find a new place to hang your hat. The civil and criminal liabilities are pretty steep for the compromise of financial transaction information (if you are in the U.S.) and they extend to individuals inside the company, not just the board and officers.
Re:mystery worm out there (Score:1)
But by posting this question to /., and by being able to show a record of his attempts at fixing the problem, he should be off the hook. He can point to this story as a time-stamped description of his stopped efforts. The the blame would shift to his boss.
But I could be wrong.
Re:mystery worm out there (Score:2)
Re:mystery worm out there (Score:1)
I am not a lawyer, but I play one on Slashdot!
But I could be wrong.
Yup.
Re:mystery worm out there (Score:2)
If they're in Russia, and making money illegally, and not owned by organized crime -- they will be soon.
Tell ya what... (Score:5, Funny)
Re:Tell ya what... (Score:4, Funny)
Re:Tell ya what... (Score:2)
Hire an expert. (Score:4, Insightful)
My company has an outside security company run quarterly checks against our network, and they sometimes catch stuff that I miss. Just don't let them talk you into buying a over-priced checkpoint firewall when all you need is a Linux box and Iptables.
--Ajay
Re:Hire an expert. (Score:2)
Just make sure that if he does nothing it is clear that he was warned and that you tried to do the right thing. Then make a copy of the memo for yourself before giving it to him.
Finally, follow your boss's orders and get back to web programming. Until the security sweep, assume every box IS compromised and don't do foolish things like VPN into it or use the same password you use on your own systems/accounts.
If the boss does nothing and things get really bad, I suppose an anonymous tip to the FBI is in order. (After all, if MY credit card info were on one of those servers, I'd want something done.) When customers start sueing, you have your memo to use as a shield for yourself and to stick it to your boss in court.
Do your job? (Score:5, Insightful)
Re:Do your job? (Score:4, Insightful)
Re:Do your job? (Score:3, Insightful)
Welcome to the wonderful world of corporate webdev (Score:5, Insightful)
You're already doing your job, and not being listened to. Since I'm not a sysadmin, I've got no direct advice for you regarding the tracking of such activity. However, it seems to me that this is the smaller of two problems. The first is being able to do your job from a technical perspective. The second (and it seems, more immediate) problem is being able to do your job from a political perspective.
Your boss has already watched his public facing site(s) and servers go down due to his failure to listen to you. Now it sounds like he's about to make the same fatal mistake. This, of course, places you in the lovely position of having to remind him that he's about to make another major tactical error... but you also have to do so in as subtle manner as possible, so it doesn't sound like a recrimination. All I can suggest is to try to make it blatantly obvious to him without coming out and saying it, thus giving him the opportunity to "discover" his error and correct it on his own.
One other suggestion: document, document, DOCUMENT! Make sure that you can prove later (should it be necessary) that you did everything you could. This is another area of vital importance for your job security that also must be done very carefully. Simply CCing the higher-ups will likely piss off your boss (and possibly the folks you're CCing as well), and may look like unnecessary whistle-blowing or complaining. Do it as unobtrusively as possible, but make sure that you're covered, in case there's any question later.
One possible solution to both problems is to communicate all of your concerns in an e-mail. Write it during your lunch hour, so he can't get upset at you for "wasting" more company time on it. Make it clear exactly why you think there is an issue, and mention (if you can do so gently and without provoking him) the past incident. Remember that if you want someone to do something, you shouldn't tell them why *you* want them to do it. They don't care about what you want. Tell them why *they* want to do it. Best-case scenario, he listens. Worst-case scenario, he gets a little more annoyed at you, but you've got your documentation.
I don't envy your position, and wish you luck.
Run, don't walk. (Score:5, Insightful)
On the other hand, if they continue as they are, they may not survive, and you are screwed again.
Re:Run, don't walk. (Score:2)
Watch out (Score:4, Interesting)
The worst case scenario here is that you detect a problem, attempt to fix it yourself, and trigger Something Bad[tm] in the process: the cracker retaliates, or you break a working app because you upgraded something out-of-sync with glibc (or whatever), or you otherwise become the catalyst for noticeable downtime that will piss off your boss and get you fired, or worse - they turn you into a scapegoat (see the Intel case against that security chap.)
Just make sure you cover your ass. You've notified your boss, copy those emails to a nice safe place (headers and all), and don't do anything stupid.
Best scenario is to build a fresh box, backup the old box's data, restore it to the new box (clean! no code! only data!) Don't bother trying to salvage a compromised O/S installation. Too many things to miss. And, when you're building fresh, don't ssh via one of the infected boxes! Don't inadvertently give *any* info to the crackers that you're setting up a new machine. Better yet, build it with the ethernet cable unplugged, if possible. Do it from CDs.
Sounds like a breakin... (Score:2, Insightful)
Also, one good thing to do is place a clean box in between the comprimised server and its internet connect and run tcpdump/tethereal on the brided connection. The first thing you need to do is be able to identify _all_ traffic going in and out of the box. If you dont know what something is, research it. You never know what you might stumble upon.
Good luck.
shut up, or quit (Score:0)
Once you step into the kitchen, you want to throw up. Cockaroaches everywhere. Nobody washes anything. They scrape off the filth before using the pots and then toss them in a pile. When the wind is right, you smell rotting food.
You point this out and the boss says, sorry you're a chef, not the cleaning crew, mind your business.
You have two choices:
#1: do what the boss says (and DOCUMENT all conversations because at this kind of outfit, you will be the first to get blamed when the restaurant gets sued by a sick and angry customer).
#2: quit, find another job, even if you have to take a pay cut to find one. Find a place that harmonizes with YOUR idea of a good place to work.
I would take #2 every time. Your example is not the kind of place I'd like to work, either as a programmer or a sysadmin. EVERYBODY should be acutely aware of security issues, and willing to hear anything that improves security. It's like not having insurance, or bathing in sewage with open wounds. You'll eventually regret it.
If youre boss is not a techie (Score:4, Insightful)
You'll be given all the time and budget to fix it. FUD doesnt always require proof, unless someone calls in some consultant.
Resume time (Score:2)
think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security.
You've got a money hemorrage on your servers. Don't bother trying to talk sense to your boss, just strengthen up your arms and be read to jump ship and swim when the next boat comes along. Start shopping your resume now, and remember: when everyone is out to get you, paranoia is just good sense.
Undetectable spammers (Score:-1, Offtopic)
Sue them. (Score:1)
Re:Sue them. (Score:2)
And therein lies the problem. As a consumer, there are thousands of companies I could buy from online, but as long as security is given such low priority, how can I trust any of them?
Let the boss know... (Score:2)
Tell him that keeping your servers secure IS paying work. As a customer, if I know that a company is not keeping my credit cards, etc. secure, I will not do business with them. Ask him what would happen if there was a big 'scandal' (for lack of better terms), and thousands of customers credit was exploited. How much of an opportunity would you have to sell your so called 'paying' work then?
Sounds Kinda like my job.....LOL (Score:2, Interesting)
At my job, I am one of two web developers. Besides us, there are the two owners and our systems admin. The owners want to become a viable commercial hosting service with secure storefronts, etc. Fine says me.
The problem lies in that one of the two owners (The husband) is a pig-headed idiot. Recently he asked us to implement a RAID solution for the webserver (notice the lack of an 's' at the end of webserver). Not a problem says sysadmin and myself, we come up with the plan, and present it to him, it involves RAID-5, blah, blah, blah, all the standard normal stuff that people do...He quickly scoffs at the idea, hands us a OLD P.O.S. [ebay.com] with a couple 10'ish gig drives and says make it out of this and use RAID-1, and promises to order some large drives for the machine, so we can implement his RAID-1 solution but insists that nobody in their right mind would ever use RAID-5...
We of course are like what the hell? You want something that is enterprise level, and expect us to make it out of this P.O.S.???
We resign ourselves to doing the best we can with it and get a crummy webserver up and running with Slackware.. It is not the fastest machine, but it works for now. We currently have no RAID, becuase the large drives he said he would order have not been ordered for 2 months.
A couple weeks ago, we are talking about a file server for our internal software, etc. and he loves the idea..GREAT!!! So we spec out a modest system that will fulfill our needs and he says, oh I have a perfectly good fileserver at home that you guys can use to make it, and the COOL thing about it is that it runs on something similar to (but not) this [rebyte.com]. We research his little linux memory card thing and yes, it is cool, however it is not capable of doing what we need to have it do, and from what we can find out about it online it is not capable of performing one of the tasks without substantial work being done, the least of which is compiling and installing netatalk (which is no big deal) for some machines that cannot connect via Samba (MacOS 9 that would require DAVE [thursby.com] that he is not willing to purchase).
So we add in the 2 120-Gig Drives to the PII/166 with 64M RAM that he gave us to make a file server from and find out that
So we switch out to some other P.O.S. motherboard he has lying around and find that it has some popped caps, so it won't work either.
We eventually come up with a working P.O.S. motherboard, put some WRONG entries in the BIOS to make it recognize the 120Gig Drives and install a 4.xG drive to use as the system drive running samba and netatalk. All is looking well...
So we get FreeBSD installed today and are in the process of setting up the Xserver so he can have his GUI, since he doesn't know jack about the command line and then we are going to implement his RAID-1 that he loves so well on this machine also. We are stoked to say the least that after all that hassle we have a working system and FreeBSD sees the entirity of the 120G drives...
So he comes into work today and sees XWindows (twm) up and running and asks what we are doing, and we tell him, making the file server like we had talked about. He asks what it is running and we tell him FreeBSD 5.2.1 and we are finishing the config, then implementing the RAID and it will be ready to go. This is where it gets good.
He flies off the handle and says, did I tell you to use FreeBSD?? We are like...ummmmmmm...we talked about that memory card thing and explained to you why it wouldn't work and therefore have implemented a solution that fits our needs like we talked about. His response was "I told you
Re:Sounds Kinda like my job.....LOL (Score:1)
What I'm actually surprised at is that three established guys (for which you say have 40+ years of experience between you.. hope you're not counting high school computer classes
Find another job, it's not going to work out.
Re:Sounds Kinda like my job.....LOL (Score:3, Insightful)
Maybe ask for an IT budget whereby you and the others get a fixed amount of cash to spend on hardware, since he's obviously paranoid about spending (forgetting that he pays you tons more to deal with the crap he buys) and wants to avoid getting something more expensive than it needs to be, and you want to avoid spending on things that don't meet your needs. Even a measly $500 can go a long way when you, the expert, has to do the shopping.
Re:data managers (Score:3, Insightful)
What I said to one guy like that is "Sir, we respect you and will do whatever you want us to do here, because you're the boss and it's your company, but you hired us to take the load off you so you'd be able to do less work and make more money. Trust us to do a good job and we will. We might not do everything exactly the same as you would, but we know the end result will still make you happy."
Of course, it takes more than that to change a personality flaw that massive, but it's a good start.
Re:Sounds Kinda like my job.....LOL (Score:1)
The longer you stay under these conditions the worse your reputation will be. The money you are making had better be enough to cover personal attacks on you by your boss that may haunt you in the future.
Nuke/Pave (Score:2)
simple, ntop (Score:2)
Forget running ethereal or other packet sniffers, they're far too fine-grained for what you're trying to do: prove they're being abused.
Connect a small box running your favorite *nix running ntop [ntop.org] to the service port (or whatever they call it, I'm half asleep) of the switch/router to which the box(es) in question are connected.
That's it.
Ntop will give you very nice graphs to print out for the guy who needs a clue, showing not only the IPs involved, but the ports, percentages of traffic per protocal/port/whatever.
If they're being used for SPAM, for example, you'll see tons of outgoing port-25 traffic.
Just make sure that's not what the box is supposed to be doing!
If this isn't enough for him, and you're sure this is something against the company's policy, bring it to his boss' attention immediately (calmly) with a full explanation of why you are doing so.
If you delay this action, you risk having your boss "poison" his perception of the situation, and end up thinking that you're a troublemaker (that is, if the politics are as bad as I'm guessing they are there).
So either fight it all the way, or drop it... or be prepared to find a new job (these situations always suck).
Security is Paying Work (Score:2)
Security is paying work, unless your company has a reputation without worth. You can get that sentiment out of any of the good security books out there, but saddly management can only see the spending cycle, not the averted crash-burn-patch-fix cycle. Remind him that if we were all "paranoid" to begin with, we wouldn't have had a hacker tell us to patch our stuff. What if the hacker didn't like you so much?
Security really isn't a game or a buzz word. Folks have to own that concept, not just parrot it.
You are a web programmer. (Score:1)
So my best advise to you is that if it's not in your job description, leave it the fuck alone. You don't want to end up being the scape goat should something happen or should you step on someones toes in the process of honestly trying to help. Besides; if they want network security, they need to open a position and pay for it.
Re:You are a web programmer. (Score:1)
If you find a problem, whether it's security or something else, notify the appropriate people (CC your boss). If they chose not to fix it, that's their problem. At least your a** will be clean if/when there's legal action resulting from the issue.
explain it to your boss (Score:0)
This sounds like a job for ... (Score:2)
But seriously, get a Snort box installed and be more active(*) in your intrusion detection. Surely your boss can't object if you slap down a printed snort long on his desk, and show him proof of intrusion.
dave
(*) I absolutely refuse to use the word 'proactive'. I'm not playing buzzword bingo here.
Nuke the site from orbit. (Score:1)
Also have a look at EtherApe (Score:2)
The main reason for that is that etherape actually maps out live traffic patterns (and volume) on a network. I personally am quite surprised it's not a default part of more security bootdisks. You'll spot an infection straightaway as they tend to 'broadcast' on the network.
As for protection, host checksumming has one disadvantage: it's AFTER the fact...
Good luck - you'll need it. You may want to remind your boss that being negligent with CC data might lose you your merchant status, and losing customer date could result in lawsuits from clients or, in EU countries, a harsh fine from data protection regulators. IANAL but frequently involved in cleanign up situation you just described...
The Bottom Line (Score:0)
Even if they don't take the CC data, its costing your company bandwidth (you can't serve up your data if your pipe is full serving the spammers and viruses), disk space (with viruses), CPU time from running those "extra" programs, and downtime either from being hacked by someone who is tired of your contributions to the network, your provider taking you offline because your causing them problems from the GIGO, or the worm itself killing the system.
It will also cost your company lost revenue due to the loss of business from your customers moving else where due to the services issues resulting from the above.
Even if he does want you to work on something that will "profit" the company, how much will it cost vrs save the company from bringing in a consultant to evaluate the systems, practics and policies? I bet its cheaper than going out of business.
warezzzzzzzzz (Score:1)