We've Been Hacked... or Have We?

hidden_fire asks: "I recently got a job as a Web Programmer at a web company that hosts many sites. The company had many badly firewalled Windows and Linux servers without any security patches, and a shared administrator password. I warned them that they needed to improve their security, but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised, and the Sasser Worm took us offline. Now, I've been allowed to rebuild the compromised box and tighten our firewalling, but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, a Linux kernel lockup, strange ports being open, and performance issues. I think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security. Has anybody else been in this situation? How can I detect these guys if their tools don't show in virus scans?"

How to spot what is happening

[Karora]

Here's a couple of things you could do:

Download and build chkrootkit [chkrootkit.org]. This will detect a lot (most?) stealthed kits on Linux systems, and it is always my first port of call when I'm invited in to clean up after a breakin.

Plug in a hub (so all traffic can be seen by multiple machines - a switch ain't as good, unless it has a monitoring port) in front of the machine(s) and run tcpdump or ethereal on another system to watch traffic from the machine. This will let you watch exactly what traffic is happening on those weird ports, or watch outbound SMTP traffic for spammer activity.

We don't put Windows-based systems on the internet, partly for security reasons, and partly because we don't have any Windows specialists, so I can't help for on-the-box detection there, although I would expect a commercial virus scanner should find everything.

Watch out

[schmaltz]

Since you're asking "how do I disinfect these boxes" (essentially), you make it pretty clear you don't already know how. You could download Nessus and chkrootkit, and maybe get lucky - e.g. the cracker who's built a nest inside your server doesn't detect your attempts to detect him.

The worst case scenario here is that you detect a problem, attempt to fix it yourself, and trigger Something Bad[tm] in the process: the cracker retaliates, or you break a working app because you upgraded something out-of-sync with glibc (or whatever), or you otherwise become the catalyst for noticeable downtime that will piss off your boss and get you fired, or worse - they turn you into a scapegoat (see the Intel case against that security chap.)

Just make sure you cover your ass. You've notified your boss, copy those emails to a nice safe place (headers and all), and don't do anything stupid.

Best scenario is to build a fresh box, backup the old box's data, restore it to the new box (clean! no code! only data!) Don't bother trying to salvage a compromised O/S installation. Too many things to miss. And, when you're building fresh, don't ssh via one of the infected boxes! Don't inadvertently give *any* info to the crackers that you're setting up a new machine. Better yet, build it with the ethernet cable unplugged, if possible. Do it from CDs.

Sounds Kinda like my job.....LOL

[Anonymous Coward]

While my situation isn't *exactly* similar, I am interested in people's thoughts...Here goes:

At my job, I am one of two web developers. Besides us, there are the two owners and our systems admin. The owners want to become a viable commercial hosting service with secure storefronts, etc. Fine says me.

The problem lies in that one of the two owners (The husband) is a pig-headed idiot. Recently he asked us to implement a RAID solution for the webserver (notice the lack of an 's' at the end of webserver). Not a problem says sysadmin and myself, we come up with the plan, and present it to him, it involves RAID-5, blah, blah, blah, all the standard normal stuff that people do...He quickly scoffs at the idea, hands us a OLD P.O.S. [ebay.com] with a couple 10'ish gig drives and says make it out of this and use RAID-1, and promises to order some large drives for the machine, so we can implement his RAID-1 solution but insists that nobody in their right mind would ever use RAID-5...

We of course are like what the hell? You want something that is enterprise level, and expect us to make it out of this P.O.S.???

We resign ourselves to doing the best we can with it and get a crummy webserver up and running with Slackware.. It is not the fastest machine, but it works for now. We currently have no RAID, becuase the large drives he said he would order have not been ordered for 2 months.

A couple weeks ago, we are talking about a file server for our internal software, etc. and he loves the idea..GREAT!!! So we spec out a modest system that will fulfill our needs and he says, oh I have a perfectly good fileserver at home that you guys can use to make it, and the COOL thing about it is that it runs on something similar to (but not) this [rebyte.com]. We research his little linux memory card thing and yes, it is cool, however it is not capable of doing what we need to have it do, and from what we can find out about it online it is not capable of performing one of the tasks without substantial work being done, the least of which is compiling and installing netatalk (which is no big deal) for some machines that cannot connect via Samba (MacOS 9 that would require DAVE [thursby.com] that he is not willing to purchase).

So we add in the 2 120-Gig Drives to the PII/166 with 64M RAM that he gave us to make a file server from and find out that

• Floppy drive is bad
• 1 on-board IDE controller is shot
• CD-ROM Drive is broken

So we switch out to some other P.O.S. motherboard he has lying around and find that it has some popped caps, so it won't work either.

We eventually come up with a working P.O.S. motherboard, put some WRONG entries in the BIOS to make it recognize the 120Gig Drives and install a 4.xG drive to use as the system drive running samba and netatalk. All is looking well...

So we get FreeBSD installed today and are in the process of setting up the Xserver so he can have his GUI, since he doesn't know jack about the command line and then we are going to implement his RAID-1 that he loves so well on this machine also. We are stoked to say the least that after all that hassle we have a working system and FreeBSD sees the entirity of the 120G drives...

So he comes into work today and sees XWindows (twm) up and running and asks what we are doing, and we tell him, making the file server like we had talked about. He asks what it is running and we tell him FreeBSD 5.2.1 and we are finishing the config, then implementing the RAID and it will be ready to go. This is where it gets good.

He flies off the handle and says, did I tell you to use FreeBSD?? We are like...ummmmmmm...we talked about that memory card thing and explained to you why it wouldn't work and therefore have implemented a solution that fits our needs like we talked about. His response was "I told you