Ask Slashdot: Experience Handling DDoS Attacks On a Mid-Tier Site? 197
New submitter caboosesw writes "A customer of mine recently was hit by a quick and massive DDoS attack. As we were in the middle of things, we learned that there are proxy services of varying maturity to deal with these kinds of outbreaks from the small and mysterious (DOSArrest, ServerOrigin, BlackLotus, DDOSProtection, CloudFlare, etc.) to the large and mature (Prolexic, Verisign, etc.) Have you guys used any of these services? Especially on the lower price point that a small e-commerce (not pr0n or gambling) company could afford? Is a DDoS service really mandatory as Gartner now puts this type of service in the same tier as SEIM, firewalls, IPS, etc?"
Lived Through This (Score:5, Interesting)
Re:Lived Through This (Score:4, Interesting)
Re:Misunderstanding (Score:5, Interesting)
From our experience packet flooding attacks are rare, most are application layer attacks because they're cheaper:
- If your landing page is dynamic chances are a small site can be choked at the database from a few hundred zombies, and it's much harder to detect the zombies from the genuine clients in a safe automated fashion
- If you don't have a lot of CPU at your firewall layer you can't create long enough rule tables to stop the bad traffic as you detect it
- Often you can simplify your rules but just starting by blocking China, Russia, Korea, then smaller countries that are hosting bots.
If they are genuine flood attacks:
The idea that your ISP will block a "list of addresses" is comical, it's not nearly responsive enough, and if you're lucky your ISP will agree to block countries and only if you have a business account which you're paying over the odds access fees for. Some will even null route YOUR IP instead of the attackers to save their own infrastructure: http://www.abc.net.au/4corners/content/2009/s2658405.htm [abc.net.au]
ANDREW FOWLER: The Russian cyber attack was so sustained it backed up through Telstra's network, knocking out the whole of Alice Springs, part of Adelaide, and Telstra central in Sydney.
DAN CRANE, FORMER TECHNOLOGY MANAGER, MULTIBET: And that's when they sort of started to panic a bit I think because all of a sudden it wasn't just a, you know run of the mill attack, this was a pretty hardcore attack because that's when it started, that's when it took out Alice Springs, that's when it degraded Adelaide and that's when it melted their routers in Sydney so that's when they said that's it, we don't want a bar of it.
ANDREW FOWLER: According to Dan Crane, Telstra stopped accepting any of Multibet's internet traffic from entering Australia.
Not to mention even creating this list is a continual task. Botnets rent out "so many connections", but the computers that are active at any time rotate in and out of the pool. We saw probably around 1000 computers at a time hitting the firewall, but from a pool of more like 100,000 addresses we discovered over the course of a week. We initially took a strategy of programmatically blocking individual IPs as they came in at a response rate of about 5 seconds with some scripting, but soon moved to blocking entire countries that we didn't do business with and doing some daily post processing of the IP list as well to consolidate IPs into /27s and sometimes as far as /24s
Our last client to have this issue used Black Lotus and they seemed to do a good job for the price and be quite responsive, though they were still learning their trade at that time... I don't think they were terribly cheap. And botnets are much much cheaper, so unless you're lucky and it's someone that loses interest and not a competitor attacking you it can end up making your web hosting very expensive.
First Hand Experience at Small Company (Score:4, Interesting)
Posting AC as I would prefer not to expose my employer in anyway.
I went through this exact situation the week before last Thanksgiving last year. I work for a gifting retailer that makes all of its money in Q4. Not a good situation. We're a small - mid sized business with about $20 million in sales from our Ecommerce site.
We went the cheap route first. The proxy service cost about $500/month and guaranteed 10 Mbps clean traffic to the site. Our DNS was changed swinging our domains to the proxy service and ACLs put in place on the "backend" to only allow connections from our new gateway in the proxy.
Things were fine for about 24 hours when the attack was stepped up. The service was seeing 450 Mbps inbound to our main domain. That is not a mistake - 450 Mbps is easily attained using a botnet or simply focusing the attention of some lurkers on pastebin links. We now had to change DNS AGAIN to "upgrade" to their better platform that could handle this attack. As we started this work, we also began talking to a couple of the higher end services...
After the $500/month service capped out and blew a gasket, we made the tough decision to go with the Cadillac. It was costly and they had us over a barrel (day before Thanksgiving, cheap service not working out, "sure would hate to see your site go down on Black Friday" mob pressure). But we knew even half a day of lost demand would pay for the yearly service (yes, it is yearly - no month to month option).
The difference was amazing. As soon as we had swung our DNS over to the new guys, the attack was mitigated within 5 minutes and abated within 20. This, of course, leads the paranoid to wonder whether it was the service doing the attacking to begin with, but we are a high profile target in the minds of the Occupy movement, so it made sense (I do not share my employers sense of community - it is only a job).
We have been attacked since then and every time the attack was mitigated within 5 minutes. If you require this type of uptime, build this service into your budget from the beginning.
Re:Gambling (Score:4, Interesting)
I used to run infosec for one of the mid-tier online gaming operations run out of the Caribbean. We got extorted by one of these gangs, and ended up paying Prolexic (they were Digidefense at the time) to solve this for us.
As for weather you can risk doing without it depends strongly on what your user tolerance for downtime is and how bursty your revenue stream is. The lower the tolerance and/or the more bursty the revenue stream the more vulnerable you are to these sort of attack methodology, as the opposition pays for the time they are actually attacking you, so if you can weather the attack they'll eventually give it up. If on the other hand they can cost you significant sums of cash by taking you out for 6 hrs (say sports betting, target the opening day games), that increases your susceptibility to these attacks.
Feel free to drop me a line if you have any more questions (my /. listed email will get to me).
Min
Re:Load balancing and an experienced sysadmin (Score:5, Interesting)
The essence comes down to two things. Neither is particularly complicated in principle, although getting it right can be a bit fiddly.
1) Detect attacking IPs.
HTTP Flood DDOS bots aren't (at least not yet) smart enough to look and behave EXACTLY like people using web browsers. They do wierd things like load web pages repeatedly while never loading images/running javascript/loading CSS stylesheets. They make sequential requests from the same IP address - but with different user agents. They might load a web page that uses cookies - but never return the cookies that are set. Or they might return a cookie - but from a different source address or with a different user agent. They might send user agents that haven't been in widespread use in half a decade. They might not set the 'referer' header, or some other header that a browser DOES set correctly. They probably don't follow HTTP redirects. What you are looking for is any behavior that distinguishes the good traffic and the bad traffic.
So I 'tailed' the web server log and analyzed it in one to ten minute chunks to detect abnormal accesses. All detected addresses were added to a persistent database of blacklisted addresses.
2) Add the detected attacking addresses to an efficient firewall.
A naive firewall blacklist might try to just put each addresses in one big long list. This doesn't scale well beyond a couple of hundred attacking addresses. On the older machine I had, I used a 'divide and conquer' approach: I created a few hundred filter chains based on a /n subnet division of the attacking ip addresses. I then wrote a set of rules that divided incoming traffic into those chains based on the /n they were a member of. That made the number of rules required to filter n attacking IP addresses scale as about O(log n). If I had had a more recent kernel I could have used a hashed map of addresses to take that down to O(1).
After that it became a slow game of cat and mouse. The attacker would alter his attack to try and slip by the detection, I would update the detection software to detect something else he wasn't getting perfect if he managed to by-pass the filters. After about two weeks they quit attacking the web server.
The largest issue I had really was that I was starting my defense from a 'standing start': I had to write all the needed scripts from scratch while the attack was still on going.
Re:Best defense: Overprovisioning and cutoffs (Score:4, Interesting)
Typically, yes (assuming your OS platform of choice doesn't have some other resource that can be remotely exhausted more cheaply then bandwidth). The problem is one of the standard defender delimas: The attacker needs bandwidth for a short period of time (typically), as their goal is to make you say "Uncle" weather that means paying their ransom, capitulating to some demand or whatever. You as a defender have to incur a cost for your defensive strategy that is either (relatively) low, non-scalable, and continuing (trying to out provision the attacker) or a high cost outsourcing solution. The attacker on the other hand rents 10,000 nodes for 200$/day. Figure that's about 5gigs conservatively (we'll say .5mbit upload as an average per node). Now assuming your data center will handle a sudden 5gig burst without cutting you off (good ones will, cheap ones will just cut you off) your hosting bill just went up by 54TB (5*3600*24/8) per day. That's not going to be sustainable for long.
That's why the outsourcing solution tends to be the way to go if you're being targeted by anyone willing to spend halfway decent money on attacking you. The ROI from the attacker POV looks pretty good. Say they ransom you for 50K (an average number for such things). If they have to keep you under DDOS for even a week till you cave, (378 TB worth of data) that nets them 48600. That's a pretty good business case from their point of view.
It's one of those moments when it sucks to be the good guys.
Min
Re:Change Apache to nginx (Score:4, Interesting)
Amazon AWS bills you bandwidth directly. A DDoS could get very expensive.
Re:Best defense: Overprovisioning and cutoffs (Score:4, Interesting)
In the case I was involved with it was wired via Western Union to a place in Moscow where (according to the PI we hired) it was picked up by call girls and taken back to the culprits. They did eventually get nailed but it took years due to the complexities of law enforcement in an international environment.
We eventually signed with Prolexic to stop them coming back.
Min