Bad Behavior on the 'Net - Who Pays the Bandwidth Bill?

rakolam asks: "I am involved with network management in the hosting department of a fairly large ISP. Constantly we have customers who dispute inbound bandwidth spikes and demand service credits on their burstable connections. Events such as the Slammer Virus literally have everyone knocking on their salesperson's door at the end of the billing cycle. My position is that the internet is a public space, and by placing themselves in that space, one has to realize the consequences (and the implications of burstable billing). I'd like Slashdot's perspective on this. Should ISP's ultimately eat the costs of malicious behavior? Is the customer ultimately responsible for the bandwidth they've generated, regardless if it's desired or not? Is this a new frontier for insurance companies?"

Re:Users just won't pay

[Gaijin42]

Thats because they pass that cost on to the vendor, for not validating enough information about who the purchaser was.

The CC company doesn't eat that. The vendor does for accepting the stolen card

If you control, you are responsible

[jrpascucci]

If you are a co-loc provider, where the person configures and runs their own machine and firewall and can take steps to minimize this sort of attack, then you have no responsibility: you are merely providing bandwidth.

If you control shared servers and/or if you do not give users a configurable blocking mechanism (firewall, IP addr/range blocker, for web services a bogus URL block or the ability to ban individuals who spam sites) then you are, in fact, responsible for the bogus bandwidth usage.

Here's the problem Jerky...

[Jim Ethanol]

The problem with billing for excessive inbound traffic is that the user has absolutely no control over what they receive.

You can have the most sophisticated firewall on the planet, but due the immutable laws of IPv4 you can NOT drop a packet until you see the packet. At which point you've already used the bandwidth (and incurred the cost) required to transport the packet that you're just going to drop.

This has nothing to do with patching your server. If you don't patch your server, and you get hit with a worm, and your box starts consuming huge amounts of bandwidth to attack other hosts, then it's your fault, and its OUTBOUND traffic, and you absolutely should pay for it. But having your server patched does not stop you from receiving inbound packets. They may not harm your server when they get to it, but you already paid for the transit.

BTW, This is why it's illegal for a telemarketer to call you on your cell phone. Because in theory you had to answer the call (and incur expense) BEFORE you knew who was on the other end.

This is a similar issue, except that we're not talking about telemarketers... which are businesses that more or less follow the rules. We're talking about script kiddies that don't care about the rules. Or in a worse case, we're talking about a competitor, or enemy, or rival that just wants to DOS you for a month until you go out of business because of all the excess bandwidth charges you're paying!

The technology limits the liability of the consumer. The ISP must take some responsibility here and put systems in place that protect the consumer.

-JE

Re:analogous to water/electric company IMHO

[ReverendRyan]

"What happens to you if someone runs an extension cord from your house or if you spring an unknown water leak? You get a huge bill and you fix the problem. How is this different?"

Its not like that at all. The original question is about UNREQUESTED inbound traffic. Do you want to pay for the bandwidth "you" used if I flood ping you with large packets for a half hour? Is that YOUR fault? Can you do anything about it at the time? No. The most you can do is not use any outgoing bandiwdth in responce (in this case, dont respond to ICMP_ECHO packets).

To answer the part of the question

"Should ISP's ultimately eat the costs of malicious behavior? Is the customer ultimately responsible for the bandwidth they've generated, regardless if it's desired or not?"

Yes, the customer should be responcable for *generated* packets, as well as *legitimate* inbound traffic. However, just like a creditcard, the customer should be protected from use outside the normal scope/intention, especially with malitious intent (ie recieving 1000 copies of the slapper worm per second, when you are running a patched SQL server).

Re:OT: What makes up bandwidth costs?

[linuxwrangler]

I've always found the method most large hosting companies use to charge for bandwidth very reasonable. Most of the ones I've dealt with use a variation of the following:

1) Total the bytes every 5 minutes (basically average usage in 5 minute groups).

2) Sort all the samples from highest to lowest for the month.

3) Throw away the top samples - usually contracts specify something like 5% or 10%.

4) Bill based on the highest remaining sample

It is quite logical - once you have installed the infrastructure to support the bandwidth it doesn't matter if you push one bit or a billion over that. In other words, the relevant cost is to support a large enough pipe (which is determined by peak usage) for the customer. Pushing 1megabit 24/7 takes the same pipe as pushing it for an hour a day.

Remember that the typical connection to a cage is 100Mb so you can really push bits for a few seconds as long as you don't get too many high 5-minute samples.

Commentary: I feel that the same logic should apply elsewhere - if I have a 384k DSL then I'm automatically capped to that bandwidth and should be allowed to use it for brief periods or constantly. Fortunately that's exactly how my local home ISP sees things as well - static IP for everyone, no restriction on # of computers, servers, etc.

Back to the original discussion: if the hosting company offers to the customer to set a rate limit then I think the customer loses otherwise there is some culpability on the part of the ISP.

In reality the current business climate will mean the ISP eats it if they want to keep their customers. The ISP is probably not out any money anyway since (see above) the cost is in the pipe, not the usage (unless, of course, they buy their bandwidth using similar contracts from an upstream provider).

Re:Here's the problem Jerky...

[JohnnyBolla]

Ok, I work for an ISP, and a damn big one at that. When one of our circuits gets hit with a Ddos, we call our upstream provider and have them block the attack at their router. We incur no cost for this, it's covered under our contract.
Of course this is for leased lines, not metered bandwidth in most cases, but the concept remains the same. We watch our own backyard, when something happens we react and get the problem resolved. If one of our cable modems is spamming or spewing slammer all over the Earth, we notice and shut off the offender. If we didn't care to look, we would get negatively impacted, just like the guy that doesn't notice his machine spewing out slammers, or nimda, or getting slashdotted.
Take an active role in your internet usage and you are largely immune to this sort of billing. You are responsible for your own stuff, if you aren't taking care of your stuff, I sure as hell shouldn't be expected to eat the cost.
It is YOUR FAULT if you get four hundred and eighty million hits. You put up the site. If you get slammer, you should have patched. Quit crying about your bill and administer your system.
Ounce of prevention, blah blah blah.

Re:analogous to water/electric company IMHO

[gmack]

There are several Apache mods that will either limit total useage or shut off files on the end of large spikes.

The original question though is what should the ISP have done. IMO they should have firewalled access to the affected ports and then split the cost.

Re:Charge on sent traffic.

[Jay L]

No one company/institution on this planet could justify a dedicated oc12.

And you post this from hotmail? Are you just trying to supply a counterexample in the same breath?

When I worked at AOL, OC48 installations were a regular occurrence.

Re:analogous to water/electric company IMHO

[kolevam]

I've got a web hosting account right now with Verizon, and in the control panel there's a combo box that prompts something like "when your max bandwidth is reached, what do you want to do?" which I've got set to "Suspend my account". I've never looked into it, but I assume it means that when the bandwidth is reached, requests will get some kind of error and I won't be billed for the excess.

Re:In other words

[Anonymous Coward]

" they HAVE TO know the /. effect is going to be too much for a page. "

No. Back in July I did some benchmarking of ext3 vs Reiserfs and it got on Slashdot.

Although we got 2 million hits in 24 hours, it was NOT too much for webserver and our T1 line.

I was happy about the traffic as we received quite a bit of exposure.

Dax Kelson
Guru Labs

Re:Fairer - sent or solicited - a modest proposal:

[Sloppy]

What you describe sounds very fair. But.

TCP: You get billed if you make, attempt to make, or accept, a connection. You don't get billed for attempted connections you refuse or that don't get completed (i.e. SYN and other DOS attacks).

Suppose the case of a customer who runs a SMTP server. A spammer tries to connects to it, the server accepts. The spammer sends a few gig of spam to him, which procmail or something ends up throwing away. Technically, this is "solicited" since the user's machine did accept the connection. But it is abuse, and wasn't really "solicited" in the way that we humans normally think of it.

What would you do? Bill anyway, since that's part of the risk of running a SMTP server? Maybe this user needs a smarter server that rejects spam at the time of connection. Hmm..

Re:analogous to water/electric company IMHO

[-Surak-]

Presumably this refers to hosted server connections, rather than a simple virtual web server account. For this sort of connection, I would want a true Internet connection, instead of some firewalled lan port. I would be very upset if the ISP did ANY filtering on my connection without my specific request or knowledge. It's none of the ISP's business what I do with my end of the network cable (aside from spam policies) - they don't need to know if I'm running a web server, SQL server, or some custom game server that happens to use UDP/1443.

Most colo providers I'm familiar with bill on 95th percentile bandwidth, which means that they drop the top 5% of samples (typically 5-minute average) and bill you for the bandwidth of the highest remaining sample. This means that you can absorb short-term heavy bandwidth spikes without being charged, up to about a day and a half worth of time per month.

In any case, the ISP should have no way of knowing WHAT traffic creates the bandwidth spike, unless I specifically request that they monitor my port. Of course, smart ISPs will exploit these incidents by offering firewalling services as a value-add, even if it's just stateless filtering at the router, as a way for customers to "insure against unexpected traffic spikes from virus/worm activity".

Of course, if I was paying for virtual web service, rather than a server colo and bandwidth fee, I should not be charged for non-web traffic, and I doubt any ISP would have the balls to do so.

Re:Simple policy

[slarti]

I run a hosting facility and I can tell you what would happen. The clients would opt for the 'cheaper' option and still complain when they get nailed.

We do bandwidth monitoring for all our clients and provide 24/7 access to the reports so clients know exactly where they stand with regard to their usage.

As I've only read the comments down to this point I haven't seen anyone discuss how bandwidth utilization is actually calculated and billed.

For the most part the comments are in regard to ISP's providing consumer Internet access as opposed to collocation, or hosting which is a different beast.

When we sell a client a T1 they get the bandwidth that will go over a T1.

Collocated clients you have to monitor via switch/router interfaces, NetFlow, et. al. The resources it would take to discern 'real' traffic from 'invalid' traffic would make it not worth the effort of the provider.

As I mentioned we provide clients access to utilization graphs updated every 5 minutes. We explain to them what they mean and get them to understand their own usage. If we or a client detect unusual usage we research it. If it's an attack we attempt to shut it down, if it's legit it stays. That doesn't make the client not responsible for bandwidth directed to or originating from the equipment they chose to put on the Internet.

Re:analogous to water/electric company IMHO

[dhogaza]

The City of Portland Water Bureau will forgive excess water bills due to undetected leaks or the like if you show that you've fixed the problem. Often leaks aren't detectable and a large water bill is the first clue the homeowner sees (western Oregon is very wet, water water everywhere)

Re:analogous to water/electric company IMHO

[bofkentucky]

You know, there is a soultion for email, its called the MX records, say your primary acount is at foohost.com, but you hit a bandwidth throttle/cutoff. You could contract with a friend/other provider to allow a small mail machine in their datacenter, just provide a backup MX record to fallover to the other provider and you are set. If you only have it as a MX record, your web traffic won't touch the other box. I'd call a small local ISP and talk to one of their systems dudes to see if they would allow it.

How it works here

[ziegast]

I currently work for an ISP that offers shared and dedicated web services. The Terms of Service that the customer signs are pretty explicit about their being responsible for bandwidth usage.

A few notes about charging for bandwidth:

• As a hosting provider, we get charged for traffic in the greater of two directions - outbound. We don't normally charge customers for inbound bandwidth.
  
  
• We rate limit traffic from all servers to 10Mbps as a precaution to protect ourselves. Being a relatively small provider, it is VERY rare that we or a customer of ours runs a server that generates more than 1-2 MBps of traffic. Everyone has a 10/100 port though, so the potential for a customer (or a customer's hacked machine) to do damage is possible. If someone wants the rate limit removed, we warn them again that they are responsible for their traffic.
  
  
• We offer rate limiting to our customers if they are afraid about bandwidth costs. This might normally be a 1.5x the rate they're normally budgeting each month. Many customers find that rate limiting makes their site too slow, but riding a bike with training wheels is slow too (but you're less likely to fall down).
  
  
• We charge by GigaBytes per mo. It's easy to track in web logs and packet counters and customers can write scripts to monitor how much they've used during the month and take appropriate steps toward teh end of the month. This amounts to our charging for average (50th percentile) pricing. We charge enough so that even if they spiked at twice their average, we wouldn't lose money on our bandwidth costs. On average, though, we make money.
  
  
• If a customer doesn't pay, we shut them off and can take them to small claims court based on the TOS agreement.
  

These are some of the steps we use to protect ourselves and our customers. Your milage may vary.

(We use packeteer for rate limiting, but I keep eyeballing OpenBSD/AltQ/PF for both rate limiting and firewalling for our customers).

Not vandalism, wireless spam

[rnapier]

There's been a lot of talk comparing this to vandals coming and screwing with your server or your property. This isn't like that. If your server gets trashed, that's your problem. The issue here is incoming bandwidth that you didn't ask for and have done everything in your power to make go away.

Compare this to someone constantly text-messaging spam to your wireless phone. You could quickly run up an insane bill that way, and there's really nothing you could do about it. The wireless company is contractually in its rights to charge you.

But it won't.

That's how they work. Someone screws with you, typically the provider eats it, especially if there was nothing you could do about it. That puts the incentive back onto the one entity who can actually do something about it: the providers. True for wireless. True for credit cards. True for just about anything where the end user can't do anything to stop the abuse.

The ISPs can do something about it. They have chosen not to because of how we (the geeks) developed the internet. It's too trusting. But at the end of the day, your ISP does know who you are, because they send you a bill. And they could apply uniform terms of service if they chose to, and only talk to other ISPs who have similar terms.

The RBLs are the future. They just don't go far enough. When they're willing to not just cut off SMTP but entire connectivity to other ISPs who aren't willing to play by uniform rules, then we'll start to see some changes. What kinds of rules? Here's some for starters:

• Authenticated mail only. Yep, this looks like banks' "know your customer" rules. You can be anonymous all you like up to the point that you connect to the mailer. But the guy who forwards mail for you is going to be held responsible for your behavior. Yes, that will radically change the free-service providers (yahoo, hotmail, etc). They're free to come up with solutions that don't require them to know exactly who you are, but if they host spammers, we're not going to talk to them. This is just the logical extension of RBLs.
• Same deal for acting as a DDoS zombie. The owner of the unpatched box is responsible, but it's the responsibility of the ISP to be able to identify that person for legal action. If they can't or won't, then we don't talk to them.

None of this says that you can't be anonymous most of the time. It just says that if you're disrupting service and causing real losses due to your actions or lack of actions, your ISP is going to have to hand you over, or they're going to be held responsible. The right to privacy has to be balance with responsibility for your actions.

The old-world networks (phones) have worked this way for years. I can block my out-bound caller-id. I can have an unlisted phone number. I can be very anonymous on the phone. But if I'm named in a law suit or criminal complaint, the phone company will hand me over in a heart beat. The only way around this is pay phones with cash. It's hard to run a large-scale scam that way.

And no, this doesn't mean that an ISP's logs are free game to the RIAA. But it does mean that if the RIAA wants to name a specific "unknown party" in a lawsuit, the ISP is obligated to identify them. Before you get excited, that's exactly the current situation. The RIAA just wants to get the info without actually suing you (which is wrong, and luckily some ISPs have resisted). ISPs need to be willing to say they will only interconnect with other ISPs who play by the same rules.

Yes, this will fragment the internet for a short period of time. So do the RBLs. But economics will fix it fast enough, especially if entire connectivity is cut off.

Re: maximum exposure

[Comen]

I disagree, this was the biggest problem with the Dot com fad, everyone was so excited about the technology and the money that might be able to be made, that no one was bothering to actually add up the bills.
It is very hard to even make a buck now a days running a ISP, for years allot of smaller ISP's got by, and made themselves (their business) look better than it was. Providing people with unlimited bandwidth for 50$ a month is hard to do when you figure out all the costs.
Allot of times they didn't have the actual bandwidth they said they did, just to bring in a profit etc...
Most these companies where hoping they would get bought out in the dot com craze and did. The bigger telecoms bought up the smaller guys knowing there would be a loss but also where thirsty for what might the biggest thing in their future, and not wanting to be left out of a good thing.
Now a days things have changed a little, allot of the smaller ISPs gone, and you might still be able to get a cheap line, but more and more people and businesses have learned that you get what you pay for and don't mind paying a little bit more to know that the company wont fold tomorrow leaving them stuck with allot of problems that they would have had from a bigger more respected company that has to play the publicity game more often.
I think bandwidth is still pretty expensive right now, the charges on an OC3 connection are not cheap and most ISPs pay by the bandwidth used.

Plus you got the problem mentioned here.
I do not believe this problem is about Web services, that may be a problem for some, but I think that I parked server that uses up allot of bandwidth for their website should just pay up.
For allot of the reasons already mentioned from others.
But the problems really happen when a Virus or a Bug leads to unreasonable bandwidth usage.
Code Red hurt allot, but after the first couple hours we had filters in place that blocked most the negative traffic from the Virus at the core routers.
Also the recent SQL bug was blocked pretty fast so that people didn't accrue a huge bill. So we are learning fast how to help our customers and ourselves not get into these problems, But there have been some times when a person has hacked a server and loaded a FTP with games and porn whatever and caused them to have a bill in the tens of thousands of dollars and the customer didn't have a clue any of that was coming.
I think we will learn to avoid allot of this, even though it may still be the customers responsibility to configure the server, The ISPs are learning that to keep customers and not get into these problems the have to do more monitoring and check the network more for anything unusual like this. A simple script that runs every night that checks for anonymous FTP PUTS can save everyone allot grief.

Allot of ISPs are just starting to turn a buck again getting spending down to something reasonable that is more inline with their income.
And keeping allot of the more talented people that can really help in these situations will be key to better service. These lines will get cheaper as the money initially invested gets

Re:Don't understand bandwith charge

[Sandman1971]

Your analogy makes very little sense in the real world.

You have your T1. So do 3000 other people. The ISP has calculated that on average, only 15% of your T1, alone with everyone else's, is used in any given month.

That T1 has to connect to something, don't it? It's not a point to point connection to every single site you go to. Your T1 will drop into a DS3, ATM, POS connection. The ISP has calculated what they need to run in the back end, and what they need at the various peering points with other providers.

Let's say the ISP only has 3000 T1 customers. That's a total available bandwidth of 4632 Mb/s for all T1s combined. But since on average only 30% of that is used, that falls to 694. They play it safe and decide that on the backbone they triple that amount (which is not the case. Usually it's less than double). That's still only 2084 Mb/s (or 13 DS3s). Your price for a T1 has been calculated using these numbers. Suddenly everyone uses their T1 at full capacity 24/7. The ISP has to put in more pipes to accomadate this. This means their bill to the backbone have skyrocketed. Since your original price was based on 15% utilisation, and now it's 100% utilisation all the time, what do you think will happen? Your bill will go up significantly. The ISP is in business to make money. If it has to put in another 16 DS3s that will run at 100%, they've more than doubled their operating costs. Why should they take a loss? They are totally justified in raising their prices.

This is how the real world operates.

ISPs aren't 'the internet'

[adri]

So far, I think many posters have forgotten one simple fact.

ISPs don't have infinite bandwidth.

I know, its quite a strange idea. But think of this.

If you're a ISP in a single location, chances are you're buying a few (hundred?) megabits off your upstreams. Unless your upstreams are happy to filter traffic they send to you (and unless its a very large DDoS, most of them will take a while to implement any access control), the ISP will still be charged for traffic sent to a customer even if the customer chooses to reject it.

Similarly, if the ISP provides filtering support for their customers, they still receieve the traffic and bite the usage.

Now, if you're a large ISP and have links to other peering exchanges. Even, say, you peer enough to not really need transit. These inter-state links still cost money. And they're fixed. So if a customer is hit with a DDoS they'll still be carrying it _somewhere_.

Even if this mythical tier-${LOWNUM} ISP with lots of fat peering links has some magical scripts to filter out DDoS traffic to a given customer range, it still will hit their border routers. So their peering cross connects have already been filled. The only way around this is to deal with their peers.. .. Now for the juicy bits. This happens. Every day. The large network NOCs are in constant communication with each other about large DDoS attacks. The little ones slip through the cracks until people complain but generally the large network NOCs will have many other issues to deal with so in a way I don't really blame them.

But they don't really have the incentive to spend all their time dealing with smaller networks being attacked. They'd be worried with keeping their network from melting under a few larger ones.

The flipside. If you're an ISP with enough bandwidth (and not high-profile sites like irc servers or pr0n) you might be willing to bite the costs of various attacks as part of a marketing point. Customers may come to you because you have a reputation of being lenient under attacks. Perhaps. But thats a delicate line.

Me, I dig flatrate pipes. Usage based pipes is just asking to be owned by excess traffic. If I buy a megabit then all I really have to worry about is service degradation due to DoS. ISPs, in my experience, will help you with that. But if you're on a usage based pipe which then gets owned by a DDoS you're struggling after the fact to get a rebate. Good luck.

(Although, that said, perhaps you guys should consider asking for usage based pipes that _have_ a bandwidth cap. Figure out what your maximum spend amount is, say 5mbit, and then ask for a usage-based pipe based on that. That way you limit your liability _AND_ getting the cheaper transit. Most of the time.)

Problem is...

[karlm]

Slammer was UDP, so people got full traffic even if they only had port 80 open. Unless customers have the option of port-based filtering on the upstream side of the connection and/or putting a cap on total bandwidth usage for the account, it's hard to make the claim that it's a risk the customer should have dealt with. Fluctuations in thousands of percents over the previous month's bill is really painful. It seems irresponsible to open up customers to such risks without giving them any ability whatsoever to mitigate the risks. ISPs also have a responsibility to the community not to be lazy and "piss in the communal pool" by standing by and not offering (via phone or email) to filter out traffic (bi-directionally at the customner's discretion) from these internet-wide security macro-events.

Ideally you'd be able to roll over bandwidth for exactly one month as in subtracting the previous month's rollover at the end of the month. Your bandwith would be continously throttled to the rate at which you'd expend all of your bandwdth at the end of the month. Without rollover, the ISPs would have a huge sawtooth pattern in monthly load and one of the sides of the teeth being nearly vertical. The rollover is more for the benefit ofthe ISPs than anything, so is upstream port blocking, allowing ISPs to blockunwanted traffic at its boarders.

Re: maximum exposure

[yintercept]

I worked for awhile in telecom. For the most part, the expenses of the telephone company are fixed. You have switches and T1 connections going in and out. Those are fixed costs.

A telephone company would build a system for anticipated peak service and would add some room for expansion. As a result, the telephone company would build an expensive system with excess capacity.

Although costs were fixed, telecom companies would bill customers for time used. To do this, they would set a rate for normal usage that would be high enough to cover the costs of the peak usage network.

I imagine that the Internet is somewhat the same way. Internet companies build for peak usage and set a rate for normal usage that will cover the cost of the peak usage network.

The thing that happens in a DOS attack is that the DOS attack pushes the services used from the normal level to peak usage levels for a prolonged period.

Since most of the network's costs are fixed, the DOS attack really doesn't cost the network that much more. A DOS attack doesn't spontaneously generate more routers and fiber optic connections.

The end effect of the attack is that it screws up billing. Remember the normal usage rates are set high enough to cover the cost of peak capacity. The DOS attack creates a situation where the end user is suddenly being charged the rate calculated for normal usage at the volume of peak usage.

Now, I realize the Internet has an extremely layers of service provides. Many ISPs are just a middlemen paying metered rates. The ISP is caught in the same trap of screwed up billing. The cost of the ISP providers didn't go up during the attack.

The big bills for both the ISP and end user are the result of flaws in the billing and metering processes and not actual higher network costs. The challenge is to keep the charges from the DOS attack from screwing up the billing systems.

BTW, I do not mean to imply in this thread that DOS attacks are cost free. Just that the bandwidth consumed during the attack is really not costing the network that much more. The machines, cables and wires have more stuff going through them. The DOS attacks cost the the support people in the ISP time, and have a cost in lost opportunity, they also create billing nightmares. The DOS attack does not actually cost the real dollar amounts that suddenly appear on bills.