Morality of Throttling a Local ISP? 640
An anonymous reader writes "I work for a small (400 customers) local cable ISP. For the company, the ISP is only a small side business, so my whole line of expertise lies in other areas, but since I know the most about Linux and networking I've been stuck into the role of part-time sysadmin. In examining our backbone and customer base I've found out that we are oversubscribed around 70:1 between our customers' bandwidth and our pipe. I've gone to the boss and showed him the bandwidth graphs of us sitting up against the limit for the better part of the day, and instead of purchasing more bandwidth, he has asked me to start implementing traffic shaping and packet inspection against P2P users and other types of large downloaders. Because this is in a certain limited market, the customers really only have the choice between my ISP and dial-up. I'm struggling with the desire to give the customers I'm administering the best experience, and the desire to do what my boss wants. In my situation, what would you do?"
Re:bill, don't throttle (Score:5, Informative)
I agree, but with the caveat that you have to do what your boss tells you to do. By all means, present this idea to the boss, but be absolutely sure that you are complying with the requirements of the job you are assigned: after all, in this economy, you do not want to give your boss a reason to fire you.
You will definitely have to consult your boss about this, and you would be remiss in not telling your boss to send the TOS to your company's attorney and have him advise on the legalities regarding whatever plan you and your boss ends up deciding on. You don't want your company to get sued and you don't want anyone to say it's your fault because that would be another reason you might get fired.
In the end, look over the TOS, and if your boss asked you to shape it and shaping doesn't meet with the TOS, by all means CYA and ask your boss to send his request to you in writing. Preferrably signed. Digitally signed e-mail might be okay, too. Just make sure you have some proof of what you were ordered to do, because you want to be sure if there is any fallout from the shaping that you can prove you were just doing as ordered.
It bears repeating so I'll say it again: always CYA.
400 / 70 = what? (Score:4, Informative)
I have no problem with you scheduling low-latency traffic over filesharing traffic, filtering, or whatever, but it seems a little short-sighted that it only takes 5.71 users to completely muck up your network. (I.E if you sell 1mbit connections, you could "theoretically" support 420 customers on a 6mibt pipe (6*70=420 at a 70:1 oversell ratio).
Make what the customer wants available... (Score:3, Informative)
I think P2P is servers used this way are a great tool helping ISP's reduce their upstream bandwidth costs. My ISP does it and, for example, has mirrors of Fedora and Ubuntu update repositories plus a whole library of popular downloads that I don't get charged for if I use their servers to download (and it's faster too). Furthermore their servers will download files via P2P and make that available to all their other users.
Re:bill, don't throttle (Score:5, Informative)
$1 per GB is a little steep, isn't it?
These guys [teksavvy.com] only charge $0.10/GB.
Your suggestion seems like the best way to go. Up here, Telus(big ISP) has caps at 10GB, 60GB, 100GB per month based on how much you're paying.
very few cable companies contract to provide that (Score:5, Informative)
I've never seen a cable-ISP contract that provided service at a specified rate in Mbps. You can get those contracts as a business user, but they're not the standard ones home users have. Usually home contracts say something along the lines of "up to xx Mbps; actual speeds may vary and are not guaranteed".
Re:400 / 70 = what? (Score:2, Informative)
Put that way... you do need to do throttling, but throttling is not enough, you also do need to increase the amount of bandwidth immediately by about 60%....
Re:Is throttling really cheaper? (Score:3, Informative)
Throttling is dirt-cheap.
I work for an ISP that's probably comparable (wireless, so each connection is slower than a cable connection, but there's more of them). If you want to roll your own stuff, a juicy PC with two network cards and some layer-7 rules should be doable for under $1000.
You can also buy one of these [mikrotikrouter.com], and configure it to do the shaping for about $1500, if you want a sexy rackmount unit and support.
They work quite well for basically everything except encrypted BitTorrent (and I'm sure that's just a matter of time).
Re:bill, don't throttle (Score:3, Informative)
I haven't looked at the site you linked to, but my experience has been that hosts that offer really low per GB prices typically either A) only offer such a price with an already expensive base package where there is already a really nice profit, or B) host for a LOT of high bandwidth customers, and it's more like dealing in bulk. It might be that neither apply in this scenario (for a small ISP, I'd suspect not).
Re:bill, don't throttle (Score:5, Informative)
Minor point, but it was an FCC hearing against Comcast [gigaom.com] not a court case. Part of the problem was that Comcast ran around terminating connections behind your back -- and without notifying customers via TOS or any other method.
When it comes to throttling, seanadams had it exactly right: you have to provide the auto-throttle option so that people don't get slammed with a huge bill at the end of the month. Very few people want to sit around adding up their monthly bandwidth usage, so it's a good idea to start warning users as they approach the limit. Unless, of course, slamming people with a huge overage bill is part of your revenue-maximizing business model.
Re:bill, don't throttle (Score:3, Informative)
Astraweb sells 25GB for $10 (that is their most expensive rate). They probably deal with some pretty serious volume, but they will meter that 25 GB across months.
To Throttle or not to Throttle (Score:2, Informative)
What to do (Score:3, Informative)
In my situation, what would you do?
First, at 70:1 oversubscription there is no bandwidth shaping policy which will improve the user experience, so you'll piss off the top 10% of your users without making the other 90% any happier.
I'd explain to the boss that the accepted norm for residential oversubscription is 10:1 and that oversubscription rates in excess of 20:1 flat out don't work. You either need to increase your system bandwidth reduce your subscriber bandwidth. In other words, you either buy more T1s at the head end or you drop those 5 meg lines to 768kbps and be honest about it.
Next, implement traffic shaping for ports other than UDP 53, TCP 22, 25, 80 and 443 during the prime time hours on your graph. You'll piss off the torrent freaks in the top 10%, but oh well.
Re:bill, don't throttle (Score:5, Informative)
I thought the problem was also because of they way they did the dropping the connection (via a "reset" [arstechnica.com]); they "masqueraded" as someone else which is a no-no under the law.
Here's a simple diagram:
A is downloading from B.
C (Comcast/ISP) "throttles" by telling A that it's B and makes the changes that way.
Essentially, a "man-in-the-middle" situation.
If Comcast was some poor sap, it would be in the federal pokey...hopefully without soap on a rope.
Re:bill, don't throttle (Score:3, Informative)
Re:BS. (Score:5, Informative)
The solution for better or for worse is for the US to implement download caps like the rest of the world. It'll be unpopular and it'll have disadvantages, but laying cable still costs money and the current all you can eat payment schemes just don't work.
No, the solution is for broadband providers to do what they've already gotten billions of taxpayer dollars to do but didn't, build out broadband. These companies are trying to double dip, first take taxpayer money then bill customers more.
Falcon
Re:bill, don't throttle (Score:5, Informative)
One particular Australian ISP I was looking at, I forget who. It may have been Dodo or something, it always seems right to blame Dodo for these things, sold ridiculously low download caps (in the less than a gigabyte range) coupled with reasonable speeds (so as to very quickly eat the allotted cap up), and charged excess usage at 10c per megabyte. And they had the audacity to throttle usage after the cap was exceeded.
I've said it before and I'll say it again: If you think your ISP might not be using lube when it fucks you, try spending some time in Australia.
Re:Screw morality. Get pragmatic: prioritize traff (Score:3, Informative)
Morality is a tool for the herd to feel more important than their leaders.
Who would have guessed that Dick Cheney was posting on slashdot?
Re:shape and/or prioritize that traffic (Score:3, Informative)
My little neck of the woods cable in Michigan is 500K. Yes, 1/2 a meg. Sparsely populated areas just are not profit motivators for rolling out new services. If you have a few lakes of say, 50 houses per lake, but only 2 are year around residences, the people aren't willing to pay what's required to have broadband.
Heck, when I sold off my dialup ISP years ago I kept a few T1s to do a small area WISP so I would have something near reasonable speeds for myself. I'd love to have something cheaper even if I didn't provide it myself.
Much of rural America can be out of reach of broadband and the telco monopolies make it very hard to do much about it.
What are you selling them? (Score:3, Informative)
A contention ratio of 70-1 is really high. What exactly are you selling your customers?
Most ISPs around my part run on an contention ratio of between 20-1 and 50-1. In practise it sits closer to the 20-1 than the 50-1. At 70-1 I'm not surprised that the pipe's constantly full: it's twice what it should be.
Unless, of course, you're selling an 'lite' package. But as you've got an monopoly, it sounds like you're probably selling an 'lite' package at 'premium' prices.
Re:bill, don't throttle (Score:3, Informative)
Your analogy would have been written more accurately like this:
A - Peer
B - Comcast, A's ISP
C - Peer
A and C are communicating through B. B doesn't like the traffic, so Comcast tells A it is C and to kindly please STFU we're done talking to each other.
And you're right, if an individual did this, they'd be in prison by now. It's only ok because it is Comcast doing it to their own customers and this isn't misrepresentation, it's creative traffic management.
Re:Do it by usage, not by protocol. (Score:5, Informative)
HTB is Hierarchical Token Bucket, a CBQ (Class Based Queueing) discipline for Linux. It lets you create a hierarchy of queues for a network link. The "Token Bucket" part means each leaf and node in the tree has a "bucket" that constantly, slowly fills with tokens. Sending a byte removes a token. So, on average, you're only guaranteed the fill rate, but if you haven't used it for a bit, you can send a burst until your bucket is empty. Extra tokens can be borrowed between nodes if they're not used by the others, up to the max rate. Thus you get minimum guarantees, max limits, and bursts, such as being able to quickly fetch a web page even if the link is full from others' usage, if you haven't used up your tokens.
For instance, you could have Customer A, Customer B, and Customer C at the top level, and then they each have a second level of HTTP, BitTorrent, and SSH. Customer A and B get a rate of 128k, and C gets 512k since he pays extra as a business customer. They all have a max rate of 6M, since that's the speed of their DSL lines, and a burst size of 1MB. Then, they have SSH (with a small rate and a small burst), HTTP (with a high rate and a large burst), and BitTorrent (with a 1k rate, and a small burst).
As long as Customer C isn't using any bandwidth, A and B can use it all. As soon as C wants to use some, he first gets his guaranteed 512k - no matter what - and then they all split any leftovers in proportion to their committed rates (So A gets a share, B gets a share, C gets four shares). If C only wants 512k, A and B each get to split all the leftovers evenly.
If A is using BT like mad, but then opens an HTTP connection, it'll be allowed most of his net connection (it has a high rate, but still lower than the full line speed). BT will automatically (and instantly) be throttled until HTTP is done. When he types on the SSH connection, it'll use little bits of its burst speed to refresh the window instantly, but its small rate won't let it consume the whole net if he accidentally cats /dev/urandom.
Sounds great, right? There are a few gotchas: You can only queue packets like this when *sending*. What're you going to do, receive a packet from the slow link and then delay it before sending it over the fast one that's not saturated? (Well, yes, you can, and it makes a limited amount of sense to fine tune TCP's flow control, in addition to selectively dropping packets to make it back off, and other tricks.) It's good, but it doesn't necessarily make optimal tradeoffs between latency and bandwidth - HFSC is an attempt to address this. Also, this is a moderately heavyweight way to do things. It has to spend some CPU classifying packets, and memory to track the buckets' state, so other queueing disciplines and schedulers exist that work on other methods (such as statistical, instead of discrete tracking), that are more appropriate for very large ISPs. Also, as a large ISP, you're going to be using Cisco, not Linux, for routing. :) But Cisco has sophisticated QOS as well.
Despite how complex this sounds, even using the simplest case on your home router will make a huge improvement in the weak side of your DSL line, the uplink. Several of the open source WIFI router firmwares support it out of the box for this reason. I have survived having my web site on my DSL linked to the front page of a popular site known to bring servers to their knees, without any lag in SSH or games, or interruption of mail or other services. We only noticed because our bulk transfers slowed to a crawl, as intended.
Learn more:
HTB: http://luxik.cdi.cz/~devik/qos/htb/ [luxik.cdi.cz] (the user guide has a good overview and pretty graphs)
HFSC: http://linux-ip.net/articles/hfsc.en/ [linux-ip.net] (More pretty graphs and good explanation)
Linux Advanced Routing and Traffic Control list: http://lartc.org/ [lartc.org] (The howto is out of date, but very enlightening)
But we're talking a 70X oversubsctiption here! (Score:3, Informative)
Re:But we're talking a 70X oversubsctiption here! (Score:3, Informative)
I'm sure usage for the average customer has gone up since I was in the business, but 6:1 to 11:1 was typical oversell on local capacity and (assuming you hosted your own mail servers on that local network) 4:1 to 6:1 was a reasonable local to backhaul oversell to actually never peak the backhaul lines or to peak them rarely for a few minutes at a time (like for 5-10 minutes at a time between 4pm and 10 pm the day a hot new patch for a major game came out).
At the time, a DS-3 at 6 Mb/s 95% burstable to the full 45 Mb/s ran between $2,500 and $7,500 a month depending on the market. Let's say conservatively you're using a 6:1 local usage ratio and a 25% of local users hitting the backhaul line at any given second. That means a 6 Mbps burstable line could cover 144 Mbps of sold bandwidth. If you're selling that bandwidth to 400 customers, they only get 0.36 Mbps each even assuming old usage data.
A quick Google search for "DS-3 pricing" shows a full DS-3 for $2200. That means 45 Mbps, so overselling that based on 5-year-old usage data means an ISP could sell 1080 Mbps. Across 400 customers, that's 2.7 Mbps each for $2,200 a month.
What are these customers paying for Internet service? $50 a month? That's $20,000 a month in revenue. So lease two DS-3 backhauls, and assume they're double the rate I found advertised. You'd still only be paying $8,800 a month for bandwidth. For three at that inflated price it'd be $13,200 a month for the lines.
With 3 DS-3s, a 4:1 local usage oversell and a 3:1 local usage to backhaul oversell, you'd have about 4 Mbps service and would be making $6,800 a month to pay other expenses to offer the service. The cable infrastructure is already paid for out of your cable TV service, I'm sure. If anything was left after that, then that'd be profit. Considering I doubled the advertised price of the line and most providers give a handsome discount for ordering multiple lines, this cable company could likely still make a few thousand dollars a month on Internet access service while offering a fairly modern customer experience.
This company by overselling 70:1 is raking in excess profit now at the expense of losing these customers when someone who cares about the customer's experience opens shop in town. The problem with overcharging a captive audience because you can is that they'll remember it when they are no longer captive.
Right now, unless the advertised speed that's being oversold so aggressively is in the tens of megabits per second, the customers are just as well off with satellite at 40 kbps and high latency, and aren't much better off than using dialup.
Re:bill, don't throttle (Score:1, Informative)
They're not different from the guy who rents his cabin to 3 dozen different people for the summer, hoping that no more than one will show up at a time.
Or airlines that sell 50 tickets on a plane that only has 38 seats.