Bandwidth Limiting Policies for Web Hosting? 30
Silas asks: "I run a small website development and hosting company. We're trying to develop creative, fair, but standard policies in limiting the bandwidth of our individual hosting accounts. I seek the opinions of Slashdot readers who have experience as hosting providers or hosting users. More details below. We're running Apache, and have pretty much decided on using mod_throttle as our bandwidth limiting technology. I know it's not everyone's favorite, but it looks great for us. We have less than 200 domains being hosted, all with varying degrees of bandwidth requirements. As you might suspect, we've got our own ideas and have done our own research about the answers to these, but now I'm interested in yours."
"The basic question is 'what's fair and standard' in these areas:
- Our two hosting packages offer 5 GIG/month and 10GIG/month respectively, with the option to upgrade in $5 per 1 GIG/month increments. Other hosting providers seem to be all over the board - what's the average hosting account want/need?
- The policy that seems common is 'allow a certain amount of data to go through in a certain time period, and then start rejecting requests until the end of the time period'. Is that fair? What policies do other hosts use? When is it appropriate to delay the response to a request instead of rejecting it?
- What should the user be able to do automatically in terms of upgrading/controlling their bandwidth usage? If a user is fine with 5 GIG/month but then gets slashdotted, what should their options be (right away, within 24 hours, etc.)?
My solution (Score:3, Funny)
Re:My solution (Score:2, Insightful)
I would think that if you set up creative policies, it should allow for DOS-like conditions as long as they were one-time, and not the fault of the site's owner. Now I'm not trying to flame/troll slashdot, but I'm quite sure the continued reckless linking will continue for the imediate future, and I would hate to be a site owner talking to an IT department after a mid-day slashdoting.
I'd LIKE a cap on my accounts. (Score:4, Insightful)
Re:I'd LIKE a cap on my accounts. (Score:3, Informative)
Re:I'd LIKE a cap on my accounts. (Score:1)
Huh? (Score:2, Interesting)
Re:Huh? (Score:1)
Re:Huh? (Score:1)
Because, one possible answser to my inquiry might be "lots of hosting providers these days offer unlimited bandwidth. Why can't you do that?" or "You charge too much for bandwidth as it is!" While those may be valid answers, I wanted to clarify that we're not operating on a scale where we can make those sort of adjustments as easily, and so I'm interested in answers that focus on managing existing resources better instead of conforming to what may be standard for the big powerhouse hosting companies. I hope that helps.
Put it to the users (Score:2, Informative)
Policies (Score:5, Insightful)
1) Automatic cut-off. If a customer has 5GB/Month, and cannot afford more, make sure their site goes unavailable and they are not billed. You do NOT need to continue service, they understand! Just make sure point 2 is made...
2) Notification of cut-off. If above customer runs out, they want to know! Make sure they get an e-mail, but more preferrably a call. It's important, very important!
3) Options for automatically extending the plan. Make sure that customers have the option to have their bandwidth upgraded (of course at additional cost) automatically. This is something a lot of customers will ask about, the type of customers who never want their site to go down, regardless if it costs them. Many customers think "More traffic, more profits", and if their site goes down due to exceeded bandwidth usage, they will think its your fault.
4) Be upfront with all of these issues. Many providers arent verbose enough with customers, and it ends up with them being confused. By laying it all on the table, they will see the above strategy and they will like it. It gives them options that are very important.
Re:Policies (Score:2)
Re:Policies (Score:1)
Re:Policies (Score:4, Insightful)
Also, ask your customers what is most important to them. It will vary, and you will find out what to spend your time and efforts on to make them happy.
Up the Bandwidth! (Score:1)
Throttling simply does not work (Score:5, Informative)
I assume that you want to limit bandwidth that your customers use because the bandwidth to your server(s) is limited (i.e. you don't have a 100mbit connect to the internet). In this case, Apache modules will not do what you want. Someone puts up a 10mb file. That file gets downloaded and uses up all of your outgoing bandwidth. While it is being downloaded, the Apache throttle module refuses other requests. This is obviously not what you want.
So suppose you end up using Zeus, or find some other way to do real throttling. Now what do you set the throttle speed to? 5gb over a month averages a little less than 2k/sec. Say you set it higher, like 20k/sec, and there are ten connections downloading that user's files (which can easily happen with certain browsers). What does the average clueless user or webmaster think? They don't understand throttling. They just think that the website is slow and that your service sucks.
We would throttle down user's sites when their bandwidth ran out. Customers did not understand that they had run out of bandwidth, even though they were notified via email. They just thought their sites were slow. We received a lot complaints about that.
We found that the best thing to do is to not throttle and to presell bandwidth cheap. Our different packages come with different amounts of bandwidth, ranging from 5gb to 180gb. After that, customers can purchase extra bandwidth (for $0.50/gb). Customers receive a notification via email when their bandwidth is running low and again when it is completely gone. When their bandwidth is gone, we redirect their sites to a page stating that they used up all their bandwidth.
This solution is simple and it works. Customers always know how much bandwidth they have left and can buy more at anytime. We never have to worry about users running up a huge bill and not paying it, since everything is prepaid.
Re:Please tell us HOW you did this! (Score:4, Informative)
It's really not that difficult, if you have a good hosting setup. Our hosting system is designed to be 100% automated, all controlled from our custom written control panel.
We use Zeus' virtual server feature. Every customer has their own virtual server. In Zeus, everything is a virtual server. There is no concept of a main server like in Apache. Thus each virtual server can have its own configuration and be independently controlled.
Each virtual server can have subservers. Subservers is similar to Apache's mass virtual hosting (but better). A virtual server has a subserver directory, which is a directory full of symlinks with the name of the site hostname, pointing to the site content directory. When a customer runs out of bandwidth, we simply place a
If I had to set this up under Apache (ick), I would use the mass virtual hosting [apache.org] module. The virtual host directory would be full of symlinks with the name of the site hostname, pointing to the site content, as with Zeus. The difference with Apache is that all sites for all customers would be in one directory. Apache configuration is much more messy and not easily scriptable. To redirect the disabled sites, the symlinks would be changed to point to a different directory.
Accumulating bandwidth statistics for this can be done in different ways. We use a custom ISAPI module which sends logging info through a socket to a daemon which accumulates statistics and periodically saves them to a MySQL database. Doing a SQL query for every web request would not work.
For Apache, a similar module could probably be written. Or server logs could be processed. Logs could be written to a pipe and the process on the other end could accumulate bandwidth and other statistics (or write out logs for something like Analog to process).
A script runs from cron that processes the bandwidth statistics in the database. It updates the bandwidth counters in the database for the user, making it easy to see how much bandwidth is left from the control panel. When a customer is low or runs out, the script sends a notification email. When the customer is out of bandwidth, the script updates a table which tells the backend hosting daemon to actually disable the sites. After the customer buys more bandwidth, the script updates the table telling the daemon to enable them again.
(Hint: SQL databases are a very nice way to let daemons on different servers communicate with each other.)
Re:Throttling simply does not work (Score:1)
FYI, the Sun Cobalt [sun.com] RaQ appliances allow you to set througput limits in kilobits/second (low=10, high=??) per IP address (that's for all IP traffice - FTP, HTTP, mail, etc), known as "Sun Cobalt Bandwidth management"... (It's done via a kernel module...) No auto-cutoff after x GB/time period though, just setting the pipe size...
Re:Throttling simply does not work (Score:1)
Simple solution (Score:2, Interesting)
Seriously, your a small outfit, if you meter the client's usage you can charge accordingly. Write it into the contract about how much you charge.
If you somehow get a spammer or a Warez site, they make themselves quite obvious if YOU AEW paying attention to bandwidth, have yourself paged when a user exceeds a certain amount and look at what they are doing.
If they got slashdotted, fine, but if your seeing a lot of SMTP going out or large files getting downloaded yank them quick. Upstream vendors don't care about what's coming into your sites, it's what's going out that they charge you on.
mod_throttle.. (Score:4, Interesting)
I tried to setup mod_throttle for a site where the requirement was 'No more than 1gig per hour'.
I actually found this incredibly hard to do, it's fine for slowing down the incoming requests when lots of them start coming in, but hard to make it keep track of total traffic served.
I think it should be fairly straightforward using the 'volume' directive, but I could never quite get it to work out properly.
My solution was mod_curb [freshmeat.net] which is dedicated to just doing that. (It doesnt handle virtualhosts yet, but it will do by next weekend.
My experiences from a user perspective (Score:2, Insightful)
What I guess I'm practically saying, is try for 'soft-limits'. If you have a 'hard limit' of 5Gb/month and someone uses up 5.01Gb in a single month (previous months were less than 2Gb) - would you want to loose their custom? Ok, if the next month is also 5.01Gb (or higher) - then it's "contact customer" time - but just "arbitarily shutting down sites" is, IMHO, not a good idea (unless, of course, they are causing _significant_ harm to your business: ie saturating more than 60% of your pipe on their own - but you should have really noticed that before hand _or_ check the source of the referers: it may just be a spike for an hour or two due to slashdot or similar).
Calculating monthly transfer needs ... help! (Score:1)
Let others buy bandwidth as well (Score:1)