Forgot your password?
typodupeerror
News

Breakdown of Bandwidth Costs? 246

Posted by Cliff
from the who-pays-whom dept.
WCityMike asks: "What is the origin of the cost of bandwidth? For instance, if I'm being charged for an apple, I know that, theoretically, the cost of that apple is going towards the purchase of apple seeds, the land on which the apple trees are grown, the fertilizer and water that helps the trees grow, and the salaries of those who pick the apples, clean them, box them, and send them to market. When an Internet provider charges someone hundreds of dollars in bandwidth costs because they were Slashdotted (or Farked) and their bandwidth use shot up, what costs have the Internet provider incurred, and why does it cost them what it does? Is there usually any sort of markup going on along the line, or are people just passing along their own expenses down the line to the end user?" It would be interesting to note the most important factor contributing to bandwidth costs. How much of the total costs are tied to infrastructure versus the human component (technicians, sysadmins, technical support and so forth)?
This discussion has been archived. No new comments can be posted.

Breakdown of Bandwidth Costs?

Comments Filter:
  • Disincentive? (Score:3, Insightful)

    by David McBride (183571) <david+slashdot@dwm. m e .uk> on Sunday January 05, 2003 @10:15AM (#5019388) Homepage
    The high metered cost of bandwidth may partially be to encourage an ISP's customers not to be gratuitous with their use of a shared resource rather than from a need to recoup increased costs due to load.
    • Re:Disincentive? (Score:2, Interesting)

      by miratrix (601203)
      The fact that services like congent communication [cogentco.com] exist means that even with shared resources, companies can make money off people while providing unlimited service (of course, the question now is whether cogent does in fact make money providing non-metered connections).

      However, the cost probably adds up when you need mission critical servers requiring connections to several backbones. Now, you need to take into account several links to several service providers as well as the extra hardware, etc. I think that's the reason why brand-name colocation/hosting bandwidth is so much more expensive as compared to, say, getting a quote for a T1 to your basement.
    • Re:Disincentive? (Score:5, Informative)

      by gremlin_591002 (548935) on Sunday January 05, 2003 @10:59AM (#5019517) Journal
      Perhaps, but when (a long time ago, in a galaxy far, far away) I ran an end user ISP. I got hammered once by a customer. His box was rooted and used in a denial of service attack.

      Anyway, my bandwidth was paid for. I had a full T1, Sprint didn't care how much traffic I sent across it, they were set to cover it.

      He was very sorry about it. It cost me about 2 hours of downtime for the rest of my customers. It took me about an hour to respond once I found out there was a problem and pulled the plug on his box. I issued refunds to customers that were affected.

      I charged him the cost of those refunds, plus $35 for an hour of my time.

      Note that this probably didn't cover all of my costs. I wasted most of a day adjusting bills of those affected, which I didn't charge for. The agreement I had with this customer was special. He was allowed to use all extra bandwidth for his peering box, but was not allowed to infringe upon the rest of the customer's requests. This was before QOS packet stuff so it was kind of a manual 'keep on eye on the MRTG graphs and make sure were aren't maxed out' kind of arrangement.

  • by spacedx (458227) on Sunday January 05, 2003 @10:21AM (#5019407)
    I had always thought that if you were being charged for an apple, that cost was actually going towards Steve Jobs' personal Gulfstream V jet?
  • Not So Complicated (Score:3, Insightful)

    by MimsyBoro (613203) on Sunday January 05, 2003 @10:21AM (#5019408) Journal
    The reason is that ISPs are commercial entities. Shockingly they are in buisness to make money! It's all about supply and demand. There is a limited amount of bandwidth a connection can server and many many more people who want to use it. They couldn't charge users a one-time-fee because then their revenue would slowly die off.
    They couldn't charge users a fixed-flat-monthly fee because then one (or a few) users could take up all the bandwith (terabytes of data) for a flat fee, thus bringing the ISPs servers to a halt.
    The pay-per-bandwidth model secures the ISP a fixed amount of revenue as long as it has demand, and it does.
    • by onenil (624773)
      I'm taking a bit of a stab in the dark here, but I think this guy actually wants to know *what* he's paying for - so he can perhaps engage his ISP in some informed discussion so as to bargain the costs to him incurred from say, a slashdotting, down. Not all ISPs out there would be bastards, let him be the judge of whether the ISP will engage in this conversation or not. Don't patronise him for raising the question.
    • by SerpentMage (13390) <ChristianHGross.yahoo@ca> on Sunday January 05, 2003 @10:54AM (#5019505)
      I have been reading in business week that the telco's have only utilized about 15% of all the capacity that they have. So now comes my question, what gives?

      Here in Europe (Switzerland specifically) it used to be fixed bandwidth and extra costs. Now most providers for a higher price are giving unlimited. And it seems to have worked.
      • by warpSpeed (67927) <slashdot@fredcom.com> on Sunday January 05, 2003 @11:12AM (#5019567) Homepage Journal
        I have been reading in business week that the telco's have only utilized about 15% of all the capacity that they have. So now comes my question, what gives?

        That is probably 15% of thier fiber capacity, not total capacity. The other cost of hooking up the fiber is the routers and integrating it into the "net". The reason is that the price of getting bandwidth is so low now that is not cost effective for them to hook the fiber up.

        Counter-intuitive? The fiber is only one part of the cost and it is fixed since it is in the ground alredy. There is the capital required, and the human cost to support hooking up and maintaining the fiber. If they hooked up all the fiber they would drive down the costs of bandwidth even more, and thus lose more money. So they leave the fiber dark until it make economic sence for them to light it up.

      • The reason is that although the prices are set on the supply/demand model since laying out new fiber is SOOOO expensive the ISPs perferred to lay a lot more fiber then they need. In order to not overflow the market with too much bandwidth thus causing the price of bandwidth through the floor the ISPs only use a certain portion of it. Think about it in terms of inflation. If the ISPs released too much fiber the price of a megabyte of data would lower to the point where the ISPs won't be making any money. Hence they leave a certain percent of the fiber untouched till demand grows.
      • Yeah, it's probably true that on any given day there is only about 15% utilization. That is averaged throughout the entire day. I'm responsible for maintaining 4 channelized DS3 (broken down into to DS1s (T1s), and then DS0(ISDN)) for voice. And our company averages about 10-20% utilization for the entire day. But on peak days and at peak hours we can max the whole thing out. Even maxxing out all of our trunks (aprox 2500) for an couple of hours we'll still probably only make about 50% utilization. (That's extremely high in the Telecom world).

        So, when you hear that telcos are only using 15%, that's because they need the other stuff for peak times. Wether they are foreseeable or not.
      • About the 15% capacity. They could be referring to fiber capacity, as was already brought up, or they could simply be referring to their actual capacity. I run a web serving company, and even at peak hours I only use 30% of my line. The reason for this being that nobody wants to be served by a server that's running at max capacity. It's dreadfully slow.

        My rule of thumb is if I'm nearing 50% capacity, I buy more bandwidth for my customers to use. So far I've never received a complaint.
    • by TheSync (5291)
      Another issue is thta moving bits over fiber at gigabits per second is relatively cheap. Routing at gigabits per second is significantly more expensive. Both are getting cheaper thanks to Moore's law.

      I regularly see DS3 quotes for $3k/month, I remember when it was more like $30k/month, and before that it was just nuts.

      But last-mile is still a problem. I recently priced out a DS3 network to 180 locations in the US. The biggest cost was last-mile connectivity.
    • by misleb (129952)
      The reason is that ISPs are commercial entities. Shockingly they are in buisness to make money! It's all about supply and demand. There is a limited amount of bandwidth a connection can server and many many more people who want to use it. They couldn't charge users a one-time-fee because then their revenue would slowly die off.

      Perhaps this is how it works for a colocated customer. But for most end users (buisiness and residential), the majority of the monthly cost goes to the line charges from the phone company and other infrastructure costs. There isn't really that high of a markup, at least for us, on the cost of, say, a T1. Not once you add in the cost of our DS-3's/OC-3. Then we have to pay for OUR upstream connection.

      They couldn't charge users a fixed-flat-monthly fee because then one (or a few) users could take up all the bandwith (terabytes of data) for a flat fee, thus bringing the ISPs servers to a halt. The pay-per-bandwidth model secures the ISP a fixed amount of revenue as long as it has demand, and it does.

      Again, maybe in a colocated situation, but ISPs could NEVER charge a one time fee unless that fee covered the estimated line charges for the length of the contract. Really, "bandwidth" doesn't cost anything. The only real cost is the infraastructure. And then there is a markup (like with any product or service) so that the ISP can make some money. Actual bandwidth used is only factored in to make the people who use the infrastructure more pay more to offset cost of maintaining things. Usually its not factored in at all, at least for us, since the majority of our cost is in delivering the physical connection.

      To answer the original question, for us, as a T1 reseller, the majority of the cost is line charges from the phone company. Thats the cost of the loop and a fraction of the cost of our OC-3. Then we markup maybe 30% to pay for our upstream connection (which may be broken down differently) and to pay for administrative costs. We don't make a whole lot of money in the end. Really just enough to stay in business.

      -matthew

    • by billstewart (78916) on Sunday January 05, 2003 @09:04PM (#5022508) Journal
      For big ISPs, it's *terribly* complicated, because supply and demand are related to marginal costs, and to fixed costs, and to prices charged by the competition, and to equipment amortization, and all of these things are getting yanked around rapidly by technology change; the only people who have a clue are the small ISPs who have to buy most of their services from other people, but even they can keep haggling down their prices a bit. There are people out there charging ~$50/Mbps for bandwidth, and doing peering for free, and charging $500/Mbps for bandwidth, and that's just in the US - once you start crossing oceans, or national boundaries, it gets far weirder, except in places that haven't liberalized their telecom and have oppressive stupid PTTs who can tell their customers what they have to pay.

      Colocation centers and Dense Wavelength Division Multiplexing are to blame for most of this, along with Moore's Law making computers (and therefore routers) cheaper. A decade ago, it cost a lot of money to put fiber in the ground, and the fiber had a fixed capacity (typically 1.7gbps), and you could get some economies of scale by putting a bundle of fibers in the same trench, since the cost of the fiber was much less than the right-of-way or installation cost. DWDM lets you run many wavelengths on the same fiber. In ~1996, this meant 8 wavelengths of 2.4Gbps; today it's 80-160 wavelengths of 10 Gbps. Adding more capacity still costs you money, because you need to add repeaters in the middle and expensive electronics at the edges, but the costs of those keep dropping and repeater distances keep growing, so it makes sense to buy the edge equipment you need now and upgrade later when you need it, since the prices of the hardware are diving fast. The marginal costs on a backbone fiber aren't zero, but they're pretty close.

      Colocation centers also change the economics radically, because they can buy a small number of fat pipes, which have much lower price per bit than skinny pipes, avoid paying the telcos much for access lines, because they can be located near big ISPs, can support thousands of machines in the same location, and to the extent that their customers are sending bits to each other instead of the outside world, can provide lots of bandwidth on very cheap LANs inside their buildings instead of to an ISP who charges them money. Before Exodus fell apart, there was a huge ecosystem of providers who sold services to each other inside their colo spaces, and some of the other hosting providers did that also.

      A related business is carrier-neutral colo centers like Equinix or the Seattle Westin Building or LA's 1 Wilshire, which convince a bunch of telecom and ISP carriers to build connections into their buildings, rather than selling to hosting users, which lets the carriers connect to each other using cheap interconnects. So a big ISP would buy an OC48 pipe into the building from an access provider, buy a rack to put their routers on, and then use Gigabit Ether to connect to the other ISPs. Also, they can put a telco fiber into the center, so if a small customer wants a T1 to one of the carriers, they can just add the circuit onto the fiber. But how do you price bandwidth in this environment? For the colo company, it's easy; they're selling real estate and charging a flat fee for fiber patch panel connections (how they make money doing that is a different question, but at least they know their costs, so they can do prices that aren't artificial.) But for the carriers, is there any reason to run a peering connection at 100 Mbps vs. 1Gbps? Not really; their router hardware costs a bit more, but it's still a drop in the bucket.

      Then there's Dark Fiber access and Dim Fiber access. It's not as common as George Gilder predicted, but there are places where it's available. The basic model is that the access provider isn't messing with routers or switches; they're acting more like a construction company, installing fiber and renting right-of-way (or conduit space) and letting the customer light it up themselves at whatever bandwidth they feel like. Sometimes this is a hybrid business - a provider might have a big lit ring around San Francisco Bay and sell wavelengths on the ring connecting to dark fiber on the last mile, or alternatively they'd have a big bundle of hundreds of fibers around the Bay and do patch panels. Some of that is technology dependent, because of distance limitations for different speeds, and therefore it's also influenced by geography - anything inside Manhattan doesn't need a a repeater, or from Manhattan to nearby parts of New Jersey where the real estate's a bit cheaper and there's less backhoe risk, but some distances in the San Francisco Bay Area are a bit too far for repeaterless operation at some speeds, and Los Angeles is really huge. Also, while fiber technology used to be just for long-haul telecom or lower-cost LANs, the Storage Area Network people have recently developed technologies like Fibre Channel for connecting disk drives to CPUs at 20km distances - there are starting to be a lot of banks and brokerages in Manhattan that have disk farms in New Jersey colo centers, and instead of their mainframes being all in the same building as the traders, they may be spread out geographically, often with primary capacity in one location and backup in another.

  • Cable ships (Score:4, Interesting)

    by IamTheRealMike (537420) <mike@plan99.net> on Sunday January 05, 2003 @10:22AM (#5019412) Homepage
    Cable ships are a large part of the cost of bandwidth, most of the major western top level bandwidth providers maintain fleets of undersea cable maintenance ships. These things are fantastically expensive to build and run, and part of the cost of them filters down to all of us. For instance, part of my £20/month connection costs are helping to pay for the undersea link that connects me to slashdot. Ditto for satellite/microwave links, they all cost a lot to maintain and run.

    That's not the only source of expense of course, but it is one major one. Don't forget supply and demand of course - people charge what people are willing to pay.

    • by zephc (225327)
      If they simply used an Interocitor [shipbrook.com], a device that can lay roads at a mile a minute, it can of course also lay cable, cheap! DIRT cheap! Just gotta get those folks with the funny, large heads to sell em....
  • by kraksmoka (561333) <grant&grantstern,com> on Sunday January 05, 2003 @10:28AM (#5019429) Homepage Journal
    well, kind of like an apple, it all starts with the seed money from unwitting investors who realize not that their hard earned money is about to be thrown in a hole in the ground. then companies hire construction folks like Mas Tec (Miami, Fl) to quickly and quietly plant the seed money along with some fiber optic cables, or repeater boxes with really expensive chips, or whatever the potential bandwidth provider needs to throw into the hole to make some /.'ing possible. then, once the investor and the company throw all their money into said hole, a wonderful economy based on advertising emerges and the bandwith is carried to market by all the mom and pop bandwith growers and sold everywhere from local farmers markets to big grocery stores. UH OH, advertising didn't work! now we have to pay the saps, er investors, who gave us their money to pour into our special hole, something called a return on investment in a hole. ok, we'll either declare bankruptcy, or now, we'll charge alot of money for bandwidth for a while since our humble little hole grew into a mountain of debt.

    and everyone lived happily ever after (except for northpoint, directv dsl, worldcom and anyone else who didn't own a regional telephone monopoly to cover idiotic spending levels and tremendous waste). the end

  • by Anonymous Coward on Sunday January 05, 2003 @10:28AM (#5019430)
    ...the problem is burst. Period.

    When you buy an apple, you can eat the whole apple at once (tossing the core into the compost pile when you're through), or you can cut the apple up into slices and eat those slices over a few hours' time. If you eat the entire apple in one sitting, it doesn't cost any more than if you'd cut the apple into 16 slices and eaten one per hour.

    Bandwidth isn't like an apple.

    Bandwidth is purchased in one of three ways (or sometimes several or all three ways at once):

    1. A connection with a maximum of X amount of throughput at any given time, "unmetered" (ha ha)

    2. A connection with a maximum of X amount of throughput at any given time, plus a restriction on how much throughput can take place in a given month

    3. A connection with a maximum of X amount of throughput most of the time, which can increase on the fly to some multiple of X for short periods of time, with a restriction on how much throughput can take place in a given month

    The third type of connection is usually known as "burstable." What it means is that the ISP sells you a line which is physically capable of, say, 3Mbps transfer; but they sell you this line with the understanding that 99% of the time you're only going to be using a maximum of, say, 1Mbps of that capacity. You're allowed to "burst" over your 1Mbps average up to 3Mbps.

    Some ISPs restrict these "bursts" to certain lengths of time, some restrict the "bursts" to certain amounts of data (e.g. you can burst to 3Mbps for 1GB at a time). Either way, you're typically given a certain allotment of "bursts" per month. Any more "bursts" and you pay extra.

    When your site is Slashdotted, you generate hella "burst," hella quick, and hella strong. As a result, your hosting company may wind up exceeding the allowable "burst" (either in time or in bytes) according to their contract.

    Unlike the apple - where you can "burst" it all into your stomach at once with no penalty - bandwidth is sold on the contingency that it's going to be used over time. Use it all at once and you get fucked. That's pretty much it.
    • by Anonymous Coward
      You stopped just short from the cause: the same game repeats also one level higher.
      This means that ISP has to shape his traffic (a sum of all users') into some other contract.
      I don't know exactly but I assume that in these contracts "bandwidth" is bound by the same parameters: traffic, bandwidth (speed), pattern, ...

      And they have to discipline the traffic pattern because they all do the so called statistical multiplexing what we normally call overbooking.

      If you eat apples all year long and you get them from your supplier which supplies to whole street, he will be interested in knowing or otherwise predicting (defining, binding) your pattern of usage since he can't store the maximum-worst-case heap of apples because they will rot most of the time.

      The same way of thinking do ISPs. They pay for their uplinks, say a fixed amount for a fixed bandwidth. To get the bet out of this setup, they have to keep the line as full as possible all the time, so they have to overbook.
      On the other side they have to put some limitations on downlinks so that over time there won't be too much traffic all the time.

      It's all about getting the most out of what resources you have.

      Another example is calculating the cost of a mile/kilometer car drive. It's lower if you do 10,000 per month or only 100 per month since you have quite some fixed cost.

      PG
      • Bandwidth costs (Score:5, Informative)

        by scoove (71173) on Sunday January 05, 2003 @12:11PM (#5019779)
        Thought there have been several responses relating to burstable pricing, it's important to understand that burstable models are more of a marketing concept than a network costing one.

        Considering that IP service is in the class of packet sharing telecom - meaning you do not have a fully reserved, "100% yours and only yours" pipe from location A to location Z (end to end), your costing models have to be a bit more complex than adding the sum of the parts. But it should probably start there. Packet share != Overbooking, although packet share networks /can/ permit overbooking to occur.

        To explain costing on a more common tier two or smaller level (tier one costing has much more to do with transport costs and a bunch more variables not interesting here), take your typical tier two or smaller ISP who we'll say is connected only to a single upstream, and that upstream provider is a tier one (multihomed at x+ bilateral and/or multilateral facilities; e.g. Sprint or AT&T Worldnet).

        That ISP will have:
        - an egress transport cost: e.g. a T1 to Sprint and perhaps a local loop cost to get to the tier one carrier; sometimes this is bundled by the tier one provider
        - a facility cost: router, colocation, switching and all the associated things to figure out what to do with customer traffic and send it where
        - an ingress transport cost: this is the transport cost from the customer to the ISP (e.g. T1 to a local frame relay cloud from the customer, then another portion from the cloud to the ISP, or a similar model in DSL, or a fixed point-to-point line)

        Plus applicable switching/routing/data center/network core/backoffice/customer support/billing costs.

        Then the fun comes in calculating an aggregate the ISP wishes to use in predicting how many seats it can sell on its T1 until the plane can get full - understanding that if it oversubscribes the T1, it may have the possibility of bumping passengers (such as slow performance; people getting less than what they paid for).

        (Note: You'd be surprised how many smaller ISPs don't even understand the concept of aggregation - many rural ISPs in our parts think Internet comes in T1 and put 500 DSL home connections on the T1 without thinking twice. "Order another T1? Why? We already have Internet!" Plus there are other reasons they never go past a single T1 - their Cisco 2600 only has two ports (T1 in, T1 out), they don't know how to load balance more than one T1, and fundamentally it's too expensive for them)

        Aggregation becomes the process of determining how to share that cost, since it's a shared network. It includes variables like time of day (business customers will usually demand more bandwidth during business hours; residential after hours, etc.), whether the customer has a bandwidth guarantee in their service level agreement, etc.

        After all of this calculation and checking of assumptions, the ISP may calculate it into a cost per Mbps as we do.

        so they have to overbook. It's all about getting the most out of what resources you have.

        Not always true. Overbooking can be perceived as ineffective loading - an error, in otherwords, to be avoided. Your power utility doesn't run a constantly overbooked network - that'd have serious regulatory and technical consequences. But when you've got very little backoffice expertise in a smaller ISP, they won't know how to develop traffic models and purchase effectively.

        Overbooking can be avoided both by developing good subscriber models and by enforcing them (ugly reality: if you're using 1 Mbps up/down sustained and are paying $35/month, you're not paying for what you're consuming and you'll cause a problem somewhere along the line). Additionally, you're going to see better aggregation and closer matches to "what you pay is what you get" as the last mile part of the business matures. Much of the success of an ISP is in how it buys and how closely it matches what was bought with what was sold - you can't run a $2 all-you-can-eat prime rib buffet, nor can you charge $10 for one that has nothing but dinner roles.

        Really, overbooking and the opposite end of the spectrum (giving excessive bandwidth without charging for it) are mistakes made early in the development of the business. Successful providers will tighten up both ends, providing the service people ordered and paid for, and smart customers will understand that in order to get what they paid for, they shouldn't also expect a free lunch.

        *scoove*
        • it's important to understand that burstable models are more of a marketing concept than a network costing one.

          I disagree. A T1 customer who only averages 1500 Kbps costs less then one who avereages 1 Mbps. An ISP can put about 5 of the former customers on one outgoing T1 while it could only have one of the latter customers on an outgoing T1 line.

          Burstable pricing is a way to ration the outgoing bandwidth. The more you consume, the more you pay.

          • I disagree. A T1 customer who only averages 1500 Kbps costs less then one who avereages 1 Mbps.

            Not sure I follow that reasoning; 1500 Kbps 1000 Kbps? And how do you put five 1500 Kbps averages into a single 1500 Kbps line? IP compression? :-) If their average is a sustained 1500, then they're using 1500 - not 300 (1/5). Perhaps you meant peak? Understand there is not a 1:5 aggregation law you can pull out either. It's a bit more detailed than that.

            I should clarify per my classification of burstable as a marketing concept; there really are two costing models for a packet share network: fixed and measured.

            "Burstable" is a marketing concept for part fixed, part measured. It is even more marketing concept when you discover that many (and initially most) service providers provided it as a short term fix to the problems of price sensitivity and bandwidth consumption cost.

            In fact, many of the burstable products offered at first couldn't be monitored, and their sales persons would even announce this to prospective customers ("it says burstable, but the secret is that we don't yet know how to measure it so you can really use all you want").

            Of course, many carriers have finally figured out how to measure and bill their burstable products, but not all.

            *scoove*

            • Not sure I follow that reasoning; 1500 Kbps 1000 Kbps?

              Typo. That should be 150 Kbps. What I was trying to say was that a customer who only uses 10% of the line will cost less outgoing bandwidth, due to the Law of Large Numbers, then one who averages 80% of the line.

              "Burstable" is a marketing concept for part fixed, part measured.

              That is nothing unique to the ISP industry. Rental cars do the same with a fixed daily charge plus milage charge.

              The term "bandwidth" actually has two components: speed of the link and traffic through the link. The first is a fixed cost. It is the local loop fee that you pay to the telco to run the line from the ISP to the customer.

              The latter is a variable cost due to the packet switching nature of the internet. A residential DSL or cable user who just wants faster down loads of their web pages require less upstream bandwidth then someone running a P2P server. A low usage web site requires less upstream bandwidth then a highly popular website even if they have the same network speed.

              The burstable pricing allows the two costs to be seperate from each other.

        • I can't begin to comment on how accurate this is. As part of a hobby, I began hosting websites for free on a Linux box attached to a cable modem. Demand was big, and I liked what I was doing, so I turned it into a business. However, I had no idea about the costs involved.

          Obviously I couldn't use a cable modem anymore, so I bought a t-1. The costs in that are huge. Local loop charges. Setup fees. Hardware fees. Then, when it's all set up, you still have to pay huge monthly bills. Not cheap.

          And obviously you can't run a web hosting company on a single linux box. Add more hardware to the bill. Now that you've got more hardware, you need more electricity, more UPS systems, more cables, more switches, etc. etc.

          A large part of what people pay for bandwidth is also advertising. The sad thing is, the customer pays for their own courting. When I pay hundreds of dollars to advertise on a website, the money comes from the customers I gain from doing so. You might think you're paying for bandwidth, but you're really paying for banner ads.
          • You'll get much better reliability and probably a lower cost putting your boxes into a colocation center. Depending on the place, you can either buy managed machines, or unmanaged machines, or rack space with an internet connection. Typical prices for a full rack with a 10Mbps Ethernet feed and some amount of bandwidth on it tend to be in the same range as a T1 line, but you're in a building with two or more physically separate pipes to the internet, no backhoes (though more technicians, who are about as dangerous), highly reliable power systesms, and infinite expansion capability. Some colo vendors give you a limit on GB/month (probably a bad deal compared to buying a straight T1), but many some of them charge you for 95th percentile bandwidth peaks, and since you're using Linux machines you can play with different traffic shaping tools to manage that.

            Disclaimer - I work for a big company that sells this sort of stuff, but we also sell T1s and T3s and OC48s and several different flavors of hosting, so I can be pretty neutral about technology tradeoffs

    • by Myxx (21264) on Sunday January 05, 2003 @10:55AM (#5019507)
      I work for a Tier 1 company that does Enterprise Hosting. We use the burstable method and the 95th percentile. You commit to say 1 mbps. As long as you stay under that you know your costs. You can burst 5% of the time the link is up for a month and not pay anything extra. That equates to something like 20+ hours you could burst, which is good for those who take a /. hit.

      I also know that we mark it up so much that I think it is ludicrous, but we can because we can.

    • "Unlike the apple - where you can "burst" it all into your stomach at once with no penalty - bandwidth is sold on the contingency that it's going to be used over time."

      This is the worst analogy that I can remember reading on Slashdot.
    • by Blkdeath (530393) on Sunday January 05, 2003 @02:21PM (#5020434) Homepage
      ...the problem is burst. Period.

      While your summary is pretty decent (as far as layman terms go), the one thing I haven't seen mentioned thus far is peering and transit costs.

      We all know that the Internet is a great mesh of providers of bandwidth, to whom providers of content connect (and oft-times these providers will be one and the same, but I digress). End-users then connect to this mesh and utilize the content provided via the bandwidth links.

      To break it down, let's say we have five companies in a mesh. Ideally, all five companies would maintain a link with all four other companies but rarely does the world ever work out in the most idyllic sense, so we have a partial mesh instead.

      Company A connects to B and E.
      Company B connects to A, C, and D.
      Company C connects to B, D, and E.
      Company D connects to B and C.
      Company E connects to A and C.

      Now we have a problem. Company B wants to transmit data from company E - but they have no direct route. They must transmit portions of their data through A and/or C in order to reach the destination, and vice versa for the return trip. So now a request that involves only two companies has now involved a third party company who really has no interest in this request. To go a step further, we could introduce company F, who maintains a connection with only company D. Now companies A and E must send their transit via no less than two companies to reach their destination. Run a traceroute to some of your favourite websites, large and small, and count the companies your data path crosses along the way. For spice; try probing sites across an ocean or two.

      To accomplish the harmony that is our global Internet, companies (vis; corporations, transit providers) utilize peering agreements. These agreements make statements of amounts of data and/or ratios. The relationship between A and B could involve something along the lines of a 2:1 ratio, where the smaller company is permitted to 'generate' 2 times the amount of traffic for every 1 amount that the larger company 'generates'. In the case of a slashdotting, this could throw that ratio, as well as that of many other interim providers, out of balance.

      Since this extra bandwidth, although provisions have been made for it, is unexpected and can cause other customers of these providers to experience increased lag until it ends. It can also cause technicians to be called in to adjust routing metrics to mitigate the damage caused by the newly opened floodgates.

      Since these top level ("tier 1") providers have to bear these additional expenses, as well as the second level ("tier 2") providers, the cost is translated onto the customer who caused the increase in traffic.

      Now this might seem horrifically unfair, many might think - what about when I'm being (D)DoS'ed? Many (most?) providers have provisions in place for such contingencies, including 24x7x365 staffed NOCs with people trained to, again, mitigate the damage and attempt to trace the problem back to its origin and stop it from causing further harm to the network. Again, this service costs money and is an invaluable service to the function and utility of the Internet. If techs had to be roused at 5AM and drive into work in all sorts of weather, consume copious quantities of coffee, then appraise the situation before getting around to solving it DoS attacks would extraordinarily harmful, to the point of "taking down the Internet" for very, very large segments of its userbase.

      There are other nominal costs that are incurred and have to be accounted for, such as medium (fibre, copper, microwave, satellite, etc.) to transport bandwidth, equipment to switch, route, shape, filter the data, NOCs to manage the equipment, support centres to handle customer enqueries, sales staff to sell the bandwidth, (management to talk about all of this {nyuk}), real estate in which to house these facilities and their respective staff, etc. When bandwidth usage increases, more medium is required, which comes down to more equipment to connect the medium, more staff to manage it, etc. etc.

      In the simplistic sense of 'bandwidth'; once the line is installed and equipment placed at both ends (and paid for), and the recurring costs of the facilities are paid for - you could load that line to wire capacity 24 hours a day for as long as it pleased you. ICMP, NFS, HTTP, whatever traffic suited your fancy. However - as soon as you put a third entity in the middle of that traffic path, the story changes. Most source <---> destinations on the 'net involve two or more interim companies, if not substantially more.

      Bandwidth has just as many tangible costs associated with it as fruit after all. :)

      • A minor note on the DDOS issue - Small internet providers may need to wake somebody up at 5am to stop a DDOS before it does major damage, but Tier 1 internet providers have people 7x24 (and coffee pots 7x24 :-), so the only times they have a slow response to that kind of problem are when the attack is a new technique that's too complex for the night shift (or when you get the dumb phone-answering person, but that can be any time of day.)

        On Peering and Transit - There are several different sets of definitions depending on technical subtleties, but the fundamental difference is that transit costs money, while peering is "free", but has restrictions on what you can do. In the transit model, Carrier A provides services that Customer B wants, so B pays A for them, and A transports B's packets anywhere that B wants; any restrictions are of the form "your pipe is this big" or "if you send more than X GB/month, it costs extra".

        Peering is a connection between two carriers that both think that they get roughly equal value from connecting to each other, so they do some allocation of the costs of connecting their networks, don't charge each other for carrying the packets, and do some monitoring and limitation to make sure they really are getting roughly equal value. A traditional peering arrangement would be that Carrier A and Carrier B are North-America-wide carriers, and they put big "peering point" connections on the East and West Coasts, and A hands any traffic for B's customers to B at the nearest peering point and vice versa. Sometimes the peering points were "private peering", where they build a direct connection, but for smaller peering, they might just both buy pipes into MAE-West and MAE-East. The subtleties are typically about traffic to destinations other than A and B's direct customers. If A has a peering connection to Carrier C, and B doesn't, typical peering policies don't let B use its peering connections with A to reach C; it either has to peer with C, or buy transit from A or C or D to get there. (This is more common for US Tier 1 ISPs than for smaller ISPs or non-US peering points.) Also, there's typically a traffic ratio like 2:1 that defines whether the players are getting equal value from each other or not.

        It's not always that simple technically, and there's often politics involved. Consider a dial ISP and a hosting center. The traffic may be very asymmetric, say 10:1 from the hosting center to the dial ISP, but it may be worthwhile for both of them to peer with each other rather than buying transit from somebody else. The dial ISP gets to offer its customers more content value, and the hosting center gets to offer its customers more eyeballs. Assuming they're near enough to each other for the peering infrastructure to be cheap, it's a win for everybody, right? But if the dial ISP is AOL and the hosting center is Joe's Garage, Joe will probably have to pay AOL for the privilege, while if the dial ISP is Joe's Garage and the hosting center is Exodus or Abovenet, Joe will probably also have to pay. During the boom, of course, the problem was often resolved by one side buying the other for stock, causing the market to overvalue the combined stock even more :-) Other examples of politics - for many private interconnection arrangements, especially in the past, the terms were often secret, because one side didn't want to admit that it had to pay somebody else to carry some of their traffic instead of using their own backbone, or one of the big carriers didn't want to admit that it *wasn't* charging money to a medium-sized ISP because being a "peer" with somebody small makes it look like you're not the 800-pound gorilla, plus it makes other medium-sized carriers think that maybe they can peer with you for free instead of paying you.

        One other common definition of "peering" is "using BGP to connect to each other", which is a description of the technology rather than the cash flow. The Border Gateway Protocol is a routing protocol that's used to exchange information about who connects to what networks, and it's the main way that big ISPs talk to each other. However, it's also used for customer-to-ISP connections even if you're just buying transit, usually for customers who are buying service from two ISPs for reliability reasons. BGP is an amazingly complex protocol with a dozen or so different knobs to tweak to give you very fine-grained control over complex routing situations and different kinds of load-balancing, but if you're not a Tier 1 ISP, the amount you actually need to do to use it effectively is pretty minimal. In the past, BGP only worked on really big routers, but these days you can use even a Cisco 2600 unless you need to get full Internet routes (which burn too much memory), and you can also get Linux Zebra routers to do BGP.

    • You described it well enough that I got a basic idea of your answer. I think you understand ISP bandwidth overbooking pretty well.

      However, maybe you don't know the world of apples.

      Burst costs can be exactly like an apple.

      Normal usage: Take apple, take bite, chew, swollow. Take another bite, chew swollow. Repeat until apple is consumed. Throw out core.

      Burst usage: Take apple, put in mouth. With two fingers, shove hard. Spend time calculating costs.

      >
      > WARNING: THIS IS THE KIND OF THING TO BE DONE
      > ONLY BY PROFESSIONAL WEBSERVER ADMINS, WHEN MY
      > INTENET CONNECTION IS DOWN. FOR THOSE OF YOU
      > AT HOME, DO NOT, I REPEAT, DO NOT TRY THIS AT
      > HOME!
      >

      (And actually, I'm joking about the webserver admins. Please, you guys don't try it either. I am spelling this out for you so that you do not mistake my intent).

  • How we do it. (Score:5, Informative)

    by standards (461431) on Sunday January 05, 2003 @10:29AM (#5019432)
    For my organization, about 45% of the customer's cost goes to pay for bandwidth. The rest is mostly people costs.

    - We provide very limited end-user technical support.
    - We provide a very specific service. We don't try to do "anything that could bring in a buck". I think a lot of ISPs get into the trap of "anything for money"... which results in a LOT of hidden costs. This should be MBA 101 stuff, but it's amazing how trapped people get doing things for zero benefit.
    - Before we implement anything, we look at the potential costs and benefits.
    - Everything has to have a complete process in place before we move it into production. Otherwise, on-going support costs skyrocket.
    - We have a very clear contract with our customers. We trust them; they tryust us; they don't misuse us, we don't misuse them.

    This is simply basic IT business stuff. But most small bandwidth resellers (and many large providers) fail at it.
    • "- We have a very clear contract with our customers. We trust them; they tryust us; they don't misuse us, we don't misuse them."

      Really? Your contracts don't have that "We can kick you off our servers in case you do something we don't like, that we haven't thought of yet."?
      • Clearly we have an obligation to all our customers before any one single customer.

        In any case, talk with your attorney... you'll likely hear that such blanket, unilateral clauses are legally much weaker than being clear and explicit.

        It's better to be explicit when possible.
    • by kfg (145172)
      "For my organization, about 45% of the customer's cost goes to pay for bandwidth. The rest is mostly people costs."

      Which rather brings up the question, "What is the origin of the cost of bandwidth?"

      Maybe someone should do an Ask Slashdot on that. :-)

      KFG
    • Economies of scale (Score:5, Interesting)

      by DABANSHEE (154661) on Sunday January 05, 2003 @01:29PM (#5020106)
      Is the main determiner of bandwidth costs.

      The fixed costs of telcos & cable networks are huge relativelly speaking.

      That's why govt telco utility monopolies (including govt cable infrasture monopolies, as opposed to content) will always have the lowest sustainable costs long-term. There's no way Singapore would be able to provide broadband to everyone on th island at the costs they do without Singapore Telecom (a 100% govt own utility that subsidies tax-payers with dividends every year) having a cable infrastructure monopoly.

      Look how universal cable is in the Netherlands, & at very low per customer costs to the provider. Going by what I was told, basically every house gets a cable connection with all the free-to-air channels on it as part of the Dutch equilivent of the British BBC TV license. So everyone gets the Dutch free to air channels, about 6 channels, including their BBC equilivents, plus they get about 3 free to air channels each from Germany, Belgium, France & the UK. Which means about 15 channels & a cable connection all on the standard national TV license (the Dutch equilivent of the Brits BBC license). The main side effect of this is is that its universal nature makes provider costs low & you don't have millions of TV antennas ruining the skyline (I spent 3 months there 2 years ago & I only saw 2 TV antennas in the whole country). If one wants pay TV they simply subsribe to whatever extra channels they want & it gets piped in on the 'free-to-air' cable with the govt getting a percentage from the pay TV content provider for providing the infrastructure. The whole country comes pre-wired by one single cable network, making things really simple if one wants a cable internet connection

      The US wouldn't have the problem with broadband internet providers going bust all the time if it had a govt telco & cable Infrastructure utility monopoly. As the only way to get cheap prices for broadband sustainably is through the economies of scale of a monopoly.

      Of course this goes for all utilities. Take electricity, the only state in Oz that has had Californian style problems is Victoria, the only Oz state with privatised electricity. Of course private monopolies are bad (legislation's then needed to protect the consumer), but govt utility monopolies are good - if they rise prices too much the politicians get voted out.

      Really the only reason the US doesn't have govt utility monopolies is because of ideology. American capitalist ideology believe's it's bad for govts to provide commercial services. But putting ideology before pragmatism is a losers game. Pragmatism dictates that where economies of scale are king govt monopolies are the go.

      Lets look at the example of Singapore again. Singapore Telecom provides billions to the govt coffers every year in dividends (there-by subsidising the tax-payers). Look at Singapore Airlines, its one of the most successful & profitable airlines in the world. Singapore also has govt owned stock broking firms that run multi-billion dollar stock portfolios in both Singapore & elseware. Plus govts have a legislative & competitive advantage which is the gain of Singapore's tax-payers too. The govt buys up & invests in firms where it helps ST or SA. The govt can legislature in favour of the companies it owns, there-by giving the tax-payer an advantage. Of course it's tough luck to anyone competing against the govt, but fair competition doesn't exist anyware on the planet anyway. The fact is the only purpose of a business is to make a profit, so businesses use every advantage they can, no matter how unfair it is to their competition, so why should govts be different when they are providing commercial services & protecting the tax-payers dividends. Of cours govts have to very circumspect & sef-controlled - go to far & they can drive away investment BTW the Singapore govt also has share portfolios managed through front companies meaning theire's many compaines arround the world, including the US, that are part owned or fully owned by the Singapore govt & they don't even know about.

      Really this is the legacy of that ultimate pragmatist Lee Quan Yew. Singapore doesn't hesitate to use policies of the far-right, centre & far-left if they feel its the right policy for the problem. Letting oneself be constrained by ideology, like the US, is a mugs game. Can you imagine the stink in the US if the govt dared to provided commercial services at a profit & make billions in divendends? Look at the controversy that surrounded Carter lending Chrysler some short term cash. Really the massive growth rates in places like Taiwan, South Korea & Japan from 1950 to 1990, have all come from putting pragmatism 1st & not giving a shit about ideology.
      • Sure, Singtel is great, but you really need to use VSNL services in India until you get this silly monopoly-is-good idea out of your head. And if you want to argue that that's just because they're a third-world kleptocracy, you should go use Telstra for a while - They're no longer quite a monopoly, but they're still almost one, and they've got the reputation of being the First World's Most Clueless Data Service Provider. It took a long, long time to get them to understand that some people wanted to use 2Mbps E1 circuits for single data channels, not for bundles of 64kbps data channels, and they seem to think that volume-capping cable modems is a good idea. After that, I invite you to set your clock back 5-10 years and buy bandwidth or Internet connectivity in still-PTT-monopoly Europe.


        Economies of scale are important, but that doesn't mean that governments have any real advantages. They buy hardware from the same commercial vendors. They can try to hire employees from the same pools of very smart technical people and the same pools of not-so-smart grunt-workers that the free market hires from, but they've typically got much more restrictive internal rules on wages and benefits. They do have some artificial advantages - they don't have to pay taxes, unlike businesses in most countries, and they don't have to put up with clueless business regulators, but they've got the artificial disadvantage that clueless government policymakers have an easier time pushing them around.

        The one real advantage they have is ability to get right-of-way, because any place that has government-run telcos has government-run highways, so negotiations for burying cable along or across highways are much much easier for them than for commercial businesses. (That's a real sore point in the San Francisco Bay Area :-) Similarly, restrictions on above-ground cables are something that commercial telcos often have to put up with that government telcos don't. Also, if the railroads are government-run, it's easier for government telcos to negotiate with them than for free-market telcos, but if the railroads are privately owned, they're on a more equal footing. Similarly for running telco along power-line routes, but that's a relatively new issue because high-speed copper-based telecom and high-voltage power don't always get along, unlike low-speed telecom and low-voltage copper.

        Well-regulated private monopolies aren't much better than government-run telcos - at least in the US, they typically got to make a fixed percentage profit, so they did have incentives to grow their markets by providing better and somewhat cheaper services, and to use internal cost accounting mechanisms that let the pricing be somewhat related to the costs, and when there were social agendas like making business customers pay more to subsidize residential customers, at least they knew the actual costs that they were subsidizing. But they were still big and stupid. And they got to hire some of the best engineers and scientists in the world (shameless plug here :-) because they had more incentive to develop and deploy technology than most of the PTT-dominated telcos (with a few possible exceptions, but not most of them.)

        The big advantage that competitive telecom companies over monopolies (government or not) is that having a market with competitors forces them to innovate, and to understand what their customers want, and to evaluate whether their value-for-money tradeoffs make sense compared to what the competition is offering. Most monopolies simply don't have real feedback, and many of them depend on technological innovation from other countries rather than doing their own.

        The other big advantage that government telcos have over non-government telcos of all types, which they totally failed to take advantage of until cellphones gave them the technology, is the ability to use radio spectrum to build rural communications. They could have done it decades earlier, because fixed-wireless is a much simpler problem than moving-wireless, but they didn't. In the US, the radio spectrum was effectively the oligopoly of broadcasters, with exceptions like amateur radio (which had anti-free-speech provisions to prevent business use of radio) and eventually CB, and the phone companies didn't have enough incentive to develop it to make it worth getting the FCC to let them have the spectrum space. But in lots of other countries, it's different - radio-based telephony would have made sense, and with the government running the phone companies, they should have been able to get permission to use the spectrum without a problem.

        Disclaimer: I work for a company that used to be a regulated monopoly, and is now partially regulated, though mostly unregulated. But these opinions are mine, not theirs.

  • IMHO I guess that the money goes towards the cost of the cables, routers, servers, manpower and all the other costs of running an ISP.
    And I guess that once that's paid for, there is the cost of upgrading.
    And then there's PROFIT

    Just a guess thought
    • IT equipment is usually leveraged...

      They're not going to make enough money to pay back the costs of building an ISP-sized network in a month. They idea is that they'll lay out all that money, and then devote all of the usage fees from the next 24 months to paying the equipment off. For the next few months, they have little costs and the equipment loan is paid off... it's time to PROFIT!

      Of course, eventually the equipment will come to the end of its lifespan and have to be replaced. Rinse. Wash. Repeat.
      • At this point though, the ISP could afford to buy new equipment for a third of the network or something, perhaps getting a deal because the networking hardware company won't have to wait two years for payoff. Wait one year and do that for another third of the network. Throughout, I would imagine profits being enough to comfortable field these costs.
  • by RhettR (632157) on Sunday January 05, 2003 @10:29AM (#5019437)
    I just finished an economics course in college, and ISPs are probably what can be termed "price searchers." I imagine there is certainly some extra cost per extra unit of bandwidth, but largely I imagine this is negligible--the bandwidth is there whether or not it is used.

    As a few other posters have noted, ISPs are out to make money. A price searcher has a set cost that is basically unrelated to the amount of service they sell. However, they are faced with a demand curve sloping downwards: the higher they charge, the fewer customers they get. They simply match up the price/quantity demanded at that price that leads to the highest revenue and thus profit.
    • ISPs are probably what can be termed "price searchers."

      That may (or may not) be true of top tier providers (roughly, those who are big enough to have quid pro quo peering on the backbone). For most others, there is a cost they will incur for the bandwidth and they must pass it on.

      Getting to the question, the cost of bandwidth is a combination of the upstream bandwidth charges, infrastructure such as racks, air conditioning, backup power, cardlocks, security cameras, switches and routers. More bandwidth consumed by a customer means needing more (or bigger) switches and more upstream bandwidth. More servers means more space, air conditioning, and power.

      Then there are the costs for manpower to maintain that hardware and deal with customer support. If you want reliability, you'll face more costs for a border router, an ASN, a block (or blocks) of IP addresses, and multiple upstream providers. You also end up paying more to have techs manning the NOC 24/7 and others on-call should a situation arise.

      All of this is all the more expensive because customers demand 5 9's even if they don't need it.

      As for people who provide the long haul connections, they face costs for right of way and repair (such as when idiots see the call before digging signs and dig up and cut the fibre anyway).

    • ISPs are probably what can be termed "price searchers."

      Ugh... and this merited an insightful? Must be lots of leftover mod points:-) Don't mean to harp on an aspiring young mind, but...

      but largely I imagine this is negligible--the bandwidth is there whether or not it is used.

      Your professor has fallen prey to the fiber glut myth. Yes, even in our parts of flyover country, it is "fiber, fiber everywhere, but nowhere to connect." Countless miles of 100-lane highways with no onramps.

      But when you realize that most of the cost is in the onramp and the interchange (last mile and switching, respectfully), there isn't this mythical nearly free bandwidth.

      As a few other posters have noted, ISPs are out to make money.

      As witnessed from the events of Worldcom, Global Crossing, Level3 and countless other carriers, probably not enough were out to make money. Please explain though, why would this comment be relevant (not to mention "insightful")? If your employer is not out to make money, RUN. Got stock in a company that is not out to make money? SELL. Buying service from a company not out to make money? LEAVE.

      Making money permits us to be around tomorrow to provide you service, give people jobs, and pay our vendors and shareholders. If you feel otherwise, a good cure would be some reading [amazon.com] to discharge the mush your professor presented.

      A price searcher has a set cost that is basically unrelated to the amount of service they sell.

      IF sum(ISP) = bandwidthcost, AND bandwidthcost is fixed, then this would be true. Neither is correct. I'm buying from a multihomed, dual OC3 IP carrier and have a 1 gigabit pipe to them myself, and pay around $200/month per Mbps of bandwidth. Doesn't seem fixed to me. It isn't to my upstream either, and last time I looked, Sprint and AT&T didn't give free extra T1s after the first one was bought.

      They simply match up the price/quantity demanded at that price that leads to the highest revenue and thus profit.

      This model might apply more to someone like Microsoft, Nokia or Motorola who have a set R&D cost to engineer a product and very little reproduction and distribution cost (e.g. reducing cellphone circuitry to ASIC). But it doesn't fit the carrier business.

      *scoove*

  • I know a person that makes a healthy sum of money (enough to support a family easily) to sit and watch people dig near buried fiber runs. That right, he just sits in his truck and watches.

    It's cheaper for the fiber owner to pay him to be on-site than it is for them to lose the money on lost bandwidth should the cable be cut and communication be down for an hour or two of response time.

    He does have to re-certify on his splicing ability a few times a year, but that's about all he really does.

  • by wackybrit (321117) on Sunday January 05, 2003 @10:42AM (#5019471) Homepage Journal
    All of the replies so far in this thread are banging on about general business issues and supply/demand, but what about the problems caused by the fragmented nature of the Internet itself?

    I'd like to see some replies from 'people in the know' on how peering agreements, backbone interconnections and peering centers like LINX affect things.

    After all, the average packet from the UK to the US doesn't just go over one provider. It goes over my ISP, a UK backbone, through LINX, to the US on a DIFFERENT ISP, then hits a NOC in NYC, goes to ANOTHER backbone.. and so on.

    How do all of these different ISPs interact with each other? Do the larger ones set up networks then charge the smaller ones (like my ISP) for bandwidth which is then passed on? Or do they have 'back and forth' arrangements where the ISPs only pay for the difference between in and out traffic?
    • by aufait (45237) on Sunday January 05, 2003 @11:39AM (#5019656) Homepage
      Do the larger ones set up networks then charge the smaller ones (like my ISP) for bandwidth which is then passed on?

      Yes. Only the big boys have free peering between themselves. And, even they have costs associated with peering: cost of colocation at a peering point, bandwidth between the two service providers, etc.

      Or do they have 'back and forth' arrangements where the ISPs only pay for the difference between in and out traffic?

      Some do this. However, it has always been a hot button issue with ISPs. Some ISPs, e.g. UUNET, have a higher percentage of web servers. While others, e.g. Earthlink, have a higher percentage of users. Who exactly is providing more value to the other. Is Earthlink supplying the viewers to the web sites? Or, is UUNET supplying the web sites to the viewers?

      • Actually, 'peers' set up peering relationships between each other, not only 'the big boys'. The Tier-1 ISPs have specific peering requirements that other ISPs have to meet (regarding the size of the other's backbone, traffic ratios between both networks, many other factors, and a whole lot of legal crap). Once those peering reqs. are met, then usually connections will be set up in as many places as are needed to sustain traffic flow. The main costs of high-bw internet lines are port costs (for the physical port on the router) and line costs. Usually in peering, both isps throw out their port costs, and then split the line costs (1 peer buys local loops for 1 ckt, the other gets the next one, etc). This provides tranit between the two networks at close to 0 cost for both providers. Now, if ISPs dont meet other's peering requirements, the smaller ISP either have to have a transit provider (a provider which will allow them to route traffic through their network to reach other ISPs'), which is very expensive, or they will have to purchase lines/ports from the large ISP, be they transit or not. Some companies have very open and lenient peering policies, and thus foster more open connectivity on the internet. However, an increasing number of ISPs are tightening their policies in an attempt to make more money. //Phizzy
      • Smaller ISPs also peer with one-another on a "free" basis.
    • by scoove (71173) on Sunday January 05, 2003 @01:22PM (#5020075)
      How do all of these different ISPs interact with each other?

      There's been a bit of battling that got us to the current model of interchange between networks. A bit of history helps explain why we have the model we have now:

      At first, we had the NSFNET system - which was a government contract administered network consisting of a national network (NSFNET, run under contract by ANS, which I believe was a venture between IBM and MCI), and then distributed to regionals, e.g. NEARNET, SURANET, MIDNET, etc. The regionals were initially nonprofits, occasionally directly administered and sometimes done via contract (I believe PSINet was formed by operating the regional contract in the central east cost).

      NSFNET didn't permit true commercial traffic, due to its AUP and the fact that taxpayers paid for the network and it'd be an abuse. Still, many regionals blatently ignored this rule. (They also charged obscene rates like $60,000/year for a 56 Kbps leased line, which I was quoted for a community connection in 1993 by our regional - guess someone had to pay for the retired college professors doing research on their payroll).

      NSFNET, its pols and associated lobbyists from the Baby Bells, decided to push for a commercial network monopoly that would create a commercial national network, regional NAPs which would be Bell-operated (granting Internet monopoly to them), and then service providers could buy access to the NAP and compete (yea right) with the Bells. We actually saw this model take place with DSL unbundling and the massive failure of DSL and CLEC providers when the Bells played paperwork games (such as SWBell, which allegedly had a single phone line for all the fax requests for DSL orders from competitive carriers - and gosh, guess what kept running out of paper!) Incidentally, Al "Father of the Internet" Gore was a big proponent of this Bell monopoly grant. I guess it is kind of true that he was the father of the aborted Internet Bell Monopoly.

      Fortunately for all of us, UUNET, PSINet, Sprint and a few others started up the Commercial Internet Exchange and created a multilateral exchange - where all could tie together by joining the organization and peer for free.

      Naturally, as each one joined, they wanted to stop multilateral connections, as they wanted all newcomers to buy through them instead. Then came the MAEs (metropolitan area ethernets, administered then by MFS) which had peering that was bilateral instead of multilateral - meaning you could show up at a MAE but have no connections. You had to arrange for peering with each party one-by-one. Sometimes they'd play, sometimes not.

      As things grew, we ended up with what former SprintLink head Bob Collet called "NSPs and ISPs" - network service providers being the backbone providers that were interconnected and bilaterally peered at numerous peerpoints and bilateral interconnects, and ISPs being those that buy from the NSPs and perhaps occasionally partially bilaterally peer.

      Do the larger ones set up networks then charge the smaller ones (like my ISP) for bandwidth which is then passed on?

      Yes. NSPs charge for transit.

      Or do they have 'back and forth' arrangements where the ISPs only pay for the difference between in and out traffic?

      This is an excellent question, given that it is very much the rule in international voice telecom. Between international voice carriers, there usually is a "settlement" arrangement - where each tracks traffic sent to each other, and then if there is any significant difference, it is either carried over to the next month or it is paid for to bring the balance back to zero.

      Whether the Internet will mature in this manner is anyone's guess, though I have a hunch it might. Understanding that we still have a big battle over which is more important: consumer eyeballs or content, and we can't build settlement models until we resolve this.

      Are Earthlink and AOL's masses worth money to a content-heavy network? Or do content providers (like cable TV) have material that Earthlink should pay for? If that's ever meshed out, I think we might see settlement models become a real possibility.

      *scoove*

  • Peering arrangements (Score:5, Informative)

    by GLX (514482) on Sunday January 05, 2003 @10:42AM (#5019472) Homepage
    Not all ISP's peer for free with other ISP's. In a lot of cases, they have to "settle up" for their peering costs at the end of the month, per megabyte, with their peers. UUNet is big on doing this with their peers and it's where they make 50% of their income...

    Cogent and AOL just had a fallout about this - Cogent felt that they should have free peering with AOL but AOL felt Cogent should pay for usage because most of the traffic going to AOL was for Cogent customers and not vice versa.

    (see 'Peering' Dispute With AOL Slows Cogent Customer Access [washingtonpost.com] and Paid Peering [carrierhotels.com] )

    So yes, the short answer is that upstream ISP's do pay per Mb in a good amount of instances if they aren't a huge transit hub. Obviously some ISP's aren't subject to this and just charge by the MB figuring the more traffic you're getting, the more money you're making from their connection.
    • Of course, not all "ISP's" (especially those in the UK) are actual businesses.

      Those in the academic community have their own networks, in addition to connections to commercial providers, and, as they are educational institutions, don't necessarily have to show a profit....

      This can kind of distort your thinking if you try and see per-MB usage as a pure business function - not all those whose networks are used are for-profit organisations.

      Just a thought!
  • I know of at least two shared-host web servers that run $10-20 month and claim to offer unlimited bandwidth, arrowweb.com and ezpublishing.com. I think part of the catch is they have rather small disk quotas in the 50-100meg range for that price. I don't know how they prevent abuse, but I've been pretty happy with them.
  • As a small ISP.. (Score:4, Informative)

    by warpSpeed (67927) <slashdot@fredcom.com> on Sunday January 05, 2003 @11:01AM (#5019522) Homepage Journal
    My cost for Bandwidth is about 50% of my revenue. I am a one man show, and my ISP is not the bulk of my income. I make my money consulting.

    I do not know how the big boys do it, they have to have teams of skilled techs, phone support people, support infrasturture, people to manage said infrastructure, oh yeah, and the actual network to provide the bandwidth.

    I do not begrudge that fact that I have to pay a lot for bandwidth. I am actualy happy that I pay so little for what I get. However the barrier to entry into the market will make many people thing twice about doing it. If I knew then what I know now... well I probably would not have gone into this business.

    As for the costs to the ISP when a site gets slashdoted, the cost is slow connectivity and frustrated clients. It is a delecate balance, making sure you have enough capacity, and staying in the black. There is a huge lead time to getting additional bandwith avaiable, unless you are a large organization and can afford standby bandwidth.

    I have to keep a close eye on operations to make sure that bandwidth is not being over taxed by a single client, and that that client knows about thier usage. I have had several clients get hacked (open ftp Windows servers are a favorite). Suddenly they are pushing a full T1 of data out and they are scrathing thier head wondering why the Internet is so slow today...

  • This question has bugged me for years. I know that if you're a medium sized isp you might have to pay bandwidth charges to a backbone provider. Therefore you're probably stuck passing the charges on to consumers. But what is the reason for this and where are some details ? I'd really appreciate some good links or references to books.

    One thing that really sticks in my mind is that 5 years ago Pacific Bell was laying fiberoptic cable in my neighborhood (San Jose, California, USA). Right before they were about to deploy some kind of gigabit network of the future SBC bought them out and put a stop to it. Later they even had the fiber dug out, maybe to make sure that no one used it. What could the reason for this have really been ? I can't come up with anything except that they didn't want to give consumers too good a break too fast, they were hung up on 1980's penny-per-pixel pricing schemes, and that businesses paying thousands per month for T1 might want a break too.

    Another thing thats bugged me for years is what the bandwidth situation would be if Al Gore hadn't privatized the internet around 1996 in exchange for a couple hundred thousand in contributions to the DNC by MCI and others. Wouldn't the current bandwidth scenario be more like the Information Highway and less like the Information Tollroad ?
    • If I'm not mistaken, in the early 80's most of the telco's started laying fiber. One of the main reasons it was done was the government subsidized it. When it was done, the government said "you have to share it, since it was partly funded by tax dollars." Well the telco's didn't like that idea. The government said if the telco's didn't share the lines with competitors, they would have to dig it out. Guess what? All of them decided to dig it out. Pure and simple it was because the Telco's didn't want to share the fiber. The end result is the tax dollars spent on laying fiber was a total waste. The only people who benefitted were the top execs who got bonuses for pushing fiber and the politicians that got kick backs.
  • by davew (820) on Sunday January 05, 2003 @11:06AM (#5019538) Homepage Journal

    I guess what you want to know is why your charge increases per bandwidth consumed rather than just a list of the various expenses ISPs incur (which other people have covered pretty well), which theoretically could be dealt with by a flat charge. Here's my understanding of that. Some of it's outside my area of direct expertise - I'll mark the point where we hit that.

    As the bandwidth use in an ISP increases, the overall quality of service provided to its customers goes down (i.e. contention increases), until the ISP does the following:

    • Upgrade their external (upstream) links (more on this in a minute)
    • Upgrade their internal infrastructure (which might also be telco links between cities or countries, or might be 10M hubs going to 100M or Gigabit Ethernet switches)
    • Upgrade their supporting infrastructure (proxy caches, mail servers, billing systems, people on the tech support line, abuse department - this last one is a problem that seems to increase exponentially with size)

    You're probably already familiar with the difference between server hardware for 100 users and hardware for 10,000. Switches and routers tend to be a step upgrade; they work fine for three years then BANG you need to spend forty thousand euro, and that'll do you for another three years, or whatever.

    Telco bandwidth costs money, and upstream ISP service over that bandwidth costs more. The increase in that is usually sublinear - 4Mbps costs a little less than 2*2Mbps, 8Mbps costs a little less than twice that. The reasons for those costs are where I start to leave the stuff I do for a living, but my understanding is this.

    For internet service (this is different and separate from just getting a leased line!), the same principles as above apply, just on a bigger scale. The larger bandwidth user takes a larger chunk of the provider's resources, therefore they'll (within certain parameters) get charged commensureately more.

    For the cost of the leased line itself:

    • It costs money to put fibre in the ground, or under the sea, or whatever, and that needs to be recouped. This is a once-off, and it takes ages to pay back (hence the massive debts that the Worldcoms of this world are operating under).
    • The equipment that goes at each end of the wire costs a fuckload. At really high speeds (10 gig and thereabouts) it could cost more than the fibre itself. This isn't quite a once-off, but is a step upgrade a bit like your own switch infrastructure.
    • People and systems need to monitor the network, work out what bits have broken and need fixes, what's just had a digger put through it, etc. etc.

    That, of course, is without all the associated costs of running a business with more than two people in it, which are (to put it politely) non-trivial.

    Dave

  • bandwidth cost (Score:5, Informative)

    by Anonymous Coward on Sunday January 05, 2003 @11:07AM (#5019546)
    First off, an increase in load does not mean you need to increase the number of people who work for you (sysadmins, techs, etc.)

    Second, costs are obviously passed to end users with a mark up (however little).

    Third, cost of bandwidth DECREASES AS BANDWIDTH ITSELF INCREASES. At least with ISPs. ISPs who buy bandwidth in bulk actually reduce their cost per Kbps (that is, we get charged a lower per Kbps rate if we buy 45Mbps outright than 64Kbps). For example, we used to pay $1600 for a 64Kbps link but now we pay only $6000 for a 2 Mbps link (we're outside the U.S. so this includes physical connections) instead of $51200 ([2048Kbps / 64Kbps] x 1600).

    HOWEVER, buying in excess usually result in part of the bandwidth being unused but the ISP still pays for it. That's where the numbers game starts, where the ISP divide the cost of the bandwidth plus a mark up, to get what they charge customers.

    Obviously they can't charge customers on a per Kbps rate to recoup costs otherwise the customers would be up in arms. Now here's a secret (ssshhh!), most Internet traffic is bursty, that is, customers don't always utilize the entire bandwidth (at least, not at the same time) that the ISP guaranteed(?) - you access a webpage, read it (while you read, you don't utilize bandwidth) and access another page. ISPs rely on that fact when coming up with prices. That is the reason most do not like it when you upload stuff in P2P nets. Most probably they just use the RIAA, DRM, whatever, as an excuse to block access to P2P. Why? Because the minute you offer upload access over P2P nets, chances are high that you'll be utilizing whatever bandwidth they promised you. They don't like that because if a significant number do that it'll throw off their numbers (remember, they didn't expect you to actually consume the entire bandwidth so they didn't charge you based on a per Kbps rate). Noticed this trend lately with so-called broadband providers who "guarantee" you so-and-so amount of bandwidth/speed?

    You asked for the breakdown of bandwidth cost. Yes, most of it is caused by having to get additional bandwidth (larger pipe to handle larger traffic) but it is actually holding on to that large bandwidth (with very few users paying for it) which is the cause of you getting slapped with a huge bill if you go over the allotted amount. Their cost would be lower if they didn't have to keep excess bandwidth to give to you when the time comes that you actually need to burst over when you're slashdotted.

    We solved this problem of having to offer competitive prices while still being able to pay for bandwidth by charging on a per-MB basis (with a minimum of course). We give enough free MB to cover a month's worth of surfing/browsing (done thru extensive study of usage patterns) but the minute customers go to P2P nets, we hit them with an excess usage fee. This limits them from keeping their connections 24 hours doing nothing but downloading mp3's and mpegs.

    And we don't even have to hide behind RIAA or DRM.

    ALL our customers btw, are satisfied with this arrangement. Only those who go to P2P nets and download recklessly are the ones complaining (until now, all those who exceeded our allotment by a large margin have ALL been caught using Kazaa, iMesh or Morpheus).
    • So, why are you anonymous today? Tell us a little about yourself. Then explain why you have not considered those of us that would like to use free software tools for installs, voice over IP, or share home made movies with our friends. Slapping your customers will ruin you, if indeed you are who you say you are.

      Clamping down on bandwith is going to screw you, much to the delight of large ISPs charging you peering fees. They would rather gouge your customers than have you offer anoying competition. I don't think I'd fit into your extensive usage pattern study and I'd be unhappy with your service, espeically if you offered less than I could download via old fashion modem. (Hint, it takes about 15 days at 4kBps to download 5 gigs). Cutting off my download is sure to piss me off. Charging me four times dialup prices and then some more for the ability to download is really rude. I won't get into serving questions because you have not mentioned that and may just hand out fixed IP adresses and allow your clients to do as they please.

  • I can probably give you a few ideas as to how that cost is composed, on a departmental level, at least. :What is the origin of the cost of bandwidth?

    To the seller or the buyer?

    To the seller, assuming they are somewhere below the carrier grade level, their cost for the bandwidth itself is whatever they are paying the carrier(s) that they are renting their own bandwidth from. Of course, then you have to add in all the costs that come with doing something useful with that bandwidth - equipment, employees, et cetra.

    To the buyer..it depends on the selling organization. In some places, Marketing gets to determine the cost, based on the current market value. This often leads to doom because hey, if they knew how to do math, they probably wouldn't have ended up in Marketing where everyone sits around and looks at shiny things. In other organizations, Engineering and Provisioning (or other Operational groups) determine the actual cost to the company, then they let either Sales or Marketing tack on a little more for overhead/profit. Depends on which departments in a company have power.

    Yes, I realize that seems ludicrous, but that's generally how it works. People are people, and all that. I worked at a company where I gradually watched a shift in the balance of pricing power over a few years from Marketing/Sales to Engineering/Ops, and let me tell you, it's a fascinating thing to watch.

  • by eyeball (17206) on Sunday January 05, 2003 @11:15AM (#5019583) Journal
    I'm not an economist or financial expert, but it would seem that it would be more difficult to put a price on a service than on a tangible good. It's like comparing (sorry) apples and oranges. Seriously, it's easy to see where the money goes: raw materials, processing equipment, employees, insurance, etc.

    Since it sounds like you are asking where the money goes to help understand why it costs what it does, consider this:

    There are equipment costs for service delivery (i.e.: routers in the case of an ISP, or trains on the case of Amtrak), and related expenses (i.e.: electricity for routers, fuel for trains). But the more important costs that aren't obvious are intellectual expenses (engineers to design and run networks, enginners to run trains). Not to mention repayment for investors for risking their hard-earned money in the first place.

    Think of other services and how difficult it would be to determine where your money goes, and why it costs what it does:
    • Lawn mowing / Landscaping
    • Medical Care
    • Cable TV
    • Hotel Service
    • Auto Repair


  • I recommend you read Telecosm: The World After Bandwidth Abundance [amazon.com] by George Gilder.
  • Some Factors (Score:4, Informative)

    by aufait (45237) on Sunday January 05, 2003 @11:24AM (#5019609) Homepage
    When an Internet provider charges someone hundreds of dollars in bandwidth costs because they were Slashdotted (or Farked) and their bandwidth use shot up, what costs have the Internet provider incurred, and why does it cost them what it does?

    That is not a simple question to answer since there are several factors that have to be taken into consideration in determining price. Different type services, e.g. dialin lines, colocation, dedicated bandwidth, etc, have different factors.

    The primary thing to remember is that the ISP business model is based on setting prices for the "average user". They do not directly pass on the cost of each user to that user. For example, a dialin phone line costs $25. That is the cost of just the line. It doesn't include equipment, employess, or bandwidth. The reason ISPs charge $20 or less is that the average user doesn't tie up the line 24/7 so that they can 7 to 10 users per dialin line.

    The same is true with bandwidth. An average customer who gets a dedicated T1 usually averages 50% or less of the lines capacity. So the ISP can sell T1 service to two or three customers for each outgoing T1.

    Another factor is peak vs average load. The ISP has to provide facilities to handle the peak load. Yet, most prices are set according to average load. Check this site [angband.oook.cz] if you want a graphic example of the slashdot effect.

    All this presents a problem for ISPs in pricing for users want fast speeds but have low averages, e.g. colocated web sites or DSL lines. They can't charge for the maximum possible load. If they did, DSL lines would run closer to $1000 per month instead of $50. If they stictly base it on average load, the low users are subsidizing the bandwidth of the high users.

    To resolve it, they have come up with a price tier that gives an incentive to users to, accurately as possible, estimate their usage. They make it cheaper for the user to pay for the next tier above their average monthly usage rather then paying for the tier below their monthly usage.

    • Re:Some Factors (Score:3, Insightful)

      by LostCluster (625375)
      This is kind of like the pricing of cell phones...

      Your usage costs the provider the same no matter what pricing plan you're on. Providing you with 800 minutes of service over a month taxes their equipment exactly the same way whether you've prepaid for 250, 500, 750, or 1000 minutes per month. The cell towers don't know and don't care what plan you are on.

      However, when it comes to billing, the user who uses 800 minutes will get punished badly if they were on the plan that allocated only 250 minutes to them. The user who has the 750 minute plan will feel only a mild sting, and the user who's on the 1000 minute plan is likely getting the best deal.

      Why do the providers work this way? Because they 1. Want to encurage users to not use the service excessively and 2. Want users to to declare their intended useage in advance.

      See, those extra minutes of airtime usually cost a very small incrimental cost when you consider those minutes alone. However, somebody's minute of airtime is going to be a very expensive incrimental minute... it's the minute that causes the cell tower to hit its limit, and it means expensive network upgrades need to happen or customers are going to start getting turned away.

      The provider would rather people not unexpectedly increase their usage, and instead call them and warn them that they're going to increase usage by moving to a higher usage plan. When too many users are on the higher plans, the provider then knows it's time for an upgrade.

      Moving back to bandwidth, it's the same thing just on much smaller units, bits instead of minutes are the measurement unit of choice.

      Unlimited pricing models are simply unnatural, because it treats very light and very heavy users as if they're one and the same. Simply put, they're not. It's highly unlikely the provider really has the equipment to provide everybody with top-speed service at the same moment. So long as they have enough equipment to provide service at the moment of highest demand, it's good enough and nobody complains. If they don't, then whenever they break the limit of the equipment, everybody slows down. The provider gets a bad reputation, and either has to quickly upgrade, or drop the "unlimited" offer.

      So really, the price of bandwidth has to be based on two factors. The bandwidth that is on the table that you can possibly use, and the bandwith that you're actually going to use. If you plan on eating everything on the table, be prepared to pay much more than the person who leaves most of it behind.
  • Costs of Bandwidth (Score:3, Informative)

    by JWSmythe (446288) <jwsmytheNO@SPAMjwsmythe.com> on Sunday January 05, 2003 @11:38AM (#5019654) Homepage Journal
    Well, from what I understand, there are a lot of factors.. Bandwidth itself doesn't cost anything. You're paying for the right to use it. :) Kind of like driving down a toll road. The road is there. It's going to be there. If you use it or not, it's still there. But when you use it, you pay $0.50.

    The ISP already owns (or leases) a bunch of fiber going wherever. They already own a bunch of routing hardware. Those were initial costs, which may not come again, unless business is good.

    There are a bunch of costs that they continue to have.

    Staff. Everyone from the salesman trying to sell you a line, to the support staff answering the phones at 3am when you can't remember what your netmask is suppose to be.

    Building costs. Like, those lines run somewhere, right?

    Fees. Everything else. :) It costs money to run your fiber into someone elses equipment (peerings), or to share fiber across an ocean, or a satellite link.

    So, they can't directly make money back for all those costs. They just turn around and say, "We'll charge you for the bandwidth usage", and voila, they do.

    From what I understand, the actual cost of running a DS3 (45Mb) or a OC3 (155Mb) isn't too far apart.. But they'll sure charge you a lot more for it. :) That's how they make money..

    Like with dialup providers, they expect to need so many customers to cover expenses, then they turn a profit. If they can support 200 customers, and need 20 to make a profit, they don't start charging all the customers less as the number of customers increase.. The ISP owners just start driving nicer cars. :)

  • by Frightened_Turtle (592418) on Sunday January 05, 2003 @11:41AM (#5019667)
    How much money does it cost an ISP to be a good ISP? A lot more than you expect! Take an OC-3 line. This can run you upwards of US$50,000 PER MONTH! This is just to connect your customers to the major communications backbones of the internet, so they get reasonable bandwidth.

    But now you need to connect those communications lines to something so that your customers can actually get onto the internet. How 'bout a couple of routers. A couple of new Ciscos could run you upwards of US$100,000. Oh, wait a sec! You need servers so your customers can put up their websites, and to manage their accounts and stuff? Servers are the "creeping featurism" of providing service. The more customers you want, the more servers you have to add. For good, high-quality, industrial grade servers, let's take an average price of $2,000. I haven't even mentioned all the Cat-5 cable and switches you're going to need internally to connect all your equipment together.

    For that matter, how on earth are your customers going to physically connect to your service? Dial-up? Well, you will probably need a CLEC to be installed at the local phone company. Be a cable provider? You'll have to string lines up around town. That's expensive! Either way, you are now a utility, and you will have to license yourself as such. THAT costs some bills. The more customers you have, the more stuff is filling up your OC-3 bandwidth. In order to maintain a higher quality of service, you're probably going to need to add another OC-3 -- or more economically add a couple of DC-3's to help spread out the load. $! $! $! It's really starting to add up! Whew!

    Just to connect alone, you are looking at a recurring cost of better than $500,000 per year. Equipment costs will probably run you another $300,000. But you have to maintain all that. There's another cost for maintenance contracts. Believe me! You can't run all this by yourself! You will need to hire people just to help run this. And sell your services. And deal with irate customers. And accountants to watch your books so you don't put yourself too deep in the red. Parasites- er, uh, "lawyers," to keep you out of trouble with RIAA when some cheese-eating high school boy decides that paying someone for their music so the musician can make a living goes against the constitutional rights to free listening...

    And here's something that few people stop to think about: all these gadgets feed on electricity. There's yet another whopping, monthly recurring cost! Oh, well...

    These are the things that your money is going towards. Admittedly, I'm listing out all the top tier stuff, here. When you want to handle industrial and commercial uses, many customers at that level want guarranteed levels of service. Just to run this level of ISP can very easily run you over US$1,500,000 per year. You had better keep your customers VERY happy in order to make enough money just to cover your costs.

    At a lower level -- say you just want to be a neighborhood server for people in your part of town. Or maybe, all you want to do is be a hosting service, where people can park their web sites and pay you per month. A Mom-an'-Pop type set up could probably be pieced together for under $50,000, but you still have to connect to the internet so your customers sites serve to the world at a reasonable bandwidth. A DC-3 line could handle this at under $15,000 per month. You'd need quite a few customers to cover that cost, at least 200 at $75 just to match that price. More to make a profit.

    A long time ago, before Broadband. When 2400 baud modems were considered lightning fast. It was quite possible to set yourself up with a couple of computers, a couple dozen modems and a T1 and call yourself an ISP. In fact, that's how a few of today's ISPs got started. Take a look at Pair.com" [pair.com]. They are a classic example of a world class hosting service that got started in a garage. Nowadays, just to keep up with the competition, you have to put out over 200 time more cash just to get started.

    Gee, I bet you wish you had that $350 million dollar Powerball lottery ticket now, don't ya?
  • I have done a fair amount of shopping on hosting and spent 70,000 USD on it last year. At the top, are the big network providers (UUNet, Qwest, AT&T, etc.) These companies have substantial costs associated with technology and connectivity. Their customers are typically end user ISPs, hosting companies, and colo (colocation)facilities, so support is a very minor cost. They have a commisioned sales force. Next in the chain, we have end user ISPs, hosting companies, and colo facilities. End user ISPs provide an obvious service. Hosting companies and colo facilities are somewhat intertwined. Large hosting companies have their own datacenters and connections. Colo facilities have separate racks that are leased out. The customer is responsible for the connectivity, but these places are typically prewired with every sort of network provider available. As an added complication, Many colo customers are small hosting companies. Let's look at the cost impacting factors at each of those levels: Network providers: Infrastructure: High Connectivity: High End user support: Nearly nonexistant End user ISPs: Infrastructure: Moderate Connectivity: Moderate End user support: Very High Hosting Infrastructure: Moderate Connectivity: Varies on account size End user support: Varies on account size Here is the problem with hosting: The largest companies are running large *nix servers with huge numbers of accounts on them. These are the 10-300 USD accounts. You pay for a minor amount of bandwidth and hardware, but the fixed costs of collecting payment and providing support make up a large portion of your bill. The high overage charges are there to discourage you from exceeding your planned bandwidth. These companies are paying for a fixed connection typically in 100Mbps and 1Gbps units. They need to keep a certain level of utilization and keep burst down to maximize the return from their connectivity costs. An additional cost of overage is in hardware. Those large servers are balanced with a high, but manageable number of accounts. When several accounts are at high burst, there is a possibility of slowing other accounts and getting complaints. The cheapest way is using dedicated servers and a fixed connection size. The problem with this is that it is not cost effective until you are using a fair amount of bandwidth. The reason it becomes so much cheaper here is that there is no uncertainty for the hosting company. You are also generally responsible for administration of these servers. Colo is pretty much the same except that you install your own servers in a facility instead of leasing a server from a hosting company. I hope this helps. I have explored many options and I have learned a great deal about hosting from being a large account.
  • by release7 (545012) on Sunday January 05, 2003 @11:44AM (#5019678) Homepage Journal
    Theories about supply and demand look good on paper but in the real world, the law is much simpler: get as much cash as you can anyway you can. Unlike natural laws, there are many ways to artificially bend the laws of supply of demand in such a way that serve self-interested parties. Take OPEC for example. It's a group nations working cooperatively to maximize profits for the member nations by either creating artificial scarcity or oversupply.

    Oil is a little different than bandwidth in that it is a very tangible good. You get mechanical devices to pump the stuff out of the ground, pipe it down to a ship, get it to an oil refinery, and distribute the stuff. OPEC may be able to influence the market somewhat, but when capital is tied very closely to physical product, the laws of supply and demand become harder to control and manipulate.

    But look at what happens when you try to apply the law of supply and demand to a much more intangible good like electricity. Electrons can very easily be "hidden" and then magically reappear practically at the flip of a few powerline circuits. It instantaneously can be shuttled over great distances from one place to another at little or no cost. And this makes it a market ripe for manipulation and scamming a la Enron. They took OPEC one step further by creating an entire artificial markets to sell electricity.

    Enron was also about to create an artificial market for Internet bandwidth. Just like electricity, you can play a lot of games with the "supply" of bandwidth and play a lot of bullshit games with the actual cost of the good. I definitely don't trust corporations to set the price of bandwidth.

    It's extrememly difficult to place a cost on bandwidth especially when the corporations that control bandwidth have their fingers in many different other communication services. Basically, companies will charge whatever they can get away with, the "true" cost of bandwidth be damned.

    And that's precisely the problem. When utility company's first started out, they were entirely unregulated and they were the only game in town. They charged whatever they could get away with until government stepped in and allowed them to charge just enough but still make a handsome profit.

    The Internet is not much different. Though we have more than one ISP to choose from, there are only a small group of players who control access to the Internet's backbone. It wouldn't be much of a trick for them to fix bandwidth prices and make a killing.

    So to answer the question "what are the true costs of bandwidth", I'd say, "However much gouging the average consumer can accept. The public be damned."

    • by the eric conspiracy (20178) on Sunday January 05, 2003 @12:08PM (#5019765)
      Take OPEC for example. It's a group nations working cooperatively to maximize profits for the member nations by either creating artificial scarcity or oversupply.

      One thing that prevents OEC from doing anything it wants with oil prices is that there are other sources of energy out there. If the price of oil gets to be over $45/barrel for a significant period of time, those other sources become competitive.

      The Saudis understand this issue and have publically stated that they will not let the price of oil rise to the point where alternative energy sources are competitive.

      Even cartels are subject to supply/demand laws if they distort a market too much.

      • You missed my point. I wasn't arguing that OPEC is not subject to the laws of supply and demand. I was simply stating that they exist to tilt the supply/demand curve in their favor. And as you rightly point out, when you are dealing with a tangible good like oil, manipulating the market is not always easy.

        But when you are buying and selling intangible goods like electricity and bandwidth on the open market, it becomes extremely easy to manipulate the laws of supply and demand. Do you think it's really possible for UUNet, Sprint, and AT&T to precisely calculate the cost of transmitting a MB from point A to point B through their vast communication networks? It's an impossible task and subject to a lot of bullshit accounting procedures. There's no law of supply and demand being followed by these bandwidth cartels. And they certainly don't have to worry about setting the price to high. What are you going to do, start laying wire for your own personal Internet?

        • But when you are buying and selling intangible goods like electricity and bandwidth on the open market,

          I haven't looked into the issue of bandwidth cost in detail, but I know for sure that it is possible to get a good hard number on the cost of producing a kilowatt-hour of electricity, and it is done all the time. Companies know what their capital investments, fuel costs and operating expenses are or are likely to be, or can estimate the effects of changes in those costs. Before ANY power plant is built you can bet all of this is calculated to a fare-thee-well by the utility company. And there is no bullshit involved in this calculation. I know, I used to do that sort of calculation as part of my job (in a different, but related industry).

          The situation with internet bandwidth may be a bit more complicated by fact of the matter is that it's an industry where things are unsettled and changing rapidly, both from an economic and regulatory point of view so making predictions is likely to be a lot riskier. But accounting practices are not the issue - cost engineering is a well established and definitely non-bullshit field. It's the unsettled nature of the business that makes predictions difficult.

          Do you think it's really possible for UUNet, Sprint, and AT&T to precisely calculate the cost of transmitting a MB from point A to point B through their vast communication networks?

          I think that they know very well how much per day it costs them to run their network, what their capital investment is and how much traffic in MB it carries each day. From that it is actually very easy to calculate what the cost is to move a MB from a to b. I am also sure they have a very good idea what their marginal costs are.

          There's no law of supply and demand being followed by these bandwidth cartels.

          I disagree with that strongly. Supply and demand is ALWAYS in force. If you raise the price too high, the demand will go down. Period. Look at the current situation with consumers today - we have many people using dialup because they don't want to pay the higher cost of the increased bandwidth of broadband.

          And they certainly don't have to worry about setting the price to high. What are you going to do, start laying wire for your own personal Internet?

          If the price goes to high you can BET somebody will start launching satellites or running more cable, or launching stratospheric planes, or installing wide area wireless, or deploying whatever other technology seems best at the time.

          • The situation with internet bandwidth may be a bit more complicated by fact of the matter is that it's an industry where things are unsettled and changing rapidly, both from an economic and regulatory point of view so making predictions is likely to be a lot riskier.

            Good analogy and good points. Power companies know the cost of building a power plant of any particular size. The trick is to estimate the capacity required. Build one too large and you have to pass the cost of the unused capacity onto your users. Build one too small, and you have brownouts and blackouts. ISPs can also calculate the exact costs associated with supplying bandwidth.

            There are several differences between the two industries. One is that most power companies are government regulated industry. If they guess too high, they can pass the cost of the unused capacity directly along to the users. The only choice the user has is to pay the higher costs or reduce their use of electricity. They can't switch providers. If they guess too low, the consumer has to put up with the brownouts until more generating capacity can be built.

            ISPs don't have that luxury. If they guess to high, they have to eat the cost of the unused capacity since their users can easily switch to cheaper providers. Guess too low, and they lose customers to providers that aren't maxed out.

            It's the unsettled nature of the business that makes predictions difficult.

            This is true in both industries also. I am sure that the power companies calculations were shot to hell when air conditioning, TVs, etc were introduced. The advantage the power companies have is that they are stictly usage based. They will earn more money as people increase their usage.

            ISPs pricing is a mix of flat rate and usage based pricing. The flat rate portion is based on the "average user". When something occurs to change what the average user consumes, the ISP has to eat the cost of the additional usage until they can adjust their prcing plans. Due to the compeition, it is not as easy for them to increase their costs.

            DSL is an example of this. The original pricing models were based on the assumption that residential users would download the same amount of traffic as they did when using dialup. They assumed that the only difference would be in the "burstiness" of the traffic. However, P2P made that assumption invalid.

    • You say, So to answer the question "what are the true costs of bandwidth", I'd say, "However much gouging the average consumer can accept. The public be damned."

      I say that soon people will find a way to supply free bandwith. It's in everyones best interest but current telcom, govenment, media. That may sound like a lot of opposition and it is. The same interests also stand against free software. Those that would continue to screw you can be defeated.

      Rise up, wireles mesh!

  • Why? How else are you going to charge for access to your network?

    Pretend you are starting from scratch, and buildingl a network. You lease/build lines from city to city, within a city, etcetera. This is all fixed cost stuff to you. Now, you only have so much bandwidth between cities... and given that you are selling internet, which is packet switched, and not full circuits.. you aren't selling dedicated city to city bandwidth to people. That's what some people call "overselling" though the term has a negative connotation when there is none; that's the whole point of packet switching. So how do you charge for use of those shared backbones that you own? Flat fee for every customer? Sure you can do that; some ISPs can and do. the cost ofen reflects this.
    The natural thing to do is divide it up by bandwidth. Set your pricing so that you have a good idea of how much bandwidth each customer is going to use in reality, so you can properly manage your network for all your customers. If they exceed certain limits, the pricing goes up... this is a deterrent, as well as a money maker.

    There are many ways to do this.. if you are just buying transit on some long distance cable, often it's a flat per-byte fee. Use as much as you want, but you pay.
    Some do what they call "burstable" where thyings are based on an average throughput over a month... where a certain percentage of sample periods are permitted to be over the limit.
    (384kbps burstable to 1.5Mbps.. your average speed has to be 384kbps max, but you are permitted up to, say, 5% of your traffic to be over 1.5Mbps). This is a convenient way for some customers to look at things.

  • Bandwidth costs (Score:3, Interesting)

    by scoove (71173) on Sunday January 05, 2003 @12:32PM (#5019858)
    Per the original post:

    Is there usually any sort of markup going on along the line, or are people just passing along their own expenses down the line to the end user?"

    The original poster refered to a "rate increase" scenario, such as being slashdotted (though this is almost never the condition seen; more often, it's things like abusive P2P, sustained large-pipe VPN connections to another network, or nonstop downloads of DVDs, MP3s, videos, etc.)

    Contrary to common net paranoia, most ISPs don't go chasing after bandwidth abusers unless they really are a problem. Network "policemen" are an expensive distraction from the primary focus of our business, so they don't usually send the heat out unless someone's really out of control.

    So per "passing along expense" vs. "marking things up," I wish!!! By the time a customer has been notified of bandwidth or AUP abuse, they've not only consumed much more resource than they've paid for (which typically won't be paid back, even if a rate increase is applied to the subscriber), but they've incurred a hefty administrative expense.

    Take your typical /. tech salary, add social security and both sides of income tax, plus health insurance, backoffice costs divided by employee, etc., and you'll have a nice $30+/hour number minimum. Multiply that times a the hours to put into place the management systems, policing time, customer contact time, etc. and you've got a cost well over $500, not including the cost of consumed bandwidth. At the $35 you're paying me, minus the cost of your service, it'll take me 5 or more years to make back that $500 you incurred. I'd rather tighten the noose and encourage you to abuse my competition.

    How much of the total costs are tied to infrastructure versus the human component (technicians, sysadmins, technical support and so forth)?

    Very good question to raise. In our network, bandwidth costs are only #3 - they follow payroll (people cost) and network infrastructure capital costs (e.g. routers, buildings, towers, switches, servers).

    When you're down to a per Mbps/month cost of less than $350 (for both upstream egress to tier one networks and ingress from your customer), bandwidth abusers still can cost you a bit of money (chewing up a sustained 1 Mbps/month while paying $30-$40 for residential service), but the people cost associated with dealing with the abuse is more significant.

    So what's the best way to deal with this? Information retrieval charging [www.thur.de]

    Actually, I'd expect more and more providers to pursue a tighter bandwidth model (down from the ~3 Mbps unlimited of early cable modem years) with bandwidth caps being more common. Understand that by making 95% of the customers overjoyed, and either making the 5% bandwidth hogs pay for their use or go to the competition, it makes service better and lower cost for everyone else.

    And a final thought: We've seen threads on /. before about profiling customers in other industries - e.g. Delta Skymiles. Don't think your ISP won't model who the good and bad customers are. Instead, count on it and use the system to your advantage, or understand that by being in the bad category, you're going to be treated just as someone with a lousy credit rating has at the bank. We already score customers on support cost - and know when you call in if you're a recurring problem user (blaming us because your MSWord doesn't work right), or if you're a prompt payer that rarely complains.

    *scoove*
  • by RebornData (25811) on Sunday January 05, 2003 @12:38PM (#5019879)
    Pricing and cost analysis is a rather complex topic that there's a lot of academic theory for, but in my experience many / most companies rely on a combination of very basic analysis, "gut feel" and "look what the other guy is doing" to figure out. I worked for 6 years as a product manager for various kinds of service providers, so I've had direct experience with it. BTW: service providers often use the terms "product" and "service" interchangeably for what the outside world would call a service, so my apologies if I switch back and forth. It goes something like this.

    Let's look at cost first. Typically, service providers think of costs as some percentage of the dollars coming in for a given service. They break these down into several categories. Although each company is different, here's a simple example:

    • Cost of goods sold: These are dollars that the companies spends to provide the specific service. More on this later.
    • "SG&A": Sales, General and Administrative. Otherwise known as "overhead" -- cost associated with running the company that aren't tied specifically to a service (management salaries, office space, etc...). Sometimes sales salaries and marketing are broken out from this on a per-service basis, other times they are lumped into the general company overhead costs.
    • Margin: This is the money left over after the company's costs are taken care of. If the company sells everything directly at retail price, this is equal to profit. However, if a company has resellers or other types of "sales channels", they will be paid a certain percentage of the margin for selling the product. Discounts also come out of this chunk.
    Very rough rule of thumb: cost of goods sold should *never* exceed 50% of "retail" (undiscounted) revenue in the telecom / service provider space. Of course, if a company is desperate for business they may give away very large portions of the margin percentage in the form of discounts, etc...

    The "cost of goods sold" for a typical hosted dedicated server generally break down into several major categories, that may or may not be broken out as different "services":

    • Hardware cost & software licensing: This is the fixed purchase cost of the dedicated server itself, and any associated software licenses. This is typically paid directly by the customer up-front, or broken down into lease-like payments and buried in a flat monthly fee.
    • Data center: These are costs of having the server sitting in the data center, including lease cost of the space, supporting rack and LAN infrastructure, electricity, cooling, security, etc... Typically these costs are figured based on averages per square ft. of server room space or by rack space.
    • Operations / management: These are the costs of management services provided for the server: NOC staffing, management / monitoring system cost, tape backup costs, etc... In the case of colocation, this may be minimal or non-existant, while it's very significant for dedicated / managed services, since you're paying for system administrators to upgrade your OS, apply patches, etc...
    • Customer service / support: Costs for the right to call someone on the phone and get help with your service. This primarily includes call center infrastructure & support staff salaries. Again, may vary widely depending on the level of service selected.
    • Bandwidth: At a minimum, this is the cost of the routing infrastructure, the cost of the WAN part of the NOC staff and systems, and the monthly costs paid to the upstream providers for big dedicated pipes. Your average hosting company is not running their own fiber or even buying dedicated circuits: they are buying IP transport from a large ISP, and pay for a certain amount of bandwidth. Usually, they'll lay in a very large fiber pipe to the ISP's local POP, and then activate additional bandwidth as needed. I can't break down the long-haul ISP's costs for providing that bandwidth, but presumably it breaks down similarly.
    Now, a few words about pricing, and specifically bandwidth pricing, since the poster was interested in that. Aside from bandwidth, the costs above are reasonably predictable from customer to customer and month to month. The more predictable a cost is, the more likely a service provider is to lump it in with a bunch of other costs and charge a flat fee. The less predictable a cost is, the more likely it's going to be metered and broken out.

    In general, customers demand / want flat, fixed, predictable prices. This is why ISPs charge $19.95/month for dialup rather than $.05 / hr. Of course, at some level the ISPs price for dialup is proportional to actual use, but they've figured out that, on average, they can make money at a $19.95 flat fee, and customers are willing to pay it. Of course, customers who use less are subsidizing the service of customers who use more. But as long as the *average* cost remains low enough that the low-usage customers feel they are paying a fair price, it works out.

    Server bandwidth is a case where it doesn't work out. The amount of bandwidth that a single static web server is capable of consuming is quite stunning, and goes up along with Moore's law. Many hosting customers choose a dedicated server not because they need gallons of bandwidth, but because they have some sort of custom app or want full administrative control of their system. These folks aren't willing to pay a price that would cover the "average" cost of bandwidth across all an ISPs customers, which may include large streaming media systems or pr0n hosters. So the ISP meters / measures the bandwidth and charges each customer appropriately.

    However, it's a bit more complicated than that. You'll notice that, if a customer buys a lot of bandwidth, they pay a lot less per GB than a small customer. The smallest customers may pay ridiculous prices for bandwidth if they exceed their plan (case in point the Farked "boobies" page that racked up $400 in bandwidth charges in two days). It's obvious that SPs are charging much more than their "costs" for the bandwidth in these cases, and you'll notice that the same thing is true for cell phone plans, etc... What's going on?

    The answer is that it's all about predictability. A service provider must maintain adequate capacity for providing service. If they don't have enough upstream bandwidth, service quality for their entire user population goes down the toilet. It takes a long time to add additional capacity- new fiber needs to be run, new equipment purchased, etc... As a result, service providers are always buying new capacity in advance of demand, and to do this accurately they must be able to forecast demand. This boils down to getting accurate forecasts from their customers, which is impossible to do directly. Instead, as capitalists, they put economic incentives in place to motivate customers to predict their maximum demand accurately by pricing in tiers, or packages. Buy buying a particular bandwidth package (say, 20GB/month) you are effectively telling the provider that's how much you are planning to use, and they plan their capacity accordingly. Deviating from their master capacity plan is very, very costly for the service provider, and accordingly, the "right" to deviate from your plan as a customer is proportionally more expensive.

    Another way to look at it. Customers who buy the smallest bandwidth package are the least "valuable" to the company because they are making no revenue commitments. They are also the most easily able to double, triple, or quadruple their demand from month to month because their servers are capable of consuming so much more bandwidth than they are buying. On the other hand, a large, multi-server customer is a more valuable customer, and is likely to be using a larger portion of the available capacity of their servers, and thus less likely to have wild changes in the amount of bandwidth they consume. The small customer thus pays much more per GB of bandwidth than the large customer, especially if they exceed their plan. In a lot of ways it's like the airline industry or any other industry where buying additional capacity is expensive and/or time consuming: you pay a premium for being able to use that capacity on short-notice.

    Well, that's more than you probably ever wanted to know. Hope it's been educational.

    -R

  • Somebody earlier mentioned about the "burstiness" of internet traffic. To be more precise it is the On-Off nature of a single traffic source, or a user. Simply put, the user is ON when he/she, for example, downloads a page, and is OFF while reading it. Internet users cycle through this behaviour.

    Since many users are dynamically using bandwidth the ISP benefits from the fact that their peak usage is uncorrelated in time (lets stay from fractality for simplicities sake). Thus burstiness leads to what is called "statistical multiplexing" which actually offers powerful economic advantage.

    Thus traffic aggregation is a key design feature of todays internet.

    Every user has a peak-bandwidth usage (P) during ON phase. This is interleaved by long periods of silence, or OFF phase. Understandably the average-bandwidth (A) of each user is much less than peak usage, or A is less than P.

    When ISP provisions his network what should he consider? A or P?

    If he commits to P then he will always be in a position to fullfil the user demand whenever it may happen. User gets QoS but ISP gets shafted as the network will be most of the time underutilized.

    If he commits to A then each transfer would be spread out in time, and each user would feel the that service is unsatifactory. But the ISP benefits from the fact that he can have "more" customers as compared to the earlier case.

    Example (units ignored)
    P = 20
    A = 10
    Bandwidth = 100

    ISP committing P will have 100/20 = 5 customers
    ISP committing A will have 100/10 = 10 customers

    Now i did mention that users would feel that "service is unsatisfactory". It is due to the fact that more that 85% of internet traffic is TCP and is elastic in nature which roughly means that each additional user of the bandwidth will result in decreased available bandwidth to every other user using it.

    Theoratically there is an optimal point, or a certain number of users, upto which aggregation is desirable. Under this point, aggregation is small but QoS is high. Beyond this point the aggregation is high but the users begin to feel the impact in shape of degraded QoS.

    Congestion pricing is one of the ways of moving the number-of-users to that optimal operating point.

  • Going price for a T1 (1.5 Mbps) routed onto the internet, from a second-tier provider, has been around the $1000 mark for several years. (OK, it's easy to pay more, but it's hard to pay much less.) Of this cost, typically:
    1. $100-$150 per month goes to the telco for the copper pairs. This can actually vary quite a lot, depending on whether it terminates at a central office or has to go out to a ISP's equipment room and the price is closely related to tarrifs that vary state-by-state and sometimes locality-by-locality. Depending on the exact type of service, this may be a "normal" telephone pair (as in recent T1-over-SDSL via a single copper pair) or it may have been a special trenching job just to run the T1 pairs (which probably would've been mucho $ you had to pay up front.)
    2. The rest is for the cost of hooking to the backbone. Again, this varies quite a bit, depending on how much oversubscribing your provider does. Some oversubscribe by as much as fifteen to one (this'd be like an airline selling 3000 tickets for a flight on a 737) and have much lower costs than a provider that doesn't oversubscribe at all.
  • Assume that it's a big hosting service and buys bandwidth in quantity. Say that physically they're using an OC-3, that's about 155 Mbps. Their contract is more likely to be based on usage, say $300 per month per Mbps, and the number of Mbps is determined by measuring usage for every 15-minute interval in the month and using the 95th percentile. If that figure is 100 Mbps, then they pay $30,000 for the month. Of course there's also a minimum usage that puts a floor under the payment.

    If a site they are hosting gets /.'ed, it creates several 15-minute intervals with abnormally high traffic. This will raise the 95th percentile figure. If that goes from 100 Mbps to 105 Mbps, the hosting service gets charged an extra $1500 for the month. Of course they can avoid that by throttling themselves back at a gateway router, but then every site they are hosting will experience degraded service. Complex traffic shaping on this scale uses significant processing in a router, reducing its capacity and raising those costs. TANSTAAFL. Watching a network grow rapidly has renewed my appreciation for some of the problems of scale.

    Actually, an OC-3 isn't all that much bandwidth. I know there are service providers connected to some of the big backbone networks that have multiple OC-12s. I have seen contracts in those cases where there's a sliding scale for bandwidth, so an incremental Mbps-month costs less as you use more.

    • This comment is right on the money. ISP's like most other businesses need to plan for a certain level of load. They don't have infinite resources or margins. They provide 'enough' headroom to accommodate reasonable loads, but they can't afford 'too much' headroom -- if they did, they'd be wasting capital that could be used for other things. (Obviously what 'enough' and 'too much' represent are a function of their SLA's, service quality, size, etc.; a good provider won't run into trouble often.)

      When they run out of gas, everything goes into panic mode. People work overtime. Unexpected capital expenditures are made (new routers, new servers, new DASD, etc.). Surcharges are tacked on from upstream vendors.

      All those costs add up, and because they're unexpected, they're painful. So to an extent, the surcharge that WE receive is to discourage us from whacking the pipes too often; or, if we need it, to arrange for the capacity in advance. So if the higher costs seem disproportionate, it's because they're supposed to be. Enough of these events can drive a smaller provider out of business.

      JMO
  • One thing I didn't see many if any people mention is reliability and redundancy. Where we live, the historical bandwidth providers are the telcos, who have flatly had a terrible record of keeping their connections working. The FIBER might be up, but the connections to the providers the telco'suse for IP may be down. One example is that a local telco uses nothing but UUNet/Alternet for their entire IP connectivity if you buy a internet pipe from them. They have 1 link to Indianapolis, and another to St. Louis. When they've had problems in the past it goes to pot quickly, especially since if there's a UU/Alternet issue, it doesn't matter which path is 'up', there's still an issue.

    Other Internet providers deal with this by selling local customers a 'local' T1 or DS3 or fiber into their data center. From there, said data center will have 2, 3 or more IP carriers, all hopefully riding different fiber providers via totally different physical connection paths to the outside world, hopefully going through physically diverse pieces of network gear.

    That's how we do it, for example :-) It does mean that customers pay for that duality of gear and network connectivity. However, the flip side is that theoretically, they shouldn't go down, unless the single piece of gear their single T1 plugs into goes down for maintenance or failure (and since you've cut the failure prone pieces of gear down to 1, and that 1 is a incredibly simple piece of gear compared to the other stuff and can be replaced in minutes if not seconds, your failure times go down). If you HAVE to be up, then get a second T1 at your location to another Provider who hopefully uses a different IP carrier, using a different fiber provider, using a different physical path, and who hopefully has his own 2 IP carriers, etc etc etc.

    That's where the real cost of bandwidth can be inflated. Just having a 100 Mb/s line to the internet can be cheap. Having one that's always up, regardless of physical, fiber, or IP carrier failures may not be especially if you don't have the expertise to manage BGP and gear that supports it.
  • Human resources are probably the most expensive. CCIE's and JNCIE's are at least 100K a year each. Lower level engineers are less. Then you have the onsite techs and the usuall overhead people like HR, finance, marketing etc. And don't forget the taxes paid on employees like FICA and unemployment insurance. Then there are health and other benefits. A true cost of an employee is almost twice their salary.

    Taxes to the various levels of government are another biggie. The rate is around 1/3 of revenues before deductions. Not everyone can be like Cisco and MS and not pay any taxes. Most companies will pay 10%-20% of revenues to taxes. So next time all you socialists complain about the tax breaks on business, think about who really pays business taxes. You do.

    Equipment costs. Something is always getting upgraded and new stuff is always being bought. Then there are the maintenance contracts for existing equipment and software. Our Veritas Netbackup Datacenter support is $15K a year. And that is just one application.

    Then other fixed costs like bandwith to higher level ISP's, elecricity, and real estate. Electricity is expensive. Say you pay $100 a month for your home and you turn of everything when not in use. Now think of an ISP with thousands of servers, routers, PC's and other junk running 24x7 with UPS's and digital quality electricity. And not to mention a special deal with the power company to be on the priority list in case of power shortages. That is some serious cash.
  • by Elequin (137149) on Sunday January 05, 2003 @03:13PM (#5020699)
    I think the answer probably has something to do with the fact that ISPs have to have higher bandwidth available than is used on average. Say that on average, an ISPs customer base takes up 30 megabit... To serve it's customers, the ISP could probably get away with having just one DS3. However, almost no ISP that size would have only one peer, so the ISP would most likely have 2 or even 3 DS3s to seperate peers, for redundancy in case there was a problem with one, through BGP they could just temporarily switch traffic to the other two. (say a DS3 goes down because of a DDoS attack, or some equipment fails..) They would also get this extra bandwidth to be prepared for things such as /.'ing or other bandwidth killing anolomies.

    So. The ISP has 3 DS3s, but it's customer base only uses about 30 mbit of all that bandwidth on average. It probably wouldn't be fair to charge all the customers the same for their usage, because #1, there are quite a few customers that are never going to pass much traffic, and #2, if you did that, your prices would not be competitive, and you wouldn't get many customers..

    Solution: the customers that DO take up more than average bandwidth get to bear the burden of helping to pay for that extra bandwidth. While Joe and Jim have colocated servers that stay right around 500kbits per second average, but Bob has a colocated server that jumps up to 10 megabit for a few days, it makes sense to get Bob to pay a higher charge for the bandwidth the ISP makes available for instances such as that. Also, as more and more customers start to take up more and more bandwidth, they're also going to be paying for the costs of adding bandwidth as it's needed. New routers, new cabling and the manpower needed for such an event, not to mention the costs of the new DS3 or OC3 or whatever are probably quite a hit at startup, financially.

    If the ISP didn't have the extra bandwidth available, when something did happen, customers would get very upset about the bog down that would inevitably happen. Upset customers = losing money..
  • by seac0rd (557659) on Sunday January 05, 2003 @05:13PM (#5021237)

    It all comes down to the cost of a pizza. In 1982, Judge Green Ruled on the Modified Final Judgment (MFJ), also known as the ruling that broke up the Bell Telephone Company into RBOCs. Anyway, part of the judgment was that residential users shouldn't pay more a month than the cost of a medium, one topping pizza. Its a true story. In order to do that, the commercial customer have to pick up part of the tab. That's done through connection charges. AT&T charges your RBOC for every minute that they use their lines, your RBOC turns around and charges ISP's.

    With a mess of calculations, statistics, and QOS numbers it can be calculated that if you use 3 MB a month instead of 1 MB then the RBOC will incurred an increased fee from AT&T. They pass that along to the ISP. However, this will all change when Qwest gets authorization to do inter-LATA calls. They can go around AT&T and connect directly and not incurred the connection charge.

    Expect big changes in the near future.

  • by puzzled (12525) on Sunday January 05, 2003 @06:41PM (#5021750) Journal
    I run a small wireless ISP that covers a five county area, three of them in a 700k population urban area, two rural counties that border the city.

    A single T1 from a reputeable provider (Sprint, UUNet, AT&T on good days) still costs around $1,000 in this part of the country. You can get slightly cheaper T1s from [IC]LECs ($800 or so) but if you later multihome with a tier 1 provider you'll end up lopsided. If you grow enough to need a significant fraction of a DS3 your cost per T1 equivalent can drop from the $1,000 mark to about $500.

    I've posted this little equation a number of times before and I don't think its worn out yet.

    one 256k full time music/movie trader == 4 channels on a T1 == (4 channels/24 total channels) * $1,000 ~= $160/mo cost to an ISP, dropping to $80/mo if they're a big ISP.

    In addition to circuit costs I pay antenna site leases for building roofs, towers, and the like. I've got a small office and the debt on the equipment used in the network.

    Unlike various venture capital created wireless beasts flopping around in my market, my company is 'organic' - it has financing (money used to buy equipment) but no funding (money used for operations). My partners and I all have other sources of income (consulting, equipment sales, very tolerant spouses, etc) and we won't draw paychecks until we've got the recurring revenue to support that.

    We face one other wireless player in our space with a first generation bridged only network and a not so good reputation (I love you guys! Don't change a thing!). We also face a large, agressive cable company turned CLEC, DSL is widely available from the ILEC and another CLEC, and the ILEC has a fairly useable, cheap ISDN offering.

    At the entry level for business we get between $75 and $130 for a connection up to 768k. We charge $250 per megabit with the assumption that anyone using that sort of speed is going to have VPN apps on the connection and they'll require additional hand holding.

    In addition to the business customers we have one of the rural counties that is actively producing residential customers @ $29.95 monthly. I wouldn't miss them if they all dropped dead tomorrow, since they're mostly those that wore out their welcome at their local DSL provider and now they're Kazaaing up my network.

    T1s are full duplex. Did you know this? DSL/cable users are amazed when they learn this fact. I like to explain it to them on a phone call rather than in person, so I can interrupt them while they're telling me a T1 can't be 1.5 mbit symetric. Think about it :-)

    So, I've got N T1s x 1.5 mbits of inflow and the same amount of outflow. The business customers use bandwidth from about 0700 - 1700, the residential customers from about 1500 when kids get home from school till about 2400 - basically two completely separate markets I can serve with the same inflow bandwidth. The ratio of inflow to outflow is about 5:1 with 'normal' customers.

    The Kazaa thieves upset this 5:1 rule of thumb greatly and should really be counted as a wholesale symetric customer and charged accordingly. The CEO won't approve this charge plan, so I'm approaching it BOFH style, and anyone running Kazaa or similar services has just volunteered to be a victim^h^h^h^h^h^h subject in my various QoS experiments.

    How did we decide how much to charge?

    You need to understand that while I have an in depth understanding of what people are doing with the IP network I manage I *don't care* what they're doing, so long as they're not annoying the others trying to use it, or doing stuff that will get various three letter government agencies serving subpoenas to my office.

    I happen to like playing with IP networks which makes being CTO a pretty fun job, but we could just as well have a warehouse full of various widgets that we buy for $2 and sell for $3 - its just an investment.

    All networks built with more equipment than you can carry in one large RubberMaid(tm) storage container are investments. They depreciate a certain amount monthly, they have various expenses involved in operation including the next tier ISP cost, and hopefully, if you're doing things right, you're ending up with more $$$ on the 31st of December than you had January 1 of the same year.

    So, there you have it. Its a *business*. Take a business class if this is all fuzzy to you, or better yet just start an ISP on your own and we'll see how long you think sharing MP3s and movies via the internet is a 'right'.
  • <NITPICK>

    Apples aren't grown from seeds as each seed in an apple is genetically different from the tree it came from (and from each other). This is how the apple as a species got so diverse. To reliably grow the same variety over and over, scions of the desired variety are grafted onto rootstocks.

    </NITPICK>

Aren't you glad you're not getting all the government you pay for now?

Working...