Internet Partitioning - Cogent vs Level 3? 450
slashmicah asks: "Internet partitioning and Tier 1 ISPs are something most people don't know much about (myself included). Today, however, some Slashdot readers might have run into some issues involving these two topics. Cogent Communications and Level 3, both Tier 1 ISPs, are apparently having some 'undisclosed' disagreements, causing an Internet partition by turning-off or deactivating their peering point. Cogent Co. has released a statement explaining their side of the problem, however they have no mention of when the problem will be fixed, or when they will sort it out. This partitioning is a problem because any [single-homed] computers that are connected through Cogent Co, can not connect to [single-homed] computers connected through Level 3. Having spent all day sorting out this problem, I ask Slashdot: Isn't there a better way that the issue of peering can be handled/regulated? If not, does the future hold a scenario in which the Internet is split into several separate networks, only to be connected at the whims of large corporations?"
Your statement is incorrect (Score:1, Informative)
For an example of this visit https://www.linx.net/www_public/our_members/peerin g_matrix/ [linx.net]
that shows the peering matrix at the London Internet Exchange.
PITA but move along (Score:5, Informative)
At least it's not like UUNET more than one, some years ago, wanting to charge other Tier 1's per packet for transfer when peering while their traffic they wanted to pass for free. They were a big dog and were trying to make everyone pay. No one did and threatened to or did kill off traffic until UUNET got the sh*t together. But the did try to pull it off more than once.
quote attribution (Score:4, Informative)
Been dealing with this all day. (Score:5, Informative)
Re:Your statement is incorrect (Score:3, Informative)
Re:Consider switching to someone less petulant (Score:5, Informative)
As to the notion by another poster of not expecting peering with someone bigger for free forever: 38.0.0.0/8 Class A is Cogent/PSI... how much bigger than being an entire Class A (and then some?!) does one have to be to be considered [ahem] "equal"?
It was a mutual arrangement: they both allowed transit for one another's packets... pretty fair given the size and stature of them both, I'd say.
Cogent's message (via NANOG) (Score:5, Informative)
Level 3's official statement (Score:5, Informative)
They're saying Cogent is intentionally not advertising routes to them via other providers, presumably because they're upset about not having a peering agreement in place. Anyone affected by this presumably needs to harass Cogent.
http://ws.arin.net/whois?queryinput=AS174 [arin.net]
Re:Interesting scenario, though most likely untrue (Score:3, Informative)
"Cogent will offer any Level 3 customer, who is single homed to the Level 3 network as of October 5, 2005,
one year of full Internet transit free of charge at the same bandwidth currently being supplied by Level 3.
Cogent will provide this connectivity in over 1,000 locations throughout North America and Europe."
Not that I really know what that means, or whether their claim that Level3 cut things off really makes Level3 the bad guys. Anyone want to explain for those of us that don't get it?
Some thoughts on this mess (Score:5, Informative)
Second, it is dishonest for Level-3 to blame Cogent for this exclusively. Level-3 had a peering arrangement with Cogent for a long time. If you look at Level-3's interconnection policy page:
http://www.level3.com/1511.html [level3.com]
It still looks like Cogent and Level-3 could peer under these terms. It was Level-3 that pulled the plug, not Cogent.
What is really annoying is that this is only traffic from Level-3 to Cogent, not to other parts of the internet. Level-3 wants money for Cogent customers to connect to Level-3's network but does not understand that this is a two-way connection and that Cogent's customers and Level-3's customer both benefit from this equally.
Up until this point, I was very happy with Level-3. They run an excellent network and I pay top-dollar to be on it. This blatent disregard for the impact on their customers is a diservice to their customers, to their reputation, and only begs for regulation.
That's transit (Score:5, Informative)
Re:OK, WTF time here (Score:4, Informative)
Cogent is not a bonified teir 1 as they still pay for some of there transit.
Peering 101 (Score:5, Informative)
Provider A and Provider B peer, be it public or private, normally they do this in several places and alternate who pays for the circuit, etc. Now, under normal circumstances, they both push enough traffic from one to the other to justify this mutual payment plan. However, in some cases, you find that B is either intentionally dumping traffic into A thinking A won't notice, or A discovers that its sending so little traffic to B in comparison to the amount B is sending to A that its not worth the continued cost.
When the first sort of thing happens, it usually gets resolved -REALLY- quick, that sort of behavior is not tolerated and will result in B getting de-peered by A (and potentially others once the abusive behavior is discovered and known) exceptionally quick unless B can show that it wasn't done knowingly or intentionally.
When the second instance happens
Based on Cogent's 'oh poor us' post from this morning, I'm leaning towards them having given L3 the finger when L3 said 'look, this isn't equitable, we're going to have to re-arrange the money'.
YMMV of course, but I'm betting I'm not terribly far off.
Re:Consider switching to someone less petulant (Score:5, Informative)
Yeah, Level 3 is really out of line in my opinion. It's not that they shut down the peering link. That wouldn't be that big of a deal. The traffic would just flow through other providers on less efficient routes. It's not as though every single backbone carrier peers with every other. But I just checked my BGP sessions, and Level 3 is not advertising the Cogent route at all. And you know for a fact that Level 3 is receiving the Cogent route from many of it's other peers. But it appears that they are intentionally filtering out the Cogent route. Which is pretty much not playing by the rules. It's one thing to shut down a peering agreement. It's something else entirely to refuse to accept that route from any of your other peers.
Re:WRONG!!! (Score:4, Informative)
How to Complain (Score:4, Informative)
720-888-2518 (Level3 Investor Relations)
and complain.
Or call 877-453-8353 (Main customer service number).
Re:Connect through another peer? (Score:3, Informative)
Re:Interesting scenario, though most likely untrue (Score:5, Informative)
What they are essentially saying is: "We haven't done anything. We haven't made any changes on our side" Level 3 have terminated their connection to Cogent "Without cause". Now, that's probably legal speak on Cogent's side for we haven't got the letter in triplicate yet telling us what the reason is. Or otherwise whoever put up the notification about it doesn't know.
Now, Cogent may have tried to change the peering arrangement, or Level 3 may have too, one side probably didn't agree, or otherwise an agreement ran out and the switch got flipped. This has happened previously with Cogent in their peering arrangement with AOL. [findlaw.com]
What Cogent are trying to do is get business from Level 3 customers because Level 3 stopped the connection. Cogent is offering them connections to the Cogent network (And subsequently Cogent's customers) for a year with no fee on the amount of data they put through. That connection itself they will obviously have to pay for, but the customer can connect into (presumably) the closest of any of 1000 points across North America and Europe.
Now some people are already connected to both Cogent and Level 3. These people won't have any problems as they will be able to go direct into either ISP. These people would probably have never have used the interconnect between Cogent and Level 3 either, unless one of their connections into either Cogent or Level 3 went down.
I understand this is still rather technical, for a simpler version, take a look through the document that I linked to.
Re:OK, WTF time here (Score:5, Informative)
When Paul Baran [wikipedia.org] had the task of designing a network that could withstand a nuclear attack, he envisioned a "distributed network". By today's lingo, it's a mesh network where each router is connected to approximately the same number of other routers. But now that routing infrastructure is driven commercially, with tit-for-tat contracts between Tier 1 ISPs, we ended up with what he said was a "decentralized network" -- that is, power law. Not what Paul Baran had in mind. If the underlying topology were his distributed network, you wouldn't be reading this story.
You can read his paper here [rand.org]. The Internet could withstand one nuclear attack. Several well-placed nuclear attacks? That's debatable...
- shadowmatter
Re:Connect through another peer? (Score:2, Informative)
Re:Level3 has a reason to be scared of Cogent (Score:4, Informative)
NANOG Archives (Score:3, Informative)
Re:Interesting scenario, though most likely untrue (Score:3, Informative)
Re:Consider switching to someone less petulant (Score:2, Informative)
Has nothing to do with size of IP address range.
Has everything to do with how much travels from Cogent to L3 for L3's customers and how much travels from L3 to Cogent for Cogent's customers. As long as those amounts are roughly the same (someone I consider a network guru once told me 4:1 or closer) then it makes sense for both peers to allow the traffic for the other's customers without charging anything for that service.
I don't know if this is why L3 cut off the connection, but if it is I think they are in the right. In peering, each side controls a toll road and are letting the others' customers use their road so that their customers can use the other's road. If Cogent's customers are using L3's road but L3's aren't using Cogent's, why should L3 continue the peering relationship? If L3's customers complain, then L3 will have to do something about it. Otherwise, Cogent will have to make a deal with L3 to reopen the pipe - I imagine huge amounts of money may be involved.
Partitioning occasionally happens (Score:5, Informative)
I'd bet L3's argument is that they will not provide transit across their AS to Cogent. It's a play that's been made several times before. The first time I know of it being done was in 1995 when Sean Doran pulled this at the CIX-W router, preferring to take commercial traffic via NSFNET or Sprint reseller service. Not only didn't it work, but it caused some immediate political backlash as Sean's action (presumably made without his boss's approval, who was the chairman of the CIX board and took some political grief for Sean's latest stunt) caused several state's to literally drop off the map.
If my memory's right, I think this got pulled again around 1998 timeframe on Exodus by someone like Genuity (I may be wrong about the culprit), only for the higher ups at the culprit to discover they couldn't see half of the world's worthwhile websites and search engines. Much of this was in the transit battle - e.g. if you had consumers, you felt your eyeballs were the value of the Internet and all other ISPs should pay you to get to your consumers, while if you were a content provider, you had the stuff all those consumers were paying their ISP to get to and someone had better pay you for that content.
What can you do about it? Let your ISP know you're not paying them for 80% of the Internet. When UUNET considered pulling this stunt around 1997, I worked for a small software shop that had a couple bonded UUNET T1's and we let them know we were going to drop them the moment they were only selling partial Internet. Then follow through if they do (UUNET backed off). Bilateral agreements are weird things in the world of settlement-free IP exchange, so unless you want a settlement-driven Internet (which will have unusual effects you might not want, like driving a per-packet pricing model), just expect this occasionally and drop those who don't play well with others. When L3 drops customer base, even the Denver boys will figure out their customers aren't happy.
*scoove*
Level 3 = Failing Business = Retribution (Score:5, Informative)
The arrogance of Jim Crowe [workign on his 7th manion and 8th large layoff at Level 3] and Kevin O'Hara (President, CEO) is only matched by the jailbird Bernie Ebbers. They only reason they have yet to decalre bankruptcy and liquidate thier debt (and clear away their bad business model with a fresh debt-free start) is that all their Omaha cronies have tied up money in the company stock, which would be flushed.
That they would resort to stunts like this against companies that undermine their pricing is not surprising. Level 3 have amassed BILLIONS in debt that they cannot service at current pricing levels, while Cogent and other more nimble competitors can sustain operations and drain Level3 into bankruptcy. So Level 3 execs do what arrogant desperate people do: lash out.
Level 3 is playing the "Sampson" card - if they cant make people price it their way, they will take the internet down with them.
And they did this trying to kill XO and now Cogent. Watch for more until they finally admit their business model is a failed one, and they declare bankruptcy, wipe the debt, and then begin to price lower and rake in the profits that their debt service is now eating.
It's happened.... (Score:4, Informative)
That either network corporation allowed this to occur is without pardon.
What I'm afraid of, is when this is all over and people realize how singificant it was, the solution to mangers will be "buy service to each, so we never have to worry about being partioned". Which is exactly what both companies would like to see.
Re:Level 3 = Failing Business = Retribution (Score:5, Informative)
Putting on my investor's cap, and taking a quick look at financials, its obvious that Level 3's burn-rate on cash, and billions in debt is not looking good for them if they cannot start generating both higher margins and more revenues. Neither one individually will save them at this point. The debt service is eating what EBITDA revenue they have coming in faster than they can produce it. And with companies like Cogent undercutting them Level 3 is dying; it seems the only question now is how much interconnectivity they will destroy in fits of pique like this.
I think you may also be right on another point, after considering it and runnnign the numbers: if Level 3 were to reorganize in bankruptcy court, dump the current shareholders, turn the debt holders into stock holders to ditch the debt, then they would probably be very profitable at even lower pricing levels. After all, that is what a lot of their competition has done. If they do that, Level 3 will cut the throats of every company out there, and make a bundle doing it, free-market style. Pretty interesting scenario.
But first they have to drop the stockholders, and from your post, it sounds like cronyism is a big factor, so its only going to happen when there has been far too much damage to Level 3 as a company. Thats a shame, because looking at their web site, they have some good ideas, but the wrong time and place for them.
Thanks for the post AC (wow an AC that actually said something useful!)
Re:WRONG!!! (Score:2, Informative)
Several things:
1) The size of your ISP's ARIN/APNIC/RIPE netblock allocation is not exactly related to the concept of peering parity. (more on this in a moment..)
2) If you use the Internet for "mission critical" applications, YOU should a) be multi-homed on multiple ISP backbones, verify that they have good peering with backbones you need to transit and have your own BGP AS OR b) leverage a single providers' network to the extent possible, thereby elininating problems like these.
3) You buy from Cogent - YOU GET WHAT YOU PAY FOR.
4) Peering. From dictionary.com "Peer" - 1: a person who is of equal standing with another in a group. Notice the use of the word "Equal"? Its there for a reason. If you take the time to read the fine print of peering agreements (http://global.mci.com/uunet/peering/ [mci.com]) , you will see that section 1.2 states:
Traffic Exchange Ratio. The ratio of the aggregate amount of traffic exchanged between the Requester and the MCI Internet Network with which it seeks to interconnect shall be roughly balanced and shall not exceed 1.8:1.
Translation: If you dump more traffic on us than we dump to you, then we have an asymmetric relationship. You are not worthy of being my peer because you take more than you give, which lumps you in with the rest of my customers who must PAY for access.
The above peering language is similar for all major Internet backbones.
5) I would expect that Cogent is present in at least one of the public peering points (Mae+pick your favorite ordinal direction) - so their BGP reachability information should be flowing through the MAE's.... Should...
Let this be the lesson - if you build a network on $10/mbit access, you get $10/mbit access. Usually, it's great, but sometimes bad things happen. Even to good fiscally responsible people, like yourselves. Good luck in your next job..
Transit vs. nontransit service (Score:5, Informative)
When you buy Internet bandwidth from your ISP, you are getting transit service. This means that you can use the link to send traffic to that ISP and to other ISP's upstream from it.
Nontransit service means that the link is to be used exclusively for sending traffic to that one ISP.
All of the Tier 1 ISP's provide nontransit service to each other, because at tier 1 there is no such thing as "upstream." This is not people playing stupid, this is how it's done at the top. It's the reason why the major peering points exist.
Any ISP who wants to shut off a peering arrangement for stupid business-o-political purposes is creating a hole in its own connectivity, and therefore shooting itself in the foot, plain and simple.
Level 3/Cogent Deal (Score:5, Informative)
Funny... but True. (Score:2, Informative)
If a nuke took out the original peer points, there's a chance software would fix the rest. (though, i'm not certain, the configs been written at this point. But if a nuke was the source of this problem, we'd mostly likely not have noticied the down time, only the large flash and radiation).
Re:This is bad. Very bad. (Score:3, Informative)
Cable modem users and DSL users are unaffected (unless Cogent is their provider's only upstream, and I haven't heard of any that are in that situation). They can use the vast majority of interesting services on the 'net just fine. The vast majority of high-traffic web sites are on multi-homed connectivity. There *are* popular sites on cruddy connectivity, but they'll learn quick with this, as they should. Never keep all your eggs in one basket. That lesson is as old as the commercialized Internet.
Those who are truly affected are businesses who have web sites and are unwilling to pay for enterprise-class connectivity. Not L3's problem man, ya get what you pay for. If your customers suffer, then I suggest you buy some real bandwidth. And if you're complaining because the free sites can't afford it, well, that's just tough. There's no SLA from ANY provider that says "you can reach 100% of the internet 100% of the time," let alone from a $19.95/month web host or a $50/mo cable provider. Again, you get what you pay for. Good connectivity costs good money.
As for the home users who are whining about this, y'all need to get a life and just go outside when your favorite site is offline for a day or two. It ain't the end of the world.
I design and build high-traffic internet-connected web sites on a hardware and software level for a living. I've never managed to work in this arena for someone who would build a completely single-homed system. Most serious Internet businesses recognize the need for multi-homing before they begin the install. Any failure to implement redundancy in those environments is what we call "acceptable risk." The folks that don't realize the need quickly change their willingness level after their first major outage.
Most, though, just lease space at a colo that has decent connectivity already installed.
On another note, I *am* on RR. Cogent doesn't exist from my net.perspective. I read quite a few sites regularly, and I have yet to run accross one that's affected. The impact of this event is quite small unless you happen to have the misfortune of being a Cogent customer.
BTW, as far as "it's not your job", yes, it most certainly is if you're in Internet-related support. Internet support techs (be it for an ISP or a web site or what have you) have had to explain outage causes to normal people since day 1. And once those folks realize that Cogent sucks, they'll move on. Cogent screwed themselves by not playing ball with L3 from what I've heard, and ultimately they're the ones that will pay for that business decision when customers leave them in droves.
--S
Re:Interesting scenario, though most likely untrue (Score:5, Informative)
Read about it here [2sparrows.org] - warning: it's a 120-page pdf...
-David
Re:Interesting scenario, though most likely untrue (Score:4, Informative)
BGP really *IS* automated, and you clearly have never worked for a large ISP. Disclaimer: I've worked for both of the two largest ISPs, and had backbone access at each, within the last five years. I am not currently employed by either of them.
If you're connected to an ISP who has connections to both Cogent and L3, you're fine. By definition, that includes any actual Tier 1 ISP (UUNet, AT&T, Qwest, etc)
If you're a customer of an ISP who is a customer of one of those particular Tier One providers, you're okay. Your packets will route to either L3 or Cogent as appropriate
The real problem is if you're either a customer of or a customer's customer of Cogent or L3 - at that point there's a disconnect. Both L3 and Cogent are significant wholesale dialup providers, so a lot of dial customers are affected.
-David
Re:Some thoughts on this mess (Score:3, Informative)
Re:This is bad. Very bad. (Score:4, Informative)
Explicit dual-homing directly with tier-1s, no.
But I do IT at a medium-small business, and we have a fairly simple solution to this.
We get our internet service from a multi-homed tier-2.
Problem solved.
I agree, this seems very, very bad - Not so much the situation itself, but the fact that, at any random moment, ALL the tier-1s could arbitrarily choose to end their peering agreements, turning the internet quite literally into the Bushism "internets".
But for any individual customer, they do have the power to prevent one such schism from limiting their connectivity simply by their choice of an ISP.
As an aside, I have to admit I don't really understand why Level-3 would do this. Regardless of the dominant direction of traffic between the two networks, every packet sent still has two sides involved - One a paying customer of Level 3, and one a paying customer of Cogent. So which side should pay for which direction? The question doesn't even make sense - A peering agreement improves both sides.
Re:Interesting scenario, though most likely untrue (Score:2, Informative)
Ah, there isn't any routing info that has to change at the modem end. The way routing works is that you don't know the whole route from end to end, it's figured out as you go.
From your computer there's only 1 path it can take, out the modem and to the ISP (unless you have multiple connections to your home network). Where to goes from there depends on the destination and how the ISP has their network setup. A pretty typical situation would be something like this:
So, there's nothing your modem has to "know" about the routing, it just sends it to the ISP's routers for them to figure it out. If it's taken them that long to fix the problem then they probably don't have a good multi-homed setup or they have a lot of static routing that needs to be changed or Level 3 and/or Cogent are still advertising the route as valid but are blocking the traffic.
TWRR can change their routing tables all the time. They could completely change backbone providers for that matter. The only effect the end user would see could be changes in response times and transfer speeds to various sites since it's taking a new route. There can be some disruption in the initial change over until the routing change propogates but that should be fairly quick on the local network.Re:Reminds me of... (Score:1, Informative)
[jlasman@da1 jlasman]$
traceroute to c.root-servers.net (192.33.4.12), 30 hops max, 38 byte packets
1 ge-6-1-155.hsa1.Tustin1.Level3.net (65.58.240.129) 0.295 ms !H
* 0.346 ms !H
No, wrong, again. (Score:4, Informative)
Listen, You seem to keep responding while ignoring what I'm actually saying, so i'm going to spell it out to you.
Customers of some ISPs that have routes out both to the L3 side and Cognent side CAN NOT access any Cognent controlled networks (AS174). In some cases it has to do with not knowing another route to that network. In other cases it has to do with Cognent blocking a path they just don't want used. Case A is Level3's issue, Case B is Cognents. Either way, the downstream guy is screwed.
Look. Here's me trying to get to Level3 side:
And here's me trying to get to something on the Cognent side:
The fact that RoadRunner is sending my packets via ATDN via Verio to get to AS174 shows me that the pinned route RR previously had (ie, all traffice for cognent side, haul via Verio which Cognent bought) is still up, but Cognent is actively blocking the traffic. If they didn't block it, we wouldn't know they were depeered and this would be a non-story. Now, I can't tell you that previously the data was backhauled via the AS3356 (Level3) network, but this is my guess. I just don't have any tracerts from then.
:
But not that Cognent is the only bad guy in this, Level3 has no advertised routes to AS174. Check http://www.level3.com/LookingGlass/ [level3.com]
And from what I read on NANOG they are filtering advertisments of the AS174 routes from reaching anyone on their side. So even if you could route through L3 to Sprint to get to Cognent, you wouldn't know.
Re:Funny... but True. (Score:3, Informative)
There have been a lot of mistaken things said about this: neither side is actually manipulating the routing table - L3 just removed the only way for it and Cogent to exchange routes.
-David
Follow the Money! Cogent's lost before (Score:4, Informative)
Cogent's business model is to sell large bandwidths for a low price, usually in multi-tenant office buildings. So they'd drop a fiber into the basement, and sell 100 Mbps ethernet connections to businesses in the building for about the price other carriers would charge for a T1 (that was back when a T1 was typically $1000 instead of $300; I haven't followed Cogent's prices in the last year or two.) Could you expect to get 100 Mbps consistently all the time? Not realistically, but you *could* expect to get lots more bandwidth than a T1 almost all the time, so it was a pretty competitive deal.
But at the end of the Interent boom, every carrier's finances looked pretty unstable, and a very aggressive business model that depends on getting free peering from big carriers while stealing their business customers looked extremely volatile :-) So does it make business sense for a Tier 1 provider to peer with Cogent as opposed to charging them money for Transit? Maybe, maybe not, and it looks like Level3 used to give them free peering but has changed their mind about it. Not the first time something like that has happened to Cogent - they've been back and forth on this with one or more carriers over the last few years. L3 seems to have decided that there's not enough reason to care about Cogent customers to give them free service.
50 days advanced warning; played chicken (Score:3, Informative)
Re:It's happened.... (Score:3, Informative)
Cogent has to pay L3 because they aren't as significant a player as L3. Their "portion of the internet", as they call it, just isn't that relevant to a lot of people.
--S
Re:50 days advanced warning; played chicken (Score:3, Informative)
Afaict the buisness model of cognet is to keep as much of thier traffic as possible as peering rather than upstream to allow them to profitablly offer very low bandwidth prices to thier customers.