Internet Partitioning - Cogent vs Level 3?

slashmicah asks: "Internet partitioning and Tier 1 ISPs are something most people don't know much about (myself included). Today, however, some Slashdot readers might have run into some issues involving these two topics. Cogent Communications and Level 3, both Tier 1 ISPs, are apparently having some 'undisclosed' disagreements, causing an Internet partition by turning-off or deactivating their peering point. Cogent Co. has released a statement explaining their side of the problem, however they have no mention of when the problem will be fixed, or when they will sort it out. This partitioning is a problem because any [single-homed] computers that are connected through Cogent Co, can not connect to [single-homed] computers connected through Level 3. Having spent all day sorting out this problem, I ask Slashdot: Isn't there a better way that the issue of peering can be handled/regulated? If not, does the future hold a scenario in which the Internet is split into several separate networks, only to be connected at the whims of large corporations?"

Interesting scenario, though most likely untrue.

While it is always fun to entertain such doomsday scenarios in ones' mind, I don't think that anything like this is possible. Current demands of most large corporations (Microsoft, Apple, any number of others) along with the internet-using public are for a universally-connected internet. Any company that simply creates its own network is going to face a huge revenue loss.

Re:Interesting scenario, though most likely untrue

It's possibly on a par with the scenario of countries cutting others off their internet connections. Not that it can't be done, but the repercussions are akin to MAD. Although these days with the effective merger of state and corporate interests anything can happen...

Re:Interesting scenario, though most likely untrue

Anyone want to explain for those of us that don't get it?

What they are essentially saying is: "We haven't done anything. We haven't made any changes on our side" Level 3 have terminated their connection to Cogent "Without cause". Now, that's probably legal speak on Cogent's side for we haven't got the letter in triplicate yet telling us what the reason is. Or otherwise whoever put up the notification about it doesn't know.

Now, Cogent may have tried to change the peering arrangement, or Level 3 may have too, one side probably didn't agree, or otherwise an agreement ran out and the switch got flipped. This has happened previously with Cogent in their peering arrangement with AOL. [findlaw.com]

What Cogent are trying to do is get business from Level 3 customers because Level 3 stopped the connection. Cogent is offering them connections to the Cogent network (And subsequently Cogent's customers) for a year with no fee on the amount of data they put through. That connection itself they will obviously have to pay for, but the customer can connect into (presumably) the closest of any of 1000 points across North America and Europe.

Now some people are already connected to both Cogent and Level 3. These people won't have any problems as they will be able to go direct into either ISP. These people would probably have never have used the interconnect between Cogent and Level 3 either, unless one of their connections into either Cogent or Level 3 went down.

I understand this is still rather technical, for a simpler version, take a look through the document that I linked to.

Really easy solution all net-ops already know

Don't expect to peer with people bigger than you for free forever.

"A true peer relationship is based on the supposition that either party
can terminate the interconnection relationship and that the other party
does not consider such an action a competitively hostile act. If one
party has a high reliance on the interconnection arrangement and the
other does not, then the most stable business outcome is that this
reliance is expressed in terms of a service contract with the other
party, and a provider/client relationship is established"

Level3 is threatened by Cogent's bandwidth pricing model, and is using it's weight to threaten that model, forcing Cogent to buy transit if it wants to reach its network. THat's how things work: you can't get free bandwidth from everyone, you're going to have to be willing to step up and pay for your link.

PITA but move along

Okay, it's a pain in the ass for customers and others BUT they are businesses that negotiate peering, sometimes with monopoly money changing hands, sometimes without. It's a business dispute and it's not like Germany and France closing roads and making you drive through Belgium. They'll resolve it or lose business. Their's more of a back story we don't know.

At least it's not like UUNET more than one, some years ago, wanting to charge other Tier 1's per packet for transfer when peering while their traffic they wanted to pass for free. They were a big dog and were trying to make everyone pay. No one did and threatened to or did kill off traffic until UUNET got the sh*t together. But the did try to pull it off more than once.

Been dealing with this all day.

we got an email this afternoon from our provider, who let us know that cogent will be reachable by their second link, which is WilTel. However, the link is slower than the Level3 link. There will also be more traffic being routed through less points, meaning congestion. (and obvious lack of redundancy, if the WilTel connection has problems, no Level3) We have had users complaining about sites being unreachable at random times this afternoon. One of our providers very big customers is the OSU Open Source Lab, home of Drupal, mozilla download servers, master.Kernel.org servers, and many, many others. If your having problems reaching these sites, that is probably why.

public peering!

Everyone knows what a success MAE-East, MAE-West and the rest of the public peering points have been. Let's build a few more of them! Or, even better, encourage the federal government to get involved. Perhaps spending some of the federal budget on this problem would be a good idea. I think I recall a peering point clause in the constitution somewhere.

In all seriousness, these private companies will work it out when they realize that their paying customers are pissed and leaving because they're no longer selling very complete connectivity. Just like in the past, it won't take long. If TV has taught me anything, these problems are usually wrapped up pretty nicely in about 28 minutes.

As a rule...

I have been kicking around the fringes of the high-speed data stuff for a number of years, and there's one true lesson to be learned;

Telcos suck.

ALL of them do in their own special way.

Cogent's message (via NANOG)

I can't get to Cogent's website, but according to a NANOG post, this is whatit says:

Cogent Network Status/DNS Server Status Description:
Date: 10/05/2005

Level 3 has partitioned its part of the Internet from Cogent's part of the Internet by denying Level 3's customers access to Cogent's customers and denying Cogent's customers access to Level 3 customers. Level 3 terminated its peering with Cogent without cause (as permitted under its peering agreement with Cogent) even though both Cogent and Level 3 remained in full compliance with the previously existing interconnection agreement.

Cogent will offer any Level 3 customer, who is single homed to the Level 3 network on the date of this notice, one year of full Internet transit free of charge at the same bandwidth currently being supplied by Level 3. Cogent will provide this connectivity in over 1,000 locations throughout North America and Europe.

Cogent is committed to an open Internet. The existing interconnection facilities between Level 3 and Cogent remain intact. Cogent hopes that Level 3 will reactivate these connections, restoring a full level of service to their customers.

For more information about the sales offer, please contact the numbers listed below.
NORTH AMERICA: 1-877-875-4432
ANYWHERE ELSE IN EUROPE: [phone]

Level 3's official statement

"Recently, certain peers have been disconnected from their direct connection to the Level 3 IP network. Some disconnected peers may elect to block access to certain IP addresses as a result of the disconnection. If a peer elects not to restore connectivity to the Level 3 network through alternative means, customers seeking continued access to the Level 3 network should make alternate arrangements."

They're saying Cogent is intentionally not advertising routes to them via other providers, presumably because they're upset about not having a peering agreement in place. Anyone affected by this presumably needs to harass Cogent.

http://ws.arin.net/whois?queryinput=AS174 [arin.net]

Seen it before in Australia

Reminds me on an incident that happened in Australia a couple of years back. Telstra and Optus were pretty much owned all the links outside of Australia, but Telstra lost their major one in a shipping incident. (Sharp anchors?) With nowhere for their data to go they rerouted everything through Optus to let them handle it.

Optus didn't appreciate that and promptly blocked all data between themselves and Telstra. Customers with Telstra were pretty much screwed because they couldn't contact anything and with their network going nuts even sites within Telstra sucked a lot. Still, for a couple of days there, it was two halves of an internet available in here. Was amusing to watch really.

Easy to Fix

There's an easy technical fix to this problem: Start a nuclear war at the location of this peering point. Then by design the Internet will route around that area, and communications will be reestablished.

Some thoughts on this mess

First, I think that Level-3 is within it's legal rights in terms of dealing with Cogent, but is probably in trouble with it's customers. I am a customer of Level-3 and of Cogent (in the same facility). When I buy IP transit from Level-3, I am not buying "part of the internet". This peering issue places 45+ Million IP addresses out of reach of the Level-3 network (and vice versa). Level-3 did not notify me that they were making this type of change. There is nothing on Level-3's website that even implies that everything is not hunky dory. If you buy a Level-3 line today, will they disclose to you that you are not connecting to the entire internet. I know I am being a little niave here, but not disclosing such a large change of policy is unconscionable.

Second, it is dishonest for Level-3 to blame Cogent for this exclusively. Level-3 had a peering arrangement with Cogent for a long time. If you look at Level-3's interconnection policy page:

    http://www.level3.com/1511.html [level3.com]

It still looks like Cogent and Level-3 could peer under these terms. It was Level-3 that pulled the plug, not Cogent.

What is really annoying is that this is only traffic from Level-3 to Cogent, not to other parts of the internet. Level-3 wants money for Cogent customers to connect to Level-3's network but does not understand that this is a two-way connection and that Cogent's customers and Level-3's customer both benefit from this equally.

Up until this point, I was very happy with Level-3. They run an excellent network and I pay top-dollar to be on it. This blatent disregard for the impact on their customers is a diservice to their customers, to their reputation, and only begs for regulation.

This is bad. Very bad.

I have reviewed all information available at this time, including discussion threads on many sites more specialised than Slashdot. This is bad. Very bad. Right now, there are millions of Internet users with partial connectivity.

But the action of Level3 is not merely an inconvenience to end users; it is hurting a great many small businesses, badly. There are thousands of small businesses that depend on single-homed Internet connectivity and that cannot afford dual-homing. There are dozens of low-cost datacenters that provide single-homed bandwith to tens of thousands of servers.

As we speak, the livelyhoods of thousands of entrepreneurs are being threatened. Many people depend on being able to offer internet services to any peer on the net. But today, Level3 has changed the rules of the game, and have split the Internet into two somewhat isolated internets.

This is happening on a very large scale. Sure, most of the affected people and businesses are going to get through it just fine. But given the sheer scale of the Internet, a small percentage of those depending on full connectivity will not escape this ordeal unscathed.

You can be sure that a few small businesses will close because of this, the reputations of a few persons will be damaged, and there will be a few bankruptcies - all because of Level3's evil actions. You won't hear about it in the media - nobody cares about such small-scale damage. But the damage is already done, and it is getting worse with every passing hour.

I urge you to join me in a five-minute hate against Level3 and all that their evil discriminative ways stand for. While Cogent is widely recognized for its shitty cut-rate network, they are the good guys here. In the past few years, Cogent has been a major driving force for lowering bandwith costs. Level3 is fighting back, and they long for the days where they charged 5000$/mbps. I say: down with Level3 !

Peering 101

The short version goes something like this:

Provider A and Provider B peer, be it public or private, normally they do this in several places and alternate who pays for the circuit, etc. Now, under normal circumstances, they both push enough traffic from one to the other to justify this mutual payment plan. However, in some cases, you find that B is either intentionally dumping traffic into A thinking A won't notice, or A discovers that its sending so little traffic to B in comparison to the amount B is sending to A that its not worth the continued cost.

When the first sort of thing happens, it usually gets resolved -REALLY- quick, that sort of behavior is not tolerated and will result in B getting de-peered by A (and potentially others once the abusive behavior is discovered and known) exceptionally quick unless B can show that it wasn't done knowingly or intentionally.

When the second instance happens .. well .. you get what happened today (I'm making an educated guess here based on what I know of the two carriers involved). A decides that spending 30 grand a month for what is a very lopsided bandwidth agreement is no longer economically feasible or reasonable. They go to B and say 'look, we're not doing this anymore, we're basically paying a hell of a lot of money every month for you to send a ton of traffic to us, and we don't send much of anything to you. You can either pay for all (or some larger portion of) the circuits, pony up some $$ per megabit, or we'll just cut it off at the stub and be done.'

Based on Cogent's 'oh poor us' post from this morning, I'm leaning towards them having given L3 the finger when L3 said 'look, this isn't equitable, we're going to have to re-arrange the money'.

YMMV of course, but I'm betting I'm not terribly far off.

More Regulation != Solution

I can't believe I'm hearing a call for more regulation, even U.N. control. The lack of rational thought and disregard of unintended consequences amazes me.

The Internet has flourished without much control, run by Both large and small businesses for one reason: profit. Information is free yet its distribution is profitable.

If we give government control (taxation, censorship and worse(, we'll see less freedom.

Why did this jinx happen? Because the top tier providers aren't making a profit. But their calls for support go unheard, so they found a way to make it news.

When businesses that rely on the infrastructure paid for by private industries, they have high expectations. But they're not paying for that infrastructure!

Trust me, no one wants to bifurcate the Internet. Its a ploy to show a problem that needs to be solved. You will Never see it done for control, censorship or monopoly powers. You'll only see it when consumers don't pay for what they use. See California's old electric company that was forced to sell energy at a loss. They went bankrupt.

Re:Come on

> Why can't we all (Cogent and Level 3 included) just get along?

The Internet's insecure enough without introducing race conditions into it.

Besides, that's so 1990s. The 21st century equivalent is to yell "Tier 1 ISPs don't care about spam victims!"

Re:Consider switching to someone less petulant

As a cogent customer, it's *really* not their fault, IMHO. I gather L3 pulled the same stunt with XO last week.

As to the notion by another poster of not expecting peering with someone bigger for free forever: 38.0.0.0/8 Class A is Cogent/PSI... how much bigger than being an entire Class A (and then some?!) does one have to be to be considered [ahem] "equal"?

It was a mutual arrangement: they both allowed transit for one another's packets... pretty fair given the size and stature of them both, I'd say.

Re:Consider switching to someone less petulant

As a cogent customer, it's *really* not their fault, IMHO. I gather L3 pulled the same stunt with XO last week.

Yeah, Level 3 is really out of line in my opinion. It's not that they shut down the peering link. That wouldn't be that big of a deal. The traffic would just flow through other providers on less efficient routes. It's not as though every single backbone carrier peers with every other. But I just checked my BGP sessions, and Level 3 is not advertising the Cogent route at all. And you know for a fact that Level 3 is receiving the Cogent route from many of it's other peers. But it appears that they are intentionally filtering out the Cogent route. Which is pretty much not playing by the rules. It's one thing to shut down a peering agreement. It's something else entirely to refuse to accept that route from any of your other peers.

Re:OK, WTF time here

Yes if only there were some mechanisms built into the internet as a whole that would allow for the forwarding of packets through a number of alternate routes if one link were to go down... "Routing" if you will.

I envision such a system could be seriously robust and would possibly withstand a nuclear attack.

Re:OK, WTF time here

  I envision such a system could be seriously robust and would possibly withstand a nuclear attack.

Wow, sounds like something that could be of great strategic military value. I wonder if the DOD would be interested in developing this idea...?

Re:OK, WTF time here

The Internet is a power law network, meaning there are some very well-connected routers out there that a lot of the end-to-end transfers through the Internet go through. These are typically the peering points, owned by Tier 1 ISPs. It's not inconceiveable that if two ISPs don't peer with each other anymore, at some level a partition is created.

When Paul Baran [wikipedia.org] had the task of designing a network that could withstand a nuclear attack, he envisioned a "distributed network". By today's lingo, it's a mesh network where each router is connected to approximately the same number of other routers. But now that routing infrastructure is driven commercially, with tit-for-tat contracts between Tier 1 ISPs, we ended up with what he said was a "decentralized network" -- that is, power law. Not what Paul Baran had in mind. If the underlying topology were his distributed network, you wouldn't be reading this story.

You can read his paper here [rand.org]. The Internet could withstand one nuclear attack. Several well-placed nuclear attacks? That's debatable...

- shadowmatter

Re:OK, WTF time here

Okay, first off if your physician is using any IP-based service and ISN'T using a dedicated connection then no one's to blame except the fucktard who set it up in the first place.

1. Set down coffee cup;
2. Open front door;
3. Take deep breath;
4. Introduce yourself to reality.

Physicians are subject to the greater economic pressures than any other small business. Insurance companies, government regulations, litigation risks, patient scheduling, qualified and reliable staffing, emergency on-call, and obligitory hospital fundraising contributions. Given a choice between an $1,800/month point-to-point circuit PLUS provider termination and service fees, or a $59/month xDSL for probably 4x the bandwidth, which do you think most will use? If in doubt, give your physician a call and report back.

That's transit

Peering is when you agree to send traffic destined to network X directly to network X via a direct connection between you and X. If you're using X's network to send traffic to Y, that's transit, and X will naturally expect you to pay for the privilege.

Re:Reminds me of...

Actually it appears that L3 has done more than just drop direct peering with Cogent. L3 seems to be filtering all traffic sourced from Cogent controlled IP blocks. So unless Cognet sets up a NAT arrangement with other peers, there is no way around this problem. L3 is actively blocking Cogent traffic.

If your company or ISP uses only Cogent for bandwidth, it is currently impossible to reach L3 only connected services. I believe L3 to Cogent is being blocked as well.