Cogent/Level 3 depeering

Wed Oct 5 19:22:55 UTC 2005

On Wed, Oct 05, 2005 at 10:43:51AM -0700, Jeff Shultz wrote:
> 
> Undereducated rant to follow...
> 
> While I realize that the "nuke survivable" thing is probably an old 
> wives tale, it seems ridiculous that "the Internet" can't adjust by 
> routing any packets that used to go directly from Cogent to Level 3 
> though some 3rd (and) 4th (and) 5th set of providers that are connected 
> in some fashion to both...
> 
> Level 3 and Cogent can't be operating in a vacuums - if we can get to 
> Kevin Bacon in 6 degrees, Level 3 and Cogent should be able to get to 
> each other in under 30 hops through other providers.
> 
> And why isn't this apparently happening automatically? Pardon the 
> density of my brain matter here, but I thought that was what BGP was all 
> about?
> 
> I welcome any education the group wishes to drop on me in this matter.

Internet connectivity is only as good as the people who are willing to buy 
it. If you wanted to connect to the Internet, you would pay someone money 
to deliver the packets to/from you to the complete Internet, yes? They do 
this by either connecting with every other network out there, or by in 
turn paying someone else to deliver the data that you paid them to 
deliver. This is called transit.

Now, sometimes when two networks of roughly equal size and value to each 
other have customer bases that need to talk to each other, they will set 
up circuits between the two and not charge each other for the traffic 
passed over it, for the SOLE purposes of exchanging traffic with each 
others' customer base. This is called peering.

If you carry this trend all the way out to the maximum extent possible, 
you end up with a network that is so big that it doesn't have to pay 
anyone else to "deliver the bits for it", it interconnects with EVERYONE 
else that it would send bits to via peering, and everything else is a 
customer. This is called a "tier 1", of which there are only a handful 
(not counting marketing-land, where everyone claims to be a tier 1).

So, what you have here is a battle of wills between two very large 
networks. One is a legitimate "tier 1" (and one of the biggest IP networks 
in the world), the other is "really really close", only a couple of 
networks away from being a tier 1. The network who is "really really 
close" is still buying transit to reach a few destinations, but they want 
to be a tier 1. This means that the transit they are buying is not "full 
transit" in the way that you would normally think of it, instead they are 
buying "selected routes" to the few remaining networks they don't peer 
with. This is a kind of "tier 1 by technicality", not having "earned" it 
through true "settlement free peering" the way that a true tier 1 has, but 
by intentionally paying your transit provider to "emulate" peering with 
the remaining networks who they don't peer with directly.

Now, when said "big" peer comes along and says "we don't want to swap 
traffic for free with you any more", the smaller network doesn't want to 
let them go. Besides the obvious fact that they don't want to have to 
start paying money for traffic that was previously free, they don't want 
to look "weak" by caving in and buying transit, incase other networks who 
previously peered with them decide that they can depeer and force said 
network to pay THEM money for transit too. So, the smaller network 
intentionally chooses to remain unreachable and not buy transit, under the 
hopes that the customers of the larger network will complain enough that 
they are forced to "repeer".

So, the bottom line is that the two networks "could" be reachable to each 
other if they wanted to, but they are intentionally choosing not to do so. 
Level 3 "could" repeer Cogent (which Cogent wants but Level 3 doesn't), 
and Cogent "could" buy transit (which Level 3 wants but Cogent doesn't), 
but it is currently a matter of waiting to see which side will blink first 
under the pain of pissed off customers who can't reach the full Internet. 
Whichever one blinks first loses. Cogent has successfully used this tactic 
in the past (Teleglobe), and unsuccessfully tried it as well 
(OpenTransit).

But that said, the Internet is working the way that it is intended. I 
believe folks have reported that Level 3 saw a loss of around 1200 
prefixes from Cogent, and Cogent saw a loss of around 4300 prefixes from 
Level 3. Out of a customer base of 11k and 57k respectively, this is 
relatively small (11% of Cogent's customer base and 7.5% of Level 3's 
customer base), since only single homed customers are affected. 
Unfortunately you can't make two networks who don't want to directly 
connect with each other or pay someone else to connect to the other 
network talk to each other if they don't want to. Usually these things 
iron themselves out within a few days, but these are certainly two of the 
largest and most pigheaded networks to go up against each other, so it 
could be interesting. Whining about it as a customer is one way to try and 
convince one side or the other to cave sooner, but you can pretty much be 
guaranteed that someone will end it before some judicial, regulatory, or 
law making body steps and makes them. :)

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)