Level 3's side of the story

JC Dill lists05 at equinephotoart.com
Sat Oct 8 14:24:06 UTC 2005


Eric Louie wrote:
> 
> DISCLAIMER:  From one of the clueless

As a disclaimer, I will point out that there are some in this debate who 
consider me clueless as well.  However, I don't believe that any of the 
following is in error.

> During this entire debaucle, I never saw any mention of:

I've mentioned most of it in prior posts but perhaps it hasn't been 
explained well enough for you to fully grasp it.  I hope this helps.

> 1)  Cogent sending "transit" traffic to Level3, which leads me to 

Cogent was sending peering traffic, not transit traffic.

> believe that all the traffic from Cogent through the peering points was 
> actually *destined* for Level3 customers.   

Yes, that's what peering is. Peering gets you the other ISP's customers 
but NOT all the rest of the internet that is reachable thru that ISP. 
Transit gets you all the internet that is reachable thru that ISP.

> Does the routing support this idea?  

Yes.

> Is it safe to assume the opposite, also... that only traffic 
> destined for Cogent customers came through the Level3 peering points?  

Yes.

> And that Level3 had one and only one path to Cogent (no one else 
> providing transit for them to Cogent AS'es?)

Yes.

> 2)  Level3 making any contingency for their own customers to reach 
> Cogent networks (any announcements to their own customers)

L3 made no contingency plans for their customers to reach Cogent (and 
visa versa).

For the L3 customers that were multi-homed, they had other paths to 
Cogent thru their other service providers.  Those that were not 
multi-homed had no path to Cogent when L3 cut the peering, because as a 
"tier 1" (the widely accepted definition of "tier-1" is that the network 
only has SFI (Settlement Free Interchange i.e. peering) links to other 
networks and thus no transit links) none of the other networks L3 
connects to will carry L3's traffic to a third network (another peer).

Cogent was a "tier 1" until prior de-peering incidents left them unable 
to reach other networks.  They solved this by buying filtered transit 
thru Verio to reach the networks they couldn't reach via peering.

L3 was hoping to force Cogent to increase that transit to include the 
traffic destined for L3's customers, thus raising Cogent's transport 
costs at no additional (transport) cost to L3.

> 3)  Possible traffic issues.  Was Cogent guilty of not transporting the 
> Level3-bound packets within the Cogent network to the closest 
> point-of-entry peer to the host in the Level3 network, therefore 
> "costing" Level3 transit of their own packets?  

Possible, in fact probable.  Most ISPs hand off traffic to peers under a 
"hot potato" policy, they hand it off at the closest point where they 
connect.  If the traffic is equal in both directions then this works. 
If the traffic is not equal, then this lowers the cost of the network 
that has high outbound traffic, as the other network bears the brunt of 
the total cost for transporting the combined traffic between their 
respective customers.

> In other words is it also a traffic engineering issue?

Or rather, that it could be mostly solved with a traffic engineering fix 
called "cold potato" routing, where the sending network carries the 
traffic as far as they can before handing it off to the recipient network.

Consider a simple hypothetical closest-exited network setup (hot potato 
routing) between 2 peers:


ISP Eyeballs:  Router-E1----2,000 Mile Link----Router-E2----Customer
                      |                                 |
                      |                                 |
                    peering                           peering
                       |                                 |
                       |                                 |
ISP Content: Server--Router-C1---2,000 Mile Link-----Router-C2

When the customer on ISP E (Eyeballs) requests content (web page, music 
file, etc.) from the server on ISP C (Content), packets travel like this:

Customer->Router-E2->Router-C2->Router-C1->Server

When the server returns traffic to the customer, traffic goes like this:

Server->Router-C1->Router-E1->Router-E2->Customer

The problem is the customer->server direction would typically be a 500 
byte request and 64 byte ACK packets, where as the server->customer data 
includes many 1500 byte data packets.  So, ISP Eyeballs may carry 2Mbs 
of data over its 2,000 mile link, where as ISP Content will only carry 
128Kbs over its 2,000 mile link.

Even though both companies met in the middle, ISP Content shifted  some 
of its costs to ISP Eyeballs.

Back when most ISPs had the same types of traffic (even mixes of content 
and eyeballs), they had even ratios which equalized this effect, as it 
was happening the same amount in both directions.  But as some ISPs 
started specializing in one type of content or the other, uneven flows 
were produced.  Some bean-counter felt that these uneven flows meant 
that the network that was sending more traffic should now pay for 
transit, even though this traffic was traffic that their own customers 
were requesting and paying them to transmit!

There are ways to deal with it though, like cold potato routing.

> Are some of the business issues solvable by proper engineering and 
> filtering (or statistics-jockeying)?

Yes, see above.

jc  (no coffee yet, errors possible)




More information about the NANOG mailing list