TWT - Comcast congestion

Wed Dec 1 03:24:47 UTC 2010

On Tue, Nov 30, 2010 at 06:45:57PM -0800, Leo Bicknell wrote:
> Actually it appears to be Level 3 who fired the first PR salvo running 
> to the FCC, if the date stamps on the statements are right. So it's 
> really Level 3 framing as a net neutrality peering issue the fact that 
> Comcast balked at paying them more.

I never said otherwise. The PR is pretty clear: Level 3 says that 
Comcast, their TRANSIT CUSTOMER, demanded that Level 3 pay them because 
of a ratio imbalance. Level 3, not wanting to cause massive disruptions 
to their other customers who would then no longer be able to reach 
Comcast (or depending on your point of view, because of an extreme lack 
of testicular fortitude), complied, and then put out a PR whining about 
it.

In some ways it IS a net neutrality issue. Comcast is effectively "too 
big" to turn off, and has used the threat of disruption to it's massive 
customer base to bully a transit provider into paying its customer for 
the right to deliver service. Comcast has made it quite clear that their 
goal is to charge content companies for access to their customers, which 
if I'm not mistaken is what the whole net neutrality thing (at least 
originally) was all about. :)

> Netflix is today apparently delivered via Akamai, who has nodes deep 
> inside Comcast.  Maybe Akamai pays Comcast, I actually don't think 
> that is the case from an IP transit point of view, but I think they do 
> pay for space and power in Comcast data centers near end users.  But 
> anyway, this Netflix data is close to the user, and going over a 
> settlement free, or customer connection.

Netflix is today delivered by LimeLight and Akamai, who are both very 
clearly and publicly acknowledged customers of Comcast (though the LLNW 
deal is VERY fresh), as well as by Level 3 CDN. Level 3 CDN recently 
(and very publicly) won a lot of Netflix's business, but they're by no 
means new customers.

> Level 3 appears to have sucked Netflix away, and wants to double dip 
> charging Netflix for the transit, and Comcast for the transit. Worse, 

Absolutely they wanted to double dip. If you've seen the prices that 
Level 3 is selling it's CDN services for, you'd know they'll need to 
quadruple dip just to break even. :)

Comcast wants to double dip too. They're not satisfied with receiving 
the traffic via a peer for free, they want to be paid on both sides.

So yes you effectively have a battle of two companies who want to double 
dip. The major difference is that Level 3 accomplished its double dip by 
providing quality service at a reasonable price in an environment with a 
significant amount of competition, while Comcast accomplished its double 
dip by hosting its (mostly captive) customer base hostage, and 
intentionally creating congestion via every alternate path. If Comcast 
was winning customers by offering better, cheaper, faster service, they 
would have a leg to stand on, but the reality is the only thing they're 
offering is access to their captive eyeball customers.

The funny part is that Level 3 was clearly ill prepared for the PR war, 
whereas Comcast, being the first mover (if not the first PR issuer), was 
well prepared.

> they get to triple dip, since they are Comcast's main fiber provider.  
> Comcast will have to buy more fiber to haul the bits from the Equinix 
> handoffs to the local markets where Akamai used to dump it off.  Worse 
> still, Level 3 told them mid-novemeber that the traffic would be there 
> in december.  Perhaps 45 days to provision backbone and peering to 
> handle this, during the holiday silly season. Perhaps Level 3 wanted 
> to quadruple dip with the expedite fees.

I think you're making a lot of assumptions which have no basis in fact 
above, unless you know something I don't, which based on what I've read 
so far I don't think you do. Again, there is no peering, Comcast is a 
Level 3 transit customer. Until a month ago a lot of this content was 
being delivered by LLNW via Global Crossing, until Comcast threatened 
LLNW with intentional congestion of it GX paid peers, and forced them to 
buy directly to keep Netflix happy. This is far from the first time this 
issue has come up, and Comcast has established a very clear pattern of 
trying everything in its power to force content companies to pay for 
uncongested access.

> Yet with all of this Level 3 runs to the FCC screaming net neutrality. 
> Wow.  That takes balls.  Comcast did itself no favors respnding with 
> "it's a ratio issue" rather than laying out the situation.

If you refused to pay your transit provider, they'd probably just shut 
you off. The problem is that Comcast is too big to just shut off, and 
would no doubt tell it's customers that "Level 3 did it" (just like they 
have every other time someone has complained about their congested 
transits), that's why they're whining.

> What I wonder is why Netflix and Comcast are letting middle men like 
> Level 3 and Akamai jerk both of them around.  These two folks need to 
> get together and deal with each other, cutting out the middle man....

Netflix is a Comcast customer too (again well established publicly and 
easily provable via the global routing table), but they don't run their 
own server infrastructure, and Comcast doesn't offer a CDN service...

The reality is that Level 3 offered Netflix a cut-throat price on CDN 
service to steal the business from Akamai, probably only made possible 
by the double dipping mentioned above. They were already in for a world 
of hurt based on their CDN infrastructure investment and the revenue 
they were able to extract from it, this certainly isn't going to help 
things. :)

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)