Level 3 Communications Issues Statement Concerning Comcast's Actions

Tue Nov 30 20:57:47 UTC 2010

GigaOm has begin tracking this story:

http://gigaom.com/2010/11/30/a-play-by-play-on-the-comcast-and-level-3-spat

On Tue, Nov 30, 2010 at 1:02 PM, Leo Bicknell <bicknell at ufp.org> wrote:
>
> Having been involved with a few peering spats in the past I know
> what is said publically rarely matches the reality behind the scenes.
> In this particular case my spidy sense tells me there is absolutely
> something interesting behind the scenes, but the question is what.
>
> I'd never really paid attention to how Netflix delivers its content.
> It's obviously a lot of bandwidth, and likely part of the issue
> here so I thought I would investigate.
>
> Apparently Akamai has been the primary Netflix streaming source
> since March.  LimeLight Networks has been a secondary provider, and
> it would appear those two make up the vast majority of Netflix's
> actual streaming traffic.  I can't tell if Netflix does any streaming
> out of their own ASN, but if they do it appears to be minor.
>
> Here's a reference from the business side of things:
> http://www.fool.com/investing/general/2010/11/11/netflix-takes-streaming-to-a-new-level.aspx
>
> This is also part of the reason I went back to the very first message in
> this thread to reply:
>
> In a message written on Mon, Nov 29, 2010 at 05:28:18PM -0500, Patrick W. Gilmore wrote:
>> <http://www.marketwatch.com/story/level-3-communications-issues-statement-concerning-comcasts-actions-2010-11-29?reflink=MW_news_stmp>
>>
>> I understand that politics is off-topic, but this policy affects operational aspects of the 'Net.
>
> Patrick works for Akamai, it seems likely he might know more about
> what is going on.  Likely he can't discuss the details, but wanted
> to seed a discussion.  I'd say that worked well.
>
> I happen to be a Comcast cable modem customer.  Gooling for people
> who had issues getting to Netflix streaming turned up plenty of
> forum posts with traceroutes to Netflix servers on Akamai and
> Limelight.  I did traceroutes to about 20 of them from my cable
> modem, and it's clear Comcast and Akamai and Comcast and Limelight
> are interconnected quite well.  Akamai does not sell IP Transit,
> and I'm thinking it is extremely unlikely that Comcast is buying
> transit from Limelight.  I will thus conclude that these are either
> peering relationships, or that they have cut some sort of special
> "CDN Interconnect" deal with Comcast.
>
> But what about Level 3?  One of my friends I was chatting with on AIM
> said they thought Comcast was a Level 3 customer, at least at one time.
> Google to the rescue again.
>
> Level 3 provides fiber to Comcast (20 year deal in 2004):
> http://blog.tmcnet.com/blog/tom-keating/voip/level-3-and-comcast.asp
>
> Level 3 provides voice services/support to Comcast:
> http://cable.tmcnet.com/news/2005/jul/1168088.htm
>
> Perhaps the most interesting though is looking up an IP on Comcast's
> local network here in my city in L3's looking glass:
> http://lg.level3.net/bgp/bgp.cgi?site=sjo1&target=68.86.240.141
>
> Slightly reformatting for your viewing pleasure, along with my comments:
>
>      Community: North_America
>                 Lclprf_100
>                 Level3_Customer       # Level 3 thinks they are a customer
>                 United_States
>                 San_Jose
>                 EU_Suppress_to_Peers
>                 Suppress_to_AS174     # Cogent
>                 Suppress_to_AS1239    # Sprint
>                 Suppress_to_AS1280    # ISC
>                 Suppress_to_AS1299    # Telia
>                 Suppress_to_AS1668    # AOL
>                 Suppress_to_AS2828    # XO
>                 Suppress_to_AS2914    # NTT
>                 Suppress_to_AS3257    # TiNet
>                 Suppress_to_AS3320    # DTAG
>                 Suppress_to_AS3549    # GBLX
>                 Suppress_to_AS3561    # Savvis
>                 Suppress_to_AS3786    # LG DACOM
>                 Suppress_to_AS4637    # Reach
>                 Suppress_to_AS5511    # OpenTransit
>                 Suppress_to_AS6453    # Tata
>                 Suppress_to_AS6461    # AboveNet
>                 Suppress_to_AS6762    # Seabone
>                 Suppress_to_AS7018    # AT&T
>                 Suppress_to_AS7132    # AT&T (ex SBC)
>
> So it would appear Comcast is a transit customer of Level 3 (along with
> buying a lot of other services from them).  I'm going to speculate that
> the list of supressed ASN's are peers of both Level 3 and Comcast, and
> Comcast is going that so those peers can't send some traffic through
> Level 3 in attempt to game the ratios on their direct connections to
> Comcast.
>
> Now a more interesting picture emerges.  Let me emphasize that this is
> AN EDUCATED GUESS on my part, and I can't prove any of it.
>
> Level 3 starts talking to Netflix, and offers them a sweetheart deal to
> move traffic from Akamai to Level 3.  Part of the reason they are
> willing to go so low on the price to Netflix is they will get to double
> dip by charging Netflix for the bits and charging Comcast for the bits,
> since Comcast is a customer!  But wait, they also get to triple dip,
> they provide the long haul fiber to Comcast, so when Comcast needs more
> capacity to get to the peering points to move the traffic that money
> also goes back to Level 3!  Patrick, from Akamai, is unhappy at losing
> all the business, and/or bemused at the ruckus this will cause and
> chooses to kick the hornets nest on NANOG.
>
> One last thing, before we do some careful word parsing.  CDN's like
> Akamai and LimeLight want to be close to the end user, and the
> networks with end users want them to be close to the end user.  It
> doesn't make sense to haul the bits across the country for any party
> involved.  Akamai does this by locating clusters inside providers
> networks, LimeLight does it by provisioning bandwidth from their
> data centers directly to distribution points on eyeball networks.
>
> So let's go back and look at the public statements now:
>
> Level 3 said:
> http://www.marketwatch.com/story/level-3-communications-issues-statement-concerning-comcasts-actions
> -2010-11-29?reflink=MW_news_stmp
>
>  "On November 19, 2010, Comcast informed Level 3 that, for the first
>   time, it will demand a recurring fee from Level 3 to transmit Internet
>   online movies and other content to Comcast's customers who request such
>   content."
>
> Comcast said:
> http://blog.comcast.com/2010/11/comcast-comments-on-level-3.html
>
>  "Comcast has long established and mutually acceptable commercial
>   arrangements with Level 3's Content Delivery Network (CDN) competitors
>   in delivering the same types of traffic to our customers. Comcast
>   offered Level 3 the same terms it offers to Level 3's CDN competitors
>   for the same traffic."
>
> You can make both of these statements make sense if the real situation
> is that Comcast told Level 3 they needed to act like a CDN if they were
> going to host Netflix.  Rather than having Comcast pay as a customer,
> they needed to show up in various Comcast distribution centers around
> the country where they could drop traffic off "locally".  To Comcast
> this is the same deal other CDN's get, it matches their statement.  To
> Level 3, this means paying a fee for bandwidth to these locations, and
> being that they are Comcast locations it may even mean colocation fees
> or other charges directly to Comcast.  Comcast said if you're not going
> to show up and do things like the CDN players then we're going to hold
> you to a reasonable peering policy, like we would anyone else making us
> run the bits the old way.
>
> The most interesting thing to me about all of this is these companies
> clearly had a close relationship, fiber, voice, and IP transit all on
> long term deals.  If my speculation is right I'm a bit surprised Level 3
> would choose to piss off such a long term large customer by bringing
> Netflix to the party like this, which is one of the reasons I doubt my
> speculation a bit.
>
> But, to bring things full circle, neither of the public statements tell
> the whole story here.  They each tell one small nugget, the nugget that
> side wants the press to run with so they can score political points.
>
> Business is messy, and as I've said throughout this thread this isn't
> about peering policies or ratios, there are deeper business interests
> on both sides.  This article:
> http://www.marketwatch.com/story/level-3-outlines-network-expansion-on-netflix-pact-lvlt-nflx-llnw-akam-2010-11-11
>
> Suggests Level 3 is adding 2.9 Terabits of capacity just for Netflix.
> I'm sure a lot of that is going to Comcast users, since they are the
> largest residential broadband ISP.
>
> Messy.  Very messy.
>
> --
>       Leo Bicknell - bicknell at ufp.org - CCIE 3440
>        PGP keys at http://www.ufp.org/~bicknell/
>