Level 3 Communications Issues Statement Concerning Comcast's Actions
brunopeter at gmail.com
Tue Nov 30 14:57:47 CST 2010
GigaOm has begin tracking this story:
On Tue, Nov 30, 2010 at 1:02 PM, Leo Bicknell <bicknell at ufp.org> wrote:
> Having been involved with a few peering spats in the past I know
> what is said publically rarely matches the reality behind the scenes.
> In this particular case my spidy sense tells me there is absolutely
> something interesting behind the scenes, but the question is what.
> I'd never really paid attention to how Netflix delivers its content.
> It's obviously a lot of bandwidth, and likely part of the issue
> here so I thought I would investigate.
> Apparently Akamai has been the primary Netflix streaming source
> since March. LimeLight Networks has been a secondary provider, and
> it would appear those two make up the vast majority of Netflix's
> actual streaming traffic. I can't tell if Netflix does any streaming
> out of their own ASN, but if they do it appears to be minor.
> Here's a reference from the business side of things:
> This is also part of the reason I went back to the very first message in
> this thread to reply:
> In a message written on Mon, Nov 29, 2010 at 05:28:18PM -0500, Patrick W. Gilmore wrote:
>> I understand that politics is off-topic, but this policy affects operational aspects of the 'Net.
> Patrick works for Akamai, it seems likely he might know more about
> what is going on. Likely he can't discuss the details, but wanted
> to seed a discussion. I'd say that worked well.
> I happen to be a Comcast cable modem customer. Gooling for people
> who had issues getting to Netflix streaming turned up plenty of
> forum posts with traceroutes to Netflix servers on Akamai and
> Limelight. I did traceroutes to about 20 of them from my cable
> modem, and it's clear Comcast and Akamai and Comcast and Limelight
> are interconnected quite well. Akamai does not sell IP Transit,
> and I'm thinking it is extremely unlikely that Comcast is buying
> transit from Limelight. I will thus conclude that these are either
> peering relationships, or that they have cut some sort of special
> "CDN Interconnect" deal with Comcast.
> But what about Level 3? One of my friends I was chatting with on AIM
> said they thought Comcast was a Level 3 customer, at least at one time.
> Google to the rescue again.
> Level 3 provides fiber to Comcast (20 year deal in 2004):
> Level 3 provides voice services/support to Comcast:
> Perhaps the most interesting though is looking up an IP on Comcast's
> local network here in my city in L3's looking glass:
> Slightly reformatting for your viewing pleasure, along with my comments:
> Community: North_America
> Level3_Customer # Level 3 thinks they are a customer
> Suppress_to_AS174 # Cogent
> Suppress_to_AS1239 # Sprint
> Suppress_to_AS1280 # ISC
> Suppress_to_AS1299 # Telia
> Suppress_to_AS1668 # AOL
> Suppress_to_AS2828 # XO
> Suppress_to_AS2914 # NTT
> Suppress_to_AS3257 # TiNet
> Suppress_to_AS3320 # DTAG
> Suppress_to_AS3549 # GBLX
> Suppress_to_AS3561 # Savvis
> Suppress_to_AS3786 # LG DACOM
> Suppress_to_AS4637 # Reach
> Suppress_to_AS5511 # OpenTransit
> Suppress_to_AS6453 # Tata
> Suppress_to_AS6461 # AboveNet
> Suppress_to_AS6762 # Seabone
> Suppress_to_AS7018 # AT&T
> Suppress_to_AS7132 # AT&T (ex SBC)
> So it would appear Comcast is a transit customer of Level 3 (along with
> buying a lot of other services from them). I'm going to speculate that
> the list of supressed ASN's are peers of both Level 3 and Comcast, and
> Comcast is going that so those peers can't send some traffic through
> Level 3 in attempt to game the ratios on their direct connections to
> Now a more interesting picture emerges. Let me emphasize that this is
> AN EDUCATED GUESS on my part, and I can't prove any of it.
> Level 3 starts talking to Netflix, and offers them a sweetheart deal to
> move traffic from Akamai to Level 3. Part of the reason they are
> willing to go so low on the price to Netflix is they will get to double
> dip by charging Netflix for the bits and charging Comcast for the bits,
> since Comcast is a customer! But wait, they also get to triple dip,
> they provide the long haul fiber to Comcast, so when Comcast needs more
> capacity to get to the peering points to move the traffic that money
> also goes back to Level 3! Patrick, from Akamai, is unhappy at losing
> all the business, and/or bemused at the ruckus this will cause and
> chooses to kick the hornets nest on NANOG.
> One last thing, before we do some careful word parsing. CDN's like
> Akamai and LimeLight want to be close to the end user, and the
> networks with end users want them to be close to the end user. It
> doesn't make sense to haul the bits across the country for any party
> involved. Akamai does this by locating clusters inside providers
> networks, LimeLight does it by provisioning bandwidth from their
> data centers directly to distribution points on eyeball networks.
> So let's go back and look at the public statements now:
> Level 3 said:
> "On November 19, 2010, Comcast informed Level 3 that, for the first
> time, it will demand a recurring fee from Level 3 to transmit Internet
> online movies and other content to Comcast's customers who request such
> Comcast said:
> "Comcast has long established and mutually acceptable commercial
> arrangements with Level 3's Content Delivery Network (CDN) competitors
> in delivering the same types of traffic to our customers. Comcast
> offered Level 3 the same terms it offers to Level 3's CDN competitors
> for the same traffic."
> You can make both of these statements make sense if the real situation
> is that Comcast told Level 3 they needed to act like a CDN if they were
> going to host Netflix. Rather than having Comcast pay as a customer,
> they needed to show up in various Comcast distribution centers around
> the country where they could drop traffic off "locally". To Comcast
> this is the same deal other CDN's get, it matches their statement. To
> Level 3, this means paying a fee for bandwidth to these locations, and
> being that they are Comcast locations it may even mean colocation fees
> or other charges directly to Comcast. Comcast said if you're not going
> to show up and do things like the CDN players then we're going to hold
> you to a reasonable peering policy, like we would anyone else making us
> run the bits the old way.
> The most interesting thing to me about all of this is these companies
> clearly had a close relationship, fiber, voice, and IP transit all on
> long term deals. If my speculation is right I'm a bit surprised Level 3
> would choose to piss off such a long term large customer by bringing
> Netflix to the party like this, which is one of the reasons I doubt my
> speculation a bit.
> But, to bring things full circle, neither of the public statements tell
> the whole story here. They each tell one small nugget, the nugget that
> side wants the press to run with so they can score political points.
> Business is messy, and as I've said throughout this thread this isn't
> about peering policies or ratios, there are deeper business interests
> on both sides. This article:
> Suggests Level 3 is adding 2.9 Terabits of capacity just for Netflix.
> I'm sure a lot of that is going to Comcast users, since they are the
> largest residential broadband ISP.
> Messy. Very messy.
> Leo Bicknell - bicknell at ufp.org - CCIE 3440
> PGP keys at http://www.ufp.org/~bicknell/
More information about the NANOG