TWT - Comcast congestion

Wed Dec 1 03:53:25 UTC 2010

In a message written on Tue, Nov 30, 2010 at 09:24:47PM -0600, Richard A Steenbergen wrote:
> I never said otherwise. The PR is pretty clear: Level 3 says that 
> Comcast, their TRANSIT CUSTOMER, demanded that Level 3 pay them because 
> of a ratio imbalance. Level 3, not wanting to cause massive disruptions 
> to their other customers who would then no longer be able to reach 
> Comcast (or depending on your point of view, because of an extreme lack 
> of testicular fortitude), complied, and then put out a PR whining about 
> it.

I'm not privy to the deal, but I will point out as reported it makes no
sense, so there is something else going on here.  This is where both
sids are hiding the real truth.  I suspect it's one of two scenarios:

- Comcast demanded a lower price from Level 3, which Level 3 has spun
  as paying Comcast a monthly fee.

- Comcast said they would do settlment free peering with Level 3, in
  addition to, or in place of transit.  Level 3 is spinning the cost
  of turning this up as paying Comcast a fee.

I suspect we'll not know what terms were offered for many years.

> In some ways it IS a net neutrality issue. Comcast is effectively "too 
> big" to turn off, and has used the threat of disruption to it's massive 
> customer base to bully a transit provider into paying its customer for 
> the right to deliver service. Comcast has made it quite clear that their 
> goal is to charge content companies for access to their customers, which 
> if I'm not mistaken is what the whole net neutrality thing (at least 
> originally) was all about. :)

Yes and no.  First off, network neutrality is a vaguely defined
term, so I'm not going to use it.  Rather I'm going to say I think
many people agree there is a concept that when it comes to traffic
between providers there should be roughly similar terms for all
players.  Comcast shouldn't give Netflix a sweetheart deal while
making Youtube pay through the nose.

The problem is that many of the folks want to conflate the ability
to be treated equal, with the ability to do whatever they want.
For instance, consider these "equivilent" interconnect models:

1 GE in 100 cities.
10 GE in 10 cities.
100 GE in 1 city.

All of these could support a 70G traffic flow between networks, but
the costs to provision all three in ports, backbone, and mangement
are wildly different.  If two networks have 70G of traffic does
network neutrailty mean one can demand 1GE in 100 cities, and the
other can get a single 100GE in 1 city and the person on the other
end has to deal with both like it or not?

> The funny part is that Level 3 was clearly ill prepared for the PR war, 
> whereas Comcast, being the first mover (if not the first PR issuer), was 
> well prepared.

Really?  I just checked google news again, and the first statement I can
find by either side was a Level 3 submission to business wire:

http://www.marketwatch.com/story/level-3-communications-issues-statement-concerning-comcasts-actions-2010-11-29?reflink=MW_news_stmp

If you can find a Comcast story before that I'd love to read it.

> > What I wonder is why Netflix and Comcast are letting middle men like 
> > Level 3 and Akamai jerk both of them around.  These two folks need to 
> > get together and deal with each other, cutting out the middle man....
> 
> Netflix is a Comcast customer too (again well established publicly and 
> easily provable via the global routing table), but they don't run their 
> own server infrastructure, and Comcast doesn't offer a CDN service...

Right, Netflix is a Comcast customer for www.netfix.com, e.g. the
web site where you select movies.  No streaming comes from that
source as far as I can tell, so it's really a sort of red herring
in this discussion.

I realize Netflix is chosing to outsource their streaming, but
there's no reason they can't outsouce the running of the servers
while controlling a direct IP relationship with Comcast, if they
don't want to run the servers in house.

> The reality is that Level 3 offered Netflix a cut-throat price on CDN 
> service to steal the business from Akamai, probably only made possible 
> by the double dipping mentioned above. They were already in for a world 
> of hurt based on their CDN infrastructure investment and the revenue 
> they were able to extract from it, this certainly isn't going to help 
> things. :)

I feel you undercut your network neutrality argument right here, because
you make an argument that this is just two competitive businesses trying
to get a leg up on each other.  You can't have the fairness part of
network neutrality and try and stab each other in the back at every
step.

To be clear, I don't think either Level 3 or Comcast is in the right
here, or well, really in the wrong.  It's easy to make both arguments:

Level 3: They have been our customer for a long time, and now want
         a lower price, or a fee, or to convert to peering just because
         we added a customer, how is that fair?

Comcast: These guys cut a deal to move 10's of Gigabits of traffic from
         entering our network at one point to entering at different
         locations far away, and then gave us ~45 days
         notice that we just have to suck it up and deal with it.  How
         is that fair?

But it is business, as much as us technical folks like to think
about peering in technical terms, of how to move traffic between
two networks and share costs all of the top 20-30 networks treat
peering as a weapon.  They weild it to force other networks to
connect where they want and how they want.  It's a continuous game
of brinksmanship and screwing each other, typically done in private.

Neither Level 3 nor Comcast here are interested in the fairness of
network neutraility, or even interested in helping their customers.
They are interested in hurting their "competitors" and boosting
their own bottom line.

I bet the cash spent on lawyers and lobbiests taking this to the FCC on
both sides could pay for enough backbone bandwidth and router ports to
make this problem go away on both sides many times over.  If they really
cared about the customers experience and good network performance they
would put away the press release swords, the various VP and CxO's egos,
and come up with a solution.

That will never happen.

-- 
       Leo Bicknell - bicknell at ufp.org - CCIE 3440
        PGP keys at http://www.ufp.org/~bicknell/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 826 bytes
Desc: not available
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20101130/81607cce/attachment.sig>