The FCC is planning new net neutrality rules. And they could enshrine pay-for-play. - The Washington Post

Mon Apr 28 03:52:19 UTC 2014

Apologies that I dropped offlist as I was out for the day.  I think the bulk of my thoughts on this have already been covered by others since, including e.g. Matt's poor grandmother and her phone dilemma in the "What Net Neutrality should and should not cover" thread.

Basically I think we're on the same page for the most part, with maybe some misunderstandings between us.

> I covered this scenario in more detail in my post "What Net Neutrality should and should not cover" but if you expand on the assumption that paying for an internet connection also pays for the direct connection of every party who you exchange traffic with then you have a scenario where only half the people connected to the Internet should have to pay at all for their connection because any scenario where people simply buy their own pipe would be considered "double billing".

I don't think anyone on the Netflix^H^H^H^H^H^H $ContentProvider side of this was saying that $ContentProvider should get everything handed to them on a silver platter.  $ContentProvider pays for transit sufficient to handle the traffic that their customers request.  $EyeballNetwork's customers pay it for internet access, i.e. to deliver the content that they request, e.g. from $ContentProvider.  That covers both directions here.  Links between $ContentProvider's transit provider and $EyeballNetwork were getting congested, and $EyeballNetwork refuses to upgrade capacity.  Where we were getting into the double-dip was $EyeballNetwork saying to $ContentProvide:  "Hey, we know you already pay for transit, but you're gonna have to pay us as well if you want us to actually accept the traffic our customers requested".

The alternate arrangement between $ContentProvider and $EyeballNetwork seems to be private peering, where again it would seem to be fair for each side to bring the needed transport and ports to peering points.  In recent history, though, it seems that $EyeballNetwork came out ahead in that agreement somehow.  Now, Tore brought up a good point on paid peering in cases where e.g. $EyeballNetwork is already exchanging traffic with $ContentProvider through existing peering or below their CDR on existing transit, and indeed it seems that was the case for $EyeballNetwork via peering with $CheapTransitProvider that $ContentProvider was using.  But it seems that $EyeballNetwork was having a pissing match with $CheapTransitProvider and refusing to upgrade ports.

"Okay", says $ContentProvider.  "How about we just peer directly."
"Sounds great," says $EyeballNetwork.  "Since we have to allocate capacity for this discrete from our existing peering capacity, you'll need to foot the bill for that."
"Huh?" says $ContentProvider.  "This could have been fixed by you increasing your peering capacity to match the traffic volume your users are requesting, but you didn't want to do that because of your tiff with $CheapTransitProvider.  Tell me again why we're paying for your side of this *in addition* to our own when we're only going this route because of a decision *you* made?"
"Because you need to reach our customers, and we're the only path to them, so we have leverage."
*blank stare*
"So you're willing to give your customers crappy service because your customers don't have alternate options and you think we need this more than you do?"
"That's a possibility."
"I hate you."
"I know; sign here please."

But, again, this is outside looking in.  For now, I'll pick up a copy of Bill Norton's Internet Peering book as per Bob's suggestion, for some light Sunday night reading.

Cheers,

--
Hugo

________________________________
From: Rick Astley <jnanog at gmail.com>
Sent: Sunday, April 27, 2014 8:45 AM
To: Hugo Slabbert
Cc: nanog at nanog.org
Subject: Re: The FCC is planning new net neutrality rules. And they could enshrine pay-for-play. - The Washington Post

If it were through a switch at the exchange it would be on each of them to individually upgrade their capacity to it but at the capacities they are at it they are beyond what would make sense financially to go over an exchange switch so they would connect directly instead. It's likely more along the lines of needing several 100G ports as Netflix is over 30% of peak usage traffic in North America:

"Netflix (31.6%) holds its ground as the leading downstream application in North America and together with YouTube (18.6%) accounts for over 50% of downstream traffic on fixed networks."  (source https://www.sandvine.com/trends/global-internet-phenomena/ )

That amount of data is massive scale. I don't see it as double dipping because each party is buying the pipe they are using. I am buying a 15Mbps pipe to my home but just because we are communicating over the Internet doesn't mean the money I am paying covers the cost of your connection too. You must still buy your own pipe in the same way Netflix would. I covered this scenario in more detail in my post "What Net Neutrality should and should not cover" but if you expand on the assumption that paying for an internet connection also pays for the direct connection of every party who you exchange traffic with then you have a scenario where only half the people connected to the Internet should have to pay at all for their connection because any scenario where people simply buy their own pipe would be considered "double billing".

The cost for residential broadband is high enough in the US without a policy like that in place. If there is one policy that would keep poor families from being able to afford broadband it would be that one.

On Sun, Apr 27, 2014 at 2:58 AM, Hugo Slabbert <hslabbert at stargate.ca<mailto:hslabbert at stargate.ca>> wrote:

> ...but if that point of congestion is the links between Netflix and Comcast...

Which, from the outside, does appear to have been the case.

> ...then Netflix would be on the hook to ensure they have enough capacity to Comcast to get the data at least gets TO the Comcast network.

Which I don't believe was a problem?  Again, outside looking in, but the appearances seemed to indicate that Comcast was refusing to upgrade capacity/ports, whereas I didn't see anything indicating that Netflix was doing the same.  So:
> I have gear; you have gear.  I upgrade or add ports on my side; you upgrade or add ports on your side.

> The argument at hand is if Comcast permitted to charge them for the links to get to their network or should they be free/settlement free. I think it should be OK to charge for those links as long as its a fair market rate and the price doesn't basically amount to extortion.

Are we talking here about transport between Netflix's POPs and Comcast's?  I definitely don't expect Comcast to foot the bill for transport between the two, and if Netflix was asking for that I'm with you that would be out of line.  If there are existing exchange points, though, would it not be reasonable to expect each side to up their capacity at those points?

> Once that traffic is given directly to comcast no other party receives payment for delivering it so there is no double billing.

The "double-dip" reference was to charging both the content provider and the ISP's own customer to deliver the same bits.  If the traffic from Netflix was via Netflix's transit provider and Comcast then again was looking to bill Netflix to accept the traffic, we'd hit double billing.

I guess that's the question here:  If additional transport directly been POPs of the two parties was needed, somebody has to pay for the links.  Releases around the deal seemed to indicate that the peering was happening at IXs (haven't checked this thoroughly), so at that point it would seem reasonable for each party to handle their own capacity to the peering points and call it even.  No?

--
Hugo

________________________________
From: Rick Astley <jnanog at gmail.com<mailto:jnanog at gmail.com>>
Sent: Saturday, April 26, 2014 11:23 PM
To: Hugo Slabbert
Cc: nanog at nanog.org<mailto:nanog at nanog.org>
Subject: Re: The FCC is planning new net neutrality rules. And they could enshrine pay-for-play. - The Washington Post

>How is this *not* Comcast's problem?  If my users are requesting more traffic than I banked on, how is it not my responsibility to ensure I have capacity to handle that?  I have gear; you have gear.  I upgrade or add ports on my side; you upgrade or add ports on your side.  Am I missing something?

Sort of yes, it's Comcasts problem to upgrade subscriber lines but if that point of congestion is the links between Netflix and Comcast then Netflix would be on the hook to ensure they have enough capacity to Comcast to get the data at least gets TO the Comcast network. The argument at hand is if Comcast permitted to charge them for the links to get to their network or should they be free/settlement free. I think it should be OK to charge for those links as long as its a fair market rate and the price doesn't basically amount to extortion. Sadly the numbers are not public so I couldn't tell you one way or the other aside from I disagree with the position Netflix seems to be taking that they simply must be free. Once that traffic is given directly to comcast no other party receives payment for delivering it so there is no double billing.

This diagram best describes the relationship (ignoring pricing): http://www.digitalsociety.org/files/gou/free-and-paid-peering.png

"Content provider" would be Netflix and Comcast would be Broadband ISP 1.

On Sun, Apr 27, 2014 at 1:56 AM, Hugo Slabbert <hslabbert at stargate.ca<mailto:hslabbert at stargate.ca><mailto:hslabbert at stargate.ca<mailto:hslabbert at stargate.ca>>> wrote:
Okay, I'm not as seasoned as a big chunk of this list, but please correct me if I'm wrong in finding this article a crock of crap.  With Comcast/Netflix being in the mix and by association Cogent in the background of that there's obviously room for some heated opinions, but here goes anyway...

>A long, long time ago when the Internet was young and few, if any had thought
>to make a profit off it, an unofficial system developed among the network
>providers who carried the traffic: You carry my traffic and I'll carry yours
>and we don't need money to change hands. This system has collapsed under
>modern realities.

I wasn't aware that settlement-free peering had "collapsed".  Not saying it's the "only way", but "she ain't dead yet".

Seltzer uses that to set up balanced ratios as the secret sauce that makes settlement-free peering viable:
"The old system made sense when the amount of traffic each network was sending to the other was roughly equivalent."

...and since Netflix sends Comcast more than it gets, therefor Netflix needs to buck up:
"Of course Netflix should pay network providers in order to get the huge amounts of bandwidth they require in order to reach their customers with sufficient quality."

But this isn't talking about transit; this is about Comcast as an edge network in this context and Netflix as a content provider sending to Comcast users the traffic that they requested.  Is there really anything more nuanced here than:

1.  Comcast sells connectivity to their end users and sizes their network according to an oversubscription ratio they're happy with.  (Nothing wrong here; oversubscription is a fact of life).
2.  Bandwidth-heavy applications like Netflix enter the market.
3.  Comcast's customers start using these bandwidth-heavy applications and suck in more data than Comcast was betting on.
4.  Comcast has to upgrade connectivity, e.g. at peering points with the heavy inbound traffic sources, accordingly in order to satisfy their customers' usage.

How is this *not* Comcast's problem?  If my users are requesting more traffic than I banked on, how is it not my responsibility to ensure I have capacity to handle that?  I have gear; you have gear.  I upgrade or add ports on my side; you upgrade or add ports on your side.  Am I missing something?

Overall it seems like a bad (and very public) precedent & shift towards double dipping, and the pay-for-play bits in the bastardized "Open Internet" rules don't help on that front.  Now, Comcast is free to leverage their customers as bargaining chips to try to extract payments, and Randy's line of encouraging his competitors to do this sort thing seems fitting here.  Basically this doesn't harm me directly at this point.  Considering the lack of broadband options for large parts of the US, though, it seems that end users are getting the short end of the stick without any real recourse while that plays out.

--
Hugo

________________________________________
From: NANOG <nanog-bounces at nanog.org<mailto:nanog-bounces at nanog.org><mailto:nanog-bounces at nanog.org<mailto:nanog-bounces at nanog.org>>> on behalf of Larry Sheldon <LarrySheldon at cox.net<mailto:LarrySheldon at cox.net><mailto:LarrySheldon at cox.net<mailto:LarrySheldon at cox.net>>>
Sent: Saturday, April 26, 2014 4:58 PM
To: nanog at nanog.org<mailto:nanog at nanog.org><mailto:nanog at nanog.org<mailto:nanog at nanog.org>>
Subject: Re: The FCC is planning new net neutrality rules. And they could enshrine pay-for-play. - The Washington Post

h/t Suresh Ramasubramanian

FCC throws in the towel on net neutrality

http://www.zdnet.com/fcc-throws-in-the-towel-on-net-neutrality-7000028770/

Forward!  On to the next windmill, Sancho!
--
Requiescas in pace o email           Two identifying characteristics
                                         of System Administrators:
Ex turpi causa non oritur actio      Infallibility, and the ability to
                                         learn from their mistakes.
                                           (Adapted from Stephen Pinker)