Observations of an Internet Middleman (Level3) (was: RIP Network Neutrality (was: Wow its been quiet here...

Fri May 16 10:25:27 UTC 2014

>What you're missing is that the transit provider is
selling full routes.  The access network is selling
paid peering, which is a tiny fraction of the routes.

Considering they charge on a $per/mb basis I don't think its just routes
they are selling. It looks a lot like they are selling bits. From the
perspective of a content provider it looks like they swivel chair most
those bits to access networks for delivery. That "tiny fraction of routes"
on the access networks make up most delivered content. In total network
size the access networks are larger although less spread out globally.
Being globally connected is useful but it doesn't make a legitimate case
for having exclusive rights to charge for content being delivered in North
America. If you are planning to serve large scale data over oceanic fiber
it's a strong selling point but that's not the case here. If instead of
$per/mb traffic delivery you want to get into arbitrary justifications
access networks have more directly assigned IP addresses than transit
networks. I'm not making the case that a middle man should never be used,
I'm making the case that they shouldn't be used where there isn't a
requirement for one. Bypassing the middleman is generally better for
everyone but the middle man.

>So, at the end of the week, I *had* been paying $10/mb to
send traffic through transit to reach the whole rest of the
internet.  Now, I'm paying $5+$4+$4+$5+$2, or $30, and
I don't have a full set of routes, so I've still got to keep
paying the transit provider as well at $10.

If this is the math you are using to justify your stance it's probably
worth reconsidering. You ignore that each of those if sent through transit
would have been $10 so the cost of $5, $4, and $2/Mb represent a savings of
$5, $6,and $8. Why would you add them? Sure there are factors you have to
evaluate like putting yourself under a minimum commit with $transit or if
the amount of traffic is worth peering over but you would generally have to
make those evaluations for peering anyway. The real difference is the
volume of traffic needed before a $2/mb savings is worth peering directly
for is higher than if the savings were the full $10 but that doesn't mean
its never worth it. There is a difference between saying "I did the math
and transit remains the cheaper option" and saying "Paid peering would save
us both money and improve performance at the same time but we refuse to do
it anyway on principal".

The concept of fair gets brought up a lot when talking about the ability of
a startup to come in to compete against bigger players in the content space
but really what do you think the impact is if the largest established
content providers peer freely and smaller newcomers only have paid options
available for traffic?

Some other things I also want to get to:
>On Vi's analogy vs Amazon prime

One major different I think people overlook is overusing Amazon prime would
mean buying too many things from Amazon. Even when you purchase through
companies selling through Amazon they get a cut of the sale and some of
that I assume gets applied to covering any additional shipping costs not
covered by Prime. If Internet traffic used the same model would ISP's
receive a portion of proceeds for ad revenue on places like Youtube or a
percentage of Netflix subscription fees? I'm not making the case that thats
the model that should be used I'm only pointing out that analogies are best
to break things down into simple terms for people but have diminishing
returns in usefulness when getting into details.

The other problem with Vi's analogy is the shipping company delivers to the
driveway of the customer where a more real life scenario would be something
closer to Amazon having a distribution center in that city, and both
Comcast and FedEx are already both sitting idle in the parking lot. Amazon
pays FedEx to give the package to Comcast in the next parking space, who
then drives it to the customers house. Comcast says to Netflix, since I am
the one driving this from the parking lot to the customers house, why not
just pay me instead of paying FedEx more money to just put it on my truck?
Amazon says, but FedEx will deliver the package to France if I tell them
to. Comcast says, but you don't even serve france out of this distribution
center, and I am not asking to be charged for all packages, only the ones I
deliver instead of FedEx. Amazon says, you are right, we have technology to
give your packages directly to you and stuff going to France to Fedex, and
it would be best for both of us to do it, but unless you'll deliver my
packages for free I'm going to keep paying FedEx to just keep loading them
on your truck. Comcast says have at it, there are 5 trucks for FedEx to
load freely now but if you need more you have to compromise with us on a
deal that works better for both of us. Amazon says, when we are done with
you in the media we won't need to compromise.

>Government regulation of interconnects

I agree with Owen Delong on this who said "That set of regulations would be
utterly impossible to meaningfully enforce because so much of it depends on
subjective evaluation." and gave some reasons why.
I also think that in order for the government to meaningfully review all
interconnect relationships between companies there would be more legal
scrutiny within companies needing to justify the process, forms would need
to be filed with government for review, some reviewing entity would need to
perform a traffic study potentially needing access to netflow data, and
"tier 1" if it would continue to exist would likely require expensive
certification to come with the responsibility and there would need to be
some process for new companies seeking "tier 1" status to apply for it. We
would be left with a pile of costs, nothing beneficial gained from it, and
an advantage to companies with more lawyers than engineers.

>Broadband is too expensive in the US compared to other places

I have seen this repeated so many times that I assume it's true but I have
never seen anything objective as to why. I can tell you if you look at
population density by country the US is 182nd in the world and the average
broadband speed (based on OOKLA:
http://www.netindex.com/download/allcountries/) is 30th in the world. South
Korea that is well known for its fast broadband speeds has a density of
505/km vs the US at 32/km. We have about 1/15 of the population density and
about 1/2 the average broadband speed. Hong Kong, Singapore, Netherlands,
Japan, Macau etc. all have more than 10x the population density in the US
so definitely not all countries with fast broadband make for a fair
comparison and there are likely fewer that do. The UK is only beating the
US by 2Mbps but has a population density of 262/km.

So while its a fair assessment that broadband in the US is very bias to
ignore some of the other factors involved. Another mistake I see people
keep making is in comparing the cost of broadband in the US in $USD to
other countries around the world. The cost of broadband in Estonia is only
about $30/month. OMG, I can't believe broadband is cheaper in Estonia! What
people ignore is everything is cheaper in Estonia, the average household
income in Estonia is $14k vs $55k here. By that measure broadband is more
expensive for families there than it is in the US. This is another point
people repeat without bothering to qualify. This would be like my
grandfather comparing the costs of a candy bar from back when he was a kid
to today but ignoring inflation.

>Broadband companies are making money hand over fist

This may be true but I have honestly not attempted to index a bunch of
major companies and compare their profit vs revenue so see if broadband
companies are really on the top of the pile as people making this point
imply.  I have to confess to being skeptic that the people making the claim
have done this either.

>ad hominem attacks

Inevitable but no, I don't have financial gain in any of this. My stance is
essentially that if ISP's are forced to choose between higher prices,
metered billing, or adopting paid peering then paid peering is the best
solution of those 3 and pushing for legislation prohibiting it only serves
to take what I think is the best solution off the table. Especially in
cases where content providers are monetizing a service sold over the top I
think resistance to this option is a bit stubborn and I'd like to see the
industry solve the dispute without the government taking the opportunity to
land grab for expanded power over the Internet. If they pick just
ratcheting up pricing for unlimited plans in auto pilot as costs rise it
will only harm the "Broadband is too expensive in the US compared to other
places" numbers and I think people have been pretty clear in their
objection to metered billing. Metered billing would also probably hurt
content providers more than paid peering would so it's the worst option all
around. I read complaints about the way things are handled all the time and
complaining is easy but proposing better solutions is harder.

On Wed, May 14, 2014 at 4:11 AM, Matthew Petach <mpetach at netflight.com>wrote:

>
>
>
> On Sat, May 10, 2014 at 8:04 AM, Rick Astley <jnanog at gmail.com> wrote:
>
>> [...]
>>
>> The reality is an increasingly directly peered Internet doesn't sit well
>> if
>> you are in the business of being the middle man. Now if you will, why do
>> transit companies themselves charge content companies to deliver bits? How
>> is it fair to be in the business of charging companies to receive their
>> bits and hand them to a settlement free peer on the hook to deliver them,
>> but not fair for content to just bypass the transit company and enter a
>> paid peering agreement with the company delivering the bits? In this case
>> paid peering is mutually beneficial to both companies involved and is
>> typically cheaper for the content company than it would cost to send that
>> traffic over transit.
>>
>
> What you're missing is that the transit provider is
> selling full routes.  The access network is selling
> paid peering, which is a tiny fraction of the routes.
> If I pay transit provider X $10/mb (i know, not realistic,
> but it makes my math work) to reach the entire internet,
> it might seem reasonable to pay access network C $5/mb
> to hand traffic to them, and bypass the transit provider,
> avoiding potentially congested links.
>
> But then access network A decides they want to cut out
> the middleman as well--so they do the same thing, run
> their ports to transit provider X hot; to avoid that, I can
> pay the cheap price of $4/mb to reach them.
>
> Now access networks F and D want to do the same thing;
> their prices for their routes are $4 and $5/mb, respectively.
>
> Finally, little access provider T wants in at $2/mb for their
> routes.
>
> So, at the end of the week, I *had* been paying $10/mb to
> send traffic through transit to reach the whole rest of the
> internet.  Now, I'm paying $5+$4+$4+$5+$2, or $30, and
> I don't have a full set of routes, so I've still got to keep
> paying the transit provider as well at $10.  Depending on
> port counts, locations, and commit volumes, your "typically
> cheaper for the content company than it would cost to send
> that traffic over transit" has flown completely out the window.
> It could even end up being many times more expensive to
> handle the traffic that way.
>
> In order for the costs to work out, you'd really need
> to apply a formula along the lines of
> C(n) <= T(n) * C(t)
> where
> T(n) =fraction of traffic volume destined for access network X
> C(t)=cost of transit (ie, full routes, reachability to the entire internet)
> C(n)=cost of paid peering to access network X
>
> So, if you're an access network and want to charge
> for paid peering, and you represent 1/20th of my
> traffic, there's no reason for me to pay more than
> 1/20th of my transit cost for your routes; otherwise,
> it's more cost effective for me as a business to
> continue to pay a transit provider.
>
> I'm constantly amazed at how access networks
> think they can charge 2/3 the price of full transit
> for just their routes when they represent less than
> 1/10th of the overall traffic volume.  The math just
> doesn't work out.  It's nothing about being tier 1, or
> bigger than someone else; it's just math, pure and
> simple.
>
> Matt
> (currently not being paid by anyone for my time
> or thoughts, so take what I'm saying as purely
> my own thoughts on the matter, nothing more)
>
>
>