S.Korea broadband firm sues Netflix after traffic surge

Matthew Petach mpetach at netflight.com
Fri Oct 1 18:36:28 UTC 2021

On Fri, Oct 1, 2021 at 9:08 AM Laura Smith via NANOG <nanog at nanog.org>

> > The bad news now, is, there are plenty of many, small, local
> > and regional ISP's who are willing to do whatever it takes to
> > work with the content providers. All that's required is some
> > network, a half-decent data centre and an exchange point. Gone
> > are the days where customers clamored to sign up with Big
> > Telco.
> Speaking as one of those smaller ISPs willing to do whatever it takes,
> perhaps you could answer me this riddle.....
> - PoP in one of your "half-decent data centres" ... tick.
> - Connnection to one of your "exchange point" ... tick.
> - $certain_large_cdn present on said "exchange point" ... tick.
> And yet .....
> - $certain_large_cdn publishes routes on route server ? Nope.
> - $certain_large_cdn willing to establish direct peering session ? Nope.
> I am well aware of the "big boys club" that operates at most exchanges
> where the large networks see it beneath them to peer with (or publish
> routes for the benefit of) the unwashed masses.
> But I struggle to comprehend why $certain_large_cdn would effectively cut
> off their nose to spite their face ?
Having worked on building out a global content delivery network,
I can hopefully shed some light here.

Let's imagine a scenario where you have a CDN node attached to
a large peering location.

At that peering location, you have Nx100G direct peering links to
large eyeball networks, and Nx10G direct peering links to midsized
eyeball networks.

You have a historical connection into the peering switch as well,
which you've kept around and upgraded as time went by.  Life
is generally good.  You've got your own backbone between
your locations, and you capacity engineer your infrastructure
so that if your peering sessions go down at that site, users
are redirected to the next closest CDN node.

Then one day, Bad Juju(tm) happens.  The router that your
Nx100G links with Large Eyeball Network A terminate on
goes down, doesn't matter if it's on their end or your end,
those direct sessions go down.  But instead of your traffic
failing over to site B instead, it continues flowing--but now
it's going through the route server sessions, across the
now grossly-undersized connection into the public peering
switch, and *nobody* is happy about that.  You tried putting
import and export lists into the route server, but they don't
seem to be taking effect, and the phone is ringing off the
hook, so you simply stop exporting routes into the route
server, and suddenly traffic is now failing over to site B
for all the eyeball requests from Large Eyeball Network A,
and you get to explain to your boss how those sessions
with the route servers caused the failover plan to fail badly.

Decision is made that route server peering sessions are
too far removed from direct control to be safe, and the
only way to effectively prevent this scenario from happening
again is to stop announcing routes to the route servers.

You're a smaller network; you're caught up in this, because you'd
been getting routes from the route server, and now you're not.
You request direct peering; unfortunately, the ports on the existing
line cards on the CDN edge routers are full, and it's going to be
months before the next capacity upgrade is planned; and those
line cards are all 100G cards, with no 10G ports planned.  It's
decided that letting that traffic get handled by the transit ports is
cheaper than trying to get some additional 10G peering port capacity
at the site.  And so, your request for direct peering is denied; there's
just no place to plug you in on the router on the other side, unless
you can drum up enough traffic to justify a 100G peering link.

Establishing direct peering across the peering fabric becomes
the only possible point of commonality, then; but that requires
adding explicit neighbor configuration into the router, unlike
the route-server mediated prefix exchange; and now you're
still running into limitations on the edge routers--though in this
case it's not physical port limitations, it's neighbor adjacency
limits on the routers.  The more neighbors you configure on an
edge router, the longer the configs are, and depending on how
the neighbors are established, the more processing power and
memory it takes on the router.  At a certain point, there's just not
enough to go around, and the CDN makes the decision to not add
any new BGP neighbors at that site.  A few years down the line,
when hardware is added/upgraded, they may revisit that decision,
or it may persist, even after the upgrade, due simply to corporate
inertia--nobody goes back to revisit the decision, and see if it still
makes sense or not.

Sometimes it's just a case of limited humans; without good automation
tools in place, adding BGP adjacencies takes a ticket being created
in a request system, and then another more highly skilled human
going into the router to configure the neighbor.  If you've got a limited
number of skilled humans available (and every company has fewer
of those than they really need, but that's always going to be the case),
the decision generally gets made to focus their attention on efforts
that bring in more revenue--and configuring public peering sessions
generally doesn't fall into that bucket.  :(

And sometimes, financial considerations come into play; the CDN
network has a default upstream transit provider to catch the traffic
that isn't handled by direct peering, and the transit provider gives
a really nice price break if the traffic makes it up into the next tier;
so, CDN that's just *below* the cheaper tier looks for traffic it can
shunt onto transit, to nudge the volume over the line, to qualify for
the lower per-mbit price point--and the easiest traffic to use to nudge
them over is the smaller public peering neighbor traffic.  And so, your
request for direct peering is looked at, and your traffic volume is
deemed to be the perfect fit to help fill out the transit commitment
needed to bump them into the cheaper tier.

In a perfect world, these would all be solvable issues, and exchanging
traffic directly with the CDN across the public switches would make
sense for everybody involved.

Unfortunately, we live in an imperfect world, with route server filters
that don't react quickly and with the level of precision needed, we
have limitations on how many BGP adjacencies routers can handle,
we have limited port counts on line cards, we have limited numbers
of humans to make changes to router configurations, and we have
economic pressures to meet volume thresholds to get better pricing
that all end up creating the scenario you find yourself in; a completely
reasonable request for direct peering nonetheless is ignored, or
denied, due to one or more of the reasons I've discussed.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20211001/7c49b8eb/attachment.html>

More information about the NANOG mailing list