Verizon Public Policy on Netflix

Sun Jul 13 00:22:24 UTC 2014

This is Brett Glass; I have been alerted to some of the responses to my
message (which was cross-posted by a third party) and have temporarily
joined the list to chime in. The following is my response to his message,
edited slightly to include some new information.

Dave Temkin wrote:

>First and foremost, we built our CDN, Open Connect, 

"Open Connect" is not, in fact, a CDN. Nor is it "peering." It is merely a set
of policies for direct connection to ISPs, and for placing servers in ISPs' 
facilities, that is as favorable as possible in every way to Netflix. It costs 
Netflix as little as possible and the ISP as much as possible.

>with the intention to
>deploy it as widely as possible in order to save ISPs who are delivering
>our traffic money 

It does not, in fact, appear to save ISPs money. Note that Comcast
asked for, and was given, additional payments even after it did all of
the things that are part of Netflix' "Open Connect" program. Netflix,
exercising inappropriate market power, has not offered smaller ISPs such
as my own the same amount per customer. In fact, it has offered us no
money at all -- even though our costs per Netflix customer are higher. 
Netflix thus discriminates against and threatens smaller ISPs, and by 
doing so, harms broadband competition.

>and improve our mutual customer experience. This goes for
>ISPs large and small, domestic and international, big endian and little
>endian. We've never demanded payment from an ISP nor have we ever charged
>for an Open Connect Appliance.

The power and bandwidth consumed by an "Open Connect Appliance" (which is
really just a hosted Netflix server) are a substantial expense for any ISP. 
Especially because the server is not a cache; it is stocked with content 
whether it is used or not and therefore wastes bandwidth on content and
formats that will never be used even once.

>When we first launched almost three years ago, we set a lower boundary for
>receiving a Netflix Open Connect Appliance (which are always free) at
>5Gbps. Since then we've softened that limit to 3.5Gbps due to efficiencies
>of how we pre-load our appliances (more on that below).

Most small ISPs (the average in the US, in fact) have 1,000 to 2,000 accounts.
If every one of those streams at 1 Mbps at the same time, which is highly
unlikely, this still does not reach 3.5 Gbps. Therefore, most ISPs are excluded
simply by this requirement if not by others (such as the requirement that the
ISP alone pay for a dedicated connection to one of Netflix' relatively few 
"peering points").

>We explicitly call our "cache" an Appliance because it's not a demand
>driven transparent or flow-through cache like the Akamai or Google caches.
>We do this because we know what's going to be popular the next day or even
>week and push a manifest to the Appliance to tell it what to download
>(usually in the middle of the night, but this is configurable by the ISP).

What Netflix does not say here is that (a) it can only somewhat predict what
will be in demand or go "viral;" (b) it wastes bandwidth by sending multiple
copies of each video to its server in different formats, rather than
transcoding locally or saving bandwidth on lesser used formats via caching; 
and (c) its server consumes large amounts of energy and bandwidth. A cache 
can be much more efficient and can be owned and managed by the ISP.

>The benefit of this architecture is that a single Appliance can get 70+%
>offload on a network, and three appliances clustered together can get 90+%
>offload, while consuming approximately 500 watts of power, using 4U of rack
>space, and serving 14Gbps per appliance. 

To put this in perspective: LARIAT builds its own caches which consume
as little as 20 watts and can saturate a 10 Gbps Ethernet port. The 
Netflix servers are large, bloated power hogs compared to a well designed 
cache.

>The downside of this architecture
>is that it requires significant bandwidth to fill; in some ISPs cases
>significantly more than they consume at peak viewing time. This is why our
>solution may not work well for some small ISPs and we instead suggest
>peering, which has 100% offload.

Because it requires expensive bandwidth that's dedicated solely to Netflix,
"peering" (as Netflix calls it; it's really just a dedicated link) has 0%,
not 100%, offload. The ISP is paying for all of the bandwidth, and it
cannot be used for anything else.

>We've put a lot of effort into localizing our peering infrastructure
>worldwide. As you can see from this map (sorry for the image), we're in 49
>locations around the world with the significant bulk of them in the US
>(blue pins = 1 location, red pins = >1 location in a metro) - more detailed
>version at <http://goo.gl/eDHpHU>http://goo.gl/eDHpHU and in our PeeringDB record (
><http://as2906.peeringdb.com>http://as2906.peeringdb.com) :

Our ISP connects to the Internet in Cheyenne, Wyoming (a major Internet
"crossroads;" it's where I-80 meets I-25) and Denver, Colorado (which is,
if anyplace can make the claim, the "center" of the entire Internet).

Netflix' small and relatively sparse network of "peering" points does not
include either of those locations. (I've noted, just today, that they have
only recently listed a presence at a private data center outside of Denver
in Englewood, Colorado -- not one of the major Denver exchanges, such as
1850 Pearl, to which we're already connected at great expense.) To get to it, 
we would have to spend several thousand dollars per month on an expensive 
connection, with no reimbursement of this expense from Netflix.

If Netflix were a good citizen, it would (a) let ISPs cache content; (b) pay them 
equitably for direct connections (smaller and more remote ISPs have higher costs 
per customer and should get MORE per account than Comcast, rather than receiving
nothing); and (c) work with ISPs to develop updated technology that makes streaming
more efficient. Bandwidth is expensive, and unicast streaming without caching is by 
far the most inefficient conceivable way of delivering "fat" content to the consumer.

--Brett Glass