Traffic engineering and peering for CDNs

Mon Jun 6 14:36:19 UTC 2016

Hello,

> On Jun 6, 2016, at 7:36 AM, Graham Johnston <johnstong at westmancom.com> wrote:
> 
> Lately I have been putting in some effort to maximize our IX connections by trying to work with the top 5-ish list of ASNs that still send us traffic via a paid transit connection despite the fact that we are both present on the same IX(s). In one case I missed the fact that one ASN wasn't using the IXs route-servers, that's on me for not spotting that one.
> 
> Even with proper IX peering in place though it seems like some CDNs are better at using the IX connections than others.  ASN 15169 for instance does an excellent job sending more than 99.99% of traffic via the IX connection; thank you. While others only seem to manage to send 60 - 80% of traffic via the IX.  What I am not understanding about the respective CDN's network wherein they don't send traffic to me through a consistent path? Is the content coming from widely different places and rather than transport it across their own network from a remote site they would rather hot-potato it out a local transit connection?  Are their transit costs so low that they don't care about using an IX connection over transit unlike a small operator like me? Is this just a non-obvious issue wherein they maybe just can't originate enough of the traffic near the IX and therefore don't make use of the IX connection, again a hot-potato phenomenon?

Most CDN’s do not have a backbone.  Transit costs are not free, but as most traffic is served by local nodes from cache, the costs of transport between locations in many cases is higher than just sending via transit. In some cases, the CDN may not have good mapping and may not be certain which node is best to serve your customers. In other cases, not all content exists on all nodes, and they may redirect to serve from the nodes which have the content. Finally, there may be an outage or capacity limits from the closest location, and another location may be serving to make up the shortfall.

> 
> Secondly can someone explain to me why some CDNs want a gigabit or two of traffic to be exchanged between our respective networks before they would peer with me via a public IX? I totally get those kinds of thresholds before engaging in a private interconnect but I don't understand the reluctance with regard to a public IX, that they are already established at. Is it again just a simple case of bandwidth economics that operate at a different scale than I can comprehend?
> 

This sounds like a surprisingly high threshold, but to some extent it boils down like this — setting up sessions requires some time. In the ideal world, the peer is intelligent and has everything set up properly, but even in this case, it still requires some time for making sure things go up properly. Some (but not all) CDN’s have it automated to reduce this time. Some potential peering networks are poorly run, and will leak routes, not announce all of their routes, will not configure the sessions properly, etc — this adds up to significantly more time. Before the CDN starts setting up peering with another network, it is not necessarily obvious if the potential peer is run by competent people or not. Many CDN’s are members of the route servers.  If you are exchanging a small amount of traffic, and both you and the CDN are on the Route Server for the IX, there maybe no reason to set up direct sessions which will require both more coordination time for configuration, and more router cpu time/ram on an ongoing basis. From the perspective of the CDN, most likely, 1Gbps or less is a perfectly reasonable amount of traffic to exchange to peers who are learned only via the route server, and not directly.

> I'm hoping the community can shed some light on this for me as I'm trying to avoid grilling the operators that are working with me as I don't expect those front line individuals to necessarily have a full view of the factors at play.
> 
> Thanks,
> Graham Johnston
> Network Planner
> Westman Communications Group
> 204.717.2829
> johnstong at westmancom.com<mailto:johnstong at westmancom.com>
> P think green; don't print this email.
> 

Best Regards,
-Phil Rosenthal
ISPrime