Google's peering, GGC, and congestion management

Thu Oct 15 14:35:41 UTC 2015

On Oct 14, 2015, at 1:07 PM, Baptiste Jonglez <baptiste at bitsofnetworks.org> wrote:

> In its peering documentation [https://peering.google.com/about/traffic_management.html],
> Google claims that it can drive peering links at 100% utilisation:
> 
>> Congestion management
>> 
>> Peering ports with Google can be run at 100% capacity in the short term,
>> with low (<1-2%) packet loss. Please note that an ICMP ping may display
>> packet loss due to ICMP rate limiting on our platforms. Please contact
>> us to arrange a peering upgrade.
> 
> How do they achieve this?

The 100% number is silly. My guess? They’re at 98%.

That is easily do-able because all the traffic is coming from them. Coordinate the HTTPd on each of the servers to serve traffic at X bytes per second, ensure you have enough buffer in the switches for micro-bursts, check the NICs for silliness such as jitter, and so on. It is non-trivial, but definitely solvable.

Google is not the only company who can do this. Akamai has done it far longer. And Akamai has a much more difficult traffic mix, with -paying customers- to deal with.

> More generally, is there any published work on how Google serves content
> from its CDN, the Google Global Cache?  I'm especially interested in two
> aspects:
> 
> - for a given eyeball network, on which basis are the CDN nodes selected?

As for picking which GGC for each eyeball, that is called “mapping”. It varies among the different CDNs. Netflix drives it mostly from the client. That has some -major- advantages over other CDNs. Google has in the past (haven’t checked in over a year) done it by giving each user a different URL, although I think they use DNS now. Akamai uses mostly DNS, although they have at least experimented with other ways. Etc., etc.

> - is Google able to spread traffic over distinct peering links for the
>  same eyeball network, in case some of the peering links become
>  congested?  If so, how do they measure congestion?

Yes. Easily.

User 1 asks for Stream 1, Google sends them them to Node 1. Google notices Link 1 is near full. User 2 asks for Stream 2, Google sends them to Node 2, which uses Link 2.

This is possible for any set of Users, Streams, Nodes, and Links.

It is even possible to send User 2 to Node 2 when User 2 wants Stream 1. Or sending User 1 to Node 2 for their second request despite the fact they just got a stream from Node 1. There are few, if any, restrictions on the combinations.

Remember, they control the servers. All CDNs (that matter) can do this. They can re-direct users with different URLs, different DNS responses, 302s, etc., etc. It is not BGP.

Everything is much easier when you are one of the end points. (Or both, like with Netflix.) When you are just an ISP shuffling packets you neither send nor receive, things are both simpler and harder.

--
TTFN,
patrick

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 872 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20151015/d967b1b8/attachment.sig>