AW: Facebook more specific via Level3 ?

Tue Mar 21 19:38:02 UTC 2017

Hi Doug,

looks like this is also affecting other prefixes:

157.240.3.0/24     *[BGP/170] 18w5d 16:50:37, MED 0, localpref 150
                      AS path: 3356 32934 I, validation-state: unverified
                    > to 80.239.128.178 via ae9.0

I understand that FB is using some type of DNS geo-loadbalancing and other
mechanism to redirect users to (possibly) nearer mirrors. The used DNS is
directly requesting the root DNS and not any other public DNS (e.g. not
8.8.8.8). Running some requests within 3 minutes gives me the below results:

www.facebook.com => star-mini.c10r.facebook.com. => 31.13.77.36
www.facebook.com => star-mini.c10r.facebook.com. => 157.240.2.35
www.facebook.com => star-mini.c10r.facebook.com. => 31.13.93.36
www.facebook.com => star-mini.c10r.facebook.com. => 31.13.76.68

Trace to 31.13.77.36:
                                                      Packets
Pings
 Host                                               Loss%   Snt   Last   Avg
Best  Wrst StDev
[...]
 6. win-b4-link.telia.net                            0.0%     3   21.5  27.5
21.5  35.7   7.3
 7. level-ic-1573273-wien-b4.c.telia.net             0.0%     3   21.7  20.8
20.0  21.7   0.8
 8. ae-1-9.edge2.sanjose3.level3.net                 0.0%     3  177.1 179.8
177.1 182.6   2.8
 9. 4.53.210.78                                      0.0%     3  178.3 178.8
178.3 179.2   0.5
10. po131.asw04.sjc1.tfbnw.net                       0.0%     3  184.6 180.0
177.6 184.6   4.0
11. po241.psw02.sjc2.tfbnw.net                       0.0%     3  177.8 177.6
177.3 177.8   0.3
12. 173.252.67.79                                    0.0%     3  176.3 177.4
176.3 177.9   0.9
13. edge-star-mini-shv-01-sjc2.facebook.com          0.0%     3  177.6 177.4
177.1 177.6   0.3

Trace to 157.240.2.35:
                                                      Packets
Pings
 Host                                               Loss%   Snt   Last   Avg
Best  Wrst StDev
[...]
 6. win-b4-link.telia.net                            0.0%     3   20.9  25.8
20.8  35.8   8.6
 7. level-ic-1573273-wien-b4.c.telia.net             0.0%     3   20.5  21.4
20.2  23.4   1.8
 8. ???
 9. 4.15.85.102                                      0.0%     3  142.9 138.5
135.8 142.9   3.8
10. po121.asw01.ord3.tfbnw.net                       0.0%     3  135.5 135.7
135.5 136.0   0.4
11. po211.psw01c.ort2.tfbnw.net                      0.0%     2  136.1 135.7
135.3 136.1   0.6
12. 173.252.67.127                                   0.0%     2  151.6 147.6
143.6 151.6   5.7
13. edge-star-mini-shv-01-ort2.facebook.com          0.0%     2  136.0 135.9
135.8 136.0   0.2

Trace to 31.13.93.36 => this one is going via the FB DE-CIX peering and
latency of 34ms is perfect:
                                                      Packets
Pings
 Host                                               Loss%   Snt   Last   Avg
Best  Wrst StDev
 1. router                                           0.0%     5    0.5   0.5
0.5   0.6   0.0
 2. 192.168.8.3                                      0.0%     4    0.7   0.7
0.7   0.8   0.1
 3. 37.252.236.88                                    0.0%     4   37.8  38.0
31.8  46.1   6.0
 4. er-03.00-09-23.anx04.vie.at.anexia-it.com        0.0%     4   25.3  25.2
23.1  29.2   2.9
 5. cr-02.0v-08-72.anx03.vie.at.anexia-it.com        0.0%     4   21.3  30.4
20.6  40.1  10.9
 6. cr-01.0v-08-73.anx25.fra.de.anexia-it.com        0.0%     4   45.7  39.1
33.1  45.7   5.3
 7. ae1.br02.fra1.tfbnw.net                          0.0%     4   33.0  34.6
32.1  40.3   3.8
 8. po114.asw01.fra2.tfbnw.net                       0.0%     4   33.6  35.6
33.3  41.7   4.0
 9. po211.psw01d.fra3.tfbnw.net                      0.0%     4   40.8  36.6
32.9  40.8   3.4
10. 173.252.67.171                                   0.0%     4   41.2  35.6
33.3  41.2   3.7
11. edge-star-mini-shv-01-fra3.facebook.com          0.0%     4   34.6  33.9
33.2  34.6   0.6

Trace to 31.13.76.68:
                                                      Packets
Pings
 Host                                               Loss%   Snt   Last   Avg
Best  Wrst StDev
[...]
 6. win-b4-link.telia.net                            0.0%    14   28.9  24.7
20.3  29.3   3.5
 7. level-ic-1573273-wien-b4.c.telia.net             0.0%    14   22.8  22.5
20.0  30.9   3.7
 8. ae-2-3613.edge1.seattle3.level3.net              0.0%    13  195.2 189.2
186.0 195.2   3.5
 9. 4.59.232.46                                      0.0%    13  187.9 188.2
186.7 196.1   2.5
10. po101.psw03.sea1.tfbnw.net                       0.0%    13  194.7 188.3
186.5 194.7   2.0
11. 173.252.67.125                                   0.0%    13  188.0 188.8
186.9 195.5   2.2
12. edge-star-mini-shv-01-sea1.facebook.com          0.0%    13  186.6 189.0
186.2 196.0   3.1

Best regards
Jürgen

-----Ursprüngliche Nachricht-----
Von: Doug Porter [mailto:dsp at fb.com] 
Gesendet: Dienstag, 21. März 2017 18:41
An: Jürgen Jaritsch <juergen at jaritsch.at>; nanog at nanog.org
Betreff: Re: Facebook more specific via Level3 ?

> 31.13.70.0/24
>
> This more specific is only visible via AS3356 ... Facebook isn’t even 
> announcing it via direct peering.

This specific, and many others, are only announced to peers in the metro
they originate in.  To receive this prefix directly you'll need to peer with
us in Los Angeles.

It appears you're in Austria though, which means there's likely no use in
you peering with us in LA.  We globally load balance people to the best
point of presence.  You shouldn't see much, if any, traffic to or from the
above prefix.

> Any Facebook admin on or off list available to get this debugged and 
> tracked down?

Please reach out to noc at fb.com in the future.

Regards,
--
dsp