v6 cdn problems

Sun Nov 9 21:53:42 UTC 2014

On 11/8/14 1:02 PM, Frank Bulk wrote:
> The Google angle is also being discussed on outages.  Initial suspicions are PTB packets not flowing through tunneled connections.

you can also have problems in the other direction e.g. if your tunnel
ingress sends a ptb towards a load balanced service it may not arrive.

https://tools.ietf.org/html/draft-v6ops-pmtud-ecmp-problem-00

if you're tunneled it does help a lot if your mss reflects the cost of
the tunnel you know exists.

> Frank
> 
> -----Original Message-----
> From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Pete Carah
> Sent: Saturday, November 08, 2014 4:56 PM
> To: nanog at nanog.org
> Subject: v6 cdn problems
> 
> Prefix this - I'm on fios in the Baltimore area, using a HE tunnel
> terminating in ashburn.
> (*still* no native v6 on fios :-(  Speedtest shows little or no
> congestion, and didn't change significantly when I reduced mtu by 8. 
> (interestingly, speedtest.net usually reads faster than verizon's
> internal speedtest, and rarely averages less than my billed speed.)
> 
> I've recently had problems (started a few weeks ago with www.att.com,
> 4-5 days ago with *.google.com)
> which unfortunately happy eyeballs doesn't help.
> att.com uses akamai, google uses their own cdn (per dns; I don't know if
> there are any connections
> behind the scenes.)  This occurs on several google sites, all of which
> resolve to the same netblocks from here (maps.google.com,
> www.google.com, maps.gstatic.com, and at least one of the ad servers).
> 
> Symptom with akamai is that it connects immediately then data transfer
> times out.
> With google, symptom involves both slow connection, and data transfer
> timing out.  I don't know if chrome's happy eyeballs is working since if
> it was, and absent address caching, I shouldn't see the slow connection.
> 
> v6 connections to my hosts in Los Angeles (not on HE address space, but
> we do peer with them on
> any2) work fine transferring graphics and large database files both
> ways, so I'm pretty sure I don't have an mtu problem nor some other fios
> or HE problem.  Just to be sure, I dropped the 1500 to 1492 on the fios
> link and did the same adjustment to the mtu on my tunnel (becomes
> 1472).  No change on my hosts.  att.com appears a little better, though
> still very slow.  Google shows no change at all.
> 
> I saw some of the same problem yesterday from Frederick on comcast (only
> to google, didn't look at att), but couldn't take the time to do
> traceroutes.  If desired, I'm likely to go out there tomorrow and can do
> that too.  (we use a freebsd+pf router there).
> 
> Is this a provisioning problem where v6 eyeballs are outstripping cdn
> provisioning (since win7 and 8 both default to v6)?  Or is something
> else going on.  Since this seems to affect more than one cdn, it seems odd.
> 
> I run my own resolver locally instead of using verizon's.  (and my own
> (vyatta) router instead of theirs.  Actually I'm still using theirs as a
> bridge to talk to the set-top box (I don't know if Motorola still makes the
> coax terminal that would bridge it directly...)
> 
> Resolve and traceroutes of relevant sites:
> 
> [pete at port5 ~]$ host www.att.com
> www.att.com is an alias for prod-www.zr-att.com.akadns.net.
> prod-www.zr-att.com.akadns.net is an alias for www.att.com.edgekey.net.
> www.att.com.edgekey.net is an alias for e2318.dscb.akamaiedge.net.
> e2318.dscb.akamaiedge.net has address 23.76.217.145
> e2318.dscb.akamaiedge.net has IPv6 address 2600:807:320:202:9200::90e
> e2318.dscb.akamaiedge.net has IPv6 address 2600:807:320:202:8600::90e
> 
> Traceroute (v4) to this shows it is in Newark or environs:
> [pete at port5 ~]$ traceroute e2318.dscb.akamaiedge.net
> traceroute to e2318.dscb.akamaiedge.net (23.76.217.145), 30 hops max, 60 byte packets
>  1  rtr.east.altadena.net (192.168.170.1)  2.008 ms  2.450 ms  3.091 ms
>  2  L300.BLTMMD-VFTTP-64.verizon-gni.net (108.3.150.1)  9.021 ms  9.054 ms  9.045 ms
>  3  G0-7-4-5.BLTMMD-LCR-21.verizon-gni.net (100.41.195.94)  10.670 ms  10.683 ms  10.677 ms
>  4  ae4-0.RES-BB-RTR2.verizon-gni.net (130.81.209.230)  9.002 ms ae20-0.RES-BB-RTR1.verizon-gni.net (130.81.151.112)  8.995 ms so-1-1-0-0.RES-BB-RTR1.verizon-gni.net (130.81.199.2)  8.953 ms
>  5  * * *
>  6  * * *
>  7  0.xe-5-0-4.XL3.EWR6.ALTER.NET (140.222.225.73)  51.202 ms  41.102 ms  40.345 ms
>  8  0.ae1.XL4.EWR6.ALTER.NET (140.222.228.41)  33.065 ms TenGigE0-6-0-3.GW8.EWR6.ALTER.NET (152.63.19.158)  11.550 ms TenGigE0-6-0-6.GW8.EWR6.ALTER.NET (152.63.25.10)  11.659 ms
>  9  TenGigE0-7-0-1.GW8.EWR6.ALTER.NET (152.63.19.166)  19.854 ms akamai-gw.customer.alter.net (152.179.185.126)  1766.402 ms TenGigE0-7-0-7.GW8.EWR6.ALTER.NET (152.63.25.30)  18.227 ms
> 10  akamai-gw.customer.alter.net (152.179.185.126)  1747.269 ms a23-76-217-145.deploy.static.akamaitechnologies.com (23.76.217.145)  10.672 ms  11.263 ms
> 
> Traceroute6 to it appears to be local (but is hard to tell.  Next-to-last hop looks like either a router or 
> load-balancer is overloaded.  Same for the v4 traceroute...
> 
> [pete at port5 ~]$ traceroute6 www.att.com
> traceroute to www.att.com (2600:807:320:202:9200::90e), 30 hops max, 80 byte packets
>  1  rtr.east.altadena.net (2001:470:e160:1::1)  5.182 ms  5.274 ms  5.254 ms
>  2  altadenamd-1.tunnel.tserv13.ash1.ipv6.he.net (2001:470:7:126::1)  11.452 ms  15.040 ms  18.592 ms
>  3  ge4-12.core1.ash1.he.net (2001:470:0:90::1)  20.273 ms  20.574 ms  20.567 ms
>  4  eqx.br6.iad8.verizonbusiness.com (2001:504:0:2::701:1)  20.522 ms  20.232 ms  20.475 ms
>  5  * * *
>  6  2600:802:44f::2 (2600:802:44f::2)  1283.113 ms  1296.115 ms  1296.181 ms
>  7  2600:807:320:200::1743:f397 (2600:807:320:200::1743:f397)  20.181 ms  16.169 ms  14.073 ms
> 
> 
> [pete at port5 ~]$ host www.google.com
> www.google.com has address 74.125.228.16
> www.google.com has address 74.125.228.20
> www.google.com has address 74.125.228.17
> www.google.com has address 74.125.228.19
> www.google.com has address 74.125.228.18
> www.google.com has IPv6 address 2607:f8b0:4004:800::1012
> 
> Traceroute (v4) to this shows something odd, but I don't know where "burl" is for verizon. Also I appear to
> hit two nodes for the terminal. At least one google node appears to be ashburn (or environs)
> :
> [pete at port5 ~]$ traceroute www.google.com
> traceroute to www.google.com (74.125.228.20), 30 hops max, 60 byte packets
>  1  rtr.east.altadena.net (192.168.170.1)  2.646 ms  2.816 ms  3.536 ms
>  2  L300.BLTMMD-VFTTP-64.verizon-gni.net (108.3.150.1)  4.109 ms  4.194 ms  4.186 ms
>  3  G0-7-4-4.BLTMMD-LCR-22.verizon-gni.net (130.81.170.84)  7.928 ms  8.096 ms  8.088 ms
>  4  ae20-0.PHIL-BB-RTR1.verizon-gni.net (130.81.151.120)  10.881 ms ae20-0.PHIL-BB-RTR2.verizon-gni.net (130.81.151.124)  11.074 ms so-6-1-0-0.PHIL-BB-RTR2.verizon-gni.net (130.81.199.4)  11.047 ms
>  5  0.xe-7-0-2.XL2.IAD8.ALTER.NET (152.63.4.93)  14.872 ms 0.xe-3-0-1.XL3.IAD8.ALTER.NET (152.63.3.61)  37.703 ms  12.268 ms
>  6  0.xe-9-2-0.GW9.IAD8.ALTER.NET (152.63.36.30)  12.866 ms 0.xe-11-2-1.GW9.IAD8.ALTER.NET (152.63.42.2)  14.442 ms 0.xe-9-2-0.GW9.IAD8.ALTER.NET (152.63.36.30)  11.918 ms
>  7  * 0.xe-10-1-1.GW9.IAD8.ALTER.NET (152.63.35.113)  16.901 ms pool-96-236-104-66.burl.east.verizon.net (96.236.104.66)  136.110 ms
>  8  pool-96-236-104-66.burl.east.verizon.net (96.236.104.66)  137.977 ms 216.239.46.248 (216.239.46.248)  13.875 ms pool-96-236-104-66.burl.east.verizon.net (96.236.104.66)  134.602 ms
>  9  216.239.46.248 (216.239.46.248)  15.918 ms  10.708 ms  10.162 ms
> 10  72.14.238.173 (72.14.238.173)  11.347 ms  12.111 ms iad23s05-in-f20.1e100.net (74.125.228.20)  12.769 ms
> 
> Corresponding traceroute6 (shows lack of reverse on most hits...):
> [pete at port5 ~]$ traceroute6 www.google.com
> traceroute to www.google.com (2607:f8b0:4004:800::1011), 30 hops max, 80 byte packets
>  1  rtr.east.altadena.net (2001:470:e160:1::1)  1.640 ms  1.811 ms  1.801 ms
>  2  altadenamd-1.tunnel.tserv13.ash1.ipv6.he.net (2001:470:7:126::1)  11.977 ms  15.279 ms  19.265 ms
>  3  ge4-12.core1.ash1.he.net (2001:470:0:90::1)  19.779 ms  20.776 ms  22.303 ms
>  4  2001:4860:1:1:0:1b1b:0:d (2001:4860:1:1:0:1b1b:0:d)  22.267 ms  22.514 ms  22.507 ms
>  5  2001:4860::1:0:9ff (2001:4860::1:0:9ff)  22.508 ms  22.471 ms  22.455 ms
>  6  2001:4860:0:1::551 (2001:4860:0:1::551)  22.467 ms  19.139 ms  19.116 ms
>  7  2607:f8b0:8000:18::c (2607:f8b0:8000:18::c)  19.054 ms 2607:f8b0:8000:18::f (2607:f8b0:8000:18::f)  7.716 ms 2607:f8b0:4004:800::1b (2607:f8b0:4004:800::1b)  8.379 ms
> 
> Again shows multiple terminals for the given address.
> 
> 
> Ping works fine to all of the addresses, both v4 and v6, and the att one
> always connects immediately.  The google one doesn't always.
> 
> When I disable the HE tunnel, (and restart the browser - apparently
> happy-eyeballs caches), everything works just fine, so the problem does
> appear to relate to v6.
> 
> For reference, I mostly use chrome in linux.  My daughter sees the same
> problem with google, mostly using chrome in win 7.  I see the problem
> with firefox (in linux) also (to both sites).
> 
> -- Pete
> 
> 
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 243 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20141109/ddf9d2bc/attachment.sig>