Extra latency at ATT exchange for UVerse

Fri Nov 12 01:56:21 UTC 2010

On Thu, Nov 11, 2010 at 05:11:47PM -0500, Srikanth Sundaresan wrote:
> Here are the traceroutes (without the first 3 hops)

(Note: NANOG is not really the right place to troubleshoot everyone's 
home connectivity, I'm mostly just posting this as an educational 
example of how to do inter-network troubleshooting... though in 
retrospect this may not be the worlds best example :P).

> >From ADSL:
> traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
> 
>  4  12.81.16.32  30.196 ms  32.292 ms  35.161 ms
>  5  12.81.16.25  37.774 ms  40.627 ms  44.209 ms
>  6  74.175.192.78  48.008 ms  50.841 ms  53.946 ms
>  7  12.122.140.186  59.278 ms  61.510 ms  61.824 ms
>  8  12.123.22.129  61.111 ms  59.803 ms  59.382 ms
>  9  12.88.97.6  116.059 ms  115.757 ms  116.331 ms
> 10  72.14.233.54  59.856 ms  60.354 ms  61.088 ms
> 11  72.14.232.213  61.312 ms  78.592 ms 209.85.254.243  60.396 ms
> 12  209.85.253.137  105.800 ms  100.558 ms 209.85.253.141  96.095 ms
> 13  8.8.8.8  96.571 ms  98.721 ms  98.514 ms
> 
> >From UVerse:
> 
>  4  76.201.204.10  24.020 ms  24.321 ms  24.250 ms
>  5  76.201.208.22  25.754 ms  25.701 ms  25.633 ms
>  6  76.201.208.8  25.558 ms  25.230 ms *
>  7  70.159.177.248  24.910 ms  22.452 ms  23.436 ms
>  8  12.81.16.2  24.478 ms  24.420 ms  24.514 ms
>  9  12.81.16.21  128.798 ms  127.685 ms  126.821 ms
> 10  74.175.192.90  22.999 ms  21.932 ms  23.057 ms
> 11  12.122.140.186  24.397 ms 12.122.141.186  24.647 ms  24.594 ms
> 12  12.123.22.5  32.763 ms 12.123.22.129  22.016 ms 12.123.22.5  26.850 ms
> 13  * * *
> 14  72.14.233.54  40.287 ms 72.14.233.56  40.716 ms  40.660 ms
> 15  209.85.254.241  41.964 ms  41.909 ms  41.842 ms
> 16  209.85.253.137  51.698 ms 209.85.253.133  44.534 ms 209.85.253.145
>  39.621 ms
> 17  8.8.8.8  41.278 ms  42.124 ms  42.718 ms
> 
> Both the homes are in the same city.  The entry point to Google is the
> same: 72.14.233.54 (from whois).

Actually the entry point to Google is probably the hop before that, 
12.88.97.6. In all likelihood this is the /30 between the two networks, 
where .5 is the AT&T side and .6 is the Google side. The IP space of the 
demarc point belongs to AT&T of course, but this is what you'd expect in 
a provider->customer relationship. :) In an ordinary network you would 
be able to confirm this with DNS and/or some traceroutes to the routers, 
but both AT&T and Google have intentionally obfuscated the hell out of 
their networks from the outside world (no dns, blocking traceroutes 
directly to router IPs, etc), so that won't help you much. There is also 
no Google looking glass (at least that I can find), nor do they support 
record-route, so you're probably SOL on the reverse path too.

> >From ADSL, latency to that google router is about 10ms:
> rtt min/avg/max/mdev = 9.461/13.137/59.856/7.841 ms
> 
> from UVerse, it's about 40ms.
> rtt min/avg/max/mdev = 38.923/44.503/70.535/7.162 ms
> 
> There isn't enough jitter to justify this difference. And it's not
> just to Google. i tested to another server  (where ATT hands off to
> Qwest), and it's the same. It can't be congestion/location, because if
> it were, the ADSL gateway should see it too. Reverse path effects,
> perhaps.

Well we can start by eliminating the possibility that the 8.8.8.8 node 
you're hitting is a significant distance away once you hit Google's 
network. What little bit of DNS AT&T does have working shows that this 
is coming out of Atlanta, which could also be confirmed with a few 
traceroutes from route-server.ip.att.net. From there, it's trivial to 
find a network with a looking glass and direct Google connectivity in 
Atlanta, and match up the exact same path:

 2  72.14.233.54 (72.14.233.54)  0.944 ms  0.902 ms
    72.14.233.56 (72.14.233.56)  0.720 ms
 3  209.85.254.241 (209.85.254.241)  1.005 ms
    209.85.254.243 (209.85.254.243)  16.214 ms 
    72.14.232.215 (72.14.232.215)  1.264 ms
 4  209.85.253.141 (209.85.253.141)  1.797 ms
    209.85.253.133 (209.85.253.133)  1.937 ms
    209.85.253.137 (209.85.253.137)  1.408 ms
 5  google-public-dns-a.google.com (8.8.8.8)  1.413 ms  1.539 ms  1.481 ms

Honestly I've got to question the measurement that you're taking above, 
since in your first (DSL) traceroute it looks like you're actually 
seeing higher latency than you are on the second (Uverse) path. Without 
being able to actually repeat the traceroute multiple times and verify 
that the reading was accurate it's obviously hard to say for certain, 
but your numbers look VERY consistent, showing a clear progression with 
very little jitter from ~30ms at the first visible hop, to ~60ms at the 
Google handoff. If there was really a measurement artifact, you would 
expect at least a healthy percentage of those numbers to be 
significantly different. 

As for the ~17ms jump between Uverse and Google in the second 
traceroute, I can't tell for certain without full IPs, but my gut says 
that the reverse path might be going back via Ashburn once it hits the 
Google side. Remember AT&T is actually composed of classic AT&T, 
SBC/AS7132, and Bellsouth/AS6389, each with their own unique routing 
policies. The latency jump would be a near perfect fit for there still 
being some direct AS7132 peering sessions up, but only in Ashburn and 
not Atlanta.

If nothing else, this illustrates one key point of troubleshooting with 
traceroute. The actual output of the traceroute is often worthless 
without knowing the source and destination IPs that were being tested, 
so *ALWAYS* provide those along with your traceroutes if you want to 
ever have any hope of having your problem solved. :)

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)