Extra latency at ATT exchange for UVerse

Thu Nov 11 22:11:47 UTC 2010

Here are the traceroutes (without the first 3 hops)

>From ADSL:
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets

 4  12.81.16.32  30.196 ms  32.292 ms  35.161 ms
 5  12.81.16.25  37.774 ms  40.627 ms  44.209 ms
 6  74.175.192.78  48.008 ms  50.841 ms  53.946 ms
 7  12.122.140.186  59.278 ms  61.510 ms  61.824 ms
 8  12.123.22.129  61.111 ms  59.803 ms  59.382 ms
 9  12.88.97.6  116.059 ms  115.757 ms  116.331 ms
10  72.14.233.54  59.856 ms  60.354 ms  61.088 ms
11  72.14.232.213  61.312 ms  78.592 ms 209.85.254.243  60.396 ms
12  209.85.253.137  105.800 ms  100.558 ms 209.85.253.141  96.095 ms
13  8.8.8.8  96.571 ms  98.721 ms  98.514 ms

>From UVerse:

 4  76.201.204.10  24.020 ms  24.321 ms  24.250 ms
 5  76.201.208.22  25.754 ms  25.701 ms  25.633 ms
 6  76.201.208.8  25.558 ms  25.230 ms *
 7  70.159.177.248  24.910 ms  22.452 ms  23.436 ms
 8  12.81.16.2  24.478 ms  24.420 ms  24.514 ms
 9  12.81.16.21  128.798 ms  127.685 ms  126.821 ms
10  74.175.192.90  22.999 ms  21.932 ms  23.057 ms
11  12.122.140.186  24.397 ms 12.122.141.186  24.647 ms  24.594 ms
12  12.123.22.5  32.763 ms 12.123.22.129  22.016 ms 12.123.22.5  26.850 ms
13  * * *
14  72.14.233.54  40.287 ms 72.14.233.56  40.716 ms  40.660 ms
15  209.85.254.241  41.964 ms  41.909 ms  41.842 ms
16  209.85.253.137  51.698 ms 209.85.253.133  44.534 ms 209.85.253.145
 39.621 ms
17  8.8.8.8  41.278 ms  42.124 ms  42.718 ms

Both the homes are in the same city.  The entry point to Google is the
same: 72.14.233.54 (from whois).

>From ADSL, latency to that google router is about 10ms:
rtt min/avg/max/mdev = 9.461/13.137/59.856/7.841 ms

from UVerse, it's about 40ms.
rtt min/avg/max/mdev = 38.923/44.503/70.535/7.162 ms

There isn't enough jitter to justify this difference. And it's not
just to Google. i tested to another server  (where ATT hands off to
Qwest), and it's the same. It can't be congestion/location, because if
it were, the ADSL gateway should see it too. Reverse path effects,
perhaps.

- Srikanth

On Thu, Nov 11, 2010 at 4:19 PM, Richard A Steenbergen <ras at e-gerbil.net> wrote:
> On Thu, Nov 11, 2010 at 03:39:42PM -0500, Srikanth Sundaresan wrote:
>> Can anyone explain why ATT's UVerse adds significant delay to packets
>> compared to their ADSL service?
>>
>> For example, pinging 8.8.8.8 from an ADSL gateway shows a latency of
>> ~10ms. From an UVerse gateway, it's about 40ms. Of the extra 30ms,
>> about 10ms can be explained by the fact that UVerse last hop is
>> interleaved. ADSL seems to have Fastpath enabled more often than not
>> (at least in my city).
>>
>> The extra 20ms is more interesting. By pinging each hop obtained by
>> tracerouting to 8.8.8.8, the extra latency seems to be added on the
>> exchange between ATT and Google. It's not just for 8.8.8.8. The same
>> holds for other hosts too. ATT seems to add 20ms when it hands off a
>> (UVerse) packet at an exchange.
>
> First off, this thread is useless without actual traceroutes. :)
>
> Whenever you see the latency change significantly at the boundry between
> networks, the two most obvious things to look for are congestion, and an
> asymmetric reverse path.
>
> Congestion is usually pretty easy to spot, if you're seeing it with high
> latency you'll usually find that latency to be pretty jittery (as tcp
> windows probe for more capacity, then back off), and you'll see the
> associated packet loss starting at the link in question.
>
> Asymmetric reverse paths are responsible for a lot of other issues too.
> Traceroute measures the round-trip latency but only shows you the path
> in a single direction, leaving the entire return trip completely
> invisible. There is no guarantee that the packet will come back to you
> the same way that you sent it, so what you may be seeing is the traffic
> returning via a different exit between networks. The best way to
> troubleshoot something like this is to get a copy of a traceroute in the
> opposite direction. For more information, see:
>
> http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf
>
> One other thing to keep in mind is that a company like Google may be
> more interested in keeping their servers located somewhere with ample
> (and cheap) space and power, than they are with ensuring close proximity
> to an Internet interconnection point. For example, Google is well known
> for building a datacenter in The Dalles Oregon, which is a significant
> distance away from ANY network interconnection. From Chicago, directly
> connected to Google, 8.8.8.8 is actually located an rtt of 12ms away:
>
>  1  core1-2-2-0.ord.net.google.com (206.223.119.21)  1.509 ms  1.769 ms  1.409 ms
>  2  72.14.236.176 (72.14.236.176)  1.677 ms  1.579 ms  1.878 ms
>  3  72.14.232.141 (72.14.232.141)  12.555 ms
>    209.85.241.22 (209.85.241.22)  12.150 ms  12.013 ms
>  4  209.85.241.37 (209.85.241.37)  11.974 ms
>    209.85.241.35 (209.85.241.35)  12.591 ms
>    209.85.241.37 (209.85.241.37)  12.125 ms
>  5  209.85.240.49 (209.85.240.49)  12.944 ms
>    72.14.239.189 (72.14.239.189)  21.509 ms
>    209.85.240.45 (209.85.240.45)  25.000 ms
>  6  google-public-dns-a.google.com (8.8.8.8)  12.890 ms  12.487 ms  12.770 ms
>
> This would put the fiber distance at around 500+ miles, i.e. this
> datacenter could actually be in Kansas City MO for all you know. Without
> the original traceroute to verify your assumptions about where the
> interconnection point between networks is, it's entirely possible that
> you could be seeing something like this too.
>
> --
> Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
> GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
>