Possible explanations for a large hop in latency

Fri Jun 27 03:31:50 UTC 2008

Just google "tbr1.sl9mo.ip.att.net" and it's clear that high latency through
that point has occurred before.  And guess what kind of customer complained
to me about the latency?  A gamer.

Frank

-----Original Message-----
From: Frank Bulk - iNAME [mailto:frnkblk at iname.com] 
Sent: Thursday, June 26, 2008 9:27 PM
To: 'Robert Richardson'; John T. Yocum
Cc: nanog list
Subject: RE: Possible explanations for a large hop in latency

<snip>

Wondering why the RTT dropped to 24 msec for that hop, I entered both
69.28.226.192 and the IP address that my customer has been complaining about
(12.129.255.4) into PingPlotter and I see that those behave very
differently.  I'm now guessing that AT&T is routing back traffic sent to
12.129.255.4 in a different way (perhaps asymmetrically) than traffic sent
to 69.28.226.192, but it doesn't show up until it hits 12.122.112.22.
Perhaps it's all those 1's and 2'. ;)

I notice that in the low RTT trace router 12.88.71.13 goes to
tbr2.sl9mo.ip.att.net (12.122.112.78), but in the high RTT trace, roouter
12.88.71.13 goes to tbr1.sl9mo.ip.att.net (12.122.112.22).   Must be
something about the way AT&T gets to tbr1.sl9mo.ip.att.net (12.122.112.22).
I can't traceroute to either of those networks directly.  In fact, I don't
appear to be able to traceroute to any of the 12.122.x.x or 12.129.x.x I see
in my traceroutes, perhaps because AT&T uses some of that space internally
and doesn't advertise it.

Frank

From: Robert Richardson [mailto:bobrmr at gmail.com]
Sent: Thursday, June 26, 2008 8:20 PM
To: John T. Yocum
Cc: frnkblk at iname.com; nanog list
Subject: Re: Possible explanations for a large hop in latency

They probably don't propagate TTL w/in their MPLS core.  Depending on how
they have MPLS implemented, you may only see 2 hops on the network; the
ingress and egress routers.  If the ingress router was in NYC and the egress
in Seattle, you could understandably expect a large jump in RTT.

Not an ATT customer but do know other providers run their MPLS core's this
way...

-Robert

On Thu, Jun 26, 2008 at 6:09 PM, John T. Yocum <john at fluidhosting.com>
wrote:

The explanation I got, was that the latency seen at the first hop was
actually a reply from the last hop in the path across their MPLS network.
Hence, all the following hops had very similar latency.

Personally, I thought it was rather strange for them to do that. And, I've
never seen that occur on any other network.

Perhaps someone from ATT would like to chime in.

--John

Frank Bulk - iNAME wrote:

Did that satisfy you?  I guess with MPLS they could tag the traffic and send
it around the country twice and I wouldn't see it at L3.

Frank

-----Original Message-----
From: John T. Yocum [mailto:john at fluidhosting.com] Sent: Thursday, June 26,
2008 7:04 PM
To: frnkblk at iname.com
Cc: nanog list
Subject: Re: Possible explanations for a large hop in latency

When I asked ATT about the sudden latency jump I see in traceroutes,
they told me it was due to how their MPLS network is setup.

--John

Frank Bulk wrote:

Our upstream provider has a connection to AT&T (12.88.71.13
<http://12.88.71.13/> ) where I
relatively consistently measure with a RTT of 15 msec, but the next hop
(12.122.112.22 <http://12.122.112.22/> ) comes in with a RTT of 85 msec.
Unless AT&T is sending

that

traffic over a cable modem or to Europe and back, I can't see a reason why
there is a consistent ~70 msec jump in RTT.  Hops farther along the route
are just a few msec more each hop, so it doesn't appear that 12.122.112.22
<http://12.122.112.22/>
has some kind of ICMP rate-limiting.

Is this a real performance issue, or is there some logical explanation?

Frank