Possible explanations for a large hop in latency

Frank Bulk - iNAME frnkblk at iname.com
Fri Jun 27 02:27:29 UTC 2008

Interestingly enough, when I trace from my Cisco router it seems to show
some MPLS labels after the hop of interest ( to,
only 24 msec here!).  I'm not sure how our Cisco box derives these from a
foreign network.




Type escape sequence to abort.

Tracing the route to


  1 sxct.sxcy.mtcnet.net ( 0 msec 0 msec 0 msec

  2 siouxcenter.sxcy.137.netins.net ( 4 msec 4 msec 4 msec

  3 ins-b12-et-4-0-112.desm.netins.net ( 8 msec 8 msec 8 msec

  4 ins-h2-et-1-10-127.desm.netins.net ( 8 msec 8 msec 8 msec

  5 ins-c2-et-pc2-0.desm.netins.net ( 8 msec 8 msec 8 msec

  6 28 msec 24 msec 28 msec

  7 tbr2.sl9mo.ip.att.net ( [MPLS: Label 30663 Exp 0] 52 msec
48 msec 52 msec

  8 cr2.sl9mo.ip.att.net ( [MPLS: Label 17306 Exp 0] 52 msec 52
msec 52 msec

  9 cr2.cgcil.ip.att.net ( [MPLS: Label 16558 Exp 0] 52 msec 52
msec 52 msec

 10 cr1.cgcil.ip.att.net ( [MPLS: Label 17002 Exp 0] 48 msec 52
msec 52 msec

 11 cr1.n54ny.ip.att.net ( [MPLS: Label 17033 Exp 0] 52 msec 52
msec 48 msec

 12 tbr1.n54ny.ip.att.net ( [MPLS: Label 32364 Exp 0] 52 msec
52 msec 52 msec

 13 48 msec 48 msec 52 msec

 14 60 msec 60 msec 64 msec

 15 oc48-po2-0.tor-151f7-cor-2.peer1.net ( 52 msec 52 msec
68 msec

 16 oc48-po7-0.tor-151f-dis-1.peer1.net ( 52 msec 52 msec 48

 17 tor-fe3-5a.ne.peer1.net ( 52 msec 52 msec *



Wondering why the RTT dropped to 24 msec for that hop, I entered both and the IP address that my customer has been complaining about
( into PingPlotter and I see that those behave very
differently.  I'm now guessing that AT&T is routing back traffic sent to in a different way (perhaps asymmetrically) than traffic sent
to, but it doesn't show up until it hits
Perhaps it's all those 1's and 2'. ;)


I notice that in the low RTT trace router goes to
tbr2.sl9mo.ip.att.net (, but in the high RTT trace, roouter goes to tbr1.sl9mo.ip.att.net (   Must be
something about the way AT&T gets to tbr1.sl9mo.ip.att.net (
I can't traceroute to either of those networks directly.  In fact, I don't
appear to be able to traceroute to any of the 12.122.x.x or 12.129.x.x I see
in my traceroutes, perhaps because AT&T uses some of that space internally
and doesn't advertise it.




From: Robert Richardson [mailto:bobrmr at gmail.com] 
Sent: Thursday, June 26, 2008 8:20 PM
To: John T. Yocum
Cc: frnkblk at iname.com; nanog list
Subject: Re: Possible explanations for a large hop in latency


They probably don't propagate TTL w/in their MPLS core.  Depending on how
they have MPLS implemented, you may only see 2 hops on the network; the
ingress and egress routers.  If the ingress router was in NYC and the egress
in Seattle, you could understandably expect a large jump in RTT.


Not an ATT customer but do know other providers run their MPLS core's this



On Thu, Jun 26, 2008 at 6:09 PM, John T. Yocum <john at fluidhosting.com>

The explanation I got, was that the latency seen at the first hop was
actually a reply from the last hop in the path across their MPLS network.
Hence, all the following hops had very similar latency.

Personally, I thought it was rather strange for them to do that. And, I've
never seen that occur on any other network.

Perhaps someone from ATT would like to chime in.


Frank Bulk - iNAME wrote:

Did that satisfy you?  I guess with MPLS they could tag the traffic and send
it around the country twice and I wouldn't see it at L3.


-----Original Message-----
From: John T. Yocum [mailto:john at fluidhosting.com] Sent: Thursday, June 26,
2008 7:04 PM
To: frnkblk at iname.com
Cc: nanog list
Subject: Re: Possible explanations for a large hop in latency

When I asked ATT about the sudden latency jump I see in traceroutes,
they told me it was due to how their MPLS network is setup.


Frank Bulk wrote:

Our upstream provider has a connection to AT&T (
<> ) where I
relatively consistently measure with a RTT of 15 msec, but the next hop
( <> ) comes in with a RTT of 85 msec.
Unless AT&T is sending


traffic over a cable modem or to Europe and back, I can't see a reason why
there is a consistent ~70 msec jump in RTT.  Hops farther along the route
are just a few msec more each hop, so it doesn't appear that
has some kind of ICMP rate-limiting.

Is this a real performance issue, or is there some logical explanation?





More information about the NANOG mailing list