impossible circuit

Jon Lewis jlewis at lewis.org
Sun Aug 17 06:08:09 UTC 2008


On Tue, 12 Aug 2008, Jon Lewis wrote:

>> What would happen if you pinged the Ocala router such that the TTL was 1 
>> when travelling over the DS3? From your traceroute it seems it travelled 
>> two IP hops that did not send ICMP error messages, but it might just be 
>> that the ICMP errors from the Ocala router are arriving first.
>
> Based on where the dupes are coming from, I assume pinging across the DS3 
> with TTL tuned to expire at the Ocala side would result in TTL exceeded 
> messages from both Ocala and the Sprint router where the packets are injected 
> into Sprint's network.  It doesn't look as if IOS gives the option to set TTL 
> on ping...so I'd try this from a Linux machine in our data center.

I just went ahead and "re-broke" the circuit for a bit by turning it back 
to hdlc to see if the issue is still there and to run some additional 
tests.  Someone is still cross connecting our Orlando->Ocala traffic over 
to Sprint.

I did your suggested ping with short TTL and the result was close to what 
I expected.

$ traceroute ocalflxa-br-1
traceroute to ocalflxa-br-1.atlantic.net (209.208.6.229), 30 hops max, 38 
byte packets
  1  209.208.25.165 (209.208.25.165)  0.539 ms  0.426 ms  0.388 ms
  2  69.28.72.162 (69.28.72.162)  0.246 ms  0.351 ms  0.223 ms
  3  andc-br-3-f2-0 (209.208.9.138)  0.559 ms  0.435 ms  0.471 ms
  4  ocalflxa-br-1-s1-0 (209.208.112.98)  2.735 ms *  2.656 ms

So, I need a TTL of 4 to get there from this machine.

$ ping -t4 ocalflxa-br-1
PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.68 ms
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=2.72 ms
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252 time=2.88 ms

Decrease ttl by one, and I get the expected ttl exceeded from the Orlando 
side of the circuit.

$ ping -t 3 ocalflxa-br-1
PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
>From andc-br-3-f2-0.atlantic.net (209.208.9.138) icmp_seq=0 Time to live 
exceeded

Now, here's a mild surprise.  You'll notice that in the above -t4 trace, I 
didn't hear back from Sprint.

$ ping -t 5 ocalflxa-br-1
PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.89 ms
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=3.10 ms
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252 time=2.97 ms
hmm...still no ttl exceeded from Sprint?

$ ping -t 6 ocalflxa-br-1
PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.95 ms
>From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=0 Time to live exceeded
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=2.78 ms
>From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=1 Time to live exceeded

$ ping -t 7 ocalflxa-br-1
PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 time=2.88 ms
>From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=0 Time to live exceeded
64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 time=2.84 ms
>From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=1 Time to live exceeded

Is it just coincidence that there are 2 private IP hops in some 
traceroutes between us and Sprint?  i.e. Look at this trace from cogent:

Tracing the route to 209.208.33.1

   1 fa0-8.na01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 0 msec 4 msec 4 msec
   2 gi3-9.3507.core01.dca01.atlas.cogentco.com (66.28.67.225) 160 msec 4 msec 8 msec
   3 te3-1.ccr02.dca01.atlas.cogentco.com (154.54.3.158) 0 msec 0 msec 4 msec
   4 vl3493.mpd01.dca02.atlas.cogentco.com (154.54.7.230) 28 msec 4 msec
     te4-1.mpd01.dca02.atlas.cogentco.com (154.54.2.182) 52 msec
   5 vl3494.mpd01.iad01.atlas.cogentco.com (154.54.5.42) 4 msec 4 msec
     vl3497.mpd01.iad01.atlas.cogentco.com (154.54.5.66) 4 msec
   6 timewarner.iad01.atlas.cogentco.com (154.54.13.250) 4 msec
     peer-01-ge-3-1-2-13.asbn.twtelecom.net (66.192.252.217) 4 msec 12 msec
   7 66-194-200-202.static.twtelecom.net (66.194.200.202) 28 msec 28 msec 32 msec
   8 66-194-200-202.static.twtelecom.net (66.194.200.202) 32 msec 32 msec 28 msec
   9 andc-br-3-f2-0.atlantic.net (209.208.9.138) 32 msec 32 msec 32 msec
  10 172.22.122.1 32 msec 32 msec 32 msec
  11 10.247.28.205 32 msec 32 msec 32 msec
  12 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 32 msec 32 msec 28 msec
  13 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 28 msec 32 msec 32 msec
  14 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 32 msec 32 msec 28 msec
  15 vlan79.csw2.Washington1.Level3.net (4.68.17.126) 28 msec
     vlan89.csw3.Washington1.Level3.net (4.68.17.190) 32 msec
     vlan79.csw2.Washington1.Level3.net (4.68.17.126) 40 msec
  16 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 28 msec
     ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 28 msec
     ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 32 msec
  17 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 48 msec 48 msec 56 msec
  18 ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 44 msec 48 msec
     ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52 msec
  19 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 56 msec 104 msec 56 msec
  20 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 52 msec 52 msec 56 msec
  21 unknown.Level3.net (63.209.98.66) 52 msec 52 msec 56 msec
  22 andc-br-3-f2-0.atlantic.net (209.208.9.138) 52 msec 52 msec 56 msec
  23 172.22.122.1 52 msec 56 msec 52 msec
  24 10.247.28.205 52 msec 52 msec 56 msec
  25 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 52 msec 56 msec 52 msec
  26 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 56 msec 56 msec 56 msec
  27 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 52 msec 52 msec 52 msec
  28 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 52 msec
     vlan69.csw1.Washington1.Level3.net (4.68.17.62) 56 msec
     vlan89.csw3.Washington1.Level3.net (4.68.17.190) 56 msec
  29 ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 64 msec
     ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 52 msec 56 msec
  30 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 76 msec 72 msec 72 msec

I've seen the 172.22.122.1 & 10.247.28.205 hops before.  They occasionally
show up in traces when the traffic is jumping over to Sprint.  Sometimes
they don't show up though. i.e. Tracing from my house:

traceroute to 209.208.33.1 (209.208.33.1), 30 hops max, 40 byte packets
  1  172.31.0.1 (172.31.0.1)  0.336 ms  0.272 ms  0.268 ms
  2  10.210.160.1 (10.210.160.1)  10.109 ms  11.719 ms  14.265 ms
  3  gig7-0-4-101.orldflaabv-rtr1.cfl.rr.com (24.95.232.100)  15.302 ms  15.324 ms  16.687 ms
  4  198.228.95.24.cfl.res.rr.com (24.95.228.198)  16.688 ms  18.812 ms  18.816 ms
  5  te-3-3.car1.Orlando1.Level3.net (4.79.116.145)  20.084 ms  19.946 ms te-3-1.car1.Orlando1.Level3.net (4.79.116.137)  21.328 ms
  6  unknown.Level3.net (63.209.98.66)  19.900 ms  14.714 ms  14.689 ms
  7  andc-br-3-f2-0.atlantic.net (209.208.9.138)  104.058 ms  11.932 ms  13.584 ms
  8  ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98)  15.872 ms  15.886 ms  17.238 ms
  9  * * *
10  sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174)  41.277 ms  41.964 ms  41.955 ms
11  sl-st20-ash-10-0.sprintlink.net (144.232.20.152)  43.360 ms  44.578 ms  35.635 ms
12  te-10-1-0.edge2.Washington4.level3.net (4.68.63.209)  37.035 ms  37.062 ms  33.185 ms
13  vlan89.csw3.Washington1.Level3.net (4.68.17.190)  44.060 ms  44.057 ms vlan99.csw4.Washington1.Level3.net (4.68.17.254)  39.603 ms
14  ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137)  38.123 ms ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141)  39.546 ms ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133)  38.115 ms
15  ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85)  46.284 ms  46.275 ms  46.274 ms
16  ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18)  52.523 ms ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2)  53.338 ms ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18)  53.299 ms
17  ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149)  34.964 ms  39.582 ms  38.088 ms
18  ae-6-6.car1.Orlando1.Level3.net (4.69.133.77)  36.701 ms  38.144 ms  36.949 ms
19  unknown.Level3.net (63.209.98.66)  36.902 ms  37.750 ms  37.717 ms
20  andc-br-3-f2-0.atlantic.net (209.208.9.138)  37.729 ms  35.812 ms  35.048 ms
21  ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98)  37.485 ms  37.601 ms  36.495 ms
22  * * *
23  sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174)  56.459 ms  56.449 ms  57.709 ms
24  sl-st20-ash-10-0.sprintlink.net (144.232.20.152)  57.694 ms  57.692 ms  60.243 ms
25  te-10-1-0.edge2.Washington4.level3.net (4.68.63.209)  103.257 ms  100.829 ms  82.571 ms
26  vlan99.csw4.Washington1.Level3.net (4.68.17.254)  70.401 ms vlan89.csw3.Washington1.Level3.net (4.68.17.190)  69.262 ms vlan99.csw4.Washington1.Level3.net (4.68.17.254)  82.700 ms
27  ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137)  74.132 ms ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129)  74.135 ms ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137)  75.540 ms
28  ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85)  58.656 ms  60.838 ms  54.346 ms
29  ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18)  59.323 ms ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2)  59.336 ms ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18)  63.323 ms
30  ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149)  127.652 ms  57.884 ms  57.851 ms

>From the traces I've seen, it seems if the first Sprint hop is sl-bb20-dc, 
the private IP hops don't show up.  If the first Sprint hop is sl-crs2-dc, 
then the private IP hops are there.  I wonder if anyone from Sprint can 
shed some light on that?

Unfortunately, the Sprint engineer I intitially made contact with who was 
helpful and seemed curious about the issue seems to have vanished and 
isn't returning my calls or emails.  Anyone else from Sprintlink care to 
play?

----------------------------------------------------------------------
  Jon Lewis                   |  I route
  Senior Network Engineer     |  therefore you are
  Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________




More information about the NANOG mailing list