impossible circuit

Paul Wall pauldotwall at gmail.com
Mon Aug 18 15:46:29 CDT 2008


Jon,

I think we can safely conclude from the information provided that
you're looking at some sort of a misconfigured traffic mirroring or
[un]lawful intercept.

Sadly, as neither Sprint nor your loop provider will fess up, I don't
think you're going to get much further on here.

Probably best to order a new loop and cancel the existing one.

Drive Slow,
Paul

- Original message -
I just went ahead and "re-broke" the circuit ...

On 8/17/08, Jon Lewis <jlewis at lewis.org> wrote:
> On Tue, 12 Aug 2008, Jon Lewis wrote:
>
>>> What would happen if you pinged the Ocala router such that the TTL was 1
>>> when travelling over the DS3? From your traceroute it seems it travelled
>>> two IP hops that did not send ICMP error messages, but it might just be
>>> that the ICMP errors from the Ocala router are arriving first.
>>
>> Based on where the dupes are coming from, I assume pinging across the DS3
>> with TTL tuned to expire at the Ocala side would result in TTL exceeded
>> messages from both Ocala and the Sprint router where the packets are
>> injected
>> into Sprint's network.  It doesn't look as if IOS gives the option to set
>> TTL
>> on ping...so I'd try this from a Linux machine in our data center.
>
> I just went ahead and "re-broke" the circuit for a bit by turning it back
> to hdlc to see if the issue is still there and to run some additional
> tests.  Someone is still cross connecting our Orlando->Ocala traffic over
> to Sprint.
>
> I did your suggested ping with short TTL and the result was close to what
> I expected.
>
> $ traceroute ocalflxa-br-1
> traceroute to ocalflxa-br-1.atlantic.net (209.208.6.229), 30 hops max, 38
> byte packets
>   1  209.208.25.165 (209.208.25.165)  0.539 ms  0.426 ms  0.388 ms
>   2  69.28.72.162 (69.28.72.162)  0.246 ms  0.351 ms  0.223 ms
>   3  andc-br-3-f2-0 (209.208.9.138)  0.559 ms  0.435 ms  0.471 ms
>   4  ocalflxa-br-1-s1-0 (209.208.112.98)  2.735 ms *  2.656 ms
>
> So, I need a TTL of 4 to get there from this machine.
>
> $ ping -t4 ocalflxa-br-1
> PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
> time=2.68 ms
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
> time=2.72 ms
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252
> time=2.88 ms
>
> Decrease ttl by one, and I get the expected ttl exceeded from the Orlando
> side of the circuit.
>
> $ ping -t 3 ocalflxa-br-1
> PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
> >From andc-br-3-f2-0.atlantic.net (209.208.9.138) icmp_seq=0 Time to live
> exceeded
>
> Now, here's a mild surprise.  You'll notice that in the above -t4 trace, I
> didn't hear back from Sprint.
>
> $ ping -t 5 ocalflxa-br-1
> PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
> time=2.89 ms
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
> time=3.10 ms
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252
> time=2.97 ms
> hmm...still no ttl exceeded from Sprint?
>
> $ ping -t 6 ocalflxa-br-1
> PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
> time=2.95 ms
> >From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=0 Time to
> live exceeded
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
> time=2.78 ms
> >From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=1 Time to
> live exceeded
>
> $ ping -t 7 ocalflxa-br-1
> PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
> time=2.88 ms
> >From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=0 Time to
> live exceeded
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
> time=2.84 ms
> >From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=1 Time to
> live exceeded
>
> Is it just coincidence that there are 2 private IP hops in some
> traceroutes between us and Sprint?  i.e. Look at this trace from cogent:
>
> Tracing the route to 209.208.33.1
>
>    1 fa0-8.na01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 0 msec 4
> msec 4 msec
>    2 gi3-9.3507.core01.dca01.atlas.cogentco.com (66.28.67.225) 160 msec 4
> msec 8 msec
>    3 te3-1.ccr02.dca01.atlas.cogentco.com (154.54.3.158) 0 msec 0 msec 4
> msec
>    4 vl3493.mpd01.dca02.atlas.cogentco.com (154.54.7.230) 28 msec 4 msec
>      te4-1.mpd01.dca02.atlas.cogentco.com (154.54.2.182) 52 msec
>    5 vl3494.mpd01.iad01.atlas.cogentco.com (154.54.5.42) 4 msec 4 msec
>      vl3497.mpd01.iad01.atlas.cogentco.com (154.54.5.66) 4 msec
>    6 timewarner.iad01.atlas.cogentco.com (154.54.13.250) 4 msec
>      peer-01-ge-3-1-2-13.asbn.twtelecom.net (66.192.252.217) 4 msec 12 msec
>    7 66-194-200-202.static.twtelecom.net (66.194.200.202) 28 msec 28 msec 32
> msec
>    8 66-194-200-202.static.twtelecom.net (66.194.200.202) 32 msec 32 msec 28
> msec
>    9 andc-br-3-f2-0.atlantic.net (209.208.9.138) 32 msec 32 msec 32 msec
>   10 172.22.122.1 32 msec 32 msec 32 msec
>   11 10.247.28.205 32 msec 32 msec 32 msec
>   12 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 32 msec 32 msec 28
> msec
>   13 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 28 msec 32 msec 32
> msec
>   14 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 32 msec 32 msec 28
> msec
>   15 vlan79.csw2.Washington1.Level3.net (4.68.17.126) 28 msec
>      vlan89.csw3.Washington1.Level3.net (4.68.17.190) 32 msec
>      vlan79.csw2.Washington1.Level3.net (4.68.17.126) 40 msec
>   16 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 28 msec
>      ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 28 msec
>      ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 32 msec
>   17 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 48 msec 48 msec 56 msec
>   18 ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 44 msec 48 msec
>      ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52 msec
>   19 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 56 msec 104 msec 56 msec
>   20 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 52 msec 52 msec 56 msec
>   21 unknown.Level3.net (63.209.98.66) 52 msec 52 msec 56 msec
>   22 andc-br-3-f2-0.atlantic.net (209.208.9.138) 52 msec 52 msec 56 msec
>   23 172.22.122.1 52 msec 56 msec 52 msec
>   24 10.247.28.205 52 msec 52 msec 56 msec
>   25 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 52 msec 56 msec 52
> msec
>   26 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 56 msec 56 msec 56
> msec
>   27 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 52 msec 52 msec 52
> msec
>   28 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 52 msec
>      vlan69.csw1.Washington1.Level3.net (4.68.17.62) 56 msec
>      vlan89.csw3.Washington1.Level3.net (4.68.17.190) 56 msec
>   29 ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 64 msec
>      ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 52 msec 56 msec
>   30 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 76 msec 72 msec 72 msec
>
> I've seen the 172.22.122.1 & 10.247.28.205 hops before.  They occasionally
> show up in traces when the traffic is jumping over to Sprint.  Sometimes
> they don't show up though. i.e. Tracing from my house:
>
> traceroute to 209.208.33.1 (209.208.33.1), 30 hops max, 40 byte packets
>   1  172.31.0.1 (172.31.0.1)  0.336 ms  0.272 ms  0.268 ms
>   2  10.210.160.1 (10.210.160.1)  10.109 ms  11.719 ms  14.265 ms
>   3  gig7-0-4-101.orldflaabv-rtr1.cfl.rr.com (24.95.232.100)  15.302 ms
> 15.324 ms  16.687 ms
>   4  198.228.95.24.cfl.res.rr.com (24.95.228.198)  16.688 ms  18.812 ms
> 18.816 ms
>   5  te-3-3.car1.Orlando1.Level3.net (4.79.116.145)  20.084 ms  19.946 ms
> te-3-1.car1.Orlando1.Level3.net (4.79.116.137)  21.328 ms
>   6  unknown.Level3.net (63.209.98.66)  19.900 ms  14.714 ms  14.689 ms
>   7  andc-br-3-f2-0.atlantic.net (209.208.9.138)  104.058 ms  11.932 ms
> 13.584 ms
>   8  ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98)  15.872 ms  15.886 ms
> 17.238 ms
>   9  * * *
> 10  sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174)  41.277 ms  41.964 ms
> 41.955 ms
> 11  sl-st20-ash-10-0.sprintlink.net (144.232.20.152)  43.360 ms  44.578 ms
> 35.635 ms
> 12  te-10-1-0.edge2.Washington4.level3.net (4.68.63.209)  37.035 ms  37.062
> ms  33.185 ms
> 13  vlan89.csw3.Washington1.Level3.net (4.68.17.190)  44.060 ms  44.057 ms
> vlan99.csw4.Washington1.Level3.net (4.68.17.254)  39.603 ms
> 14  ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137)  38.123 ms
> ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141)  39.546 ms
> ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133)  38.115 ms
> 15  ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85)  46.284 ms  46.275 ms
> 46.274 ms
> 16  ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18)  52.523 ms
> ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2)  53.338 ms
> ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18)  53.299 ms
> 17  ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149)  34.964 ms  39.582 ms
> 38.088 ms
> 18  ae-6-6.car1.Orlando1.Level3.net (4.69.133.77)  36.701 ms  38.144 ms
> 36.949 ms
> 19  unknown.Level3.net (63.209.98.66)  36.902 ms  37.750 ms  37.717 ms
> 20  andc-br-3-f2-0.atlantic.net (209.208.9.138)  37.729 ms  35.812 ms
> 35.048 ms
> 21  ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98)  37.485 ms  37.601 ms
> 36.495 ms
> 22  * * *
> 23  sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174)  56.459 ms  56.449 ms
> 57.709 ms
> 24  sl-st20-ash-10-0.sprintlink.net (144.232.20.152)  57.694 ms  57.692 ms
> 60.243 ms
> 25  te-10-1-0.edge2.Washington4.level3.net (4.68.63.209)  103.257 ms
> 100.829 ms  82.571 ms
> 26  vlan99.csw4.Washington1.Level3.net (4.68.17.254)  70.401 ms
> vlan89.csw3.Washington1.Level3.net (4.68.17.190)  69.262 ms
> vlan99.csw4.Washington1.Level3.net (4.68.17.254)  82.700 ms
> 27  ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137)  74.132 ms
> ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129)  74.135 ms
> ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137)  75.540 ms
> 28  ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85)  58.656 ms  60.838 ms
> 54.346 ms
> 29  ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18)  59.323 ms
> ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2)  59.336 ms
> ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18)  63.323 ms
> 30  ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149)  127.652 ms  57.884 ms
> 57.851 ms
>
> >From the traces I've seen, it seems if the first Sprint hop is sl-bb20-dc,
> the private IP hops don't show up.  If the first Sprint hop is sl-crs2-dc,
> then the private IP hops are there.  I wonder if anyone from Sprint can
> shed some light on that?
>
> Unfortunately, the Sprint engineer I intitially made contact with who was
> helpful and seemed curious about the issue seems to have vanished and
> isn't returning my calls or emails.  Anyone else from Sprintlink care to
> play?
>
> ----------------------------------------------------------------------
>   Jon Lewis                   |  I route
>   Senior Network Engineer     |  therefore you are
>   Atlantic Net                |
> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
>
>

-- 
Sent from Gmail for mobile | mobile.google.com




More information about the NANOG mailing list