impossible circuit
Paul Wall
pauldotwall at gmail.com
Mon Aug 18 20:46:29 UTC 2008
Jon,
I think we can safely conclude from the information provided that
you're looking at some sort of a misconfigured traffic mirroring or
[un]lawful intercept.
Sadly, as neither Sprint nor your loop provider will fess up, I don't
think you're going to get much further on here.
Probably best to order a new loop and cancel the existing one.
Drive Slow,
Paul
- Original message -
I just went ahead and "re-broke" the circuit ...
On 8/17/08, Jon Lewis <jlewis at lewis.org> wrote:
> On Tue, 12 Aug 2008, Jon Lewis wrote:
>
>>> What would happen if you pinged the Ocala router such that the TTL was 1
>>> when travelling over the DS3? From your traceroute it seems it travelled
>>> two IP hops that did not send ICMP error messages, but it might just be
>>> that the ICMP errors from the Ocala router are arriving first.
>>
>> Based on where the dupes are coming from, I assume pinging across the DS3
>> with TTL tuned to expire at the Ocala side would result in TTL exceeded
>> messages from both Ocala and the Sprint router where the packets are
>> injected
>> into Sprint's network. It doesn't look as if IOS gives the option to set
>> TTL
>> on ping...so I'd try this from a Linux machine in our data center.
>
> I just went ahead and "re-broke" the circuit for a bit by turning it back
> to hdlc to see if the issue is still there and to run some additional
> tests. Someone is still cross connecting our Orlando->Ocala traffic over
> to Sprint.
>
> I did your suggested ping with short TTL and the result was close to what
> I expected.
>
> $ traceroute ocalflxa-br-1
> traceroute to ocalflxa-br-1.atlantic.net (209.208.6.229), 30 hops max, 38
> byte packets
> 1 209.208.25.165 (209.208.25.165) 0.539 ms 0.426 ms 0.388 ms
> 2 69.28.72.162 (69.28.72.162) 0.246 ms 0.351 ms 0.223 ms
> 3 andc-br-3-f2-0 (209.208.9.138) 0.559 ms 0.435 ms 0.471 ms
> 4 ocalflxa-br-1-s1-0 (209.208.112.98) 2.735 ms * 2.656 ms
>
> So, I need a TTL of 4 to get there from this machine.
>
> $ ping -t4 ocalflxa-br-1
> PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
> time=2.68 ms
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
> time=2.72 ms
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252
> time=2.88 ms
>
> Decrease ttl by one, and I get the expected ttl exceeded from the Orlando
> side of the circuit.
>
> $ ping -t 3 ocalflxa-br-1
> PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
> >From andc-br-3-f2-0.atlantic.net (209.208.9.138) icmp_seq=0 Time to live
> exceeded
>
> Now, here's a mild surprise. You'll notice that in the above -t4 trace, I
> didn't hear back from Sprint.
>
> $ ping -t 5 ocalflxa-br-1
> PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
> time=2.89 ms
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
> time=3.10 ms
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252
> time=2.97 ms
> hmm...still no ttl exceeded from Sprint?
>
> $ ping -t 6 ocalflxa-br-1
> PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
> time=2.95 ms
> >From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=0 Time to
> live exceeded
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
> time=2.78 ms
> >From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=1 Time to
> live exceeded
>
> $ ping -t 7 ocalflxa-br-1
> PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data.
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252
> time=2.88 ms
> >From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=0 Time to
> live exceeded
> 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252
> time=2.84 ms
> >From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=1 Time to
> live exceeded
>
> Is it just coincidence that there are 2 private IP hops in some
> traceroutes between us and Sprint? i.e. Look at this trace from cogent:
>
> Tracing the route to 209.208.33.1
>
> 1 fa0-8.na01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 0 msec 4
> msec 4 msec
> 2 gi3-9.3507.core01.dca01.atlas.cogentco.com (66.28.67.225) 160 msec 4
> msec 8 msec
> 3 te3-1.ccr02.dca01.atlas.cogentco.com (154.54.3.158) 0 msec 0 msec 4
> msec
> 4 vl3493.mpd01.dca02.atlas.cogentco.com (154.54.7.230) 28 msec 4 msec
> te4-1.mpd01.dca02.atlas.cogentco.com (154.54.2.182) 52 msec
> 5 vl3494.mpd01.iad01.atlas.cogentco.com (154.54.5.42) 4 msec 4 msec
> vl3497.mpd01.iad01.atlas.cogentco.com (154.54.5.66) 4 msec
> 6 timewarner.iad01.atlas.cogentco.com (154.54.13.250) 4 msec
> peer-01-ge-3-1-2-13.asbn.twtelecom.net (66.192.252.217) 4 msec 12 msec
> 7 66-194-200-202.static.twtelecom.net (66.194.200.202) 28 msec 28 msec 32
> msec
> 8 66-194-200-202.static.twtelecom.net (66.194.200.202) 32 msec 32 msec 28
> msec
> 9 andc-br-3-f2-0.atlantic.net (209.208.9.138) 32 msec 32 msec 32 msec
> 10 172.22.122.1 32 msec 32 msec 32 msec
> 11 10.247.28.205 32 msec 32 msec 32 msec
> 12 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 32 msec 32 msec 28
> msec
> 13 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 28 msec 32 msec 32
> msec
> 14 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 32 msec 32 msec 28
> msec
> 15 vlan79.csw2.Washington1.Level3.net (4.68.17.126) 28 msec
> vlan89.csw3.Washington1.Level3.net (4.68.17.190) 32 msec
> vlan79.csw2.Washington1.Level3.net (4.68.17.126) 40 msec
> 16 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 28 msec
> ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 28 msec
> ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 32 msec
> 17 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 48 msec 48 msec 56 msec
> 18 ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 44 msec 48 msec
> ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52 msec
> 19 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 56 msec 104 msec 56 msec
> 20 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 52 msec 52 msec 56 msec
> 21 unknown.Level3.net (63.209.98.66) 52 msec 52 msec 56 msec
> 22 andc-br-3-f2-0.atlantic.net (209.208.9.138) 52 msec 52 msec 56 msec
> 23 172.22.122.1 52 msec 56 msec 52 msec
> 24 10.247.28.205 52 msec 52 msec 56 msec
> 25 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 52 msec 56 msec 52
> msec
> 26 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 56 msec 56 msec 56
> msec
> 27 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 52 msec 52 msec 52
> msec
> 28 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 52 msec
> vlan69.csw1.Washington1.Level3.net (4.68.17.62) 56 msec
> vlan89.csw3.Washington1.Level3.net (4.68.17.190) 56 msec
> 29 ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 64 msec
> ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 52 msec 56 msec
> 30 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 76 msec 72 msec 72 msec
>
> I've seen the 172.22.122.1 & 10.247.28.205 hops before. They occasionally
> show up in traces when the traffic is jumping over to Sprint. Sometimes
> they don't show up though. i.e. Tracing from my house:
>
> traceroute to 209.208.33.1 (209.208.33.1), 30 hops max, 40 byte packets
> 1 172.31.0.1 (172.31.0.1) 0.336 ms 0.272 ms 0.268 ms
> 2 10.210.160.1 (10.210.160.1) 10.109 ms 11.719 ms 14.265 ms
> 3 gig7-0-4-101.orldflaabv-rtr1.cfl.rr.com (24.95.232.100) 15.302 ms
> 15.324 ms 16.687 ms
> 4 198.228.95.24.cfl.res.rr.com (24.95.228.198) 16.688 ms 18.812 ms
> 18.816 ms
> 5 te-3-3.car1.Orlando1.Level3.net (4.79.116.145) 20.084 ms 19.946 ms
> te-3-1.car1.Orlando1.Level3.net (4.79.116.137) 21.328 ms
> 6 unknown.Level3.net (63.209.98.66) 19.900 ms 14.714 ms 14.689 ms
> 7 andc-br-3-f2-0.atlantic.net (209.208.9.138) 104.058 ms 11.932 ms
> 13.584 ms
> 8 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 15.872 ms 15.886 ms
> 17.238 ms
> 9 * * *
> 10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 41.277 ms 41.964 ms
> 41.955 ms
> 11 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 43.360 ms 44.578 ms
> 35.635 ms
> 12 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 37.035 ms 37.062
> ms 33.185 ms
> 13 vlan89.csw3.Washington1.Level3.net (4.68.17.190) 44.060 ms 44.057 ms
> vlan99.csw4.Washington1.Level3.net (4.68.17.254) 39.603 ms
> 14 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 38.123 ms
> ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141) 39.546 ms
> ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 38.115 ms
> 15 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 46.284 ms 46.275 ms
> 46.274 ms
> 16 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52.523 ms
> ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 53.338 ms
> ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 53.299 ms
> 17 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 34.964 ms 39.582 ms
> 38.088 ms
> 18 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 36.701 ms 38.144 ms
> 36.949 ms
> 19 unknown.Level3.net (63.209.98.66) 36.902 ms 37.750 ms 37.717 ms
> 20 andc-br-3-f2-0.atlantic.net (209.208.9.138) 37.729 ms 35.812 ms
> 35.048 ms
> 21 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 37.485 ms 37.601 ms
> 36.495 ms
> 22 * * *
> 23 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 56.459 ms 56.449 ms
> 57.709 ms
> 24 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 57.694 ms 57.692 ms
> 60.243 ms
> 25 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 103.257 ms
> 100.829 ms 82.571 ms
> 26 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 70.401 ms
> vlan89.csw3.Washington1.Level3.net (4.68.17.190) 69.262 ms
> vlan99.csw4.Washington1.Level3.net (4.68.17.254) 82.700 ms
> 27 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 74.132 ms
> ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 74.135 ms
> ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 75.540 ms
> 28 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 58.656 ms 60.838 ms
> 54.346 ms
> 29 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 59.323 ms
> ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 59.336 ms
> ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 63.323 ms
> 30 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 127.652 ms 57.884 ms
> 57.851 ms
>
> >From the traces I've seen, it seems if the first Sprint hop is sl-bb20-dc,
> the private IP hops don't show up. If the first Sprint hop is sl-crs2-dc,
> then the private IP hops are there. I wonder if anyone from Sprint can
> shed some light on that?
>
> Unfortunately, the Sprint engineer I intitially made contact with who was
> helpful and seemed curious about the issue seems to have vanished and
> isn't returning my calls or emails. Anyone else from Sprintlink care to
> play?
>
> ----------------------------------------------------------------------
> Jon Lewis | I route
> Senior Network Engineer | therefore you are
> Atlantic Net |
> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
>
>
--
Sent from Gmail for mobile | mobile.google.com
More information about the NANOG
mailing list