Bad bad routing problems?

Sat Aug 31 14:54:42 UTC 2002

We are seeing bad routing problems from outside our network. Can anyone
corroborate this or help?

We are on AS4276 and all traffic from us to our upstream seems good. Great
way to spend holiday weekend. /me wonders if anyone is even awake on the
NANOG list. :-)

2 addresses in our network I've tested with are:

216.223.200.14
and 216.223.192.68

Many many traces done, some interesting ones below:

Here is a succesful traceroute:
traceroute to 216.223.200.14 (216.223.200.14), 30 hops max, 40 byte
  1   <10 ms   <10 ms   <10 ms  rt1.altair7.com [209.11.155.129]
  2   <10 ms    15 ms    16 ms  209.10.41.130
  3   <10 ms    16 ms    15 ms  ge-6-1-0.core1.sjc1.globix.net [209.10.2.193]
  4   <10 ms    16 ms    16 ms  so-4-2-0.core1.sjc4.globix.net [209.10.11.221]
  5    63 ms    46 ms    47 ms  so-1-0-0.core2.cgx2.globix.net [209.10.10.150]
  6    62 ms    63 ms    47 ms  so-0-0-0.core1.cgx2.globix.net [209.10.10.157]
  7    78 ms    94 ms    94 ms  so-1-0-0.core2.nyc8.globix.net [209.10.10.162]
  8    78 ms    94 ms    94 ms  pos15-0.core2.nyc1.globix.net [209.10.11.169]
  9    78 ms    94 ms    94 ms  s5-0-peer1.nyc3.globix.net [209.10.12.18]
 10    78 ms    94 ms    94 ms  nyiix.peer.nac.net [198.32.160.20]
 11    78 ms    94 ms    93 ms  internetchannel.customer.nac.net [209.123.10.34]
 12    78 ms    94 ms    94 ms  auth-2.inch.com [216.223.200.14]

> These show failures before reaching our network:
>
> traceroute to 216.223.192.68 (216.223.192.68): 1-30 hops, 38 byte packets
>  1  paleagw4.hpl.external.hp.com (192.6.19.2)  1.95 ms  1.95 ms  5.86 ms
>  2  palgwb01-vbblpa.americas.hp.net (15.243.170.49)  0.976 ms  0.977 ms
> 0.976 ms
>  3  svl-edge-15.inet.qwest.net (65.115.64.25)  0.976 ms !H  *  0.977 ms !H
>  4  * * *
>  5  * * *
>
>
>
> traceroute to 216.223.192.68 (216.223.192.68), 30 hops max, 40 byte
> packets
>  1  e3-13.foundry1.cs.wisc.edu (198.133.224.116)  2.195 ms  1.754 ms
> 1.663 ms
>  2  e15.extreme1.cs.wisc.edu (128.105.1.1)  0.856 ms  0.765 ms  0.737 ms
>  3  144.92.128.194 (144.92.128.194)  1.550 ms  1.171 ms  1.264 ms
>  4  * * *
>  5  * * *
>
> All global crossing traces seem to loop within their router:
> traceroute to 216.223.192.68 (216.223.192.68), 30 hops max, 40 byte
> 1 64.214.13.1 (64.214.13.1) 0.695 ms 0.889 ms 0.485 ms 0.453 ms 0.350 ms
> 2 64.214.13.1 (64.214.13.1) 4.535 ms !N ms * ms 0.574 ms !N ms
> 3 * (64.214.13.1) 64.214.13.1 ms 0.408 ms !N ms * ms 0.425 ms
> 4 64.214.13.1 (64.214.13.1) 0.741 ms !N ms * ms 0.542 ms !N ms
>
> traceroute to 216.223.200.14 (216.223.200.14), 30 hops max, 40 byte
>  1  ROUTER6.VINEYARD.NET (204.17.195.236)  0.401 ms  0.342 ms  0.325 ms
>  2  bos-edge-01.inet.qwest.net (65.115.97.141)  7.249 ms  7.089 ms  6.943
> ms
>  3  * * *
>  4  * * bos-edge-01.inet.qwest.net (65.115.97.141)  9.885 ms !H
>  5  * * *
>  6  * * *
>  7  bos-edge-01.inet.qwest.net (65.115.97.141)  7.330 ms !H * *
>  8  * * *
>  9  * * *
> 10  * * *
> 11  * * *
> 12  * * *
> 13  * * *
> 14  * * *
> 15  * bos-edge-01.inet.qwest.net (65.115.97.141)  7.021 ms !H *
> 16  * bos-edge-01.inet.qwest.net (65.115.97.141)  172.856 ms !H *
> 17  * * *
> 18  * * *
> 19  * * bos-edge-01.inet.qwest.net (65.115.97.141)  7.127 ms !H
> 20  * * bos-edge-01.inet.qwest.net (65.115.97.141)  6.787 ms !H
> 21  * * bos-edge-01.inet.qwest.net (65.115.97.141)  61.972 ms !H
> 22  * bos-edge-01.inet.qwest.net (65.115.97.141)  22.525 ms !H *
> 23  * bos-edge-01.inet.qwest.net (65.115.97.141)  6.944 ms !H *
> 24  bos-edge-01.inet.qwest.net (65.115.97.141)  15.922 ms !H * *
> 25  bos-edge-01.inet.qwest.net (65.115.97.141)  11.212 ms !H *  6.895 ms
>
> This one works:
>
> traceroute to 216.223.192.68 (216.223.192.68), 30 hops max, 40 byte
> 1 tin-V8.cac.washington.edu (140.142.3.1) 1 ms 0 ms 1 ms
> 2 uwbr2-GE1-1.cac.washington.edu (140.142.155.24) 1 ms 1 ms 0 ms
> 3 cnsp2-wes-GE0-1-0.pnw-gigapop.net (198.107.151.5) 1 ms 1 ms 1 ms
> 4 ge-7-2-0.r04.sttlwa01.us.bb.verio.net (129.250.10.77) 0 ms 0 ms 1 ms
> 5 p4-1-0-0.r03.sttlwa01.us.bb.verio.net (129.250.2.230) 0 ms 1 ms 0 ms
> 6 p16-0-1-0.r20.sttlwa01.us.bb.verio.net (129.250.2.14) 1 ms 0 ms 1 ms
> 7 p16-0-1-2.r21.plalca01.us.bb.verio.net (129.250.5.83) 23 ms 23 ms 23 ms
> 8 p64-0-0-0.r20.plalca01.us.bb.verio.net (129.250.3.76) 22 ms 23 ms 22 ms
> 9 p16-1-1-1.r20.dllstx01.us.bb.verio.net (129.250.4.104) 57 ms 57 ms 57 ms
> 10 p64-0-0-0.r21.dllstx01.us.bb.verio.net (129.250.3.41) 57 ms 57 ms 57 ms
> 11 p16-2-0-0.r00.stngva01.us.bb.verio.net (129.250.5.35) 87 ms 87 ms 88 ms
> 12 p16-0-0-0.r02.stngva01.us.bb.verio.net (129.250.5.15) 87 ms 88 ms 87 ms
> 13 p16-7-0-0.r02.mclnva02.us.bb.verio.net (129.250.5.47) 88 ms 88 ms 88 ms
> 14 p4-3-0.r00.mclnva02.us.bb.verio.net (129.250.5.249) 88 ms 88 ms 88 ms
> 15 mae-east-atm.peer.nac.net (198.32.187.103) 89 ms 89 ms 90 ms
> 16 a9-0-1064.msfc1.nyc.nac.net (209.123.11.41) 95 ms 95 ms 96 ms
> 17 internetchannel.customer.nac.net (209.123.10.34) 96 ms 96 ms 96 ms
> 18 s0-varick-gw.inch.com (216.223.200.206) 100 ms 100 ms 100 ms
> 19 rubber-biscuit.inch.com (216.223.192.98) 99 ms 99 ms 99 ms
> 20 rubber-biscuit.inch.com (216.223.192.98) 99 ms !H 100 ms !H 98 ms !H