Level3 funkiness

Alex Thurlow alex at blastro.com
Wed Apr 15 20:07:58 UTC 2009


Same result from Cogent in Texas.  Dying at ge-6-2.hsa1.Denver1.Level3.net

# traceroute level3.net
traceroute to 63.211.236.36 (63.211.236.36), 30 hops max, 46 byte packets
  1  gi1-1.ccr01.aus02.atlas.cogentco.com (38.104.4.37)  0.493 ms  0.393 
ms  0.496 ms
  2  te4-4.ccr01.aus01.atlas.cogentco.com (154.54.25.157)  1.040 ms 
te4-3.ccr01.aus01.atlas.cogentco.com (154.54.25.153)  0.645 ms 
te4-4.ccr01.aus01.atlas.cogentco.com (154.54.25.157)  0.662 ms
  3  te8-2.mpd01.iah01.atlas.cogentco.com (154.54.1.66)  3.972 ms  3.999 
ms  3.926 ms
  4  te4-1.mpd01.dfw01.atlas.cogentco.com (154.54.2.14)  9.289 ms *  
9.249 ms
  5  te3-3.mpd01.dfw03.atlas.cogentco.com (154.54.6.94)  9.227 ms 
te7-3.mpd01.dfw03.atlas.cogentco.com (154.54.6.66)  9.670 ms 
te3-3.mpd01.dfw03.atlas.cogentco.com (154.54.6.94)  9.806 ms
  6  te-3-2.car3.Dallas1.Level3.net (4.68.110.109)  9.670 ms  9.813 ms  
9.431 ms
  7  * vlan79.csw2.Dallas1.Level3.net (4.68.19.126)  19.579 ms  20.438 ms
  8  ae-72-72.ebr2.Dallas1.Level3.net (4.69.136.141)  15.812 ms  15.417 
ms  18.290 ms
  9  ae-2.ebr1.Denver1.Level3.net (4.69.132.105)  26.905 ms *  24.321 ms
10  ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131)  24.518 ms !H *  
24.644 ms !H

Alex Thurlow
Blastro Networks

http://www.blastro.com
http://www.roxwel.com
http://www.yallwire.com


On 4/15/2009 2:49 PM, Dixon, Justin wrote:
>> -----Original Message-----
>> From: J. Oquendo [mailto:sil at infiltrated.net]
>> Sent: Wednesday, April 15, 2009 15:36
>> To: nanog at nanog.org
>> Subject: Level3 funkiness
>>
>>
>> Anyone else experience sporadic funkiness via
>> Level3? I can't even reach the main website from who
>> knows how many networks I've tried. Also friends
>> and former colleagues have tried to reach the site
>> to no avail.
>>
>> One of my machines on AT&T:
>> # traceroute level3.net
>> traceroute to level3.net (63.211.236.36), 30 hops max, 40 byte packets
>>
>> 4  cr1.n54ny.ip.att.net (12.122.105.58)  11.285 ms  21.702 ms  21.477
>>      
> ms
>    
>> 5  ggr2.n54ny.ip.att.net (12.122.131.141)  12.712 ms  10.194 ms
>>      
> 16.393 ms
>    
>> 6  so-8-0-0.car3.NewYork1.Level3.net (4.68.127.149)  9.975 ms  10.019
>>      
> ms  10.833 ms
>    
>> 7  vlan79.csw2.NewYork1.Level3.net (4.68.16.126)  10.162 ms  10.189 ms
>>      
> 14.474 ms
>    
>> 8  ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69)  15.763 ms  11.166
>>      
> ms  9.725 ms
>    
>> 9  ae-3-3.ebr4.Washington1.Level3.net (4.69.132.93)  16.139 ms  30.616
>>      
> ms  16.275 ms
>    
>> 10  ae-64-64.csw1.Washington1.Level3.net (4.69.134.178)  15.684 ms
>>      
> ae-74->74.csw2.Washington1.Level3.net (4.69.134.182)  21.870 ms
> ae-84->84.csw3.Washington1.Level3.net (4.69.134.186)  28.729 ms
>    
>> 11  ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157)  17.035 ms
>>      
> ae-62->62.ebr2.Washington1.Level3.net (4.69.134.145)  17.041 ms
> ae-72->72.ebr2.Washington1.Level3.net (4.69.134.149)  21.940 ms
>    
>> 12  ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69)  31.671 ms  42.407 ms
>>      
> 45.774 ms
>    
>> 13  ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113)  31.922 ms  32.115
>>      
> ms  38.135 ms
>    
>> 14  ae-3.ebr2.Denver1.Level3.net (4.69.132.61)  75.265 ms  67.528 ms
>>      
> 67.937 ms
>    
>> 15  ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35)  62.587 ms !H
>>      
> ge-9->1.hsa1.Denver1.Level3.net (4.68.107.99)  62.543 ms !H
> ge-9-2.hsa1.Denver1.Level3.net>(4.68.107.163)  75.797 ms !H
>    
>>
>> (From Texas through Above.net)
>> $ traceroute level3.net|tail -n 1
>> traceroute to level3.net (63.211.236.36), 64 hops max, 40 byte packets
>> 11  ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131)  21.473 ms !H *
>>      
> ge-6->0.hsa1.Denver1.Level3.net (4.68.107.3)  21.547 ms !H
>    
>> Confirmed it can't be reached from Travelers Ins, The
>> Hartford, none of my connections. Anyone else seeing
>> issues? I'm seeing drop off from clients going through
>> their Atlanta interconnects with Charter and two other
>> providers, which I can't make sense of. I DO KNOW they
>> experienced some sort of issue with a TDM switch or so
>> they said... Very broad statements: "We know teh
>> interwebs are down please stand by"
>>
>> I know websites are one thing, but the chances of the
>> website going down, a TDM switch being wacky and now
>> clients traversing their networks complaining all at
>> once seems a little out of the ordinary.
>>
>> =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+>=+=+
>> J. Oquendo
>> SGFA, SGFE, C|EH, CNDA, CHFI, OSCP
>>
>> "It takes 20 years to build a reputation and five minutes to
>> ruin it. If you think about that, you'll do things
>> differently." - Warren Buffett
>>
>> 227C 5D35 7DCB 0893 95AA  4771 1DCE 1FD1 5CCD 6B5E
>> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x5CCD6B5E
>>
>>      
> Looks similar from Sprints perspective via
> https://www.sprint.net/lg/lg_start.php as well...
>
> Sprint Source Region: Anaheim, CA (sl-bb20-ana)
> IP Destination: 63.211.236.36
> Performing: ICMP Traceroute
> Tracing the route to 63.211.236.36
>    1 sl-crs2-ana-0-13-5-0.sprintlink.net (144.232.1.177) 4 msec 0 msec 0
> msec
>    2 144.232.19.227 4 msec 24 msec
>      sl-st30-la-0-4-5-0.sprintlink.net (144.232.24.41) 0 msec
>    3 xe-7-3-0.edge1.losangeles9.Level3.net (4.68.111.169) [AS 3356] 4
> msec
>      xe-7-2-0.edge1.losangeles9.Level3.net (4.68.111.89) [AS 3356] 0 msec
> 0 msec
>    4 vlan69.csw1.LosAngeles1.Level3.net (4.68.20.62) [AS 3356] 16 msec
>      vlan99.csw4.LosAngeles1.Level3.net (4.68.20.254) [AS 3356] 12 msec
>      vlan89.csw3.LosAngeles1.Level3.net (4.68.20.190) [AS 3356] 24 msec
>    5 ae-63-63.ebr3.LosAngeles1.Level3.net (4.69.137.33) [AS 3356] 4 msec
> 12 msec
>      ae-93-93.ebr3.LosAngeles1.Level3.net (4.69.137.45) [AS 3356] 4 msec
>    6 ae-2.ebr3.SanJose1.Level3.net (4.69.132.9) [AS 3356] 40 msec 16 msec
> 16 msec
>    7 ae-73-73.csw2.SanJose1.Level3.net (4.69.134.230) [AS 3356] 16 msec
>      ae-63-63.csw1.SanJose1.Level3.net (4.69.134.226) [AS 3356] 12 msec
> 12 msec
>    8 ae-82-82.ebr2.SanJose1.Level3.net (4.69.134.217) [AS 3356] 12 msec
>      ae-92-92.ebr2.SanJose1.Level3.net (4.69.134.221) [AS 3356] 32 msec
> 12 msec
>    9 ae-3.ebr1.Denver1.Level3.net (4.69.132.58) [AS 3356] 60 msec 48 msec
> 52 msec
>   10 ge-6-1.hsa1.Denver1.Level3.net (4.68.107.67) [AS 3356] !H
>      ge-6-2.hsa1.Denver1.Level3.net (4.68.107.131) [AS 3356] !H  *
>
> Completed - Wed Apr 15 15:43:21 EDT 2009
>
> Sprint Source Region: Atlanta, GA (sl-bb20-atl)
> IP Destination: 63.211.236.36
> Performing: ICMP Traceroute
> Tracing the route to 63.211.236.36
>    1 sl-bb21-atl-14-0.sprintlink.net (144.232.12.142) 0 msec 0 msec 0
> msec
>    2 sl-crs1-atl-0-4-0-3.sprintlink.net (144.232.12.148) 4 msec 0 msec 0
> msec
>    3 sl-crs1-rly-0-0-5-0.sprintlink.net (144.232.20.177) 20 msec 16 msec
> 16 msec
>    4 sl-crs1-dc-0-8-0-0.sprintlink.net (144.232.19.213) 20 msec 20 msec
> 24 msec
>    5 sl-st31-ash-0-4-0-1.sprintlink.net (144.232.19.4) 20 msec 20 msec 24
> msec
>    6 4.68.63.205 [AS 3356] 20 msec
>      te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) [AS 3356] 20
> msec 24 msec
>    7 vlan79.csw2.Washington1.Level3.net (4.68.17.126) [AS 3356] 24 msec
> 28 msec
>      vlan89.csw3.Washington1.Level3.net (4.68.17.190) [AS 3356] 28 msec
>    8 ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) [AS 3356] 24
> msec
>      ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS 3356] 24
> msec 24 msec
>    9 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) [AS 3356] 28 msec 28
> msec 28 msec
>   10 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) [AS 3356] 28 msec
> 28 msec 28 msec
>   11 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 48 msec 48 msec
> 52 msec
>   12 ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) [AS 3356] !H
>      ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) [AS 3356] !H  *
> Completed - Wed Apr 15 15:44:17 EDT 2009
>
> Sprint Source Region: Chicago, IL (sl-bb20-chi)
> IP Destination: 63.211.236.36
> Performing: ICMP Traceroute
> Tracing the route to 63.211.236.36
>    1 sl-crs1-chi-0-0-0-0.sprintlink.net (144.232.26.115) 0 msec 4 msec 0
> msec
>    2 sl-st20-chi-4-0-0.sprintlink.net (144.232.18.153) 0 msec
>      sl-st20-chi-13-0-0.sprintlink.net (144.232.20.3) 0 msec 0 msec
>    3 144.232.19.174 0 msec 0 msec 4 msec
>    4 ae-31-53.ebr1.Chicago1.Level3.net (4.68.101.94) [AS 3356] 8 msec
>      ae-31-51.ebr1.Chicago1.Level3.net (4.68.101.30) [AS 3356] 8 msec
>      ae-31-55.ebr1.Chicago1.Level3.net (4.68.101.158) [AS 3356] 4 msec
>    5 ae-6.ebr1.Chicago2.Level3.net (4.69.140.190) [AS 3356] 0 msec 4 msec
> 0 msec
>    6 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 44 msec 44 msec
> 36 msec
>    7 ge-9-0.hsa1.Denver1.Level3.net (4.68.107.35) [AS 3356] !H
>      ge-9-2.hsa1.Denver1.Level3.net (4.68.107.163) [AS 3356] !H  *
>
> Completed - Wed Apr 15 15:44:49 EDT 2009
>
> Sprint Source Region: Relay, MD (sl-bb20-rly)
> IP Destination: 63.211.236.36
> Performing: ICMP Traceroute
> Tracing the route to 63.211.236.36
>    1 sl-crs2-rly-0-1-0-0.sprintlink.net (144.232.3.96) 0 msec 0 msec 4
> msec
>    2 sl-crs2-dc-0-12-2-0.sprintlink.net (144.232.19.221) 0 msec 0 msec 0
> msec
>    3 sl-st31-ash-0-8-0-0.sprintlink.net (144.232.19.6) 0 msec
>      sl-st31-ash-0-1-0-0.sprintlink.net (144.232.19.230) 0 msec
>      sl-st31-ash-0-8-0-0.sprintlink.net (144.232.19.6) 4 msec
>    4 4.68.63.205 [AS 3356] 4 msec
>      te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) [AS 3356] 0
> msec
>      4.68.63.205 [AS 3356] 0 msec
>    5 vlan69.csw1.Washington1.Level3.net (4.68.17.62) [AS 3356] 0 msec
>      vlan89.csw3.Washington1.Level3.net (4.68.17.190) [AS 3356] 4 msec 12
> msec
>    6 ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS 3356] 4 msec
>      ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) [AS 3356] 0 msec
>      ae-82-82.ebr2.Washington1.Level3.net (4.69.134.153) [AS 3356] 4 msec
>    7 ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69) [AS 3356] 20 msec 20
> msec 20 msec
>    8 ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113) [AS 3356] 20 msec
> 24 msec 20 msec
>    9 ae-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] 52 msec 64 msec
> 52 msec
>   10 ge-9-1.hsa1.Denver1.Level3.net (4.68.107.99) [AS 3356] !H  *  !H
>
> Completed - Wed Apr 15 15:45:37 EDT 2009
>
>
>
>    

-- 
Alex Thurlow
Blastro Networks

http://www.blastro.com
http://www.roxwel.com
http://www.yallwire.com






More information about the NANOG mailing list