level3 issue in chicago (was: FYI, Level 3 issues in Dallas)

Thu Nov 20 00:29:26 UTC 2014

Appears to have been resolved after seven hours.  My ticket just says:

"We isolated the routing issue and resolved it.
The issue was due to a misconfiguration on one our core routers."

Now that the issue is corrected, interestingly enough, my trace goes
from Tampa to Chicago instead of Dallas.

David

-----Original Message-----
From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Cool Hand Luke
Sent: Wednesday, November 19, 2014 11:23 AM
To: nanog at nanog.org
Subject: Re: level3 issue in chicago (was: FYI, Level 3 issues in
Dallas)

like david below, we've been getting reports this morning from customers
unable to reach various web sites.

investigating a number of these reports, the one commonality is level3
in chicago. sites are reachable from level3/cincinnati but not
level3/chicago.

traceroutes make it one hop past our bgp peer and seem to die there.

for one particular remote web server (node1104.follett.com, and possibly
others) traceroutes and pings from level3's looking glass (using chicago
as the site) complete successfully, while neither does when ran from my
gear through the same site.

On 11/19/2014 10:46 AM, garya at completeweb.net wrote:
> Also peering problems in Chicago, had to drop BGP to them until they
figure it out.
> Reported it 4 hours ago, no good response yet.
> Gary
>
>> We have some customers unable to access their websites, seeing this 
>> on the way to them:
>>
>>   4  ae-0-11.bar2.Tampa1.Level3.net (4.69.137.110)  0.343 ms  0.423 
>> ms
>> 0.406 ms
>>   5  ae-12-12.ebr1.Dallas1.Level3.net (4.69.137.118)  23.940 ms  
>> 23.470 ms  23.426 ms
>>   6  ae-71-71.csw2.Dallas1.Level3.net (4.69.151.137)  24.026 ms 
>> ae-81-81.csw3.Dallas1.Level3.net (4.69.151.149)  24.007 ms 
>> ae-61-61.csw1.Dallas1.Level3.net (4.69.151.125)  23.855 ms
>>   7  ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146)  23.361 ms 
>> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203)  23.329 ms 
>> ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  23.936 ms
>>   8  vlan90.csw4.Dallas1.Level3.net (4.69.145.254)  23.378 ms 
>> vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.387 ms 
>> vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  23.963 ms
>>   9  ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  23.357 ms 
>> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16)  22.779 ms 
>> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203)  62.640 ms 10  
>> vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  24.179 ms  23.297 ms
>> 23.403 ms
>> 11  ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  65.617 ms 
>> ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75)  23.286 ms 
>> ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18)  23.553 ms
>> 12  vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.303 ms 
>> vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  24.027 ms  24.028 ms
>> 13  ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139)  24.018 ms 
>> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146)  23.374 ms 
>> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203)  23.342 ms
>> 14  vlan90.csw4.Dallas1.Level3.net (4.69.145.254)  24.068 ms 
>> vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.361 ms  23.348 ms
>> 15  ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  24.023 ms 
>> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16)  22.740 ms 
>> ae-4-90.edge10.Dallas1.Level3.net (4.69.145.210)  23.963 ms
>> 16  vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  23.420 ms 
>> vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.400 ms 
>> vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  25.317 ms
>> 17  ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82)  23.985 ms 
>> ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139)  23.476 ms 
>> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144)  51.762 ms
>> 18  vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  23.416 ms 
>> vlan90.csw4.Dallas1.Level3.net (4.69.145.254)  24.062 ms 
>> vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  24.051 ms
>> 19  ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144)  44.798 ms 
>> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146)  22.856 ms 
>> ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18)  22.891 ms 20  
>> vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  23.433 ms 
>> vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  25.333 ms 
>> vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.367 ms
>> 21  ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75)  23.475 ms 
>> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208)  23.476 ms 
>> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16)  64.753 ms
>> 22  vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  24.008 ms  24.043 
>> ms vlan90.csw4.Dallas1.Level3.net (4.69.145.254)  23.508 ms
>> 23  ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139)  23.505 ms 
>> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144)  23.958 ms 
>> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208)  24.058 ms
>> 24  vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.429 ms 
>> vlan90.csw4.Dallas1.Level3.net (4.69.145.254)  23.623 ms  25.547 ms
>> 25  ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82)  23.491 ms 
>> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144)  23.551 ms 
>> ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  24.078 ms
>> 26  vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  24.472 ms  24.144 
>> ms 24.080 ms
>> 27  ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18)  28.121 ms 
>> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146)  22.848 ms 
>> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208)  24.090 ms
>> 28  vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  23.509 ms  24.056 
>> ms vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  23.522 ms
>> 29  ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75)  24.161 ms  24.153

>> ms ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208)  24.130 ms 30  
>> vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.606 ms  23.482 ms 
>> vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  23.580 ms
>>
>
>