level3 issue in chicago (was: FYI, Level 3 issues in Dallas)

Wed Nov 19 16:22:40 UTC 2014

like david below, we've been getting reports this morning from customers 
unable to reach various web sites.

investigating a number of these reports, the one commonality is level3 
in chicago. sites are reachable from level3/cincinnati but not 
level3/chicago.

traceroutes make it one hop past our bgp peer and seem to die there.

for one particular remote web server (node1104.follett.com, and possibly 
others) traceroutes and pings from level3's looking glass (using chicago 
as the site) complete successfully, while neither does when ran from my 
gear through the same site.

On 11/19/2014 10:46 AM, garya at completeweb.net wrote:
> Also peering problems in Chicago, had to drop BGP to them until they figure it out.
> Reported it 4 hours ago, no good response yet.
> Gary
>
>> We have some customers unable to access their websites, seeing this on
>> the way to them:
>>
>>   4  ae-0-11.bar2.Tampa1.Level3.net (4.69.137.110)  0.343 ms  0.423 ms
>> 0.406 ms
>>   5  ae-12-12.ebr1.Dallas1.Level3.net (4.69.137.118)  23.940 ms  23.470
>> ms  23.426 ms
>>   6  ae-71-71.csw2.Dallas1.Level3.net (4.69.151.137)  24.026 ms
>> ae-81-81.csw3.Dallas1.Level3.net (4.69.151.149)  24.007 ms
>> ae-61-61.csw1.Dallas1.Level3.net (4.69.151.125)  23.855 ms
>>   7  ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146)  23.361 ms
>> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203)  23.329 ms
>> ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  23.936 ms
>>   8  vlan90.csw4.Dallas1.Level3.net (4.69.145.254)  23.378 ms
>> vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.387 ms
>> vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  23.963 ms
>>   9  ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  23.357 ms
>> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16)  22.779 ms
>> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203)  62.640 ms
>> 10  vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  24.179 ms  23.297 ms
>> 23.403 ms
>> 11  ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  65.617 ms
>> ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75)  23.286 ms
>> ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18)  23.553 ms
>> 12  vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.303 ms
>> vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  24.027 ms  24.028 ms
>> 13  ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139)  24.018 ms
>> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146)  23.374 ms
>> ae-4-90.edge2.Dallas1.Level3.net (4.69.145.203)  23.342 ms
>> 14  vlan90.csw4.Dallas1.Level3.net (4.69.145.254)  24.068 ms
>> vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.361 ms  23.348 ms
>> 15  ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  24.023 ms
>> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16)  22.740 ms
>> ae-4-90.edge10.Dallas1.Level3.net (4.69.145.210)  23.963 ms
>> 16  vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  23.420 ms
>> vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.400 ms
>> vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  25.317 ms
>> 17  ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82)  23.985 ms
>> ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139)  23.476 ms
>> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144)  51.762 ms
>> 18  vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  23.416 ms
>> vlan90.csw4.Dallas1.Level3.net (4.69.145.254)  24.062 ms
>> vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  24.051 ms
>> 19  ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144)  44.798 ms
>> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146)  22.856 ms
>> ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18)  22.891 ms
>> 20  vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  23.433 ms
>> vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  25.333 ms
>> vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.367 ms
>> 21  ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75)  23.475 ms
>> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208)  23.476 ms
>> ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16)  64.753 ms
>> 22  vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  24.008 ms  24.043 ms
>> vlan90.csw4.Dallas1.Level3.net (4.69.145.254)  23.508 ms
>> 23  ae-3-80.edge2.Dallas1.Level3.net (4.69.145.139)  23.505 ms
>> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144)  23.958 ms
>> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208)  24.058 ms
>> 24  vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.429 ms
>> vlan90.csw4.Dallas1.Level3.net (4.69.145.254)  23.623 ms  25.547 ms
>> 25  ae-2-70.edge10.Dallas1.Level3.net (4.69.145.82)  23.491 ms
>> ae-3-80.edge9.Dallas1.Level3.net (4.69.145.144)  23.551 ms
>> ae-1-60.edge2.Dallas1.Level3.net (4.69.145.11)  24.078 ms
>> 26  vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  24.472 ms  24.144 ms
>> 24.080 ms
>> 27  ae-1-60.edge10.Dallas1.Level3.net (4.69.145.18)  28.121 ms
>> ae-3-80.edge10.Dallas1.Level3.net (4.69.145.146)  22.848 ms
>> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208)  24.090 ms
>> 28  vlan60.csw1.Dallas1.Level3.net (4.69.145.62)  23.509 ms  24.056 ms
>> vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  23.522 ms
>> 29  ae-2-70.edge2.Dallas1.Level3.net (4.69.145.75)  24.161 ms  24.153 ms
>> ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208)  24.130 ms
>> 30  vlan70.csw2.Dallas1.Level3.net (4.69.145.126)  23.606 ms  23.482 ms
>> vlan80.csw3.Dallas1.Level3.net (4.69.145.190)  23.580 ms
>>
>
>