AW: Level3 routing issue US west coast

Jürgen Jaritsch jj at anexia.at
Fri Jul 10 13:06:45 UTC 2015


Hi Joseph,

in the meantime I have ~20 verified paths which are affected and Level3 is simply not competent enough to reroute/drop the affected path ...

FYI: my private ticket # is 9446435

###

There is also a new global ticket available:

Network Event Detail
Network Event Summary: 	Multiple devices were unreachable in Europe impacting IP services.  
Event Ticket ID: 	9446797
Market Area Affected: 	Multiple Markets in Europe  
Ticket Create Date: 	7/10/15 11:15:39 AM GMT  
Impacted For: 	12 minutes  
Event Status: 	Active  
Time Since Last Update: 	1 hour 39 minutes  

7/10/15 11:26:39 AM GMT 	The IP NOC reported multiple devices were simultaneously unreachable in Europe impacting IP services. The devices recovered without any intervention from Level 3 after an interruption lasting twelve minutes. Due to the possibility of additional impacted, the IP NOC will monitor services until the completion of multiple ongoing events in the region. A final update will be se

###

best regards

Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: jj at anexia.at 
Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601


-----Ursprüngliche Nachricht-----
Von: NANOG [mailto:nanog-bounces at nanog.org] Im Auftrag von Joseph Jenkins
Gesendet: Freitag, 10. Juli 2015 14:55
An: nanog at nanog.org
Betreff: Re: Level3 routing issue US west coast

Level3 had an issue with one of their core routers in Los Angeles last night(7pm Pacific) and early this morning(1am Pacific). Last update to my trouble ticket had the issue still being reviewed by engineering, but that a core router was dropping packets.

> On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj at anexia.at> wrote:
> 
> Hi,
> 
> does anyone else experience issues with the Level3 network at the US west coast? We see lots of broken paths like this:
> 
>                                                      Packets               Pings
> Host                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
> 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com        0.0%   231    0.6   0.5   0.2  18.1   1.2
> 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com        0.0%   231    0.5   9.9   0.3 361.1  40.1
> 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com        0.0%   230    6.5   7.7   6.3  49.7   5.3
> 4. win-b4-link.telia.net                            0.0%   230    6.6   6.8   6.4  20.2   1.5
> 5. level-ic-1573273-wien-b4.c.telia.net             0.0%   230    6.6   9.3   6.3  69.1   9.6
> 6. ae-2-70.edge1.SanJose3.Level3.net               38.4%   230  164.8 165.0 164.5 194.9   2.6
> 7. ae-2-70.edge1.SanJose3.Level3.net               45.9%   230  164.7 164.8 164.5 174.1   0.9
> 8. 4.53.208.102                                    34.3%   230  634.9 310.7 168.5 680.1 199.7
> 9. TenGE5-4.br01.seo01.pccwbtn.net                 34.1%   230  412.0 455.2 304.9 954.6 203.4
> 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net     40.6%   230  323.4 441.4 323.1 822.0 182.1
> 11. 211.115.201.92                                  38.4%   230  289.8 412.9 289.6 846.7 185.4
> 12. 61.250.89.2                                     35.8%   230  290.6 439.4 290.2 804.3 205.1
> 13. ???
> 
> Trace from NYC is also broken:
> 
>                                                          Packets               Pings
> Host                                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
> 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com            0.0%    30    0.4   4.3   0.4  57.8  13.3
> 2. nyk-b5-link.telia.net                                0.0%    30    0.3   0.4   0.3   0.9   0.1
> 3. ???
> 4. ae-3-80.edge1.SanJose3.Level3.net                   17.2%    30   71.7  73.2  71.7  98.7   5.5
> 5. ae-3-80.edge1.SanJose3.Level3.net                    0.0%    30   71.8  71.8  71.7  72.0   0.1
> 6. 4.53.208.102                                        31.0%    30  569.6 250.7  70.5 579.3 231.2
> 7. 63.218.250.73                                       31.0%    30  672.6 355.5 178.0 672.6 232.1
> 
> 
> At 10:24 UTC+2 it was even more broken:
> 
>                                                      Packets               Pings
> Host                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
> 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com        0.0%   326    0.4   0.5   0.3  39.7   2.2
> 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com        0.0%   326    0.5   6.7   0.3 198.1  26.3
> 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com        0.0%   326    6.6   7.6   6.4  43.6   4.5
> 4. win-b4-link.telia.net                            0.0%   326    6.7   7.4   6.3  43.1   3.2
> 5. level-ic-1573273-wien-b4.c.telia.net             0.0%   326    6.9   9.2   6.3  73.2  10.1
> 6. ae-1-60.edge5.LosAngeles1.Level3.net            62.6%   326  164.7 165.5 164.5 176.7   1.5
>    ae-2-70.edge1.SanJose3.Level3.net
> 7. ae-1-60.edge5.LosAngeles1.Level3.net            63.1%   326  164.8 165.8 164.6 204.2   3.9
>    ae-2-70.edge1.SanJose3.Level3.net
> 8. 205.129.5.70                                    74.2%   326  799.9 487.2 169.0 799.9 305.7
>    4.53.208.102
> 9. TenGE5-4.br01.seo01.pccwbtn.net                 77.2%   326  1359. 701.0 308.7 3716. 510.6
> 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net     75.1%   326  960.4 643.0 323.4 960.4 307.6
> 11. 211.115.201.92                                  68.9%   326  925.3 674.2 289.8 932.3 296.6
> 12. 61.250.89.2                                     72.9%   326  928.5 637.2 291.9 928.5 304.3
> 13. ???
> 
> 
> best regards
> 
> Jürgen Jaritsch
> Head of Network & Infrastructure
> 
> ANEXIA Internetdienstleistungs GmbH
> 
> Telefon: +43-5-0556-300
> Telefax: +43-5-0556-500
> 
> E-Mail: jj at anexia.at<mailto:jj at anexia.at>
> Web: http://www.anexia.at<http://www.anexia.at/>
> 
> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
> Geschäftsführer: Alexander Windbichler
> Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
> 




More information about the NANOG mailing list