AS2828 XO / Verizon Route Flapping?

Mon Sep 7 13:26:09 UTC 2020

Greetings,

We're seeing a pretty consistent drop of traffic every 60s for about 5s
inside of AS2828 (formerly xo.net, now Verizon).

Started seeing alerts due to this at Sun, 06 Sep 2020 06:04:04 +0000.

During the 5s we're getting 'No route to host' bounces, eg:

connect to address 2a02:c0:301:0:ffff::31 and port 80: No route to host

MTR:

Keys:  Help   Display mode   Restart statistics   Order of fields   quit

                             Last 181 pings
 1. 2604:a580:2::2           .....................................................................................................................................................................................
 2. 2604:a580::1:0:0:12      .....................................................................................................................................................................................
 3. 2604:a580::1:0:0:d       .....................................................................................................................................................................................
 4. 2610:18:18c:4000::5      .....................................................................................................................................................................................
 5. 2610:18::2046            .......................................?????..........................>......................?????......................................................................?????........
 6. 2001:7f8:4:1::847:1      .......................................?????.................................................?????......................................................................?????.......?
 7. 2001:4600:0:100::29      .......................>...............?????.................................................?????......................................................................?????.......
 8. 2001:4600:0:100::3       .......................................?????.................................................?????......................................................................?????.......
 9. 2001:4600:0:100::2a      .........>.....>.....>...............>?????.........................>.......................?????......................................>.............................>.?????........
10. 2001:4600:0:200::62      ...>..................................?????........>..........>..................>>.........?????.....>.......>............................>>...>......................?????........
11. 2001:4600:9:100::2e      ..............>.......................?????............................................>....?????.............................>...................>....................?????........
12. 2a02:c0:1:1::6           ......................................?????.................................................?????......................................................................?????........
13. 2a02:c0:1::4:1           ......................................?????.....?????????????????????????????????????????????????.....??????????????????????????????????????????????????????????????????????.....???
14. 2a02:c0:1:401::25        ......................................?????.................................................?????......................................................................?????........
15. 2a02:c0:301:0:ffff::34   ......................................?????.................................................?????......................................................................?????........

Scale:  .:3 ms  1:8 ms  2:14 ms  3:26 ms  a:56 ms  b:85 ms  c:137 ms

Traceroute:

traceroute to wrigleys (2a02:c0:301:0:ffff::34), 30 hops max, 80 byte packets
 1  2604:a580:2::2 (2604:a580:2::2)  0.357 ms  0.320 ms  0.243 ms
 2  2604:a580::1:0:0:12 (2604:a580::1:0:0:12)  0.641 ms  0.302 ms  0.271 ms
 3  2604:a580::1:0:0:d (2604:a580::1:0:0:d)  0.472 ms  0.427 ms  0.313 ms
 4  2610:18:18c:4000::5 (2610:18:18c:4000::5)  0.873 ms  0.906 ms  0.855 ms
 5  cir1.london2-eng.us.xo.net (2610:18::2046)  91.539 ms  91.471 ms  91.413 ms
 6  2001:7f8:4:1::847:1 (2001:7f8:4:1::847:1)  92.361 ms  92.570 ms  92.513 ms
 7  ti3004c400-lo0-0.ti.telenor.net (2001:4600:0:100::5)  123.541 ms  123.460 ms  123.388 ms
 8  ti0001c360-lo0-0.ti.telenor.net (2001:4600:0:100::23)  123.516 ms  123.474 ms  123.419 ms
 9  ti0275c360-lo0-0.ti.telenor.net (2001:4600:0:100::2)  123.305 ms  123.332 ms  123.264 ms
10  ti0169a400-lo0-0.ti.telenor.net (2001:4600:0:200::62)  129.801 ms  129.767 ms  129.717 ms
11  2001:4600:9:100::2e (2001:4600:9:100::2e)  128.751 ms  128.703 ms cir1.london2-eng.us.xo.net (2610:18::2046)  91.658 ms !H

MTR going the other way:

regula (2a02:c0:301:0:ffff::38)                                                                                                                                                           2020-09-07T13:24:34+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit

                             Last 181 pings
 1. 2a02:c0:301:0:ffff::1    .....................................................................................................................................................................................
 2. 2a02:c0:1:401::22        .....................................................................................................................................................................................
 3. 2a02:c0:1::1             .....................................................................................................................................................................................
 4. 2a02:c0:1:1::7           .....................................................................................................................................................................................
 5. 2001:4600:9:100::2d      .....................................................................................................................................................................................
 6. 2001:4600:0:100::2       .....................................................................................................................................................................................
 7. 2001:4600:0:100::23      .....................................................................................................................................................................................
 8. 2001:4600:0:100::5       .....................................................................................................................................................................................
 9. 2001:4600::4             .....................................................................................................................................................................................
10. 2001:7f8:4:1::b0c:1      ...............................?????................................................?????......................................................................?????.................
11. 2610:18::30c4            ...............................?????................>...............................?????......................................................................?????.................
12. 2610:18:18c:4000::6      ...............................?????................................................?????......................................................................?????.................
13. 2604:a580::1:0:0:e       ..........>....................?????.>.........................>....................?????........................>............................>................?????...........>>..??
14. 2604:a580:2:0:5254:ff:fe ...............................?????................................................?????......................................................................?????................

Anyone from Verizon watching who could look into this?  Anyone else
seeing similar issues?

Feel free to contact me off-list if that'd help.  We're also trying to
reach out to our provider to see if they can help.

Thanks!

Stephen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20200907/321b6227/attachment.sig>