Packet loss on XO's network

Sun Feb 28 22:33:38 UTC 2016

Terri,

Except the initial traces show that hop 6 is clean.  If the problem really
started at hop 4 then the issue would likely show up in hop 6 (unless there
was a hashing on the return path based on the far end router's source IP
that caused just some traffic to be thrown away).

Hop 4 and 5 always have packet loss - I believe it's due to ICMPv6 rate
limiters protecting their CPU:

Frank

From: Fullenkamp, Terri L [mailto:terri.l.fullenkamp at xo.com] 
Sent: Sunday, February 28, 2016 2:59 PM
To: Frank Bulk <frnkblk at iname.com>; nanog at nanog.org
Cc: # SUP - NCC Level 1 Data <NCCLevel1Data at xo.com>
Subject: RE: Packet loss on XO's network

>From the traces provided the packet loss starts accruing at hop 4 and 5 and
that is before it gets into the XO network,  If you are a customer of XO
please contact care @ 888-575-6398 with any other questions

Trace from hop 6

tfullenk at MCR2.Minneapolis-MN <mailto:tfullenk at MCR2.Minneapolis-MN> >
traceroute enterprise.com 

traceroute6 to enterprise.com (2620:103:e049:31a7::b4) from 2610:18::30bc,
64 hops max, 12 byte packets

1  mcr1.minneapolis-mn.us.xo.net (2610:18::30b8)  29.325 ms  28.950 ms
31.327 ms

     MPLS Label=318081 CoS=0 TTL=1 S=0

     MPLS Label=2 CoS=0 TTL=1 S=1

2  2610:18::5802 (2610:18::5802)  32.680 ms  29.684 ms  32.643 ms

     MPLS Label=19268 CoS=0 TTL=255 S=0

     MPLS Label=2 CoS=0 TTL=255 S=1

3  2610:18::5804 (2610:18::5804)  30.629 ms  33.384 ms  81.985 ms

     MPLS Label=757571 CoS=0 TTL=1 S=0

     MPLS Label=2 CoS=0 TTL=3 S=1

4  2610:18::5803 (2610:18::5803)  65.305 ms  28.876 ms  39.141 ms

     MPLS Label=419427 CoS=0 TTL=1 S=0

     MPLS Label=2 CoS=0 TTL=4 S=1

5  2610:18::5201 (2610:18::5201)  36.762 ms  28.776 ms  28.718 ms

     MPLS Label=694064 CoS=0 TTL=1 S=0

     MPLS Label=2 CoS=0 TTL=5 S=1

6  2610:18::2070 (2610:18::2070)  29.487 ms  28.920 ms  42.500 ms

7  2610:18:16::32 (2610:18:16::32)  28.619 ms 2600:803:2::9 (2600:803:2::9)
28.706 ms  28.692 ms

8  2600:807:21f::22 (2600:807:21f::22)  47.997 ms  48.042 ms  50.504 ms

9  2600:807:21f::22 (2600:807:21f::22)  48.026 ms  48.084 ms  49.172 ms

Terri Fullenkamp  
NCC Specialist V / IP Data /  <http://www.xo.com/> XO Communications / 2020
Westport Center Drive, Maryland Heights, MO 63146
T: 866-966-8975 F: 314-787- 7799 E: terri.l.fullenkamp at xo.com
<mailto:terri.l.fullenkamp at xo.com> 

From: Frank Bulk [mailto:frnkblk at iname.com] 
Sent: Sunday, February 28, 2016 12:25
To: nanog at nanog.org <mailto:nanog at nanog.org> 
Subject: Packet loss on XO's network

Since ~11:08 am Central our monitoring system has been reporting
intermittent HTTPv6 timeouts to www.sprint.net <http://www.sprint.net> ,
www.qwest.com <http://www.qwest.com> , enterprise.com, and enterprise.ca.
It goes bad for a while and then clears.

Using mtr, these all have XO's network in common, and the packet loss to the
endpoint sustains over 40%.  Seems to be between 2610:18:18b:c000::11 and
mcr1.minneapolis-mn.us.xo.net [2610:18::30b8]

Frank

To enterprise.com:

Host
Loss%   Snt   Last   Avg  Best  Wrst StDev

1. router-core.mtcnet.net
0.0%   929    0.3   0.6   0.2  37.6   2.5

2. sxct.sxcy.mtcnet.net
0.0%   929    0.3   0.3   0.2  11.2   0.6

3. v6-premier.sxcy-mlx.fbnt.netins.net
0.0%   929    1.6   1.5   1.5  12.6   0.9

4. v6-ins-db4-te-0-6-0-4-219.desm.netins.net
52.4%   929    8.1   8.0   7.8  10.5   0.3

5. v6-ins-dc1-te-7-4.desm.netins.net
52.4%   929    7.9   8.0   7.6  10.9   0.5

6. 2610:18:18b:c000::11
0.0%   929   13.9  13.2  12.8  46.8   3.0

7. mcr1.minneapolis-mn.us.xo.net
50.3%   929   54.7  71.2  54.6  86.7   7.7

8. 2610:18::5802
50.8%   929   55.7  72.8  54.6  84.3   7.8

9. 2610:18::5202
51.4%   929   54.4  72.9  54.4  85.0   8.0

    2610:18::5804

10. 2610:18::2070
52.3%   928   54.4  71.5  54.4  90.7   7.9

    2610:18::5304

11. 2600:803:2::9
41.6%   928   54.5  56.8  47.8  96.7   3.4

    2610:18::5303

12. 2610:18::5200
96.0%   928   54.5  55.8  54.4  75.6   3.7

13. 2610:18::5202
96.0%   928   55.5  56.6  54.6  58.5   1.2

14. 2610:18::2070
96.0%   928   61.8  55.1  54.6  61.8   1.2

15. 2600:803:2::9
95.9%   928   45.0  45.1  45.0  48.0   0.5

16. ???

To www.sprint.net <http://www.sprint.net> :

Packets               Pings

Host
Loss%   Snt   Last   Avg  Best  Wrst StDev

1. router-core.mtcnet.net
0.0%   846    0.4   0.8   0.2  91.5   3.9

2. sxct.sxcy.mtcnet.net
0.0%   846    0.9   0.3   0.2  10.3   0.4

3. v6-premier.sxcy-mlx.fbnt.netins.net
0.0%   846    1.5   1.6   1.5  12.0   0.8

4. v6-ins-db4-te-0-6-0-4-219.desm.netins.net
55.4%   846    7.9   8.1   7.8  14.8   0.4

5. v6-ins-dc1-te-7-4.desm.netins.net
54.3%   846    7.8   8.2   7.6  19.2   1.0

6. 2610:18:18b:c000::11
0.0%   846   12.9  14.0  12.8  73.0   5.4

7. mcr1.minneapolis-mn.us.xo.net
49.8%   846   54.7  70.0  54.6 100.1   8.1

8. 2610:18::5802
47.4%   846   58.3  71.9  54.6  82.9   8.2

9. 2610:18::5202
45.0%   846   54.4  72.1  54.4  83.3   8.4

    2610:18::5804

10. 2610:18::2070
50.6%   845   54.4  70.4  54.3  88.8   8.1

    2610:18::5304

11. sl-st31-ash-te0-15-2-0.v6.sprintlink.net
43.4%   845   57.5  59.1  53.1  80.5   2.3

    2610:18::5303

12. sl-crs3-dc-be13.v6.sprintlink.net
46.7%   845   54.6  64.7  54.4  69.1   3.7

    2610:18::5200

13. sl-crs1-ffx-be2.v6.sprintlink.net
44.1%   844   55.0  71.8  55.0  79.7   5.3

    2610:18::5202

14. sl-crs1-orl-bu-1.v6.sprintlink.net
41.7%   844   54.9  75.7  54.7  96.3   7.4

    2610:18::2070

15. sl-lkdstr2-p1-0.v6.sprintlink.net
41.0%   844   46.4  73.5  45.8  81.7   7.0

    sl-st31-ash-te0-15-2-0.v6.sprintlink.net

16. www.sprint.net <http://www.sprint.net>
43.1%   844   54.5  74.6  53.5  81.6   5.0

    sl-crs3-dc-be13.v6.sprintlink.net