Update: Packet Loss through Level 3 in Southern California?

Gregorio Focaccio gfocaccio at cari.net
Fri Dec 1 01:06:14 CST 2017


Hi All,

We have continued our investigation and data seem to show a more focused issue at the peering between Level 3 (CenturyLink) and MSN in LA.  Can someone look at our data (new data below) and see if this seems like a reasonable conclusion?

>From our network (San Diego) to an MSN Azure sever through the "Cogent | MSN Portal in LA" has no loss, but the same server has loss when going through the "Level 3 | MSN Portal in LA"

Also, we have data showing no packet loss through the same Level 3 LA (LosAngeles1) nodes to Cogent to Texas, so data seems to clear the Level 3 Los Angeles and Cogent peering.

Thanks,
Greg

Testing data:

IPERF UDP results looks good on path through Level 3 (CenturyLink) in LA (LosAngeles1) when going to a server in Texas (non-MSN Azure)

gfxc0.localdomain (0.0.0.0)                                                                                                            Thu Nov 30 16:16:02 2017
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                       Packets               Pings
Host                                                                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
1. 216.75.40.1                                                                                                       0.0%    86    0.3   7.1   0.3 172.4  27.4
2. xe-8-3-3.bar1.SanDiego1.Level3.net                                                                                0.0%    86    3.3   3.6   3.3  25.4   2.4
3. ae-3-3.ebr1.LosAngeles1.Level3.net                                                                               90.6%    86    3.6   3.6   3.6   3.7   0.0
4. ae-1-51.ear2.LosAngeles1.Level3.net                                                                              95.3%    86  7225. 7157. 7134. 7225.  45.4
5. Cogent-level3-100G.LosAngeles1.Level3.net                                                                         0.0%    86    3.8   3.9   3.6   5.7   0.2
6. be3360.ccr42.lax01.atlas.cogentco.com                                                                             0.0%    86    3.6   3.7   3.5   4.0   0.0
7. be2932.ccr32.phx01.atlas.cogentco.com                                                                             0.0%    86   12.6  12.6  12.4  13.0   0.0
8. be2930.ccr21.elp01.atlas.cogentco.com                                                                             0.0%    85   20.7  20.9  20.6  22.5   0.2
9. be2928.ccr42.iah01.atlas.cogentco.com                                                                             0.0%    85   36.8  36.6  36.4  37.1   0.0
10. be2443.ccr32.dfw01.atlas.cogentco.com                                                                             0.0%    85   41.6  41.7  41.5  42.6   0.0
11. be2939.rcr21.dfw04.atlas.cogentco.com                                                                             0.0%    85   42.6  42.7  42.3  44.0   0.2
12. te0-0-1-1.nr12.b028597-0.dfw04.atlas.cogentco.com                                                                 0.0%    85   43.2  43.2  43.1  43.5   0.0
13. 38.122.200.202                                                                                                    0.0%    85   42.4  42.4  42.3  42.7   0.0
14. 138.128.243.167                                                                                                   0.0%    85   42.6  42.7  42.4  48.2   0.7

[root at gfxc0 ~]# iperf3 -uZVc 138.128.243.167 -b10m -t15 --get-server-output
iperf 3.1.3
Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64
Time: Fri, 01 Dec 2017 00:17:19 GMT
Connecting to host 138.128.243.167, port 5201
      Cookie: gfxc0.localdomain.1512087439.341597.
[  4] local 216.75.40.2 port 42277 connected to 138.128.243.167 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  1.09 MBytes  9.11 Mbits/sec  139
[  4]   1.00-2.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   2.00-3.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   3.00-4.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   4.00-5.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   5.00-6.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   6.00-7.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   7.00-8.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   8.00-9.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   9.00-10.00  sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]  10.00-11.00  sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]  11.00-12.00  sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]  12.00-13.00  sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]  13.00-14.00  sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]  14.00-15.00  sec  1.19 MBytes  9.96 Mbits/sec  152
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-15.00  sec  17.8 MBytes  9.94 Mbits/sec  0.057 ms  0/2274 (0%)
[  4] Sent 2274 datagrams
CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.1% (0.1%u/0.0%s)

Server output:
Accepted connection from 216.75.40.2, port 43300
[  5] local 138.128.243.167 port 5201 connected to 216.75.40.2 port 42277
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  1.08 MBytes  9.04 Mbits/sec  0.068 ms  0/138 (0%)
[  5]   1.00-2.00   sec  1.20 MBytes  10.0 Mbits/sec  0.060 ms  0/153 (0%)
[  5]   2.00-3.00   sec  1.19 MBytes  9.96 Mbits/sec  0.058 ms  0/152 (0%)
[  5]   3.00-4.00   sec  1.20 MBytes  10.0 Mbits/sec  0.058 ms  0/153 (0%)
[  5]   4.00-5.00   sec  1.19 MBytes  9.96 Mbits/sec  0.061 ms  0/152 (0%)
[  5]   5.00-6.00   sec  1.20 MBytes  10.0 Mbits/sec  0.121 ms  0/153 (0%)
[  5]   6.00-7.00   sec  1.19 MBytes  9.96 Mbits/sec  0.057 ms  0/152 (0%)
[  5]   7.00-8.00   sec  1.20 MBytes  10.0 Mbits/sec  0.056 ms  0/153 (0%)
[  5]   8.00-9.00   sec  1.20 MBytes  10.0 Mbits/sec  0.106 ms  0/153 (0%)
[  5]   9.00-10.00  sec  1.19 MBytes  9.96 Mbits/sec  0.083 ms  0/152 (0%)
[  5]  10.00-11.00  sec  1.20 MBytes  9.99 Mbits/sec  0.069 ms  0/153 (0%)
[  5]  11.00-12.00  sec  1.19 MBytes  10.0 Mbits/sec  0.054 ms  0/152 (0%)
[  5]  12.00-13.00  sec  1.20 MBytes  10.0 Mbits/sec  0.061 ms  0/153 (0%)
[  5]  13.00-14.00  sec  1.20 MBytes  10.0 Mbits/sec  0.063 ms  0/153 (0%)
[  5]  14.00-15.00  sec  1.19 MBytes  9.96 Mbits/sec  0.057 ms  0/152 (0%)


iperf Done.
[root at gfxc0 ~]#



Results still look NOT OK when going through same Level 3 (CenturyLink) LA (LosAngeles1) nodes ebr1 and ear2 to an MSN Azure server

                                                                     My traceroute  [v0.87]
gfxc0.localdomain (0.0.0.0)                                                                                                            Thu Nov 30 16:24:54 2017
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                       Packets               Pings
Host                                                                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
1. 216.75.40.1                                                                                                       0.0%    95    0.3  10.9   0.2 199.5  38.9
2. xe-8-3-3.bar1.SanDiego1.Level3.net                                                                                0.0%    95    3.3   3.7   3.2  20.6   1.9
3. ae-3-3.ebr1.LosAngeles1.Level3.net                                                                               98.9%    95    4.0   4.0   4.0   4.0   0.0
4. ae-1-51.ear2.LosAngeles1.Level3.net                                                                              96.8%    95  7101. 7046. 7004. 7101.  50.0
5. Microsoft-level3-20G.LosAngeles1.Level3.net                                                                       0.0%    95   15.2  17.7   9.3  20.8   2.4
6. be-63-0.ibr01.lax03.ntwk.msn.net                                                                                  1.1%    95   29.9  30.1  22.7  34.0   2.6
7. be-4-0.ibr01.by2.ntwk.msn.net                                                                                     1.1%    95   31.6  30.2  20.4  33.9   2.6
8. 104.44.7.198                                                                                                      0.0%    94   29.8  29.9  20.3  33.7   2.7
9. ae102-0.icr02.by21.ntwk.msn.net                                                                                   0.0%    94   35.7  29.2  20.5  41.4   2.7
10. ???

[root at gfxc0 ~]# iperf3 -uZVc 13.91.55.110 -b10m -t15 --get-server-output
iperf 3.1.3
Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64
Time: Fri, 01 Dec 2017 00:25:50 GMT
Connecting to host 13.91.55.110, port 5201
      Cookie: gfxc0.localdomain.1512087950.883346.
[  4] local 216.75.40.2 port 34611 connected to 13.91.55.110 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  1.09 MBytes  9.11 Mbits/sec  139
[  4]   1.00-2.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   2.00-3.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   3.00-4.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   4.00-5.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   5.00-6.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   6.00-7.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   7.00-8.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   8.00-9.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   9.00-10.00  sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]  10.00-11.00  sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]  11.00-12.00  sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]  12.00-13.00  sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]  13.00-14.00  sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]  14.00-15.00  sec  1.19 MBytes  9.96 Mbits/sec  152
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-15.00  sec  17.8 MBytes  9.94 Mbits/sec  0.141 ms  102/2274 (4.5%)
[  4] Sent 2274 datagrams
CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.1% (0.0%u/0.1%s)

Server output:
-----------------------------------------------------------
Accepted connection from 216.75.40.2, port 44374
[  5] local 10.0.0.4 port 5201 connected to 216.75.40.2 port 34611
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  1.08 MBytes  9.04 Mbits/sec  0.174 ms  0/138 (0%)
[  5]   1.00-2.00   sec  1.20 MBytes  10.0 Mbits/sec  0.203 ms  0/153 (0%)
[  5]   2.00-3.00   sec  1.17 MBytes  9.83 Mbits/sec  0.196 ms  2/152 (1.3%)
[  5]   3.00-4.00   sec  1.20 MBytes  10.0 Mbits/sec  0.110 ms  0/153 (0%)
[  5]   4.00-5.00   sec  1.16 MBytes  9.69 Mbits/sec  0.120 ms  4/152 (2.6%)
[  5]   5.00-6.00   sec  1.11 MBytes  9.31 Mbits/sec  0.106 ms  11/153 (7.2%)
[  5]   6.00-7.00   sec  1.13 MBytes  9.50 Mbits/sec  0.313 ms  7/152 (4.6%)
[  5]   7.00-8.00   sec   992 KBytes  8.13 Mbits/sec  0.163 ms  29/153 (19%)
[  5]   8.00-9.00   sec  1.06 MBytes  8.91 Mbits/sec  0.129 ms  17/153 (11%)
[  5]   9.00-10.00  sec  1.12 MBytes  9.37 Mbits/sec  0.131 ms  9/152 (5.9%)
[  5]  10.00-11.00  sec  1.19 MBytes  9.96 Mbits/sec  0.155 ms  1/153 (0.65%)
[  5]  11.00-12.00  sec  1.12 MBytes  9.44 Mbits/sec  0.336 ms  5/149 (3.4%)
[  5]  12.00-13.00  sec  1.09 MBytes  9.18 Mbits/sec  0.166 ms  16/156 (10%)
[  5]  13.00-14.00  sec  1.20 MBytes  10.0 Mbits/sec  0.109 ms  0/153 (0%)
[  5]  14.00-15.00  sec  1.18 MBytes  9.90 Mbits/sec  0.141 ms  1/152 (0.66%)


iperf Done.
[root at gfxc0 ~]#

"tcp slowdown"

[root at gfxc0 ~]# iperf3 -ZVc 13.91.55.110 -b20m -t15 --get-server-output
iperf 3.1.3
Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64
Time: Fri, 01 Dec 2017 00:30:16 GMT
Connecting to host 13.91.55.110, port 5201
      Cookie: gfxc0.localdomain.1512088216.384940.
      TCP MSS: 1428 (default)
[  4] local 216.75.40.2 port 44388 connected to 13.91.55.110 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  2.33 MBytes  19.6 Mbits/sec   13    411 KBytes
[  4]   1.00-2.00   sec  2.38 MBytes  19.9 Mbits/sec   13    201 KBytes
[  4]   2.00-3.00   sec  2.38 MBytes  19.9 Mbits/sec    3    105 KBytes
[  4]   3.00-4.00   sec  2.45 MBytes  20.6 Mbits/sec    3   64.1 KBytes
[  4]   4.00-5.00   sec  1.91 MBytes  16.1 Mbits/sec   10   44.6 KBytes
[  4]   5.00-6.00   sec  1.26 MBytes  10.5 Mbits/sec    8   36.3 KBytes
[  4]   6.00-7.00   sec  1.26 MBytes  10.6 Mbits/sec    6   20.9 KBytes
[  4]   7.00-8.00   sec   803 KBytes  6.58 Mbits/sec    3   22.3 KBytes
[  4]   8.00-9.00   sec   697 KBytes  5.71 Mbits/sec    3   25.1 KBytes
[  4]   9.00-10.00  sec  1.27 MBytes  10.6 Mbits/sec    0   48.8 KBytes
[  4]  10.00-11.00  sec  1.53 MBytes  12.9 Mbits/sec    4   37.7 KBytes
[  4]  11.00-12.00  sec  1.21 MBytes  10.1 Mbits/sec    5   33.5 KBytes
[  4]  12.00-13.00  sec  1.35 MBytes  11.3 Mbits/sec    1   40.4 KBytes
[  4]  13.00-14.00  sec  1.33 MBytes  11.1 Mbits/sec    1   51.6 KBytes
[  4]  14.00-15.00  sec  2.28 MBytes  19.1 Mbits/sec    2   53.0 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-15.00  sec  24.4 MBytes  13.6 Mbits/sec   75             sender
[  4]   0.00-15.00  sec  24.1 MBytes  13.5 Mbits/sec                  receiver
CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.2% (0.0%u/0.2%s)

Server output:
-----------------------------------------------------------
Accepted connection from 216.75.40.2, port 44386
[  5] local 10.0.0.4 port 5201 connected to 216.75.40.2 port 44388
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  2.21 MBytes  18.5 Mbits/sec
[  5]   1.00-2.00   sec  2.38 MBytes  19.9 Mbits/sec
[  5]   2.00-3.00   sec  2.38 MBytes  19.9 Mbits/sec
[  5]   3.00-4.00   sec  2.23 MBytes  18.7 Mbits/sec
[  5]   4.00-5.00   sec  1.93 MBytes  16.2 Mbits/sec
[  5]   5.00-6.00   sec  1.26 MBytes  10.6 Mbits/sec
[  5]   6.00-7.00   sec  1.27 MBytes  10.6 Mbits/sec
[  5]   7.00-8.00   sec   800 KBytes  6.56 Mbits/sec
[  5]   8.00-9.00   sec   692 KBytes  5.67 Mbits/sec
[  5]   9.00-10.00  sec  1.27 MBytes  10.7 Mbits/sec
[  5]  10.00-11.00  sec  1.53 MBytes  12.9 Mbits/sec
[  5]  11.00-12.00  sec  1.21 MBytes  10.2 Mbits/sec
[  5]  12.00-13.00  sec  1.35 MBytes  11.3 Mbits/sec
[  5]  13.00-14.00  sec  1.32 MBytes  11.1 Mbits/sec
[  5]  14.00-15.00  sec  2.26 MBytes  18.9 Mbits/sec


iperf Done.
[root at gfxc0 ~]#



...but then results look OK to same Azure server when traffic forced to get to MSN via Cogent

                                                                     My traceroute  [v0.87]
gfxc0.localdomain (0.0.0.0)                                                                                                            Thu Nov 30 16:20:50 2017
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                       Packets               Pings
Host                                                                                                                Loss%   Snt   Last   Avg  Best  Wrst StDev
1. 216.75.40.1                                                                                                       0.0%    19    0.3   9.9   0.3 124.0  30.6
2. 216.98.153.90                                                                                                     0.0%    19   63.8  10.0   0.3 117.3  29.8
3. te0-0-2-2.agr11.san01.atlas.cogentco.com                                                                          0.0%    19    1.6   0.9   0.7   1.6   0.0
4. te0-0-0-3.rcr12.san01.atlas.cogentco.com                                                                          0.0%    19    1.0   1.0   0.8   1.2   0.0
5. be2937.rcr12.sna02.atlas.cogentco.com                                                                             0.0%    19    3.2   3.3   3.2   3.5   0.0
6. be2463.agr22.lax01.atlas.cogentco.com                                                                             0.0%    19    4.0   4.3   4.0   6.7   0.5
7. be2586.ccr41.lax01.atlas.cogentco.com                                                                             0.0%    19    4.1   4.1   4.0   4.4   0.0
8. be3271.ccr41.lax04.atlas.cogentco.com                                                                             0.0%    19    4.9   4.7   4.0   9.2   1.0
9. 38.142.33.250                                                                                                     0.0%    19    4.0   3.9   3.8   4.0   0.0
10. be-63-0.ibr01.lax03.ntwk.msn.net                                                                                  0.0%    19   17.0  16.6  15.6  17.4   0.2
11. be-4-0.ibr01.by2.ntwk.msn.net                                                                                     0.0%    19   17.2  16.9  15.9  17.9   0.0
12. 104.44.7.198                                                                                                      0.0%    19   16.1  17.0  16.1  17.8   0.2
13. ae103-0.icr02.by4.ntwk.msn.net                                                                                    0.0%    19   15.2  15.3  15.1  17.3   0.4
14. ???

[root at gfxc0 ~]# iperf3 -uZVc 13.91.55.110 -b10m -t15 --get-server-output
iperf 3.1.3
Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64
Time: Fri, 01 Dec 2017 00:21:51 GMT
Connecting to host 13.91.55.110, port 5201
      Cookie: gfxc0.localdomain.1512087711.332144.
[  4] local 216.75.40.2 port 43882 connected to 13.91.55.110 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  1.09 MBytes  9.11 Mbits/sec  139
[  4]   1.00-2.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   2.00-3.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   3.00-4.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   4.00-5.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   5.00-6.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   6.00-7.00   sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]   7.00-8.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   8.00-9.00   sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]   9.00-10.00  sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]  10.00-11.00  sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]  11.00-12.00  sec  1.19 MBytes  9.96 Mbits/sec  152
[  4]  12.00-13.00  sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]  13.00-14.00  sec  1.20 MBytes  10.0 Mbits/sec  153
[  4]  14.00-15.00  sec  1.19 MBytes  9.96 Mbits/sec  152
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-15.00  sec  17.8 MBytes  9.94 Mbits/sec  0.046 ms  0/2274 (0%)
[  4] Sent 2274 datagrams
CPU Utilization: local/sender 0.7% (0.1%u/0.7%s), remote/receiver 0.0% (0.0%u/0.0%s)

Server output:
-----------------------------------------------------------
Accepted connection from 216.75.40.2, port 44356
[  5] local 10.0.0.4 port 5201 connected to 216.75.40.2 port 43882
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  1.08 MBytes  9.04 Mbits/sec  0.036 ms  0/138 (0%)
[  5]   1.00-2.00   sec  1.20 MBytes  10.0 Mbits/sec  0.057 ms  0/153 (0%)
[  5]   2.00-3.00   sec  1.19 MBytes  9.96 Mbits/sec  0.047 ms  0/152 (0%)
[  5]   3.00-4.00   sec  1.20 MBytes  10.0 Mbits/sec  0.049 ms  0/153 (0%)
[  5]   4.00-5.00   sec  1.19 MBytes  9.96 Mbits/sec  0.064 ms  0/152 (0%)
[  5]   5.00-6.00   sec  1.20 MBytes  10.0 Mbits/sec  0.059 ms  0/153 (0%)
[  5]   6.00-7.00   sec  1.19 MBytes  9.96 Mbits/sec  0.056 ms  0/152 (0%)
[  5]   7.00-8.00   sec  1.20 MBytes  10.0 Mbits/sec  0.055 ms  0/153 (0%)
[  5]   8.00-9.00   sec  1.20 MBytes  10.0 Mbits/sec  0.061 ms  0/153 (0%)
[  5]   9.00-10.00  sec  1.19 MBytes  9.96 Mbits/sec  0.051 ms  0/152 (0%)
[  5]  10.00-11.00  sec  1.20 MBytes  10.0 Mbits/sec  0.057 ms  0/153 (0%)
[  5]  11.00-12.00  sec  1.19 MBytes  9.96 Mbits/sec  0.053 ms  0/152 (0%)
[  5]  12.00-13.00  sec  1.20 MBytes  10.0 Mbits/sec  0.055 ms  0/153 (0%)
[  5]  13.00-14.00  sec  1.20 MBytes  10.0 Mbits/sec  0.063 ms  0/153 (0%)
[  5]  14.00-15.00  sec  1.19 MBytes  9.96 Mbits/sec  0.046 ms  0/152 (0%)


iperf Done.
[root at gfxc0 ~]#

From: Gregorio Focaccio
Sent: Tuesday, November 28, 2017 2:05 PM
To: 'nanog at nanog.org' <nanog at nanog.org>
Subject: Packet Loss through Level 3 in Southern California?

Hi All,

We are a multi-datacenter MSP in San Diego that also offers Colo and Cloud hosting.  We are multi-homed with Level 3 and Cogent.

A client reported newly slow FTP transfers, so we started a network investigation.

Our data (see below) seem to show packet loss through Level 3 with associated slow TCP based data transfers.

Is anyone else seeing packet loss and consequent slow TCP based transfers when going through Level 3 in Southern California?

Thanks,
Greg Focaccio
CARI.net<https://cari.net/>


Testing Data:

********** Internal tests - OK **********
FTP transfers from client server to another server 3 hops away in our adjacent datacenter was normal


********** Level 3 data - packet loss **********
External tests via Level 3  - 12% Packet Loss - see data below
  UDP IPERF testing from our data center (through Level 3 and Microsoft - trace below) to an Azure server showed repeatable packet loss
   TCP based testing - such as FTP or SCP transfers - the rate was very slow about 4Mbps

[root at raynor ~]# iperf3 -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[  4] local 71.6.220.101 port 55684 connected to 40.80.156.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1001 KBytes  8.20 Mbits/sec   34   9.76 KBytes
[  4]   1.00-2.00   sec   503 KBytes  4.12 Mbits/sec   12   11.2 KBytes
[  4]   2.00-3.00   sec   502 KBytes  4.11 Mbits/sec    3   25.1 KBytes
[  4]   3.00-4.00   sec   502 KBytes  4.11 Mbits/sec   12   11.2 KBytes
[  4]   4.00-5.00   sec   377 KBytes  3.08 Mbits/sec    9   15.3 KBytes
[  4]   5.00-6.00   sec   502 KBytes  4.11 Mbits/sec   10   19.5 KBytes
[  4]   6.00-7.00   sec   377 KBytes  3.09 Mbits/sec   13   6.97 KBytes
[  4]   7.00-8.00   sec   251 KBytes  2.06 Mbits/sec    9   5.58 KBytes
[  4]   8.00-9.00   sec   251 KBytes  2.06 Mbits/sec    6   9.76 KBytes
[  4]   9.00-10.00  sec   251 KBytes  2.06 Mbits/sec   10   12.6 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  4.41 MBytes  3.70 Mbits/sec  118             sender
[  4]   0.00-10.00  sec  4.24 MBytes  3.56 Mbits/sec                  receiver
iperf Done.

[root at raynor ~]# iperf3 -u -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[  4] local 71.6.220.101 port 39221 connected to 40.80.156.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  85.0 MBytes   712 Mbits/sec  62402
[  4]   1.00-2.00   sec  94.0 MBytes   789 Mbits/sec  69043
[  4]   2.00-3.00   sec  92.6 MBytes   777 Mbits/sec  68030
[  4]   3.00-4.00   sec  76.5 MBytes   641 Mbits/sec  56153
[  4]   4.00-5.00   sec  94.9 MBytes   796 Mbits/sec  69662
[  4]   5.00-6.00   sec  97.7 MBytes   819 Mbits/sec  71713
[  4]   6.00-7.00   sec  98.5 MBytes   826 Mbits/sec  72347
[  4]   7.00-8.00   sec  92.7 MBytes   778 Mbits/sec  68085
[  4]   8.00-9.00   sec  91.3 MBytes   765 Mbits/sec  67045
[  4]   9.00-10.00  sec  59.3 MBytes   498 Mbits/sec  43551
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec   883 MBytes   740 Mbits/sec  0.014 ms  75647/647955 (12%)
[  4] Sent 647955 datagrams
iperf Done.

[root at raynor ~]# traceroute 40.80.156.2
traceroute to 40.80.156.2 (40.80.156.2), 30 hops max, 60 byte packets
1  gateway (71.6.220.97)  6.170 ms  7.827 ms  10.718 ms
2  209.126.134.65 (209.126.134.65)  0.426 ms  0.420 ms  0.543 ms
3  216.98.153.21 (216.98.153.21)  159.936 ms  160.215 ms  160.209 ms
4  gi1-2.gw65-50.c5.sdcix.net (216.98.153.89)  170.977 ms  170.972 ms  170.964 ms
5  xe-8-3-3.bar1.SanDiego1.Level3.net (4.16.105.93)  0.547 ms  0.586 ms  0.533 ms
6  * * *
7  * * *
8  Microsoft-level3-20G.LosAngeles1.Level3.net (4.68.111.122)  17.296 ms  18.026 ms  20.200 ms
9  be-61-0.ibr01.lax03.ntwk.msn.net (104.44.8.104)  30.108 ms  32.300 ms  32.283 ms
10  be-4-0.ibr01.by2.ntwk.msn.net (104.44.4.3)  30.364 ms  30.961 ms  28.960 ms
11  104.44.7.198 (104.44.7.198)  31.870 ms  30.120 ms  29.636 ms
12  ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194)  27.828 ms ae101-0.icr01.by4.ntwk.msn.net (104.44.11.193)  29.611 ms ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194)  27.710 ms
13  * * *

[root at raynor ~]# scp ubuntu-14.04.5-desktop-amd64.iso carinet at 40.80.156.2:/home/carinet/ubuntunew11.iso<mailto:carinet at 40.80.156.2:/home/carinet/ubuntunew11.iso>
ubuntu-14.04.5-desktop-amd64.iso                     100% 1053MB 730.8KB/s   24:35

LEVEL3 - summary
CARIcloud to Azure
2 Mbits/s TCP iperf
800 Mbits/s UDP iperf
24m and 35s 1053 MB upload to Azure through SCP


********** Cogent data - OK **********
External tests via Cogent - OK - No significant loss - see IPERF and trace data below

[root at raynor ~]# iperf3 -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[  4] local 71.6.220.101 port 45076 connected to 40.80.156.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   149 MBytes  1.25 Gbits/sec    0   4.58 MBytes
[  4]   1.00-2.00   sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
[  4]   2.00-3.00   sec   178 MBytes  1.50 Gbits/sec    0   4.58 MBytes
[  4]   3.00-4.00   sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
[  4]   4.00-5.00   sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
[  4]   5.00-6.00   sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
[  4]   6.00-7.00   sec   179 MBytes  1.50 Gbits/sec    0   4.58 MBytes
[  4]   7.00-8.00   sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
[  4]   8.00-9.00   sec   181 MBytes  1.52 Gbits/sec    0   4.58 MBytes
[  4]   9.00-10.00  sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.73 GBytes  1.48 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  1.73 GBytes  1.48 Gbits/sec                  receiver
iperf Done.

[root at raynor ~]# iperf3 -u -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[  4] local 71.6.220.101 port 39130 connected to 40.80.156.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   101 MBytes   844 Mbits/sec  73869
[  4]   1.00-2.00   sec   103 MBytes   861 Mbits/sec  75344
[  4]   2.00-3.00   sec  99.0 MBytes   830 Mbits/sec  72700
[  4]   3.00-4.00   sec  97.9 MBytes   821 Mbits/sec  71885
[  4]   4.00-5.00   sec  98.0 MBytes   822 Mbits/sec  71979
[  4]   5.00-6.00   sec   100 MBytes   841 Mbits/sec  73640
[  4]   6.00-7.00   sec  69.8 MBytes   585 Mbits/sec  51237
[  4]   7.00-8.00   sec  94.9 MBytes   796 Mbits/sec  69650
[  4]   8.00-9.00   sec   102 MBytes   855 Mbits/sec  74804
[  4]   9.00-10.00  sec   100 MBytes   839 Mbits/sec  73407
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec   965 MBytes   809 Mbits/sec  0.004 ms  3194/708505 (0.45%)
[  4] Sent 708505 datagrams
iperf Done.

[root at raynor ~]# traceroute 40.80.156.2
traceroute to 40.80.156.2 (40.80.156.2), 30 hops max, 60 byte packets
1  gateway (71.6.220.97)  9.892 ms  10.883 ms  15.574 ms
2  209.126.134.65 (209.126.134.65)  1.541 ms  1.966 ms  1.959 ms
3  216.98.153.21 (216.98.153.21)  0.523 ms  0.517 ms  0.511 ms
4  216.98.153.114 (216.98.153.114)  0.589 ms  0.480 ms  0.687 ms
5  te0-0-1-0.nr11.b005949-0.san01.atlas.cogentco.com (38.104.122.61)  1.045 ms  1.039 ms  1.072 ms
6  te0-0-0-3.agr11.san01.atlas.cogentco.com (154.24.32.53)  1.023 ms  0.823 ms te0-0-0-3.agr12.san01.atlas.cogentco.com (154.24.32.65)  1.359 ms
7  te0-0-1-3.rcr11.san01.atlas.cogentco.com (154.24.31.21)  0.919 ms te0-0-1-3.rcr12.san01.atlas.cogentco.com (154.24.31.37)  0.834 ms  0.874 ms
8  be2936.rcr11.sna02.atlas.cogentco.com (154.54.45.166)  3.947 ms  3.726 ms be2937.rcr12.sna02.atlas.cogentco.com (154.54.45.173)  3.221 ms
9  be2463.agr22.lax01.atlas.cogentco.com (154.54.80.61)  4.095 ms  3.960 ms  4.087 ms
10  be2584.ccr41.lax01.atlas.cogentco.com (154.54.29.33)  4.364 ms  3.961 ms be2586.ccr41.lax01.atlas.cogentco.com (154.54.29.245)  4.290 ms
11  be3360.ccr41.lax04.atlas.cogentco.com (154.54.25.150)  4.283 ms be3271.ccr41.lax04.atlas.cogentco.com (154.54.42.102)  4.020 ms  4.003 ms
12  38.142.33.250 (38.142.33.250)  3.828 ms  3.818 ms  3.481 ms
13  be-61-0.ibr01.lax03.ntwk.msn.net (104.44.8.104)  15.565 ms  15.291 ms  15.173 ms
14  be-4-0.ibr01.by2.ntwk.msn.net (104.44.4.3)  14.290 ms  13.903 ms  16.168 ms
15  104.44.7.198 (104.44.7.198)  15.445 ms  14.098 ms  13.866 ms
16  ae102-0.icr02.by21.ntwk.msn.net (104.44.11.198)  13.752 ms ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194)  14.378 ms ae101-0.icr01.by4.ntwk.msn.net (104.44.11.193)  13.813 ms

[root at raynor ~]# scp ubuntu-14.04.5-desktop-amd64.iso carinet at 40.80.156.2:/home/carinet/ubuntunew10.iso<mailto:carinet at 40.80.156.2:/home/carinet/ubuntunew10.iso>
ubuntu-14.04.5-desktop-amd64.iso                              100% 1053MB 111.5MB/s   00:09

COGENT - summary
CARIcloud to Azure
1.5 Gbits/s TCP iperf
850 Mbits/s UDP iperf
9s 1053 MB upload to Azure through SCP



More information about the NANOG mailing list