Update: Packet Loss through Level 3 in Southern California?
Gregorio Focaccio
gfocaccio at cari.net
Fri Dec 1 01:06:14 UTC 2017
Hi All,
We have continued our investigation and data seem to show a more focused issue at the peering between Level 3 (CenturyLink) and MSN in LA. Can someone look at our data (new data below) and see if this seems like a reasonable conclusion?
>From our network (San Diego) to an MSN Azure sever through the "Cogent | MSN Portal in LA" has no loss, but the same server has loss when going through the "Level 3 | MSN Portal in LA"
Also, we have data showing no packet loss through the same Level 3 LA (LosAngeles1) nodes to Cogent to Texas, so data seems to clear the Level 3 Los Angeles and Cogent peering.
Thanks,
Greg
Testing data:
IPERF UDP results looks good on path through Level 3 (CenturyLink) in LA (LosAngeles1) when going to a server in Texas (non-MSN Azure)
gfxc0.localdomain (0.0.0.0) Thu Nov 30 16:16:02 2017
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 216.75.40.1 0.0% 86 0.3 7.1 0.3 172.4 27.4
2. xe-8-3-3.bar1.SanDiego1.Level3.net 0.0% 86 3.3 3.6 3.3 25.4 2.4
3. ae-3-3.ebr1.LosAngeles1.Level3.net 90.6% 86 3.6 3.6 3.6 3.7 0.0
4. ae-1-51.ear2.LosAngeles1.Level3.net 95.3% 86 7225. 7157. 7134. 7225. 45.4
5. Cogent-level3-100G.LosAngeles1.Level3.net 0.0% 86 3.8 3.9 3.6 5.7 0.2
6. be3360.ccr42.lax01.atlas.cogentco.com 0.0% 86 3.6 3.7 3.5 4.0 0.0
7. be2932.ccr32.phx01.atlas.cogentco.com 0.0% 86 12.6 12.6 12.4 13.0 0.0
8. be2930.ccr21.elp01.atlas.cogentco.com 0.0% 85 20.7 20.9 20.6 22.5 0.2
9. be2928.ccr42.iah01.atlas.cogentco.com 0.0% 85 36.8 36.6 36.4 37.1 0.0
10. be2443.ccr32.dfw01.atlas.cogentco.com 0.0% 85 41.6 41.7 41.5 42.6 0.0
11. be2939.rcr21.dfw04.atlas.cogentco.com 0.0% 85 42.6 42.7 42.3 44.0 0.2
12. te0-0-1-1.nr12.b028597-0.dfw04.atlas.cogentco.com 0.0% 85 43.2 43.2 43.1 43.5 0.0
13. 38.122.200.202 0.0% 85 42.4 42.4 42.3 42.7 0.0
14. 138.128.243.167 0.0% 85 42.6 42.7 42.4 48.2 0.7
[root at gfxc0 ~]# iperf3 -uZVc 138.128.243.167 -b10m -t15 --get-server-output
iperf 3.1.3
Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64
Time: Fri, 01 Dec 2017 00:17:19 GMT
Connecting to host 138.128.243.167, port 5201
Cookie: gfxc0.localdomain.1512087439.341597.
[ 4] local 216.75.40.2 port 42277 connected to 138.128.243.167 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 1.09 MBytes 9.11 Mbits/sec 139
[ 4] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 10.00-11.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 11.00-12.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 152
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-15.00 sec 17.8 MBytes 9.94 Mbits/sec 0.057 ms 0/2274 (0%)
[ 4] Sent 2274 datagrams
CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.1% (0.1%u/0.0%s)
Server output:
Accepted connection from 216.75.40.2, port 43300
[ 5] local 138.128.243.167 port 5201 connected to 216.75.40.2 port 42277
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 1.08 MBytes 9.04 Mbits/sec 0.068 ms 0/138 (0%)
[ 5] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 0.060 ms 0/153 (0%)
[ 5] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 0.058 ms 0/152 (0%)
[ 5] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 0.058 ms 0/153 (0%)
[ 5] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 0.061 ms 0/152 (0%)
[ 5] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 0.121 ms 0/153 (0%)
[ 5] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 0.057 ms 0/152 (0%)
[ 5] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 0.056 ms 0/153 (0%)
[ 5] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 0.106 ms 0/153 (0%)
[ 5] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 0.083 ms 0/152 (0%)
[ 5] 10.00-11.00 sec 1.20 MBytes 9.99 Mbits/sec 0.069 ms 0/153 (0%)
[ 5] 11.00-12.00 sec 1.19 MBytes 10.0 Mbits/sec 0.054 ms 0/152 (0%)
[ 5] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 0.061 ms 0/153 (0%)
[ 5] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 0.063 ms 0/153 (0%)
[ 5] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 0.057 ms 0/152 (0%)
iperf Done.
[root at gfxc0 ~]#
Results still look NOT OK when going through same Level 3 (CenturyLink) LA (LosAngeles1) nodes ebr1 and ear2 to an MSN Azure server
My traceroute [v0.87]
gfxc0.localdomain (0.0.0.0) Thu Nov 30 16:24:54 2017
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 216.75.40.1 0.0% 95 0.3 10.9 0.2 199.5 38.9
2. xe-8-3-3.bar1.SanDiego1.Level3.net 0.0% 95 3.3 3.7 3.2 20.6 1.9
3. ae-3-3.ebr1.LosAngeles1.Level3.net 98.9% 95 4.0 4.0 4.0 4.0 0.0
4. ae-1-51.ear2.LosAngeles1.Level3.net 96.8% 95 7101. 7046. 7004. 7101. 50.0
5. Microsoft-level3-20G.LosAngeles1.Level3.net 0.0% 95 15.2 17.7 9.3 20.8 2.4
6. be-63-0.ibr01.lax03.ntwk.msn.net 1.1% 95 29.9 30.1 22.7 34.0 2.6
7. be-4-0.ibr01.by2.ntwk.msn.net 1.1% 95 31.6 30.2 20.4 33.9 2.6
8. 104.44.7.198 0.0% 94 29.8 29.9 20.3 33.7 2.7
9. ae102-0.icr02.by21.ntwk.msn.net 0.0% 94 35.7 29.2 20.5 41.4 2.7
10. ???
[root at gfxc0 ~]# iperf3 -uZVc 13.91.55.110 -b10m -t15 --get-server-output
iperf 3.1.3
Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64
Time: Fri, 01 Dec 2017 00:25:50 GMT
Connecting to host 13.91.55.110, port 5201
Cookie: gfxc0.localdomain.1512087950.883346.
[ 4] local 216.75.40.2 port 34611 connected to 13.91.55.110 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 1.09 MBytes 9.11 Mbits/sec 139
[ 4] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 10.00-11.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 11.00-12.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 152
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-15.00 sec 17.8 MBytes 9.94 Mbits/sec 0.141 ms 102/2274 (4.5%)
[ 4] Sent 2274 datagrams
CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.1% (0.0%u/0.1%s)
Server output:
-----------------------------------------------------------
Accepted connection from 216.75.40.2, port 44374
[ 5] local 10.0.0.4 port 5201 connected to 216.75.40.2 port 34611
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 1.08 MBytes 9.04 Mbits/sec 0.174 ms 0/138 (0%)
[ 5] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 0.203 ms 0/153 (0%)
[ 5] 2.00-3.00 sec 1.17 MBytes 9.83 Mbits/sec 0.196 ms 2/152 (1.3%)
[ 5] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 0.110 ms 0/153 (0%)
[ 5] 4.00-5.00 sec 1.16 MBytes 9.69 Mbits/sec 0.120 ms 4/152 (2.6%)
[ 5] 5.00-6.00 sec 1.11 MBytes 9.31 Mbits/sec 0.106 ms 11/153 (7.2%)
[ 5] 6.00-7.00 sec 1.13 MBytes 9.50 Mbits/sec 0.313 ms 7/152 (4.6%)
[ 5] 7.00-8.00 sec 992 KBytes 8.13 Mbits/sec 0.163 ms 29/153 (19%)
[ 5] 8.00-9.00 sec 1.06 MBytes 8.91 Mbits/sec 0.129 ms 17/153 (11%)
[ 5] 9.00-10.00 sec 1.12 MBytes 9.37 Mbits/sec 0.131 ms 9/152 (5.9%)
[ 5] 10.00-11.00 sec 1.19 MBytes 9.96 Mbits/sec 0.155 ms 1/153 (0.65%)
[ 5] 11.00-12.00 sec 1.12 MBytes 9.44 Mbits/sec 0.336 ms 5/149 (3.4%)
[ 5] 12.00-13.00 sec 1.09 MBytes 9.18 Mbits/sec 0.166 ms 16/156 (10%)
[ 5] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 0.109 ms 0/153 (0%)
[ 5] 14.00-15.00 sec 1.18 MBytes 9.90 Mbits/sec 0.141 ms 1/152 (0.66%)
iperf Done.
[root at gfxc0 ~]#
"tcp slowdown"
[root at gfxc0 ~]# iperf3 -ZVc 13.91.55.110 -b20m -t15 --get-server-output
iperf 3.1.3
Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64
Time: Fri, 01 Dec 2017 00:30:16 GMT
Connecting to host 13.91.55.110, port 5201
Cookie: gfxc0.localdomain.1512088216.384940.
TCP MSS: 1428 (default)
[ 4] local 216.75.40.2 port 44388 connected to 13.91.55.110 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 2.33 MBytes 19.6 Mbits/sec 13 411 KBytes
[ 4] 1.00-2.00 sec 2.38 MBytes 19.9 Mbits/sec 13 201 KBytes
[ 4] 2.00-3.00 sec 2.38 MBytes 19.9 Mbits/sec 3 105 KBytes
[ 4] 3.00-4.00 sec 2.45 MBytes 20.6 Mbits/sec 3 64.1 KBytes
[ 4] 4.00-5.00 sec 1.91 MBytes 16.1 Mbits/sec 10 44.6 KBytes
[ 4] 5.00-6.00 sec 1.26 MBytes 10.5 Mbits/sec 8 36.3 KBytes
[ 4] 6.00-7.00 sec 1.26 MBytes 10.6 Mbits/sec 6 20.9 KBytes
[ 4] 7.00-8.00 sec 803 KBytes 6.58 Mbits/sec 3 22.3 KBytes
[ 4] 8.00-9.00 sec 697 KBytes 5.71 Mbits/sec 3 25.1 KBytes
[ 4] 9.00-10.00 sec 1.27 MBytes 10.6 Mbits/sec 0 48.8 KBytes
[ 4] 10.00-11.00 sec 1.53 MBytes 12.9 Mbits/sec 4 37.7 KBytes
[ 4] 11.00-12.00 sec 1.21 MBytes 10.1 Mbits/sec 5 33.5 KBytes
[ 4] 12.00-13.00 sec 1.35 MBytes 11.3 Mbits/sec 1 40.4 KBytes
[ 4] 13.00-14.00 sec 1.33 MBytes 11.1 Mbits/sec 1 51.6 KBytes
[ 4] 14.00-15.00 sec 2.28 MBytes 19.1 Mbits/sec 2 53.0 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-15.00 sec 24.4 MBytes 13.6 Mbits/sec 75 sender
[ 4] 0.00-15.00 sec 24.1 MBytes 13.5 Mbits/sec receiver
CPU Utilization: local/sender 0.7% (0.1%u/0.6%s), remote/receiver 0.2% (0.0%u/0.2%s)
Server output:
-----------------------------------------------------------
Accepted connection from 216.75.40.2, port 44386
[ 5] local 10.0.0.4 port 5201 connected to 216.75.40.2 port 44388
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 2.21 MBytes 18.5 Mbits/sec
[ 5] 1.00-2.00 sec 2.38 MBytes 19.9 Mbits/sec
[ 5] 2.00-3.00 sec 2.38 MBytes 19.9 Mbits/sec
[ 5] 3.00-4.00 sec 2.23 MBytes 18.7 Mbits/sec
[ 5] 4.00-5.00 sec 1.93 MBytes 16.2 Mbits/sec
[ 5] 5.00-6.00 sec 1.26 MBytes 10.6 Mbits/sec
[ 5] 6.00-7.00 sec 1.27 MBytes 10.6 Mbits/sec
[ 5] 7.00-8.00 sec 800 KBytes 6.56 Mbits/sec
[ 5] 8.00-9.00 sec 692 KBytes 5.67 Mbits/sec
[ 5] 9.00-10.00 sec 1.27 MBytes 10.7 Mbits/sec
[ 5] 10.00-11.00 sec 1.53 MBytes 12.9 Mbits/sec
[ 5] 11.00-12.00 sec 1.21 MBytes 10.2 Mbits/sec
[ 5] 12.00-13.00 sec 1.35 MBytes 11.3 Mbits/sec
[ 5] 13.00-14.00 sec 1.32 MBytes 11.1 Mbits/sec
[ 5] 14.00-15.00 sec 2.26 MBytes 18.9 Mbits/sec
iperf Done.
[root at gfxc0 ~]#
...but then results look OK to same Azure server when traffic forced to get to MSN via Cogent
My traceroute [v0.87]
gfxc0.localdomain (0.0.0.0) Thu Nov 30 16:20:50 2017
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 216.75.40.1 0.0% 19 0.3 9.9 0.3 124.0 30.6
2. 216.98.153.90 0.0% 19 63.8 10.0 0.3 117.3 29.8
3. te0-0-2-2.agr11.san01.atlas.cogentco.com 0.0% 19 1.6 0.9 0.7 1.6 0.0
4. te0-0-0-3.rcr12.san01.atlas.cogentco.com 0.0% 19 1.0 1.0 0.8 1.2 0.0
5. be2937.rcr12.sna02.atlas.cogentco.com 0.0% 19 3.2 3.3 3.2 3.5 0.0
6. be2463.agr22.lax01.atlas.cogentco.com 0.0% 19 4.0 4.3 4.0 6.7 0.5
7. be2586.ccr41.lax01.atlas.cogentco.com 0.0% 19 4.1 4.1 4.0 4.4 0.0
8. be3271.ccr41.lax04.atlas.cogentco.com 0.0% 19 4.9 4.7 4.0 9.2 1.0
9. 38.142.33.250 0.0% 19 4.0 3.9 3.8 4.0 0.0
10. be-63-0.ibr01.lax03.ntwk.msn.net 0.0% 19 17.0 16.6 15.6 17.4 0.2
11. be-4-0.ibr01.by2.ntwk.msn.net 0.0% 19 17.2 16.9 15.9 17.9 0.0
12. 104.44.7.198 0.0% 19 16.1 17.0 16.1 17.8 0.2
13. ae103-0.icr02.by4.ntwk.msn.net 0.0% 19 15.2 15.3 15.1 17.3 0.4
14. ???
[root at gfxc0 ~]# iperf3 -uZVc 13.91.55.110 -b10m -t15 --get-server-output
iperf 3.1.3
Linux gfxc0.localdomain 4.12.5-300.fc26.x86_64 #1 SMP Mon Aug 7 15:27:25 UTC 2017 x86_64
Time: Fri, 01 Dec 2017 00:21:51 GMT
Connecting to host 13.91.55.110, port 5201
Cookie: gfxc0.localdomain.1512087711.332144.
[ 4] local 216.75.40.2 port 43882 connected to 13.91.55.110 port 5201
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 15 second test
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 1.09 MBytes 9.11 Mbits/sec 139
[ 4] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 10.00-11.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 11.00-12.00 sec 1.19 MBytes 9.96 Mbits/sec 152
[ 4] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 153
[ 4] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 152
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-15.00 sec 17.8 MBytes 9.94 Mbits/sec 0.046 ms 0/2274 (0%)
[ 4] Sent 2274 datagrams
CPU Utilization: local/sender 0.7% (0.1%u/0.7%s), remote/receiver 0.0% (0.0%u/0.0%s)
Server output:
-----------------------------------------------------------
Accepted connection from 216.75.40.2, port 44356
[ 5] local 10.0.0.4 port 5201 connected to 216.75.40.2 port 43882
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 1.08 MBytes 9.04 Mbits/sec 0.036 ms 0/138 (0%)
[ 5] 1.00-2.00 sec 1.20 MBytes 10.0 Mbits/sec 0.057 ms 0/153 (0%)
[ 5] 2.00-3.00 sec 1.19 MBytes 9.96 Mbits/sec 0.047 ms 0/152 (0%)
[ 5] 3.00-4.00 sec 1.20 MBytes 10.0 Mbits/sec 0.049 ms 0/153 (0%)
[ 5] 4.00-5.00 sec 1.19 MBytes 9.96 Mbits/sec 0.064 ms 0/152 (0%)
[ 5] 5.00-6.00 sec 1.20 MBytes 10.0 Mbits/sec 0.059 ms 0/153 (0%)
[ 5] 6.00-7.00 sec 1.19 MBytes 9.96 Mbits/sec 0.056 ms 0/152 (0%)
[ 5] 7.00-8.00 sec 1.20 MBytes 10.0 Mbits/sec 0.055 ms 0/153 (0%)
[ 5] 8.00-9.00 sec 1.20 MBytes 10.0 Mbits/sec 0.061 ms 0/153 (0%)
[ 5] 9.00-10.00 sec 1.19 MBytes 9.96 Mbits/sec 0.051 ms 0/152 (0%)
[ 5] 10.00-11.00 sec 1.20 MBytes 10.0 Mbits/sec 0.057 ms 0/153 (0%)
[ 5] 11.00-12.00 sec 1.19 MBytes 9.96 Mbits/sec 0.053 ms 0/152 (0%)
[ 5] 12.00-13.00 sec 1.20 MBytes 10.0 Mbits/sec 0.055 ms 0/153 (0%)
[ 5] 13.00-14.00 sec 1.20 MBytes 10.0 Mbits/sec 0.063 ms 0/153 (0%)
[ 5] 14.00-15.00 sec 1.19 MBytes 9.96 Mbits/sec 0.046 ms 0/152 (0%)
iperf Done.
[root at gfxc0 ~]#
From: Gregorio Focaccio
Sent: Tuesday, November 28, 2017 2:05 PM
To: 'nanog at nanog.org' <nanog at nanog.org>
Subject: Packet Loss through Level 3 in Southern California?
Hi All,
We are a multi-datacenter MSP in San Diego that also offers Colo and Cloud hosting. We are multi-homed with Level 3 and Cogent.
A client reported newly slow FTP transfers, so we started a network investigation.
Our data (see below) seem to show packet loss through Level 3 with associated slow TCP based data transfers.
Is anyone else seeing packet loss and consequent slow TCP based transfers when going through Level 3 in Southern California?
Thanks,
Greg Focaccio
CARI.net<https://cari.net/>
Testing Data:
********** Internal tests - OK **********
FTP transfers from client server to another server 3 hops away in our adjacent datacenter was normal
********** Level 3 data - packet loss **********
External tests via Level 3 - 12% Packet Loss - see data below
UDP IPERF testing from our data center (through Level 3 and Microsoft - trace below) to an Azure server showed repeatable packet loss
TCP based testing - such as FTP or SCP transfers - the rate was very slow about 4Mbps
[root at raynor ~]# iperf3 -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[ 4] local 71.6.220.101 port 55684 connected to 40.80.156.2 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1001 KBytes 8.20 Mbits/sec 34 9.76 KBytes
[ 4] 1.00-2.00 sec 503 KBytes 4.12 Mbits/sec 12 11.2 KBytes
[ 4] 2.00-3.00 sec 502 KBytes 4.11 Mbits/sec 3 25.1 KBytes
[ 4] 3.00-4.00 sec 502 KBytes 4.11 Mbits/sec 12 11.2 KBytes
[ 4] 4.00-5.00 sec 377 KBytes 3.08 Mbits/sec 9 15.3 KBytes
[ 4] 5.00-6.00 sec 502 KBytes 4.11 Mbits/sec 10 19.5 KBytes
[ 4] 6.00-7.00 sec 377 KBytes 3.09 Mbits/sec 13 6.97 KBytes
[ 4] 7.00-8.00 sec 251 KBytes 2.06 Mbits/sec 9 5.58 KBytes
[ 4] 8.00-9.00 sec 251 KBytes 2.06 Mbits/sec 6 9.76 KBytes
[ 4] 9.00-10.00 sec 251 KBytes 2.06 Mbits/sec 10 12.6 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 4.41 MBytes 3.70 Mbits/sec 118 sender
[ 4] 0.00-10.00 sec 4.24 MBytes 3.56 Mbits/sec receiver
iperf Done.
[root at raynor ~]# iperf3 -u -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[ 4] local 71.6.220.101 port 39221 connected to 40.80.156.2 port 5201
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 85.0 MBytes 712 Mbits/sec 62402
[ 4] 1.00-2.00 sec 94.0 MBytes 789 Mbits/sec 69043
[ 4] 2.00-3.00 sec 92.6 MBytes 777 Mbits/sec 68030
[ 4] 3.00-4.00 sec 76.5 MBytes 641 Mbits/sec 56153
[ 4] 4.00-5.00 sec 94.9 MBytes 796 Mbits/sec 69662
[ 4] 5.00-6.00 sec 97.7 MBytes 819 Mbits/sec 71713
[ 4] 6.00-7.00 sec 98.5 MBytes 826 Mbits/sec 72347
[ 4] 7.00-8.00 sec 92.7 MBytes 778 Mbits/sec 68085
[ 4] 8.00-9.00 sec 91.3 MBytes 765 Mbits/sec 67045
[ 4] 9.00-10.00 sec 59.3 MBytes 498 Mbits/sec 43551
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 883 MBytes 740 Mbits/sec 0.014 ms 75647/647955 (12%)
[ 4] Sent 647955 datagrams
iperf Done.
[root at raynor ~]# traceroute 40.80.156.2
traceroute to 40.80.156.2 (40.80.156.2), 30 hops max, 60 byte packets
1 gateway (71.6.220.97) 6.170 ms 7.827 ms 10.718 ms
2 209.126.134.65 (209.126.134.65) 0.426 ms 0.420 ms 0.543 ms
3 216.98.153.21 (216.98.153.21) 159.936 ms 160.215 ms 160.209 ms
4 gi1-2.gw65-50.c5.sdcix.net (216.98.153.89) 170.977 ms 170.972 ms 170.964 ms
5 xe-8-3-3.bar1.SanDiego1.Level3.net (4.16.105.93) 0.547 ms 0.586 ms 0.533 ms
6 * * *
7 * * *
8 Microsoft-level3-20G.LosAngeles1.Level3.net (4.68.111.122) 17.296 ms 18.026 ms 20.200 ms
9 be-61-0.ibr01.lax03.ntwk.msn.net (104.44.8.104) 30.108 ms 32.300 ms 32.283 ms
10 be-4-0.ibr01.by2.ntwk.msn.net (104.44.4.3) 30.364 ms 30.961 ms 28.960 ms
11 104.44.7.198 (104.44.7.198) 31.870 ms 30.120 ms 29.636 ms
12 ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194) 27.828 ms ae101-0.icr01.by4.ntwk.msn.net (104.44.11.193) 29.611 ms ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194) 27.710 ms
13 * * *
[root at raynor ~]# scp ubuntu-14.04.5-desktop-amd64.iso carinet at 40.80.156.2:/home/carinet/ubuntunew11.iso<mailto:carinet at 40.80.156.2:/home/carinet/ubuntunew11.iso>
ubuntu-14.04.5-desktop-amd64.iso 100% 1053MB 730.8KB/s 24:35
LEVEL3 - summary
CARIcloud to Azure
2 Mbits/s TCP iperf
800 Mbits/s UDP iperf
24m and 35s 1053 MB upload to Azure through SCP
********** Cogent data - OK **********
External tests via Cogent - OK - No significant loss - see IPERF and trace data below
[root at raynor ~]# iperf3 -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[ 4] local 71.6.220.101 port 45076 connected to 40.80.156.2 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 149 MBytes 1.25 Gbits/sec 0 4.58 MBytes
[ 4] 1.00-2.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes
[ 4] 2.00-3.00 sec 178 MBytes 1.50 Gbits/sec 0 4.58 MBytes
[ 4] 3.00-4.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes
[ 4] 4.00-5.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes
[ 4] 5.00-6.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes
[ 4] 6.00-7.00 sec 179 MBytes 1.50 Gbits/sec 0 4.58 MBytes
[ 4] 7.00-8.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes
[ 4] 8.00-9.00 sec 181 MBytes 1.52 Gbits/sec 0 4.58 MBytes
[ 4] 9.00-10.00 sec 180 MBytes 1.51 Gbits/sec 0 4.58 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.73 GBytes 1.48 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.73 GBytes 1.48 Gbits/sec receiver
iperf Done.
[root at raynor ~]# iperf3 -u -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[ 4] local 71.6.220.101 port 39130 connected to 40.80.156.2 port 5201
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-1.00 sec 101 MBytes 844 Mbits/sec 73869
[ 4] 1.00-2.00 sec 103 MBytes 861 Mbits/sec 75344
[ 4] 2.00-3.00 sec 99.0 MBytes 830 Mbits/sec 72700
[ 4] 3.00-4.00 sec 97.9 MBytes 821 Mbits/sec 71885
[ 4] 4.00-5.00 sec 98.0 MBytes 822 Mbits/sec 71979
[ 4] 5.00-6.00 sec 100 MBytes 841 Mbits/sec 73640
[ 4] 6.00-7.00 sec 69.8 MBytes 585 Mbits/sec 51237
[ 4] 7.00-8.00 sec 94.9 MBytes 796 Mbits/sec 69650
[ 4] 8.00-9.00 sec 102 MBytes 855 Mbits/sec 74804
[ 4] 9.00-10.00 sec 100 MBytes 839 Mbits/sec 73407
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 965 MBytes 809 Mbits/sec 0.004 ms 3194/708505 (0.45%)
[ 4] Sent 708505 datagrams
iperf Done.
[root at raynor ~]# traceroute 40.80.156.2
traceroute to 40.80.156.2 (40.80.156.2), 30 hops max, 60 byte packets
1 gateway (71.6.220.97) 9.892 ms 10.883 ms 15.574 ms
2 209.126.134.65 (209.126.134.65) 1.541 ms 1.966 ms 1.959 ms
3 216.98.153.21 (216.98.153.21) 0.523 ms 0.517 ms 0.511 ms
4 216.98.153.114 (216.98.153.114) 0.589 ms 0.480 ms 0.687 ms
5 te0-0-1-0.nr11.b005949-0.san01.atlas.cogentco.com (38.104.122.61) 1.045 ms 1.039 ms 1.072 ms
6 te0-0-0-3.agr11.san01.atlas.cogentco.com (154.24.32.53) 1.023 ms 0.823 ms te0-0-0-3.agr12.san01.atlas.cogentco.com (154.24.32.65) 1.359 ms
7 te0-0-1-3.rcr11.san01.atlas.cogentco.com (154.24.31.21) 0.919 ms te0-0-1-3.rcr12.san01.atlas.cogentco.com (154.24.31.37) 0.834 ms 0.874 ms
8 be2936.rcr11.sna02.atlas.cogentco.com (154.54.45.166) 3.947 ms 3.726 ms be2937.rcr12.sna02.atlas.cogentco.com (154.54.45.173) 3.221 ms
9 be2463.agr22.lax01.atlas.cogentco.com (154.54.80.61) 4.095 ms 3.960 ms 4.087 ms
10 be2584.ccr41.lax01.atlas.cogentco.com (154.54.29.33) 4.364 ms 3.961 ms be2586.ccr41.lax01.atlas.cogentco.com (154.54.29.245) 4.290 ms
11 be3360.ccr41.lax04.atlas.cogentco.com (154.54.25.150) 4.283 ms be3271.ccr41.lax04.atlas.cogentco.com (154.54.42.102) 4.020 ms 4.003 ms
12 38.142.33.250 (38.142.33.250) 3.828 ms 3.818 ms 3.481 ms
13 be-61-0.ibr01.lax03.ntwk.msn.net (104.44.8.104) 15.565 ms 15.291 ms 15.173 ms
14 be-4-0.ibr01.by2.ntwk.msn.net (104.44.4.3) 14.290 ms 13.903 ms 16.168 ms
15 104.44.7.198 (104.44.7.198) 15.445 ms 14.098 ms 13.866 ms
16 ae102-0.icr02.by21.ntwk.msn.net (104.44.11.198) 13.752 ms ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194) 14.378 ms ae101-0.icr01.by4.ntwk.msn.net (104.44.11.193) 13.813 ms
[root at raynor ~]# scp ubuntu-14.04.5-desktop-amd64.iso carinet at 40.80.156.2:/home/carinet/ubuntunew10.iso<mailto:carinet at 40.80.156.2:/home/carinet/ubuntunew10.iso>
ubuntu-14.04.5-desktop-amd64.iso 100% 1053MB 111.5MB/s 00:09
COGENT - summary
CARIcloud to Azure
1.5 Gbits/s TCP iperf
850 Mbits/s UDP iperf
9s 1053 MB upload to Azure through SCP
More information about the NANOG
mailing list