Packet Loss through Level 3 in Southern California?

Gregorio Focaccio gfocaccio at cari.net
Tue Nov 28 22:59:00 CST 2017


Hi All,

We are an MSP in San Diego that also offers multi-datacenter Colo and Cloud hosting.  We are multi-homed with Level 3 and Cogent.

A physical server client reported newly slow FTP transfers, so we started a network investigation.

Our data (see below) seem to show packet loss through Level 3 with associated slow TCP based data transfers.

Is anyone else seeing packet loss and consequent slow TCP based transfers when going through Level 3 in (LosAngeles1) Southern California?

Thanks,
Greg Focaccio
CARI.net<https://cari.net/>


Testing Data:

********** Internal tests - OK **********
FTP transfers from client server to another server 3 hops away in our adjacent datacenter was normal


********** Level 3 data - packet loss **********
External tests via Level 3  - 12% Packet Loss - see data below
  UDP IPERF testing from our data center (through Level 3 and Microsoft - trace below) to an Azure server showed repeatable packet loss
   TCP based testing - such as FTP or SCP transfers - the rate was very slow about 4Mbps

[root at raynor ~]# iperf3 -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[  4] local 71.6.220.101 port 55684 connected to 40.80.156.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1001 KBytes  8.20 Mbits/sec   34   9.76 KBytes
[  4]   1.00-2.00   sec   503 KBytes  4.12 Mbits/sec   12   11.2 KBytes
[  4]   2.00-3.00   sec   502 KBytes  4.11 Mbits/sec    3   25.1 KBytes
[  4]   3.00-4.00   sec   502 KBytes  4.11 Mbits/sec   12   11.2 KBytes
[  4]   4.00-5.00   sec   377 KBytes  3.08 Mbits/sec    9   15.3 KBytes
[  4]   5.00-6.00   sec   502 KBytes  4.11 Mbits/sec   10   19.5 KBytes
[  4]   6.00-7.00   sec   377 KBytes  3.09 Mbits/sec   13   6.97 KBytes
[  4]   7.00-8.00   sec   251 KBytes  2.06 Mbits/sec    9   5.58 KBytes
[  4]   8.00-9.00   sec   251 KBytes  2.06 Mbits/sec    6   9.76 KBytes
[  4]   9.00-10.00  sec   251 KBytes  2.06 Mbits/sec   10   12.6 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  4.41 MBytes  3.70 Mbits/sec  118             sender
[  4]   0.00-10.00  sec  4.24 MBytes  3.56 Mbits/sec                  receiver
iperf Done.

[root at raynor ~]# iperf3 -u -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[  4] local 71.6.220.101 port 39221 connected to 40.80.156.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  85.0 MBytes   712 Mbits/sec  62402
[  4]   1.00-2.00   sec  94.0 MBytes   789 Mbits/sec  69043
[  4]   2.00-3.00   sec  92.6 MBytes   777 Mbits/sec  68030
[  4]   3.00-4.00   sec  76.5 MBytes   641 Mbits/sec  56153
[  4]   4.00-5.00   sec  94.9 MBytes   796 Mbits/sec  69662
[  4]   5.00-6.00   sec  97.7 MBytes   819 Mbits/sec  71713
[  4]   6.00-7.00   sec  98.5 MBytes   826 Mbits/sec  72347
[  4]   7.00-8.00   sec  92.7 MBytes   778 Mbits/sec  68085
[  4]   8.00-9.00   sec  91.3 MBytes   765 Mbits/sec  67045
[  4]   9.00-10.00  sec  59.3 MBytes   498 Mbits/sec  43551
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec   883 MBytes   740 Mbits/sec  0.014 ms  75647/647955 (12%)
[  4] Sent 647955 datagrams
iperf Done.

[root at raynor ~]# traceroute 40.80.156.2
traceroute to 40.80.156.2 (40.80.156.2), 30 hops max, 60 byte packets
1  gateway (71.6.220.97)  6.170 ms  7.827 ms  10.718 ms
2  209.126.134.65 (209.126.134.65)  0.426 ms  0.420 ms  0.543 ms
3  216.98.153.21 (216.98.153.21)  159.936 ms  160.215 ms  160.209 ms
4  gi1-2.gw65-50.c5.sdcix.net (216.98.153.89)  170.977 ms  170.972 ms  170.964 ms
5  xe-8-3-3.bar1.SanDiego1.Level3.net (4.16.105.93)  0.547 ms  0.586 ms  0.533 ms
6  * * *
7  * * *
8  Microsoft-level3-20G.LosAngeles1.Level3.net (4.68.111.122)  17.296 ms  18.026 ms  20.200 ms
9  be-61-0.ibr01.lax03.ntwk.msn.net (104.44.8.104)  30.108 ms  32.300 ms  32.283 ms
10  be-4-0.ibr01.by2.ntwk.msn.net (104.44.4.3)  30.364 ms  30.961 ms  28.960 ms
11  104.44.7.198 (104.44.7.198)  31.870 ms  30.120 ms  29.636 ms
12  ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194)  27.828 ms ae101-0.icr01.by4.ntwk.msn.net (104.44.11.193)  29.611 ms ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194)  27.710 ms
13  * * *

[root at raynor ~]# scp ubuntu-14.04.5-desktop-amd64.iso carinet at 40.80.156.2:/home/carinet/ubuntunew11.iso<mailto:carinet at 40.80.156.2:/home/carinet/ubuntunew11.iso>
ubuntu-14.04.5-desktop-amd64.iso                     100% 1053MB 730.8KB/s   24:35

LEVEL3 - summary
CARIcloud to Azure
2 Mbits/s TCP iperf
800 Mbits/s UDP iperf
24m and 35s 1053 MB upload to Azure through SCP


********** Cogent data - OK **********
External tests via Cogent - OK - No significant loss - see IPERF and trace data below

[root at raynor ~]# iperf3 -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[  4] local 71.6.220.101 port 45076 connected to 40.80.156.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   149 MBytes  1.25 Gbits/sec    0   4.58 MBytes
[  4]   1.00-2.00   sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
[  4]   2.00-3.00   sec   178 MBytes  1.50 Gbits/sec    0   4.58 MBytes
[  4]   3.00-4.00   sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
[  4]   4.00-5.00   sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
[  4]   5.00-6.00   sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
[  4]   6.00-7.00   sec   179 MBytes  1.50 Gbits/sec    0   4.58 MBytes
[  4]   7.00-8.00   sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
[  4]   8.00-9.00   sec   181 MBytes  1.52 Gbits/sec    0   4.58 MBytes
[  4]   9.00-10.00  sec   180 MBytes  1.51 Gbits/sec    0   4.58 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.73 GBytes  1.48 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  1.73 GBytes  1.48 Gbits/sec                  receiver
iperf Done.

[root at raynor ~]# iperf3 -u -c 40.80.156.2 -b 10000m
Connecting to host 40.80.156.2, port 5201
[  4] local 71.6.220.101 port 39130 connected to 40.80.156.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   101 MBytes   844 Mbits/sec  73869
[  4]   1.00-2.00   sec   103 MBytes   861 Mbits/sec  75344
[  4]   2.00-3.00   sec  99.0 MBytes   830 Mbits/sec  72700
[  4]   3.00-4.00   sec  97.9 MBytes   821 Mbits/sec  71885
[  4]   4.00-5.00   sec  98.0 MBytes   822 Mbits/sec  71979
[  4]   5.00-6.00   sec   100 MBytes   841 Mbits/sec  73640
[  4]   6.00-7.00   sec  69.8 MBytes   585 Mbits/sec  51237
[  4]   7.00-8.00   sec  94.9 MBytes   796 Mbits/sec  69650
[  4]   8.00-9.00   sec   102 MBytes   855 Mbits/sec  74804
[  4]   9.00-10.00  sec   100 MBytes   839 Mbits/sec  73407
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec   965 MBytes   809 Mbits/sec  0.004 ms  3194/708505 (0.45%)
[  4] Sent 708505 datagrams
iperf Done.

[root at raynor ~]# traceroute 40.80.156.2
traceroute to 40.80.156.2 (40.80.156.2), 30 hops max, 60 byte packets
1  gateway (71.6.220.97)  9.892 ms  10.883 ms  15.574 ms
2  209.126.134.65 (209.126.134.65)  1.541 ms  1.966 ms  1.959 ms
3  216.98.153.21 (216.98.153.21)  0.523 ms  0.517 ms  0.511 ms
4  216.98.153.114 (216.98.153.114)  0.589 ms  0.480 ms  0.687 ms
5  te0-0-1-0.nr11.b005949-0.san01.atlas.cogentco.com (38.104.122.61)  1.045 ms  1.039 ms  1.072 ms
6  te0-0-0-3.agr11.san01.atlas.cogentco.com (154.24.32.53)  1.023 ms  0.823 ms te0-0-0-3.agr12.san01.atlas.cogentco.com (154.24.32.65)  1.359 ms
7  te0-0-1-3.rcr11.san01.atlas.cogentco.com (154.24.31.21)  0.919 ms te0-0-1-3.rcr12.san01.atlas.cogentco.com (154.24.31.37)  0.834 ms  0.874 ms
8  be2936.rcr11.sna02.atlas.cogentco.com (154.54.45.166)  3.947 ms  3.726 ms be2937.rcr12.sna02.atlas.cogentco.com (154.54.45.173)  3.221 ms
9  be2463.agr22.lax01.atlas.cogentco.com (154.54.80.61)  4.095 ms  3.960 ms  4.087 ms
10  be2584.ccr41.lax01.atlas.cogentco.com (154.54.29.33)  4.364 ms  3.961 ms be2586.ccr41.lax01.atlas.cogentco.com (154.54.29.245)  4.290 ms
11  be3360.ccr41.lax04.atlas.cogentco.com (154.54.25.150)  4.283 ms be3271.ccr41.lax04.atlas.cogentco.com (154.54.42.102)  4.020 ms  4.003 ms
12  38.142.33.250 (38.142.33.250)  3.828 ms  3.818 ms  3.481 ms
13  be-61-0.ibr01.lax03.ntwk.msn.net (104.44.8.104)  15.565 ms  15.291 ms  15.173 ms
14  be-4-0.ibr01.by2.ntwk.msn.net (104.44.4.3)  14.290 ms  13.903 ms  16.168 ms
15  104.44.7.198 (104.44.7.198)  15.445 ms  14.098 ms  13.866 ms
16  ae102-0.icr02.by21.ntwk.msn.net (104.44.11.198)  13.752 ms ae100-0.icr01.by21.ntwk.msn.net (104.44.11.194)  14.378 ms ae101-0.icr01.by4.ntwk.msn.net (104.44.11.193)  13.813 ms

[root at raynor ~]# scp ubuntu-14.04.5-desktop-amd64.iso carinet at 40.80.156.2:/home/carinet/ubuntunew10.iso<mailto:carinet at 40.80.156.2:/home/carinet/ubuntunew10.iso>
ubuntu-14.04.5-desktop-amd64.iso                              100% 1053MB 111.5MB/s   00:09

COGENT - summary
CARIcloud to Azure
1.5 Gbits/s TCP iperf
850 Mbits/s UDP iperf
9s 1053 MB upload to Azure through SCP




More information about the NANOG mailing list