TCP Performance

Mon Aug 26 19:07:52 UTC 2013

Greetings all, I've got an issue I was hoping to put a few more eyes on. 
 Here's the scenario. Downloading a file at our Border is multiple orders 
of magnitude faster then a few hops out. Using the same 128MB test file, I 
tested at two different locations. As well as between them. Using multiple 
connections improves throughput, However it's the single stream issue we're 
looking at right now. All testing servers in question are Centos Linux. 
 Orlando Datapath: Cogent>Orlando Border Router (Mikrotik)>HP Procurve 
Switch> Server Results: 2013-08-29 05:04:09 (52.6 MB/s) - `128mbfile.tgz' 
saved [127926272/127926272] 
 Cocoa NOC Datapath: Cogent>Orlando Border Router (Mikrotik)>Licensed 
Microwave Link (300+Mb/s Capacity)>East Orange Router (Mikrotik)> Licensed 
Microwave Link (300+Mb/s Capacity)>Cocoa Router (Mikrotik)>Licensed 
Microwave Link (300+Mb/s Capacity)>Colo Router (Mikrotik)>NOC Router 
(Mikrotik)>HP Procurve Switch>Server Results: 2013-08-26 13:42:25 (398 
KB/s) - `128mbfile.tgz' saved [127926272/127926272] 
 Orlando-Cocoa NOC Datapath: Orlando Server>HP Procurve Switch>Orlando 
Border Router (Mikrotik)>Licensed Microwave Link (300+Mb/s Capacity)>East 
Orange Router (Mikrotik)> Licensed Microwave Link (300+Mb/s Capacity)>Cocoa 
Router (Mikrotik)>Licensed Microwave Link (300+Mb/s Capacity)>Colo Router 
(Mikrotik)>NOC Router(Mikrotik)>HP Procurve Switch>ServerResults: 
2013-08-26 13:56:25 (3.31 MB/s) - `128mbfile.tgz' saved 
[134217728/134217728] 
 Now, For the fun of it. I ran Iperf single TCP between our Cocoa and 
Orlando POP's. Just like the HTTP test above. (Server has a 100Mb/s port). 
It maxes out the port, Unlike the HTTP test. 
 [root at ded01 ~]# iperf -c 
208.90.219.18------------------------------------------------------------Cli
ent connecting to 208.90.219.18, TCP port 5001TCP window size: 16.0 KByte 
(default)------------------------------------------------------------[  3] 
local 206.208.56.130 port 47281 connected with 208.90.219.18 port 5001[ ID] 
Interval       Transfer     Bandwidth[  3]  0.0-10.0 sec   114 MBytes  95.7 
Mbits/sec

Here's associated packet captures for each transfer. As well as full wget 
output and traceroutes for each test. As you can see, The tests crossing 
the wireless links show about 3x more TCP re-transmits/dup ACK's. But I'm 
not sure I'm sold this could show such a huge drop in throughput. Other 
then that, nothing really stands out to me as to why these transfers are so 
much slower. Intra-network iperf testing shows full throughput the whole 
way with single connection. As well as UDP testing. One thing to note is 
the Iperf testing has far less TCP re-transmit/dup acks then any of the 
HTTP testing, Crossing the same Microwave Links and routers.
http://cdn.141networks.com/files/captures.zip 
I appreciate any insight anyone might have. Thanks! 
 Nick Olsen
Network Operations (855) FLSPEED  x106