Best utilizing fat long pipes and large file transfer

Lincoln Dale ltd at
Fri Jun 13 00:58:46 UTC 2008

> I'm looking for input on the best practices for sending large files over
> a long fat pipe between facilities (gigabit private circuit, ~20ms RTT).

providing you have RFC1323 type extensions enabled on a semi-decent OS, a 4MB
TCP window should be more than sufficient to fill a GbE pipe over 30msec.

with a modified TCP stack, that uses TCP window sizes up to 32MB, i've worked
with numerous customers to achieve wire-rate GbE async replication for
storage-arrays with FCIP.

the modifications to TCP were mostly to adjust how it reacts to packet loss,
e.g. don't "halve the window".
the intent of those modifications is that it doesn't use the "greater internet"
but is more suited for private connections within an enterprise customer

that is used in production today on many Cisco MDS 9xxx FC switch environments.

> I'd like to avoid modifying TCP windows and options on end hosts where
> possible (I have a lot of them). I've seen products that work as
> "transfer stations" using "reliable UDP" to get around the windowing
> problem.

given you don't want to modify all your hosts, you could 'proxy' said TCP
connections via 'socat' or 'netcat++'.



More information about the NANOG mailing list