Best utilizing fat long pipes and large file transfer

Fri Jun 13 00:58:46 UTC 2008

> I'm looking for input on the best practices for sending large files over
> a long fat pipe between facilities (gigabit private circuit, ~20ms RTT).

providing you have RFC1323 type extensions enabled on a semi-decent OS, a 4MB
TCP window should be more than sufficient to fill a GbE pipe over 30msec.

with a modified TCP stack, that uses TCP window sizes up to 32MB, i've worked
with numerous customers to achieve wire-rate GbE async replication for
storage-arrays with FCIP.

the modifications to TCP were mostly to adjust how it reacts to packet loss,
e.g. don't "halve the window".
the intent of those modifications is that it doesn't use the "greater internet"
but is more suited for private connections within an enterprise customer
environment.

that is used in production today on many Cisco MDS 9xxx FC switch environments.

> I'd like to avoid modifying TCP windows and options on end hosts where
> possible (I have a lot of them). I've seen products that work as
> "transfer stations" using "reliable UDP" to get around the windowing
> problem.

given you don't want to modify all your hosts, you could 'proxy' said TCP
connections via 'socat' or 'netcat++'.

cheers,

lincoln.