Best utilizing fat long pipes and large file transfer
sean at craigslist.org
Fri Jun 13 17:43:23 UTC 2008
Many thanks for great replies on and off-list.
The suggestions basically ranged from these options:
1. tune TCP on all hosts you wish to transfer between
2. create tuned TCP proxies and transfer through those hosts
3. setup a socat (netcat++) proxy and send through this host
4. use an alternative to plain netcat/scp for large file transfers
My needs are pretty simple: occasionally I need to push large database
files (300Gb+) around linux hosts. #4 seems like the best option for me.
People suggested a slew of methods to do this: RBUDP, gridftp, bbcp, and
many others, with programs either sending with reliable UDP or breaking
large transfers into multiple streams. Because it was easy to use right
away, I tried RBUDP and was able to copy a tarball at about 700Mb/s over
a 20ms delay link, and when factoring in destination disk write speed,
isn't too bad a starting point. GridFTP and bbcp look very useful too;
I'll be exploring them as well. The presentation links Kevin O. sent
were very interesting.
I've looked at HPN-SSH before but haven't played with it much. I'll
definitely try it out based on the feedback from this thread.
Sean Knox wrote:
> I'm looking for input on the best practices for sending large files over
> a long fat pipe between facilities (gigabit private circuit, ~20ms RTT).
> I'd like to avoid modifying TCP windows and options on end hosts where
> possible (I have a lot of them). I've seen products that work as
> "transfer stations" using "reliable UDP" to get around the windowing
> I'm thinking of setting up servers with optimized TCP settings to push
> big files around data centers but I'm curious to know how others deal
> with LFN+large transfers.
More information about the NANOG