Best utilizing fat long pipes and large file transfer

Sean Knox sean at craigslist.org
Fri Jun 13 17:43:23 UTC 2008


Many thanks for great replies on and off-list.

The suggestions basically ranged from these options:

1. tune TCP on all hosts you wish to transfer between
2. create tuned TCP proxies and transfer through those hosts
3. setup a socat (netcat++) proxy and send through this host
4. use an alternative to plain netcat/scp for large file transfers

My needs are pretty simple: occasionally I need to push large database 
files (300Gb+) around linux hosts. #4 seems like the best option for me.

People suggested a slew of methods to do this: RBUDP, gridftp, bbcp, and 
many others, with programs either sending with reliable UDP or breaking 
large transfers into multiple streams. Because it was easy to use right 
away, I tried RBUDP and was able to copy a tarball at about 700Mb/s over 
a 20ms delay link, and when factoring in destination disk write speed, 
isn't too bad a starting point. GridFTP and bbcp look very useful too; 
I'll be exploring them as well. The presentation links Kevin O. sent 
were very interesting.

I've looked at HPN-SSH before but haven't played with it much. I'll 
definitely try it out based on the feedback from this thread.

Thanks again.

sk


Sean Knox wrote:
> Hi,
> 
> I'm looking for input on the best practices for sending large files over 
> a long fat pipe between facilities (gigabit private circuit, ~20ms RTT).
> I'd like to avoid modifying TCP windows and options on end hosts where 
> possible (I have a lot of them). I've seen products that work as 
> "transfer stations" using "reliable UDP" to get around the windowing 
> problem.
> 
> I'm thinking of setting up servers with optimized TCP settings to push 
> big files around data centers but I'm curious to know how others deal 
> with LFN+large transfers.
> 
> thanks,
> Sean




More information about the NANOG mailing list