Best utilizing fat long pipes and large file transfer

Fri Jun 13 00:28:52 UTC 2008

> Hi,
> 
> I'm looking for input on the best practices for sending large 
> files 

There are both commercial products (fastcopy)
and various "free"(*) products (bbcp, bbftp,
gridftp) that will send large files.  While
they can take advantage of larger windows
they also have the capability of using multiple
streams (dealing with the inability to tune the
tcp stack).  There are, of course, competitors
to these products which you should look into.
As always, YMWV.

Some references:
http://www.softlink.com/fastcopy_techie.html
  (Some parts of NASA seem to like fastcopy)
http://nccs.gov/user-support/general-support/data-transfer/bbcp/
  (Full disclosure, bbcp was written by someone who sits
  about 3 meters from where I am sitting, but I cannot find
  a nice web reference from him about the product, so I am
  showing a different sites documentation)
http://doc.in2p3.fr/bbftp/
  (One of the first to use multistream for BaBar)
http://www.globus.org/grid_software/data/gridftp.php
  (Part of the globus toolkit.  Somewhat heavier weight
  if all you want is file transfer.)
http://fasterdata.es.net/tools.html
  (A reference I am surprised Kevin did not point to :-)
http://www.slac.stanford.edu/grp/scs/net/talk/ggf5_jul2002/NMWG_GGF5.pdf
  (A few year old performance evaluation)
www.triumf.ca/canarie/amsterdam-test/References/010402-ftp.ppt 
  (Another older performance evaluation)

Gary

(*) Some are GPL, and some (modified) BSD licenses.
    Which one is "free enough" depends on some strongly
    held beliefs of the validator.