Re: [HTCondor-users] Most of the time in Condor jobs gets wasted in I/o

[Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx>]

On 04/25/2013 11:38 AM, Dr. Harinder Singh Bawa wrote:

...  just
wondering that "cp" command will work for the data of say 150GB on one
node.(15TB divided into 100 jobs/nodes). Certainly it works but take lot of
> time to copy. Is there any other command I can use instead of cp.which is
> meant to be faster

I'd start with taking the calculator and seeing how long it should take
to push that many bits over your network. That would give you a
baseline, then you can define "faster". Then look into implementation
details.

The fastest way to copy over the network is 'cat | nc', but it's harder
to use in a script. Rsync without delta-xfer would probably be the
second fastest, but you need to set up rsync server. Scp is generally no
faster than cp over nfs since you can't turn off encryption, plus you
have set up passwordless ssh on all nodes.

However, if it takes 34 hours to send that much data over the wire, and
condor does it in 24 hours, then perhaps it's all moot anyway.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

Attachment: signature.asc
Description: OpenPGP digital signature