Re: [Condor-users] Transfer of output files very slow

["Kewley, J \(John\)" <john.kewley@xxxxxxxxxx>]

If you can't work out how to speed this up,

you could maybe submit a .bat batch file as your "executable", this would

run the program as now, and then pack up the files before returning them to you.

 

JK

---

From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of user790 user790
Sent: Wednesday, December 31, 2008 9:53 AM
To: condor-users@xxxxxxxxxxx
Subject: [Condor-users] Transfer of output files very slow

I am running Condor on two Windows XP computers. My program is an uninstrumented binary running in a vanilla universe. During its execution it writes a large (~300) number of small (~1 to 100 kB) files. Now after the program has been executed, the files that has been written locally on disk are transfered back to the submitter very slowly. It seems it is due to the number of files rather than the amount of data, because if I concatenate all the files into a single uncompressed archive, it is transferred much faster.

 

Any idea what is going on and how I could avoid the extra compression step?

 

Thanks,

 

P.