Re: [Condor-users] transfers of intermediate files

[Erik Paulson <epaulson@xxxxxxxxxxx>]

On Sun, Dec 23, 2007 at 01:58:38PM +0100, Christopher Dianthus wrote:
> Dear All,
> 
> I just installed condor on a number of machines. Now I am trying to
> put some jobs into production work. However,
> I encountedred a problem with transfering intermediate files. My
> programs produce not only a regular console output but also a number
> of files which have some valuable information. The files are updated
> during the run and normally I can gather and analyse the data
> ``on-line'' during  very long runs (the execution time counts in
> days). How can I transfer these
> data to the machine submitting the jobs? I use the following script to
> submit jobs
> 
> #-------------------------------------------------------------------------------------------------
> executable = pgm
> arguments = f$(Process).in
> universe = vanilla
> output = out.$(Process)
> error = err.$(Process)
> log = log.$(Process)
> should_transfer_output_files = yes
> should_transfer_input_files = yes
> should_transfer_files = yes
> WhenToTransferOutput = ON_EXIT_OR_EVICT
> transfer_output_files = output$(Process).out
> transfer_input_files = input.dat, input$(Process).inp, test
> transferout = true
> stream_output = true
> queue 16
> #----------------------------------------------------------------------------------------------------
> During the run I see that output[0-15].out files appear in the
> directory with the submitting
> script (on the submitting machine), but these files are empty up to
> the end of the run.
> 

Condor cannot automatically stream any output files other than stdout and stderr. 

Condor creates the output[0-15].out files as placeholders (and as a permissions
check when the job is submitted) 

If you need to stream anything other than stdout and stderr, you need to
look into Chirp. See the list archives.

-Erik