Re: [Condor-users] Create dir hierarchy for job output files?

[Matthew Farrellee <mfarrellee@xxxxxxxxxx>]

Steffen,

Do your users need access to the /scratch space after their jobs have 
run? If not, you could just put your EXECUTE (config param) directory on 
/scratch, and let Condor create&destroy temporary directories in it for 
each job.

If you need the output for the user to access later you could use the 
new Job Hooks' Job Exit hook [1] to copy your user's data out of a 
Condor controlled execute directory into the scratch space. You may be 
able to use the Prepare Hook [2], but it'll probably have the same 
problem as the wrapper script.

Best,


matt

[1] 
http://www.cs.wisc.edu/condor/manual/v7.1/4_4Job_Hooks.html#SECTION00542600000000000000
[2] 
http://www.cs.wisc.edu/condor/manual/v7.1/4_4Job_Hooks.html#SECTION00542400000000000000

Steffen Grunewald wrote:
> Hi everyone,
>
> as recommended, we're using local disk storage for job output, error, and
> log files, and specify corresponding paths in the submit file, using
> /scratch/<user>/output.$(Process) (etc., with <user> substituted with the
> real account name) expressions.
> Every now and then, a scratch disk has to be replaced after a disk failure,
> which means that the user cannot rely on the directory structure being there
> on all nodes anymore.
> Is it feasible to create the underlying directory hierarchy if necessary?
> Since the job would fail in the starter stage, there's no opportunity to
> do this in a wrapper script (which wouldn't run).
> Perhaps a new command for the submit file? (along the lines of 
> 'pre-run="mkdir -p /scratch/me"', or 'createpath=True')
> Perhaps something that's already there (and for some reason, invisible to
> me)?
>
> Cheers,
>  Steffen