[HTCondor-users] Scratch space and input subdirectories

[Michael V Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>]

Hi folks,

So we've got a fairly simple setup -
a PreCmd populates an input template file and places it into a numbered
directory it creates named with the $(Process), and then the job runs against
that template, the PostCmd compresses the various output files which may
differ from run to run, and then HTCondor will transfer back the numbered
directory and all its contents, so we end up with a directory named for
the process ID number containing all the outputs for that run, across all
the runs. By transferring the directory, we don't have to worry about figuring
out which files will be in there based on what version or scenario we're
using.

However we've got an executable that
apparently doesn't have an option to specify a run directory on the command
line, so it just works in whatever directory it's started in. This means
that it needs to be in the numbered directory, not in the root of the scratch
directory, when it starts under our current approach. And doing a "cd"
in the PreCmd doesn't work, because that doesn't carry over to the executable,
naturally.

Now I realize that we can just do a
wrapper script to change into the directory created by the PreCmd and then
launch the run, but I'd really prefer to strip off the layers of wrappers
and have the tool itself be what's run by the starter, rather than a wrapper
which runs the tool. I'd also prefer to have the final numbered output
directory not exist on the submitter's output path until the run is complete,
in an effort to keep things tidy and self-contained.

If we run it as-is without the numbered
subdirectory, HTCondor can transfer all output files it finds in the scratch
root without having to enumerate them, but can we get those into a subdirectory
on the submitter which won't exist before the job is complete?

Is there some trick for this I'm missing, or do I want to have my cake
and eat it too and should just go with a simple cd/exec wrapper?

	Michael V. Pelletier IT Program Execution Principal Engineer 978.858.9681 (5-9681) 339.293.9149 cell michael.v.pelletier@xxxxxxxxxxxx