Re: [HTCondor-users] Including data in a submit file

["R. Kent Wenger" <wenger@xxxxxxxxxxx>]

On Fri, 13 Dec 2013, Szabolcs Horvátth wrote:

> I have a DAGMan job graph, but some of the later running jobs' submit 
> settings should depend on the result of the first job.
> For example the first job generates some datastructure and based on that 
> result just a limited number of the computers are able to run the other jobs.
>
> If I could include the content of a text file into the submit files or read 
> it into a variable in the submit files than the first job could simply write 
> its requirements (or concurrency attributes) into this text file and that 
> could be used automatically in all other jobs without the need to modify 
> *all* submit files.
> I checked the documentation for such feature or solution but could not find 
> any. Can it be done somehow? Or can you suggest an alternate solution that 
> does not require
> editing all submit files or submitting a new DAG?

If I'm correctly understanding what you want to do, it shouldn't be hard.
Remember that submit files are evaluated lazily, so the PRE script of a 
node can write the submit file for that node.

So if your first node writes out requirements.txt, the PRE script of your
later node can do something like this:

  cat submit_template requirements.txt > real_submit

You probably don't want to do this:

  cat requirements.txt >> real_submit

because that won't work right if the node get retried, etc.

There are a few more details, like removing real_submit if it already 
exists, but that's the basic idea.  (To summarize, the PRE script can 
create the submit file however it wants to -- DAGMan doesn't actually read 
the submit file until it's time to submit the job.)

Kent Wenger
CHTC Team