Re: [Condor-users] dynamic dagman

["R. Kent Wenger" <wenger@xxxxxxxxxxx>]

On Mon, 29 Mar 2010, Jason Reilly wrote:

> I have a job that has 2 parts.  The output of the first job returns a zip 
> file with N entries.  For the second job, I would like to dynamically create 
> a dagman based on the number of entries in the zip file generated by the 
> output of the first job.  Is there such a thing as a dynamic dag that can 
> handle this kind of task, where I don't need to programmatically generate the 
> Condor submit file?

If I'm understanding things correctly, you should be able to do what you 
want with a little trickery.  As of 7.5.2, the trickery will no longer be
necessary.

I'm assuming that you want to do the whole thing in a DAG.  Because of the 
dynamic aspect, you'll have to use nested DAGs.

* Top-level DAG:
Job A getzip.sub
Subdag external B unzip.dag
Script Pre B generate_dag
Parent A child B

The idea is that the generate_dag script would look at the zip file and 
write the "unzip.dag" file.

The trickery is this:  In the current versions of DAGMan, a valid 
unzip.dag file must exist at the time when you submit the top-level DAG.
You can overwrite the unzip.dag file, but it must exist.  In current 
versions of DAGMan (since 7.3-something, I think), condor_submit_dag
will automatically run itself on the lower-level DAG file (with the 
-no_submit option) to generate the .condor.sub file.

In 7.5.2, by default, the running of condor_submit_dag -no_submit will
be done by DAGMan itself, just before the node job will be submitted,
so you won't have to have a "dummy" unzip.dag file in place when you
submit the top-level DAG.

Let me know if this isn't clear...

Kent Wenger
Condor Team