Re: [HTCondor-users] Image processing with HTCondor

[Michael V Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>]

From: Sean Milligan <Sean.Milligan@xxxxxxxxxxxxx>
Date: 01/18/2016 11:41 AM

> We need to run these seven algorithms over each
image, these images are 
> all in separate directories so Iâm trying to figure out how others
> approach this. I thought Iâd be able to use something like the flexible
> queue command to iterate over each image but my reading through the
> mailing list archive explained why this isnât support with DAGman.
At this
> point the only thing Iâve figured is to write a script to create
unique 
> DAG files for each image and then either submit each DAG file individually
> or wrap all of the individual DAGs into a âmasterâ DAG as SUBDAGs.
>  
> I guess Iâm ultimately asking for pointers or
what approaches have others 
> used in situations like this?

You've got the right idea. At HTCondor Week 2015,
Miron described DAGman as
something akin to "assembly language," and
that's a good thing to keep in
mind when dealing with it.

My DAGman app has a little submission script I wrote
in Perl, which accepts
the name of the input file to be run and builds out
the DAG accordingly
through each of the processing steps, with a node
for splitting up the
input file, children for running the code against
each one, then a child
of all the run nodes for combining the output again,
and finally a node
for cleaning up the DAG files upon successful completion
of the run.

Either way, you'll need a script - or Pegasus - to
write out the DAG
"assembly language."

        -Michael Pelletier.