[Condor-users] Running Lots of Jobs with Makeflow

[Douglas Thain <dthain@xxxxxx>]

In response to several recent messages about running lots of jobs:

We have developed a tool called "Makeflow" that works with Condor and
is handy for running a large number of small jobs with dependencies.
Very briefly, you specify each job as a list of input files, output
files, and a command to run:

    output.txt: input.data mysim.exe
        ./mysim.exe -t 5 input.data -o output.txt

You can configure Makeflow to submit jobs directly to Condor, or to
send them to lightweight "worker" processes deployed into your Condor
pool, or other grids or clouds.    It's sort of like a cross between
DAGMan and Master-Worker.

Makeflow is a relatively new tool, but we have been using it on a
production basis over the summer to run bioinformatics workloads with
thousands to millions of components.  It also works with other batch
systems and plain multicore computers.  If this sounds interesting,
you can get more information and download the software here:

    http://www.cse.nd.edu/~ccl/software/makeflow

Cheers -

Douglas Thain
dthain@xxxxxx
University of Notre Dame