Re: [Condor-users] 2 general kinds of jobs, different maximum requirements.

[Erik Paulson <epaulson@xxxxxxxxxxx>]

On Tue, Feb 15, 2005 at 01:52:56PM -0600, Michael Remijan wrote:
> I have a dag that consists of 2 general kinds of jobs.
>         1) Get data
>         2) Insert data into database.
> 
> I'd like to be able to limit the number of "get data" jobs to 20 and the 
> number of "insert data into database" jobs two 5.  However, I do not want 
> to make artificial parent-child relationships in the dag.  When we create 
> a big dag we can have thousands of each kind of job.  the --maxjobs switch 
> to condor_submit_dag doesn't do what we need.  Anyone else have a 
> situation like this?  How did you handle it?
> 

That'd be a nice DAGMan feature, but it's not there. Could you do it
with subdags, or is your DAG not amenable to that?

You could use Condor-C to submit jobs to two different schedds. Then,
with the different schedds, you could set different MAX_JOBS_RUNNING.
You could queue up as many as you wanted of each kind of job in the
different schedds. The DAGMan is watching jobs in the original schedd,
and when jobs complete in the schedd-for-get-data or schedd-for-database,
the job status in the original-schedd will be updated.

A crude diagram:

DAGMan     Original-Schedd          schedd-for-get-data
 |              |   | |      condor-c      |
 +--------------+   |  --------------------+
                    |  
                    +----------------+
                       condor-c      |
                                 schedd-for-database


-Erik