Re: [Condor-users] restricting the number of jobs

["R. Kent Wenger" <wenger@xxxxxxxxxxx>]

On Mon, 12 Oct 2009, Dillman, Kimberley A wrote:

> I think this is because of "maxidle" versus "maxjob".
>
> I saw a similar thing when I used a small number for "maxidle" like 3.
>
> I had a dagman of 10 and tried to limit "maxidle" to 3. Since it 
> appears that dagman submits 5 jobs at a time, you still get 5 but not 10 
> since a job isn't "idle" until it's actually submitted. Since my dagman 
> submitted 5 jobs at a time, you can't get 3 so I get 5 but at least I 
> don't get all 10.

You can control the maximum number of submits per cycle with the
DAGMAN_MAX_SUBMITS_PER_INTERVAL configuration variable.  As you surmise, 
the default for this is five.  If you're trying for really fine-grained
control of the number of jobs in the queue, you may want to set a lower 
value.

> However, if I use "maxjobs" set to 3. I get just 3 submitted at a time 
> since dagman can figure out maxjobs prior to submitting them.
>
> At least I think that's how it works because that is what I interpreted 
> from the documentation and what I saw happen on my 10 job test dagman 
> script.

Yes, that's correct -- DAGMan doesn't consider jobs idle until they're 
submitted.  One thing to keep in mind is that maxidle is kind of a rough
setting.  Jobs that are running can become idle, so you're not guaranteed 
to never exceed the maxidle setting.  DAGMan never *removes* jobs from the 
queue to try to maintain the maxidle setting -- it just throttles how fast 
it submits them.

> However, the documentation appears to indicate that "maxjobs" will only 
> count each "job" in the dagman script and doesn't count each individual 
> "job" within a single submit script (i.e. queue 500 for instance would 
> count as 1 "job"). It does indicate that the "maxidle" option (since it 
> looks at each job in the queue separately for counting purposes) will 
> throttle a "single" job in dagman with many individual "jobs" (i.e. 
> queue xxx) since it counts them AFTER they are submitted so it looks at 
> them as individual jobs. Kind of confusing but that is why the exact 
> "maxidle" doesn't appear to work to the exact number since the dagman 
> submits the jobs in "groups" of x (5 in my case but I'm not sure exactly 
> where that comes from yet) and "maxidle" doesn't take affect until after 
> they are submitted to suppress additional submissions.

Yes, that's right.  Just keep in mind that maxjobs is a "harder" limit 
than maxidle is.

> Kind of confusing but it seems to work okay especially if you keep the 
> "maxidle" to something greater than whatever the single group submission 
> number is for a dag.
>
> If anyone can explain how this works in more detail, I would love to 
> hear about it to save some time on experimentation to figure it out. :-)

Okay, here's an explanation that hopefully makes some sense...

One thing to keep in mind with both maxidle and maxjobs is that they were
really designed to work in the situation where each submit file queues a 
single job.  DAGMan's ability to even handle submit files that queue more 
than one job is a pretty recent addition.  Also, DAGMan can only throttle 
things at the level of a submit file -- so if you have submit files that 
queue 10 jobs, the smallest increment that DAGMan can work with is 10 
jobs...

The difference in how maxjobs and maxidle count things is more an artifact 
of how the implementation works for multiple-job submit files than a 
design decision.

So the background on maxidle is this: we had implemented maxjobs, but a 
user who has a big pool, shared among a number of users, wanted something 
more flexible than maxjobs.  The problem was, they couldn't really predict 
ahead of time what maxjobs should be set to, because it depended on the 
load that other users were putting on their pool.  So eventually someone 
said, "how about if DAGMan keeps submitting jobs until the jobs aren't 
getting run?" and that was how the idea of maxidle was born.  The idea is
that you set maxidle to some fraction of the number of machines in your
pool, and DAGMan will keep feeding jobs in to keep the pool fully 
utilized, without flooding the queue with lots of jobs that won't run for 
a long time.  (We have users running 500k node DAGs, so they really don't 
want to submit, say 100k jobs at one shot.)

So if you want a strict limit on the number of jobs running, you should 
use maxjobs, but if you want to maximize the utilization of your pool, you 
should use maxidle.

Kent Wenger
Condor Team