Re: [Condor-users] Dagman Newbie Questions

["R. Kent Wenger" <wenger@xxxxxxxxxxx>]

On Thu, 9 Oct 2008, Jeremy Yabrow wrote:

> It seems that condor_q only shows the dag jobs that running or that CAN 
> run.  Jobs that are blocked due to prequisite jobs not done yet, are not 
> shown.  Also, if I keep the jobs in the system and attempt to re-run 
> them (using condor_hold & condor_release), then the dependencies are 
> not obeyed during this subsequent "run"

Yes, that's right.  Once a job is submitted, DAGMan doesn't do anything to 
it (besides removing it if you remove the condor_dagman job itself).

> If I understand what's going on, the dagman job appears to be simply a 
> job that submits other jobs, and the downstream jobs are not even 
> submitted until their prerequisite jobs have run.  Subsequent runs can 
> only be run again with the .rescue file.  Is this correct?

Well, the first part of this is basically correct.  DAGMan does a little
more than just submitting jobs, but yes, jobs whose prerequisities are not
satisfied are not submitted to Condor.

As far as the rescue DAG goes, though, you only get a rescue DAG if the 
workflow fails or if you condor_rm the condor_dagman job.  If you do run
a rescue DAG, you don't re-run all of the jobs, only the ones that didn't
finish (or were not run at all) the first time around.

If you want to re-run a DAG from scratch, you need to do

    condor_submit_dag -f <whatever>.dag

This will re-run the whole DAG regardless of whether it succeeded the 
first time.

> This has consequences for us because in our business, deadlines are 
> critical and resource utilization must be maximized.  So progressive 
> estimates of completion and remaining work are necessary.  We need all 
> nodes of the entire DAG to be present in the system to estimate resource 
> use, even though many of the nodes may be blocked waiting for 
> prerequisite nodes.  Jobs that submit other jobs are a nasty surprise 
> for our resource managers.  Also re-running of any node and its 
> dependent nodes is fairly common and is often done many times during 
> pipeline troubleshooting-we don't want to have to re-submit the entire 
> DAG several times in separate runs because there may be long-running 
> nodes in the DAG we want to continue in parallel while we're working on 
> other "broken" nodes.

As far as estimates of completion go, you can get some idea by looking at 
the dagman.out file, where you'll find periodic updates like this:

7/10 17:20:40  Done     Pre   Queued    Post   Ready   Un-Ready   Failed
7/10 17:20:40   ===     ===      ===     ===     ===        ===      ===
7/10 17:20:40     1       0        0       0       2          1        0

Of course, this doesn't account for differences in resource usage between 
different nodes of the DAG.

You can have DAGMan re-try nodes if you want, but that doesn't allow you 
to manually re-start failed nodes.

If you absolutely have to have all of the jobs in the queue, though, I 
don't see any way to do this with DAGMan.

> It looks to me like we'd have a hard time getting Condor/Dagman to 
> support these needs.  I'd love any advice / comments you might have on 
> this.

Well, having DAGMan submit all of the jobs and then put them on hold if 
they're not ready (or something along those lines) would be a really
fundamental change in DAGMan, and I don't see that happening.  (And for
many users, the fact that not all of the jobs go into the queue right
away is a benefit, because it decreases the load on the Condor central 
manager and the schedd.)

Sorry I haven't been of more help...

Kent Wenger
Condor Team