Re: [HTCondor-users] Dynamic DAGs

[John N Calley <calley_john_n@xxxxxxxxx>]

Dmitry,

   I have a job like this. It’s fairly easy to do, but you have to use a SUBDAG. There are two separate mechanisms within condor to include a dag as a node in a parental dag. There’s the SUBDAG mechanism and the SPLICE mechanism. When you don’t know the structure of the child dag when you submit the parental dag, you have to use the SUBDAG mechanism. It’s much nicer to use SPLICE when you can, but in the scenario you outlined it is not possible.

 

John

 

-----------------------------------------------------

John N. Calley, Ph. D.

Ttx Genetics and Bioinformatics

Eli Lilly and Co.

Lilly Corporate Center

Indianapolis, IN 46285

 

(317) 433-3399

calley_john_n@xxxxxxxxx

 

From: htcondor-users-bounces@xxxxxxxxxxx [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Dmitry Grudzinskiy
Sent: Wednesday, June 12, 2013 3:41 PM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Dynamic DAGs

 

Hello,

My name is Dmitry and our company is currently looking at different solutions for our cluster. We are using Torque at the moment and not very unhappy with it.

HTCondor seems very attractive specifically because of DAG functionality that we would utilize a lot.

 

But the workflows that we have may require dynamic DAGs meaning that the graph may change it's structure during runtime and there is no way for us to know it before we start.

For example node A generates number of files which is unknown before we start. After the job is done we process these files in parallel and the number of jobs should be equal to the number of files - each job processing one file. Then after this n jobs are done the DAG may continue and have other jobs to run.

 

What would be the best way to solve this?

Thank you,

Dmitry