Re: [HTCondor-users] Python Binding for SPLICE

["Michaloliakos,Ioannis" <ioannis.michalol@xxxxxxx>]

Hello,

Thank you for the reply. So, I want to combine a collection of .dag files into a single DAG and then add a few children post-processing nodes. I have written the following toplevel DAG which accomplices this:

# begin combined.dag

SPLICE H1 pipeline-1.dag DIR ../pipeline-1

SPLICE H2 pipeline-2.dag DIR ../pipeline-2

SPLICE H3 pipeline-3.dag DIR ../pipeline-3

...

JOB PPROC1 PPROC1.sub DIR ../pproc1

VARS PPROC1 ...

PARENT H1 H2 H3 CHILD PPROC1

JOB PPROC2 PPROC2.sub DIR ../pproc2

VARS PPROC2 ...

PARENT PPROC1 CHILD PPROC2

.

.

.

The number of pipeline .dag's and the number of post-processing nodes (as well as the values of their variables) is not fixed, so I want to be able to write this dag file programmatically, and I am trying to achieve the above result using the HTCondor Python Bindings. I believe I could use a SUBDAG instead of a SPLICE, but reading the HTCondor manual I got the impression that I should try to use a SPLICE whenever possible.

Best regards,

Ioannis

Message: 4
Date: Fri, 4 Mar 2022 16:10:18 -0600
From: Jason Patton <jpatton@xxxxxxxxxxx>
To: htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] Python Binding for SPLICE
Message-ID: <2299a789-c5cb-4cca-9d36-42e3b5fbb451@xxxxxxxxxxx>
Content-Type: text/plain; charset="utf-8"; Format="flowed"

Hi Ioannis,


Can I ask what is the use case? For example, do you have two DAG objects and want to add the whole inner DAG in as a child of one of the outer DAG nodes?


Thanks


Jason Patton

---

From: Michaloliakos,Ioannis
Sent: Friday, March 4, 2022 4:28 PM
To: htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: Python Binding for SPLICE

 

Hello,

Is there a way to create a SPLICE in a dag using the Python API? I see the option to create a subdag with `dag.child_subdag()`, but I don't see any method of the form `dag.child_splice()`.

Thank you,

Ioannis