[HTCondor-users] DAGman: job info propagation and delayed child jobs with still active parents possible?

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi all,

while a DAG is static(?), is there a way to propagate information (by 
class ads or so) from a running parent job to child jobs started 
underneath it?

E.g., a user wants to start a dask/spark master as a job; the job 
initializes itself probing random port for a free one and then starts 
the dask master process listening on the found port. Now the user wants 
to chirps the found port as new class ad to the parent master job.

After the parent master job is up and running, the user wants to 
automatically spawn a number of child worker jobs, that need the info 
where on what port the dask parent master is listening.

Could something like that be realized as a DAG...?

Can a child job get spawned after a parent job has reached a certain 
point in its life cycle (but keeps running, i.e., has not exited as such)?
And would there be a good way to propagate information from the parent 
job to child jobs as arguments or so when they are initialized/started? 
A dag pre script would only get executed after the parent job would have 
concluded, or?

Or is a DAG maybe the wrong approach and a small demon scraping a user's 
job ads for the required info to appear and then spawning worker jobs 
would be the way to go...?

Cheers and thanks for ideas,
  Thomas

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature