Re: [HTCondor-users] condor_submit_dag independent child processes

["Nathan W. Panike" <nathan.panike@xxxxxxxxx>]

I suggest looking atÂDAGMAN_SUBMIT_DEPTH_FIRST in the HTCondor manual:Âhttps://research.cs.wisc.edu/htcondor/manual/latest/3_5Configuration_Macros.html#29939. The default is that dagman runs nodes in breadth-first order, so that once all the parent dependencies are satisfied, a job goes to the "back" of the queue and jobs that are already on the queue will be run first.

On Wed, Sep 20, 2017 at 6:20 AM, Labounek Renà <xlabou01@xxxxxxxxxxxxxxxxxx> wrote:

Small correction. I have switched processes mprage_seg and mprage2nii, because I have not deleted one output file from previous testing run.

The first mprage_seg is executed. After its finishing, mprage2nii process starts.

Other words, it is not paralelized optimally, because only 2 threads are executed in time when 3 independent threads can be executed.

Regards,
Rene Labounek


Cituji Labounek Renà <xlabou01@xxxxxxxxxxxxxxxxxx>:

Dear HT-condor Users,
I am trying to optimally paralelize our data processing pipeline via condor and I have found one issue which I am not able to solve...

In my file.dag, I have 6 different processes and following PARRENT and CHILD edges:

JOB data_convert /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-data_convert.condor
JOB dmri_preproc /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-dmri_preproc.condor
JOB mprage_bet /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-mprage_bet.condor
JOB mprage_seg /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-mprage_seg.condor
JOB mprage2diff /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-mprage2diff.condor
JOB mprage2mni /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-mprage2mni.condor
PARENT data_convert CHILD dmri_preproc mprage_bet
PARENT mprage_bet CHILD mprage_seg mprage2mni
PARENT dmri_preproc mprage_seg CHILD mprage2diff

After data_convert, both CHILD precosses (dmri_preproc and mprage_bet) start. Process dmri_preproc takes much more time than other processes. Process mprage_bet is finished much earlier. When mprage_bet is finnished, the process mprage2mni is started first and mprage_seg is started after mprage2nii is finished and not imediatelly when mprage_bet is finished.

and i have enough free threads in my pool to be able to compute all three processes at once.

Does somebody have an idea why it behaves in that way. mprage_seg and mprage2mni should be independent on themselves.

Yours faithfully,
Rene Labounek

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--

Nathan Panike