Re: [HTCondor-users] condor_submit_dag independent child processes [SOLVED]

[Labounek Renà <xlabou01@xxxxxxxxxxxxxxxxxx>]

I apologize for the spam. I have found the mistake in my code. The  
condor works perfect.
RL

Cituji Labounek Renà <xlabou01@xxxxxxxxxxxxxxxxxx>:

> Small correction. I have switched processes mprage_seg and  
> mprage2nii, because I have not deleted one output file from previous  
> testing run.
>
> The first mprage_seg is executed. After its finishing, mprage2nii  
> process starts.
>
> Other words, it is not paralelized optimally, because only 2 threads  
> are executed in time when 3 independent threads can be executed.
>
> Regards,
> Rene Labounek
>
>
> Cituji Labounek Renà <xlabou01@xxxxxxxxxxxxxxxxxx>:
>
> > Dear HT-condor Users,
> > I am trying to optimally paralelize our data processing pipeline  
> > via condor and I have found one issue which I am not able to solve...
> >
> > In my file.dag, I have 6 different processes and following PARRENT  
> > and CHILD edges:
> >
> > JOB data_convert  
> > /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-data_convert.condor
> > JOB dmri_preproc  
> > /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-dmri_preproc.condor
> > JOB mprage_bet  
> > /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-mprage_bet.condor
> > JOB mprage_seg  
> > /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-mprage_seg.condor
> > JOB mprage2diff  
> > /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-mprage2diff.condor
> > JOB mprage2mni  
> > /md1/NH-DHP/NH-DHP-002/condor_logs/NH-DHP-002-mprage2mni.condor
> > PARENT data_convert CHILD dmri_preproc mprage_bet
> > PARENT mprage_bet CHILD mprage_seg mprage2mni
> > PARENT dmri_preproc mprage_seg CHILD mprage2diff
> >
> > After data_convert, both CHILD precosses (dmri_preproc and  
> > mprage_bet) start. Process dmri_preproc takes much more time than  
> > other processes. Process mprage_bet is finished much earlier. When  
> > mprage_bet is finnished, the process mprage2mni is started first  
> > and mprage_seg is started after mprage2nii is finished and not  
> > imediatelly when mprage_bet is finished.
> >
> > and i have enough free threads in my pool to be able to compute all  
> > three processes at once.
> >
> > Does somebody have an idea why it behaves in that way. mprage_seg  
> > and mprage2mni should be independent on themselves.
> >
> > Yours faithfully,
> > Rene Labounek
> >
> > _______________________________________________
> > HTCondor-users mailing list
> > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/htcondor-users/
>
>
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/