Mailing List ArchivesAuthenticated access |
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}
|
Mailing List ArchivesAuthenticated access |
|
|
Thank you, Emily ... as you can tell the challenge in this type of cases is what to do in case of a failure ... now, the second job may fail because the node crashed .. In the case the second job will wait until the node comes back (if ever?) ... should we
copy the processed training data back to "safe" storage just in case?
We are extremely interested in these AI driven workloads and want to do it "right".
Miron
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Emily Kooistra <a66@xxxxxxxxx>
Sent: Wednesday, March 18, 2026 9:00 To: htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx> Subject: Re: [HTCondor-users] Schedule jobs after each other on same node with shared scratch (GPU preproccessing) Hi Miron,
Good question, purging the data when a job fails would be fine it does require a full rerun but that should be okay. Or alternatively, have the job copy them out back to a specified destination to be able to restore from that. Emily On 3/18/26 13:59, Miron Livny via HTCondor-users wrote: > Emily, > > What do you expect to happen if the second job fails? Do you expect the data to stay there waiting for a rerun? > > In other words who will remove the training data? > > Miron > > > Sent from my iPhone > >> On Mar 18, 2026, at 07:54, Beyer, Christoph <christoph.beyer@xxxxxxx> wrote: >> >> ïHi, >> >> there are a lot of options to do so more or less elegant with condor on-board ressources. >> >> E.g. you could allow only one of theses jobs at a time per type and then put the claime-liftime on the GPU node very high this would automatically run these jobs one after another on this one node ... >> >> Best >> christoph >> >> >> >> -- >> Christoph Beyer >> DESY Hamburg >> IT-Department >> >> Notkestr. 85 >> Building 02b, Room 009 >> 22607 Hamburg >> >> phone:+49-(0)40-8998-2317 >> mail: christoph.beyer@xxxxxxx >> >> ----- UrsprÃngliche Mail ----- >> Von: "Erik Kooistra" <a66@xxxxxxxxx> >> An: "htcondor-users" <htcondor-users@xxxxxxxxxxx> >> Gesendet: Mittwoch, 18. MÃrz 2026 12:59:34 >> Betreff: [HTCondor-users] Schedule jobs after each other on same node with shared scratch (GPU preproccessing) >> >> Hi All, >> >> At NIKHEF we are seeing more and more GPU usage, and often those jobs >> have quite a long preprocessing stage to reformat the training inputs or >> other data files takes quite some time compared to the total runtime of >> the slot. >> >> As a result of this we end up with slots that do request a GPU but don't >> actually use it for a big part of the claimed period what results in non >> optimal GPU usage. >> >> Now copying this prepared data back to a network storage, and then copy >> it back to the scratch disk of the slot with a GPU is a bit waist full >> of network bandwidth. >> >> So i was wondering if it would be possible to have in a DAG or some >> other way, a CPU intensive preproccessing job running on a node with a >> GPU, and later in the process attaching the GPU to this slot or having >> a way to have a internal copy between the two jobs. >> >> Any other suggestions that would work with the current limitations of >> condor are also more then welcome, (by for example having a node local >> scratch and having some constraints the jobs run after each other, altho >> then you miss the cleanup that condor does of the scratch) >> >> Emily Kooistra >> NIKHEF >> _______________________________________________ >> HTCondor-users mailing list >> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a >> subject: Unsubscribe >> >> The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/ >> >> _______________________________________________ >> HTCondor-users mailing list >> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a >> subject: Unsubscribe >> >> The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/ > > _______________________________________________ > HTCondor-users mailing list > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a > subject: Unsubscribe > > The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/ _______________________________________________ HTCondor-users mailing list To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a subject: Unsubscribe The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/ |