Re: [HTCondor-users] run a script when all jobs complete

Mailing List Archives Authenticated access	UW Madison Computer Sciences Department Computer Systems Lab

Hey,

I wasn't aware of DAGMan, so I wrote a log processor of the condor log in python:

Â class LogProcessor(object):Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂÂ

Â Â Â """Processor for condor logs.Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂÂ

Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â

Â Â Â This is a wrapper over HTCondor's EventIterator and keeps track ofÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â

Â Â Â state of all the submitted jobs. It can be used as a context managerÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â

Â Â Â in the following way:Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂÂ

Â Â Â with LogProcessor('job.log', num_jobs) as processor:Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â

Â Â Â Â Â while not processor.done():Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂÂ

Â Â Â Â Â Â Â processor.process()Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂÂ

Â Â Â At the exit, it will log all the details about the submitted job.Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂÂ

Â Â Â """Â Â ÂÂ

Internally, it essentially calls `htcondor.read_events` from the python api on the condor log. Might be helpful.Â

Sandeep.

On Wed, Mar 7, 2018 at 3:21 PM, Zach Miller <zmiller@xxxxxxxxxxx> wrote:

One fairly straightforward way would be to use DAGMan:
Â Â Â Â http://research.cs.wisc.edu/htcondor/manual/v8.7/2_10DAGMan_Applications.html

Your jobs don't have any dependencies, but DAGMan allows you to specify a "FINAL" node that gets run after everything else.

There's some other advantages offered by dagman, such as retrying failed jobs, recovery of a partially-completed run, throttling job submission (if you need it), running scripts before and/or after each job, etc.

Cheers,
-zach

> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf
> Of Justin Killebrew
> Sent: Wednesday, March 07, 2018 2:10 PM
> To: htcondor-users@xxxxxxxxxxx
> Subject: [HTCondor-users] run a script when all jobs complete
>
> Hello.Â I need to run a python (or bash) script when all jobs of a
> particular submission are complete.Â Should I use hooks and
> HOOK_JOB_CLEANUP?Â If so, could someone point me to a simple example since
> I have not been able to configure it properly.Â Is there
> another/better/easier way?
>
> Running HTCondor 8.6.8 on Rocks 7.0
>
> Thanks,
> JK
>
> _____________________________
> Justin Killebrew
> Senior Programmer/Analyst
> Zanvyl Krieger Mind/Brain Institute
> Johns Hopkins University
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

Mailing List Archives

Authenticated access

Re: [HTCondor-users] run a script when all jobs complete