Re: [Condor-users] Defining an exit script for condor jobs

[Jaime Frey <jfrey@xxxxxxxxxxx>]

On Oct 6, 2005, at 2:13 PM, Terrence Martin wrote:

> I asked this question a couple months ago but I wanted to put it out
> again because I did not follow up on the one response I got.
>
> My question was whether it is possible to have a script run on job  
> exit
> that can go beyond what the normal condor exit does in terms of  
> cleaning
> up areas. This is important in the current Open Science Grid  
> clusters I
> am working with since often user files are stored in temporary area  
> that
> condor does not necessarily know about. It would be nice to have this
> area cleared on exit.
>
> The answer I got was either use a wrapper or Dagman.
>
> The first solution does not work, that is if I follow the rules for
> USER_JOB_WRAPPER in the condor documentation to not have the wrapper
> fork a child and only call exec.  I can do that but it is not clear I
> should. What would be nice is that in addition to USER_JOB_WRAPPER  
> there
> was a USER_JOB_EXIT_SCRIPT which could define a script that performs
> certain cleanup steps on job exit.
>
> As far as DAGman, I am not sure how that would help. DAGman from the
> condor documentation is meta-scheduler that submits to condor. That
> sounds like it works on the outside between the user and condor.  The
> grid software I work with is already thick with schedulers to  
> condor and
> I cannot enforce what users make use of on that side. All I can  
> control
> is my condor queue and my worker nodes. Admittedly my knowledge of
> dagman extends to what I read here http://www.cs.wisc.edu/condor/ 
> dagman/
> but it does not sound like what I am looking for.
>
> I guess I have another option and try to be clever. Just before my  
> user
> wrapper drops to the actual job I could start a monitoring process  
> that
> watches for the job to exit and then try to cleanup. It would be  
> simpler
> and probably less error prone if condor could just trigger a cleanup
> process though.  This would also have to end up being an orphan  
> process
> since the parent calls an exec right after it spawns the monitor.

I see a few options available, none ideal:

1) Have the USER_JOB_WRAPPER clean up the files of the previous job.

2) If you have any control on the submit side, you can set a post- 
script in the job ad that will be run after the job. You can use  
SUBMIT_EXPRS to add it automatically to all jobs.

3) Have the USER_JOB_WRAPPER fork the job instead of exec'ing it. We  
don't mention this in the manual because it can be tricky to get  
right. The script has to not exit before the job, exit with the same  
status as the job, and catch SIGTERM and forward it to the job. If  
you run any standard universe jobs, there are several more signals  
the script has to catch and forward to the job. There may be some  
other details, but but those are the ones I can think of.

+----------------------------------+---------------------------------+
|            Jaime Frey            |  Public Split on Whether        |
|        jfrey@xxxxxxxxxxx         |  Bush Is a Divider              |
|  http://www.cs.wisc.edu/~jfrey/  |         -- CNN Scrolling Banner |
+----------------------------------+---------------------------------+