Re: [HTCondor-users] How to specify unique output for crondor jobs

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 10/19/2023 1:05 PM, Joseph Areeda wrote:

> Hi All,
>       
>       
>       We are having trouble debugging condor cron jobs and would like to
>       save stderr, stdout, and log files for each run.
>       
>       
>       It appears $(cluster), $(process) and $([ JobCurrentStartDate ])
>       do not change with each run. The log file appends its output but
>       error and out files overwrite.
>       
>       
>       Is there a variable we can use in the file specs that will change
>       with each run?
>       
>       
>       Best,
>       
>       
>       Joe and Iara
>       
>     
    
    Hi Joe and Iara,
    
    With your "crondor job", you can append to output and error files
    with each run (instead of the default behavior of overwrite) by
    adding the following to line to your submit file:
    
       erase_output_and_error_on_restart = false
       
    Or if you prefer, you can log each run instance output to individual
    files.  The trick here is to use the submit file macro that allows
    you to reference a job ClassAd _expression_ to define its value - the
    syntax for this is $$([ classad _expression_ ]). Yes, that is two
    dollar signs, followed by a parenthesis, followed by a square
    bracket, then the job attribute or _expression_.  
    
    Here is an example submit file showing output from each run of a
    crondor job into an individual file, using the job classad attribute
    "NumJobStarts" which is an integer starting at 0 that is incremented
    each time the job is launched:
    
    executable = /bin/echo
    arguments = Hello
    >
    cron_minute = *
    cron_hour = *
    cron_day_of_month = *
    cron_month = *
    cron_day_of_week = *
    output  = myjob.$$([NumJobStarts]).out
    queue
    
    After letting the above run for four minutes, I see the following
    files with "ls -l " (note they are created one minute apart):
    
    -rw-r--r-- 1 submituser submituser   13 Oct
      19 20:46 myjob.0.out
      -rw-r--r-- 1 submituser submituser   13 Oct 19 20:47 myjob.1.out
      -rw-r--r-- 1 submituser submituser   13 Oct 19 20:48 myjob.2.out
      -rw-r--r-- 1 submituser submituser   13 Oct 19 20:49 myjob.3.out
    
    Everything I mentioned above is documented in the man page for
    condor_submit, online at 
    
      
    https://htcondor.readthedocs.io/en/latest/man-pages/condor_submit.html
    
    
    Hope the above helps,
    Todd
    
    
    --
Todd Tannenbaum <tannenba@xxxxxxxxxxx>  University of Wisconsin-Madison
Center for High Throughput Computing    Department of Computer Sciences
Calendar: https://tinyurl.com/yd55mtgd  1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                   Madison, WI 53706-1685