[HTCondor-users] Queries regarding reset retries in rescue dag

[Vijay Chakravarty <vijay.chakravarty@xxxxxxx>]

Dear HTCondor experts,

We have been trying to make use of the configuration variable DAGMAN_RESET_RETRIES_UPON_RESCUE by setting it to True in a config file that we refer to with CONFIG command in the dag file. However, it doesn't seem to make any difference to the Retries. We have a simple RETRY Job <no. of retries> command that we want to reset when the number of retries are done and failure persists leading to writing of rescue file along with some PRE and POST scripts. We also don't find the variable set in the output dagman.out. 

In the backdrop of this problem, I have the following questions:

A.) Is there a conclusive way to make sure our config file is being read and the config variable being set to True besides checking for the variable in the output file? Or an alternative way to set this variable?

B.) The manual mentions the line "If the Rescue DAG file is generated before all retries of a node are completed, then the Rescue DAG file will also contain RETRY entries." 

1. I want to confirm that this "RETRY entries" is  just copying the RETRY lines from the dagman script. For instance, if my DAG file didn't have a RETRY command to begin with, would I still get a RETRY entry in the rescue? 
2. Secondly, the word "before": Our use case envisions using the latest rescue dag file generated after all the retries are completed for the first time. Once the retries are completed, we do a resubmission that uses the latest rescue file which we want to reset the retries for, so effectively run the number of retries again.  Is that not possible? What we find currently is that rescue files have only the RETRY line (same as that we pass in the actual DAG) It seems that it is copying the RETRY entry from original DAG and the reset never happens.
3. Also, it's my understanding that the rescue file should copy the original DAG PRE Job and POST lines verbatim in the rescue file from the dag file. However, we find that the rescue file has only the RETRY line after the header comments about failed DAG execution. I suspect since the Rescue file has no other line it is not working as expected. Maybe this can be resolved by setting DAGMAN_WRITE_PARTIAL_RESCUE to False. 

If we could understand how to set the Configuration variables, it would be very helpful.

Any insight will be highly appreciated.

Thanks!

Cheers,

Vijay Chakravarty