Re: [HTCondor-users] Edit job requested memory after job enters hold state

[Jason Patton <jpatton@xxxxxxxxxxx>]

Tomer,

Assuming these are jobs you have submitted and you have control over their submit files, you can add a "periodic_release" statement that releases your jobs if they go on hold for being out of memory, and you can set "request_memory" to an _expression_ that sets it to a multiple of itself each time it is resubmitted (and to a default if it hasn't been defined yet, i.e. on first submission). See slides 103 and 105 from the user tutorial from this recent HTCondor workshop for some examples: https://indico.cern.ch/event/936993/contributions/4022073/

Jason

On Sun, Feb 28, 2021 at 9:33 AM Tomer Pearl <tomerp@xxxxxxxxxxx> wrote:

Hi everyone,

I have containers that suffers from out of memory error and I don't want to configure swap on my executors. Currently to handle this situation I'm running with a very large RequestMemory value for my jobs.

Is there a mechanism in condor which allows me to "catch" jobs that enters HOLD state, edit their request memory (to a larger value) and resubmit automatically?

Multiple iterations of this process would be great but even changing the amount ones is enough for me.
I tried the hooks mechanism but couldn't find a way to be invoked when my job enters hold state.

Thanks,

Tomer.

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/