Re: [HTCondor-users] Edit job requested memory after job enters hold state

[Tomer Pearl <tomerp@xxxxxxxxxxx>]

Thanks Jason for your help!
_____________________________
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Jason Patton <jpatton@xxxxxxxxxxx>
Sent: Monday, March 1, 2021 4:22:13 AM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Edit job requested memory after job enters hold state

Tomer,

Assuming these are jobs you have submitted and you have control over their submit files, you can add a "periodic_release" statement that releases your jobs if they go on hold for being out of memory, and you can set "request_memory" to an expression that sets it to a multiple of itself each time it is resubmitted (and to a default if it hasn't been defined yet, i.e. on first submission). See slides 103 and 105 from the user tutorial from this recent HTCondor workshop for some examples: https://indico.cern.ch/event/936993/contributions/4022073/

Jason



On Sun, Feb 28, 2021 at 9:33 AM Tomer Pearl <tomerp@xxxxxxxxxxx<mailto:tomerp@xxxxxxxxxxx>> wrote:
Hi everyone,

I have containers that suffers from out of memory error and I don't want to configure swap on my executors. Currently to handle this situation I'm running with a very large RequestMemory value for my jobs.

Is there a mechanism in condor which allows me to "catch" jobs that enters HOLD state, edit their request memory (to a larger value) and resubmit automatically?

Multiple iterations of this process would be great but even changing the amount ones is enough for me.
I tried the hooks mechanism but couldn't find a way to be invoked when my job enters hold state.

Thanks,
Tomer.

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx<mailto:htcondor-users-request@xxxxxxxxxxx> with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/