Re: [HTCondor-users] request_memory

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 12/26/2013 3:57 PM, Rita wrote:
> I have a job with request_memory = 256 Megabytes but if I go over it will
> get held according to my SYSTEM_PERIODIC_HOLD policy. I would like to
> automatically triple the request_memory and then release the job. Is that
> possible to do?

Warning - this is off the top of my head, but I think the following 
would work or at least point you in the right direction.   Also I am 
assuming you are using HTCondor 8.0 or above.

I would expect your condor_config file has entries like the following:

  # Tell the schedd to hold jobs that have been restarted more than
  # 10 times, or if it uses more memory than it requested
  SYSTEM_PERIODIC_HOLD = JobRunCount > 10 || MemoryUsage > RequestMemory

  # Set a specific HoldReasonSubCode so if the job uses
  # more memory than request, so we can identify this particular
  # hold reason in automatic release expressions.  With the below
  # expression, the job's HoldReasonSubCode will be 1 if
  # the job is on hold due to memory issues, and 0 otherwise.
  SYSTEM_PERIODIC_HOLD_SUBCODE = MemoryUsage > RequestMemory

Then in your job submit file you could try something like:

 executable = foo
 # HoldReasonSubCode of 26 means SYSTEM_PERIODIC_HOLD became true,
 # and HoldReasonSubCode of 1 is configured in condor_config to mean
 # job was put on hold due to memory usage.
 periodic_release = HoldReasonCode =?= 26 && HoldReasonSubCode =?= 1
 # Request 256 MB ram unless MemoryUsage (from a previous run) is
 # already defined, in which case triple it for the next try.
 request_memory = ifthenelse(isUndefined(MemoryUsage),256,3*MemoryUsage)
 queue

Hope the above makes sense - the manual has documentation on all the 
above knobs and job classad attributes.

Note that if you have control of the configuration of the execute 
machines, consider having the execute machine itself put the job on hold 
if memory usage exceeds requested memory.  This way the job will be put 
on hold much sooner after memory usage is exceeded (perhaps even 
instantly).  For some HOWTO recipes, see
 https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=HowToLimitMemoryUsage

regards,
Todd