Re: [HTCondor-users] Are evicted jobs memory requirements automatically adjusted?

[Max Fischer <mfischer@xxxxxxxxxxxxxxxxxxxx>]

Hi Chris,

do you want to enforce such a policy as administrator of your condor pool or do you want to be able to tell your users how to do this if they require it?

If you want this enforced for all, you should set your startd nodes to make the distinction. Use *Requirements* _expression_, *Start* _expression_ or others to accept jobs only if the higher value of the MemoryUsage or RequestMemory matches.
As TJ mentioned, jobs with mismatched resource estimations are something you may tolerate but should generally not encourage. Consider changing the *Rank* _expression_ to punish jobs depending on how much they overdraw their estimated RequestMemory. You might want to go so far as to use *SYSTEM_PERIODIC_HOLD* or *SYSTEM_PERIODIC_REMOVE* to outright cancel jobs that have restarted too often and overdrawn their estimation.

If you want users to make the distinction if they feel it is needed, it's up to them to define RequestMemory as a hard value or a dynamic one

-Max

On 02/07/2013 06:52 PM, Chris Filo Gorgolewski wrote:

I;m new to Condor, but I believe that it is that by default ImageSize is not directly used in the Requirements field used for matching. It is used through RequestMemory when user did not specify it explicitly. Therefore when user specifies "request_memory", even if ImageSize is being updated is is not used for matching.

Best,

Chris

On 7 February 2013 18:47, Ian Chesal <ian.chesal@xxxxxxxxx> wrote:

TJ,

The matching is driven by ImageSize, no? And that references RequestMemory, which should be updated to 2000 MB after the job is bumped back to the queue.

(Sorry, I don't have a Condor queue on hand to check that ImageSize _expression_ that gets written out when you use request_memory in your submission file so I'm working from memory here...)

- Ian

On Thu, Feb 7, 2013 at 8:36 AM, John (TJ) Knoeller <johnkn@xxxxxxxxxxx> wrote:

As HTCondor is configured by default it's 5a.   If you want 5b the job should have

    RequestMemory = Max(500, MemoryUsage)

But you probably don't really want that.   Jobs that have MemoryUsage > RequestMemory should
probably not match anything until the user intervenes and sets RequestMemory correctly. 

Keep in mind that jobs that lie about memory usage like this are  waste time on larger and
larger slots as they hunt for a slot that can accommodate them.  

-tj

On 2/6/2013 8:48 AM, Chris Filo Gorgolewski wrote:

When a job gets suspended and evicted it's memory requirements adjusted by the actual values measured during it's execution?

For example:

1. I job with "request_memory = 500" is being submitted.

2. The job gets assigned to a node, starts running.

3. It allocates 2000MB.

4. The job gets suspended and evicted (reason does not really matter in this example)

5a. Job gets resubmitted to another node with memory requirement of 500MB (as requested by the user)

5b. Job gets resubmitted to another node with memory requirement of 2000MB (as measured during previous execution)

So which one is it 5a or 5b?

If 5a how can I achieve 5b (increasing the memory requirements by the maximum memory allocation from the previous failed execution)?

This is especially important when using policy described here: https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=HowToLimitMemoryUsage

Thanks in advance!

Best,

Chris

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/