Re: [HTCondor-users] Preemption while there are free slots

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 12/22/2020 6:34 AM, Zohar Kol wrote:

Hi,

CondorVersion 8.8.9

Setup includes:

1. Accounting group quota
2. All dynamic slots
3. allow_pslot_preemption
   

We see a lot of job evictions while there are plenty of free resources.

"Job was evicted" is followed by "Job executing on host..." in the next cycle.

This happens while there are machines that can serve the requirements of incoming job without preempting.

What configuration is the relevant place to handle this situation?

We've tried NEGOTIATOR_PRE_JOB_RANK and PREEMPTION_RANK but those evaluate the job and partitionable slot classad which have no knowledge of negotiator classad like  LastNegotiationCycleMatches<X>.

Any ideas?

    Hi Zohar,
    
    I did not test any of my guesses below, so I may be heading in the
    wrong direction, but my initial guess is the default settings for
    both NEGOTIATOR_PRE_JOB_RANK and NEGOTIATOR_POST_JOB_RANK are not
    good defaults if you set ALLOW_PSLOT_PREEMPTION=True.
    
    The HTCondor Manual, in the section that talks about
    ALLOW_PSLOT_PREEMPTION, gives this vague warning that I think may be
    relevant to your problem (see https://tinyurl.com/yct4tuks) : "When
      multiple partitionable slots match a candidate job and the various
      job rank expressions are evaluated to sort the matching slots, the
      ClassAd of the partitionable slot is used for evaluation. This may
      cause
      unexpected results for some expressions, as attributes such as
    RemoteOwner
      will not be present in a partitionable slot that matches
      with preemption of some of its dynamic slots."
    
    The default value for NEGOTIATOR_(PRE|POST)_JOB_RANK on my laptop
    (running HTCondor v8.9.10, but my guess is it is essentially the
    same in v8.8.9) are:
    
    NEGOTIATOR_PRE_JOB_RANK = (10000000 *
      My.Rank) + (1000000 * (RemoteOwner =?= UNDEFINED)) - (100000 *
      Cpus) - Memory
      NEGOTIATOR_POST_JOB_RANK = (RemoteOwner =?= UNDEFINED) *
      (ifthenElse(isUndefined(KFlops), 1000, Kflops) - SlotID -
      1.0e10*(Offline=?=True))
    
    I am guessing the clause "(RemoteOwner =?= UNDEFINED)" is causing
    the problem for you, since pslots never have RemoteOwner defined.  
    Normally this is all fine, but by also changing
    ALLOW_PSLOT_PREEMPTION to True, that means that a matched pslot that
    does not have enough free resources will start preempting dynamic
    slots (dslots) until the needed resources are available.  Thus I
    suggest changing that clause to instead sort such that pslots which 
    already have enough resources free to run the job without preemption
    are preferred; one way to do this is replace the RemoteOwner clause
    to instead  be "(WithinResourceLimits =?= True)".  
    
    So TL;DR, on your central manager, try setting the following knobs 
    (and then do a condor_reconfig):
    
    NEGOTIATOR_PRE_JOB_RANK = (10000000 *
      My.Rank) + (1000000 * (WithinResourceLimits
      =?= True)) - (100000 * Cpus) - Memory
      NEGOTIATOR_POST_JOB_RANK = (WithinResourceLimits
      =?= True) * (ifthenElse(isUndefined(KFlops), 1000, 
      
    Again, the above is just a guess...  But if it solves your
    problem please let us know, and we could improve the defaults for
    these knobs appropriately, or at least make the vague warning in the
    HTCondor Manual more helpful with an example.
    
    Hope the above helps,
    regards,
    Todd