Re: [HTCondor-users] Improving negotiator match rate

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 1/14/2021 10:19 AM, Zohar Kol wrote:

Hello,

Setup description:

Condor version 8.8.9

1.5k physical machines

18.9k cores

~200TB RAM

~50k jobs in queues

4 schedulers machines (16 cores 128GB RAM each)

Negotiator / Collector machine runs on 4 core 16GB RAM machine.

Condor configuration description:

Accounting groups quotas

Pslot preemption enabled

All partitionable slots are running within Docker universe

Issue description:

We experiencing a slow job matching rate ~15 per second when the cluster is ~50% idle.

Can anyone share their tips on how to improve this rate?

Thanks,

Zohar

    Hi Zohar,
    
    For most HTC workloads, a matching rate of 15 per second is
    reasonable because the matching rate is different than the rate at
    which jobs can start running.  When the schedd is given a matching
    slot by the negotiator, the schedd will keep reusing that slot for
    job after job after job without needing any addition matches from
    the negotiator.  The only time a slot needs to be matched again is
    when a) the slot is preempted, b) the CLAIM_WORKLIFE time has
    expired (default of 20 minutes), or c) the schedd unclaims the slot
    because it no longer has any jobs queued that match the slot.   
    Besides the Manual, some details can be found in this recorded
    HTCondor Architecture workshop presentation at
    https://indico.cern.ch/event/936993/contributions/4022092/
    
    Thus I am wondering why match time is impacting you...  some
    thoughts:
    
    Are you constantly submitting many short jobs in small bursts?  For
    example, say you are submitting a pile of 30 second jobs every 60
    seconds.  In this case, perhaps the schedd gets a bunch of matches
    and claims a bunch of slots, but ends up releasing these slots
    before the next pile of jobs is submitted because the queue is
    empty.  And then when the next pile of jobs is submitted 60 seconds
    later, the schedd has to wait for the negotiator to match slots
    again.  If this is your scenario, you can tell the schedd to "hold
    on" to slot for X amount of seconds even if there are no jobs
    remaining in anticipation that more jobs will be submitted soon.  To
    do this, use the "keep_claim_idle" setting in your job submit file -
    the man page for condor_submit discusses the "keep_claim_idle"
    setting here:
    https://htcondor.readthedocs.io/en/latest/man-pages/condor_submit.html. 
    
    
    What is the setting on your pool for CLAIM_WORKLIFE ?  The default
    is 1200 seconds (20 minutes).  Did you change it to be something
    much smaller?  
    
    Is there a lot of slot preemption occurring in your pool? 
    
    Hope the above helps
    Todd