Re: [HTCondor-users] Issue with NEGOTIATOR_PRE_JOB_RANK â Dedicated slots not prioritized

[Cole Bollig <cabollig@xxxxxxxx>]

Hi Ritik,

Sorry for the delay in responding. I have been messing around recreating your setup locally. After some discussion with my colleagues and experimenting, the pre job rank _expression_ is working correctly (i.e. The dedicated slots are given preference for matchmaking over the shared slots). However, there are two aspects of the matchmaking that are undesirable for your setup:

1. Jobs can still match to shared slots if the job fails to match to any idle dedicated slots due to requirements in the job or the slot.
2. When a job is matched to a partitionable slot, the AP (submit host) Schedd will have a claim on the whole slot for a short time while it tries to match as many similar jobs as possible (from the same user/autocluster) to the partitionable slot before returning the unclaimed resources. During this time, the same schedd or different ones will be getting more matches to slots from the negotiator which may include shared slots.

 am not sure if you want to go this direction, but using static slots instead of partitionable slots may solve the latter issue (which we think is the main one) since the slot dedicated slots will be fully matched immediately in the negotiator. 

-Cole Bollig

---

From: Ritik Raj <ritik.raj@xxxxxxxxxxxxxxxxx>
Sent: Tuesday, July 22, 2025 1:21 AM
To: Cole Bollig <cabollig@xxxxxxxx>
Cc: htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>; Aditya Rawat <aditya.rawat@xxxxxxxxxxxxxxxxx>; Pawan Mishra <pawanmishra@xxxxxxxxxxxxxx>; Pratham Kapoor <pratham.kapoor@xxxxxxxxxxxxxxxxx>
Subject: Re: [HTCondor-users] Issue with NEGOTIATOR_PRE_JOB_RANK â Dedicated slots not prioritized

 

Hi Cole, 

1. We are currently using partitionable slots on our Execution Points.

     Each Execution Point has two partitionable slots:

• One is marked with SlotPurpose = "dedicated" and is reserved for a specific user.

• The other is marked as "shared" and is available for use by anyone in the pool.

2. The _expression_ seems to be working correctly. When I run the command, I am getting 100000 for dedicated and 0 for the shared one.
Let me know if you'd like me to check anything else.

On Tue, Jul 22, 2025 at 1:17âAM Cole Bollig <cabollig@xxxxxxxx> wrote:

Hi Ritik,

A few various comments/questions:

1. Are you using static or partitionable slots on your Execution Points?
2. The _expression_ seems correct but to be safe do you see the larger number if you run the following command: condor_status -af SlotPurpose 'ifThenElse(SlotPurpose =?= "dedicated", 100000, 0)'

-Cole Bollig

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Ritik Raj via HTCondor-users <htcondor-users@xxxxxxxxxxx>
Sent: Saturday, July 19, 2025 5:40 AM
To: htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Cc: Ritik Raj <ritik.raj@xxxxxxxxxxxxxxxxx>; Aditya Rawat <aditya.rawat@xxxxxxxxxxxxxxxxx>; Pawan Mishra <pawanmishra@xxxxxxxxxxxxxx>; Pratham Kapoor <pratham.kapoor@xxxxxxxxxxxxxxxxx>
Subject: [HTCondor-users] Issue with NEGOTIATOR_PRE_JOB_RANK â Dedicated slots not prioritized

 

Hi,

We are encountering a scheduling problem in our HTCondor pool. We have marked certain slots with SlotPurpose="dedicated" and expect that jobs should always be matched to these dedicated slots first, using shared slots only when no dedicated slots are free. In practice, however, we observe that jobs are sometimes scheduled on shared slots even when idle dedicated slots exist. 

Expected vs. Observed: Our intended behavior is that idle dedicated slots get claimed preferentially. In other words, the negotiator should sort matching slots so that any slot with SlotPurpose == "dedicated" has the highest priority, regardless of other factors. Only if all dedicated slots are busy would a job be placed on a shared slot. Instead, we see jobs being assigned to shared slots while dedicated slots remain idle. This suggests that our NEGOTIATOR_PRE_JOB_RANK _expression_ is not having the intended effect on slot ordering.

Configuration (relevant excerpts): We have set up custom attributes and ranking logic as follows 

• Access Point (Submit machine) config: No special configuration; jobs are submitted normally without extra requirements or ranks.
• Execution Point (Execute host) config: We added a SlotPurpose attribute to each slot:
  STARTD_ATTRS = $(STARTD_ATTRS) SlotPurpose  
  SLOT_TYPE_1_SlotPurpose = "dedicated"  
  SLOT_TYPE_2_SlotPurpose = "shared"  
• Central Manager (Negotiator) config: We set:
  NEGOTIATOR_PRE_JOB_RANK = ifThenElse(SlotPurpose =?= "dedicated", 100000, 0)

Questions and Next Steps: Despite this configuration, the behavior is not as expected. We have confirmed that the SlotPurpose attribute appears in the slot ClassAds.

• NEGOTIATOR_PRE_JOB_RANK usage: We assumed this is the right mechanism for slot prioritization. Are there alternative approaches? 

Any advice on why our NEGOTIATOR_PRE_JOB_RANK setup isnât working, or suggestions for alternate configurations to enforce dedicated-slot usage, would be greatly appreciated. Thank you for your help.

Sincerely,

Ritik Raj

Open Futures

CONFIDENTIALITY NOTE:
This message contains confidential information of Open Futures and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.

CONFIDENTIALITY NOTE:
This message contains confidential information of Open Futures and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.