[Condor-users] Machine rank and job priorities ... again

[Horvátth Szabolcs <szabolcs@xxxxxxxxxxxxx>]

Hi,

This might be a forum classic and definitely a topic raised multiple 
times but I still can't get my head around job priorities / user 
priorities and machine rank.

My expectations are the following (they may be completely false though):

- Job priority only affects job ordering for a single user on a single 
scheduler.
- When different users' jobs are running than user priority decides the 
amount of VMs a user gets. By default it should be equal for all users, 
modified by historical data. If PRIORITY_HALFLIFE is very low than it 
should be more or less equal all the time (until the user priority is 
modified manually).
- If there are jobs with higher machine rank than these jobs are 
"preferred" by the startds, so they should run before *ANY* other lower 
ranked jobs. There might be a small time window when existing claims 
still allow lower ranked jobs to run but it should all settle quickly if 
CLAIM_WORKLIFE is set.
- If machine rank is higher but job priority is lower than other jobs' 
of a user might still execute before jobs with higher rank. Both higher 
rank *AND* higher job priority should make sure that a job runs before 
all other (less important) jobs of any user.

Now, this is the behaviour I don't get. All works well for a single 
user, job priority orders job execution reliably.
Problems arise when multiple users submit jobs to the same scheduler: in 
this case machine rank does make a slight difference
but nothing drastic like "no other jobs run until we're done" policy. I 
see more VMs occupied by the user with the higher ranked jobs for a 
while but with time other lower ranked jobs take the VMs away even when 
higher ranked jobs wait in the queue.

Which attributes might affect the matchmaking process to behave the way 
I described?
Do I still misunderstand the way priorities and rank works?
Should I get rid of the computers and become a gardener?

Thanks for any tips, pointers, exotic flower seeds in advance!

Cheers,
Szabolcs