Re: [Condor-users] Understanding Condor's "Claimed" state

[Steven Timm <timm@xxxxxxxx>]

On Wed, 29 Jun 2011, Jeff Ramnani wrote:

> Hello,
>
> I have a Condor pool with 10 dedicated compute nodes, and I'm having an issue 
> getting people's jobs scheduled the way I want.  Here's what's happening.
>
> If user1 has submitted a large batch of jobs, then users that submit jobs 
> after them aren't getting scheduled until after the first user's jobs are 
> completed, even if the users who came later have better priorities.
>
> One difference I've seen when this happens is that user1 who submits the 
> large batch of jobs does so as one job cluster that contains many jobs (in 
> this example, let's say 100 jobs. e.g. 1.0 .. 1.99), and user2 who has a 
> better priority, but submits later, does so as many clusters with one job 
> each (e.g. 2.0, 3.0, 4.0).
>
> Here's an example output of condor_q:
>
> ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
> 54713.0    user1         6/29 14:00   0+00:02:27 R  0   0.0  user1_job.py 
> 2011.0
> 54713.1    user1         6/29 14:00   0+00:02:19 R  0   122.1 user1_job.py 
> 2011.0
> 54713.2    user1         6/29 14:00   0+00:02:14 R  0   0.0  user1_job.py 
> 2011.0
> 54713.3    user1         6/29 14:00   0+00:02:06 R  0   0.0  user1_job.py 
> 2011.0
> ...
> 54713.99   user1         6/29 14:00   0+00:00:00 R  0   0.0  user1_job.py 
> 2011.0
> 54488.0    user2         6/29 12:03   0+00:00:00 I  0   732.4 user2_job.sh
> 54489.0    user2         6/29 12:03   0+00:00:00 I  0   732.4 user2_job.sh
> 54490.0    user2         6/29 12:03   0+00:00:00 I  0   732.4 user2_job.sh
> 54491.0    user2         6/29 12:03   0+00:00:00 I  0   732.4 user2_job.sh
> 54492.0    user2         6/29 12:03   0+00:00:00 I  0   732.4 user2_job.sh
>
>
>
> User2 has a better priority, so I would expect user2's job 54488.0 to be 
> scheduled on the first available machine when one of user1's jobs is 
> completed, but that's not what's happening.  It seems like user1 has a 
> "claim" on the machines that lasts longer than an individual job.  I've read 
> the following manual pages:
> http://www.cs.wisc.edu/condor/manual/v7.4.4/2_7Priorities_Preemption.html
> http://www.cs.wisc.edu/condor/manual/v7.4.4/3_4User_Priorities.html
> http://www.cs.wisc.edu/condor/manual/v7.4.4/3_5Policy_Configuration.html
> and I'm still not 100% sure I understand how jobs are scheduled in this 
> situation.  I've found the configuration setting for CLAIM_WORKLIFE in the 
> manual which states, "If provided, this expression specifies the number of 
> seconds during which a claim will continue accepting new jobs."  This leads 
> me to the following questions.
>
> * How long does a user's "claim" last on a machine?

 Others know more technical detail here but if CLAIM_WORKLIFE
is not set, the claim is infinite unless preemption kicks the user off.
If you set CLAIM_WORKLIFE, then it is just the duration of the claim
that you specify.

> * Does a job cluster keep a "claim" open on a machine until all its jobs are 
> completed?

The size of the cluster doesn't make a difference.
but if a node is claimed for a single user then as long as that user
has jobs in the queue it will keep on executing the jobs of that user,
whether in one cluster or many.

CLAIM_WORKLIFE is a very valuable setting.  I set it to 3600 seconds.
I've never quite understood why the default is infinity.

Steve


> Any help is appreciated.
>
> Sincerely,
>
> Jeff Ramnani
>
> This e-mail and any attachments may contain information that is confidential 
> and proprietary and otherwise protected from disclosure. If you are not the 
> intended recipient of this e-mail, do not read, duplicate or redistribute it 
> by any means. Please immediately delete it and any attachments and notify the 
> sender that you have received it in error. Unintended recipients are 
> prohibited from taking action on the basis of information in this e-mail or 
> any attachments. The DRW Companies make no representations that this e-mail 
> or any attachments are free of computer viruses or other defects.

--
------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525
timm@xxxxxxxx  http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Group Leader.
Lead of FermiCloud project.