Re: [HTCondor-users] How to limit number of machines used in a long list of queues

[Ralph Finch <ralphmariafinch@xxxxxxxxx>]

Excellent, this does it. I also see on further reading of the HTCondor manual that DAGMan will do this and also provide a number of other desirable features (failure detection and retry for instance). I'll try both ways.

RF

On Tue, Sep 2, 2014 at 6:25 PM, <Greg.Hitchen@xxxxxxxx> wrote:

MAX_JOBS_RUNNING = 50

Â

in the config of your submit node should do the trick.

Â

You should also be able to use $(Process) in your submit file to simplify it.

Â

e.g.

Â

initialdir = dir_$(Process)

queue 1000

Â

You may need to do something fancier as the above will produce

dir_0

dir_1

.....

dir_10

.....

dir_9999

Â

initialdir = dir_$$([$(Process)+1])

should give you 1-1000 rather than 0-9999

but still needs some formatting to produce 4 digit numbers.

Â

Cheers

Â

Greg

Â

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Ralph Finch
Sent: Wednesday, 3 September 2014 6:34 AM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] How to limit number of machines used in a long list of queues

Â

HTCondor 8.2.1 on all Windowsx64 pool

I have about 80 machines (slots) in the pool. I need to submit more than 1000 jobs in a single submit file as follows:

initialdir = dir_0001

queue

initialdir = dir_0002

queue
....

initialdir = dir_1000

queue

OK, that will work, but I would like to limit the number of machines claimed at any time to, say, 50. How can I submit 1000 jobs, but have the number of machines actually used (50) limited to less than the number of machines available (80) at all times?

I don't want to have to explicitly list each machine allowed or disallowed using a Requirements statement. Instead I'd like to say something like

Max_Machines = 50

Â

Thanks much--

Ralph Finch

Calif Dept. of Water Resources
Sacramento, CA USA

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/