Re: [HTCondor-users] Queuing N items from file

[Eremey Valetov <eremey@xxxxxxxxxxx>]

Hello Todd,

Thanks, using max_materialize is indeed a better solution!

I was setting $(njobs) from the command line like this:

condor_submit g4bl.submit njobs=10

The output of condor_version is
$CondorVersion: 8.9.6 Mar 19 2020 PackageID: 8.9.6-1 $
$CondorPlatform: X86_64-CentOS_7.7

Thanks,
Eremey

On 27-Aug-20 12:39, Todd Tannenbaum wrote:

On 8/27/2020 11:19 AM, Eremey Valetov wrote:

Hello All,

I am using a computing cluster where I am allowed to have a limited number of jobs in the queue.

I would like to queue $(njobs) at a time, each with parameters from one line of a file:

queue par1,par2,par3,par4,par5 from [0:$(njobs)] inputs.txt

Instead of doing a slice to ensure only njobs are simultaneously in the queue, I suggest just using MAX_MATERIALIZE in your submit file.
For example, say inputs.txt has 1,000,000 lines, but you are only permitted to have 500 jobs in the queue at once. You could do:

ÂÂ max_materialize = 500
ÂÂ queue x,y,z from inputs.txt

The above ensure the number of jobs in the queue for this cluster never exceeds 500... as soon as a job leaves the queue (because it completed or was removed), the next job in the cluster will be "materialized" and show up as idle. E.g. upon submission, you may get jobs 50.0 thru 50.499... as soon as job 50.0 completes, job 50.500 will get submitted. To remove the entire cluster of jobs (i.e. remove all jobs and do not materialize any more), do "condor_rm <cluster_id>", e.g. do "condor_rm 50" instead of "condor_rm 50.5".Â

But re your specific question/issue below, it would help to see how you are setting $(njobs) via the command-line. In other words, what exactly are you typing on the command line? It may also be useful to see the output from "condor_version".

Thanks
Todd

However, this is treated as if the slice were [0:], even though njobs is set using a command-line argument to condor_submit. On the other hand, the following works as expected:

queue par1,par2,par3,par4,par5 from [0:1000] inputs.txt

Is there perhaps a way to have a variable in the slice, as in [0:$(njobs)], so that it works correctly?

Thanks,
Eremey Valetov

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

-- 
Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing   Department of Computer Sciences
HTCondor Technical Lead                1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                  Madison, WI 53706-1685 

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature