Re: [HTCondor-users] Queuing N items from file

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 8/27/2020 11:19 AM, Eremey Valetov wrote:

> Hello
>       All,
>       
>       
>       I am using a computing cluster where I am allowed to have a
>       limited number of jobs in the queue.
>       
>       
>       I would like to queue $(njobs) at a time, each with parameters
>       from one line of a file:
>       
>       
>       queue par1,par2,par3,par4,par5 from [0:$(njobs)] inputs.txt
>       
>     
    
    Instead of doing a slice to ensure only njobs are simultaneously in
    the queue, I suggest just using MAX_MATERIALIZE in your submit file.
    For example, say inputs.txt has 1,000,000 lines, but you are only
    permitted to have 500 jobs in the queue at once.  You could do:
    
       max_materialize = 500 
       queue x,y,z from inputs.txt
    
    The above ensure the number of jobs in the queue for this cluster
    never exceeds 500... as soon as a job leaves the queue (because it
    completed or was removed), the next job in the cluster will be
    "materialized" and show up as idle.    E.g. upon submission, you may
    get jobs 50.0 thru 50.499... as soon as job 50.0 completes, job
    50.500 will get submitted.  To remove the entire cluster of jobs
    (i.e. remove all jobs and do not materialize any more), do
    "condor_rm <cluster_id>", e.g. do "condor_rm 50" instead of
    "condor_rm 50.5".  
    
    But re your specific question/issue below, it would help to see how
    you are setting $(njobs) via the command-line.  In other words, what
    exactly are you typing on the command line?  It may also be useful
    to see the output from "condor_version".
    
    Thanks
    Todd

However, this is treated as if the slice were [0:], even though njobs is set using a command-line argument to condor_submit. On the other hand, the following works as expected:

queue par1,par2,par3,par4,par5 from [0:1000] inputs.txt

Is there perhaps a way to have a variable in the slice, as in [0:$(njobs)], so that it works correctly?

Thanks,
Eremey Valetov

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

-- 
Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing   Department of Computer Sciences
HTCondor Technical Lead                1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                  Madison, WI 53706-1685