Re: [Condor-users] condor_submit feature request

[Nick LeRoy <nleroy@xxxxxxxxxxx>]

On Wed August 6 2008, Steffen Grunewald wrote:
> On Wed, Aug 06, 2008 at 09:38:54AM +0200, Steffen Grunewald wrote:
> > I'd like to ask for a feature related to condor_submit.
> >
> > Every now and then, I have to run a large job cluster with several 100s
> > to 1000s of jobs, each using $(Process) as the argument to the executable
> > (simply a counter, used as a seed for a RNG).
> > Often I'd like to split the large cluster into pieces submitted
> > separately, or extend an existing cluster.
> > What condor_submit is lacking is a way to start counting at a process
> > number
>
> That would be nasty because it decouples the number of the corresponding
> file from the number of the process. In your scheme, even multiple
> processes would try to write to the same files!

No, all I'm suggesting is a more flexible means of specifying file names 
(using a sprintf-like formatting).  I just expanded on Ryan's suggestion a 
bit (the ability to do basic math operations).  Ignoring the math part:

output = %03d.out, $(Process)
error  = %03d.err, $(Process)
log    = %03d.log, $(Process)

would produce: 000.out, 000.err, 000.log, 001.out, 001.err, 001.log, 002.out, 
002.err, 002.log, 003.out...

> What about an extension of the "Queue" syntax to
> 	Queue <number_of_jobs> <starting_process_number> <step_width>
> which would be close to the "seq" syntax?
> No additional keywords, backwards compatibility. Nothing sacrificed, a lot
> gained.

This would be useful, too.

> And BTW, what also would be very useful: a user-imposed limit on
> simultaneously running jobs from the same cluster ("I want to be nice to my
> colleagues, n slots are enough even if there are more available - because I
> know other's work is more important and I hate preemption")

Also useful.

-Nick

-- 
           <<< There is no spoon. >>>
 /`-_    Nicholas R. LeRoy               The Condor Project
{     }/ http://www.cs.wisc.edu/~nleroy  http://www.cs.wisc.edu/condor
 \    /  nleroy@xxxxxxxxxxx              The University of Wisconsin
 |_*_|   608-265-5761                    Department of Computer Sciences