Re: [Condor-users] Running multiple batch files under one description file or running multiple description files under the same condor_submit command

[Matthew Farrellee <matt@xxxxxxxxxx>]

condor_submit is well suited for this. you can have multiple queue 
commands in a single submit file...

universe = vanilla
executable = /bin/true
queue 1
executable = /bin/false
queue 2

...will give you two false jobs and 1 true. you can expand this too...

(below, purely illustrative)

universe = vanilla
executable = /bin/true
run_as_user = true
queue 1
run_as_user = false
queue 1

...will give you 1 job run as the user and one not.

hope that gives you a taste for what you can do.

best,


matt

p.s. if you want to do case #2 you can just concatenate all the files...

Alas, Alex [FEDI] wrote:
> Hello to all Condor users (Gurus, Experts, not so experts and the rest
> of us) again,
>
> I have a very important question, I have a condor pool of 23 Windows
> nodes (CPU's). The way we are running batching jobs today is creating a
> batch file that will encompass 500 process into one file. The way I
> understand condor when a submitting machine sends a job to the Central
> Manager Systems, it will allocate that batch file (submitting job) to
> only one CPU or the first CPU that matches its requirements, then the
> second, the third and so on until it matches the amount of time you
> stipulated to run the job in your queue. For a better understanding of
> my question I am adding a sample of the submitting job and a sample of
> the batch file we are running. As I said before some time we run some
> jobs with up to 500 process on one batch file. To clarify my question I
> will create two different scenarios, case 1 and 2; 
>
> Case 1: I need to know if there is a way I can run multiple batch files
> under the same description file for example adding the another
> testjob1.bat right after \\Sharename\Subdir1\Subdir2\testjob.bat
> <file:///\\Sharename\Subdir1\Subdir2\testjob.bat> ;
> \\Sharename\Subdir1\Subdir2\testjob1.bat
> <file:///\\Sharename\Subdir1\Subdir2\testjob1.bat> , in some cases up to
> 500 different batch files, and can those multiple and different files
> being executed on different CPU's. 
>
>  
>
> Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> x
>
> Case 1:
>
> Description File sample:
>
> UNIVERSE = VANILLA
>
> REQUIREMENTS = (Arch == "INTEL" && OpSys == "WINNT52" || Arch == "INTEL"
> && OpSys == "WINNT51")
>
> NOTIFY_USER = user1@xxxxxxxxxx,user2@xxxxxxxxxx
>
> INITIALDIR = c:\condor\execute
>
> SHOULD_TRANSFER_FILES = YES
>
> WHEN_TO_TRANSFER_OUTPUT = ON_EXIT
>
> TRANSFER_INPUT_FILES =
> \\Sharename\Subdir1\BatchProcessingPackage\Subdir3\Subdir4\lasEnvelop.ex
> e
>
> RUN_AS_OWNER = TRUE
>
> EXECUTABLE =  \\Sharename\Subdir1\Subdir2\testjob.bat
> <file:///\\Sharename\Subdir1\Subdir2\testjob.bat> ,  ||
>
>   \\Sharename\Subdir1\Subdir2\testjob1.bat
> <file:///\\Sharename\Subdir1\Subdir2\testjob1.bat>  ||
>
>   \\Sharename\Subdir1\Subdir2\testjob2.bat
> <file:///\\Sharename\Subdir1\Subdir2\testjob2.bat>  ||
>
>   \\Sharename\Subdir1\Subdir2\testjob3.bat
> <file:///\\Sharename\Subdir1\Subdir2\testjob3.bat> 
>
> OUTPUT = testjob.out.$(Process)
>
> ERROR = testjob.err.$(Process)
>
> LOG = testjob.log
>
> QUEUE 1
>
> Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> x
>
> Case 2: 
>
> If that is not possible is there a way to submit multiple jobs under one
> condor_submit command.
>
> Condor_submit Test_job.txt, Test_job1.txt, Test_job2.txt... up to n
> jobs...
>
>  
>
> Xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> x
>
> Samples:
>
> Description File sample; Test_job.txt:
>
> UNIVERSE = VANILLA
>
> REQUIREMENTS = (Arch == "INTEL" && OpSys == "WINNT52" || Arch == "INTEL"
> && OpSys == "WINNT51")
>
> NOTIFY_USER = user1@xxxxxxxxxx,user2@xxxxxxxxxx
>
> INITIALDIR = c:\condor\execute
>
> SHOULD_TRANSFER_FILES = YES
>
> WHEN_TO_TRANSFER_OUTPUT = ON_EXIT
>
> TRANSFER_INPUT_FILES =
> \\Sharename\Subdir1\BatchProcessingPackage\Subdir3\Subdir4\lasEnvelop.ex
> e
>
> RUN_AS_OWNER = TRUE
>
> EXECUTABLE =  \\Sharename\Subdir1\Subdir2\testjob.bat
>
> OUTPUT = testjob.out.$(Process)
>
> ERROR = testjob.err.$(Process)
>
> LOG = testjob.log
>
> QUEUE 1
>
>  
>
> Testjob.bat:
>
> echo off
>
> \\Sharename\Subdir1\BatchProcessingPackage_beta\Subdir3\Subdir4\Envelop.
> exe 3 -1 2 \\Sharename\Subdir1\Subdir2\Envelop.txt
> \\Sharename\Subdir1\Subdir2 a.LAS
> <file:///\\Sharename\Subdir1\Subdir2%20a.LAS>  (Process 1)
>
> \\Sharename\Subdir1\BatchProcessingPackage_beta\Subdir3\Subdir4\Envelop.
> exe 3 -1 2  \\Sharename\Subdir1\Subdir2\Envelop.txt
> \\Sharename\Subdir1\Subdir2 b.LAS
> <file:///\\Sharename\Subdir1\Subdir2%20b.LAS>  (Process 2)
>
>  
>
>  
>
> Respectfully,
>
> Alex Alas
>
> Systems Administrator
> Fugro EarthData Inc.
>
> Tel. 301-948-8550 x219 Fax 301-963-2064 E-mail: aalas@xxxxxxxxxxxxx
> <mailto:aalas@xxxxxxxxxxxxx>  
>
> 7320 Executive Way, Frederick, MD  21704
>
> Website: http://www.fugroearthdata.com <http://www.earthdata.com/> 
>
>  
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at: 
> https://lists.cs.wisc.edu/archive/condor-users/