Re: [HTCondor-users] How to submit jobs to both Windows and Linux with a single .sub file

[Greg.Hitchen@xxxxxxxx]

As John said you can use $$ in your executable name.

 

We have done this using the statement in a submit file:

 

executable = Apsim.$$(OpSys).$$(Arch).bat

 

We then have 3 different executables called:

 

Apsim.WINDOWS.INTEL.bat

Apsim.WINDOWS.X86_64.bat

Apsim.LINUX.X86_64.bat

 

for windows 32-bit , windows 64-bit, and linux 64-bit executables.

 

These are actually batch and bash scripts that set things up, e.g. download the ârealâ executables

and input data, run the ârealâ exeâs and upload output data.

 

Cheers

 

Greg

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of John M Knoeller
Sent: Tuesday, 4 August 2020 10:37 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] How to submit jobs to both Windows and Linux with a single .sub file

 

Yes,  $(OpSys)  and config if/else are evaluated at submit time, which is too soon for this use case.

 

This is why there is $$(OpSys),  the two dollar macro expansion does not expand at submit time,

it expands at match time to the OpSys of the machine that the job is going to run on.  

 

There is an example on the submit man page

https://htcondor.readthedocs.io/en/v8_9_7/man-pages/condor_submit.html

 

executable = povray.$$(OpSys).$$(Arch)

 

Hope this helps

-tj

 

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of jefhque@xxxxxxxxx
Sent: Tuesday, August 4, 2020 5:02 AM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] How to submit jobs to both Windows and Linux with a single .sub file

 

Hello everyone, 

 

I'm working on HTCondor v8.7.9 with a condor pool containing both Windows and Linux machines, and my submit node is a Windows machine. As the executable of Windows cannot execute on Linux, and the executable of Linux cannot execute on Windows either, I need to submit two kinds of executables to run on both Windows and Linux,  but I don't know how to write the condor job description (.sub file) to let condor choose the correct executable to execute. I have tried several ways to figure out this, but they don't work:

  I specified the executable with $(OpSys) macro (e.g. foo.$(OpSys).exe), but I find that the value of the macro is always "WINDOWS" when I submit with my Windows machine, regardless of the target machine is Windows or Linux. My thought is that the $(OpSys) macro is determined as the operating system of the submit machine, and thus it cannot resolve my issue.

  I also tried to use the condition description (if/else), but it fails to resolve macros like $(OpSys) and the target machine's operating system and cause the parsing error when I submit the job. It seems that the conditions in the submit description file are only supported for simple static conditions, and dynamic conditions like choosing executables according to the target machine's operating system are not supported.

  Currently, I can statically split the jobs into Windows jobs and Linux jobs and determine the job numbers of each platform according to the slot numbers of each platform, but it suffers from undesired load balance problem, as I cannot perfectly predict the execution time of each job on each machine.

Is there any way to make condor automatically choose the right executable for the target machine's operating system and schedule the jobs for both Windows and Linux without any manual scheduling?