Re: [HTCondor-users] transfer_input_files, arguments......executable not transferred

[Gergely Debreczeni <gergely.debreczeni@xxxxxxxxxxxx>]

Debugging the problem deeper it seems that the executable gets transferred only if it is defined in the "transfer_input_files" list.....

So given a

queue transfer_input_files , arguments from [0:1:1] ./filelist

kind of queue statement, what format should i use in the filelist file to be bale to list multiple input files (the executable and data files to process) and multiple arguments ?

thanks a lot,

Gergely

---

From: John M Knoeller <johnkn@xxxxxxxxxxx>
Sent: Thursday, April 4, 2019 6:49 PM
To: HTCondor-Users Mail List
Cc: Gergely Debreczeni
Subject: RE: [HTCondor-users] transfer_input_files, arguments......executable not transferred

 

For historical reasons, HTCondor will rename your executable to condor_exec.exe when it transfers it.  so if you want to know if the executable is transferred, you need to stat condor_exec.exe.  

 

Also, there is no need to use ./ before your executable name.   This should work fine

 

## Executable

executable              = sdfgen.sh

universe                = docker

docker_image            = dani_tensorflow:v_01

 

At some time in the 8.9 series, we plan to stop renaming the executable, but it is actually a lot of work to remove this behavior cleanly.

-tj

 

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Gergely Debreczeni via HTCondor-users
Sent: Thursday, April 4, 2019 8:21 AM
To: htcondor-users@xxxxxxxxxxx
Cc: Gergely Debreczeni <gergely.debreczeni@xxxxxxxxxxxx>
Subject: [HTCondor-users] transfer_input_files, arguments......executable not transferred

 

Hi !

 

I have a single executable (sdfgen.sh) that I'd like to run in thousands of times with different input files and parameters. I found the submit file example below, where one lists the input files and parameters in a text file and gives that as a parameter to the Queue "command".

 

Everything seems to work fine, except that the executable itself is not transferred, even if I add the transfer_executable=true line into my sub file.

 

The submitted jobs get started apparently with the right parameter set but are all put in 'Hold' with the following message:

 

------------------------------------------

044.000:  Request is held.

 

Hold reason: Error from slot1@scorpio005: Error running docker job: OCI runtime create failed: container_linux.go:344: starting container process caused 'exec: \'./sdfgen.sh\': stat ./sdfgen.sh: no such file or directory': unknown

-------------------------------------

 

How I can resolve this issue ?

 

Thanks in advance,

Gergely

 

 

 

 

 

The sub fiile:

--------------------------------------------------------------------------------------------

## Executable

executable              = ./sdfgen.sh

universe                = docker

docker_image            = dani_tensorflow:v_01

 

## Logs

log                     = out/sdfgen.$(Process).log

output                  = out/sdfgen.$(Process).stdout

error                   = out/sdfgen.$(Process).stderr

 

## File transfer

transfer_executable     = true

should_transfer_files   = Yes

when_to_transfer_output = ON_EXIT

line = $(Row)+1

 

transfer_output_files   = output.out

transfer_output_remaps  = "output.out=out/output$INT(line).out"

 

 

## Resources requested

request_cpus            = 1

request_GPUs            = 1

Requirements            = (ResourceType == "Dedicated") && (regexp(".*dani_tensorflow.*",LocallyAvailableDockerImages))

 

 

## Submit command

queue transfer_input_files,arguments from [0:3:1] ./filelist

 

 

 

 

The filelist file:

-------------------------------------------------------------------------------------------

 

a, a 14.3 4

b, b 2.3 3

c, c 23.4 1

 

 

---

This e-mail and any files transmitted with it contain confidential and may contain privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorized use, copying, disclosure or distribution of the material in this e-mail is strictly forbidden.