Re: [HTCondor-users] File transfer not working with docker universe ?

[Gaetan Geffroy <gage@xxxxxxxxx>]

Hi Todd,

 

The test.py is an error from my side copying the text. The actual error does say test.txt.

I am using condor 10.0.2.

 

I tried running the job with the manifest, but the in and out files are both empty:

 

About the submit method, the htcondor.Schedd().submit() caused me some problems in the past, throwing boost python errors and crashing my whole program on a segmentation fault.

I ended up switching to this as it did not appear to have the same issue.

 

Thanks,

 

GaÃtan

 

Gaetan Geffroy
Junior Software Engineer
Terma GmbH
T +49 6151 86005 43 (direct)
 


 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Todd Tannenbaum via HTCondor-users
Sent: Friday, March 10, 2023 18:59
To: htcondor-users@xxxxxxxxxxx
Cc: Todd Tannenbaum <tannenba@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] File transfer not working with docker universe ?

 

CAUTION: This email originated from outside of Terma. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On 3/10/2023 11:11 AM, Gaetan Geffroy wrote:

Hi,

 

I am submitting the following job (through python):

import htcondor

s = htcondor.Schedd()

 

job = htcondor.Submit({

    "executable": "/bin/cat",

    "arguments": "test.txt",

    "transfer_executable": False,

    "initialdir": "/tmp",

    "should_transfer_files": True,

    "transfer_input_files": "/test/test.txt",

    "universe": "docker",

    "docker_image": "python:3.8.10",

    "output": "/tmp/test.out",

    "error": "/tmp/test.err",

    "max_retries": 0,

})

 

with s.transaction() as t:

    job.queue(t)

Once it ran, test.out is empty and test.err contains the following: /bin/cat: test.py: No such file or directory

 

Hi Gaetan,

Strange...  the error message above is complaining about file "test.py" - where is that filename coming from?  If the problem is lack of file transfer as you suggest, I would expect to see the error message complain about "test.txt" not existing.   Seems like there is something else going on specific to your configuration - perhaps there is some job transform or USER_JOB_WRAPPER configured that is messing around with file name/transfers?

Couple other quick thoughts:

1. What version of HTCondor are you using?
2. To help with figuring out what is going on, perhaps add "manifest=true" to the submit description... this will produce output on the access point (submit node) showing the contents of the job sandbox at the execute point (execute node).  See the man page for condor_submit for more info about "manifest" option. (Note: you need to be running HTCondor v9.0.17 or later for this option to work).
3. Not related to the problem, but you may find it easier/safer to use the htcondor.Schedd.submit() method to submit the job instead of using the lower level htcondor.Submit.queue() method; see https://htcondor.readthedocs.io/en/latest/apis/python-bindings/tutorials/Submitting-and-Managing-Jobs.html

regards,
Todd