Re: [HTCondor-users] File transfer not working with docker universe ?

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 3/10/2023 11:11 AM, Gaetan Geffroy wrote:

Hi,

 

I am submitting the following job (through python):

import htcondor

s = htcondor.Schedd()

 

job = htcondor.Submit({

    "executable": "/bin/cat",

    "arguments": "test.txt",

    "transfer_executable": False,

    "initialdir": "/tmp",

    "should_transfer_files": True,

    "transfer_input_files": "/test/test.txt",

    "universe": "docker",

    "docker_image": "python:3.8.10",

    "output": "/tmp/test.out",

    "error": "/tmp/test.err",

    "max_retries": 0,

})

 

with s.transaction() as t:

    job.queue(t)

Once it ran, test.out is empty and test.err contains the following: /bin/cat: test.py: No such file or directory

 

    Hi Gaetan,
    
    Strange...  the error message above is complaining about file
    "test.py" - where is that filename coming from?  If the problem is
    lack of file transfer as you suggest, I would expect to see the
    error message complain about "test.txt" not existing.   Seems like
    there is something else going on specific to your configuration -
    perhaps there is some job transform or USER_JOB_WRAPPER configured
    that is messing around with file name/transfers?
    
    Couple other quick thoughts:
    
    1. What version of HTCondor are you using?
    2. To help with figuring out what is going on, perhaps add
    "manifest=true" to the submit description... this will produce
    output on the access point (submit node) showing the contents of the
    job sandbox at the execute point (execute node).  See the man page
    for condor_submit for more info about "manifest" option. (Note: you
    need to be running HTCondor v9.0.17 or later for this option to
    work).
    3. Not related to the problem, but you may find it easier/safer to
    use the htcondor.Schedd.submit() method to submit the job instead of
    using the lower level htcondor.Submit.queue() method; see
https://htcondor.readthedocs.io/en/latest/apis/python-bindings/tutorials/Submitting-and-Managing-Jobs.html
    
    regards,
    Todd