Re: [HTCondor-users] need help in settings for condo job submission using python bindings

[Xin Wang <xwang@xxxxxxxxxxxxx>]

The condor used here in the most recent stable release, i.e. 8.6.6.

 

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Xin Wang
Sent: Friday, September 22, 2017 3:44 PM
To: 'htcondor-users@xxxxxxxxxxx' <htcondor-users@xxxxxxxxxxx>; 'htcondor-admin@xxxxxxxxxxx' <htcondor-admin@xxxxxxxxxxx>
Subject: [HTCondor-users] need help in settings for condo job submission using python bindings

 

[External Message]

I’m trying to submit jobs to condor to run some python scripts. If I generate a job file and submit with condor_submit, everything works fine.

Here is the job file:

 

universe = vanilla

environment = "PYTHONHOME=/my/path/to/anaconda3"

executable = /my/path/to/anaconda3/bin/python

arguments = /my/path/to/scripts/myrun.py

log = /tmp/job.log

output = /tmp/test.log

error = /tmp/test.err

queue

 

 

For the same job, I tried to submit through python bindings, using two different methods but do not have luck with either.

 

Firstly I tried schedd.Submit with the following codes:

 

import htcondor

schedd = htcondor.Schedd()

sub = htcondor.Submit()

sub['universe'] = 'vanilla'

sub['environment'] = "PYTHONHOME=/my/path/to/anaconda3"

sub['executable'] = '/my/path/to/anaconda3/bin/python'

sub['arguments'] = '/my/path/to/scripts/myrun.py'

sub['log'] = '/tmp/job.log'

sub['output'] = '/tmp/test.log'

sub['error'] = '/tmp/test.err'

 

with schedd.transaction() as txn:

    sub.queue(txn)

 

The job was submitted without any issues, can run successfully without issues, and have log file /tmp/job.log generated successfully. However, output and error does not work, and /tmp/test.log or /tmp/test.err are generated but with size 0 (empty).

 

 

Secondly, I tried schedd.submit with the following codes:

import htcondor

schedd = htcondor.Schedd()

job_ad = {

    "cmd" : ‘/my/path/to/anaconda3/bin/python',

    "arguments" : '/my/path/to/scripts/myrun.py',

    'env': "PYTHONHOME=/my/path/to/anaconda3",

    "log": '/tmp/job.log',

    "out": '/tmp/test.log',

    "err": "/tmp/test.err",

}

clusterId = schedd.submit(job_ad)

 

The job could not run. However, /tmp/test.err can be generated proper error messages:

condor_exec.exe: error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

I suspect that the error is because the environment is not properly set, but I had no luck when I also tried to set “environment” instead of “env”.

 

How should I fix the settings so that I can submit condor task through python bindings properly? Thanks.

 

Xin

 

Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorized and regulated by the Financial Conduct Authority.

Jefferies archives and monitors outgoing and incoming e-mail. The contents of this email, including any attachments, are confidential to the ordinary user of the email address to which it was addressed. If you are not the addressee of this email you may not copy, forward, disclose or otherwise use it or any part of it in any form whatsoever. This email may be produced at the request of regulators or in connection with civil litigation. Jefferies accepts no liability for any errors or omissions arising as a result of transmission. Use by other than intended recipients is prohibited. In the United Kingdom, Jefferies operates as Jefferies International Limited; registered in England: no. 1978621; registered office: Vintners Place, 68 Upper Thames Street, London EC4V 3BJ. Jefferies International Limited is authorized and regulated by the Financial Conduct Authority.