[HTCondor-devel] Ring in the New Year with new python bindings!

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

Hi all,

Over Christmas break, I decided to tackle a small project that had been sitting in the back of my mind for awhile - python bindings for ClassAds and HTCondor.

This resulted in two new projects:
https://github.com/bbockelm/python-classad
https://github.com/bbockelm/python-condor

It utilizes boost.python to do the majority of the lifting.  I try to wrap the "90%" features - things that are needed 90% of the time.  Some of the remaining features are going to take some amount of clever thinking to expose (a gold star to someone who will write the "Why ClassAd::SetParentScope is an unusable interface" rant for me).

The end-result is quite nice for ClassAds.  It can be built on the system based only on the condor-classad-devel, boost-devel, and boost-python RPMs.

The end-result for HTCondor wrappers leaves a bit to be desired.  It requires the source code of HTCondor to be present - and, because some important things are defined in the g++ invocation for HTCondor instead of the config.h header, you may need to twiddle with platfrom-specific defines.  The correct abstractions and API to expose is also not clear; I might be fiddling more in the future.

Having in-process bindings is incredibly useful:
- It prevents developers from re-implementing, poorly, bits and pieces of the ClassAd language in order to parse output of Condor tools.
  - One reason I tackled this project is I'm afraid no sysadmin has sufficient skills to write JobHooks without it (see https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=3380).
- It allows tools to re-use established security sessions.
- It allows errors to be handled more cleanly.

I'd really like to make these into a Condor contrib module.  While python-classads is fine stand-alone, the build limitations outlined above makes a strong case for a contrib python-condor.

Anyhow - enjoy the python bindings!  If I missed your favorite ClassAd/Condor feature, patches are accepted!

Brian

[bbockelm@hcc-briantest python-classads]$ python
Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52) 
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import classad
>>> ad = classad.ClassAd()
>>> expr = classad.ExprTree("2+2")
>>> ad["foo"] = expr
>>> print ad["foo"].eval()
4
>>> ad["bar"] = 2.1
>>> ad["baz"] = classad.ExprTree("time() + 4")
>>> print list(ad)
['bar', 'foo', 'baz']
>>> print dict(ad.items())
{'baz': time() + 4, 'foo': 2 + 2, 'bar': 2.100000000000000E+00}
>>> print ad

    [
        bar = 2.100000000000000E+00; 
        foo = 2 + 2; 
        baz = time() + 4
    ]
>>> ad2=classad.parse(open("test_ad", "r"));
>>> ad2["error"] = classad.Value.Error
>>> ad2["undefined"] = classad.Value.Undefined
>>> print ad2

    [
        error = error; 
        bar = 2.100000000000000E+00; 
        foo = 2 + 2; 
        undefined = undefined; 
        baz = time() + 4
    ]
>>> ad2["undefined"]
classad.Value.Undefined
>>> 

[bbockelm@example python-condor]$ python
Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52) 
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import condor
>>> results = condor.query_collector("red-condor.unl.edu", "true", ["Name"])
>>> print len(results)
4130
>>> print results[0]

    [
        Name = "Nebraska T2@xxxxxxxxxxxxxxxxxx"; 
        MyType = "Collector"; 
        CurrentTime = time()
    ]
>>> results = condor.query_collector("red-condor.unl.edu", 'MyType =?= "Scheduler"', ["Name"])
>>> print results[0]

    [
        Name = "red-gw1.unl.edu"; 
        MyType = "Scheduler"; 
        NumUsers = 21; 
        CurrentTime = time()
    ]
>>> query = condor.JobQuery()
>>> schedd = query.locate("red-condor.unl.edu", "red-gw1.unl.edu")
>>> schedd["ScheddIpAddr"]
'<129.93.239.132:53020>'
>>> results = query.run(schedd, 'Owner =?= "cmsprod088"', ["ClusterId", "ProcID"])
>>> len(results)
439
>>> results[0]
[ MyType = "Job"; TargetType = "Machine"; ServerTime = 1356662680; ClusterId = 670932; ProcID = 0; CurrentTime = time() ]
>>>

Attachment: smime.p7s
Description: S/MIME cryptographic signature