Re: [HTCondor-devel] Ring in the New Year with new python bindings!

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

Ok, based on input from Burt, I rounded out the functionality a bit - 

The python-condor module now has much more functionality.  It can submit jobs, remove/hold/release/suspend/continue, and edit jobs.  It can advertise ads into the collector.  It also exposes the parameter subsystem as a dictionary-like object (anyone want to write a configuration validator in python?).

Submitting via a raw ClassAd interface is a touch "fun" as you get to figure out which magic attributes are required in the job (see https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=3406).  However, I don't plan on re-implementing condor_submit, so this interface will likely remain developers-only.

I took the time to refactor a bit to make the API more natural - schedd actions utilize the "Schedd" object and collector actions utilize the "Collector" object.  This seems to be the correct direction to go.

Enjoy!

Brian

PS - Food for thought: Dan noted over IM that utilizing these bindings be a pleasant way to write unit tests.

[bbockelm@example python-condor]$ python
Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52) 
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import condor
>>> import classad
>>> coll = condor.Collector("red-condor.unl.edu")
>>> results = coll.query(condor.AdTypes.Startd, "true", ["Name"])
>>> len(results)
3812
>>> results[0]
[ Name = "slot1@red-d20n35"; MyType = "Machine"; TargetType = "Job"; CurrentTime = time() ]
>>> scheddAd = coll.locate(condor.DaemonTypes.Schedd, "red-gw1.unl.edu")
>>> scheddAd["ScheddIpAddr"]
'<129.93.239.132:53020>'
>>> schedd = condor.Schedd(scheddAd)
>>> results = schedd.query('Owner =?= "cmsprod088"', ["ClusterId", "ProcId"])
>>> len(results)
63
>>> results[0]
[ MyType = "Job"; TargetType = "Machine"; ServerTime = 1356722353; ClusterId = 674143; ProcId = 0; CurrentTime = time() ]
>>> condor.param["COLLECTOR_HOST"]
'hcc-briantest.unl.edu'
>>> schedd = condor.Schedd() # Defaults to the local schedd.
>>> results = schedd.query()
>>> results[0]["RequestMemory"]
ifthenelse(MemoryUsage isnt undefined,MemoryUsage,( ImageSize + 1023 ) / 1024)
>>> results[0]["RequestMemory"].eval()
1L
>>> ad=classad.parse(open("test.submit.ad"))
>>> print schedd.submit(ad, 2) # Submits two jobs in the cluster; edit test.submit.ad to preference.
110
>>> print schedd.act(condor.JobAction.Remove, ["111.0", "110.0"])'

    [
        TotalNotFound = 0; 
        TotalPermissionDenied = 0; 
        TotalAlreadyDone = 0; 
        TotalJobAds = 2; 
        TotalSuccess = 2; 
        TotalChangedAds = 1; 
        TotalBadStatus = 0; 
        TotalError = 0
    ]
>>> print schedd.act(condor.JobAction.Hold, "Owner =?= \"bbockelm\"")'

    [
        TotalNotFound = 0; 
        TotalPermissionDenied = 0; 
        TotalAlreadyDone = 0; 
        TotalJobAds = 2; 
        TotalSuccess = 2; 
        TotalChangedAds = 1; 
        TotalBadStatus = 0; 
        TotalError = 0
    ]
>>> schedd.edit('Owner =?= "bbockelm"', "Foo", classad.ExprTree('"baz"'))
>>> schedd.edit(["110.0"], "Foo", '"bar"')
>>>

On Dec 28, 2012, at 9:44 AM, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:

> Hi all,
> 
> Over Christmas break, I decided to tackle a small project that had been sitting in the back of my mind for awhile - python bindings for ClassAds and HTCondor.
> 
> This resulted in two new projects:
> https://github.com/bbockelm/python-classad
> https://github.com/bbockelm/python-condor
> 
> It utilizes boost.python to do the majority of the lifting.  I try to wrap the "90%" features - things that are needed 90% of the time.  Some of the remaining features are going to take some amount of clever thinking to expose (a gold star to someone who will write the "Why ClassAd::SetParentScope is an unusable interface" rant for me).
> 
> The end-result is quite nice for ClassAds.  It can be built on the system based only on the condor-classad-devel, boost-devel, and boost-python RPMs.
> 
> The end-result for HTCondor wrappers leaves a bit to be desired.  It requires the source code of HTCondor to be present - and, because some important things are defined in the g++ invocation for HTCondor instead of the config.h header, you may need to twiddle with platfrom-specific defines.  The correct abstractions and API to expose is also not clear; I might be fiddling more in the future.
> 
> Having in-process bindings is incredibly useful:
> - It prevents developers from re-implementing, poorly, bits and pieces of the ClassAd language in order to parse output of Condor tools.
>  - One reason I tackled this project is I'm afraid no sysadmin has sufficient skills to write JobHooks without it (see https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=3380).
> - It allows tools to re-use established security sessions.
> - It allows errors to be handled more cleanly.
> 
> I'd really like to make these into a Condor contrib module.  While python-classads is fine stand-alone, the build limitations outlined above makes a strong case for a contrib python-condor.
> 
> Anyhow - enjoy the python bindings!  If I missed your favorite ClassAd/Condor feature, patches are accepted!
> 
> Brian
> 
> [bbockelm@hcc-briantest python-classads]$ python
> Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52) 
> [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import classad
>>>> ad = classad.ClassAd()
>>>> expr = classad.ExprTree("2+2")
>>>> ad["foo"] = expr
>>>> print ad["foo"].eval()
> 4
>>>> ad["bar"] = 2.1
>>>> ad["baz"] = classad.ExprTree("time() + 4")
>>>> print list(ad)
> ['bar', 'foo', 'baz']
>>>> print dict(ad.items())
> {'baz': time() + 4, 'foo': 2 + 2, 'bar': 2.100000000000000E+00}
>>>> print ad
> 
>    [
>        bar = 2.100000000000000E+00; 
>        foo = 2 + 2; 
>        baz = time() + 4
>    ]
>>>> ad2=classad.parse(open("test_ad", "r"));
>>>> ad2["error"] = classad.Value.Error
>>>> ad2["undefined"] = classad.Value.Undefined
>>>> print ad2
> 
>    [
>        error = error; 
>        bar = 2.100000000000000E+00; 
>        foo = 2 + 2; 
>        undefined = undefined; 
>        baz = time() + 4
>    ]
>>>> ad2["undefined"]
> classad.Value.Undefined
>>>> 
> 
> [bbockelm@example python-condor]$ python
> Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52) 
> [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import condor
>>>> results = condor.query_collector("red-condor.unl.edu", "true", ["Name"])
>>>> print len(results)
> 4130
>>>> print results[0]
> 
>    [
>        Name = "Nebraska T2@xxxxxxxxxxxxxxxxxx"; 
>        MyType = "Collector"; 
>        CurrentTime = time()
>    ]
>>>> results = condor.query_collector("red-condor.unl.edu", 'MyType =?= "Scheduler"', ["Name"])
>>>> print results[0]
> 
>    [
>        Name = "red-gw1.unl.edu"; 
>        MyType = "Scheduler"; 
>        NumUsers = 21; 
>        CurrentTime = time()
>    ]
>>>> query = condor.JobQuery()
>>>> schedd = query.locate("red-condor.unl.edu", "red-gw1.unl.edu")
>>>> schedd["ScheddIpAddr"]
> '<129.93.239.132:53020>'
>>>> results = query.run(schedd, 'Owner =?= "cmsprod088"', ["ClusterId", "ProcID"])
>>>> len(results)
> 439
>>>> results[0]
> [ MyType = "Job"; TargetType = "Machine"; ServerTime = 1356662680; ClusterId = 670932; ProcID = 0; CurrentTime = time() ]
>>>> 
> 
> 
> _______________________________________________
> HTCondor-devel mailing list
> HTCondor-devel@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-devel

Attachment: smime.p7s
Description: S/MIME cryptographic signature