Proposal and design document for inclusion into base Condor:
https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=3407
Brian
On Dec 28, 2012, at 8:32 PM, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:
> Ok, based on input from Burt, I rounded out the functionality a bit -
>
> The python-condor module now has much more functionality. It can submit jobs, remove/hold/release/suspend/continue, and edit jobs. It can advertise ads into the collector. It also exposes the parameter subsystem as a dictionary-like object (anyone want to write a configuration validator in python?).
>
> Submitting via a raw ClassAd interface is a touch "fun" as you get to figure out which magic attributes are required in the job (see https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=3406). However, I don't plan on re-implementing condor_submit, so this interface will likely remain developers-only.
>
> I took the time to refactor a bit to make the API more natural - schedd actions utilize the "Schedd" object and collector actions utilize the "Collector" object. This seems to be the correct direction to go.
>
> Enjoy!
>
> Brian
>
> PS - Food for thought: Dan noted over IM that utilizing these bindings be a pleasant way to write unit tests.
>
> [bbockelm@example python-condor]$ python
> Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52)
> [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import condor
>>>> import classad
>>>> coll = condor.Collector("red-condor.unl.edu")
>>>> results = coll.query(condor.AdTypes.Startd, "true", ["Name"])
>>>> len(results)
> 3812
>>>> results[0]
> [ Name = "slot1@red-d20n35"; MyType = "Machine"; TargetType = "Job"; CurrentTime = time() ]
>>>> scheddAd = coll.locate(condor.DaemonTypes.Schedd, "red-gw1.unl.edu")
>>>> scheddAd["ScheddIpAddr"]
> '<129.93.239.132:53020>'
>>>> schedd = condor.Schedd(scheddAd)
>>>> results = schedd.query('Owner =?= "cmsprod088"', ["ClusterId", "ProcId"])
>>>> len(results)
> 63
>>>> results[0]
> [ MyType = "Job"; TargetType = "Machine"; ServerTime = 1356722353; ClusterId = 674143; ProcId = 0; CurrentTime = time() ]
>>>> condor.param["COLLECTOR_HOST"]
> 'hcc-briantest.unl.edu'
>>>> schedd = condor.Schedd() # Defaults to the local schedd.
>>>> results = schedd.query()
>>>> results[0]["RequestMemory"]
> ifthenelse(MemoryUsage isnt undefined,MemoryUsage,( ImageSize + 1023 ) / 1024)
>>>> results[0]["RequestMemory"].eval()
> 1L
>>>> ad=classad.parse(open("test.submit.ad"))
>>>> print schedd.submit(ad, 2) # Submits two jobs in the cluster; edit test.submit.ad to preference.
> 110
>>>> print schedd.act(condor.JobAction.Remove, ["111.0", "110.0"])'
>
> [
> TotalNotFound = 0;
> TotalPermissionDenied = 0;
> TotalAlreadyDone = 0;
> TotalJobAds = 2;
> TotalSuccess = 2;
> TotalChangedAds = 1;
> TotalBadStatus = 0;
> TotalError = 0
> ]
>>>> print schedd.act(condor.JobAction.Hold, "Owner =?= \"bbockelm\"")'
>
> [
> TotalNotFound = 0;
> TotalPermissionDenied = 0;
> TotalAlreadyDone = 0;
> TotalJobAds = 2;
> TotalSuccess = 2;
> TotalChangedAds = 1;
> TotalBadStatus = 0;
> TotalError = 0
> ]
>>>> schedd.edit('Owner =?= "bbockelm"', "Foo", classad.ExprTree('"baz"'))
>>>> schedd.edit(["110.0"], "Foo", '"bar"')
>>>>
>
> On Dec 28, 2012, at 9:44 AM, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:
>
>> Hi all,
>>
>> Over Christmas break, I decided to tackle a small project that had been sitting in the back of my mind for awhile - python bindings for ClassAds and HTCondor.
>>
>> This resulted in two new projects:
>> https://github.com/bbockelm/python-classad
>> https://github.com/bbockelm/python-condor
>>
>> It utilizes boost.python to do the majority of the lifting. I try to wrap the "90%" features - things that are needed 90% of the time. Some of the remaining features are going to take some amount of clever thinking to expose (a gold star to someone who will write the "Why ClassAd::SetParentScope is an unusable interface" rant for me).
>>
>> The end-result is quite nice for ClassAds. It can be built on the system based only on the condor-classad-devel, boost-devel, and boost-python RPMs.
>>
>> The end-result for HTCondor wrappers leaves a bit to be desired. It requires the source code of HTCondor to be present - and, because some important things are defined in the g++ invocation for HTCondor instead of the config.h header, you may need to twiddle with platfrom-specific defines. The correct abstractions and API to expose is also not clear; I might be fiddling more in the future.
>>
>> Having in-process bindings is incredibly useful:
>> - It prevents developers from re-implementing, poorly, bits and pieces of the ClassAd language in order to parse output of Condor tools.
>> - One reason I tackled this project is I'm afraid no sysadmin has sufficient skills to write JobHooks without it (see https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=3380).
>> - It allows tools to re-use established security sessions.
>> - It allows errors to be handled more cleanly.
>>
>> I'd really like to make these into a Condor contrib module. While python-classads is fine stand-alone, the build limitations outlined above makes a strong case for a contrib python-condor.
>>
>> Anyhow - enjoy the python bindings! If I missed your favorite ClassAd/Condor feature, patches are accepted!
>>
>> Brian
>>
>> [bbockelm@hcc-briantest python-classads]$ python
>> Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52)
>> [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import classad
>>>>> ad = classad.ClassAd()
>>>>> expr = classad.ExprTree("2+2")
>>>>> ad["foo"] = expr
>>>>> print ad["foo"].eval()
>> 4
>>>>> ad["bar"] = 2.1
>>>>> ad["baz"] = classad.ExprTree("time() + 4")
>>>>> print list(ad)
>> ['bar', 'foo', 'baz']
>>>>> print dict(ad.items())
>> {'baz': time() + 4, 'foo': 2 + 2, 'bar': 2.100000000000000E+00}
>>>>> print ad
>>
>> [
>> bar = 2.100000000000000E+00;
>> foo = 2 + 2;
>> baz = time() + 4
>> ]
>>>>> ad2=classad.parse(open("test_ad", "r"));
>>>>> ad2["error"] = classad.Value.Error
>>>>> ad2["undefined"] = classad.Value.Undefined
>>>>> print ad2
>>
>> [
>> error = error;
>> bar = 2.100000000000000E+00;
>> foo = 2 + 2;
>> undefined = undefined;
>> baz = time() + 4
>> ]
>>>>> ad2["undefined"]
>> classad.Value.Undefined
>>>>>
>>
>> [bbockelm@example python-condor]$ python
>> Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52)
>> [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import condor
>>>>> results = condor.query_collector("red-condor.unl.edu", "true", ["Name"])
>>>>> print len(results)
>> 4130
>>>>> print results[0]
>>
>> [
>> Name = "Nebraska T2@xxxxxxxxxxxxxxxxxx";
>> MyType = "Collector";
>> CurrentTime = time()
>> ]
>>>>> results = condor.query_collector("red-condor.unl.edu", 'MyType =?= "Scheduler"', ["Name"])
>>>>> print results[0]
>>
>> [
>> Name = "red-gw1.unl.edu";
>> MyType = "Scheduler";
>> NumUsers = 21;
>> CurrentTime = time()
>> ]
>>>>> query = condor.JobQuery()
>>>>> schedd = query.locate("red-condor.unl.edu", "red-gw1.unl.edu")
>>>>> schedd["ScheddIpAddr"]
>> '<129.93.239.132:53020>'
>>>>> results = query.run(schedd, 'Owner =?= "cmsprod088"', ["ClusterId", "ProcID"])
>>>>> len(results)
>> 439
>>>>> results[0]
>> [ MyType = "Job"; TargetType = "Machine"; ServerTime = 1356662680; ClusterId = 670932; ProcID = 0; CurrentTime = time() ]
>>>>>
>>
>>
>> _______________________________________________
>> HTCondor-devel mailing list
>> HTCondor-devel@xxxxxxxxxxx
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-devel
>
> _______________________________________________
> HTCondor-devel mailing list
> HTCondor-devel@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-devel
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
|
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}