Re: [HTCondor-users] Adding new ClassAd functions to HTC

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

> On Nov 10, 2014, at 3:56 AM, Fischer, Max (SCC) <max.fischer@xxxxxxx> wrote:
> 
> Hi all,
> 
> I’m looking for a way to query a REST service on our worker nodes for job matching. Dumping the information into the job/worker ClassAds as lists is not possible due to the data volume.
> I’m constrained by having to execute the query on the worker node, but with information from the job, during START/RANK/… evaluation.
> 
> It seems the only realistic solution is a custom ClassAd function, similar to the HDFS plugin [1]. I’m lost at how to include a new ClassAd function into HTC itself, though. Is there any tutorial, guide or other information on this?

Hi Max,

I think Dan Bradley wrote this up (very well) once - but I also think the tutorial disappeared when he moved on to a new job.  :(

Here's a perhaps-simpler example custom function to start with:

https://github.com/bbockelm/condor-ce/blob/master/src/classad_ce.cpp

Basically, you compile your function into a shared library and add the shared library to CLASSAD_USER_LIB.

One thing to consider is that the negotiator may evaluate the function many, many times (note - this means that having side-effects can be very bad!).  If you want an example of a custom function whose results are cached, look here:

https://github.com/bbockelm/classad-xrootd-mapping/tree/master/src

(Very similar idea to Stephen's HDFS function; I don't recall which code came first)

> Alternatively, is there a simpler way to execute an arbitrary query from within a ClassAd?
> 

Nothing comes to mind.

Brian

PS - I'm working on a patch that allows you to write custom function in python, but it's not done / released yet.