Re: [HTCondor-devel] idle thoughts: json as classads

[Erik Paulson <epaulson@xxxxxxxxxxxx>]

On Wed, Jun 26, 2013 at 10:32 AM, Todd Tannenbaum <tannenba@xxxxxxxxxxx> wrote:

On 6/26/2013 9:51 AM, Alain Roy wrote:

On Jun 25, 2013, at 1:45 PM, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:

While ClassAds have been occasionally used outside of HTCondor, the uptake was never overwhelming and I think has fallen off (are there any external ClassAd users anymore?). We may have more luck if we started to look a little more like what the rest of the world expects.

What about ProtoBufs?  They are expressive enough to encapsulate ClassAds.

Unfortunately, most interaction with HTCondor isn't really RPC -- it's an undocumented custom binary protocol with a mix of ClassAds and binary data.

Having more RPC-like interfaces would be a good goal; the SOAP clients were a good start toward a cross-language API, although the choice of data marshaling (SOAP) was unfortunate...

Maybe what Erik is really hoping for is a modern API to Condor that is easier to use. JSON ClassAds (if there was a nice way to pull them off) would be one step towards a RESTful API.

At one level, chasing the latest RPC style for an API isn't worth the time. On the other hand, there are some pretty significant toolsets that can help people build on top of a RESTful API. There are lots of _javascript_ libraries that make it easier to use REST and JSON to build a UI. It's also easier to consume as a client in other environments when building tools. That said, I understand that it may not be the top priority for Condor.

-alain

Pithy comments:


SOAP is still supported, both native in the schedd and collector and also via the Aviary contrib.

A RESTful interface is available via the CondorAgent contrib.

Also not clear that the browser is the most important client environment -vs- a scripting environment widely used in scientific computing like Python.... and HTCondor v8.0.0 now includes Python client packages that are built on top of libcondorapi.

-Todd

p.s. awesome to have west-coast Dr Roy in the discussion!

In this instance, I'm not specifically looking for a modern RESTful API, though I wouldn't object to one. 

I'm primarily looking for ways to migrate some of the work that we've done in HTCondor to a broader audience, and sometimes that means being closer to what the rest of the world is doing. 

As an example, ClassAds are a really interesting way to express and expose data, and an interesting example of where the query language is the same as the data representation. I think that's kind of cool, and I think more people would take it up if we lowered some barriers to entry, and one way to do that is to make it easier for other systems to handle it. 

I have no specific use in mind - though obviously there are web uses, and ClassAds native in _javascript_ could be helpful there. As Todd mentioned, there are other use cases that might be more important, like Python (and while I love the Python module and think you should push it front-and-center, some of the most interesting work in the scientific python world is being done in hosted python environments and runtimes which don't lend kindly to C++-based modules, so a pure python implementation of ClassAds with the same API could be a follow-on)

I think there's a lot to be learned from pushing some of the tools outside of the HTCondor world. For example, we still don't really know how a ClassAd should think of itself as part of a larger collection - how a job ad can reference the rest of the queue, or what's happening elsewhere in the pool. There are some solutions, but I don't think we're done thinking about how it might work. I think if we had more examples of people using ClassAds in other domains it'd be helpful in finding a method that we're really happy with. 

There have been some really missed opportunities for HTCondor to be a bigger presence in some hot fields that lend themselves naturally to what it does. HTCondor's a pretty reasonable workload management and execution engine, and it's not hard to imagine a alternate timeline where it was the basis for the next gen MapReduce, or basis for the Private Cloud distributions, or the basis for the open-source Platform-as-a-Service stacks. We might have been in those conversations if more parts of HTCondor were used outside of HTCondor.

So that's where I'm coming from with this thread. This is not a serious proposal for making the change in HTCondor 8.1. I do appreciate that flipping some parts of ClassAds around is a non-trivial change in HTCondor proper and is a long way off, if it ever happens. Maybe there's an easier way, or maybe there's a way to do it on the side, I dunno. 

-Erik