Re: [HTCondor-users] How can I avoid the ClassAds matching system?

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

Hi Chongguang,

Two thoughts:

(a)  You probably want to consider extending ClassAds instead of trying to do without ClassAds.

ClassAds for many years has had a loadable library interface - you could add your own functions using C++.  Starting in 8.3.5, you can also write new ClassAd functions using python (an interface with a lower learning curve).

You could add the following to the START requirements

START = MyGreatFunction()

where all the logic is hidden in your great function (you'll get the job and machine ad through the state keyword argument).  There's an immense amount of details* I'm skipping, but that's the crux.  The negotiator will then evaluate the START expression as part of the normal matchmaking process and invoke your function.

(b) Reconsider whether you can express your function in ClassAds

Recall that you can insert arbitrary attributes into the machine's classad through the startd cron mechanism - and ClassAds is a surprisingly expressive language!  You can simply have your metrics inserted into the classads and updated often.

So, which one to chose?  (b) is the traditional approach and semi-documented; (a) is uncharted territory (and technically more difficult) but allows for completely arbitrary policies.  I can't recall a policy that you *can't* write in ClassAds, but sometimes the policy is many lines long.

Hope this helps,

Brian

* ClassAd functions should have no side-effects, may be invoked at a very large rate and hence should finish in ~1ms, START is periodically invoked without a job ad to determine if the slot should go to owner state, you'll want to consider the fact it has to work on both workers and the matchmaker (so if it's somewhat expensive, you'll maybe want it to always return True on workers).

> On May 1, 2015, at 8:11 AM, Chongguang Liu <chongguang.liu@xxxxxxxxxxxx> wrote:
> 
> Hi Guys,
> 
> I'm implementing a grid computing system in my company using HTCondor, and I would like to know if it is possible to use HTCondor without his ClassAds matching functionality?
> 
> What I want to achieve is that when I submit a job from a machine, I can ask the master the send the job to a machine I personally choose. I want to do this not because the HTCondor's matching ClassAds algorithm is not good enough. It is because we want to collect information of each machine in the grid system and do some analysis to find the best machine according to our own criteria.
> Can any of the APIs do this? I read through the documentation and I didn't find useful information on it.
> 
> If this is not possible, I'm thinking of doing a trick. On the HTCondor Manual, I read the paragraph:
> Pools of Machines can be Hooked Together.
> Flocking is a feature of HTCondor that allows jobs submitted within a first pool of HTCondor machines to execute on a second pool. The mechanism is flexible, following requests from the job submission, while allowing the second pool, or a subset of machines within the second pool to set policies over the conditions under which jobs are executed.
> 
> Can the APIs do this? Because if it is possible, what I will try to do is: I set each machine as a pool, and each machine has the role of master, submit and execute. When submitting a job, we use our matching system to find the best machine for execution. Then we hook the two pools. 
> 
> Those are ideas in my mind, and I didn't find how to do them from the HTCondor Manual. Maybe they are just impossible to do, but I still want to have opinions from all of you and the developers of HTCondor. And it is urgent for me.
> 
> I'm looking forward to having answers asap.
> 
> Thanks a lot
> 
> Chongguang
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/