Re: [HTCondor-users] Efficiency & centralization of global information gathering?

[John M Knoeller <johnkn@xxxxxxxxxxx>]

Generic ads are actually at specific type of ad.   What you want to use is
 
What you actually want is condor_status -subsytem 'mytype'

This will make a generic query to the collector for ads of type 'mytype'

On the other hand, if your ads actually ARE of type generic, then condor_status -generic is the correct query to use.

Hint: you can look at the collector log to see what it actually sees for any condor_status query, grep for 'Query info'

-tj



-----Original Message-----
From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Fischer, Max (SCC)
Sent: Thursday, January 5, 2017 4:00 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Efficiency & centralization of global information gathering?

Hi Michael,

1. Nice! I have to try that myself. ^^

Can you pull your custom classad also with condor_status -generic (instead of -any)?
Not entirely sure how condor does the filtering, but it may be more efficient.

2. Concerning the "modifying startd ClassAds" approach:
You can push values to any startd using condor_update_machine_ad [1]. I can confirm this is working persistently as long as the startd lives (yes, it did hurt a lot in our case).

Scalability wise, it shouldn't matter whether you push or pull information: That blob of meta-data seems negligible compared to what jobs consume.
If your workers go up and down a lot (e.g. VMs), pulling is easier to keep synchronised. If the value changes infrequently, pushing is more responsive.

Personally, I prefer every startd pulling its own information. Makes it easier with heterogeneous resources.

Cheers,
Max

[1]
http://research.cs.wisc.edu/htcondor/manual/current/condor_update_machine_ad.html

> Am 05.01.2017 um 01:11 schrieb Michael Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>:
> 
> Max,
> 
> Thanks for that suggestion! For my FlexLM problem, I just wrote a quick Perl script which you call like so:
> 
> 	flexlm2classad lmstat -a
> 
> This runs the "lmstat -a" command (or you can feed it flexlm data on stdin), and converts it into a ClassAd that looks like so:
> 
> MyType = "Generic"
> Name = "FlexLM"
> FlexLM_Available_a_spaceclaim_dirmod = 6
> FlexLM_Available_acfd = 3	
> FlexLM_Available_acfd_flo = 1
> FlexLM_Available_agppi = 8
> ...etc...
> 
> The identifier is the feature name for each license, and it's derived from the "Users of" lines like so:
> 
> Users of a_spaceclaim_dirmod:  (Total of 8 licenses issued;  Total of 2 licenses in use)
> 
> Then this can be pulled out for use by a startd_cron job or what have you with:
> 
> 	condor_status -any -constraint 'Name == "FlexLM"' -af:lrng FlexLM_Available_a_spaceclaim_dirmod
> 
> I noticed that in the help output for condor_advertise, there's a "MERGE_STARTD_AD" option, but it's not mentioned in the man page and it doesn't seem to let me add an attribute to an existing startd ad even if I structure it as a Query type like the invalidate commands. Maybe someone from the CHTC pantheon can enlighten us on this point.
> 
> 	-Michael Pelletier.
> 
> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Fischer, Max (SCC)
> Sent: Wednesday, January 04, 2017 1:59 PM
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] Efficiency & centralization of global information gathering?
> 
> Hi Michael,
> 
> I've found this to be best solved outside of Condor.
> 
> 1. Have a regular cron job *somewhere* fetch the data once.
> 2. Provide that data via files on shared filesystems.
> 3. Have startd_cron read from the file.
> 4. ???
> 5. Profit
> 
> The trick is just to have 1. and 3. separate. There's no problem having 1. create proper classad already, and 3. just using cat.
> 
> Note that using files for 2. is historical laziness on my part:
> You can just as well publish this information via custom ClassAds. I think condor_advertise with UPDATE_AD_GENERIC should do the trick.
> 
> Cheers,
> Max
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/