Re: [HTCondor-users] Splitting condor_history by user/task?

[Max Fischer <mfischer@xxxxxxxxxxxxxxxxxxxx>]

Hi Pete,

interesting, but not what I'm looking for for our job manager. I'll keep 
it in mind for a future plan we still have on the drawing board, though.
The point of our job-manager is that it works with only minimal 
dependencies on the batch systems. It's meant to plug into existing 
batch systems we are allowed to use but do not administrate, so it must 
be robust against configuration changes. Core components such as 
condor_q, ClassAd features, job settings, job Log or the history file 
would be reliable for us, whereas specific configuration settings or 
even other services should at best be only used for optimization when 
available.
-Max


On 01/28/2013 02:30 PM, Peter MacKinnon wrote:
> On 01/28/2013 05:04 AM, Max Fischer wrote:
> > Hi,
> >
> > our job-manager tool's Condor wrapper is showing increasing delays 
> > when extracting information from condor_history (and to a lesser 
> > extend, condor_q). This is especially the case with user pools that 
> > do not split the history file regularly and have lots of user jobs, 
> > preventing our wrapper from partitioning its search. The main problem 
> > seems to be that regardless of constraints, condor_history will go 
> > through the entire history file merely suppressing lines that do not 
> > match the constraint.
> > What I'm looking for is a way to tell Condor to split its history/q 
> > file (or make a copy of the relevant parts) either for each user or 
> > preferably for a dynamic key of the job/task. Something akin to job 
> > Log in combination with EVENT_LOG_JOB_AD_INFORMATION_ATTR, but the 
> > important thing is that we need to be able to define it solely 
> > through the job so that we do not require the pool to be configured 
> > specifically for us.
> > Cheers,
> > Max
> > _______________________________________________
> > HTCondor-users mailing list
> > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx 
> > with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/htcondor-users/
> Hi Max,
>
> Not sure it's exactly what you're looking for but I've developed a 
> contrib that (among other things) stores job history ads into mongodb. 
> There's an existing tool to query the db much like condor_history 
> based on various parameters, including a user name (which is indexed). 
> Unfortunately there is no full ClassAd constraint expression parsing 
> capability so your queries are really what you can do from a mongodb 
> client such as the provided tool (written in pymongo).
> http://htcondor-git.cs.wisc.edu/?p=condor.git;a=blob;f=src/condor_contrib/plumage/README 
> (scroll to bottom for history tool details)
> Let me know if you have further questions if any of that sounds 
> potentially useful.
> \Pete