Re: [HTCondor-users] Storing HTCondor user log files centrally

["Smith, Ian" <I.C.Smith@xxxxxxxxxxxxxxx>]

Hi Matt,

 

I think Todd touched on this in a previous thread:

 

https://lists.cs.wisc.edu/archive/htcondor-users/2012-November/msg00135.shtml

 

<quote>

Finally, re ND's Condor Log Analyzer, you may be interested in the HTCondor config knob "EVENT_LOG" (see the Manual)- this specifies a file that receives job user log events, but across all users and user's jobs on a given schedd. I added a link to Condor Log Analyzer here:

   https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=ContribModules

</quote>

 

I haven't tried this so I'm not sure if it records the usernames (my guess is it maybe just the jobIDs).

 

Here, I cheat and wrap condor_submit so log files are always created in a central location (even if the users

don't ask for them). This has the advantage that the logs are already broken down per-user,

and per-submission date but of course it is possible to get around this if users are particularly devious.

 

regards,

 

-ian.

 

-----

Dr Ian C. Smith,

Advanced Research Computing,

University of Liverpool UK.

 

http://condor.liv.ac.uk

 

 

 

 

From: htcondor-users-bounces@xxxxxxxxxxx [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Matthew Forshaw
Sent: 11 December 2012 10:03
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Storing HTCondor user log files centrally

 

Good afternoon all,

 

Hopefully a fairly simple configuration question for you all.

 

My aim is to collect detailed logs for all jobs going through our pool to carry out some analyses as part of my PhD. We currently collect fairly coarse-grained workload traces but are now interested in gaining a deeper insight into what is happening on a per-job level.

 

I was wondering if it is possible to specify a central location where all HTCondor user logs are stored, in addition to the location specified in the users' submit files.

 

1. I would like this to happen automatically without users having to change their submit scripts.

2. I would like logs to be stored even if a log file location has not been specified inside the job description.

3. Ideally I would like these logs to be in ClassAd XML format but I should be able to parse the output regardless of format.

 

If anyone has attempted something similar before, or could advise me on the best way to go about this, I would really appreciate it. Thanks in advance for any help you can offer!

 

All the best,

Matt