Re: [HTCondor-users] Event log to kafka-like system

[Samik Raychaudhuri <samikr@xxxxxxxxx>]

Hi Kevin,

Thanks for the input. Couple of questions/comments:

• I am stumbling at the very beginning: I am not able to find the "global event log" file, i.e. I do not see /var/log/condor/EventLog. Is there a setting to enable this? I could not find something similar in: [http://research.cs.wisc.edu/htcondor/manual/v8.6/3_5Configuration_Macros.html]. I am using 8.6.4 - the most recent version on a Debian VM.
  
• Given the "condor" patterns file, looks like you are using the text version of the log file, rather than the xml version.  I am assuming that those are the two options, right? (the secret question is, there is no JSON option, right? :-))
• The way you are doing this indicates that, I can roll out my own kafka producer by tailing the files myself (e.g., by using Tailer from Apache Commons IO), if I do not want Elasticsearch (ES) in the loop (although I am pretty familiar with these systems, and completely see the value of ES/Graphana). Will probably try that out.
• Curious from that perspective and slightly off-topic: since you already have the data in ES, why are you using graphana instead of kibana?
  

Regards,
-Samik

On 05-Jul-17 2:04 AM, Kevin Retzke wrote:

We currently use Logstash to tail the global event log and send it directly to Elasticsearch [1]. Logstash can easily send events to Kafka instead, or just about anywhere else (e.g. we're putting RabbitMQ into our pipeline). We've found having the event log in Elasticseach has been hugely useful for system analytics and troubleshooting.

[1] https://github.com/fifemon/logstash-config/blob/master/condor.logstash.conf

/Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Samik Raychaudhuri <samikr@xxxxxxxxx>
Sent: Saturday, July 1, 2017 9:39 AM
To: HTCondor-Users Mail List
Subject: [HTCondor-users] Event log to kafka-like system

 

Hello,

Wanted to process the event logs from a submitted job as they come in. Is there a way to get the logs in an event queue like kafka (http://kafka.apache.org/)? Is there a socket from where I can directly listen to, and read the events using an external program?

Apache Kafka

kafka.apache.org

Apache Kafka: A Distributed Streaming Platform. ... Kafka™ is used for building real-time data pipelines and streaming apps.

Just exploring!

Regards,
-Samik

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/