Re: [HTCondor-users] Event log to kafka-like system

[Kevin Retzke <kretzke@xxxxxxxx>]

We currently use Logstash to tail the global event log and send it directly to Elasticsearch [1]. Logstash can easily send events to Kafka instead, or just about anywhere else (e.g. we're putting RabbitMQ into our pipeline). We've found having the event log in Elasticseach has been hugely useful for system analytics and troubleshooting.

[1] https://github.com/fifemon/logstash-config/blob/master/condor.logstash.conf

/Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Samik Raychaudhuri <samikr@xxxxxxxxx>
Sent: Saturday, July 1, 2017 9:39 AM
To: HTCondor-Users Mail List
Subject: [HTCondor-users] Event log to kafka-like system

 

Hello,

Wanted to process the event logs from a submitted job as they come in. Is there a way to get the logs in an event queue like kafka (http://kafka.apache.org/)? Is there a socket from where I can directly listen to, and read the events using an external program?

Apache Kafka

kafka.apache.org

Apache Kafka: A Distributed Streaming Platform. ... Kafka™ is used for building real-time data pipelines and streaming apps.

Just exploring!

Regards,
-Samik