[HTCondor-users] logstash grok for XML Event Log

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi all,

I wonder, if somebody has already a logstash grok to mutate the results
into a nicer format?

I am writing our CondorCE's event log as xml [1] and put a logstash grok
onto it [2], that parses reasonably well the individual events [3.a]
into JSONs [3.b].

Thing is, that I would like to mutate the <a n="foobar"> tags.
AFAIS the `actual key` is always the tag's single n-attribute and a
value is wrapped in one of the int/str/real type-tags.
So, I am looking on how to best mutate the attribute to become the key
(instead of the tag 'a') and carve out the value from the type-tag.

Since I am not really an expert with grok, I am hoping, that maybe
somebody has already a mutate or so at hand, that I could borrow...? ;)

Cheers,
  Thomas

[1]
EVENT_LOG = /var/log/condor-ce/EventLog.xml
EVENT_LOG_MAX_SIZE =  500000000
EVENT_LOG_MAX_ROTATIONS = 4
EVENT_LOG_USE_XML=True

==============================================================

[2]
input {
  file {
    path => "/var/log/condor-ce/EventLog.xml"
    start_position => "beginning"
    sincedb_path => "/var/log/condor-ce/.EventLog.sincedb"
    exclude => "*.gz"
    type => "xml"
      codec => multiline {
        pattern => "<c>"
        negate => "true"
        what => "previous"
      }
  }
}

filter{
    xml{
        source => "message"
	store_xml => true
	target => "events"
	xpath => [
            "/stations/station/id/text()", "station_id",
            "/stations/station/name/text()", "station_name"
	]
    }
}

==============================================================

[3.a]
<c>
    <a n="SentBytes"><r>0.0</r></a>
    <a n="TotalRemoteUsage"><s>Usr 0 00:00:33, Sys 0 00:00:16</s></a>
    <a n="TotalLocalUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a>
    <a n="EventTypeNumber"><i>5</i></a>
    <a n="TotalSentBytes"><r>0.0</r></a>
    <a n="Subproc"><i>0</i></a>
    <a n="MyType"><s>JobTerminatedEvent</s></a>
    <a n="RunRemoteUsage"><s>Usr 0 00:00:33, Sys 0 00:00:16</s></a>
    <a n="EventTime"><s>2020-09-17T16:44:29.367</s></a>
    <a n="Cluster"><i>64876</i></a>
    <a n="Proc"><i>0</i></a>
    <a n="ReceivedBytes"><r>0.0</r></a>
    <a n="TerminatedNormally"><b v="t"/></a>
    <a n="TotalReceivedBytes"><r>0.0</r></a>
    <a n="ReturnValue"><i>0</i></a>
    <a n="RunLocalUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a>
</c>

==============================

[3.b]
> grep TotalRemoteUsage /tmp/logstash.eventxml.json | head -n1 | jq .
{
  "host": "grid-htcondorce0.desy.de",
  "events": {
    "a": [
      {
        "n": "SentBytes",
        "r": [
          "0.0"
        ]
      },
      {
        "n": "TotalRemoteUsage",
        "s": [
          "Usr 0 00:00:33, Sys 0 00:00:16"
        ]
      },
      {
        "n": "TotalLocalUsage",
        "s": [
          "Usr 0 00:00:00, Sys 0 00:00:00"
        ]
      },
      {
        "n": "EventTypeNumber",
        "i": [
          "5"
        ]
      },
      {
        "n": "TotalSentBytes",
        "r": [
          "0.0"
        ]
      },
      {
        "n": "Subproc",
        "i": [
          "0"
        ]
      },
      {
        "n": "MyType",
        "s": [
          "JobTerminatedEvent"
        ]
      },
      {
        "n": "RunRemoteUsage",
        "s": [
          "Usr 0 00:00:33, Sys 0 00:00:16"
        ]
      },
      {
        "n": "EventTime",
        "s": [
          "2020-09-17T16:44:29.367"
        ]
      },
      {
        "n": "Cluster",
        "i": [
          "64876"
        ]
      },
      {
        "n": "Proc",
        "i": [
          "0"
        ]
      },
      {
        "n": "ReceivedBytes",
        "r": [
          "0.0"
        ]
      },
      {
        "n": "TerminatedNormally",
        "b": [
          {
            "v": "t"
          }
        ]
      },
      {
        "n": "TotalReceivedBytes",
        "r": [
          "0.0"
        ]
      },
      {
        "n": "ReturnValue",
        "i": [
          "0"
        ]
      },
      {
        "n": "RunLocalUsage",
        "s": [
          "Usr 0 00:00:00, Sys 0 00:00:00"
        ]
      }
    ]
  },
  "type": "xml",
  "@version": "1",
  "@timestamp": "2020-09-17T15:00:21.876Z",
  "message": "<c>\n    <a n=\"SentBytes\"><r>0.0</r></a>\n    <a
n=\"TotalRemoteUsage\"><s>Usr 0 00:00:33, Sys 0 00:00:16</s></a>\n    <a
n=\"TotalLocalUsage\"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a>\n    <a
n=\"EventTypeNumber\"><i>5</i></a>\n    <a
n=\"TotalSentBytes\"><r>0.0</r></a>\n    <a n=\"Subproc\"><i>0</i></a>\n
   <a n=\"MyType\"><s>JobTerminatedEvent</s></a>\n    <a
n=\"RunRemoteUsage\"><s>Usr 0 00:00:33, Sys 0 00:00:16</s></a>\n    <a
n=\"EventTime\"><s>2020-09-17T16:44:29.367</s></a>\n    <a
n=\"Cluster\"><i>64876</i></a>\n    <a n=\"Proc\"><i>0</i></a>\n    <a
n=\"ReceivedBytes\"><r>0.0</r></a>\n    <a n=\"TerminatedNormally\"><b
v=\"t\"/></a>\n    <a n=\"TotalReceivedBytes\"><r>0.0</r></a>\n    <a
n=\"ReturnValue\"><i>0</i></a>\n    <a n=\"RunLocalUsage\"><s>Usr 0
00:00:00, Sys 0 00:00:00</s></a>\n</c>",
  "tags": [
    "multiline"
  ],
  "path": "/var/log/condor-ce/EventLog.xml"
}

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature