[HTCondor-users] condor_history JSON malformed with multiple files

["Fischer, Max (SCC)" <max.fischer@xxxxxxx>]

Hi all,

I am seeing some spurious errors when dumping condor_history as json. After about 4000/26000 jobs, there is a list close/open inserted [1].
So, the JSON is formatted as
	..., {...}][, {...}, ...
when it should be
	..., {...}, {...}, ...
with {...} as individual job data mappings.
The attributes do not seem to matter, even if I have all, 1 or no attribute, it breaks at the same number of jobs [2].

It seems the error is that condor_history adds the list open/close for every file read [3], and we have multiple history files. If files are read by the schedd via `-name $(hostname)`, the problem does not occur [4].

Cheers,
Max

[1]# condor_history -json -attributes ServerTime -match 4150 | tail -25
}
,
{
}
]
[
,
{
}
,
{
}
,
{
}
,
{
}
,
{
}
,
{
}
]

[2]
# condor_history -json -attributes ProcId | python -m json.tool >/dev/null
Extra data: line 16726 column 1 - line 104846 column 1 (char 83622 - 524198)
# condor_history -json -attributes Environment | python -m json.tool >/dev/null
Extra data: line 16726 column 1 - line 104846 column 1 (char 4081684 - 25601144)
# condor_history -json -attributes Foobar | python -m json.tool >/dev/null
Extra data: line 12548 column 1 - line 78640 column 1 (char 25094 - 157278)

[3]
https://github.com/htcondor/htcondor/blob/09f56c3a5324d8066dc63f4c49be4969937102bf/src/condor_tools/history.cpp#L1715
https://github.com/htcondor/htcondor/blob/09f56c3a5324d8066dc63f4c49be4969937102bf/src/condor_tools/history.cpp#L1467

[4]
# condor_history -json -attributes Foobar -name $(hostname) | python -m json.tool >/dev/null
# condor_history -json -attributes Foobar  | python -m json.tool >/dev/null
Extra data: line 13469 column 1 - line 79561 column 1 (char 26936 - 159120)

Attachment: smime.p7s
Description: S/MIME cryptographic signature