Re: [HTCondor-users] python multiple job event logs in parallel

[David Schultz <david.schultz@xxxxxxxxxxxxxxxx>]

Hi Cole,

That's a possibility, even though the program probably only controls 25% of the jobs on the AP. Does the global event log roll over properly, or does it just get infinitely big? That's the main problem with writing to a single user log, when you start to talk about 50k or more jobs a day and running this "forever."

Best,

David

On Fri, Oct 6, 2023 at 10:48âAM Cole Bollig via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

Hi David,

Depending on what the code is doing, it may be beneficial to turn on the Global event log, and just read all events from the single file. This could be useful for a program that monitors the entire AP since the global event log records all events for all jobs in a single file.

-Cole Bollig

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of David Schultz <david.schultz@xxxxxxxxxxxxxxxx>
Sent: Thursday, October 5, 2023 4:00 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] python multiple job event logs in parallel

Â

Hi,

I'm looking at migrating code from querying the schedd directly to reading job event logs, but I would need to query multiple log files. The current python bindings make this a sequential operation, even trying from multiple threads, due to a global mutex.

So, requests:

1. If possible, could this mutex be per-log?

2. Allow polling of the job event log (a true 0 timeout). This would allow checking multiple logs for updates sequentially much faster. Since I can see that this might break existing code, allowing a float for a 1 millisecond timeout would also work well, as it seems like the C++ code natively uses milliseconds.

Best,

David

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/