Re: [HTCondor-users] How history file rotation works in HTCondor

[Cole Bollig <cabollig@xxxxxxxx>]

Hi Maria,

Yes, this history archive file is timestamped with the time it was rotated.

The main factor that causes history files to be rotated is based on the configuration option MAX_HISTORY_LOG. Which specifies the maximum size of a history file. When the writer goes to write new history, the writer will see if the current size of the file plus the size of the record about to be written is greater than the max size. By default, this value is set to 20 MB.

The history file can also be specified to rotate every day and month (checked when a record is written) controlled respectively by ROTATE_HISTORY_DAILY and ROTATE_HISTORY_MONTHLY. You can even control the number of old rotated files left around via MAX_HISTORY ROTATIONS.

One final thing, you may be interested in using the PER_JOB_HISTORY_DIR option. This will inform the history writer to write the job record to both the full history file and a unique per job file in the specified directory. You can make your reader monitor this directory for new files and process the job records when noticed. An important thing to note about the job history directory is HTCondor does not manage/cleanup the records. HTCondor assumes the reader will remove/cleanup the per job record files. If not cleaned up then HTCondor will keep writing until the filesystem is full.

Cheers,

Cole Bollig

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Maria Alandes Pradillo via HTCondor-users <htcondor-users@xxxxxxxxxxx>
Sent: Wednesday, June 18, 2025 8:29 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Cc: Maria Alandes Pradillo <Maria.Alandes.Pradillo@xxxxxxx>
Subject: [HTCondor-users] How history file rotation works in HTCondor

 

Hi,

 

I’m trying to understand how HTCondor rotates history files. I’m using them for our accounting and I’m trying to validate accounting data from different accounting systems (all using HTCondor history files one way or another), against the “raw” history files themselves.

 

Some examples from our testing pool where schedulers are used infrequently:

-rw-r--r--. 1 condor condor 215779 Apr 15 13:04 history.20250526T115455

-rw-r--r--. 1 condor condor  12979 May 26 12:43 history.20250527T140056

-rw-r--r--. 1 condor condor  18276 May 27 16:27 history.20250611T161637

 

My understanding is that the name of the history file is the moment when it’s rotated.

 

In our production pool, where there are a lot of jobs, history files are rotated basically every day at midnight. So what’s the criteria to rotate a file? Is this configured somewhere?

 

Thanks a lot,

Maria