Re: [HTCondor-users] POOL_HISTORY_MAX_STORAGE seems to be ignored on Windows

[John M Knoeller <johnkn@xxxxxxxxxxx>]

This is indeed what is happening.  The code is assuming that rename will delete the file it is trying to rename to, but that does not happen on Windows.  this is now bug 7438

https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=7438

 

Thank you for letting us know.

-tj

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Alexey Smirnov
Sent: Wednesday, December 18, 2019 11:34 AM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] POOL_HISTORY_MAX_STORAGE seems to be ignored on Windows

 

Hi!

We are running condor 8.8.0 on Windows cluster and noticed that the history files directory grows indefinitely regardless the settings "POOL_HISTORY_MAX_STORAGE = 50000000":

 Directory of C:\condor\history

03/07/2019  07:08 AM    <DIR>          .
03/07/2019  07:08 AM    <DIR>          ..
12/18/2019  05:35 PM        18,616,459 viewhist0.0.new
11/01/2018  12:21 AM         1,666,782 viewhist0.0.old
12/18/2019  05:35 PM         3,405,423 viewhist0.1.new
01/08/2019  09:29 PM         1,666,711 viewhist0.1.old
12/18/2019  05:03 PM         1,269,576 viewhist0.2.new
12/18/2019  05:35 PM       898,349,280 viewhist1.0.new
10/10/2018  10:41 PM         1,667,005 viewhist1.0.old
12/18/2019  05:35 PM       225,003,773 viewhist1.1.new
10/12/2018  10:17 AM         1,668,903 viewhist1.1.old
12/18/2019  05:03 PM        56,321,841 viewhist1.2.new
10/18/2018  07:37 AM         1,676,836 viewhist1.2.old
[...]
              24 File(s)  1,260,128,938 bytes 

CollectorLog confirms that all settings are correct:
12/18/19 11:44:17 Configuration: SAMPLING_INTERVAL=60, MAX_STORAGE=50000000, MaxFileSize=1666666, POOL_HISTORY_DIR=C:\condor\history

If I manually delete ".old" files then almost immediately those big files are renamed into ".old". Therefore my guess is that the problem is related to some bug in condor when it rotates history files reached the size limit. Most probably "rename" function is used without prior deleting of old file. This is crucial in Windows compare to Linux since rename simply fails if destination file already exists.

Any ideas if anything else could be a reason?

Thank you!

Alexey