Re: [HTCondor-users] Ever expanding Accountantnew.log

[Greg.Hitchen@xxxxxxxx]

I also meant to add that on this Central Manager, whenever condor is restarted the condor_negotiator

chews up 100% cpu for 30 minutes. I assume this is reading the Accountantnew.log file?

Excerpt from Negotiator log at startup.

 

04/05/17 13:57:30 MAX_ACCOUNTANT_DATABASE_SIZE=1000000

04/05/17 14:27:51 About to rotate ClassAd log /home/condor/spool/Accountantnew.log

04/05/17 14:27:51 failed to rotate log: safe_open_wrapper(/home/condor/spool/Accountantnew.log.tmp) returns -1

04/05/17 14:27:51 Accountant::Initialize - LogFileName=/home/condor/spool/Accountantnew.log

 

Cheers

 

Greg

 

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Greg.Hitchen@xxxxxxxx
Sent: Wednesday, 5 April 2017 12:06 PM
To: htcondor-users@xxxxxxxxxxx
Subject: [ExternalEmail] [HTCondor-users] Ever expanding Accountantnew.log

 

Hi All

 

We have noticed that one of our Central Managers has an ever increasing Accountantnew.log file

 

 

Running on SLES11SP3

 

$CondorVersion: 8.2.6 Dec 10 2014 BuildID: 287355 $

$CondorPlatform: x86_64_RedHat5 $

 

This is the relevant entry from NegotiatorLog (with NEGOTIATOR_DEBUG = D_MATCH D_ACCOUNTANT).

 

04/05/17 13:20:15 About to rotate ClassAd log /home/condor/spool/Accountantnew.log

04/05/17 13:20:15 failed to rotate log: safe_open_wrapper(/home/condor/spool/Accountantnew.log.tmp) returns -1

04/05/17 13:20:15 Accountant::UpdatePriorities - truncating database (prev size=16356650949)

04/05/17 13:20:15 Database has grown, expanding MAX_ACCOUNTANT_DATABASE_SIZE to -1646436470

 

The partition this file lives on shows this disk usage (nearly all of which is the Accountantnew.log file):

 

# df -k

Filesystem     1K-blocks     Used Available Use% Mounted on

/dev/sda3       41274688 16171032  23007016  42% /home/condor/spool

 

# ll

total 15990844

-rw-r--r-- 1 condor condor 16357128606 Apr  5 13:49 Accountantnew.log

-rw-r--r-- 1 condor condor      744905 Nov  3 18:20 history

-rw------- 1 condor condor      668019 Nov  3 18:20 job_queue.log

-rw------- 1 condor condor       53365 Nov  3 18:20 job_queue.log.1

-rw------- 1 condor condor         158 Nov  3 18:20 job_queue.log.4

drwxrwxrwt 2 condor condor        4096 Nov  3 18:20 local_univ_execute

-rw-r--r-- 1 condor condor          59 Nov  3 18:20 spool_version

 

Is there any way we can fix this? e.g. is there an independent tool to rotate/compress/truncate this file?

 

We would rather not lose the data/information.

 

Even so, just renaming the file and starting from scratch gives:

 

04/05/17 13:56:15 ERROR "failed to open log /home/condor/spool/Accountantnew.log, errno = 13" at line 176 in file /slots/05/dir_20841/userdir/src/condor_utils/classad_log.cpp

 

and condor_negotiator keeps bombing out and restarting.

 

Thanks for any info/help.

 

Cheers

 

Greg