Re: [HTCondor-users] Configuration of logging

[Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx>]

Dear Jaime, 

Am 18.04.2018 um 16:53 schrieb Jaime Frey:
>> On Apr 18, 2018, at 4:32 AM, Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx> wrote:
>>
>> Dear HTCondor experts,
>>
>> I just realized HTCondor takes care of logging and rotating the logs by itself, without leveraging existing tools such as logrotate. 
>> This caused an issue because we had a schedd logging (due to a problematic job) almost continuously over night, and this morning, the 2 GB log partition was full. 
>> Of course, I could reduce the log sizes and number of rotations using HTCondor parameters - but then I would (in the worst case) not be able to log more than a day in a few GB of storage. 
>>
>> Is there any way to trigger automatic compression of logfiles after rotation, similar as to what logrotate can do out of the box? 
>> For logfiles, this tends to save orders of magnitude of space. 
>>
>> Also, is there a way to specify how long logs are kept (instead of going by size or number of rotations)? 
>> Especially with the new GDPR it's common practice to (and I think everybody in Europe is even forced to...) specify how long log files are kept in units of "time" instead of "size". 
> 
> 
> HTCondor doesnât have a way to automatically compress old log files. I suppose you could set up a cron job to do so.
> 
> You can specify a time instead of a file size for when the daemon logs should be rotated, like so:
> 
> # Save a dayâs worth of logs
> MAX_SCHEDD_LOG = 4Hr
> MAX_NUM_SCHEDD_LOG = 6
> 
> See MAX_DEFAULT_LOG in the manual for details.

thanks, this is a valid workaround, indeed I did not know one could also specify a time instead of bytes there! 

However, my cronjob for compression would also need to: 
- Check which files to compress (e.g. by matching all uncompressed files with date-extension). 
- Take care by itself about expiring and deleting old logs. 

In the end, this feels like I have to reinvent parts of logrotate. 
It's sad I have to solve a problem once more which had been solved for good on all Unix platforms over two decades ago. 

Cheers,
	Oliver

> 
> Thanks and regards,
> Jaime Frey
> UW-Madison HTCondor Project
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
>

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature