[Condor-devel] request for discussion: unified file rotation code

[Erik Paulson <epaulson@xxxxxxxxxxx>]

CondorDB is creating another type of logfile for Condor, similiar to the
sql logfiles that quill++ uses. As part of that, we'll need to kill old
logfiles from time to time. 

It turns out that a lot of Condor needs to rotate/rollover a file
before writing a record to it. Just off the top of my head, dprintf,
the job_queue, the accountant log, and the schedd history file. The view
collector may do some rotation as well. All of them implement writing
records and rotating the file differently - some of them are very paranoid
(like dprintf, which blocks some signals) and others are more haphazard.

I think it would be good to consolidate much of this code into a new
object. I think we're fortunate in that most of the potential clients
have a "write this record to the file and rotate it if necessary" sort
of interface. My gut feeling right now is that wrapping FILE* might
be the way to go - just like we've got MyString around char*, maybe we
want a MyFile (though it'd be nice if it had a less dumb name). My goal
would be to have an object has basically all of the functionality of a
plain old FILE*, so we could route as much file output to this object,
but have an opportunity to be clever in just one object if we wanted to
(maybe we could use this to re-enable remote logging, or shuffling data
into one daemon to be logged there). I think one key is that it doesn't
have any semantic knowledge of what's being logged, so it can be the
job queue or the user log or whatever. This is not trying to 
replicate log4j or dprintf, but it's the sort of thing we could build
undernearth them.

Some things that I think would be good to support:

MyFile::Init(char *basefilename, (int *)(void *data)onRotationCallback, 
			bool deleteOldFiles=true, int maxNumberToKeep=1, 
			int max_size=64k, char *extenstionFormatStr=NULL);
MyFile::WriteRecord(char *record, int record_size);

MyFile::GetCurrentSize();
MyFile::SetMaxSize(int max_size);
MyFile::SetLockFunction( bool lock_fn_ptr);

The idea for the initialization funtion is that it could handle the case where:
 -- we want to do some action before rotating the file, ala clean the job queue
 -- maybe we don't want the old files deleted at all, because something else is
    going to consume them
 -- or maybe we know we only want one old one around, ala dprintf
 -- the extension string could be something like "%4d", which would get
    sprintf'd onto the end. The data could be like the number of rotations
    that have happened, or the timestamp, or maybe the onRotationCallback could 
    set it. 

It'd be good to shove locking into the instance of the object itself, so we
could provide a default locking method, but override it as needed - or at least
be able to put specific data into the object that the lock function could get
at. One place to use that could be for user logs, so they could put their lock 
files somewhere off of NFS. 

I'm not sure if the object should know what user to do the write as, or if
should assume the higher level has done the appropriate set_user_priv() call. 

Thoughts?

-Erik