Re: [Condor-devel] Bug report: log file rotation and COLLECTOR_QUERY_WORKERS

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

At 03:43 PM 10/4/2005, Michael Yoder wrote:

> If COLLECTOR_QUERY_WORKERS is some positive number, there is a risk of
> the log file rotation code killing the collector.  The risk is greatly
> increased with a small log file size (lots of rotations) and a high
> debugging output, like D_ALL.


Thanks for the report!

Does adding the following into condor_config work as a workaround ?


COLLECTOR_LOCK = $(LOCK)/CollectorLock

If it does, maybe a fix could be to make Collector_Lock default to
NULL (nothing) if COLLECTOR_QUERY_WORKERS==0, and default to the above
if COLLECTOR_QUERY_WORKERS > 0.

Any thoughts/comments on this approach?

thanks,
Todd



> Here's what happens.  See preserve_log_file().
>
> 1) unlink ( CollectorLog.old )
> 2) link ( CollectorLog, CollectorLog.old )
> 3) unlink ( CollectorLog )
> 4) stat ( CollectorLog ) to see if CollectorLog exists.
>
> The parent collector gets to step #3 and is switched out.  A child
> collector wants to write to the (same) log file, sees that it doesn't
> exist, and creates it.  The parent comes back in, does step #4, and dies
> with:
>
> sprintf( msg_buf, "unlink(%s) succeeded but file still exists!",
>          DebugFile[debug_level] );
> _condor_dprintf_exit( save_errno, msg_buf );
>
> For a real-life example, please see
>
> http://docs.optena.com/display/CONDOR/2005/10/04/Collector+Log+Rotation
>
> I believe that the extra stat(), while nice and paranoid, isn't needed.
> If you really wanted to keep it and be both paranoid and correct, you
> could stat() the log file beforehand and compare their inode numbers.
>
> Have a nice day,
> -Mike
>
>
> _______________________________________________
> Condor-devel mailing list
> Condor-devel@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/condor-devel


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Todd Tannenbaum                       University of Wisconsin-Madison
Condor Project Research               Department of Computer Sciences
tannenba@xxxxxxxxxxx                  1210 W. Dayton St. Rm #4257
http://www.cs.wisc.edu/~tannenba      Madison, WI 53706-1685
Phone: (608) 263-7132  FAX: (608) 262-9777