[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Delay in job log writing



On 8/12/2013 9:17 AM, éææ wrote:
Hi, I find that there is delay in job log file writing. In my test,
condor_q showed one job were running, but there were only a submit event of
it in the log file. After a while, more events occurs and they would be
wrote together. I think this is because of the buffer in write/fwrite
function.

I hope that my log reader program can know events as soon as possible, so
is there  any configuration can do this? I think condor_shadow should fsync
after writing one event into log file.



Events written to the job event log (specified via "log=<file>" in the job submit file) are fsynced to disk by default. Events written to the schedd-wide event log (specified via "EVENT_LOG=<file>" in the condor_config file) are not fsynced by default. In either case, you can enable/disable fsyncing via the condor_config knobs

  ENABLE_USERLOG_FSYNC = True | False
  EVENT_LOG_FSYNC = True | False

Given that fsync is probably happening by default in your case, I think the delay you are observing is due to the following:

The condor_q tool reports a job as "running" as soon as a condor_shadow process is spawned for the job. However, the execute event does not show up in the event log until the moment before the job is spawned on the remote execute machine. Thus the time it takes the shadow to activate the claim on the execute machine (aka start a condor_starter) and potentially transfer any input files will show up as running time with condor_q, but an execute event will not appear in the log until these operations complete.

Hope the above helps
Todd



Thanks!



_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/



--
Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing   Department of Computer Sciences
HTCondor Technical Lead                1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                  Madison, WI 53706-1685