Re: [Condor-users] job's log file: date does not indicate the year !?!

[Matthew Farrellee <matt@xxxxxxxxxx>]

Great catch, and it's messy.

ULogEvent::writeHeader in condor_c++_util/condor_event.cpp isn't very 
flexible about the header,

   // write header
   retval = fprintf (file, "%03d (%03d.%03d.%03d) %02d/%02d 
%02d:%02d:%02d ",
                 eventNumber,
                 cluster, proc, subproc,
                 eventTime.tm_mon+1, eventTime.tm_mday,
                 eventTime.tm_hour, eventTime.tm_min, eventTime.tm_sec);

There's a ticket about unifying log timestamps,

https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=81

and it expresses concern about changing the format of the event logs 
(UserLog and EVENT_LOG).

Unfortunately, ULogEvent::readHeader, from the same file, is also not 
very flexible when it comes to the date format,

   // read from file
   retval = fscanf (file, " (%d.%d.%d) %d/%d %d:%d:%d ",
                &cluster, &proc, &subproc,
                &(eventTime.tm_mon), &(eventTime.tm_mday),
                &(eventTime.tm_hour), &(eventTime.tm_min),
                &(eventTime.tm_sec));

   // check if all fields were successfully read
   if (retval != 8)
   {
      return 0;
   }

A change in the format would certainly have forward compatibility 
issues. The issues would be present for users of the so-called 
libcondorapi, or at least those who statically link it. Oh yeah, does 
anyone want to admit to using libcondorapi? For instance, an old DAGMan 
would have an issue reading a new UserLog, though some thought would 
need to go into evaluating if that's a likely scenario.

If you're interested in a possible change to the event log header 
format, it might be something to consider for the 7.7 development series.

Best,


matt

On 11/18/2010 08:54 PM, Edier Alberto Zapata Hernández wrote:
> Good night Rob,
>
>   I just tested the Matt's method and it was great with the Condor's log
> files but not with the Condor Job's log files. I'd check the log_xml
> option in the submit files and got a better date field as you can see next.
>
> *************************The Test Submit
> File*********************************
> ##
> # Test Submit File
> ##
> # Use: condor_submit testTask.condor
> should_transfer_files = Yes
> when_to_transfer_output = ON_EXIT_OR_EVICT
> requirements = Memory >= 512 && (Arch=="X86_64" || Arch=="INTEL")
> transfer_executable=false
>
> Executable = /bin/hostname
> Universe = vanilla
> Output = hostOut.$(Process)
> Error = hostErr.$(Process)
> Log = hostLog.txt
> log_xml=true
> Queue
>
> *******************The Test Command and log file*******************
> [condor:~] condor_submit testTask1.condor ; tail -f hostLog.txt
> Submitting job(s).
> Logging submit event(s).
> 1 job(s) submitted to cluster 31.
> </c>
> <c>
> <a n="MyType"><s>SubmitEvent</s></a>
> <a n="EventTypeNumber"><i>0</i></a>
> <a n="EventTime"><s>2010-11-18T20:45:25</s></a>
> <a n="Cluster"><i>31</i></a>
> <a n="Proc"><i>0</i></a>
> <a n="Subproc"><i>0</i></a>
> <a n="SubmitHost"><s>&lt;192.168.20.156:57390&gt;</s></a>
> </c>
> <c>
> <a n="MyType"><s>ExecuteEvent</s></a>
> <a n="EventTypeNumber"><i>1</i></a>
> <a n="EventTime"><s>2010-11-18T20:45:31</s></a>
> <a n="Cluster"><i>31</i></a>
> <a n="Proc"><i>0</i></a>
> <a n="Subproc"><i>0</i></a>
> <a n="ExecuteHost"><s>&lt;192.168.2.12:37860&gt;</s></a>
> </c>
> <c>
> <a n="MyType"><s>JobTerminatedEvent</s></a>
> <a n="EventTypeNumber"><i>5</i></a>
> <a n="EventTime"><s>2010-11-18T20:45:31</s></a>
> <a n="Cluster"><i>31</i></a>
> <a n="Proc"><i>0</i></a>
> <a n="Subproc"><i>0</i></a>
> <a n="TerminatedNormally"><b v="t"/></a>
> <a n="ReturnValue"><i>0</i></a>
> <a n="RunLocalUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a>
> <a n="RunRemoteUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a>
> <a n="TotalLocalUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a>
> <a n="TotalRemoteUsage"><s>Usr 0 00:00:00, Sys 0 00:00:00</s></a>
> <a n="SentBytes"><r>1.000000000000000E+01</r></a>
> <a n="ReceivedBytes"><r>0.000000000000000E+00</r></a>
> <a n="TotalSentBytes"><r>1.000000000000000E+01</r></a>
> <a n="TotalReceivedBytes"><r>0.000000000000000E+00</r></a>
> </c>
> *****************************End of Log file*****************************
>
> Maybe this help you.
>
> Bye.
>
> On Thu, Nov 18, 2010 at 8:22 PM, Matthew Farrellee <matt@xxxxxxxxxx
> <mailto:matt@xxxxxxxxxx>> wrote:
>
>     On 11/18/2010 08:12 PM, Rob wrote:
>
>
>         Hi,
>
>         I'm in the process of analyzing loads of log files in order to
>         provide
>         statistics on how efficient condor is handling certain types of
>         jobs in our
>         condor pool.
>
>         For that I need to know when a job is submitted and when it has
>         terminated. My
>         approach is to read the log file of those jobs and analyze the
>         time difference
>         between these two.
>
>         I then realized that the year is not in the log file. For short
>         and "normal"
>         jobs this is not an issue (for as long as you're not near the
>         year-end
>         transition!), but much longer jobs may have the year bumbed up
>         without it being
>         indicated in the log file.
>
>         So in theory, if a job is processed in a condor pool for little
>         over a year, the
>         log file then indicates that just a little time has passed
>         between its
>         submission and its termination (if I ignore all the other stuff
>         that is listed
>         in the log file between submission and termination).
>
>         OR: is the year automatically added when condor notices the date
>         has changed to
>         the next year?
>         If not, wouldn't adding the year into the log file make the file
>         more complete
>         and robust for analysis?
>
>         Thank you.
>         Rob.
>
>
>     For 7.5.1+, the year is present,
>
>     https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=1030
>
>     For 7.5.1-, you can add it yourself via DEBUG_TIME_FORMAT=%m/%d/%y
>     %H:%M:%S,
>
>     http://www.cs.wisc.edu/condor/manual/v7.5/3_3Configuration.html#15352
>
>     Best,
>
>
>     matt
>
>     _______________________________________________
>     Condor-users mailing list
>     To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
>     <mailto:condor-users-request@xxxxxxxxxxx> with a
>     subject: Unsubscribe
>     You can also unsubscribe by visiting
>     https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
>     The archives can be found at:
>     https://lists.cs.wisc.edu/archive/condor-users/
>
>
>
>
> --
> ----
> Edier Alberto Zapata Hernández
> Est. Ingeniería de Sistemas
> Universidad de Valle
>
>
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/