Re: [HTCondor-users] python api about history

["jiangxw@xxxxxxxxxxxxxxx" <jiangxw@xxxxxxxxxxxxxxx>]

Hi Iain,

    Thanks for your advice.

    I can catch your meaning.

    I implemented that function like yours.

    And the space of history files in our environments is 132G totally.

    When I  execute the script with the lines

        requirements = "EnteredCurrentStatus <= %s && EnteredCurrentStatus > %s" % (current_time,pre_time) 

        job_history = self.schedd.history(requirements, projection, 0),

    the script will take too long times.  I think self.schedd.history() will look over all 132G history files.

    How can I optimize it?

    Thanks.

Cheers,

Jiang Xiaowei

---

NAME：Jiang Xiaowei

MAIL：jiangxw@xxxxxxxxxxxxxxx

TEL：010 8823 6024

DEPARTMENT：Computing Center of IHEP

 

From: Iain Steers

Date: 2016-11-15 17:16

To: HTCondor-Users Mail List

Subject: Re: [HTCondor-users] python api about history

Hi,

 

history() can take a Requirement constraint as one of its arguments.

 

In order to do this efficiently i would suggest something like the following.

 

start_dt = datetime.datetime.now() - datetime.timedelta(minutes=30)

start_stamp = calendar.timegm(d.timetuple())

schedd = htcondor.Schedd()

jobs = schedd.history('EnteredCurrentStatus >= {0}'.format(start_stamp))

 

This should return an iterator of all the jobs which were added to the history in the last half hour.

 

Cheers, Iain

 

________________________________________

From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of jiangxw@xxxxxxxxxxxxxxx [jiangxw@xxxxxxxxxxxxxxx]

Sent: 15 November 2016 10:08

To: htcondor-users

Subject: [HTCondor-users] python api about history

 

Hi all,

    I used python api to get job history information then insert job information into database.

       The function is htcondor.Schedd.history().

    But when the amount of the history files in "/var/lib/condor/spool/" became large,

    htcondor.Schedd.history() will be executed slowly.

    I want to gain the job history information every half hour. So I just need to look over the history file generated last half hour.

    But the htcondor.Schedd.history() may look over all history files in "/var/lib/condor/spool" every time.

    I can not find  the parameter from htcondor.Schedd.history() like "condor_history -file".

    Are there some ways to solve it?

    Thanks for help.

 

Best regards,

Jiang XiaoWei

________________________________

NAME：Jiang Xiaowei

MAIL：jiangxw@xxxxxxxxxxxxxxx

TEL：010 8823 6024

DEPARTMENT：Computing Center of IHEP

_______________________________________________

HTCondor-users mailing list

To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a

subject: Unsubscribe

You can also unsubscribe by visiting

https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

 

The archives can be found at:

https://lists.cs.wisc.edu/archive/htcondor-users/