Re: [HTCondor-users] python api about history

Mailing List Archives Authenticated access	UW Madison Computer Sciences Department Computer Systems Lab

Also worth noting that the remote history option is limited to 10k jobs by default. So, regardless of how much data is on disk, no more than 10k ads should be parsed.

Sent from my iPhone

On Nov 15, 2016, at 5:31 AM, Fischer, Max (SCC) <max.fischer@xxxxxxx> wrote:

Hi,

since history apparently is implemented as a generator, you should break by *yourself* in python code. The history itself has no way of knowing that an arbitrary constraint is satisfied at a certain point.

Pseudocode:

   max_age = 60*30 # only care for jobs in the last 30 minutes
   new_jobs = []
   for job in htcondor.Schedd.history():
       if job['EnteredCurrentStatus'] < time.time() - max_age:
           break # exit early, only older jobs after this point
       new_jobs.append(job)

Cheers,
Max

Am 15.11.2016 um 11:09 schrieb jiangxw@xxxxxxxxxxxxxxx:

Thank you very mach.
Hope history() can provide a parameter setting history file like "-file" in condor_history command.

NAMEïJiang Xiaowei
MAILïjiangxw@mail.ihep.ac.cn
TELï010 8823 6024
DEPARTMENTïComputing Center of IHEP

From: Iain Steers
Date: 2016-11-15 17:56
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] python api about history
Ah I was thinking of the since and stop constraints.

https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=5642,56

Looks like they haven't been added to the python bindings yet.
________________________________________
From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of jiangxw@xxxxxxxxxxxxxxx[jiangxw@xxxxxxxxxxxxxxx]
Sent: 15 November 2016 10:53
To: htcondor-users
Subject: Re: [HTCondor-users] python api about history

Hi Iain,
   Thanks for your advice.
   I can catch your meaning.
   I implemented that function like yours.
   And the space of history files in our environments is 132G totally.
   When I execute the script with the lines
       requirements = "EnteredCurrentStatus <= %s && EnteredCurrentStatus > %s" % (current_time,pre_time)
       job_history = self.schedd.history(requirements, projection, 0),
   the script will take too long times. I think self.schedd.history() will look over all 132G history files.
   How can I optimize it?
   Thanks.

Cheers,
Jiang Xiaowei
________________________________
NAMEïJiang Xiaowei
MAILïjiangxw@mail.ihep.ac.cn
TELï010 8823 6024
DEPARTMENTïComputing Center of IHEP

From: Iain Steers<mailto:iain.steers@xxxxxxx>
Date: 2016-11-15 17:16
To: HTCondor-Users Mail List<mailto:htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] python api about history
Hi,

history() can take a Requirement constraint as one of its arguments.

In order to do this efficiently i would suggest something like the following.

start_dt = datetime.datetime.now() - datetime.timedelta(minutes=30)
start_stamp = calendar.timegm(d.timetuple())
schedd = htcondor.Schedd()
jobs = schedd.history('EnteredCurrentStatus >= {0}'.format(start_stamp))

This should return an iterator of all the jobs which were added to the history in the last half hour.

Cheers, Iain

________________________________________
From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of jiangxw@xxxxxxxxxxxxxxx[jiangxw@xxxxxxxxxxxxxxx]
Sent: 15 November 2016 10:08
To: htcondor-users
Subject: [HTCondor-users] python api about history

Hi all,
   I used python api to get job history information then insert job information into database.
      The function is htcondor.Schedd.history().
   But when the amount of the history files in "/var/lib/condor/spool/" became large,
   htcondor.Schedd.history() will be executed slowly.
   I want to gain the job history information every half hour. So I just need to look over the history file generated last half hour.
   But the htcondor.Schedd.history() may look over all history files in "/var/lib/condor/spool" every time.
   I can not find the parameter from htcondor.Schedd.history() like "condor_history -file".
   Are there some ways to solve it?
   Thanks for help.

Best regards,
Jiang XiaoWei
________________________________
NAMEïJiang Xiaowei
MAILïjiangxw@mail.ihep.ac.cn
TELï010 8823 6024
DEPARTMENTïComputing Center of IHEP
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

Mailing List Archives

Authenticated access

Re: [HTCondor-users] python api about history