Re: [HTCondor-users] python api about history

["Fischer, Max (SCC)" <max.fischer@xxxxxxx>]

Hi,

since history apparently is implemented as a generator, you should break by *yourself* in python code. The history itself has no way of knowing that an arbitrary constraint is satisfied at a certain point.

Pseudocode:

    max_age = 60*30  # only care for jobs in the last 30 minutes
    new_jobs = []
    for job in htcondor.Schedd.history():
        if job['EnteredCurrentStatus'] < time.time() - max_age:
            break  # exit early, only older jobs after this point
        new_jobs.append(job)

Cheers,
Max


> Am 15.11.2016 um 11:09 schrieb jiangxw@xxxxxxxxxxxxxxx:
> 
> Thank you very mach.
> Hope history() can provide a parameter setting history file like "-file" in condor_history command.
> 
> NAMEïJiang Xiaowei
> MAILïjiangxw@xxxxxxxxxxxxxxx
> TELï010 8823 6024
> DEPARTMENTïComputing Center of IHEP
>  
> From: Iain Steers
> Date: 2016-11-15 17:56
> To: HTCondor-Users Mail List
> Subject: Re: [HTCondor-users] python api about history
> Ah I was thinking of the since and stop constraints.
>  
> https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=5642,56
>  
> Looks like they haven't been added to the python bindings yet.
> ________________________________________
> From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of jiangxw@xxxxxxxxxxxxxxx[jiangxw@mail.ihep.ac.cn]
> Sent: 15 November 2016 10:53
> To: htcondor-users
> Subject: Re: [HTCondor-users] python api about history
>  
> Hi Iain,
>     Thanks for your advice.
>     I can catch your meaning.
>     I implemented that function like yours.
>     And the space of history files in our environments is 132G totally.
>     When I  execute the script with the lines
>         requirements = "EnteredCurrentStatus <= %s && EnteredCurrentStatus > %s" % (current_time,pre_time)
>         job_history = self.schedd.history(requirements, projection, 0),
>     the script will take too long times.  I think self.schedd.history() will look over all 132G history files.
>     How can I optimize it?
>     Thanks.
>  
> Cheers,
> Jiang Xiaowei
> ________________________________
> NAMEïJiang Xiaowei
> MAILïjiangxw@xxxxxxxxxxxxxxx
> TELï010 8823 6024
> DEPARTMENTïComputing Center of IHEP
>  
> From: Iain Steers<mailto:iain.steers@xxxxxxx>
> Date: 2016-11-15 17:16
> To: HTCondor-Users Mail List<mailto:htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] python api about history
> Hi,
>  
> history() can take a Requirement constraint as one of its arguments.
>  
> In order to do this efficiently i would suggest something like the following.
>  
> start_dt = datetime.datetime.now() - datetime.timedelta(minutes=30)
> start_stamp = calendar.timegm(d.timetuple())
> schedd = htcondor.Schedd()
> jobs = schedd.history('EnteredCurrentStatus >= {0}'.format(start_stamp))
>  
> This should return an iterator of all the jobs which were added to the history in the last half hour.
>  
> Cheers, Iain
>  
> ________________________________________
> From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of jiangxw@xxxxxxxxxxxxxxx[jiangxw@mail.ihep.ac.cn]
> Sent: 15 November 2016 10:08
> To: htcondor-users
> Subject: [HTCondor-users] python api about history
>  
> Hi all,
>     I used python api to get job history information then insert job information into database.
>        The function is htcondor.Schedd.history().
>     But when the amount of the history files in "/var/lib/condor/spool/" became large,
>     htcondor.Schedd.history() will be executed slowly.
>     I want to gain the job history information every half hour. So I just need to look over the history file generated last half hour.
>     But the htcondor.Schedd.history() may look over all history files in "/var/lib/condor/spool" every time.
>     I can not find  the parameter from htcondor.Schedd.history() like "condor_history -file".
>     Are there some ways to solve it?
>     Thanks for help.
>  
> Best regards,
> Jiang XiaoWei
> ________________________________
> NAMEïJiang Xiaowei
> MAILïjiangxw@xxxxxxxxxxxxxxx
> TELï010 8823 6024
> DEPARTMENTïComputing Center of IHEP
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>  
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>  
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

Attachment: smime.p7s
Description: S/MIME cryptographic signature