|
What version of HTCondor are you running?
There was a bug regarding sorting of history information that was
fixed in 8.2.10.
https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=5305
...Tim
Dear Biranï
 ÂÂThanks for the advice
about job history.
  I tryed to limite the amount of jobs from
history file.
  Condor_history doesn't looks over history files
in time ordor,
  so I can not get the latest
jobs from 10K jobs limited sometimes.
  Whether in our
environment there's something wrong?
Cheers,
Jiang Xiaowei
NAMEïJiang Xiaowei
TELï010 8823 6024
DEPARTMENTïComputing Center
of IHEP
Â
Date:Â2016-11-18Â09:01
Subject:ÂRe: [HTCondor-users] python api about
history
Also worth noting that
the remote history option is limited to 10k jobs by
default. ÂSo, regardless of how much data is on disk, no
more than 10k ads should be parsed.
Sent from my iPhone
Hi,
since history apparently is implemented as a
generator, you should break by *yourself* in python
code. The history itself has no way of knowing that an
arbitrary constraint is satisfied at a certain point.
Pseudocode:
ÂÂÂmax_age = 60*30 Â# only care for jobs in the
last 30 minutes
ÂÂÂnew_jobs = []
ÂÂÂfor job in htcondor.Schedd.history():
ÂÂÂÂÂÂÂif job['EnteredCurrentStatus'] <
time.time() - max_age:
ÂÂÂÂÂÂÂÂÂÂÂbreak Â# exit early, only older jobs
after this point
ÂÂÂÂÂÂÂnew_jobs.append(job)
Cheers,
Max
Am
15.11.2016 um 11:09 schrieb jiangxw@xxxxxxxxxxxxxxx:
Thank
you very mach.
Hope
history() can provide a parameter setting history
file like "-file" in condor_history command.
NAMEïJiang
Xiaowei
MAILïjiangxw@mail.ihep.ac.cn
TELï010
8823 6024
DEPARTMENTïComputing
Center of IHEP
From:
Iain Steers
Date:
2016-11-15 17:56
To:
HTCondor-Users Mail List
Subject:
Re: [HTCondor-users] python api about history
Ah
I was thinking of the since and stop constraints.
https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=5642,56
Looks
like they haven't been added to the python bindings
yet.
________________________________________
From:
HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx]
on behalf of jiangxw@xxxxxxxxxxxxxxx[jiangxw@xxxxxxxxxxxxxxx]
Sent:
15 November 2016 10:53
To:
htcondor-users
Subject:
Re: [HTCondor-users] python api about history
Hi
Iain,
ÂÂÂThanks for your advice.
ÂÂÂI can catch your meaning.
ÂÂÂI implemented that function like yours.
ÂÂÂAnd the space of history files in our
environments is 132G totally.
ÂÂÂWhen I Âexecute the script with the lines
ÂÂÂÂÂÂÂrequirements = "EnteredCurrentStatus <= %s
&& EnteredCurrentStatus > %s" %
(current_time,pre_time)
ÂÂÂÂÂÂÂjob_history =
self.schedd.history(requirements, projection, 0),
ÂÂÂthe script will take too long times. ÂI think
self.schedd.history() will look over all 132G
history files.
ÂÂÂHow can I optimize it?
ÂÂÂThanks.
Cheers,
Jiang
Xiaowei
________________________________
NAMEïJiang
Xiaowei
MAILïjiangxw@mail.ihep.ac.cn
TELï010
8823 6024
DEPARTMENTïComputing
Center of IHEP
From:
Iain Steers<mailto:iain.steers@xxxxxxx>
Date:
2016-11-15 17:16
To:
HTCondor-Users Mail List<mailto:htcondor-users@xxxxxxxxxxx>
Subject:
Re: [HTCondor-users] python api about history
Hi,
history()
can take a Requirement constraint as one of its
arguments.
In
order to do this efficiently i would suggest
something like the following.
start_dt
= datetime.datetime.now() -
datetime.timedelta(minutes=30)
start_stamp
= calendar.timegm(d.timetuple())
schedd
= htcondor.Schedd()
jobs
= schedd.history('EnteredCurrentStatus >=
{0}'.format(start_stamp))
This
should return an iterator of all the jobs which were
added to the history in the last half hour.
Cheers,
Iain
________________________________________
From:
HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx]
on behalf of jiangxw@xxxxxxxxxxxxxxx[jiangxw@xxxxxxxxxxxxxxx]
Sent:
15 November 2016 10:08
To:
htcondor-users
Subject:
[HTCondor-users] python api about history
Hi
all,
ÂÂÂI used python api to get job history information
then insert job information into database.
ÂÂÂÂÂÂThe function is htcondor.Schedd.history().
ÂÂÂBut when the amount of the history files in
"/var/lib/condor/spool/" became large,
ÂÂÂhtcondor.Schedd.history() will be executed
slowly.
ÂÂÂI want to gain the job history information every
half hour. So I just need to look over the history
file generated last half hour.
ÂÂÂBut the htcondor.Schedd.history() may look over
all history files in "/var/lib/condor/spool" every
time.
ÂÂÂI can not find Âthe parameter from
htcondor.Schedd.history() like "condor_history
-file".
ÂÂÂAre there some ways to solve it?
ÂÂÂThanks for help.
Best
regards,
Jiang
XiaoWei
________________________________
NAMEïJiang
Xiaowei
MAILïjiangxw@mail.ihep.ac.cn
TELï010
8823 6024
DEPARTMENTïComputing
Center of IHEP
_______________________________________________
HTCondor-users
mailing list
To
unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx
with a
subject:
Unsubscribe
You
can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The
archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users
mailing list
To
unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx
with a
subject:
Unsubscribe
You
can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The
archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users
mailing list
To
unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx
with a
subject:
Unsubscribe
You
can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The
archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
--
Tim Theisen
Release Manager
HTCondor & Open Science Grid
Center for High Throughput Computing
Department of Computer Sciences
University of Wisconsin - Madison
4261 Computer Sciences and Statistics
1210 W Dayton St
Madison, WI 53706-1685
+1 608 265 5736
|
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}