Re: [HTCondor-users] Python bindings and per job history files

[Iain Bradford Steers <iain.steers@xxxxxxx>]

Hi Luke,

Quite possibly.

In terms of the job ads.

The JobRouter is the equivalent of your example and those values will end up in the *routed* ad. i.e. the classad on the batch schedd.

Those extra values do not get brought back to the CE resource classad.

You could probably inject them back into the CE ads with hooks but Iâve never bothered.

I just have PER_JOB_HISTORY_DIR set up on the batch schedds anyway and use that one, which allows me access to the routed classad attributes. i.e. hepspec, datacentre etc.

Cheers, Iain

On Mar 7, 2016, at 21:41, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:

But naively I would expect an iterator of 1 still be valid.

Anyway, I will see what I can do. Thanks for the hints.

Regarding LastRemoteHost, is there no way to inject it into the job ad? I've seen examples of things like MemoryUsage:

STARTD_JOB_EXPRS = $(START_JOB_EXPRS) MemoryUsage

This should work for HEPSPEC06 as well or do I misunderstand the procedure?

Cheers,

Luke

On 7 March 2016 at 18:24, Iain Bradford Steers <iain.steers@xxxxxxx> wrote:

Also in reference to your original issue.

classad.parseAds expects a stream of classads e.g. (âcondor_history -l -match 4â) whereas youâre only giving it one file.

IIRC the function you need is parseOne(f).

Cheers, Iain

On Mar 7, 2016, at 19:21, Iain Steers <iain.steers@xxxxxxx> wrote:

Hi Luke,

Ah the htcondor-ce history files.

Unless something has changed in this latest release the advise is to parse the batch schedd history files instead.

IIRC thereâs some missing fields that are added in job routing and by the batch schedd that donât yet make it back to the CE schedd.

LastRemoteHost being one of the missing fields makes it difficult to apply per machine scaling factors to the jobs.

Cheers, Iain

On Mar 7, 2016, at 19:12, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:

Dear experts,

I am currently trying to parse our per job history files. The example I got together so far is
import classad;
f = open("/var/lib/gratia/condorce_data/history.10.0");
ads = classad.parseAds(f);
for c in ads:
    print c

where the printout looks similar to that of a classAd (indeed the type is a class ad).

However,
print c["JobStatus"]
fails with
Traceback (most recent call last):
  File "test_history.py", line 7, in <module>
    print c["JobStatus"]
StopIteration

From the examples I've seen, this should be possible.

Fun fact: if I add
print dir(c )

to the for-loop, it works.

Equivalently, if I change the iterator to a list
for c in ads: -> for c in list(ads):

it works as well.

Is this a bug in the way the iterator is constructed or am I missing something?

The history file can be seen here:
https://gist.github.com/kreczko/9056874bbb54a1b4a21b#file-gistfile1-txt

and the python example:
https://gist.github.com/kreczko/9056874bbb54a1b4a21b#file-python

working python:
https://gist.github.com/kreczko/9056874bbb54a1b4a21b#file-working-python

I am running htcondor 8.4.3 and python 2.6.6.

Cheers,

Luke

--

*********************************************************
  Dr Lukasz Kreczko           
  Research Associate

  Department of Physics

  Particle Physics Group

  University of Bristol
  HH Wills Physics Lab
  University of Bristol
  Tyndall Avenue
  Bristol
  BS8 1TL

  +44 (0)117 928 8724 

  L.Kreczko@xxxxxxxxxxxxx

  

  A top 5 UK university with leading employers (2015)

  A top 5 UK university for research (2014 REF)

  A world top 40 university (QS Ranking 2015)

*********************************************************

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--

*********************************************************
  Dr Lukasz Kreczko           
  Research Associate

  Department of Physics

  Particle Physics Group

  University of Bristol
  HH Wills Physics Lab
  University of Bristol
  Tyndall Avenue
  Bristol
  BS8 1TL

  +44 (0)117 928 8724 

  L.Kreczko@xxxxxxxxxxxxx

  

  A top 5 UK university with leading employers (2015)

  A top 5 UK university for research (2014 REF)

  A world top 40 university (QS Ranking 2015)

*********************************************************

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

Attachment: smime.p7s
Description: S/MIME cryptographic signature