Re: [HTCondor-users] virtualenv, pip and htcondor (python)

[L Kreczko <L.Kreczko@xxxxxxxxxxxxx>]

Hi Brian & all,

Thanks for the help so far, I think I am almost there.

I've written the setup.py route off as a failure, as I cannot build older HTCondor versions (even after applying the patches).

Instead, I decided to try out `condaÂbuild` which turned out more successful so far.

I can now install htcondor python bindings on Ubuntu and SL6 under python 2.7 following the instructions in [1] (please give it a try).

However, a few things are missing/out of place:

1. probably worth adding more tests if the bindings are working (currently just testing import of htcondor and classad)
   
2. Although boost 1.63 is present & found by cmake, condor still downloads 1.49
   
3. lots of binaries are compiled as well (and installed) - ideally none of this is needed. In addition to preventing themÂfrom overwriting the system ones (condaÂenv will be earlier in PATH), I would like to reduce the current size of 51 MB to around 20 MB if possible.
4. condor is installing the python libraries into $PREFIX/lib/python instead of $PREFIX/lib/python2.7 (or $PREFIX/lib/python3.5)
   

The build file can be found in [2], the full build log in [3] and the currently available builds in [4].

This was an interesting side-project, now it is time to adjust the Luigi PR :).

Cheers,

Luke

[1]

https://github.com/kreczko/hep-conda-recipes/blob/master/README.md

[2]

https://github.com/kreczko/hep-conda-recipes/blob/master/htcondor-python/build.sh

[3]

https://gist.github.com/kreczko/59302165216dcded7d9a6ca6675fc3eb

[4]

https://anaconda.org/kreczko/htcondor-python/files

On 3 January 2017 at 17:11, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:

> On Jan 3, 2017, at 10:29 AM, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:
>
> Hi Jaime,
>
> Thanks a lot.
>
> Finally got around to work a bit more on the setup.py.
>
> I've created a few docker images to be able to test the procedure in a more reliable manner:
> git clone https://github.com/kreczko/htcondor-1.git
> git checkout worlds-weirdest-setup.py
> cd docker
> # build containers
> docker-compose build htcondor-ubuntu
> docker-compose build htcondor-centos6
> docker-compose build htcondor-centos7
>
> # you can also try to directly install the modules:
> python setup.py install
>
> However, on my local machine the configuration fails for python, since cmake discovers two versions of python (2.7.11 & 3.5.2) and attempts to find headers for the latter only (for some reason I can find only 3.5.1 headers.
> Is there a way to point the configure script/cmake to the python version currently in use rather than the latest available?
> This is important for pip install with virtualenv or conda.

I think you can force a particular version by setting the following CMake variable:

PYTHONLIBS_VERSION_STRING

I.e.,

cmake -DPYTHONLIBS_VERSION_STRING="2.7.11"

>
> Am I considering all libraries (or too many)? Currently, I am copying
>Â - ./src/python-bindings/classad.so
>Â - ./src/python-bindings/htcondor.so
> to the python site-packages directory
> and
>Â - ./src/python-bindings/libpyclassad2.7_X_Y_X.so
>Â - ./src/condor_utils/libcondor_utils_X_Y_Z.so
> to /usr/lib (ubuntu) or /usr/lib64 (centos)
>

Those sound correct for the direct dependencies - did you use 'ldd' to suss out all the system dependencies?

> This one is also needed, but should already be present after a condor install:
>Â ./bld_external/classads-X.Y.Z/install/lib/libclassad.so
>

Well, it should actually look for something like "libclassad.so.7". The major version ("7") is relatively important because that should catch ABI breaks.

Brian

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--

*********************************************************
 Dr Lukasz Kreczko     ÂÂ
 Research Associate

 Department of Physics

 Particle Physics Group

 University of Bristol
 HH Wills Physics Lab
 University of Bristol
 Tyndall Avenue
 Bristol
 BS8 1TL

 +44 (0)117 928 8724Â

 L.Kreczko@xxxxxxxxxxxxx

ÂÂ

 A top 5 UK university with leading employers (2015)

 A top 5 UK university for research (2014 REF)

 A world top 40 university (QS Ranking 2015)

*********************************************************