Re: [HTCondor-users] virtualenv, pip and htcondor (python)

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

> On Feb 2, 2017, at 5:54 AM, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:
> 
> Hi Brian & all,
> 
> Thanks for the help so far, I think I am almost there.
> I've written the setup.py route off as a failure, as I cannot build older HTCondor versions (even after applying the patches).
> Instead, I decided to try out `conda build` which turned out more successful so far.
> 
> I can now install htcondor python bindings on Ubuntu and SL6 under python 2.7 following the instructions in [1] (please give it a try).
> However, a few things are missing/out of place:
> 
> 	â probably worth adding more tests if the bindings are working (currently just testing import of htcondor and classad)

For this -

cd src/python-bindings/
./test_driver tests/classad_tests.py
./test_driver tests/htcondor_version_tests.py
./test_driver tests/htcondor_tests.py

The last one is quite complex; probably best to do it in a Docker container.

> 	â Although boost 1.63 is present & found by cmake, condor still downloads 1.49

Hm - are you doing "PROPER" build?  That bypasses all the external builds.

Otherwise, I think it'll always download condor's preferred version.

> 	â lots of binaries are compiled as well (and installed) - ideally none of this is needed. In addition to preventing them from overwriting the system ones (conda env will be earlier in PATH), I would like to reduce the current size of 51 MB to around 20 MB if possible.
> 	â condor is installing the python libraries into $PREFIX/lib/python instead of $PREFIX/lib/python2.7 (or $PREFIX/lib/python3.5)
> 

Yeah - I think this might require quite a "rm" and "mv" invocations.  This is done by the RPM's spec file.

> The build file can be found in [2], the full build log in [3] and the currently available builds in [4].
> 
> This was an interesting side-project, now it is time to adjust the Luigi PR :).
> 

Sigh.  Now I need to go learn more about conda build!

> Cheers,
> Luke
> 
> [1]
> https://github.com/kreczko/hep-conda-recipes/blob/master/README.md
> 
> [2]
> https://github.com/kreczko/hep-conda-recipes/blob/master/htcondor-python/build.sh
> 
> [3]
> https://gist.github.com/kreczko/59302165216dcded7d9a6ca6675fc3eb
> 
> [4]
> https://anaconda.org/kreczko/htcondor-python/files
> 
> On 3 January 2017 at 17:11, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:
> 
> > On Jan 3, 2017, at 10:29 AM, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:
> >
> > Hi Jaime,
> >
> > Thanks a lot.
> >
> > Finally got around to work a bit more on the setup.py.
> >
> > I've created a few docker images to be able to test the procedure in a more reliable manner:
> > git clone https://github.com/kreczko/htcondor-1.git
> > git checkout worlds-weirdest-setup.py
> > cd docker
> > # build containers
> > docker-compose build htcondor-ubuntu
> > docker-compose build htcondor-centos6
> > docker-compose build htcondor-centos7
> >
> > # you can also try to directly install the modules:
> > python setup.py install
> >
> > However, on my local machine the configuration fails for python, since cmake discovers two versions of python (2.7.11 & 3.5.2) and attempts to find headers for the latter only (for some reason I can find only 3.5.1 headers.
> > Is there a way to point the configure script/cmake to the python version currently in use rather than the latest available?
> > This is important for pip install with virtualenv or conda.
> 
> I think you can force a particular version by setting the following CMake variable:
> 
> PYTHONLIBS_VERSION_STRING
> 
> I.e.,
> 
> cmake -DPYTHONLIBS_VERSION_STRING="2.7.11"
> 
> >
> > Am I considering all libraries (or too many)? Currently, I am copying
> >  - ./src/python-bindings/classad.so
> >  - ./src/python-bindings/htcondor.so
> > to the python site-packages directory
> > and
> >  - ./src/python-bindings/libpyclassad2.7_X_Y_X.so
> >  - ./src/condor_utils/libcondor_utils_X_Y_Z.so
> > to /usr/lib (ubuntu) or /usr/lib64 (centos)
> >
> 
> Those sound correct for the direct dependencies - did you use 'ldd' to suss out all the system dependencies?
> 
> > This one is also needed, but should already be present after a condor install:
> >  ./bld_external/classads-X.Y.Z/install/lib/libclassad.so
> >
> 
> Well, it should actually look for something like "libclassad.so.7".  The major version ("7") is relatively important because that should catch ABI breaks.
> 
> Brian
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> 
> 
> -- 
> *********************************************************
>   Dr Lukasz Kreczko            
>   Research Associate
>   Department of Physics
>   Particle Physics Group
> 
>   University of Bristol
>   HH Wills Physics Lab
>   University of Bristol
>   Tyndall Avenue
>   Bristol
>   BS8 1TL
> 
>   +44 (0)117 928 8724  
>   L.Kreczko@xxxxxxxxxxxxx
>   
>   A top 5 UK university with leading employers (2015)
>   A top 5 UK university for research (2014 REF)
>   A world top 40 university (QS Ranking 2015)
> *********************************************************
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/