Re: [HTCondor-users] virtualenv, pip and htcondor (python)

[L Kreczko <L.Kreczko@xxxxxxxxxxxxx>]

Hi all,

Since my work on an HTCondor task implementation for Spotify's Luigi [1], this topic is again becoming of interest.

In order to do 'pip install htcondor', htcondor would need a setup.py that compiles the HTCondor Python bindings and distributes the libraries htcondor.so and classad.so the same way the condor-python RPM does.

As I understand from the instructions [2] all I need to call in the setup.py is

1) match code to the installed condor version

2) ./configure_uw

3) make package

4) point python to the two libraries it needs to install.

Is this a safe way? Is it possible to compile just the python bindings (I am asking because I cannot build condor on 16.04 [3])?

Do htcondor.so and classad.so need to be compiled in the same version as the condor install or does some wiggle-room exist?

Cheers,

Luke

[1]

https://github.com/spotify/luigi/pull/1955

[2]

https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=BuildingHtcondorOnUnix

[3]

https://gist.github.com/kreczko/91811e46f17d0178ea47b7dac9861a9f

On 10 July 2015 at 15:27, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:

Hi Tom, hi Brian,

Thanks a lot for your answers. Good to see that I am not the only one trying this :).

As for Tom's solution: that looks neat. The only problem is that the auto-completion possibly won't work in the IDE. Have to try it out.

@Brian: Yes, I was thinking of that, but after the first attempt to compile failed, I did not want to spend more time it for now. Maybe later.

I am also experimenting with conda (http://conda.pydata.org/docs/) for environment setup, but this is something further down the line after the first rush of analyses this year.

If I manage to come up with something that will work on, say, travis-ci.org, I will let you know.

Cheers,

Luke

On 10 July 2015 at 13:31, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:

On Jul 10, 2015, at 5:46 AM, L Kreczko <L.Kreczko@xxxxxxxxxxxxx> wrote:

Dear all,

I am currently trying to use the htcondor python bindings in our code, but I could not find a good way to make this possible.

The setup:

The experiment framework sets up a separate python version (2.7) that is newer than the OS default. On top of that our code uses virtualenv (https://github.com/pypa/virtualenv) that allows you to create an isolated python environment.

While the htcondor python bindings are available for the OS python (2.6) they are not for anything above.

So, is it possible to setup htcondor via pip or similar?

A possible solution is to symlink the OS libraries to the virtualenv installation, but there must be a better way than this:

ln -s /usr/lib64/python2.6/site-packages/htcondor.so external/vpython/lib/python2.7/site-packages/htcondor.so
ln -s /usr/lib64/python2.6/site-packages/classad.so external/vpython/lib/python2.7/site-packages/classad.so

Hi Luke,

You can always recompile HTCondor and point it to an alternate python install. Thatâs what I do when I need to use a non-OS python.

However, Iâm stumped on how it could be converted to something I can upload into PyPI. Suggestions?

Brian

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/