Re: [Condor-devel] Surprise with classad.so, new batlab and rpath: Are you linking what I'm linking?

[Jaime Frey <jfrey@xxxxxxxxxxx>]

On Oct 10, 2011, at 11:44 AM, Dan Bradley wrote:

> On 10/10/11 10:36 AM, Brian Bockelman wrote:
>> On Oct 10, 2011, at 10:21 AM, Tim St Clair wrote:
>> 
>>> If we hold true to shared library version-ing mantra this becomes a non-issue.  We simply need to up the rev for classads.
>>> 
>> What if you want to test a bugfix that doesn't break the ABI?  I think it's very useful for developers to be able to feed their own libs.
>> 
>> Sigh - Greg pointed out to me that my suggestion is dense, and you can't do per-.so RPATH.  This led me to lookup "man ld.so".  To quote:
>> 
>>        The shared libraries needed by the program are searched for in various places:
>> 
>>        o      (ELF only) Using the DT_RPATH dynamic section  attribute  of  the  binary  if  present  and
>>               DT_RUNPATH attribute does not exist.  Use of DT_RPATH is deprecated.
>> 
>>        o      Using  the  environment  variable LD_LIBRARY_PATH.  Except if the executable is a set-user-
>>               ID/set-group-ID binary, in which case it is ignored.
>> 
>>        o      (ELF only) Using the DT_RUNPATH dynamic section attribute of the binary if present.
>> 
>>        o      From the cache file /etc/ld.so.cache which contains a compiled list of candidate  libraries
>>               previously  found in the augmented library path. If, however, the binary was linked with -z
>>               nodeflib linker option, libraries in the default library paths are skipped.
>> 
>>        o      In the default path /lib, and then /usr/lib.  If the binary was  linked  with  -z  nodeflib
>>               linker option, this step is skipped.
>> 
>> Note that with RPATH that LD_LIBRARY_PATH does *not* override things.
> 
> Good point.  I misunderstood this.
> 
>>   What if we set DT_RUNPATH to /usr/lib64:$ORIGIN/..libs?  Then,
>> 
>> 1) In glideins / testing, one can provide a custom version of the libs using trusty ol' $LD_LIBRARY_PATH.
>> 2) If not overridden, /usr/lib64 is used.
>> 3) If not in /usr/lib64, then $ORIGIN/..libs is used.
>> 
>> We don't lose flexibility for developers, and it is effectively the same for users.
> 
> The flexibility seems necessary to me.  The cost of flexibility is potential mixups with LD_LIBRARY_PATH.
> 
> In glideins, it seems clear that for libraries closely tied to Condor, such as libclassads, the desired option is to use the lib that comes with Condor, not the one that happens to be lying around on the system.  I don't know about other libs such as openssl.  I believe these two classes of libraries are segregated in case one wants to treat them differently (the external libs are in $ORIGIN/../lib/condor and the condor libs are in $ORIGIN/../lib).
> 
> This suggests to me that at least $ORIGIN/../lib could be moved to the front of the search path.
> 
> --Dan

This sounds like the best compromise at present. Use RUNPATH instead of RPATH and move $ORIGIN/../lib to the front of RUNPATH.
Then LD_LIBRARY_PATH can be set as the final arbiter of where to look first for the libraries, and the correct version of classads will be found in the common case when LD_LIBRARY_PATH isn't set.

I don't like revving the classads so version aggressively. It would make it needlessly difficult for other programs to link against it, at which point we may as well keep it a private library.

 -- Jaime

>>> Cheers,
>>> Tim
>>> 
>>> ----- Original Message -----
>>> From: "Dan Bradley"<dan@xxxxxxxxxxxx>
>>> To: condor-devel@xxxxxxxxxxx
>>> Sent: Monday, October 10, 2011 10:10:12 AM
>>> Subject: Re: [Condor-devel] Surprise with classad.so, new batlab and rpath:  Are you linking what I'm linking?
>>> 
>>> 
>>> This would affect glideins as well.  Good catch!
>>> 
>>> I suppose LD_LIBRARY_PATH could work around this.  What's the wisdom on
>>> having the system lib directories come before $ORIGIN directories?  If
>>> the order had been the other way around, this particular problem would
>>> go away, right?
>>> 
>>> --Dan
>>> 
>>> On 10/9/11 9:48 PM, Greg Thain wrote:
>>>> In the new batlab, the base Condor is installed as native packages,
>>>> where available.  This means that condor binaries are in /usr/bin and
>>>> /usr/sbin, and the .so's are in /usr/lib64.  I made some changes to
>>>> the classad library, and submitted a build&  test run.
>>>> 
>>>> I was surprised to see my changes not get executed in the test suite.
>>>> 
>>>> This is because the RPATH is set to /usr/lib/64:some_other_stuff, and
>>>> the base native condor has a libclassad.so installed in /usr/lib64,
>>>> and the condor-under-test, finds that one first.  Ick.
>>>> 
>>>> We can fix this by aggressive revving of the .so version number during
>>>> development, but this seems like an accident waiting to happen.
>>>> 
>>>> Thoughts?
>>>> 
>>>> -Greg

+--------------------------------+-----------------------------------+
|           Jaime Frey           | I used to be a heavy gambler.     |
|       jfrey@xxxxxxxxxxx        | But now I just make mental bets.  |
|                                | That's how I lost my mind.        |
+--------------------------------+-----------------------------------+