Re: [Condor-devel] Surprise with classad.so, new batlab and rpath: Are you linking what I'm linking?

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

On Oct 10, 2011, at 10:21 AM, Tim St Clair wrote:

> If we hold true to shared library version-ing mantra this becomes a non-issue.  We simply need to up the rev for classads. 
> 

What if you want to test a bugfix that doesn't break the ABI?  I think it's very useful for developers to be able to feed their own libs.

Sigh - Greg pointed out to me that my suggestion is dense, and you can't do per-.so RPATH.  This led me to lookup "man ld.so".  To quote:

       The shared libraries needed by the program are searched for in various places:

       o      (ELF only) Using the DT_RPATH dynamic section  attribute  of  the  binary  if  present  and
              DT_RUNPATH attribute does not exist.  Use of DT_RPATH is deprecated.

       o      Using  the  environment  variable LD_LIBRARY_PATH.  Except if the executable is a set-user-
              ID/set-group-ID binary, in which case it is ignored.

       o      (ELF only) Using the DT_RUNPATH dynamic section attribute of the binary if present.

       o      From the cache file /etc/ld.so.cache which contains a compiled list of candidate  libraries
              previously  found in the augmented library path. If, however, the binary was linked with -z
              nodeflib linker option, libraries in the default library paths are skipped.

       o      In the default path /lib, and then /usr/lib.  If the binary was  linked  with  -z  nodeflib
              linker option, this step is skipped.

Note that with RPATH that LD_LIBRARY_PATH does *not* override things.  What if we set DT_RUNPATH to /usr/lib64:$ORIGIN/..libs?  Then,

1) In glideins / testing, one can provide a custom version of the libs using trusty ol' $LD_LIBRARY_PATH.
2) If not overridden, /usr/lib64 is used.
3) If not in /usr/lib64, then $ORIGIN/..libs is used.

We don't lose flexibility for developers, and it is effectively the same for users.

Brian

> Cheers,
> Tim
> 
> ----- Original Message -----
> From: "Dan Bradley" <dan@xxxxxxxxxxxx>
> To: condor-devel@xxxxxxxxxxx
> Sent: Monday, October 10, 2011 10:10:12 AM
> Subject: Re: [Condor-devel] Surprise with classad.so, new batlab and rpath:  Are you linking what I'm linking?
> 
> 
> This would affect glideins as well.  Good catch!
> 
> I suppose LD_LIBRARY_PATH could work around this.  What's the wisdom on 
> having the system lib directories come before $ORIGIN directories?  If 
> the order had been the other way around, this particular problem would 
> go away, right?
> 
> --Dan
> 
> On 10/9/11 9:48 PM, Greg Thain wrote:
>> 
>> In the new batlab, the base Condor is installed as native packages, 
>> where available.  This means that condor binaries are in /usr/bin and 
>> /usr/sbin, and the .so's are in /usr/lib64.  I made some changes to 
>> the classad library, and submitted a build & test run.
>> 
>> I was surprised to see my changes not get executed in the test suite.
>> 
>> This is because the RPATH is set to /usr/lib/64:some_other_stuff, and 
>> the base native condor has a libclassad.so installed in /usr/lib64, 
>> and the condor-under-test, finds that one first.  Ick.
>> 
>> We can fix this by aggressive revving of the .so version number during 
>> development, but this seems like an accident waiting to happen.
>> 
>> Thoughts?
>> 
>> -Greg
>> _______________________________________________
>> Condor-devel mailing list
>> Condor-devel@xxxxxxxxxxx
>> https://lists.cs.wisc.edu/mailman/listinfo/condor-devel
> _______________________________________________
> Condor-devel mailing list
> Condor-devel@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/condor-devel
> _______________________________________________
> Condor-devel mailing list
> Condor-devel@xxxxxxxxxxx
> https://lists.cs.wisc.edu/mailman/listinfo/condor-devel