Re: [DynInst_API:] spack dyninst/package.py file changes?

[Benjamin Welton <welton@xxxxxxxxxxx>]

Jim,

We are likely okay using either 2019 or 2018.6. So while Dyninst may pull 2019 (which is apparently a week older than 2018.6), it should be fine. The real reason I want a limit in the depends_on spec is to make sure dyninst isn't built with a crazy old version of TBB (and a week older isn't a big deal).

Go ahead and submit those changes to Spack (assuming everything builds on your end OK). We will revisit these at a later date if need be.

Mark,

> It's actually a pain to get spack to use a previously installed package
because you have to write a dependency spec that *exactly* matches the
installed version

This is the problem with TBB in specific. While most of Dyninst's dependencies have very specific recipies that must be built, TBB (at least as per my understanding) does not. In this case, there is a much higher chance (which could still be low) of a match to an existing version. There is a zero chance of a match with an incompatible version if we specify that we require at least a certain version of the dependency. 

The whole reason I am pushing for this is purely that it eliminates any chance of a crazy bug being reported by a user that turns out to be caused by linking to some ancient version of TBB with spack.

As for going forward, I am going to open a dialog with the spack folks about how to best handle package updates (and best practices for doing so).

Ben

On Sun, Nov 4, 2018 at 5:54 PM Mark W. Krentel <krentel@xxxxxxxx> wrote:

On 11/04/18 15:46, Benjamin Welton wrote:
> Mark,
>
> Packages.yaml is site/user specific (at least as per my understanding).
>
> > Spack will take the latest version from intel-tbb by default.
>
> I do not believe this is true. Spack will only do this if no version
> of TBB was installed by spack already. If a site has the oldest
> version of TBB supported by spack already installed, that will be used
> with Dyninst (likely something we dont want).
>

No, spack spec resolution does not depend on what's currently
installed.  That would actually be hard to add (and I claim NP-hard in
general).

What actually happens is that it resolves the partial spec into a
completely specified spec and if that *exact* hash has already been
installed, then it uses it, otherwise it builds the new hash.

Here's a simple example.  I installed an old version of libiberty+pic
2.28.1.

  $ ./spack find -v -l
==> 1 installed packages.
-- linux-fedora26-x86_64 / gcc@xxxxx ----------------------------
dz45txs libiberty@xxxxxx+pic

Then, I use 'spack spec' to see how it would build dynisnt (omitting
many transitive dependencies).

$ ./spack spec dyninst
dyninst@xxxxx%gcc@7.3.1~stat_dysect arch=linux-fedora26-x86_64
     ^boost@xxxxxx%gcc@7.3.1+atomic+chrono~clanglibcpp cxxstd=default
+date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy
patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199
+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave
arch=linux-fedora26-x86_64
     ^cmake@xxxxxx%gcc@7.3.1~doc+ncurses+openssl+ownlibs
patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0
~qt arch=linux-fedora26-x86_64
     ^elfutils@xxxxx%gcc@7.3.1~bzip2+nls~xz arch=linux-fedora26-x86_64
     ^libdwarf@20180129%gcc@7.3.1 arch=linux-fedora26-x86_64
     ^libiberty@xxxxxx%gcc@7.3.1+pic arch=linux-fedora26-x86_64

So, it would build dyninst with 2.31.1, even though 2.28.1 is already
installed.

It's actually a pain to get spack to use a previously installed package
because you have to write a dependency spec that *exactly* matches the
installed version.  If this is what you want, the easiest way is to
use '/hash' for the dependency.

$ ./spack spec dyninst ^libiberty /dz45txs
dyninst@xxxxx%gcc@7.3.1~stat_dysect arch=linux-fedora26-x86_64
     ^libiberty@xxxxxx%gcc@7.3.1+pic arch=linux-fedora26-x86_64

Here, '/dz45txs' refers to the installed version.  Note that
/dz45txs must have +pic or else it won't match dyninst's needs.

Also, I should have clarified what I said about the preferred version.
If a version specifies 'preferred=True', then that's preferred (duh).
Otherwise, it's the latest rev according to some ordering.

Jim G, that's why the order you put them in the file doesn't matter.
2019 > 2018 no matter what order you write them.

Btw, you can call this a deficiency is spack's features.  What you'd
like to say is that you *prefer* a version >= 2018.06 without
requiring that.  But AFAIK, spack doesn't have a way to say this.

That's the problem with depends_on().  It's a requirement, not a
preference.  You're saying you refuse to build with anything earlier.
And I think that's unnecessary.

The spack way of handling this is with packages.yaml.

Btw, it makes sense for you to keep a packages.yaml file in your repo.
This would specify the reference point that you want to build with.
That's what I plan to do with hpctoolkit.

Also AFAIK, spack doesn't have a good way to incorporate your
packages.yaml preferences for a default dyninst build.  You'd like to
write packages.yaml in every directory as default preferences, but I
don't think that's allowed.  I regard this is a significant deficiency
in spack.

--Mark