Re: [DynInst_API:] spack dyninst/package.py file changes?

[Jim Galarowicz <jeg@xxxxxxxxxxxxx>]

Hi Ben,

No matter if I put the 2018.6 line before or after the 2019 line, 2019 will be built
because it is numerically higher than 2018.6. I believe that is the logic spack uses.

class IntelTbb(Package):
ÂÂÂ """Widely used C++ template library for task parallelism.
ÂÂÂ Intel Threading Building Blocks (Intel TBB) lets you easily write parallel
ÂÂÂ C++ programs that take full advantage of multicore performance, that are
ÂÂÂ portable and composable, and that have future-proof scalability.
ÂÂÂ """
ÂÂÂ homepage = "http://www.threadingbuildingblocks.org/"

ÂÂÂ # See url_for_version() below.
ÂÂ version('2018.6', '9a0f78db4f72356068b00f29f54ee6bc')
ÂÂÂ version('2019',ÂÂ '2119f1db2f905dc5b423482d7689b7d6')
ÂÂÂ version('2018.5', 'ff3ae09f8c23892fbc3008c39f78288f')
ÂÂÂ version('2018.4', '5e2e6ba0e25624a94331c945856551c2')
ÂÂÂ version('2018.3', 'cd2e136598ffa5c136f077ee85a35b4c')
ÂÂÂ version('2018.2', '0b8dfe30917a54e40828eeb0ed7562ae')
ÂÂÂ version('2018.1', 'b2f2fa09adf44a22f4024049907f774b')

==> elfutils is already installed in /home/jeg/OSS_SPACK/spack/opt/spack/linux-fedora27-x86_64/gcc-7.2.1/elfutils-0.173-5jj6vo36rcaqvyqe2yh25q3sobtt4xly
==> Installing intel-tbb
==> Fetching https://github.com/01org/tbb/archive/2019.tar.gz
######################################################################## 100.0%
==> Staging archive: /home/jeg/OSS_SPACK/spack/var/spack/stage/intel-tbb-2019-yibspv36ojjkcjj4tq7iwsyct4lfixj6/2019.tar.gz
==> Created stage in /home/jeg/OSS_SPACK/spack/var/spack/stage/intel-tbb-2019-yibspv36ojjkcjj4tq7iwsyct4lfixj6
==> Applied patch tbb_cmakeConfig.patch
==> Building intel-tbb [Package]
==> Executing phase: 'install'
==> Successfully installed intel-tbb
 Fetch: 1.40s. Build: 14.28s. Total: 15.68s.
[+] /home/jeg/OSS_SPACK/spack/opt/spack/linux-fedora27-x86_64/gcc-7.2.1/intel-tbb-2019-yibspv36ojjkcjj4tq7iwsyct4lfixj6
==> libiberty is already installed in /home/jeg/OSS_SPACK/spack/opt/spack/linux-fedora27-x86_64/gcc-7.2.1/libiberty-2.31.1-7vfoqwkbgkrslrpuwzuyehuna6v5dvp7
==> Installing dyninst
==> Cloning git repository: https://github.com/dyninst/dyninst.git on branch master
==> No checksum needed when fetching with git
==> Already staged dyninst-develop-23i3tj4axmub7plsfyy2vid5tzjfd5gn in /home/jeg/OSS_SPACK/spack/var/spack/stage/dyninst-develop-23i3tj4axmub7plsfyy2vid5tzjfd5gn
==> No patches needed for dyninst
==> Building dyninst [Package]
==> Executing phase: 'install'
==> Successfully installed dyninst
 Fetch: 3.55s. Build: 7m 30.67s. Total: 7m 34.22s.
[+] /home/jeg/OSS_SPACK/spack/opt/spack/linux-fedora27-x86_64/gcc-7.2.1/dyninst-develop-23i3tj4axmub7plsfyy2vid5tzjfd5gn

So, if that is ok with the dyninst team, then the spack modifications/suggestions work to build dyninst with tbb.

Discuss it with your team and Mark K. and let me know how you want to proceed.
I will do some execution testing to see if it works with O|SS now.

Thanks,
Jim G

On 11/04/2018 03:46 PM, Benjamin Welton wrote:

Mark,

Packages.yaml is site/user specific (at least as per my understanding).Â

> Spack will take the latest version from intel-tbb by default.ÂÂ

I do not believe this is true. Spack will only do this if no version of TBB was installed by spack already. If a site has the oldest version of TBB supported by spack already installed, that will be used with Dyninst (likely something we dont want).Â

Jim, if you want to give it a try my suggested changes are the following (if these work, then it is likely good to submit to spack as a pull request):

TBB Package.py:

version('2018.6', ' 9a0f78db4f72356068b00f29f54ee6bc')ÂÂ

Dyninst Package.py:

depends_on("tbb@2018.6:")

(then your other changes).

As for going forward, I am going to suggest on Monday that we (dyninst) keeps a running github fork of spack where we make changes to the packages for dyninst. That way we have a single location that we can grant access to and allow for commits/testing before submitting a pull request to the main spack repo.

Ben

On Sun, Nov 4, 2018 at 1:24 PM Mark W. Krentel <krentel@xxxxxxxx> wrote:

I know that dyninst will download and build TBB, elfutils, boost if
they're not specified. That's fine that you go the extra mile, but
that's not the spack way of doing things.

Spack wants to build all the prereqs itself, through the intel-tbb
package.py file. (So, it's important to allow -D vars for all the
external prereqs.)

As for version, I suggest you leave this unspecified in the dyninst
recipe. Just use depends_on('tbb'). This is what the packages.yaml
file is for.

Spack will take the latest version from intel-tbb by default. So
unless there's some version that's completely incompatible, I suggest
don't add unnecessary constraints. Let someone specify this in
packages.yaml.

Of course, if you want to update the intel-tbb recipe, that's fine,
but that's a separate question.

Btw, I was planning on making a pass over the dyninst recipe, but I'm
waiting for 10.x to be tagged and released.

--Mark


On 11/04/18 08:33, Xiaozhu Meng wrote:
> Hi Ben,
>
> Dyninst will automatically download the following version of TBB:
>
> URL https://github.com/01org/tbb/archive/2018_U6.tar.gz
> URL_MD5 9a0f78db4f72356068b00f29f54ee6bc
>
> It looks like this version is not in your TBB version list. Is the
> above not supported by spack?
>
> Thanks,
>
> --Xiaozhu
>

_______________________________________________
Dyninst-api mailing list
Dyninst-api@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

# Copyright 2013-2018 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)

from spack import *
import glob
import inspect
import platform


class IntelTbb(Package):
    """Widely used C++ template library for task parallelism.
    Intel Threading Building Blocks (Intel TBB) lets you easily write parallel
    C++ programs that take full advantage of multicore performance, that are
    portable and composable, and that have future-proof scalability.
    """
    homepage = "http://www.threadingbuildingblocks.org/";

    # See url_for_version() below.
    version('2018.6', '9a0f78db4f72356068b00f29f54ee6bc')
    version('2019',   '2119f1db2f905dc5b423482d7689b7d6')
    version('2018.5', 'ff3ae09f8c23892fbc3008c39f78288f')
    version('2018.4', '5e2e6ba0e25624a94331c945856551c2')
    version('2018.3', 'cd2e136598ffa5c136f077ee85a35b4c')
    version('2018.2', '0b8dfe30917a54e40828eeb0ed7562ae')
    version('2018.1', 'b2f2fa09adf44a22f4024049907f774b')
    version('2018',   '7fb30d5ea2545f26ce02757d9ab05e6c')
    version('2017.8', '7240f57f1aeea0e266a5e17ae68fdc16')
    version('2017.7', '364f2a4b80e978f38a69cbf7c466b898')
    version('2017.6', 'ec21254af4fc2a29574c272f501a3138')
    version('2017.5', '85b41c64102c052e24d8a39f6193e599')
    version('2017.4', '71526b2fef098515e212302d1455de7d')
    version('2017.3', 'd7622eeaafeff8d271c7aa684bd82ddb')
    version('2017.2', '9605cbea96998a10a186fc72c35cbd76')
    version('2017.1', '6c0fe8aa7bc911a85e8e522e620511b3')
    version('2017',   '9e7f9ea684ecf84ac74dcd3c6012cfa6')
    version('4.4.6',  '20e15206f70c2651bfc964e451a443a0')
    version('4.4.5',  '531a67cd98f9b4ec8ece95c5f8193a83')
    version('4.4.4',  '61531b2e8684e06a621dcdca1a7a420e')
    version('4.4.3',  '8e3e39e1fdfb3f7c3a5ac8ec1afe186e')
    version('4.4.2',  'e92b110e8eb238741b00e3789b39969e')
    version('4.4.1',  'a02c9958f02c1b5f3626874219979ae8')
    version('4.4',    '1d512085221996eae6cec04e1a4cd3dd')

    provides('tbb')

    conflicts('%gcc@xxx:', when='@:4.4.3',
              msg='4.4.4 or later required for GCC >= 6.1.')

    variant('shared', default=True,
            description='Builds a shared version of TBB libraries')

    variant('cxxstd',
            default='default',
            values=('default', '98', '11', '14', '17'),
            multi=False,
            description='Use the specified C++ standard when building.')

    variant('tm', default=True,
            description='Enable use of transactional memory on x86')

    # Build and install CMake config files if we're new enough.
    depends_on('cmake@xxxxx:', type='build', when='@2017.0:')

    # Deactivate use of RTM with GCC when on an OS with an elderly assembler.
    patch("tbb_gcc_rtm_key.patch", level=0, when='%gcc@xxxxx: os=rhel6')
    patch("tbb_gcc_rtm_key.patch", level=0, when='%gcc@xxxxx: os=scientific6')
    patch("tbb_gcc_rtm_key.patch", level=0, when='%gcc@xxxxx: os=centos6')

    # Patch cmakeConfig.cmake.in to find the libraries where we install them.
    patch("tbb_cmakeConfig.patch", level=0, when='@2017.0:')

    # Some very old systems don't support transactional memory.
    patch("disable-tm.patch", when='~tm')

    def url_for_version(self, version):
        url = 'https://github.com/01org/tbb/archive/{0}.tar.gz'
        if (version[0] >= 2017) and len(version) > 1:
            return url.format('{0}_U{1}'.format(version[0], version[1]))
        else:
            return url.format(version)

    def coerce_to_spack(self, tbb_build_subdir):
        for compiler in ["icc", "gcc", "clang"]:
            fs = glob.glob(join_path(tbb_build_subdir,
                                     "*.%s.inc" % compiler))
            for f in fs:
                lines = open(f).readlines()
                of = open(f, "w")
                for l in lines:
                    if l.strip().startswith("CPLUS ="):
                        of.write("# coerced to spack\n")
                        of.write("CPLUS = $(CXX)\n")
                    elif l.strip().startswith("CPLUS ="):
                        of.write("# coerced to spack\n")
                        of.write("CONLY = $(CC)\n")
                    else:
                        of.write(l)

    def install(self, spec, prefix):
        # We need to follow TBB's compiler selection logic to get the proper
        # build + link flags but we still need to use spack's compiler wrappers
        # to accomplish this, we do two things:
        #
        # * Look at the spack spec to determine which compiler we should pass
        #   to tbb's Makefile;
        #
        # * patch tbb's build system to use the compiler wrappers (CC, CXX) for
        #   icc, gcc, clang (see coerce_to_spack());
        #
        self.coerce_to_spack("build")

        if spec.satisfies('%clang'):
            tbb_compiler = "clang"
        elif spec.satisfies('%intel'):
            tbb_compiler = "icc"
        else:
            tbb_compiler = "gcc"

        mkdirp(prefix)
        mkdirp(prefix.lib)

        make_opts = []

        # Static builds of TBB are enabled by including 'big_iron.inc' file
        # See caveats in 'big_iron.inc' for limits on using TBB statically
        # Lore states this file must be handed to make before other options
        if '+shared' not in self.spec:
            make_opts.append("extra_inc=big_iron.inc")

        if spec.variants['cxxstd'].value != 'default':
            make_opts.append('stdver=c++{0}'.
                             format(spec.variants['cxxstd'].value))

        #
        # tbb does not have a configure script or make install target
        # we simply call make, and try to put the pieces together
        #
        make_opts.append("compiler={0}".format(tbb_compiler))
        make(*make_opts)

        # install headers to {prefix}/include
        install_tree('include', prefix.include)

        # install libs to {prefix}/lib
        tbb_lib_names = ["libtbb",
                         "libtbbmalloc",
                         "libtbbmalloc_proxy"]

        for lib_name in tbb_lib_names:
            # install release libs
            fs = glob.glob(join_path("build", "*release", lib_name + ".*"))
            for f in fs:
                install(f, prefix.lib)
            # install debug libs if they exist
            fs = glob.glob(join_path("build", "*debug", lib_name + "_debug.*"))
            for f in fs:
                install(f, prefix.lib)

        if self.spec.satisfies('@2017.0:'):
            # Generate and install the CMake Config file.
            cmake_args = ('-DTBB_ROOT={0}'.format(prefix),
                          '-DTBB_OS={0}'.format(platform.system()),
                          '-P',
                          'tbb_config_generator.cmake')
            with working_dir(join_path(self.stage.source_path, 'cmake')):
                inspect.getmodule(self).cmake(*cmake_args)

# Copyright 2013-2018 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)

from spack import *


class Dyninst(Package):
    """API for dynamic binary instrumentation.  Modify programs while they
    are executing without recompiling, re-linking, or re-executing."""

    homepage = "https://paradyn.org";
    git      = "https://github.com/dyninst/dyninst.git";

    version('develop', branch='master')
    version('9.3.2', tag='v9.3.2')
    version('9.3.0', tag='v9.3.0')
    version('9.2.0', tag='v9.2.0')
    version('9.1.0', tag='v9.1.0')
    version('8.2.1', tag='v8.2.1')
    version('8.1.2', tag='v8.1.2')
    version('8.1.1', tag='v8.1.1')

    variant('stat_dysect', default=False,
            description="patch for STAT's DySectAPI")

    # Dyninst depends on libelf and libdwarf prior to @9.3.0
    # Dyninst depends on elfutils and libdwarf from @9.3.0 to but
    # not including @develop
    # Dyninst depends on elfutils and elfutils libdw from @develop forward
    # elf@0 is an abstaction for libelf
    # elf@1 is an abstaction for elfutils
    depends_on("elf@0", type='link', when='@:9.2.99')
    # The sorting algorithm puts numbered releases as newer than alphabetic
    # releases, but spack has special logic in place to ensure that
    # develop is considered newer than all other releases.
    # So, develop is included in the elf@1 line below.
    depends_on("elf@1", type='link', when='@9.3.0:')
    depends_on("libdwarf", when='@:9')
    depends_on("boost@xxxx:")
    depends_on('libiberty+pic')
    depends_on("tbb@xxxxxx:")
    depends_on('cmake', type='build')

    patch('stat_dysect.patch', when='+stat_dysect')
    patch('stackanalysis_h.patch', when='@9.2.0')

    # new version uses cmake
    def install(self, spec, prefix):
        if spec.satisfies('@:8.1'):
            configure("--prefix=" + prefix)
            make()
            make("install")
            return

        libelf = spec['elf'].prefix
        if spec.satisfies('@:9'):
            libdwarf = spec['libdwarf'].prefix

        with working_dir('spack-build', create=True):
            args = ['..',
                    '-DBoost_INCLUDE_DIR=%s'    % spec['boost'].prefix.include,
                    '-DBoost_LIBRARY_DIR=%s'    % spec['boost'].prefix.lib,
                    '-DBoost_NO_SYSTEM_PATHS=TRUE',
                    '-DLIBELF_INCLUDE_DIR=%s'   % join_path(
                        libelf.include, 'libelf'),
                    '-DLIBELF_LIBRARIES=%s'     % join_path(
                        libelf.lib, "libelf." + dso_suffix)]
            if spec.satisfies('@:9'):
                args.append('-DLIBDWARF_INCLUDE_DIR=%s' % libdwarf.include)
                args.append('-DLIBDWARF_LIBRARIES=%s'   % join_path(
                    libdwarf.lib, "libdwarf." + dso_suffix))
            # For @develop + use elfutils libdw, libelf is an abstraction
            # we are really using elfutils here
            if spec.satisfies('@develop'):
                args.append('-DTBB_INCLUDE_DIRS=%s' % spec['tbb'].prefix.include)
                args.append('-DTBB_LIBRARIES=%s'   % join_path(
                    spec['tbb'].prefix.lib, "libtbb." + dso_suffix))
                args.append('-DLIBDWARF_INCLUDE_DIR=%s' % libelf.include)
                args.append('-DLIBDWARF_LIBRARIES=%s'   % join_path(
                    libelf.lib, "libdw." + dso_suffix))
            if spec.satisfies('arch=linux-redhat7-ppc64le'):
                args.append('-Darch_ppc64_little_endian=1')
            args += std_cmake_args
            cmake(*args)
            make()
            make("install")

    @when('@:8.1')
    def install(self, spec, prefix):
        configure("--prefix=" + prefix)
        make()
        make("install")