Re: [DynInst_API:] Dyninst assert -- Re: Problem with running OpenSpeedShop - mpirun command?

[Bill Williams <bill@xxxxxxxxxxx>]

On 01/25/2016 01:44 PM, Mike Burklund wrote:

Bill,

Please find attached the dyninst debug information.  Thank you for your help!

Mike

The RT lib path needs to be the full filename of libdyninstAPI_RT.so, not just the directory. (This is quite probably something that should go on our "small annoyances to fix" list at some point, but it's been this way since time immemorial.)

--bw

On Mon, Jan 25, 2016 at 2:34 PM, Bill Williams <bill@xxxxxxxxxxx> wrote:

Generally this means that whatever runtime library we may have found was not compatible with the mutator (in this case, O|SS). If you run with DYNINST_DEBUG_STARTUP set to 1 in your environment, that will tell me what candidates we found and why we discarded them...

--bw

On 01/25/2016 01:24 PM, Jim Galarowicz wrote:

Hi Mike,

I think the reason you aren't seeing that data is because Dyninst aborted while we were asking it to find the loops in your application.
You should see a by function list of the performance information as part of the default view generation from the osspcsamp command.
So, we probably have a Dyninst problem (which we don't see very often).  I'm copying the Dyninst team on this one.

Bill, Dyninst team - do you have any ideas?   Have you seen this type of assert in the past?

Thanks,
Jim G

On 01/25/2016 12:28 PM, Mike Burklund wrote:

Jim,

Ok I used the full path to mpiexec.hydra and was able to get further.  It complained about not having the env variable DYNINST_API_RT_LIB set.  I set this and ran again and was able to run.  I got this message (not sure if it is bad or not):

    ossutil: /opt/openss-2.2/openspeedshop-release-2.2/BUILD/hornberg/dyninst-8.2.1/dyninstAPI/src/binaryEdit.C:918: int_variable* BinaryEdit::createTrampGuard(): Assertion `rtlib.size()' failed.

I'm just learning openspeedshop now, but when I look at the results with openss, it say:

    (There are no objects specified for the basic Detail report.)

So, I guess I need to look into how to get sampling/timing results on the function level of my test application.

Thanks again,

Mike

On Mon, Jan 25, 2016 at 11:40 AM, Jim Galarowicz <jeg@xxxxxxxxxxxxx> wrote:

Hi Mike,

You could also try setting OPENSS_MPI_IMPLEMENTATION=mpich2.   This is only really supposed to matter for experiments= mpi, mpit but it also is looked at for finding mpi driver commands.
There might be some special casing for that in the offline.py file.
You could also try the full path to the mpirun or mpirun.hydra commands.

Jim G

On 01/25/2016 10:25 AM, Mike Burklund wrote:

Jim,

Thank you for your timely reply.

Regarding my mpirun command, it is installed with the fedora 23 mpich rpm and is a sym link to mpiexec.hyrdra in the same directory.  Maybe the symbolic link is throwing things off.

I'll take a look at your sample module file to check my installation and will let you know the results.

Thanks,

Mike

On Mon, Jan 25, 2016 at 11:04 AM, Jim Galarowicz <jeg@xxxxxxxxxxxxx> wrote:

Hi Mike,

I changed the Subject line to something more meaningful and copied the oss-questions group, so that others might benefit from what we discover.

This is an example module file for setting up the environment to run OpenSpeedShop.
It looks like the build is correct (looking at the debug output).   Seems like the problem is that we are not recognizing the mpirun command as an mpi driver command.

I noticed from the output the OPENSS_RAWDATA_DIR is not being set to a shared file system directory location.  We need that to write the raw data out while the application is running, then we read from that location at the end of execution to create the database file that contains all the performance data and your application program debug symbol information we use to map the data to.

The other thing I can see from the debug startup output (Thanks for that!) is that we don't seem to be recognizing the mpirun command as the mpi driver.  The output says we are running non-MPI.
So, is your mpirun command a script or an executable?   Is it mpich based?
It must be an executable based on the output.   With no symbols I might have to put a special case in our offline.py file to force it to be known as a mpi driver.
I'm in Minnesota for a week and away from my office.  I have a Fedora 23 system there and could try to reproduce, but that won't be until next week.

So, the environment variables set in the module file are the ones that are needed for successful runs of OpenSpeedShop.

I've attached a simple program to test a sequential test program that will tell us if your installation is correct.

$ cc -o mutatee -g -O0 mutatee.c
$ osspcsamp ./mutatee
$ module load ossoff222
$ osspcsamp ./mutatee
[openss]: pcsamp experiment using the default sampling rate: "100".
[openss]: Setting up offline raw data directory in /opt/shared/offline-oss
[openss]: Running offline pcsamp experiment using the command:
"./mutatee"
work(900)=0

[openss]: Converting raw data from /opt/shared/offline-oss into temp file X.0.openss

Processing raw data for mutatee ...
Processing processes and threads ...
Processing performance data ...
Processing symbols ...
Resolving symbols for /home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee
Updating database with symbols ...
Finished ...

[openss]: Restoring and displaying default view for:
    /home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee-pcsamp.openss
[openss]: The restored experiment identifier is:  -x 1

Exclusive   % of CPU  Function (defining location)
 CPU time       Time 
       in            
 seconds.            
24.150000  50.756620  f3 (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,24)
15.530000  32.639765  f2 (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,15)
 7.890000  16.582598  f1 (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,6)
 0.010000   0.021017  work (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,33)

Thanks,
Jim G

$ cat ossoff222
#%Module1.0#####################################################################
##
## openss modulefile
##
proc ModulesHelp { } {
        global version openss

        puts stderr "\topenss - loads the OpenSpeedShop software & application environment"
        puts stderr "\n\tThis adds $oss/* to several of the"
        puts stderr "\tenvironment variables."
        puts stderr "\n\tVersion $version\n"
}

module-whatis   "Loads the OpenSpeedShop runtime environment."

# for Tcl script use only
set     version         2.2

set     root            /opt/krellroot_v2.2.2
set     oss             /opt/ossoff_222

setenv          OPENSS_DOC_DIR        $oss/share/doc/packages/OpenSpeedShop
setenv          OPENSS_MPI_IMPLEMENTATION       openmpi
setenv        OPENSS_RAWDATA_DIR    /opt/shared

prepend-path    PATH            $root/bin
prepend-path    PATH            $oss/bin
prepend-path    MANPATH            $oss/share/man

    setenv  OPENSS_PLUGIN_PATH   $oss/lib64/openspeedshop
    setenv DYNINSTAPI_RT_LIB $root/lib64/libdyninstAPI_RT.so
    prepend-path LD_LIBRARY_PATH $root/lib64
    prepend-path LD_LIBRARY_PATH $oss/lib64

On 01/25/2016 09:40 AM, Mike Burklund wrote:

Jim,

Thanks you for your help!  I ended up needing to adding the following paths to my LD_LIBRARY_PATH:

   ossoffline_v2.2/lib64
   krellroot_v2.2/lib64

These were built in a non-standard location I guess - /opt

I can now open the openss GUI successfully.

However, when I tested a simple mpi application from the command-line, I get the following messages:

   nm: /usr/lib64/mpich/bin/mpirun: no symbols
  
   env: ./mpirun: No such file or directory

   No performance measurements were made for the experiment.

I have set OPENSS_MPI_IMPLEMENTATION=mpich

Attached is the output after running with OPENSS_DEBUG_STARTUP=1.

Please let me know if you have any ideas on what else I'm missing?  Is there some part of the installation I did not do?

Thanks again,

Mike

On Fri, Jan 22, 2016 at 3:11 PM, Jim Galarowicz <jeg@xxxxxxxxxxxxx> wrote:

Hi Mike,

There are a couple of possibilities that could be causing this.
One:
The library path to the python libraries is not in the LD_LIBRARY_PATH in your module file or OpenSpeedShop runtime set-up script.

Two:
If OpenSpeedShop was compiled with a compiler that is not in the standard /usr install path, the libstdc++.so.6 is not found when starting OpenSpeedShop up.
For this, you can do one of these:

• Load the module file for the non-standard installed compiler or
  
• Add the path to the libstdc++.so.6 library to the LD_LIBRARY_PATH
• Copy that library to the krellroot install library (lib or lib64) sub-directory: cp /opt/gcc/4.8.2/snos/lib64/libstdc++.so.6 <krellroot_installdir>/lib64/.

You can do an "strace openss" to see what files are being loaded, it should show the file being looked for when it fails.

Thanks,
Jim G

On 01/22/2016 12:42 PM, Mike Burklund wrote:

Name:	Mike Burklund
Email:	mdb.inet@xxxxxxxxx
Message:	I have built openspeedshop 2.2 and the krell-root libraries on my Fedora 23 system. I saw no errors in the build process, but when I tried to run osspcsamp on a test program, I got: [openss]: pcsamp experiment using the pcsamp experiment default sampling rate: "100". [openss]: pcsamp experiment calling openss. file not found I tried executing openss directly and just get: file not found Any thoughts on what may be wrong? Thanks, Mike

_______________________________________________
Dyninst-api mailing list
Dyninst-api@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api