Re: [DynInst_API:] Dyninst assert -- Re: Problem with running OpenSpeedShop - mpirun command?

[Mike Burklund <mdb.inet@xxxxxxxxx>]

Sorry, just realized I was not pointing to the dyninstapi_rt library directly, just to its parent dir. I changed the DYNINSTAPI_RT_LIB to point to the library directly. Looks like I got my timings now:

[openss]: Restoring and displaying default view for:
ÂÂÂ /data/FVV/DESIGN/openss_test1/computePi-pcsamp-5.openss
[openss]: The restored experiment identifier is:Â -x 1

ExclusiveÂÂ % of CPUÂ Function (defining location)
ÂCPU timeÂÂÂÂÂÂ TimeÂ
ÂÂÂÂÂÂ inÂÂÂÂÂÂÂÂÂÂÂÂ
Âseconds.ÂÂÂÂÂÂÂÂÂÂÂÂ
Â0.600000Â 42.253521Â MPIDI_CH3I_Progress (libmpi.so.12.0.5)
Â0.240000Â 16.901408Â MPID_nem_tcp_connpoll (libmpi.so.12.0.5)
Â0.150000Â 10.563380Â runWorker(int&, int, int) (computePi)
Â0.110000ÂÂ 7.746479Â __random_r (libc-2.22.so)
Â0.090000ÂÂ 6.338028Â MPIDU_Sched_are_pending (libmpi.so.12.0.5)
Â0.080000ÂÂ 5.633803Â MPID_nem_network_poll (libmpi.so.12.0.5)
Â0.070000ÂÂ 4.929577Â __random (libc-2.22.so)
Â0.030000ÂÂ 2.112676Â computeRandom() (computePi)
Â0.020000ÂÂ 1.408451Â runServer() (computePi)
Â0.010000ÂÂ 0.704225Â __memcpy_sse2_unaligned (libc-2.22.so)
Â0.010000ÂÂ 0.704225Â MPIR_Bcast_intra (libmpi.so.12.0.5)
Â0.010000ÂÂ 0.704225Â MPID_Recv (libmpi.so.12.0.5)

Thank you again for your help.

Mike

On Mon, Jan 25, 2016 at 2:44 PM, Mike Burklund <mdb.inet@xxxxxxxxx> wrote:

Bill,

Please find attached the dyninst debug information. Thank you for your help!

Mike

On Mon, Jan 25, 2016 at 2:34 PM, Bill Williams <bill@xxxxxxxxxxx> wrote:

Generally this means that whatever runtime library we may have found was not compatible with the mutator (in this case, O|SS). If you run with DYNINST_DEBUG_STARTUP set to 1 in your environment, that will tell me what candidates we found and why we discarded them...

--bw

On 01/25/2016 01:24 PM, Jim Galarowicz wrote:

Hi Mike,

I think the reason you aren't seeing that data is because Dyninst aborted while we were asking it to find the loops in your application.
You should see a by function list of the performance information as part of the default view generation from the osspcsamp command.
So, we probably have a Dyninst problem (which we don't see very often). I'm copying the Dyninst team on this one.

Bill, Dyninst team - do you have any ideas?ÂÂ Have you seen this type of assert in the past?

Thanks,
Jim G

On 01/25/2016 12:28 PM, Mike Burklund wrote:

Jim,

Ok I used the full path to mpiexec.hydra and was able to get further. It complained about not having the env variable DYNINST_API_RT_LIB set. I set this and ran again and was able to run. I got this message (not sure if it is bad or not):

ÂÂÂ ossutil: /opt/openss-2.2/openspeedshop-release-2.2/BUILD/hornberg/dyninst-8.2.1/dyninstAPI/src/binaryEdit.C:918: int_variable* BinaryEdit::createTrampGuard(): Assertion `rtlib.size()' failed.

I'm just learning openspeedshop now, but when I look at the results with openss, it say:

ÂÂÂ (There are no objects specified for the basic Detail report.)

So, I guess I need to look into how to get sampling/timing results on the function level of my test application.

Thanks again,

Mike

On Mon, Jan 25, 2016 at 11:40 AM, Jim Galarowicz <jeg@xxxxxxxxxxxxx> wrote:

Hi Mike,

You could also try setting OPENSS_MPI_IMPLEMENTATION=mpich2.ÂÂ This is only really supposed to matter for experiments= mpi, mpit but it also is looked at for finding mpi driver commands.
There might be some special casing for that in the offline.py file.
You could also try the full path to the mpirun or mpirun.hydra commands.

Jim G

On 01/25/2016 10:25 AM, Mike Burklund wrote:

Jim,

Thank you for your timely reply.

Regarding my mpirun command, it is installed with the fedora 23 mpich rpm and is a sym link to mpiexec.hyrdra in the same directory. Maybe the symbolic link is throwing things off.

I'll take a look at your sample module file to check my installation and will let you know the results.

Thanks,

Mike

On Mon, Jan 25, 2016 at 11:04 AM, Jim Galarowicz <jeg@xxxxxxxxxxxxx> wrote:

Hi Mike,

I changed the Subject line to something more meaningful and copied the oss-questions group, so that others might benefit from what we discover.

This is an example module file for setting up the environment to run OpenSpeedShop.
It looks like the build is correct (looking at the debug output).ÂÂ Seems like the problem is that we are not recognizing the mpirun command as an mpi driver command.

I noticed from the output the OPENSS_RAWDATA_DIR is not being set to a shared file system directory location. We need that to write the raw data out while the application is running, then we read from that location at the end of execution to create the database file that contains all the performance data and your application program debug symbol information we use to map the data to.

The other thing I can see from the debug startup output (Thanks for that!) is that we don't seem to be recognizing the mpirun command as the mpi driver. The output says we are running non-MPI.
So, is your mpirun command a script or an executable?ÂÂ Is it mpich based?
It must be an executable based on the output.ÂÂ With no symbols I might have to put a special case in our offline.py file to force it to be known as a mpi driver.
I'm in Minnesota for a week and away from my office. I have a Fedora 23 system there and could try to reproduce, but that won't be until next week.

So, the environment variables set in the module file are the ones that are needed for successful runs of OpenSpeedShop.

I've attached a simple program to test a sequential test program that will tell us if your installation is correct.

$ cc -o mutatee -g -O0 mutatee.c
$ osspcsamp ./mutatee
$ module load ossoff222
$ osspcsamp ./mutatee
[openss]: pcsamp experiment using the default sampling rate: "100".
[openss]: Setting up offline raw data directory in /opt/shared/offline-oss
[openss]: Running offline pcsamp experiment using the command:
"./mutatee"
work(900)=0

[openss]: Converting raw data from /opt/shared/offline-oss into temp file X.0.openss

Processing raw data for mutatee ...
Processing processes and threads ...
Processing performance data ...
Processing symbols ...
Resolving symbols for /home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee
Updating database with symbols ...
Finished ...

[openss]: Restoring and displaying default view for:
ÂÂÂ /home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee-pcsamp.openss
[openss]: The restored experiment identifier is:Â -x 1

ExclusiveÂÂ % of CPUÂ Function (defining location)
ÂCPU timeÂÂÂÂÂÂ TimeÂ
ÂÂÂÂÂÂ inÂÂÂÂÂÂÂÂÂÂÂÂ
Âseconds.ÂÂÂÂÂÂÂÂÂÂÂÂ
24.150000Â 50.756620Â f3 (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,24)
15.530000Â 32.639765Â f2 (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,15)
Â7.890000Â 16.582598Â f1 (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,6)
Â0.010000ÂÂ 0.021017Â work (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,33)

Thanks,
Jim G

$ cat ossoff222
#%Module1.0#####################################################################
##
## openss modulefile
##
proc ModulesHelp { } {
ÂÂÂÂÂÂÂ global version openss

ÂÂÂÂÂÂÂ puts stderr "\topenss - loads the OpenSpeedShop software & application environment"
ÂÂÂÂÂÂÂ puts stderr "\n\tThis adds $oss/* to several of the"
ÂÂÂÂÂÂÂ puts stderr "\tenvironment variables."
ÂÂÂÂÂÂÂ puts stderr "\n\tVersion $version\n"
}

module-whatisÂÂ "Loads the OpenSpeedShop runtime environment."

# for Tcl script use only
setÂÂÂÂ versionÂÂÂÂÂÂÂÂ 2.2

setÂÂÂÂ rootÂÂÂÂÂÂÂÂÂÂÂ /opt/krellroot_v2.2.2
setÂÂÂÂ ossÂÂÂÂÂÂÂÂÂÂÂÂ /opt/ossoff_222

setenvÂÂÂÂÂÂÂÂÂ OPENSS_DOC_DIRÂÂÂ ÂÂÂ $oss/share/doc/packages/OpenSpeedShop
setenvÂÂÂÂÂÂÂÂÂ OPENSS_MPI_IMPLEMENTATIONÂÂÂÂÂÂ openmpi
setenvÂÂÂ ÂÂÂ OPENSS_RAWDATA_DIRÂÂÂ /opt/shared

prepend-pathÂÂÂ PATHÂÂÂ ÂÂÂ ÂÂÂ $root/bin
prepend-pathÂÂÂ PATHÂÂÂ ÂÂÂ ÂÂÂ $oss/bin
prepend-pathÂÂÂ MANPATHÂÂÂ ÂÂÂ ÂÂÂ $oss/share/man

 setenv OPENSS_PLUGIN_PATH $oss/lib64/openspeedshop
ÂÂÂ setenv DYNINSTAPI_RT_LIB $root/lib64/libdyninstAPI_RT.so
ÂÂÂ prepend-path LD_LIBRARY_PATH $root/lib64
ÂÂÂ prepend-path LD_LIBRARY_PATH $oss/lib64

On 01/25/2016 09:40 AM, Mike Burklund wrote:

Jim,

Thanks you for your help! I ended up needing to adding the following paths to my LD_LIBRARY_PATH:

ÂÂ ossoffline_v2.2/lib64
ÂÂ krellroot_v2.2/lib64

These were built in a non-standard location I guess - /opt

I can now open the openss GUI successfully.

However, when I tested a simple mpi application from the command-line, I get the following messages:

ÂÂ nm: /usr/lib64/mpich/bin/mpirun: no symbols
ÂÂ
ÂÂ env: ./mpirun: No such file or directory

ÂÂ No performance measurements were made for the experiment.

I have set OPENSS_MPI_IMPLEMENTATION=mpich

Attached is the output after running with OPENSS_DEBUG_STARTUP=1.

Please let me know if you have any ideas on what else I'm missing? Is there some part of the installation I did not do?

Thanks again,

Mike

On Fri, Jan 22, 2016 at 3:11 PM, Jim Galarowicz <jeg@xxxxxxxxxxxxx> wrote:

Hi Mike,

There are a couple of possibilities that could be causing this.
One:
The library path to the python libraries is not in the LD_LIBRARY_PATH in your module file or OpenSpeedShop runtime set-up script.

Two:
If OpenSpeedShop was compiled with a compiler that is not in the standard /usr install path, the libstdc++.so.6 is not found when starting OpenSpeedShop up.
For this, you can do one of these:

• Load the module file for the non-standard installed compiler or
  
• Add the path to the libstdc++.so.6 library to the LD_LIBRARY_PATH
• Copy that library to the krellroot install library (lib or lib64) sub-directory: cp /opt/gcc/4.8.2/snos/lib64/libstdc++.so.6 <krellroot_installdir>/lib64/.

You can do an "strace openss" to see what files are being loaded, it should show the file being looked for when it fails.

Thanks,
Jim G

On 01/22/2016 12:42 PM, Mike Burklund wrote:

Name:	Mike Burklund
Email:	mdb.inet@xxxxxxxxx
Message:	I have built openspeedshop 2.2 and the krell-root libraries on my Fedora 23 system. I saw no errors in the build process, but when I tried to run osspcsamp on a test program, I got: [openss]: pcsamp experiment using the pcsamp experiment default sampling rate: "100". [openss]: pcsamp experiment calling openss. file not found I tried executing openss directly and just get: file not found Any thoughts on what may be wrong? Thanks, Mike

_______________________________________________
Dyninst-api mailing list
Dyninst-api@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api