Re: [DynInst_API:] Dyninst assert -- Re: Problem with running OpenSpeedShop - mpirun command?

[Mike Burklund <mdb.inet@xxxxxxxxx>]

Bill,

Please find attached the dyninst debug information. Thank you for your help!

Mike

On Mon, Jan 25, 2016 at 2:34 PM, Bill Williams <bill@xxxxxxxxxxx> wrote:

Generally this means that whatever runtime library we may have found was not compatible with the mutator (in this case, O|SS). If you run with DYNINST_DEBUG_STARTUP set to 1 in your environment, that will tell me what candidates we found and why we discarded them...

--bw

On 01/25/2016 01:24 PM, Jim Galarowicz wrote:

Hi Mike,

I think the reason you aren't seeing that data is because Dyninst aborted while we were asking it to find the loops in your application.
You should see a by function list of the performance information as part of the default view generation from the osspcsamp command.
So, we probably have a Dyninst problem (which we don't see very often). I'm copying the Dyninst team on this one.

Bill, Dyninst team - do you have any ideas?ÂÂ Have you seen this type of assert in the past?

Thanks,
Jim G

On 01/25/2016 12:28 PM, Mike Burklund wrote:

Jim,

Ok I used the full path to mpiexec.hydra and was able to get further. It complained about not having the env variable DYNINST_API_RT_LIB set. I set this and ran again and was able to run. I got this message (not sure if it is bad or not):

ÂÂÂ ossutil: /opt/openss-2.2/openspeedshop-release-2.2/BUILD/hornberg/dyninst-8.2.1/dyninstAPI/src/binaryEdit.C:918: int_variable* BinaryEdit::createTrampGuard(): Assertion `rtlib.size()' failed.

I'm just learning openspeedshop now, but when I look at the results with openss, it say:

ÂÂÂ (There are no objects specified for the basic Detail report.)

So, I guess I need to look into how to get sampling/timing results on the function level of my test application.

Thanks again,

Mike

On Mon, Jan 25, 2016 at 11:40 AM, Jim Galarowicz <jeg@xxxxxxxxxxxxx> wrote:

Hi Mike,

You could also try setting OPENSS_MPI_IMPLEMENTATION=mpich2.ÂÂ This is only really supposed to matter for experiments= mpi, mpit but it also is looked at for finding mpi driver commands.
There might be some special casing for that in the offline.py file.
You could also try the full path to the mpirun or mpirun.hydra commands.

Jim G

On 01/25/2016 10:25 AM, Mike Burklund wrote:

Jim,

Thank you for your timely reply.

Regarding my mpirun command, it is installed with the fedora 23 mpich rpm and is a sym link to mpiexec.hyrdra in the same directory. Maybe the symbolic link is throwing things off.

I'll take a look at your sample module file to check my installation and will let you know the results.

Thanks,

Mike

On Mon, Jan 25, 2016 at 11:04 AM, Jim Galarowicz <jeg@xxxxxxxxxxxxx> wrote:

Hi Mike,

I changed the Subject line to something more meaningful and copied the oss-questions group, so that others might benefit from what we discover.

This is an example module file for setting up the environment to run OpenSpeedShop.
It looks like the build is correct (looking at the debug output).ÂÂ Seems like the problem is that we are not recognizing the mpirun command as an mpi driver command.

I noticed from the output the OPENSS_RAWDATA_DIR is not being set to a shared file system directory location. We need that to write the raw data out while the application is running, then we read from that location at the end of execution to create the database file that contains all the performance data and your application program debug symbol information we use to map the data to.

The other thing I can see from the debug startup output (Thanks for that!) is that we don't seem to be recognizing the mpirun command as the mpi driver. The output says we are running non-MPI.
So, is your mpirun command a script or an executable?ÂÂ Is it mpich based?
It must be an executable based on the output.ÂÂ With no symbols I might have to put a special case in our offline.py file to force it to be known as a mpi driver.
I'm in Minnesota for a week and away from my office. I have a Fedora 23 system there and could try to reproduce, but that won't be until next week.

So, the environment variables set in the module file are the ones that are needed for successful runs of OpenSpeedShop.

I've attached a simple program to test a sequential test program that will tell us if your installation is correct.

$ cc -o mutatee -g -O0 mutatee.c
$ osspcsamp ./mutatee
$ module load ossoff222
$ osspcsamp ./mutatee
[openss]: pcsamp experiment using the default sampling rate: "100".
[openss]: Setting up offline raw data directory in /opt/shared/offline-oss
[openss]: Running offline pcsamp experiment using the command:
"./mutatee"
work(900)=0

[openss]: Converting raw data from /opt/shared/offline-oss into temp file X.0.openss

Processing raw data for mutatee ...
Processing processes and threads ...
Processing performance data ...
Processing symbols ...
Resolving symbols for /home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee
Updating database with symbols ...
Finished ...

[openss]: Restoring and displaying default view for:
ÂÂÂ /home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee-pcsamp.openss
[openss]: The restored experiment identifier is:Â -x 1

ExclusiveÂÂ % of CPUÂ Function (defining location)
ÂCPU timeÂÂÂÂÂÂ TimeÂ
ÂÂÂÂÂÂ inÂÂÂÂÂÂÂÂÂÂÂÂ
Âseconds.ÂÂÂÂÂÂÂÂÂÂÂÂ
24.150000Â 50.756620Â f3 (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,24)
15.530000Â 32.639765Â f2 (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,15)
Â7.890000Â 16.582598Â f1 (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,6)
Â0.010000ÂÂ 0.021017Â work (/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee:/home/jeg/OpenSpeedShop/test/executables/sequential/mutatee/c_version/mutatee.c,33)

Thanks,
Jim G

$ cat ossoff222
#%Module1.0#####################################################################
##
## openss modulefile
##
proc ModulesHelp { } {
ÂÂÂÂÂÂÂ global version openss

ÂÂÂÂÂÂÂ puts stderr "\topenss - loads the OpenSpeedShop software & application environment"
ÂÂÂÂÂÂÂ puts stderr "\n\tThis adds $oss/* to several of the"
ÂÂÂÂÂÂÂ puts stderr "\tenvironment variables."
ÂÂÂÂÂÂÂ puts stderr "\n\tVersion $version\n"
}

module-whatisÂÂ "Loads the OpenSpeedShop runtime environment."

# for Tcl script use only
setÂÂÂÂ versionÂÂÂÂÂÂÂÂ 2.2

setÂÂÂÂ rootÂÂÂÂÂÂÂÂÂÂÂ /opt/krellroot_v2.2.2
setÂÂÂÂ ossÂÂÂÂÂÂÂÂÂÂÂÂ /opt/ossoff_222

setenvÂÂÂÂÂÂÂÂÂ OPENSS_DOC_DIRÂÂÂ ÂÂÂ $oss/share/doc/packages/OpenSpeedShop
setenvÂÂÂÂÂÂÂÂÂ OPENSS_MPI_IMPLEMENTATIONÂÂÂÂÂÂ openmpi
setenvÂÂÂ ÂÂÂ OPENSS_RAWDATA_DIRÂÂÂ /opt/shared

prepend-pathÂÂÂ PATHÂÂÂ ÂÂÂ ÂÂÂ $root/bin
prepend-pathÂÂÂ PATHÂÂÂ ÂÂÂ ÂÂÂ $oss/bin
prepend-pathÂÂÂ MANPATHÂÂÂ ÂÂÂ ÂÂÂ $oss/share/man

 setenv OPENSS_PLUGIN_PATH $oss/lib64/openspeedshop
ÂÂÂ setenv DYNINSTAPI_RT_LIB $root/lib64/libdyninstAPI_RT.so
ÂÂÂ prepend-path LD_LIBRARY_PATH $root/lib64
ÂÂÂ prepend-path LD_LIBRARY_PATH $oss/lib64

On 01/25/2016 09:40 AM, Mike Burklund wrote:

Jim,

Thanks you for your help! I ended up needing to adding the following paths to my LD_LIBRARY_PATH:

ÂÂ ossoffline_v2.2/lib64
ÂÂ krellroot_v2.2/lib64

These were built in a non-standard location I guess - /opt

I can now open the openss GUI successfully.

However, when I tested a simple mpi application from the command-line, I get the following messages:

ÂÂ nm: /usr/lib64/mpich/bin/mpirun: no symbols
ÂÂ
ÂÂ env: ./mpirun: No such file or directory

ÂÂ No performance measurements were made for the experiment.

I have set OPENSS_MPI_IMPLEMENTATION=mpich

Attached is the output after running with OPENSS_DEBUG_STARTUP=1.

Please let me know if you have any ideas on what else I'm missing? Is there some part of the installation I did not do?

Thanks again,

Mike

On Fri, Jan 22, 2016 at 3:11 PM, Jim Galarowicz <jeg@xxxxxxxxxxxxx> wrote:

Hi Mike,

There are a couple of possibilities that could be causing this.
One:
The library path to the python libraries is not in the LD_LIBRARY_PATH in your module file or OpenSpeedShop runtime set-up script.

Two:
If OpenSpeedShop was compiled with a compiler that is not in the standard /usr install path, the libstdc++.so.6 is not found when starting OpenSpeedShop up.
For this, you can do one of these:

• Load the module file for the non-standard installed compiler or
  
• Add the path to the libstdc++.so.6 library to the LD_LIBRARY_PATH
• Copy that library to the krellroot install library (lib or lib64) sub-directory: cp /opt/gcc/4.8.2/snos/lib64/libstdc++.so.6 <krellroot_installdir>/lib64/.

You can do an "strace openss" to see what files are being loaded, it should show the file being looked for when it fails.

Thanks,
Jim G

On 01/22/2016 12:42 PM, Mike Burklund wrote:

Name:	Mike Burklund
Email:	mdb.inet@xxxxxxxxx
Message:	I have built openspeedshop 2.2 and the krell-root libraries on my Fedora 23 system. I saw no errors in the build process, but when I tried to run osspcsamp on a test program, I got: [openss]: pcsamp experiment using the pcsamp experiment default sampling rate: "100". [openss]: pcsamp experiment calling openss. file not found I tried executing openss directly and just get: file not found Any thoughts on what may be wrong? Thanks, Mike

_______________________________________________
Dyninst-api mailing list
Dyninst-api@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api

Enabling DyninstAPI startup debug
[139779478669120][BPatch_binaryEdit.C:89] - Opening original file /data/FVV/DESIGN/openss_test1/computePi
[139779478669120]mapped_object.C[146]:  about to parseImage
[139779478669120]mapped_object.C[148]: name /data/FVV/DESIGN/openss_test1/computePi, codeBase 0x0, dataBase 0x0
[139779478669120]image.C[1067]:  about to create image
[139779478669120]image.C[1262]:  opening file /data/FVV/DESIGN/openss_test1/computePi
[139779478669120]image.C[1353]:  before findMain
[139779478669120]image.C[1384]:  before determineImageType
[139779478669120]image.C[1069]:  created image
[139779478669120]image.C[1074]: processing executable object
[139779478669120]mapped_object.C[195]:  creating mapped object
[139779478669120][mapped_object.C:105] Creating new mapped_object /data/FVV/DESIGN/openss_test1/computePi/
[139779478669120][mapped_object.C:107] 	codeBase 0x0, codeOffset 0x400000, size 5900
[139779478669120][mapped_object.C:109] 	dataBase 0x0, dataOffset 0x601d98, size 820
[139779478669120][mapped_object.C:111] 	fileDescriptor: code at 0x0, data 0x0
[139779478669120][mapped_object.C:113] 	Code: 0x400000 to 0x40170c
[139779478669120][mapped_object.C:115] 	Data: 0x601d98 to 0x6020cc
[139779478669120]mapped_object.C[200]:  leaving createMappedObject(/data/FVV/DESIGN/openss_test1/computePi)
[139779478669120]dyninstRT_name: 
[139779478669120]Invalid Dyninst RT lib name: /opt/openss-2.2/krellroot_v2.2/lib64
[139779478669120][unix.C:533] - Opening dependent file /opt/openss-2.2/krellroot_v2.2/lib64
[139779478669120]mapped_object.C[146]:  about to parseImage
[139779478669120]mapped_object.C[148]: name /opt/openss-2.2/krellroot_v2.2/lib64, codeBase 0x0, dataBase 0x0
[139779478669120]image.C[1067]:  about to create image
[139779478669120]image.C[1262]:  opening file /opt/openss-2.2/krellroot_v2.2/lib64
[139779478669120]image.C[1069]:  created image
[139779478669120]image.C[1074]: processing executable object
[139779478669120]image.C[1090]: error in processing, deleting image and returning
[139779478669120]mapped_object.C[151]:  failed to parseImage
[139779478669120]binaryEdit.C[342]: failed to create mapped object for /opt/openss-2.2/krellroot_v2.2/lib64
[139779478669120][unix.C:620] - Creation error opening /opt/openss-2.2/krellroot_v2.2/lib64
ossutil: /opt/openss-2.2/openspeedshop-release-2.2/BUILD/hornberg/dyninst-8.2.1/dyninstAPI/src/binaryEdit.C:918: int_variable* BinaryEdit::createTrampGuard(): Assertion `rtlib.size()' failed.

[openss]: Restoring and displaying default view for:
	/data/FVV/DESIGN/openss_test1/computePi-pcsamp-3.openss
[openss]: The restored experiment identifier is:  -x 1
(There are no objects specified for the basic Detail report.)