Hi Dyninst team,
Here is the abort and output from running GEMM (mpi shoc cuda
benchmark) with our CUDA experiment osscuda.
It seems like Dyninst is trying to get a Dwarf handle on this
file:Â /usr/lib64/libcuda.so.410.48 when we abort.
This is from using Dyninst master top of tree from October 23,
2018. When I go back to the September 15, 2018 top of tree I do not
see the abort.
That seems to indicate that it is a Dyninst issue, so I'm sending
this in because our spack build of O|SS depends on your top of tree,
until 10.0 is released.
Let me know if you need more information.ÂÂ I attached the file
above - gzipped.
Thanks,
Jim G
Here is the output from the run and the core traceback
information.
[jeg@p9-4V100-01 CUDA]$ ulimit -c unlimited
[jeg@p9-4V100-01 CUDA]$ ulimit -c
unlimited
[jeg@p9-4V100-01 CUDA]$ osscuda "mpirun -np 2 ./GEMM"
[openss]: cuda counting all instructions for CPU and GPU.
[openss]: cuda using default periodic sampling rate (10 ms).
[openss]: cuda configuration:
"interval=10000000,PAPI_TOT_INS,inst_executed"
Creating topology file for frontend host p9-4V100-01
Generated topology file: ./cbtfAutoTopology
Running cuda collector.
Program: mpirun -np 2 ./GEMM
Number of mrnet backends: 2
Topology file used: ./cbtfAutoTopology
executing mpi program: mpirun -np 2 cbtfrun --mpi --mrnet -c
cuda ./GEMM
MPI Task 0/1 starting....
MPI Task 1/1 starting....
Chose device: name='Tesla V100-SXM2-16GB' index=0
Chose device: name='Tesla V100-SXM2-16GB' index=1
Running single precision test
Running single precision test
Running double precision test
Running double precision test
testÂÂÂ attsÂÂÂ unitsÂÂÂ medianÂÂÂ meanÂÂÂ stddevÂÂÂ minÂÂÂ
maxÂÂÂ
DGEMM-N(max)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 9.69694ÂÂÂ 9.69694ÂÂÂ
2.91311ÂÂÂ 6.78383ÂÂÂ 12.61ÂÂÂ
DGEMM-N(mean)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 6.3096ÂÂÂ 6.3096ÂÂÂ 0.64253ÂÂÂ
5.66707ÂÂÂ 6.95213ÂÂÂ
DGEMM-N(median)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 6.44043ÂÂÂ 6.44043ÂÂÂ
0.954308ÂÂÂ 5.48612ÂÂÂ 7.39473ÂÂÂ
DGEMM-N(min)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 2.74011ÂÂÂ 2.74011ÂÂÂ
1.78164ÂÂÂ 0.958474ÂÂÂ 4.52175ÂÂÂ
DGEMM-N(stddev)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 2.12004ÂÂÂ 2.12004ÂÂÂ
1.32167ÂÂÂ 0.798364ÂÂÂ 3.44171ÂÂÂ
DGEMM-N_PCIe(max)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.986637ÂÂÂ 0.986637ÂÂÂ
0.382143ÂÂÂ 0.604494ÂÂÂ 1.36878ÂÂÂ
DGEMM-N_PCIe(mean)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.883682ÂÂÂ 0.883682ÂÂÂ
0.290739ÂÂÂ 0.592943ÂÂÂ 1.17442ÂÂÂ
DGEMM-N_PCIe(median)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.931726ÂÂÂ 0.931726ÂÂÂ
0.339714ÂÂÂ 0.592012ÂÂÂ 1.27144ÂÂÂ
DGEMM-N_PCIe(min)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.584403ÂÂÂ 0.584403ÂÂÂ
0.00570615ÂÂÂ 0.578697ÂÂÂ 0.590109ÂÂÂ
DGEMM-N_PCIe(stddev)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.119548ÂÂÂ 0.119548ÂÂÂ
0.110633ÂÂÂ 0.00891468ÂÂÂ 0.230181ÂÂÂ
DGEMM-N_Parity(max)ÂÂÂ 128ÂÂÂ NÂÂÂ 9.21747ÂÂÂ 9.21747ÂÂÂ
1.00485ÂÂÂ 8.21262ÂÂÂ 10.2223ÂÂÂ
DGEMM-N_Parity(mean)ÂÂÂ 128ÂÂÂ NÂÂÂ 6.53364ÂÂÂ 6.53364ÂÂÂ
2.00588ÂÂÂ 4.52775ÂÂÂ 8.53952ÂÂÂ
DGEMM-N_Parity(median)ÂÂÂ 128ÂÂÂ NÂÂÂ 6.54143ÂÂÂ 6.54143ÂÂÂ
1.72542ÂÂÂ 4.81601ÂÂÂ 8.26685ÂÂÂ
DGEMM-N_Parity(min)ÂÂÂ 128ÂÂÂ NÂÂÂ 3.71895ÂÂÂ 3.71895ÂÂÂ
3.09472ÂÂÂ 0.624231ÂÂÂ 6.81368ÂÂÂ
DGEMM-N_Parity(stddev)ÂÂÂ 128ÂÂÂ NÂÂÂ 1.72226ÂÂÂ 1.72226ÂÂÂ
0.519237ÂÂÂ 1.20303ÂÂÂ 2.2415ÂÂÂ
DGEMM-T(max)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 8.73437ÂÂÂ 8.73437ÂÂÂ
1.89898ÂÂÂ 6.83539ÂÂÂ 10.6334ÂÂÂ
DGEMM-T(mean)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 5.84512ÂÂÂ 5.84512ÂÂÂ
0.200883ÂÂÂ 5.64424ÂÂÂ 6.04601ÂÂÂ
DGEMM-T(median)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 6.43896ÂÂÂ 6.43896ÂÂÂ
0.641068ÂÂÂ 5.7979ÂÂÂ 7.08003ÂÂÂ
DGEMM-T(min)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 2.19582ÂÂÂ 2.19582ÂÂÂ
2.05793ÂÂÂ 0.137885ÂÂÂ 4.25375ÂÂÂ
DGEMM-T(stddev)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 2.08324ÂÂÂ 2.08324ÂÂÂ
1.28537ÂÂÂ 0.797875ÂÂÂ 3.36861ÂÂÂ
DGEMM-T_PCIe(max)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.973304ÂÂÂ 0.973304ÂÂÂ
0.368403ÂÂÂ 0.604901ÂÂÂ 1.34171ÂÂÂ
DGEMM-T_PCIe(mean)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.837045ÂÂÂ 0.837045ÂÂÂ
0.244427ÂÂÂ 0.592618ÂÂÂ 1.08147ÂÂÂ
DGEMM-T_PCIe(median)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.928633ÂÂÂ 0.928633ÂÂÂ
0.333162ÂÂÂ 0.59547ÂÂÂ 1.2618ÂÂÂ
DGEMM-T_PCIe(min)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.350296ÂÂÂ 0.350296ÂÂÂ
0.223772ÂÂÂ 0.126523ÂÂÂ 0.574068ÂÂÂ
DGEMM-T_PCIe(stddev)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.186293ÂÂÂ 0.186293ÂÂÂ
0.176799ÂÂÂ 0.00949395ÂÂÂ 0.363091ÂÂÂ
DGEMM-T_Parity(max)ÂÂÂ 128ÂÂÂ NÂÂÂ 8.61263ÂÂÂ 8.61263ÂÂÂ
1.68739ÂÂÂ 6.92524ÂÂÂ 10.3ÂÂÂ
DGEMM-T_Parity(mean)ÂÂÂ 128ÂÂÂ NÂÂÂ 6.22137ÂÂÂ 6.22137ÂÂÂ
2.28375ÂÂÂ 3.93762ÂÂÂ 8.50512ÂÂÂ
DGEMM-T_Parity(median)ÂÂÂ 128ÂÂÂ NÂÂÂ 6.67385ÂÂÂ 6.67385ÂÂÂ
2.0628ÂÂÂ 4.61105ÂÂÂ 8.73666ÂÂÂ
DGEMM-T_Parity(min)ÂÂÂ 128ÂÂÂ NÂÂÂ 3.24982ÂÂÂ 3.24982ÂÂÂ
3.16002ÂÂÂ 0.0898014ÂÂÂ 6.40984ÂÂÂ
DGEMM-T_Parity(stddev)ÂÂÂ 128ÂÂÂ NÂÂÂ 1.69809ÂÂÂ 1.69809ÂÂÂ
0.495801ÂÂÂ 1.20229ÂÂÂ 2.19389ÂÂÂ
SGEMM-N(max)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 62.1606ÂÂÂ 62.1606ÂÂÂ
7.55859ÂÂÂ 54.602ÂÂÂ 69.7191ÂÂÂ
SGEMM-N(mean)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 47.9714ÂÂÂ 47.9714ÂÂÂ
1.68053ÂÂÂ 46.2909ÂÂÂ 49.652ÂÂÂ
SGEMM-N(median)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 54.0888ÂÂÂ 54.0888ÂÂÂ
6.28268ÂÂÂ 47.8061ÂÂÂ 60.3715ÂÂÂ
SGEMM-N(min)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 20.6876ÂÂÂ 20.6876ÂÂÂ
6.84074ÂÂÂ 13.8469ÂÂÂ 27.5284ÂÂÂ
SGEMM-N(stddev)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 14.7462ÂÂÂ 14.7462ÂÂÂ
6.70088ÂÂÂ 8.0453ÂÂÂ 21.4471ÂÂÂ
SGEMM-N_PCIe(max)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 10.3257ÂÂÂ 10.3257ÂÂÂ
1.85213ÂÂÂ 8.47362ÂÂÂ 12.1779ÂÂÂ
SGEMM-N_PCIe(mean)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 9.66712ÂÂÂ 9.66712ÂÂÂ
1.9606ÂÂÂ 7.70653ÂÂÂ 11.6277ÂÂÂ
SGEMM-N_PCIe(median)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 10.0591ÂÂÂ 10.0591ÂÂÂ
1.74278ÂÂÂ 8.31632ÂÂÂ 11.8019ÂÂÂ
SGEMM-N_PCIe(min)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 7.83632ÂÂÂ 7.83632ÂÂÂ
2.1509ÂÂÂ 5.68542ÂÂÂ 9.98722ÂÂÂ
SGEMM-N_PCIe(stddev)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 0.840406ÂÂÂ 0.840406ÂÂÂ
0.206688ÂÂÂ 0.633718ÂÂÂ 1.04709ÂÂÂ
SGEMM-N_Parity(max)ÂÂÂ 256ÂÂÂ NÂÂÂ 5.35575ÂÂÂ 5.35575ÂÂÂ
1.87205ÂÂÂ 3.4837ÂÂÂ 7.22779ÂÂÂ
SGEMM-N_Parity(mean)ÂÂÂ 256ÂÂÂ NÂÂÂ 4.05043ÂÂÂ 4.05043ÂÂÂ
1.09699ÂÂÂ 2.95344ÂÂÂ 5.14743ÂÂÂ
SGEMM-N_Parity(median)ÂÂÂ 256ÂÂÂ NÂÂÂ 4.65442ÂÂÂ 4.65442ÂÂÂ
1.6043ÂÂÂ 3.05012ÂÂÂ 6.25872ÂÂÂ
SGEMM-N_Parity(min)ÂÂÂ 256ÂÂÂ NÂÂÂ 1.59593ÂÂÂ 1.59593ÂÂÂ
0.160425ÂÂÂ 1.43551ÂÂÂ 1.75636ÂÂÂ
SGEMM-N_Parity(stddev)ÂÂÂ 256ÂÂÂ NÂÂÂ 1.36836ÂÂÂ 1.36836ÂÂÂ
0.855058ÂÂÂ 0.513304ÂÂÂ 2.22342ÂÂÂ
SGEMM-T(max)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 63.4329ÂÂÂ 63.4329ÂÂÂ
10.5459ÂÂÂ 52.8869ÂÂÂ 73.9788ÂÂÂ
SGEMM-T(mean)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 45.9749ÂÂÂ 45.9749ÂÂÂ
0.620777ÂÂÂ 45.3541ÂÂÂ 46.5957ÂÂÂ
SGEMM-T(median)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 49.8315ÂÂÂ 49.8315ÂÂÂ
4.88943ÂÂÂ 44.942ÂÂÂ 54.7209ÂÂÂ
SGEMM-T(min)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 19.9912ÂÂÂ 19.9912ÂÂÂ
18.8349ÂÂÂ 1.15635ÂÂÂ 38.8261ÂÂÂ
SGEMM-T(stddev)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 13.8137ÂÂÂ 13.8137ÂÂÂ
9.37892ÂÂÂ 4.43478ÂÂÂ 23.1926ÂÂÂ
SGEMM-T_PCIe(max)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 10.3119ÂÂÂ 10.3119ÂÂÂ
1.77855ÂÂÂ 8.53333ÂÂÂ 12.0904ÂÂÂ
SGEMM-T_PCIe(mean)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 9.42068ÂÂÂ 9.42068ÂÂÂ
2.20647ÂÂÂ 7.2142ÂÂÂ 11.6272ÂÂÂ
SGEMM-T_PCIe(median)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 9.90605ÂÂÂ 9.90605ÂÂÂ
1.71467ÂÂÂ 8.19137ÂÂÂ 11.6207ÂÂÂ
SGEMM-T_PCIe(min)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 6.09928ÂÂÂ 6.09928ÂÂÂ
5.06671ÂÂÂ 1.03257ÂÂÂ 11.166ÂÂÂ
SGEMM-T_PCIe(stddev)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 1.24072ÂÂÂ 1.24072ÂÂÂ
0.950884ÂÂÂ 0.289832ÂÂÂ 2.1916ÂÂÂ
SGEMM-T_Parity(max)ÂÂÂ 256ÂÂÂ NÂÂÂ 5.52184ÂÂÂ 5.52184ÂÂÂ
2.14756ÂÂÂ 3.37428ÂÂÂ 7.66939ÂÂÂ
SGEMM-T_Parity(mean)ÂÂÂ 256ÂÂÂ NÂÂÂ 3.86213ÂÂÂ 3.86213ÂÂÂ
0.968453ÂÂÂ 2.89367ÂÂÂ 4.83058ÂÂÂ
SGEMM-T_Parity(median)ÂÂÂ 256ÂÂÂ NÂÂÂ 4.27015ÂÂÂ 4.27015ÂÂÂ
1.40277ÂÂÂ 2.86738ÂÂÂ 5.67292ÂÂÂ
SGEMM-T_Parity(min)ÂÂÂ 256ÂÂÂ NÂÂÂ 1.29853ÂÂÂ 1.29853ÂÂÂ
1.17865ÂÂÂ 0.119879ÂÂÂ 2.47717ÂÂÂ
SGEMM-T_Parity(stddev)ÂÂÂ 256ÂÂÂ NÂÂÂ 1.34366ÂÂÂ 1.34366ÂÂÂ
1.06072ÂÂÂ 0.282947ÂÂÂ 2.40438ÂÂÂ
Note: results marked with (*) had missing values such as
might occur with a mixture of architectural capabilities.
Detecting outliers based on per-process mean values.
Mild outliers (>1.5 IQR from 1st/3rd quartile):
None.
Extreme outliers (>3.0 IQR from 1st/3rd quartile):
None.
[FE 1803] EXCEPTION: Encountered multiple CUDA_SamplingConfig
for a thread.
[FE 1803] EXCEPTION: Encountered multiple CUDA_SamplingConfig
for a thread.
/opt/tools/OSS/osscbtf_v2.4.0/bin/osscuda: line 1914:Â 1803
Segmentation faultÂÂÂÂÂ (core dumped) osscollect $topology_opt
$cbtf_offline_opt --program "$1" --collector $collector
[jeg@p9-4V100-01 CUDA]$ BE: receive failure
232290224.754090: Message.c[278] Message_send - MRN_send()
failed
232290224.754194: PeerNode.c[176] PeerNode_sendDirectly -
Message_send() failed
BE: receive failure
232290224.754211: Network.c[839] Network_send_PacketToParent -
upstream.send() failed
232290224.754228: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send() failure
232290224.754328: Message.c[278] Message_send - MRN_send()
failed
232290224.754400: PeerNode.c[176] PeerNode_sendDirectly -
Message_send() failed
232290224.754416: Network.c[839] Network_send_PacketToParent -
upstream.send() failed
232290224.754432: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send() failure
^C
[jeg@p9-4V100-01 CUDA]$ lsr
total 105636
ÂÂÂ 0 drwxr-xr-x. 3 jeg openssÂÂÂÂÂÂ 18 Oct 25 09:46 ..
Â2312 -rwxr-xr-x. 1 jeg openss 2366680 Oct 25 10:14
BusSpeedDownload
Â2476 -rwxr-xr-x. 1 jeg openss 2533880 Oct 25 10:14
DeviceMemory
Â2316 -rwxr-xr-x. 1 jeg openss 2368472 Oct 25 10:14
BusSpeedReadback
Â3736 -rwxr-xr-x. 1 jeg openss 3824448 Oct 25 10:14 MaxFlops
Â2948 -rwxr-xr-x. 1 jeg openss 3017920 Oct 25 10:14 BFS
Â2792 -rwxr-xr-x. 1 jeg openss 2858208 Oct 25 10:14 FFT
Â2620 -rwxr-xr-x. 1 jeg openss 2681720 Oct 25 10:14 GEMM
Â2608 -rwxr-xr-x. 1 jeg openss 2669440 Oct 25 10:14 MD
Â2392 -rwxr-xr-x. 1 jeg openss 2445944 Oct 25 10:14 Reduction
Â2488 -rwxr-xr-x. 1 jeg openss 2544208 Oct 25 10:14 Scan
Â2356 -rwxr-xr-x. 1 jeg openss 2410744 Oct 25 10:14 Sort
Â2820 -rwxr-xr-x. 1 jeg openss 2883792 Oct 25 10:14 Spmv
Â2436 -rwxr-xr-x. 1 jeg openss 2491968 Oct 25 10:14 Triad
Â2504 -rwxr-xr-x. 1 jeg openss 2560960 Oct 25 10:14 MD5Hash
Â2444 -rwxr-xr-x. 1 jeg openss 2500432 Oct 25 10:14 NeuralNet
Â4512 -rwxr-xr-x. 1 jeg openss 4617400 Oct 25 10:14 S3D
Â2452 -rwxr-xr-x. 1 jeg openss 2508096 Oct 25 10:14 BusCont
Â2496 -rwxr-xr-x. 1 jeg openss 2551920 Oct 25 10:14 MTBusCont
Â2588 -rwxr-xr-x. 1 jeg openss 2649928 Oct 25 10:14
Stability_mpi
 444 -rw-r--r--. 1 jeg openss 451584 Oct 25 11:54
GEMM-cuda-0.openss
ÂÂÂ 4 -rw-r--r--. 1 jeg openssÂÂÂÂÂÂ 34 Oct 25 12:02
cbtfAutoTopology
ÂÂÂ 4 -rw-r--r--. 1 jeg openssÂÂÂÂÂÂ 44 Oct 25 12:02
attachBE_connections
 524 -rw-r--r--. 1 jeg openss 533504 Oct 25 12:03
GEMM-cuda-1.openss
ÂÂÂ 4 drwxr-xr-x. 2 jeg openssÂÂÂÂ 4096 Oct 25 12:03 .
53360 -rw-------. 1 jeg openss 97779712 Oct 25 12:03 core.1803
[jeg@p9-4V100-01 CUDA]$ file core.1803
core.1803: ELF 64-bit LSB core file 64-bit PowerPC or cisco
7500, version 1 (SYSV), SVR4-style, from 'osscollect --program
mpirun -np 2 ./GEMM --collector cuda', real uid: 1001, effective
uid: 1001, real gid: 1001, effective gid: 1001, execfn:
'/opt/tools/OSS/osscbtf_v2.4.0/bin/osscollect', platform:
'power9'
jeg@p9-4V100-01 CUDA]$ lsr
/usr/lib64/libcuda.so.410.48
15536 -rwxr-xr-x. 1 root root 15906448 Sep 6 05:20
/usr/lib64/libcuda.so.410.48
[jeg@p9-4V100-01 CUDA]$ lsr /usr/lib64/libcuda*
15536 -rwxr-xr-x. 1 root root 15906448 Sep 6 05:20
/usr/lib64/libcuda.so.410.48
 0 lrwxrwxrwx. 1 root root 17 Oct 9 15:49
/usr/lib64/libcuda.so.1 -> libcuda.so.410.48
 0 lrwxrwxrwx. 1 root root 17 Oct 9 15:49
/usr/lib64/libcuda.so -> libcuda.so.410.48
[jeg@p9-4V100-01 CUDA]$ lsr
/usr/local/cuda-10.0/lib64/libcuda*
Â544 -rw-rw-rw-. 1 root root 554512 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudart.so.10.0.130
1152 -rw-rw-rw-. 1 root root 1178786 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudart_static.a
Â680 -rw-rw-rw-. 1 root root 694236 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudadevrt.a
 0 lrwxrwxrwx. 1 root root 21 Oct 9 15:49
/usr/local/cuda-10.0/lib64/libcudart.so.10.0 ->
libcudart.so.10.0.130
 0 lrwxrwxrwx. 1 root root 17 Oct 9 15:49
/usr/local/cuda-10.0/lib64/libcudart.so -> libcudart.so.10.0
[jeg@p9-4V100-01 CUDA]$
[jeg@p9-4V100-01 CUDA]$ ldd /usr/lib64/libcuda.so.410.48
ÂÂÂ linux-vdso64.so.1 =>Â (0x00007fff87e50000)
ÂÂÂ libm.so.6 => /lib64/libm.so.6 (0x00007fff86de0000)
ÂÂÂ libc.so.6 => /lib64/libc.so.6 (0x00007fff86bf0000)
ÂÂÂ libdl.so.2 => /lib64/libdl.so.2 (0x00007fff86bc0000)
ÂÂÂ libpthread.so.0 => /lib64/libpthread.so.0
(0x00007fff86b80000)
ÂÂÂ librt.so.1 => /lib64/librt.so.1 (0x00007fff86b50000)
ÂÂÂ libnvidia-fatbinaryloader.so.410.48 =>
/lib64/libnvidia-fatbinaryloader.so.410.48 (0x00007fff86ae0000)
ÂÂÂ /lib64/ld64.so.2 (0x00007fff87e70000)
[jeg@p9-4V100-01 CUDA]$ file /usr/lib64/libcuda.so.410.48
/usr/lib64/libcuda.so.410.48: ELF 64-bit LSB shared object,
64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically
linked, stripped
[jeg@p9-4V100-01 CUDA]$ gdb osscollect
core.1803
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute
it.
There is NO WARRANTY, to the extent permitted by law. Type
"show copying"
and "show warranty" for details.
This GDB was configured as "ppc64le-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from
/opt/tools/OSS/osscbtf_v2.4.0/bin/osscollect...done.
[New LWP 1909]
[New LWP 1828]
[New LWP 1820]
[New LWP 1830]
[New LWP 1803]
[New LWP 1826]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `osscollect --program mpirun -np 2 ./GEMM
--collector cuda '.
Program terminated with signal 11, Segmentation fault.
#0Â size (this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
716ÂÂÂ ÂÂÂÂÂ { return _M_rep()->_M_length; }
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.ppc64le libattr-2.4.46-13.el7.ppc64le
libcap-2.22-9.el7.ppc64le libicu-50.1.2-15.el7.ppc64le
libtool-ltdl-2.4.2-22.el7_3.ppc64le sqlite-3.7.17-8.el7.ppc64le
systemd-libs-219-57.el7_5.3.ppc64le xz-libs-5.2.2-1.el7.ppc64le
(gdb) where
#0Â size (this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
#1Â compare (__str=..., this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:2227
#2Â operator< <char, std::char_traits<char>,
std::allocator<char> > (__rhs=..., __lhs=<error
reading variable: Cannot access memory at address
0xffffffffffffffe8>)
ÂÂÂ at /usr/include/c++/4.8.2/bits/basic_string.h:2571
#3Â operator() (this=<optimized out>, __y=...,
__x=<error reading variable: Cannot access memory at address
0xffffffffffffffe8>)
ÂÂÂ at /usr/include/c++/4.8.2/bits/stl_function.h:235
#4Â _M_lower_bound (this=<optimized out>,
__k="/usr/lib64/libcuda.so.410.48", __y=<optimized out>,
__x=0x7fff89c283f0) at
/usr/include/c++/4.8.2/bits/stl_tree.h:1141
#5Â find (__k="/usr/lib64/libcuda.so.410.48",
this=0x7fffa39802d8
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
ÂÂÂ at /usr/include/c++/4.8.2/bits/stl_tree.h:1792
#6Â find (__x="/usr/lib64/libcuda.so.410.48",
this=0x7fffa39802d8
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
ÂÂÂ at /usr/include/c++/4.8.2/bits/stl_map.h:822
#7Â
Dyninst::DwarfDyninst::DwarfHandle::createDwarfHandle
(filename_="/usr/lib64/libcuda.so.410.48",
file_=0x7fff89e07740)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/dwarf/src/dwarfHandle.C:255
#8Â 0x00007fffa65d29e8 in Dyninst::SymtabAPI::Object::Object
(this=0x7fff89e06ef0, mf_=0x7fff89c24f10, err_func=<optimized
out>, alloc_syms=<optimized out>,
ÂÂÂ st=<optimized out>) at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Object-elf.C:2986
#9Â 0x00007fffa6577f24 in Dyninst::SymtabAPI::Symtab::Symtab
(this=0x7fff89e06060, filename="", defensive_bin=<optimized
out>, err=@0x7fffa10dc349: false)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Symtab.C:1261
#10 0x00007fffa657838c in Dyninst::SymtabAPI::Symtab::openFile
(obj=@0x7fffa10dc800: 0x0,
filename="/usr/lib64/libcuda.so.410.48",
def_binary=<optimized out>)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Symtab.C:2077
#11 0x00007fffa673cdfc in
OpenSpeedShop::Framework::SymtabAPISymbols::getSymbols
(this=<optimized out>, addresses=..., linkedobject=...,
stm=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/SymtabAPISymbols.cxx:126
#12 0x00007fffa680d188 in
OpenSpeedShop::Framework::Callbacks::finalize ()
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1119
#13 0x00007fffa680eab8 in
OpenSpeedShop::Framework::Callbacks::linkedObjectGroup (in=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1552
#14 0x00007fffa68331e8 in
boost::detail::function::void_function_invoker1<void
(*)(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&), void,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>::invoke (function_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:118
#15 0x00007fffa685f7c4 in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#16 m_invoke<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>,
0u, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>
(
ÂÂÂ args=std::tuple containing = {...}, func=<error reading
variable: value has been optimized out>, this=<optimized
out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#17 operator()<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&,
1ul> (
ÂÂÂ args=std::tuple containing = {...}, func=<error reading
variable: value has been optimized out>, this=<optimized
out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#18
operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > > (connectionBody=...,
this=0x7fffa10dd7a0) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:134
---Type <return> to continue, or q <return> to
quit---Â
#19 dereference (this=0x7fffa10dd7b8) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/slot_call_iterator.hpp:110
#20
dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > > (f=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/iterator/iterator_facade.hpp:550
#21 operator* (this=0x7fffa10dd7b8) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/iterator/iterator_facade.hpp:656
#22
operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > > (last=..., first=...,
this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/optional_last_value.hpp:57
#23
operator()<boost::signals2::optional_last_value<void>,
boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > > (last=..., first=...,
combiner=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/result_type_wrapper.hpp:64
#24 boost::signals2::detail::signal_impl<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&), boost::signals2::optional_last_value<void>,
int, std::less<int>, boost::function<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&)>, boost::function<void
(boost::signals2::connection const&,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&)>,
boost::signals2::mutex>::operator()(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&) (this=<optimized out>,
ÂÂÂ args#0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/signal_template.hpp:247
#25 0x00007fffa685f974 in operator() (args#0=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/signal_template.hpp:722
#26
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>::valueHandler (this=<optimized out>, value=...)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/SignalAdapter.hpp:85
#27 0x00007fffa6833734 in operator() (a1=..., p=<optimized
out>, this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#28 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>
> (a=<synthetic pointer>, f=..., this=<optimized
out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#29 operator()<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>
(a1=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#30
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>, boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>*>, boost::arg<1> > >, void,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>::invoke (function_obj_ptr=...,
---Type <return> to continue, or q <return> to
quit---
ÂÂÂ a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#31 0x00007fffa68432ec in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#32
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>::operator() (
ÂÂÂ this=<error reading variable: value has been optimized
out>, value=...) at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#33 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#34 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#35 0x00007fffa643ec48 in
KrellInstitute::CBTF::Component::emitOutput (this=<optimized
out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#36 0x00007fffa63daf38 in
call<KrellInstitute::CBTF::Impl::MRNet*,
std::basic_string<char> const, KrellInstitute::CBTF::Type
const, boost::any const> (
ÂÂÂ u=<error reading variable: value has been optimized
out>, b3=..., b2=..., b1=..., this=<error reading
variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:384
#37 operator()<KrellInstitute::CBTF::Impl::MRNet*>
(u=<error reading variable: value has been optimized out>,
a3=..., a2=..., a1=...,
ÂÂÂ this=<error reading variable: value has been optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:399
#38 operator()<boost::_mfi::mf3<void,
KrellInstitute::CBTF::Component, const
std::basic_string<char>&, const
KrellInstitute::CBTF::Type&, const boost::any&>,
boost::_bi::rrlist1<const boost::any&> >
(a=<synthetic pointer>, f=<error reading variable:
value has been optimized out>,
ÂÂÂ this=<error reading variable: value has been optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:463
#39 operator()<const boost::any&> (a1=...,
this=<error reading variable: value has been optimized
out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#40
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
std::string const&, KrellInstitute::CBTF::Type const&,
boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::MRNet*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>,
boost::arg<1> > >, void, boost::any
const&>::invoke (function_obj_ptr=..., a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#41 0x00007fffa64442a4 in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#42 KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been optimized
out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#43 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#44 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#45 0x00007fffa643ec48 in
KrellInstitute::CBTF::Component::emitOutput (this=<optimized
out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#46 0x00007fffa643f308 in
call<KrellInstitute::CBTF::Impl::Network*,
std::basic_string<char> const, KrellInstitute::CBTF::Type
const, boost::any const> (
ÂÂÂ u=<error reading variable: value has been optimized
out>, b3=..., b2=..., b1=..., this=<error reading
variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:384
---Type <return> to continue, or q <return> to
quit---
#47 operator()<KrellInstitute::CBTF::Impl::Network*>
(u=<error reading variable: value has been optimized out>,
a3=..., a2=..., a1=...,
ÂÂÂ this=<error reading variable: value has been optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:399
#48 operator()<boost::_mfi::mf3<void,
KrellInstitute::CBTF::Component, const
std::basic_string<char>&, const
KrellInstitute::CBTF::Type&, const boost::any&>,
boost::_bi::rrlist1<const boost::any&> >
(a=<synthetic pointer>, f=<error reading variable:
value has been optimized out>,
ÂÂÂ this=<error reading variable: value has been optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:463
#49 operator()<const boost::any&> (a1=...,
this=<error reading variable: value has been optimized
out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#50
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
std::string const&, KrellInstitute::CBTF::Type const&,
boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::Network*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>,
boost::arg<1> > >, void, boost::any
const&>::invoke (function_obj_ptr=..., a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#51 0x00007fffa64442a4 in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#52 KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been optimized
out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#53 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#54 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#55 0x00007fffa2bd0fa0 in
emitOutput<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
> (value=..., name="LinkedObjectGroup", this=<optimized
out>)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Component.hpp:330
#56 StateManagementForCUDA::handleThreadsStateChanged
(this=0x7fff9c02e380, message=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf-argonavis/CUDA/components/StateManagementForCUDA.cpp:291
#57 0x00007fffa2bd1c94 in operator() (a1=..., p=<optimized
out>, this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#58 operator()<boost::_mfi::mf1<void,
StateManagementForCUDA, const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>
> (a=<synthetic pointer>, f=..., this=<optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#59 operator()<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>
(a1=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#60
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, StateManagementForCUDA,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
const&>,
boost::_bi::list2<boost::_bi::value<StateManagementForCUDA*>,
boost::arg<1> > >, void,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
const&>::invoke (function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#61 0x00007fffa2bd342c in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#62
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
>::operator() (
ÂÂÂ this=<error reading variable: value has been optimized
out>, value=...) at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#63 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#64 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
---Type <return> to continue, or q <return> to
quit---
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#65 0x00007fffa643ecc8 in emitOutput (value=..., type=...,
name="value", this=0x7fff9c02a700)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#66 KrellInstitute::CBTF::Impl::InputMediator::handler
(this=0x7fff9c02a700, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-xml/InputMediator.hpp:70
#67 0x00007fffa643db94 in operator() (a1=..., p=<optimized
out>, this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#68 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::InputMediator, const
boost::any&>, boost::_bi::rrlist1<const
boost::any&> > (
ÂÂÂ a=<synthetic pointer>, f=..., this=<optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#69 operator()<const boost::any&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#70
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::InputMediator, boost::any
const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::Impl::InputMediator*>,
boost::arg<1> > >, void, boost::any
const&>::invoke (function_obj_ptr=..., a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#71 0x00007fffa64442a4 in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#72 KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been optimized
out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#73 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#74 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#75 0x00007fffa2be0f44 in
emitOutput<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
> (value=<synthetic pointer>, name="out",
this=0x7fff9c0076f0)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Component.hpp:330
#76
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>::handler
(this=0x7fff9c0076f0, in=...)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/XDR.hpp:205
#77 0x00007fffa2bd67a4 in operator() (a1=..., p=<optimized
out>, this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#78 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
const boost::shared_ptr<MRN::Packet>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> >
(a=<synthetic pointer>, f=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#79 operator()<const
boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#80
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>*>,
boost::arg<1> > >, void,
boost::shared_ptr<MRN::Packet> const&>::invoke
(function_obj_ptr=..., a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#81 0x00007fffa63e357c in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#82
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<MRN::Packet>
>::operator() (this=<error reading variable: value has
been optimized out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#83 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
---Type <return> to continue, or q <return> to
quit---
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#84 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#85 0x00007fffa63c9494 in
emitOutput<boost::shared_ptr<MRN::Packet> >
(value=..., name="value", this=0x7fff9c0072d0)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:330
#86 KrellInstitute::CBTF::Impl::IncomingStreamMediator::handler
(this=0x7fff9c0072d0, packet=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/IncomingStreamMediator.cpp:77
#87 0x00007fffa63dafe8 in
call<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>,
boost::shared_ptr<MRN::Packet> const> (b1=..., u=...,
ÂÂÂ this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:156
#88
operator()<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>
> (a1=..., u=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:171
#89 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator, const
boost::shared_ptr<MRN::Packet>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> >
(a=<synthetic pointer>, f=..., this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#90 operator()<const
boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#91
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>
>, boost::arg<1> > >, void,
boost::shared_ptr<MRN::Packet> const&>::invoke
(function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#92 0x00007fffa63d1460 in operator() (a0=...,
this=0x7fff8802c6a0) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#93 KrellInstitute::CBTF::Impl::MessageHandlers::operator()
(this=0x7fff9c002a88, tag=@0x7fffa10de690: 1124, packet=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/MessageHandlers.cpp:103
#94 0x00007fffa63bab90 in
KrellInstitute::CBTF::Impl::Frontend::doMessagePump
(this=0x7fff9c002a80)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/Frontend.cpp:409
#95 0x00007fffa63bcea4 in operator() (p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:49
#96 operator()<boost::_mfi::mf0<void,
KrellInstitute::CBTF::Impl::Frontend>, boost::_bi::list0>
(a=<synthetic pointer>, f=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:259
#97 operator() (this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1294
#98 boost::detail::thread_data<boost::_bi::bind_t<void,
boost::_mfi::mf0<void,
KrellInstitute::CBTF::Impl::Frontend>,
boost::_bi::list1<boost::_bi::value<KrellInstitute::CBTF::Impl::Frontend*>
> > >::run (this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/thread/detail/thread.hpp:117
#99 0x00007fffa73323b0 in thread_proxy () from
/opt/tools/OSS/krellroot_v2.4.0/lib/libboost_thread.so.1.67.0
#100 0x00007fffa66f8af4 in start_thread (arg=0x7fffa10df170) at
pthread_create.c:310
#101 0x00007fffa5e78814 in clone () at
../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:109
(gdb)
(gdb)
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}