|
Hi Xiaozhu, all,
Thanks for your help getting past the API changes!
Unfortunately, with top of master, I still get the core dump:
Note: results marked with (*) had missing values
such as
might occur with a mixture of architectural capabilities.
Detecting outliers based on per-process mean values.
Mild outliers (>1.5 IQR from 1st/3rd quartile):
None.
Extreme outliers (>3.0 IQR from 1st/3rd quartile):
None.
[p9-4V100-01:59338] 1 more process has sent help message
help-mpi-btl-openib.txt / no active ports found
[p9-4V100-01:59338] Set MCA parameter "orte_base_help_aggregate"
to 0 to see all help / error messages
/opt/tools/OSS/osscbtf_v2.4.0dyn1026/bin/osscuda: line 1914:
59289 Segmentation faultÂÂÂÂÂ (core dumped) osscollect
$topology_opt $cbtf_offline_opt --program "$1" --collector
$collector
[jeg@p9-4V100-01 CUDA]$ BE: receive failure
BE: receive failure
232389263.030495: Message.c[278] Message_send - MRN_send()
failed
232389263.030495: Message.c[278] Message_send - MRN_send()
failed
232389263.030562: PeerNode.c[176] PeerNode_sendDirectly -
232389263.030563: PeerNode.c[176] PeerNode_sendDirectly -
Message_send() failed
232389263.030576: Network.c[839] Network_send_PacketToParent -
upstream.send() failed
Message_send() failed
232389263.030577: Network.c[839] Network_send_PacketToParent -
upstream.send() failed
232389263.030592: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send() failure
232389263.030588: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send() failure
^C
[jeg@p9-4V100-01 CUDA]$ gdb osscollect core.59289
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
..
Using host libthread_db library
"/lib64/libthread_db.so.1".
Core was generated by `osscollect --program mpirun -np 2 ./GEMM
--collector cuda '.
Program terminated with signal 11, Segmentation fault.
#0Â size (this=0x7fff6d798350) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
716ÂÂÂ ÂÂÂÂÂ { return _M_rep()->_M_length; }
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.ppc64le libattr-2.4.46-13.el7.ppc64le
libcap-2.22-9.el7.ppc64le libicu-50.1.2-15.el7.ppc64le
libtool-ltdl-2.4.2-22.el7_3.ppc64le sqlite-3.7.17-8.el7.ppc64le
systemd-libs-219-57.el7_5.3.ppc64le
tbb-4.1-9.20130314.el7.ppc64le xz-libs-5.2.2-1.el7.ppc64le
(gdb) where
#0Â size (this=0x7fff6d798350) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
#1Â compare (__str=..., this=0x7fff6d798350) at
/usr/include/c++/4.8.2/bits/basic_string.h:2227
#2Â operator< <char, std::char_traits<char>,
std::allocator<char> > (__rhs=..., __lhs=<error
reading variable: Cannot access memory at address
0xffffffffffffffe8>)
ÂÂÂ at /usr/include/c++/4.8.2/bits/basic_string.h:2571
#3Â operator() (this=<optimized out>, __y=...,
__x=<error reading variable: Cannot access memory at address
0xffffffffffffffe8>) at
/usr/include/c++/4.8.2/bits/stl_function.h:235
#4Â _M_lower_bound (this=<optimized out>,
__k="/usr/lib64/libcuda.so.410.48", __y=<optimized out>,
__x=0x7fff6d798330) at
/usr/include/c++/4.8.2/bits/stl_tree.h:1141
#5Â find (__k="/usr/lib64/libcuda.so.410.48",
this=0x7fff897602d0
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
at /usr/include/c++/4.8.2/bits/stl_tree.h:1792
#6Â find (__x="/usr/lib64/libcuda.so.410.48",
this=0x7fff897602d0
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
at /usr/include/c++/4.8.2/bits/stl_map.h:822
#7Â Dyninst::DwarfDyninst::DwarfHandle::createDwarfHandle
(filename_="/usr/lib64/libcuda.so.410.48", file_=0x7fff6d98c910)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181026/dwarf/src/dwarfHandle.C:255
#8Â 0x00007fff8c485988 in Dyninst::SymtabAPI::Object::Object
(this=0x7fff6d98c0c0, mf_=0x7fff6d7a1ab0, err_func=<optimized
out>, alloc_syms=<optimized out>, st=<optimized
out>)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181026/symtabAPI/src/Object-elf.C:2888
#9Â 0x00007fff8c42b280 in Dyninst::SymtabAPI::Symtab::Symtab
(this=0x7fff6d98ae10, filename="", defensive_bin=<optimized
out>, err=@0x7fff827cc1b9: false)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181026/symtabAPI/src/Symtab.C:1273
#10 0x00007fff8c42b6dc in Dyninst::SymtabAPI::Symtab::openFile
(obj=@0x7fff827cc670: 0x0,
filename="/usr/lib64/libcuda.so.410.48",
def_binary=<optimized out>)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181026/symtabAPI/src/Symtab.C:2091
#11 0x00007fff8c5ed8ac in
OpenSpeedShop::Framework::SymtabAPISymbols::getSymbols
(this=<optimized out>, addresses=..., linkedobject=...,
stm=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/SymtabAPISymbols.cxx:126
#12 0x00007fff8c6bd188 in
OpenSpeedShop::Framework::Callbacks::finalize ()
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1119
#13 0x00007fff8c6beab8 in
OpenSpeedShop::Framework::Callbacks::linkedObjectGroup (in=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1552
#14 0x00007fff8c6e31e8 in
boost::detail::function::void_function_invoker1<void
(*)(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&), void,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>::invoke (function_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:118
---Type <return> to continue, or q <return> to
quit---
#15 0x00007fff8c70f7c4 in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#16 m_invoke<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>,
0u, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>
(
ÂÂÂ args=std::tuple containing = {...}, func=<error reading
variable: value has been optimized out>, this=<optimized
out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#17 operator()<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&,
1ul> (
ÂÂÂ args=std::tuple containing = {...}, func=<error reading
variable: value has been optimized out>, this=<optimized
out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#18
operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > > (connectionBody=...,
ÂÂÂ this=0x7fff827cd610) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/variadic_slot_invoker.hpp:134
#19 dereference (this=0x7fff827cd628) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/slot_call_iterator.hpp:110
#20
dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > > (f=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/iterator/iterator_facade.hpp:550
#21 operator* (this=0x7fff827cd628) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/iterator/iterator_facade.hpp:656
#22
operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > > (last=..., first=...,
this=<optimized out>)
---Type <return> to continue, or q <return> to
quit---
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/optional_last_value.hpp:57
#23
operator()<boost::signals2::optional_last_value<void>,
boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > > (last=..., first=...,
ÂÂÂ combiner=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/result_type_wrapper.hpp:64
#24 boost::signals2::detail::signal_impl<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&), boost::signals2::optional_last_value<void>,
int, std::less<int>, boost::function<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&)>, boost::function<void
(boost::signals2::connection const&,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&)>,
boost::signals2::mutex>::operator()(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&) (this=<optimized out>, args#0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/signal_template.hpp:247
#25 0x00007fff8c70f974 in operator() (args#0=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/signals2/detail/signal_template.hpp:722
#26
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>::valueHandler (this=<optimized out>, value=...)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/SignalAdapter.hpp:85
#27 0x00007fff8c6e3734 in operator() (a1=..., p=<optimized
out>, this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:165
#28 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>
> (a=<synthetic pointer>, f=..., this=<optimized
out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:319
#29 operator()<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>
(a1=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#30
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>, boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>*>, boost::arg<1> > >, void,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>::invoke (function_obj_ptr=..., a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#31 0x00007fff8c6f32ec in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
---Type <return> to continue, or q <return> to
quit---
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#32
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>::operator() (this=<error reading variable: value has
been optimized out>, value=...)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#33 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#34 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#35 0x00007fff8c2eec68 in
KrellInstitute::CBTF::Component::emitOutput (this=<optimized
out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#36 0x00007fff8c28af38 in
call<KrellInstitute::CBTF::Impl::MRNet*,
std::basic_string<char> const, KrellInstitute::CBTF::Type
const, boost::any const> (
ÂÂÂ u=<error reading variable: value has been optimized
out>, b3=..., b2=..., b1=..., this=<error reading
variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:384
#37 operator()<KrellInstitute::CBTF::Impl::MRNet*>
(u=<error reading variable: value has been optimized out>,
a3=..., a2=..., a1=...,
ÂÂÂ this=<error reading variable: value has been optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:399
#38 operator()<boost::_mfi::mf3<void,
KrellInstitute::CBTF::Component, const
std::basic_string<char>&, const
KrellInstitute::CBTF::Type&, const boost::any&>,
boost::_bi::rrlist1<const boost::any&> >
(a=<synthetic pointer>, f=<error reading variable:
value has been optimized out>, this=<error reading
variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:463
#39 operator()<const boost::any&> (a1=...,
this=<error reading variable: value has been optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#40
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
std::string const&, KrellInstitute::CBTF::Type const&,
boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::MRNet*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>,
boost::arg<1> > >, void, boost::any
const&>::invoke (function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#41 0x00007fff8c2f42c4 in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#42 KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been optimized
out>, value=...)
---Type <return> to continue, or q <return> to
quit---
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#43 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#44 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#45 0x00007fff8c2eec68 in
KrellInstitute::CBTF::Component::emitOutput (this=<optimized
out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#46 0x00007fff8c2ef328 in
call<KrellInstitute::CBTF::Impl::Network*,
std::basic_string<char> const, KrellInstitute::CBTF::Type
const, boost::any const> (
ÂÂÂ u=<error reading variable: value has been optimized
out>, b3=..., b2=..., b1=..., this=<error reading
variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:384
#47 operator()<KrellInstitute::CBTF::Impl::Network*>
(u=<error reading variable: value has been optimized out>,
a3=..., a2=..., a1=...,
ÂÂÂ this=<error reading variable: value has been optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:399
#48 operator()<boost::_mfi::mf3<void,
KrellInstitute::CBTF::Component, const
std::basic_string<char>&, const
KrellInstitute::CBTF::Type&, const boost::any&>,
boost::_bi::rrlist1<const boost::any&> >
(a=<synthetic pointer>, f=<error reading variable:
value has been optimized out>, this=<error reading
variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:463
#49 operator()<const boost::any&> (a1=...,
this=<error reading variable: value has been optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#50
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void, KrellInstitute::CBTF::Component,
std::string const&, KrellInstitute::CBTF::Type const&,
boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::Network*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>,
boost::arg<1> > >, void, boost::any
const&>::invoke (function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#51 0x00007fff8c2f42c4 in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#52 KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been optimized
out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#53 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
---Type <return> to continue, or q <return> to
quit---
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#54 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#55 0x00007fff886b0fc0 in
emitOutput<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
> (value=..., name="LinkedObjectGroup", this=<optimized
out>)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/Component.hpp:330
#56 StateManagementForCUDA::handleThreadsStateChanged
(this=0x7fff8402d820, message=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf-argonavis/CUDA/components/StateManagementForCUDA.cpp:291
#57 0x00007fff886b1cb4 in operator() (a1=..., p=<optimized
out>, this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:165
#58 operator()<boost::_mfi::mf1<void,
StateManagementForCUDA, const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>
> (a=<synthetic pointer>, f=..., this=<optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:319
#59 operator()<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>
(a1=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#60
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, StateManagementForCUDA,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
const&>,
boost::_bi::list2<boost::_bi::value<StateManagementForCUDA*>,
boost::arg<1> > >, void,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
const&>::invoke (function_obj_ptr=..., a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#61 0x00007fff886b344c in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#62
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
>::operator() (this=<error reading variable: value has
been optimized out>, value=...)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#63 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#64 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#65 0x00007fff8c2eece8 in emitOutput (value=..., type=...,
name="value", this=0x7fff8402a470)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
---Type <return> to continue, or q <return> to
quit---
#66 KrellInstitute::CBTF::Impl::InputMediator::handler
(this=0x7fff8402a470, value=...) at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-xml/InputMediator.hpp:70
#67 0x00007fff8c2edbb4 in operator() (a1=..., p=<optimized
out>, this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:165
#68 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::InputMediator, const
boost::any&>, boost::_bi::rrlist1<const
boost::any&> > (a=<synthetic pointer>, f=...,
ÂÂÂ this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:319
#69 operator()<const boost::any&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#70
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::InputMediator, boost::any
const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::Impl::InputMediator*>,
boost::arg<1> > >, void, boost::any
const&>::invoke (function_obj_ptr=..., a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#71 0x00007fff8c2f42c4 in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#72 KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been optimized
out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#73 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#74 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#75 0x00007fff886c0f64 in
emitOutput<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
> (value=<synthetic pointer>, name="out",
this=0x7fff8401c3c0)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/Component.hpp:330
#76
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>::handler
(this=0x7fff8401c3c0, in=...)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0dyn1026/include/KrellInstitute/CBTF/XDR.hpp:205
#77 0x00007fff886b67c4 in operator() (a1=..., p=<optimized
out>, this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:165
#78 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
const boost::shared_ptr<MRN::Packet>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> >
(a=<synthetic pointer>, f=..., this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:319
#79 operator()<const
boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
---Type <return> to continue, or q <return> to
quit---
#80
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>*>,
boost::arg<1> > >, void,
boost::shared_ptr<MRN::Packet> const&>::invoke
(function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#81 0x00007fff8c29357c in operator() (a0=..., this=<error
reading variable: value has been optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#82
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<MRN::Packet>
>::operator() (this=<error reading variable: value has
been optimized out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#83 0x00007fff8c34b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#84 0x00007fff8c348ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=..., value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#85 0x00007fff8c279494 in
emitOutput<boost::shared_ptr<MRN::Packet> >
(value=..., name="value", this=0x7fff84001c20)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:330
#86 KrellInstitute::CBTF::Impl::IncomingStreamMediator::handler
(this=0x7fff84001c20, packet=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/IncomingStreamMediator.cpp:77
#87 0x00007fff8c28afe8 in
call<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>,
boost::shared_ptr<MRN::Packet> const> (b1=..., u=...,
this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:156
#88
operator()<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>
> (a1=..., u=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:171
#89 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator, const
boost::shared_ptr<MRN::Packet>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> >
ÂÂÂ (a=<synthetic pointer>, f=..., this=<optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:319
#90 operator()<const
boost::shared_ptr<MRN::Packet>&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1306
#91
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>
>, boost::arg<1> > >, void,
boost::shared_ptr<MRN::Packet> const&>::invoke (
---Type <return> to continue, or q <return> to
quit---
ÂÂÂ function_obj_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:159
#92 0x00007fff8c281460 in operator() (a0=...,
this=0x7fff6c021a00) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/function/function_template.hpp:769
#93 KrellInstitute::CBTF::Impl::MessageHandlers::operator()
(this=0x7fff84017c68, tag=@0x7fff827ce500: 1124, packet=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/MessageHandlers.cpp:103
#94 0x00007fff8c26ab90 in
KrellInstitute::CBTF::Impl::Frontend::doMessagePump
(this=0x7fff84017c60) at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/Frontend.cpp:409
#95 0x00007fff8c26cea4 in operator() (p=<optimized out>,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/mem_fn_template.hpp:49
#96 operator()<boost::_mfi::mf0<void,
KrellInstitute::CBTF::Impl::Frontend>, boost::_bi::list0>
(a=<synthetic pointer>, f=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:259
#97 operator() (this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/bind/bind.hpp:1294
#98 boost::detail::thread_data<boost::_bi::bind_t<void,
boost::_mfi::mf0<void,
KrellInstitute::CBTF::Impl::Frontend>,
boost::_bi::list1<boost::_bi::value<KrellInstitute::CBTF::Impl::Frontend*>
> > >::run (this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0dyn1026/include/boost/thread/detail/thread.hpp:117
#99 0x00007fff8d1e23b0 in thread_proxy () from
/opt/tools/OSS/krellroot_v2.4.0dyn1026/lib/libboost_thread.so.1.67.0
#100 0x00007fff8c5a8af4 in start_thread (arg=0x7fff827cefe0) at
pthread_create.c:310
#101 0x00007fff8bd28814 in clone () at
../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:109
On 10/26/2018 12:17 PM, Xiaozhu Meng
wrote:
Hi Jim,
Can you try with the latest Dyninst master? It looks like
the crash is during Symtab::openFile based your stack trace.
But, on my end with the latest master branch, I can open the
library file provided by you using Symtab and perform other
analysis.Â
By the way, the master branch now has the parallel parsing
merged. So, it will need two more dependencies (tbb and
openmp). Dyninst will automatically download and install tbb
and will use openmp by default. The master branch also has the
two planed API breaking change:
1. InstructionAPI::Instruction is now passed by value
rather than boost smart pointer. So, all Instruction::Ptr i
should be replaced as InstructionÂi.Â
2. Operation.h is renamed as Operation_impl.h.
Let me know if you encounter any problem with the latest
master branch.
Thanks,
--Xiaozhu
Hi all,
Posting again without the library attached - the first
email went to the moderator list.
Jim G
On
10/26/2018 09:59 AM, Jim Galarowicz wrote:
Hi Dyninst team,
Here is the abort and output from running GEMM (mpi
shoc cuda benchmark) with our CUDA experiment osscuda.
It seems like Dyninst is trying to get a Dwarf handle
on this file:Â /usr/lib64/libcuda.so.410.48 when we
abort.
This is from using Dyninst master top of tree from
October 23, 2018. When I go back to the September 15,
2018 top of tree I do not see the abort.
That seems to indicate that it is a Dyninst issue, so
I'm sending this in because our spack build of O|SS
depends on your top of tree, until 10.0 is released.
Let me know if you need more information.ÂÂ I attached
the file above - gzipped.
Thanks,
Jim G
Here is the output from the run and the core
traceback information.
[jeg@p9-4V100-01 CUDA]$ ulimit -c
unlimited
[jeg@p9-4V100-01 CUDA]$ ulimit -c
unlimited
[jeg@p9-4V100-01 CUDA]$ osscuda "mpirun -np 2
./GEMM"
[openss]: cuda counting all instructions for CPU and
GPU.
[openss]: cuda using default periodic sampling rate
(10 ms).
[openss]: cuda configuration:
"interval=10000000,PAPI_TOT_INS,inst_executed"
Creating topology file for frontend host p9-4V100-01
Generated topology file: ./cbtfAutoTopology
Running cuda collector.
Program: mpirun -np 2 ./GEMM
Number of mrnet backends: 2
Topology file used: ./cbtfAutoTopology
executing mpi program: mpirun -np 2 cbtfrun --mpiÂ
--mrnet -c cuda ./GEMM
MPI Task 0/1 starting....
MPI Task 1/1 starting....
Chose device: name='Tesla V100-SXM2-16GB' index=0
Chose device: name='Tesla V100-SXM2-16GB' index=1
Running single precision test
Running single precision test
Running double precision test
Running double precision test
testÂÂÂ attsÂÂÂ unitsÂÂÂ medianÂÂÂ meanÂÂÂ stddevÂÂÂ
minÂÂÂ maxÂÂÂ
DGEMM-N(max)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 9.69694ÂÂÂ
9.69694ÂÂÂ 2.91311ÂÂÂ 6.78383ÂÂÂ 12.61ÂÂÂ
DGEMM-N(mean)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 6.3096ÂÂÂ
6.3096ÂÂÂ 0.64253ÂÂÂ 5.66707ÂÂÂ 6.95213ÂÂÂ
DGEMM-N(median)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 6.44043ÂÂÂ
6.44043ÂÂÂ 0.954308ÂÂÂ 5.48612ÂÂÂ 7.39473ÂÂÂ
DGEMM-N(min)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 2.74011ÂÂÂ
2.74011ÂÂÂ 1.78164ÂÂÂ 0.958474ÂÂÂ 4.52175ÂÂÂ
DGEMM-N(stddev)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 2.12004ÂÂÂ
2.12004ÂÂÂ 1.32167ÂÂÂ 0.798364ÂÂÂ 3.44171ÂÂÂ
DGEMM-N_PCIe(max)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.986637ÂÂÂ
0.986637ÂÂÂ 0.382143ÂÂÂ 0.604494ÂÂÂ 1.36878ÂÂÂ
DGEMM-N_PCIe(mean)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.883682ÂÂÂ
0.883682ÂÂÂ 0.290739ÂÂÂ 0.592943ÂÂÂ 1.17442ÂÂÂ
DGEMM-N_PCIe(median)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.931726ÂÂÂ
0.931726ÂÂÂ 0.339714ÂÂÂ 0.592012ÂÂÂ 1.27144ÂÂÂ
DGEMM-N_PCIe(min)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.584403ÂÂÂ
0.584403ÂÂÂ 0.00570615ÂÂÂ 0.578697ÂÂÂ 0.590109ÂÂÂ
DGEMM-N_PCIe(stddev)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.119548ÂÂÂ
0.119548ÂÂÂ 0.110633ÂÂÂ 0.00891468ÂÂÂ 0.230181ÂÂÂ
DGEMM-N_Parity(max)ÂÂÂ 128ÂÂÂ NÂÂÂ 9.21747ÂÂÂ
9.21747ÂÂÂ 1.00485ÂÂÂ 8.21262ÂÂÂ 10.2223ÂÂÂ
DGEMM-N_Parity(mean)ÂÂÂ 128ÂÂÂ NÂÂÂ 6.53364ÂÂÂ
6.53364ÂÂÂ 2.00588ÂÂÂ 4.52775ÂÂÂ 8.53952ÂÂÂ
DGEMM-N_Parity(median)ÂÂÂ 128ÂÂÂ NÂÂÂ 6.54143ÂÂÂ
6.54143ÂÂÂ 1.72542ÂÂÂ 4.81601ÂÂÂ 8.26685ÂÂÂ
DGEMM-N_Parity(min)ÂÂÂ 128ÂÂÂ NÂÂÂ 3.71895ÂÂÂ
3.71895ÂÂÂ 3.09472ÂÂÂ 0.624231ÂÂÂ 6.81368ÂÂÂ
DGEMM-N_Parity(stddev)ÂÂÂ 128ÂÂÂ NÂÂÂ 1.72226ÂÂÂ
1.72226ÂÂÂ 0.519237ÂÂÂ 1.20303ÂÂÂ 2.2415ÂÂÂ
DGEMM-T(max)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 8.73437ÂÂÂ
8.73437ÂÂÂ 1.89898ÂÂÂ 6.83539ÂÂÂ 10.6334ÂÂÂ
DGEMM-T(mean)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 5.84512ÂÂÂ
5.84512ÂÂÂ 0.200883ÂÂÂ 5.64424ÂÂÂ 6.04601ÂÂÂ
DGEMM-T(median)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 6.43896ÂÂÂ
6.43896ÂÂÂ 0.641068ÂÂÂ 5.7979ÂÂÂ 7.08003ÂÂÂ
DGEMM-T(min)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 2.19582ÂÂÂ
2.19582ÂÂÂ 2.05793ÂÂÂ 0.137885ÂÂÂ 4.25375ÂÂÂ
DGEMM-T(stddev)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 2.08324ÂÂÂ
2.08324ÂÂÂ 1.28537ÂÂÂ 0.797875ÂÂÂ 3.36861ÂÂÂ
DGEMM-T_PCIe(max)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.973304ÂÂÂ
0.973304ÂÂÂ 0.368403ÂÂÂ 0.604901ÂÂÂ 1.34171ÂÂÂ
DGEMM-T_PCIe(mean)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.837045ÂÂÂ
0.837045ÂÂÂ 0.244427ÂÂÂ 0.592618ÂÂÂ 1.08147ÂÂÂ
DGEMM-T_PCIe(median)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.928633ÂÂÂ
0.928633ÂÂÂ 0.333162ÂÂÂ 0.59547ÂÂÂ 1.2618ÂÂÂ
DGEMM-T_PCIe(min)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.350296ÂÂÂ
0.350296ÂÂÂ 0.223772ÂÂÂ 0.126523ÂÂÂ 0.574068ÂÂÂ
DGEMM-T_PCIe(stddev)ÂÂÂ 128ÂÂÂ GFlopsÂÂÂ 0.186293ÂÂÂ
0.186293ÂÂÂ 0.176799ÂÂÂ 0.00949395ÂÂÂ 0.363091ÂÂÂ
DGEMM-T_Parity(max)ÂÂÂ 128ÂÂÂ NÂÂÂ 8.61263ÂÂÂ
8.61263ÂÂÂ 1.68739ÂÂÂ 6.92524ÂÂÂ 10.3ÂÂÂ
DGEMM-T_Parity(mean)ÂÂÂ 128ÂÂÂ NÂÂÂ 6.22137ÂÂÂ
6.22137ÂÂÂ 2.28375ÂÂÂ 3.93762ÂÂÂ 8.50512ÂÂÂ
DGEMM-T_Parity(median)ÂÂÂ 128ÂÂÂ NÂÂÂ 6.67385ÂÂÂ
6.67385ÂÂÂ 2.0628ÂÂÂ 4.61105ÂÂÂ 8.73666ÂÂÂ
DGEMM-T_Parity(min)ÂÂÂ 128ÂÂÂ NÂÂÂ 3.24982ÂÂÂ
3.24982ÂÂÂ 3.16002ÂÂÂ 0.0898014ÂÂÂ 6.40984ÂÂÂ
DGEMM-T_Parity(stddev)ÂÂÂ 128ÂÂÂ NÂÂÂ 1.69809ÂÂÂ
1.69809ÂÂÂ 0.495801ÂÂÂ 1.20229ÂÂÂ 2.19389ÂÂÂ
SGEMM-N(max)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 62.1606ÂÂÂ
62.1606ÂÂÂ 7.55859ÂÂÂ 54.602ÂÂÂ 69.7191ÂÂÂ
SGEMM-N(mean)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 47.9714ÂÂÂ
47.9714ÂÂÂ 1.68053ÂÂÂ 46.2909ÂÂÂ 49.652ÂÂÂ
SGEMM-N(median)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 54.0888ÂÂÂ
54.0888ÂÂÂ 6.28268ÂÂÂ 47.8061ÂÂÂ 60.3715ÂÂÂ
SGEMM-N(min)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 20.6876ÂÂÂ
20.6876ÂÂÂ 6.84074ÂÂÂ 13.8469ÂÂÂ 27.5284ÂÂÂ
SGEMM-N(stddev)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 14.7462ÂÂÂ
14.7462ÂÂÂ 6.70088ÂÂÂ 8.0453ÂÂÂ 21.4471ÂÂÂ
SGEMM-N_PCIe(max)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 10.3257ÂÂÂ
10.3257ÂÂÂ 1.85213ÂÂÂ 8.47362ÂÂÂ 12.1779ÂÂÂ
SGEMM-N_PCIe(mean)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 9.66712ÂÂÂ
9.66712ÂÂÂ 1.9606ÂÂÂ 7.70653ÂÂÂ 11.6277ÂÂÂ
SGEMM-N_PCIe(median)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 10.0591ÂÂÂ
10.0591ÂÂÂ 1.74278ÂÂÂ 8.31632ÂÂÂ 11.8019ÂÂÂ
SGEMM-N_PCIe(min)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 7.83632ÂÂÂ
7.83632ÂÂÂ 2.1509ÂÂÂ 5.68542ÂÂÂ 9.98722ÂÂÂ
SGEMM-N_PCIe(stddev)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 0.840406ÂÂÂ
0.840406ÂÂÂ 0.206688ÂÂÂ 0.633718ÂÂÂ 1.04709ÂÂÂ
SGEMM-N_Parity(max)ÂÂÂ 256ÂÂÂ NÂÂÂ 5.35575ÂÂÂ
5.35575ÂÂÂ 1.87205ÂÂÂ 3.4837ÂÂÂ 7.22779ÂÂÂ
SGEMM-N_Parity(mean)ÂÂÂ 256ÂÂÂ NÂÂÂ 4.05043ÂÂÂ
4.05043ÂÂÂ 1.09699ÂÂÂ 2.95344ÂÂÂ 5.14743ÂÂÂ
SGEMM-N_Parity(median)ÂÂÂ 256ÂÂÂ NÂÂÂ 4.65442ÂÂÂ
4.65442ÂÂÂ 1.6043ÂÂÂ 3.05012ÂÂÂ 6.25872ÂÂÂ
SGEMM-N_Parity(min)ÂÂÂ 256ÂÂÂ NÂÂÂ 1.59593ÂÂÂ
1.59593ÂÂÂ 0.160425ÂÂÂ 1.43551ÂÂÂ 1.75636ÂÂÂ
SGEMM-N_Parity(stddev)ÂÂÂ 256ÂÂÂ NÂÂÂ 1.36836ÂÂÂ
1.36836ÂÂÂ 0.855058ÂÂÂ 0.513304ÂÂÂ 2.22342ÂÂÂ
SGEMM-T(max)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 63.4329ÂÂÂ
63.4329ÂÂÂ 10.5459ÂÂÂ 52.8869ÂÂÂ 73.9788ÂÂÂ
SGEMM-T(mean)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 45.9749ÂÂÂ
45.9749ÂÂÂ 0.620777ÂÂÂ 45.3541ÂÂÂ 46.5957ÂÂÂ
SGEMM-T(median)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 49.8315ÂÂÂ
49.8315ÂÂÂ 4.88943ÂÂÂ 44.942ÂÂÂ 54.7209ÂÂÂ
SGEMM-T(min)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 19.9912ÂÂÂ
19.9912ÂÂÂ 18.8349ÂÂÂ 1.15635ÂÂÂ 38.8261ÂÂÂ
SGEMM-T(stddev)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 13.8137ÂÂÂ
13.8137ÂÂÂ 9.37892ÂÂÂ 4.43478ÂÂÂ 23.1926ÂÂÂ
SGEMM-T_PCIe(max)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 10.3119ÂÂÂ
10.3119ÂÂÂ 1.77855ÂÂÂ 8.53333ÂÂÂ 12.0904ÂÂÂ
SGEMM-T_PCIe(mean)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 9.42068ÂÂÂ
9.42068ÂÂÂ 2.20647ÂÂÂ 7.2142ÂÂÂ 11.6272ÂÂÂ
SGEMM-T_PCIe(median)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 9.90605ÂÂÂ
9.90605ÂÂÂ 1.71467ÂÂÂ 8.19137ÂÂÂ 11.6207ÂÂÂ
SGEMM-T_PCIe(min)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 6.09928ÂÂÂ
6.09928ÂÂÂ 5.06671ÂÂÂ 1.03257ÂÂÂ 11.166ÂÂÂ
SGEMM-T_PCIe(stddev)ÂÂÂ 256ÂÂÂ GFlopsÂÂÂ 1.24072ÂÂÂ
1.24072ÂÂÂ 0.950884ÂÂÂ 0.289832ÂÂÂ 2.1916ÂÂÂ
SGEMM-T_Parity(max)ÂÂÂ 256ÂÂÂ NÂÂÂ 5.52184ÂÂÂ
5.52184ÂÂÂ 2.14756ÂÂÂ 3.37428ÂÂÂ 7.66939ÂÂÂ
SGEMM-T_Parity(mean)ÂÂÂ 256ÂÂÂ NÂÂÂ 3.86213ÂÂÂ
3.86213ÂÂÂ 0.968453ÂÂÂ 2.89367ÂÂÂ 4.83058ÂÂÂ
SGEMM-T_Parity(median)ÂÂÂ 256ÂÂÂ NÂÂÂ 4.27015ÂÂÂ
4.27015ÂÂÂ 1.40277ÂÂÂ 2.86738ÂÂÂ 5.67292ÂÂÂ
SGEMM-T_Parity(min)ÂÂÂ 256ÂÂÂ NÂÂÂ 1.29853ÂÂÂ
1.29853ÂÂÂ 1.17865ÂÂÂ 0.119879ÂÂÂ 2.47717ÂÂÂ
SGEMM-T_Parity(stddev)ÂÂÂ 256ÂÂÂ NÂÂÂ 1.34366ÂÂÂ
1.34366ÂÂÂ 1.06072ÂÂÂ 0.282947ÂÂÂ 2.40438ÂÂÂ
Note: results marked with (*) had missing values
such as
might occur with a mixture of architectural
capabilities.
Detecting outliers based on per-process mean values.
Mild outliers (>1.5 IQR from 1st/3rd quartile):
None.
Extreme outliers (>3.0 IQR from 1st/3rd
quartile):
None.
[FE 1803] EXCEPTION: Encountered multiple
CUDA_SamplingConfig for a thread.
[FE 1803] EXCEPTION: Encountered multiple
CUDA_SamplingConfig for a thread.
/opt/tools/OSS/osscbtf_v2.4.0/bin/osscuda: line
1914:Â 1803 Segmentation faultÂÂÂÂÂ (core dumped)
osscollect $topology_opt $cbtf_offline_opt --program
"$1" --collector $collector
[jeg@p9-4V100-01 CUDA]$ BE: receive failure
232290224.754090: Message.c[278] Message_send -
MRN_send() failed
232290224.754194: PeerNode.c[176]
PeerNode_sendDirectly - Message_send() failed
BE: receive failure
232290224.754211: Network.c[839]
Network_send_PacketToParent - upstream.send() failed
232290224.754228: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send()
failure
232290224.754328: Message.c[278] Message_send -
MRN_send() failed
232290224.754400: PeerNode.c[176]
PeerNode_sendDirectly - Message_send() failed
232290224.754416: Network.c[839]
Network_send_PacketToParent - upstream.send() failed
232290224.754432: Stream.c[438] Stream_send_aux -
Network_send_PacketToParent failed
CBTF_Waitfor_MRNet_Shutdown BE: stream::send()
failure
^C
[jeg@p9-4V100-01 CUDA]$ lsr
total 105636
ÂÂÂ 0 drwxr-xr-x. 3 jeg openssÂÂÂÂÂÂ 18 Oct 25 09:46
..
Â2312 -rwxr-xr-x. 1 jeg openss 2366680 Oct 25 10:14
BusSpeedDownload
Â2476 -rwxr-xr-x. 1 jeg openss 2533880 Oct 25 10:14
DeviceMemory
Â2316 -rwxr-xr-x. 1 jeg openss 2368472 Oct 25 10:14
BusSpeedReadback
Â3736 -rwxr-xr-x. 1 jeg openss 3824448 Oct 25 10:14
MaxFlops
Â2948 -rwxr-xr-x. 1 jeg openss 3017920 Oct 25 10:14
BFS
Â2792 -rwxr-xr-x. 1 jeg openss 2858208 Oct 25 10:14
FFT
Â2620 -rwxr-xr-x. 1 jeg openss 2681720 Oct 25 10:14
GEMM
Â2608 -rwxr-xr-x. 1 jeg openss 2669440 Oct 25 10:14
MD
Â2392 -rwxr-xr-x. 1 jeg openss 2445944 Oct 25 10:14
Reduction
Â2488 -rwxr-xr-x. 1 jeg openss 2544208 Oct 25 10:14
Scan
Â2356 -rwxr-xr-x. 1 jeg openss 2410744 Oct 25 10:14
Sort
Â2820 -rwxr-xr-x. 1 jeg openss 2883792 Oct 25 10:14
Spmv
Â2436 -rwxr-xr-x. 1 jeg openss 2491968 Oct 25 10:14
Triad
Â2504 -rwxr-xr-x. 1 jeg openss 2560960 Oct 25 10:14
MD5Hash
Â2444 -rwxr-xr-x. 1 jeg openss 2500432 Oct 25 10:14
NeuralNet
Â4512 -rwxr-xr-x. 1 jeg openss 4617400 Oct 25 10:14
S3D
Â2452 -rwxr-xr-x. 1 jeg openss 2508096 Oct 25 10:14
BusCont
Â2496 -rwxr-xr-x. 1 jeg openss 2551920 Oct 25 10:14
MTBusCont
Â2588 -rwxr-xr-x. 1 jeg openss 2649928 Oct 25 10:14
Stability_mpi
 444 -rw-r--r--. 1 jeg openss 451584 Oct 25 11:54
GEMM-cuda-0.openss
ÂÂÂ 4 -rw-r--r--. 1 jeg openssÂÂÂÂÂÂ 34 Oct 25 12:02
cbtfAutoTopology
ÂÂÂ 4 -rw-r--r--. 1 jeg openssÂÂÂÂÂÂ 44 Oct 25 12:02
attachBE_connections
 524 -rw-r--r--. 1 jeg openss 533504 Oct 25 12:03
GEMM-cuda-1.openss
ÂÂÂ 4 drwxr-xr-x. 2 jeg openssÂÂÂÂ 4096 Oct 25 12:03
.
53360 -rw-------. 1 jeg openss 97779712 Oct 25 12:03
core.1803
[jeg@p9-4V100-01 CUDA]$ file
core.1803
core.1803: ELF 64-bit LSB core file 64-bit PowerPC
or cisco 7500, version 1 (SYSV), SVR4-style, from
'osscollect --program mpirun -np 2 ./GEMM
--collector cuda', real uid: 1001, effective uid:
1001, real gid: 1001, effective gid: 1001, execfn:
'/opt/tools/OSS/osscbtf_v2.4.0/bin/osscollect',
platform: 'power9'
jeg@p9-4V100-01 CUDA]$ lsr
/usr/lib64/libcuda.so.410.48
15536 -rwxr-xr-x. 1 root root 15906448 Sep 6 05:20
/usr/lib64/libcuda.so.410.48
[jeg@p9-4V100-01 CUDA]$ lsr /usr/lib64/libcuda*
15536 -rwxr-xr-x. 1 root root 15906448 Sep 6 05:20
/usr/lib64/libcuda.so.410.48
 0 lrwxrwxrwx. 1 root root 17 Oct 9 15:49
/usr/lib64/libcuda.so.1 -> libcuda.so.410.48
 0 lrwxrwxrwx. 1 root root 17 Oct 9 15:49
/usr/lib64/libcuda.so -> libcuda.so.410.48
[jeg@p9-4V100-01 CUDA]$ lsr
/usr/local/cuda-10.0/lib64/libcuda*
Â544 -rw-rw-rw-. 1 root root 554512 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudart.so.10.0.130
1152 -rw-rw-rw-. 1 root root 1178786 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudart_static.a
Â680 -rw-rw-rw-. 1 root root 694236 Aug 25 22:01
/usr/local/cuda-10.0/lib64/libcudadevrt.a
 0 lrwxrwxrwx. 1 root root 21 Oct 9 15:49
/usr/local/cuda-10.0/lib64/libcudart.so.10.0 ->
libcudart.so.10.0.130
 0 lrwxrwxrwx. 1 root root 17 Oct 9 15:49
/usr/local/cuda-10.0/lib64/libcudart.so ->
libcudart.so.10.0
[jeg@p9-4V100-01 CUDA]$
[jeg@p9-4V100-01 CUDA]$ lddÂ
/usr/lib64/libcuda.so.410.48
ÂÂÂ linux-vdso64.so.1 =>Â (0x00007fff87e50000)
ÂÂÂ libm.so.6 => /lib64/libm.so.6
(0x00007fff86de0000)
ÂÂÂ libc.so.6 => /lib64/libc.so.6
(0x00007fff86bf0000)
ÂÂÂ libdl.so.2 => /lib64/libdl.so.2
(0x00007fff86bc0000)
ÂÂÂ libpthread.so.0 => /lib64/libpthread.so.0
(0x00007fff86b80000)
ÂÂÂ librt.so.1 => /lib64/librt.so.1
(0x00007fff86b50000)
ÂÂÂ libnvidia-fatbinaryloader.so.410.48 =>
/lib64/libnvidia-fatbinaryloader.so.410.48
(0x00007fff86ae0000)
ÂÂÂ /lib64/ld64.so.2 (0x00007fff87e70000)
[jeg@p9-4V100-01 CUDA]$ fileÂ
/usr/lib64/libcuda.so.410.48
/usr/lib64/libcuda.so.410.48: ELF 64-bit LSB shared
object, 64-bit PowerPC or cisco 7500, version 1
(SYSV), dynamically linked, stripped
[jeg@p9-4V100-01 CUDA]$ gdb
osscollect core.1803
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and
redistribute it.
There is NO WARRANTY, to the extent permitted by
law. Type "show copying"
and "show warranty" for details.
This GDB was configured as
"ppc64le-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from
/opt/tools/OSS/osscbtf_v2.4.0/bin/osscollect...done.
[New LWP 1909]
[New LWP 1828]
[New LWP 1820]
[New LWP 1830]
[New LWP 1803]
[New LWP 1826]
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib64/libthread_db.so.1".
Core was generated by `osscollect --program mpirun
-np 2 ./GEMM --collector cuda '.
Program terminated with signal 11, Segmentation
fault.
#0Â size (this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
716ÂÂÂ ÂÂÂÂÂ { return _M_rep()->_M_length; }
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.ppc64le
libattr-2.4.46-13.el7.ppc64le
libcap-2.22-9.el7.ppc64le
libicu-50.1.2-15.el7.ppc64le
libtool-ltdl-2.4.2-22.el7_3.ppc64le
sqlite-3.7.17-8.el7.ppc64le
systemd-libs-219-57.el7_5.3.ppc64le
xz-libs-5.2.2-1.el7.ppc64le
(gdb) where
#0Â size (this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:716
#1Â compare (__str=..., this=0x7fff89c28410) at
/usr/include/c++/4.8.2/bits/basic_string.h:2227
#2Â operator< <char,
std::char_traits<char>,
std::allocator<char> > (__rhs=...,
__lhs=<error reading variable: Cannot access
memory at address 0xffffffffffffffe8>)
ÂÂÂ at
/usr/include/c++/4.8.2/bits/basic_string.h:2571
#3Â operator() (this=<optimized out>, __y=...,
__x=<error reading variable: Cannot access memory
at address 0xffffffffffffffe8>)
ÂÂÂ at
/usr/include/c++/4.8.2/bits/stl_function.h:235
#4Â _M_lower_bound (this=<optimized out>,
__k="/usr/lib64/libcuda.so.410.48",
__y=<optimized out>, __x=0x7fff89c283f0) at
/usr/include/c++/4.8.2/bits/stl_tree.h:1141
#5Â find (__k="/usr/lib64/libcuda.so.410.48",
this=0x7fffa39802d8
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
ÂÂÂ at /usr/include/c++/4.8.2/bits/stl_tree.h:1792
#6Â find (__x="/usr/lib64/libcuda.so.410.48",
this=0x7fffa39802d8
<Dyninst::DwarfDyninst::DwarfHandle::all_dwarf_handles>)
ÂÂÂ at /usr/include/c++/4.8.2/bits/stl_map.h:822
#7Â
Dyninst::DwarfDyninst::DwarfHandle::createDwarfHandle
(filename_="/usr/lib64/libcuda.so.410.48",
file_=0x7fff89e07740)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/dwarf/src/dwarfHandle.C:255
#8Â 0x00007fffa65d29e8 in
Dyninst::SymtabAPI::Object::Object
(this=0x7fff89e06ef0, mf_=0x7fff89c24f10,
err_func=<optimized out>,
alloc_syms=<optimized out>,
ÂÂÂ st=<optimized out>) at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Object-elf.C:2986
#9Â 0x00007fffa6577f24 in
Dyninst::SymtabAPI::Symtab::Symtab
(this=0x7fff89e06060, filename="",
defensive_bin=<optimized out>,
err=@0x7fffa10dc349: false)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Symtab.C:1261
#10 0x00007fffa657838c in
Dyninst::SymtabAPI::Symtab::openFile
(obj=@0x7fffa10dc800: 0x0,
filename="/usr/lib64/libcuda.so.410.48",
def_binary=<optimized out>)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/dyninst-20181023/symtabAPI/src/Symtab.C:2077
#11 0x00007fffa673cdfc in
OpenSpeedShop::Framework::SymtabAPISymbols::getSymbols
(this=<optimized out>, addresses=...,
linkedobject=..., stm=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/SymtabAPISymbols.cxx:126
#12 0x00007fffa680d188 in
OpenSpeedShop::Framework::Callbacks::finalize ()
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1119
#13 0x00007fffa680eab8 in
OpenSpeedShop::Framework::Callbacks::linkedObjectGroup
(in=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/openspeedshop-2.4.0/libopenss-framework/cbtf/Callbacks.cxx:1552
#14 0x00007fffa68331e8 in
boost::detail::function::void_function_invoker1<void
(*)(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&), void,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>::invoke (function_ptr=..., a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:118
#15 0x00007fffa685f7c4 in operator() (a0=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#16 m_invoke<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>,
0u, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>
(
ÂÂÂ args=std::tuple containing = {...},
func=<error reading variable: value has been
optimized out>, this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
#17 operator()<boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&,
1ul> (
ÂÂÂ args=std::tuple containing = {...},
func=<error reading variable: value has been
optimized out>, this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#18
operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >
(connectionBody=..., this=0x7fffa10dd7a0) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/variadic_slot_invoker.hpp:134
---Type <return> to continue, or q
<return> to quit---Â
#19 dereference (this=0x7fffa10dd7b8) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/slot_call_iterator.hpp:110
#20
dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > > (f=...)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/iterator/iterator_facade.hpp:550
#21 operator* (this=0x7fffa10dd7b8) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/iterator/iterator_facade.hpp:656
#22
operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >
(last=..., first=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/optional_last_value.hpp:57
#23
operator()<boost::signals2::optional_last_value<void>,
boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type,
const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >,
boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group,
boost::optional<int> >,
boost::signals2::slot<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&),
boost::function<void(const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&)>
>, boost::signals2::mutex> > >
(last=..., first=..., combiner=...,
this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/result_type_wrapper.hpp:64
#24 boost::signals2::detail::signal_impl<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&),
boost::signals2::optional_last_value<void>,
int, std::less<int>, boost::function<void
(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&)>, boost::function<void
(boost::signals2::connection const&,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&)>,
boost::signals2::mutex>::operator()(boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&) (this=<optimized out>,
ÂÂÂ args#0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/signal_template.hpp:247
#25 0x00007fffa685f974 in operator() (args#0=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/signals2/detail/signal_template.hpp:722
#26
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>::valueHandler (this=<optimized out>,
value=...)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/SignalAdapter.hpp:85
#27 0x00007fffa6833734 in operator() (a1=...,
p=<optimized out>, this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#28 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>, const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>
> (a=<synthetic pointer>, f=...,
this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#29 operator()<const
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>&>
(a1=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#30
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::SignalAdapter<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>*>, boost::arg<1> > >, void,
boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
const&>::invoke (function_obj_ptr=...,
---Type <return> to continue, or q
<return> to quit---
ÂÂÂ a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#31 0x00007fffa68432ec in operator() (a0=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#32
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
>::operator() (
ÂÂÂ this=<error reading variable: value has been
optimized out>, value=...) at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#33 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#34 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#35 0x00007fffa643ec48 in
KrellInstitute::CBTF::Component::emitOutput
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#36 0x00007fffa63daf38 in
call<KrellInstitute::CBTF::Impl::MRNet*,
std::basic_string<char> const,
KrellInstitute::CBTF::Type const, boost::any
const> (
ÂÂÂ u=<error reading variable: value has been
optimized out>, b3=..., b2=..., b1=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:384
#37
operator()<KrellInstitute::CBTF::Impl::MRNet*>
(u=<error reading variable: value has been
optimized out>, a3=..., a2=..., a1=...,
ÂÂÂ this=<error reading variable: value has been
optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:399
#38 operator()<boost::_mfi::mf3<void,
KrellInstitute::CBTF::Component, const
std::basic_string<char>&, const
KrellInstitute::CBTF::Type&, const
boost::any&>, boost::_bi::rrlist1<const
boost::any&> > (a=<synthetic
pointer>, f=<error reading variable: value has
been optimized out>,
ÂÂÂ this=<error reading variable: value has been
optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:463
#39 operator()<const boost::any&> (a1=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#40
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void,
KrellInstitute::CBTF::Component, std::string
const&, KrellInstitute::CBTF::Type const&,
boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::MRNet*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>,
boost::arg<1> > >, void, boost::any
const&>::invoke (function_obj_ptr=...,
a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#41 0x00007fffa64442a4 in operator() (a0=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#42
KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been
optimized out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#43 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#44 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#45 0x00007fffa643ec48 in
KrellInstitute::CBTF::Component::emitOutput
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#46 0x00007fffa643f308 in
call<KrellInstitute::CBTF::Impl::Network*,
std::basic_string<char> const,
KrellInstitute::CBTF::Type const, boost::any
const> (
ÂÂÂ u=<error reading variable: value has been
optimized out>, b3=..., b2=..., b1=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:384
---Type <return> to continue, or q
<return> to quit---
#47
operator()<KrellInstitute::CBTF::Impl::Network*>
(u=<error reading variable: value has been
optimized out>, a3=..., a2=..., a1=...,
ÂÂÂ this=<error reading variable: value has been
optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:399
#48 operator()<boost::_mfi::mf3<void,
KrellInstitute::CBTF::Component, const
std::basic_string<char>&, const
KrellInstitute::CBTF::Type&, const
boost::any&>, boost::_bi::rrlist1<const
boost::any&> > (a=<synthetic
pointer>, f=<error reading variable: value has
been optimized out>,
ÂÂÂ this=<error reading variable: value has been
optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:463
#49 operator()<const boost::any&> (a1=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#50
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf3<void,
KrellInstitute::CBTF::Component, std::string
const&, KrellInstitute::CBTF::Type const&,
boost::any const&>,
boost::_bi::list4<boost::_bi::value<KrellInstitute::CBTF::Impl::Network*>,
boost::_bi::value<std::string>,
boost::_bi::value<KrellInstitute::CBTF::Type>,
boost::arg<1> > >, void, boost::any
const&>::invoke (function_obj_ptr=...,
a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#51 0x00007fffa64442a4 in operator() (a0=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#52
KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been
optimized out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#53 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#54 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#55 0x00007fffa2bd0fa0 in
emitOutput<boost::shared_ptr<CBTF_Protocol_LinkedObjectGroup>
> (value=..., name="LinkedObjectGroup",
this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Component.hpp:330
#56
StateManagementForCUDA::handleThreadsStateChanged
(this=0x7fff9c02e380, message=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf-argonavis/CUDA/components/StateManagementForCUDA.cpp:291
#57 0x00007fffa2bd1c94 in operator() (a1=...,
p=<optimized out>, this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#58 operator()<boost::_mfi::mf1<void,
StateManagementForCUDA, const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>
> (a=<synthetic pointer>, f=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#59 operator()<const
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>&>
(a1=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#60
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void, StateManagementForCUDA,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
const&>,
boost::_bi::list2<boost::_bi::value<StateManagementForCUDA*>,
boost::arg<1> > >, void,
boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
const&>::invoke (function_obj_ptr=...,
a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#61 0x00007fffa2bd342c in operator() (a0=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#62
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
>::operator() (
ÂÂÂ this=<error reading variable: value has been
optimized out>, value=...) at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#63 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#64 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
---Type <return> to continue, or q
<return> to quit---
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#65 0x00007fffa643ecc8 in emitOutput (value=...,
type=..., name="value", this=0x7fff9c02a700)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:348
#66
KrellInstitute::CBTF::Impl::InputMediator::handler
(this=0x7fff9c02a700, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-xml/InputMediator.hpp:70
#67 0x00007fffa643db94 in operator() (a1=...,
p=<optimized out>, this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#68 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::InputMediator, const
boost::any&>, boost::_bi::rrlist1<const
boost::any&> > (
ÂÂÂ a=<synthetic pointer>, f=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#69 operator()<const boost::any&> (a1=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#70
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::InputMediator,
boost::any const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::Impl::InputMediator*>,
boost::arg<1> > >, void, boost::any
const&>::invoke (function_obj_ptr=...,
a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#71 0x00007fffa64442a4 in operator() (a0=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#72
KrellInstitute::CBTF::Impl::InvokerForAny::operator()
(this=<error reading variable: value has been
optimized out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerForAny.hpp:56
#73 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#74 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#75 0x00007fffa2be0f44 in
emitOutput<boost::shared_ptr<CBTF_Protocol_ThreadsStateChanged>
> (value=<synthetic pointer>, name="out",
this=0x7fff9c0076f0)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/Component.hpp:330
#76
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>::handler
(this=0x7fff9c0076f0, in=...)
ÂÂÂ at
/opt/tools/OSS/cbtf_v2.4.0/include/KrellInstitute/CBTF/XDR.hpp:205
#77 0x00007fffa2bd67a4 in operator() (a1=...,
p=<optimized out>, this=<optimized out>)
at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:165
#78 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
const boost::shared_ptr<MRN::Packet>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> >
(a=<synthetic pointer>, f=...,
this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#79 operator()<const
boost::shared_ptr<MRN::Packet>&>
(a1=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#80
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<KrellInstitute::CBTF::ConvertMRNetToXDR<CBTF_Protocol_ThreadsStateChanged>*>,
boost::arg<1> > >, void,
boost::shared_ptr<MRN::Packet>
const&>::invoke (function_obj_ptr=...,
a0=...)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#81 0x00007fffa63e357c in operator() (a0=...,
this=<error reading variable: value has been
optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#82
KrellInstitute::CBTF::Impl::InvokerFor<boost::shared_ptr<MRN::Packet>
>::operator() (this=<error reading variable:
value has been optimized out>, value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Impl/InvokerFor.hpp:70
#83 0x00007fffa649b2e0 in
KrellInstitute::CBTF::Impl::ComponentImpl::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
---Type <return> to continue, or q
<return> to quit---
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/ComponentImpl.cpp:529
#84 0x00007fffa6498ffc in
KrellInstitute::CBTF::Component::emitOutputImpl
(this=<optimized out>, name=..., type=...,
value=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/Component.cpp:214
#85 0x00007fffa63c9494 in
emitOutput<boost::shared_ptr<MRN::Packet>
> (value=..., name="value", this=0x7fff9c0072d0)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf/KrellInstitute/CBTF/Component.hpp:330
#86
KrellInstitute::CBTF::Impl::IncomingStreamMediator::handler
(this=0x7fff9c0072d0, packet=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/IncomingStreamMediator.cpp:77
#87 0x00007fffa63dafe8 in
call<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>,
boost::shared_ptr<MRN::Packet> const>
(b1=..., u=...,
ÂÂÂ this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:156
#88
operator()<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>
> (a1=..., u=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:171
#89 operator()<boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator,
const boost::shared_ptr<MRN::Packet>&>,
boost::_bi::rrlist1<const
boost::shared_ptr<MRN::Packet>&> >
(a=<synthetic pointer>, f=...,
this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:319
#90 operator()<const
boost::shared_ptr<MRN::Packet>&>
(a1=..., this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1306
#91
boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf1<void,
KrellInstitute::CBTF::Impl::IncomingStreamMediator,
boost::shared_ptr<MRN::Packet> const&>,
boost::_bi::list2<boost::_bi::value<boost::shared_ptr<KrellInstitute::CBTF::Impl::IncomingStreamMediator>
>, boost::arg<1> > >, void,
boost::shared_ptr<MRN::Packet>
const&>::invoke (function_obj_ptr=...,
a0=...) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:159
#92 0x00007fffa63d1460 in operator() (a0=...,
this=0x7fff8802c6a0) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/function/function_template.hpp:769
#93
KrellInstitute::CBTF::Impl::MessageHandlers::operator()
(this=0x7fff9c002a88, tag=@0x7fffa10de690: 1124,
packet=...)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/MessageHandlers.cpp:103
#94 0x00007fffa63bab90 in
KrellInstitute::CBTF::Impl::Frontend::doMessagePump
(this=0x7fff9c002a80)
ÂÂÂ at
/home/jeg/openspeedshop-externals/BUILD/p9-4V100-01/cbtf/libcbtf-mrnet/Frontend.cpp:409
#95 0x00007fffa63bcea4 in operator()
(p=<optimized out>, this=<optimized
out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/mem_fn_template.hpp:49
#96 operator()<boost::_mfi::mf0<void,
KrellInstitute::CBTF::Impl::Frontend>,
boost::_bi::list0> (a=<synthetic pointer>,
f=..., this=<optimized out>)
ÂÂÂ at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:259
#97 operator() (this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/bind/bind.hpp:1294
#98
boost::detail::thread_data<boost::_bi::bind_t<void,
boost::_mfi::mf0<void,
KrellInstitute::CBTF::Impl::Frontend>,
boost::_bi::list1<boost::_bi::value<KrellInstitute::CBTF::Impl::Frontend*>
> > >::run (this=<optimized out>) at
/opt/tools/OSS/krellroot_v2.4.0/include/boost/thread/detail/thread.hpp:117
#99 0x00007fffa73323b0 in thread_proxy () from
/opt/tools/OSS/krellroot_v2.4.0/lib/libboost_thread.so.1.67.0
#100 0x00007fffa66f8af4 in start_thread
(arg=0x7fffa10df170) at pthread_create.c:310
#101 0x00007fffa5e78814 in clone () at
../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:109
(gdb)
(gdb)
_______________________________________________
Dyninst-api mailing list
Dyninst-api@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
|
|
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}