Ahhhh, that explains the following and how to get around it. ThankÂyou.
WARNING: Skipping DWARF for gfs_model.x, over threshold (377978416 > 104857600)
On Mon, May 12, 2025 at 10:13âAM John Mellor-Crummey < johnmc@xxxxxxxx> wrote: Daniel,
One more thing:
While we work on resolving the issue with hpcstruct, you should be able to run hpcprof on your measurement data even if hpcstruct failed to analyze this binary. hpcprof includes the ability to read DWARF (using a different library that shouldnât crash).
When you run hpcprof, you should use
hpcprofÂ--dwarf-max-size=unlimitedÂ<measurement directory>
Best,
John -- John Mellor-Crummey    ÂProfessor Dept of Computer Science ÂÂRice University email: johnmc@xxxxxxxx  ÂÂphone: 713-348-5179
Got permission to share the executable. Link sent. I'll ask about providing the executable.
On Fri, May 9, 2025 at 1:52âPM John Mellor-Crummey < johnmc@xxxxxxxx> wrote: Hi Daniel,
Thanks for the callstack.
The problem seems to be exactly the same one recently encountered by Doug Pase for a Fortran program at Sandia. This is a problem inside the type processing by the Dyninst software written by our collaborators.Â
Can you share a binary with us to facilitate debugging? The Sandia binary is export controlled and only accessible inside their firewall. Having a non-export controlled binary for debugging would make our lives easier.
Best,
John -- John Mellor-Crummey    ÂProfessor Dept of Computer Science ÂÂRice University email: johnmc@xxxxxxxx  ÂÂphone: 713-348-5179
The application is compiledÂwith Intel ifort. HPCToolkit and its dependenciesÂare compiled with gcc-13.2.1. I attached the spec for HPCToolkit.
(gdb) run --nocache /lfs/h1/hpc/support/daniel.kokron/Tickets/2025042910000034/sorc/ufs_model.fd/build_fv3_1/gfs_model.x
Starting program: /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/hpctoolkit-2024.01.1-a3im66mlumyu3hbzmeuor3kj3l553yau/bin/hpcstruct --nocache /lfs/h1/hpc/support/daniel.kokron/Tickets/2025042910000034/sorc/ufs_model.fd/build_fv3_1/gfs_model.x
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.31-150300.63.1.x86_64
Missing separate debuginfo for /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libstdc++.so.6
Try: zypper install -C "debuginfo(build-id)=c74eca671e2dd0f063706372d103f8acef88f1e3"
Missing separate debuginfo for /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libgomp.so.1
Try: zypper install -C "debuginfo(build-id)=54684492738e640bcd600e830cee025dd8771a20"
Missing separate debuginfo for /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libgcc_s.so.1
Try: zypper install -C "debuginfo(build-id)=12f775ec4aeb94b749897b1b65638f18b61d1b1f"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Âbegin sequential analysis of CPU binary /lfs/h1/hpc/support/daniel.kokron/Tickets/2025042910000034/sorc/ufs_model.fd/build_fv3_1/gfs_model.x (size = 377978672, threads = 1)
hpcstruct: /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/boost-1.87.0-2cldxfpwec5rbbhxutja5lwcgzh6fbhc/include/boost/smart_ptr/shared_ptr.hpp:550: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = Dyninst::SymtabAPI::typeSubrange; typename boost::detail::sp_member_access<T>::type = Dyninst::SymtabAPI::typeSubrange*]: Assertion `px != 0' failed.
Program received signal SIGABRT, Aborted.
0x0000155553e2fd2b in raise () from /lib64/libc.so.6
(gdb) where
#0 Â0x0000155553e2fd2b in raise () from /lib64/libc.so.6
#1 Â0x0000155553e313e5 in abort () from /lib64/libc.so.6
#2 Â0x0000155553e27c6a in __assert_fail_base () from /lib64/libc.so.6
#3 Â0x0000155553e27cf2 in __assert_fail () from /lib64/libc.so.6
#4 Â0x0000155554d65127 in boost::enable_if<boost::integral_constant<bool, !((bool)boost::is_same<Dyninst::SymtabAPI::Type, Dyninst::SymtabAPI::typeSubrange>::value)>, boost::shared_ptr<Dyninst::SymtabAPI::Type> >::type Dyninst::SymtabAPI::typeCollection::addOrUpdateType<Dyninst::SymtabAPI::typeSubrange>(boost::shared_ptr<Dyninst::SymtabAPI::typeSubrange>) () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#5 Â0x0000155554d547e6 in Dyninst::SymtabAPI::DwarfWalker::parseSubrange() () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#6 Â0x0000155554d5a0a8 in Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#7 Â0x0000155554d5b235 in Dyninst::SymtabAPI::DwarfWalker::findAnyType(Dwarf_Attribute, bool, boost::shared_ptr<Dyninst::SymtabAPI::Type>&) ()
 Âfrom /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#8 Â0x0000155554d5b732 in Dyninst::SymtabAPI::DwarfWalker::findType(boost::shared_ptr<Dyninst::SymtabAPI::Type>&, bool) ()
 Âfrom /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#9 Â0x0000155554d5497b in Dyninst::SymtabAPI::DwarfWalker::parseArray() () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#10 0x0000155554d59fb8 in Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#11 0x0000155554d5a53b in Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#12 0x0000155554d5bab0 in Dyninst::SymtabAPI::DwarfWalker::parseModule(Dwarf_Die, Dyninst::SymtabAPI::Module*&) [clone .constprop.0] [clone .isra.0] ()
 Âfrom /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#13 0x0000155554d5c15c in Dyninst::SymtabAPI::DwarfWalker::parse() [clone ._omp_fn.0] () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#14 0x000015555403b306 in GOMP_parallel () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libgomp.so.1
#15 0x0000155554d5d2ed in Dyninst::SymtabAPI::DwarfWalker::parse() () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#16 0x0000155554d0a0c1 in Dyninst::SymtabAPI::Object::parseTypeInfo() () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#17 0x0000155554cd48a7 in Dyninst::SymtabAPI::Symtab::parseTypes() () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#18 0x0000155553fef5d7 in __pthread_once_slow () from /lib64/libpthread.so.0
#19 0x0000155554ccc8b4 in Dyninst::SymtabAPI::Symtab::parseTypesNow() () from /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0
#20 0x00000000004418c8 in Inline::openSymtab (elfFile=elfFile@entry=0x8d94b0) at Struct-Inline.cpp:132
#21 0x000000000043cb31 in BAnal::Struct::makeStructure (filename=..., outFile=outFile@entry=0x830ab0, gapsFile=gapsFile@entry=0x0, gaps_filenm=..., search_path=..., structOpts=...) at Struct.cpp:770
#22 0x000000000042cb42 in doSingleBinary (args=..., sb=sb@entry=0x7ffffffd8740) at /usr/include/c++/13/bits/basic_string.tcc:238
#23 0x0000000000412cfd in realmain (argc=<optimized out>, argv=<optimized out>) at main.cpp:209
#24 0x000000000041220a in main (argc=<optimized out>, argv=<optimized out>) at main.cpp:137 On Fri, May 9, 2025 at 11:16âAM John Mellor-Crummey < johnmc@xxxxxxxx> wrote: Hi Daniel,
You should be able to run hpcstruct under gdb and then run it directly on the offending binary as follows
gdb `which hpcstruct` run --nocache /path/to/gfs_model
Then, you can send us a call path. By any chance is this a Fortran code compiled with gfortran? We are presently looking into a complaint about that from Sandia.
Best,
John -- John Mellor-Crummey    ÂProfessor Dept of Computer Science ÂÂRice University email: johnmc@xxxxxxxx  ÂÂphone: 713-348-5179
I am encountering the following error while running hpcstruct. I cannot find the core file in any of the usual places. I have also tried running hpcstruct under gdb without getting very far.
Wondering what my debugging options are?
Âbegin concurrent analysis of CPU binary gfs_model. (size = 377978416, threads = 1)
/bin/sh: line 32: 63480 Aborted         (core dumped) /spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/hpctoolkit-2024.01.1-a3im66mlumyu3hbzmeuor3kj3l553yau/bin/hpcstruct.Â--nocache -j 1 -o $struct_name -M $meas_dir /Baseline_6Hr_WithWW3Restarts_Trace.16774.rawdata/cpubins/model.x > $warn_name 2>&1
Dan
_______________________________________________ HPCToolkit-forum mailing list HPCToolkit-forum@xxxxxxxxxxxxxxxxhttps://mailman.rice.edu/mailman/listinfo/hpctoolkit-forum
|
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}