Re: [DynInst_API:] Hung mutator on exit

[Matthew LeGendre <legendre1@xxxxxxxx>]

There's been numerous ProcControlAPI issues in this category--the Linux 
debug interface can be very inconsistent when multithreaded processes 
exit.  The lastest set of ProcControlAPI fixes for this category went into 
the dyninst GIT master branch in February.  It may be worth testing this 
with the latest Dyninst from GIT.
Marc, you can also turn on the environment variable 
DYNINST_DEBUG_PROCCONTROL, run until it hangs, and send Emily/Bill/myself 
the resulting output.  That's usually sufficient for diagnosing these.
-Matt

On Thu, 2 Apr 2015, Emily Gember-Jacobson wrote:
> Hi Marc,
> I haven't been able to reproduce this (8.2 on RHEL 6).  What OS are you on?
>
> Nothing stands out to me as problematic in your mutator. 
>
> Matt or Josh, do either of you have ideas here?
>
> Best,
> Emily
>
> On Thursday, April 2, 2015, Marc Brünink <marc@xxxxxxxxx> wrote:
>       Hi Emily,
>
>       actually I have a hanging mutator every now and then with this very
>       simple mutator/mutatee pair during destruction of the mutator. Using 8.2.
>
>       If you enable the thread in the mutatee, you can get a hanging mutator at
>       a different location.
>
>       All hangs happen on exit.
>
>       Marc
>
>
>
>
>       On 31/03/2015 21:59, Emily Gember-Jacobson wrote:
>             Hi Marc,
>
>             Bill is out of the office this week, so I'll try to look into
>             this for
>             you. Would it be possible for you to share the code to
>             reproduce this hang?
>
>             Thanks,
>             Emily
>
>             On Tue, Mar 31, 2015 at 5:55 AM, Marc Brünink <marc@xxxxxxxxx
>             <mailto:marc@xxxxxxxxx>> wrote:
>
>                 I am struggling with a hung mutator on exit of the
>             mutatee.
>                 It looks like it is related to event ordering.
>
>                 The process hangs if the post-UserThreadDestroy event is
>             queued
>                 before the pre-Exit.
>
>                 --------------
>                 [pcEventMuxer.C/83]: PCEventMuxer waiting for events,
>             non-blocking
>                 pcEventMuxer.C[407]: Begin callbackMux, process pointer =
>                 0x33ae2dc0, event post-UserThreadDestroy
>                 pcEventMuxer.C[439]: Added event post-UserThreadDestroy
>             to mailbox,
>                 size now 1
>                 pcEventMuxer.C[411]: after muxing event, mailbox size is
>             1
>                 pcEventMuxer.C[407]: Begin callbackMux, process pointer =
>                 0x33ae2dc0, event pre-LWPDestroy
>                 pcEventMuxer.C[439]: Added event pre-LWPDestroy to
>             mailbox, size now 2
>                 pcEventMuxer.C[411]: after muxing event, mailbox size is
>             2
>                 [pcEventMuxer.C:86] after PC event handling, 2 events in
>             mailbox
>                 pcEventMuxer.C[464]: Returning event
>             post-UserThreadDestroy from mailbox
>                 pcEventHandler.C[65]: attempting to handle event
>                 post-UserThreadDestroy on thread 4976/4977
>                 pcEventHandler.C[109]: stopping process for event
>             handling
>                 dynProcess.C[1048]: Stopping process 4976
>                 pcEventMuxer.C[231]: Begin callbackMux, process pointer =
>                 0x33ae2dc0, event pre-Exit
>                 [pcEventMuxer.C:232] Exit callback
>                 pcEventMuxer.C[439]: Added event pre-Exit to mailbox,
>             size now 2
>                 pcEventMuxer.C[236]: after muxing event, mailbox size is
>             2
>                 ... hang ...
>                 --------------
>
>
>                 In contrast if the pre-Exit is queued before, everything
>             work fine
>                 because the stopping of the process is skipped.
>
>
>                 --------------
>                 [pcEventMuxer.C/83]: PCEventMuxer waiting for events,
>             non-blocking
>                 [46912496468160]pcEventMuxer.__C[231]: Begin callbackMux,
>             process
>                 pointer = 0x20cde620, event pre-Exit
>                 [pcEventMuxer.C:232] Exit callback
>                 pcEventMuxer.C[439]: Added event pre-Exit to mailbox,
>             size now 1
>                 pcEventMuxer.C[236]: after muxing event, mailbox size is
>             1
>                 pcEventMuxer.C[407]: Begin callbackMux, process pointer =
>                 0x20cde620, event post-UserThreadDestroy
>                 pcEventMuxer.C[439]: Added event post-UserThreadDestroy
>             to mailbox,
>                 size now 2
>                 pcEventMuxer.C[411]: after muxing event, mailbox size is
>             2
>                 pcEventMuxer.C[407]: Begin callbackMux, process pointer =
>                 0x20cde620, event pre-LWPDestroy
>                 pcEventMuxer.C[439]: Added event pre-LWPDestroy to
>             mailbox, size now 3
>                 pcEventMuxer.C[411]: after muxing event, mailbox size is
>             3
>                 pcEventMuxer.C[407]: Begin callbackMux, process pointer =
>                 0x20cde620, event post-LWPDestroy
>                 pcEventMuxer.C[439]: Added event post-LWPDestroy to
>             mailbox, size now 4
>                 pcEventMuxer.C[411]: after muxing event, mailbox size is
>             4
>                 [pcEventMuxer.C:86] after PC event handling, 4 events in
>             mailbox
>                 pcEventMuxer.C[464]: Returning event pre-Exit from
>             mailbox
>                 pcEventHandler.C[65]: attempting to handle event pre-Exit
>             on thread
>                 4952/4952
>                 --------------
>
>
>                 A truncated backtrace:
>                 --------------
>                 #0  pthread_cond_wait@@GLIBC_2.3.2 () at
>                
>             ../nptl/sysdeps/unix/sysv/__linux/x86_64/pthread_cond___wait.S:185
>                 #1  0x00002aaaaee25c38 in wait<Mutex<false> > (m=...,
>                 this=0x2aaab40009b8)
>                      at
>             /usr/include/boost/thread/__pthread/condition_variable.__hpp:184
>                 #2  wait (this=0x2aaab40009b8) at
>                
>             /home/marc/Downloads/__DyninstAPI-8.2/common/src/__dthread.h:121
>                 #3  MailboxMT::dequeue (this=0x2aaab40008c0, block=true)
>                      at
>                
>             /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/mailbox.C:156
>                 #4  0x00002aaaaee4aa01 in
>             int_process::__waitAndHandleEvents
>                 (block=block@entry=false)
>                      at
>                
>             /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/process.C:1045
>                 #5  0x00002aaaaee733af in
>                 Dyninst::ProcControlAPI::__ProcessSet::stopProcs
>             (this=0xca39e0)
>                      at
>                
>             /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/procset.C:1495
>                 #6  0x00002aaaaee39953 in
>                 Dyninst::ProcControlAPI::__Process::stopProc
>             (this=<optimized out>)
>                      at
>                
>             /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/process.C:6406
>                 --------------
>
>
>                 Questions:
>                 1. Is this a known issue in Dyninst 8.2?
>                 2. Any idea what's wrong? It might very well be a problem
>             in my mutator.
>
>
>                 Marc
>                 _________________________________________________
>                 Dyninst-api mailing list
>                 Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxxxx>
>                 https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
>                 <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>