[DynInst_API:] Hung mutator on exit

[Marc BrÃnink <marc@xxxxxxxxx>]

I am struggling with a hung mutator on exit of the mutatee.
It looks like it is related to event ordering.

The process hangs if the post-UserThreadDestroy event is queued before 
the pre-Exit.
--------------
[pcEventMuxer.C/83]: PCEventMuxer waiting for events, non-blocking
pcEventMuxer.C[407]: Begin callbackMux, process pointer = 0x33ae2dc0, 
event post-UserThreadDestroy
pcEventMuxer.C[439]: Added event post-UserThreadDestroy to mailbox, size 
now 1
pcEventMuxer.C[411]: after muxing event, mailbox size is 1
pcEventMuxer.C[407]: Begin callbackMux, process pointer = 0x33ae2dc0, 
event pre-LWPDestroy
pcEventMuxer.C[439]: Added event pre-LWPDestroy to mailbox, size now 2
pcEventMuxer.C[411]: after muxing event, mailbox size is 2
[pcEventMuxer.C:86] after PC event handling, 2 events in mailbox
pcEventMuxer.C[464]: Returning event post-UserThreadDestroy from mailbox
pcEventHandler.C[65]: attempting to handle event post-UserThreadDestroy 
on thread 4976/4977
pcEventHandler.C[109]: stopping process for event handling
dynProcess.C[1048]: Stopping process 4976
pcEventMuxer.C[231]: Begin callbackMux, process pointer = 0x33ae2dc0, 
event pre-Exit
[pcEventMuxer.C:232] Exit callback
pcEventMuxer.C[439]: Added event pre-Exit to mailbox, size now 2
pcEventMuxer.C[236]: after muxing event, mailbox size is 2
... hang ...
--------------


In contrast if the pre-Exit is queued before, everything work fine 
because the stopping of the process is skipped.

--------------
[pcEventMuxer.C/83]: PCEventMuxer waiting for events, non-blocking
[46912496468160]pcEventMuxer.C[231]: Begin callbackMux, process pointer 
= 0x20cde620, event pre-Exit
[pcEventMuxer.C:232] Exit callback
pcEventMuxer.C[439]: Added event pre-Exit to mailbox, size now 1
pcEventMuxer.C[236]: after muxing event, mailbox size is 1
pcEventMuxer.C[407]: Begin callbackMux, process pointer = 0x20cde620, 
event post-UserThreadDestroy
pcEventMuxer.C[439]: Added event post-UserThreadDestroy to mailbox, size 
now 2
pcEventMuxer.C[411]: after muxing event, mailbox size is 2
pcEventMuxer.C[407]: Begin callbackMux, process pointer = 0x20cde620, 
event pre-LWPDestroy
pcEventMuxer.C[439]: Added event pre-LWPDestroy to mailbox, size now 3
pcEventMuxer.C[411]: after muxing event, mailbox size is 3
pcEventMuxer.C[407]: Begin callbackMux, process pointer = 0x20cde620, 
event post-LWPDestroy
pcEventMuxer.C[439]: Added event post-LWPDestroy to mailbox, size now 4
pcEventMuxer.C[411]: after muxing event, mailbox size is 4
[pcEventMuxer.C:86] after PC event handling, 4 events in mailbox
pcEventMuxer.C[464]: Returning event pre-Exit from mailbox
pcEventHandler.C[65]: attempting to handle event pre-Exit on thread 
4952/4952
--------------


A truncated backtrace:
--------------
#0  pthread_cond_wait@@GLIBC_2.3.2 () at 
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaaee25c38 in wait<Mutex<false> > (m=..., this=0x2aaab40009b8)
    at /usr/include/boost/thread/pthread/condition_variable.hpp:184
#2  wait (this=0x2aaab40009b8) at 
/home/marc/Downloads/DyninstAPI-8.2/common/src/dthread.h:121
#3  MailboxMT::dequeue (this=0x2aaab40008c0, block=true)
    at /home/marc/Downloads/DyninstAPI-8.2/proccontrol/src/mailbox.C:156
#4  0x00002aaaaee4aa01 in int_process::waitAndHandleEvents 
(block=block@entry=false)
    at /home/marc/Downloads/DyninstAPI-8.2/proccontrol/src/process.C:1045
#5  0x00002aaaaee733af in Dyninst::ProcControlAPI::ProcessSet::stopProcs 
(this=0xca39e0)
    at /home/marc/Downloads/DyninstAPI-8.2/proccontrol/src/procset.C:1495
#6  0x00002aaaaee39953 in Dyninst::ProcControlAPI::Process::stopProc 
(this=<optimized out>)
    at /home/marc/Downloads/DyninstAPI-8.2/proccontrol/src/process.C:6406
--------------


Questions:
1. Is this a known issue in Dyninst 8.2?
2. Any idea what's wrong? It might very well be a problem in my mutator.


Marc