Re: [DynInst_API:] Hung mutator on exit

[Marc BrÃnink <marc@xxxxxxxxx>]

Hi Emily,

actually I have a hanging mutator every now and then with this very 
simple mutator/mutatee pair during destruction of the mutator. Using 8.2.
If you enable the thread in the mutatee, you can get a hanging mutator 
at a different location.
All hangs happen on exit.

Marc




On 31/03/2015 21:59, Emily Gember-Jacobson wrote:
> Hi Marc,
>
> Bill is out of the office this week, so I'll try to look into this for
> you. Would it be possible for you to share the code to reproduce this hang?
>
> Thanks,
> Emily
>
> On Tue, Mar 31, 2015 at 5:55 AM, Marc BrÃnink <marc@xxxxxxxxx
> <mailto:marc@xxxxxxxxx>> wrote:
>
>     I am struggling with a hung mutator on exit of the mutatee.
>     It looks like it is related to event ordering.
>
>     The process hangs if the post-UserThreadDestroy event is queued
>     before the pre-Exit.
>
>     --------------
>     [pcEventMuxer.C/83]: PCEventMuxer waiting for events, non-blocking
>     pcEventMuxer.C[407]: Begin callbackMux, process pointer =
>     0x33ae2dc0, event post-UserThreadDestroy
>     pcEventMuxer.C[439]: Added event post-UserThreadDestroy to mailbox,
>     size now 1
>     pcEventMuxer.C[411]: after muxing event, mailbox size is 1
>     pcEventMuxer.C[407]: Begin callbackMux, process pointer =
>     0x33ae2dc0, event pre-LWPDestroy
>     pcEventMuxer.C[439]: Added event pre-LWPDestroy to mailbox, size now 2
>     pcEventMuxer.C[411]: after muxing event, mailbox size is 2
>     [pcEventMuxer.C:86] after PC event handling, 2 events in mailbox
>     pcEventMuxer.C[464]: Returning event post-UserThreadDestroy from mailbox
>     pcEventHandler.C[65]: attempting to handle event
>     post-UserThreadDestroy on thread 4976/4977
>     pcEventHandler.C[109]: stopping process for event handling
>     dynProcess.C[1048]: Stopping process 4976
>     pcEventMuxer.C[231]: Begin callbackMux, process pointer =
>     0x33ae2dc0, event pre-Exit
>     [pcEventMuxer.C:232] Exit callback
>     pcEventMuxer.C[439]: Added event pre-Exit to mailbox, size now 2
>     pcEventMuxer.C[236]: after muxing event, mailbox size is 2
>     ... hang ...
>     --------------
>
>
>     In contrast if the pre-Exit is queued before, everything work fine
>     because the stopping of the process is skipped.
>
>
>     --------------
>     [pcEventMuxer.C/83]: PCEventMuxer waiting for events, non-blocking
>     [46912496468160]pcEventMuxer.__C[231]: Begin callbackMux, process
>     pointer = 0x20cde620, event pre-Exit
>     [pcEventMuxer.C:232] Exit callback
>     pcEventMuxer.C[439]: Added event pre-Exit to mailbox, size now 1
>     pcEventMuxer.C[236]: after muxing event, mailbox size is 1
>     pcEventMuxer.C[407]: Begin callbackMux, process pointer =
>     0x20cde620, event post-UserThreadDestroy
>     pcEventMuxer.C[439]: Added event post-UserThreadDestroy to mailbox,
>     size now 2
>     pcEventMuxer.C[411]: after muxing event, mailbox size is 2
>     pcEventMuxer.C[407]: Begin callbackMux, process pointer =
>     0x20cde620, event pre-LWPDestroy
>     pcEventMuxer.C[439]: Added event pre-LWPDestroy to mailbox, size now 3
>     pcEventMuxer.C[411]: after muxing event, mailbox size is 3
>     pcEventMuxer.C[407]: Begin callbackMux, process pointer =
>     0x20cde620, event post-LWPDestroy
>     pcEventMuxer.C[439]: Added event post-LWPDestroy to mailbox, size now 4
>     pcEventMuxer.C[411]: after muxing event, mailbox size is 4
>     [pcEventMuxer.C:86] after PC event handling, 4 events in mailbox
>     pcEventMuxer.C[464]: Returning event pre-Exit from mailbox
>     pcEventHandler.C[65]: attempting to handle event pre-Exit on thread
>     4952/4952
>     --------------
>
>
>     A truncated backtrace:
>     --------------
>     #0  pthread_cond_wait@@GLIBC_2.3.2 () at
>     ../nptl/sysdeps/unix/sysv/__linux/x86_64/pthread_cond___wait.S:185
>     #1  0x00002aaaaee25c38 in wait<Mutex<false> > (m=...,
>     this=0x2aaab40009b8)
>          at /usr/include/boost/thread/__pthread/condition_variable.__hpp:184
>     #2  wait (this=0x2aaab40009b8) at
>     /home/marc/Downloads/__DyninstAPI-8.2/common/src/__dthread.h:121
>     #3  MailboxMT::dequeue (this=0x2aaab40008c0, block=true)
>          at
>     /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/mailbox.C:156
>     #4  0x00002aaaaee4aa01 in int_process::__waitAndHandleEvents
>     (block=block@entry=false)
>          at
>     /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/process.C:1045
>     #5  0x00002aaaaee733af in
>     Dyninst::ProcControlAPI::__ProcessSet::stopProcs (this=0xca39e0)
>          at
>     /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/procset.C:1495
>     #6  0x00002aaaaee39953 in
>     Dyninst::ProcControlAPI::__Process::stopProc (this=<optimized out>)
>          at
>     /home/marc/Downloads/__DyninstAPI-8.2/proccontrol/__src/process.C:6406
>     --------------
>
>
>     Questions:
>     1. Is this a known issue in Dyninst 8.2?
>     2. Any idea what's wrong? It might very well be a problem in my mutator.
>
>
>     Marc
>     _________________________________________________
>     Dyninst-api mailing list
>     Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxxxx>
>     https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
>     <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>

#include <stdio.h>
#include <pthread.h>
#include <signal.h>

void* thread_do_work(void* arg) {
  while(1);
  return NULL;
}

int main() {
  int i=0;
  pthread_t thread;
  int ret;

  for (i=0; i< 5; ++i)
    sleep(1);
return 0;
  ret = pthread_create(&thread, NULL, thread_do_work, (void*) NULL);
  if (0 != ret) {
    fprintf(stderr, "[FATAL] Could not create thread");
    return -1;
  }

//pthread_kill(thread, SIGKILL);

  return 0;
}

#include "BPatch.h"
#include "BPatch_addressSpace.h"
#include "BPatch_process.h"
#include "BPatch_binaryEdit.h"
#include "BPatch_function.h"
#include "BPatch_point.h"
#include "BPatch_flowGraph.h"


BPatch bpatch;

int main(int argc, char const *argv[])
{
	bpatch.setTrampRecursive(true);
	bpatch.setSaveFPR(false);

	char const *mutatee_argv[] = {
		"mutatee",
		NULL
	};
	BPatch_addressSpace *app = bpatch.processCreate("./mutatee", mutatee_argv);
	
	BPatch_process *appProc = dynamic_cast<BPatch_process *>(app);

	do {

		appProc->continueExecution();
		bpatch.waitForStatusChange();
	} while(!appProc->isTerminated());


	return 0;
}