Re: [DynInst_API:] Hung process

[Matthew LeGendre <legendre1@xxxxxxxx>]

On Tue, 3 Feb 2015, Bill Williams wrote:
> On 02/03/2015 11:12 AM, Josh Stone wrote:
> > On 02/03/2015 07:22 AM, Gerard wrote:
> > > Hello,
> > >
> > > I'm experiencing a strange issue while instrumenting a multithreaded
> > > application. In essence this application sends data to other processes
> > > through TCP/IP connections using multiple threads (one per connection).
> > > The problem is that after 39~41 successful connections (always the
> > > same), the whole process hangs and I don't know why this happens or how
> > > to make it to continue.
> > >
> > > Without Dyninst instrumentation the process works correctly so I don't
> > > know if it is a problem with how I'm using Dyninst or a problem that
> > > appears due to using it. If I remove the use of threads the application
> > > never hangs and all the data is sent correctly so I guess it is related.
> > >
> > > Any tips why this could happen?
> > There is a compiled constant MAX_THREADS, just 32.  If anything, I'm
> > surprised that you got up around 40 before seeing issues. :)
> >
> > In dyninstAPI_RT, it's used for DYNINST_max_num_threads, and it looks
> > like there's code to initialize larger counts dynamically, but I don't
> > see anywhere that actually tries to set it higher.  It would have to be
> > done as part of process init though.
> > _______________________________________________
> > Dyninst-api mailing list
> > Dyninst-api@xxxxxxxxxxx
> > https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
> Right now, I'd recommend bumping the compiled constant to a reasonable safe 
> ceiling. For 9.0, we're going to be eliminating MAX_THREADS, as proper 
> TLS-based trampguards make it unnecessary.
> I can put together a patch for that and send it out to the list for testing 
> and feedback soon; I thought I'd already done that but apparently not.
Another possible fix may be to disable tramp guards.  Tramp guards are 
used to prevent recursive instrumentation.  For example, if you instrument 
malloc() with instrumentation that calls malloc(), then tramp guards will 
prevent you from going into infinite recursion.
If you already know that your instrumentation can't infinitely recurse,
then disabling tramp guards will give a big performance win and may work 
around this hang.  To disable tramp guards, put a call to 
BPatch::setTrampRecursive(true) before you insert instrumentation.
-Matt