Re: [DynInst_API:] Hung process

[Gerard <nouboh@xxxxxxxxx>]

Ah ok, I didn't know that.

About how reproducible is the error, I run it three times (without the change you suggested) and every time stopped at around 32000 threads. Now I addedÂappProc->continueExecution() and it happened again after creatingÂ32322 threads, so it seems this is not the problem.

Gerard

2015-02-18 20:29 GMT+01:00 Bill Williams <bill@xxxxxxxxxxx>:

On 02/18/2015 01:00 PM, Gerard wrote:

I'm not sure at which point it hangs, but I have set up an example that
I think mimics what happens with my application. In this example the
mutatee hangs after creating 32294 threads. I'm using the latest master
version.

Okay, so there's at least one obvious problem here (and this is at least in part the fault of our manuals): if the mutatee ever becomes stopped, we won't continue it. The standard logic should be something more like:

do
{
 appProc->continueExecution();
 bpatch.waitForStatusChange();
} while(!appProc->isTerminated());

assuming that you're starting from a stopped state (e.g. create or attach and then some instrumentation). While it's about 99% safe to assume that if your mutator isn't stopping a process, it won't be stopped during its execution, it's not actually guaranteed, and waitForStatusChange will return whenever there exists a process that's become stopped or terminated.

If there's no third-party source of SIGSTOP (and there shouldn't be), there's still something weird (if rare) going on. I'm running with debug logs (DYNINST_DEBUG_PROCCONTROL), which will take some time (and some disk) but should tell me whether the above change is a sufficient fix for you.

--bw

Thanks,

Gerard

2015-02-16 18:51 GMT+01:00 Bill Williams <bill@xxxxxxxxxxx
<mailto:bill@xxxxxxxxxxx>>:

  On 02/16/2015 10:35 AM, Gerard wrote:

    Hi,

    I'm having another problem. I increased the MAX_THREADS variable to
    10240 because I need to instrument a mutatee that creates around
    10000
    threads. After increasing the variable I could instrument the
    mutatee
    without problems, but I added more snippets and the process hung
    again,
    this time after creating 3999 threads. Now, it doesn't matter if I
    increase even more the constant, it still hangs.

    Is there any other reason why the mutatee could hang or is there any
    other limit somewhere?

  I'm not aware of other hardcoded limits, and if you already had
  tramp guards disabled, it seems likely that you're running into some
  form of bad behavior at scale that's independent of MAX_THREADS.

  Is it possible for you to put together a simple reproducer
  (mutator/mutatee) and send that to me? Do you know whether the
  process is hanging at a particular point (thread
  creation/destruction, process exit)?

    Thanks,

    Gerard

    2015-02-04 11:22 GMT+01:00 Gerard <nouboh@xxxxxxxxx
    <mailto:nouboh@xxxxxxxxx>
    <mailto:nouboh@xxxxxxxxx <mailto:nouboh@xxxxxxxxx>>>:

      ÂThanks! That was exactly the problem. I have increased the
    constant
      ÂMAX_THREADS and now I don't have this problem anymore. Is
    there any
      Âreason why the constant is set to 32 instead of a higher value?

      ÂI also have tried to find how to dynamically change
      Âthe DYNINST_max_num_threads value but I haven't found where
    is it
      Âimplemented (version 8.2.1). And about disabling tramp
    guards, I
      Âalready had them disabled so it seems that this workaround
    does not
      Âsolve the problem.

      ÂGerard

      Â2015-02-03 20:13 GMT+01:00 Barton Miller <bart@xxxxxxxxxxx
    <mailto:bart@xxxxxxxxxxx>
      Â<mailto:bart@xxxxxxxxxxx <mailto:bart@xxxxxxxxxxx>>>:

        ÂDisabling tramp guards certainly works if you really
    know that
        Âyou're not recursing. That can be subtle and error
    prone, which
        Âis why tramp guards were invented. Proceed cautiously
    here.

        Â--bart


        ÂOn 2/3/2015 12:02 PM, Matthew LeGendre wrote:

          ÂAnother possible fix may be to disable tramp
    guards. Tramp
          Âguards are used to prevent recursive
    instrumentation. For
          Âexample, if you instrument malloc() with
    instrumentation
          Âthat calls malloc(), then tramp guards will prevent
    you from
          Âgoing into infinite recursion.

          ÂIf you already know that your instrumentation can't
          Âinfinitely recurse,
          Âthen disabling tramp guards will give a big
    performance win
          Âand may work around this hang. To disable tramp
    guards, put
          Âa call to BPatch::setTrampRecursive(____true)
    before you
          Âinsert instrumentation.

          Â-Matt
          Â___________________________________________________
          ÂDyninst-api mailing list
    Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxedu>
    <mailto:Dyninst-api@xxxxxxxx__edu <mailto:Dyninst-api@xxxxxxxxedu>>
    https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>

    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>>


        Â___________________________________________________
        ÂDyninst-api mailing list
    Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxedu>
    <mailto:Dyninst-api@xxxxxxxx__edu <mailto:Dyninst-api@xxxxxxxxedu>>
    https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>

    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>>





    _________________________________________________
    Dyninst-api mailing list
    Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxedu>
    https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>



  --
  --bw

  Bill Williams
  Paradyn Project
  bill@xxxxxxxxxxx <mailto:bill@xxxxxxxxxxx>

--
--bw

Bill Williams
Paradyn Project
bill@xxxxxxxxxxx