Re: [DynInst_API:] Hung process

[Gerard <nouboh@xxxxxxxxx>]

Then let's hope that debug logs will tell us where is the problem!

I'm using Gentoo Linux with kernel 3.18.3-gentoo.

Gerard

2015-02-18 20:42 GMT+01:00 Bill Williams <bill@xxxxxxxxxxx>:

On 02/18/2015 01:37 PM, Gerard wrote:

Ah ok, I didn't know that.

About how reproducible is the error, I run it three times (without the
change you suggested) and every time stopped at around 32000 threads.
Now I added appProc->continueExecution() and it happened again after
creating 32322 threads, so it seems this is not the problem.

Then it's got to be that somewhere in here, we're messing up internal stop/continue state without that propagating out to the user level. Debug logs will tell me something eventually...sadly, they're verbose and time-consuming.

Which kernel version/distribution are you using, by the way?

Gerard

2015-02-18 20:29 GMT+01:00 Bill Williams <bill@xxxxxxxxxxx
<mailto:bill@xxxxxxxxxxx>>:

  On 02/18/2015 01:00 PM, Gerard wrote:

    I'm not sure at which point it hangs, but I have set up an
    example that
    I think mimics what happens with my application. In this example the
    mutatee hangs after creating 32294 threads. I'm using the latest
    master
    version.

  Okay, so there's at least one obvious problem here (and this is at
  least in part the fault of our manuals): if the mutatee ever becomes
  stopped, we won't continue it. The standard logic should be
  something more like:

  do
  {
   ÂappProc->continueExecution();
   Âbpatch.waitForStatusChange();
  } while(!appProc->isTerminated()__);

  assuming that you're starting from a stopped state (e.g. create or
  attach and then some instrumentation). While it's about 99% safe to
  assume that if your mutator isn't stopping a process, it won't be
  stopped during its execution, it's not actually guaranteed, and
  waitForStatusChange will return whenever there exists a process
  that's become stopped or terminated.

  If there's no third-party source of SIGSTOP (and there shouldn't
  be), there's still something weird (if rare) going on. I'm running
  with debug logs (DYNINST_DEBUG_PROCCONTROL), which will take some
  time (and some disk) but should tell me whether the above change is
  a sufficient fix for you.

  --bw

    Thanks,

    Gerard

    2015-02-16 18:51 GMT+01:00 Bill Williams <bill@xxxxxxxxxxx
    <mailto:bill@xxxxxxxxxxx>
    <mailto:bill@xxxxxxxxxxx <mailto:bill@xxxxxxxxxxx>>>:

      ÂOn 02/16/2015 10:35 AM, Gerard wrote:

        ÂHi,

        ÂI'm having another problem. I increased the MAX_THREADS
    variable to
        Â10240 because I need to instrument a mutatee that
    creates around
        Â10000
        Âthreads. After increasing the variable I could
    instrument the
        Âmutatee
        Âwithout problems, but I added more snippets and the
    process hung
        Âagain,
        Âthis time after creating 3999 threads. Now, it doesn't
    matter if I
        Âincrease even more the constant, it still hangs.

        ÂIs there any other reason why the mutatee could hang or
    is there any
        Âother limit somewhere?

      ÂI'm not aware of other hardcoded limits, and if you already had
      Âtramp guards disabled, it seems likely that you're running
    into some
      Âform of bad behavior at scale that's independent of
    MAX_THREADS.

      ÂIs it possible for you to put together a simple reproducer
      Â(mutator/mutatee) and send that to me? Do you know whether the
      Âprocess is hanging at a particular point (thread
      Âcreation/destruction, process exit)?

        ÂThanks,

        ÂGerard

        Â2015-02-04 11:22 GMT+01:00 Gerard <nouboh@xxxxxxxxx
    <mailto:nouboh@xxxxxxxxx>
        Â<mailto:nouboh@xxxxxxxxx <mailto:nouboh@xxxxxxxxx>>
        Â<mailto:nouboh@xxxxxxxxx <mailto:nouboh@xxxxxxxxx>
    <mailto:nouboh@xxxxxxxxx <mailto:nouboh@xxxxxxxxx>>>>:

           Thanks! That was exactly the problem. I have
    increased the
        Âconstant
           MAX_THREADS and now I don't have this problem
    anymore. Is
        Âthere any
           reason why the constant is set to 32 instead of a
    higher value?

           I also have tried to find how to dynamically change
           the DYNINST_max_num_threads value but I haven't
    found where
        Âis it
           implemented (version 8.2.1). And about disabling tramp
        Âguards, I
           already had them disabled so it seems that this
    workaround
        Âdoes not
           solve the problem.

           Gerard

           2015-02-03 20:13 GMT+01:00 Barton Miller
    <bart@xxxxxxxxxxx <mailto:bart@xxxxxxxxxxx>
        Â<mailto:bart@xxxxxxxxxxx <mailto:bart@xxxxxxxxxxx>>
           <mailto:bart@xxxxxxxxxxx <mailto:bart@xxxxxxxxxxx>
    <mailto:bart@xxxxxxxxxxx <mailto:bart@xxxxxxxxxxx>>>>:

             Disabling tramp guards certainly works if you
    really
        Âknow that
             you're not recursing. That can be subtle and
    error
        Âprone, which
             is why tramp guards were invented. Proceed
    cautiously
        Âhere.

             --bart


             On 2/3/2015 12:02 PM, Matthew LeGendre wrote:

               Another possible fix may be to disable tramp
        Âguards. Tramp
               guards are used to prevent recursive
        Âinstrumentation. For
               example, if you instrument malloc() with
        Âinstrumentation
               that calls malloc(), then tramp guards
    will prevent
        Âyou from
               going into infinite recursion.

               If you already know that your
    instrumentation can't
               infinitely recurse,
               then disabling tramp guards will give a big
        Âperformance win
               and may work around this hang. To disable
    tramp
        Âguards, put
               a call to

    BPatch::setTrampRecursive(______true)
        Âbefore you
               insert instrumentation.

               -Matt

     _____________________________________________________
               Dyninst-api mailing list
    Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxedu>
    <mailto:Dyninst-api@xxxxxxxx__edu <mailto:Dyninst-api@xxxxxxxxedu>>
        Â<mailto:Dyninst-api@xxxxxxx.
    <mailto:Dyninst-api@xxxxxxx.>____edu
    <mailto:Dyninst-api@xxxxxxxx__edu <mailto:Dyninst-api@xxxxxxxxedu>>>
    https://lists.cs.wisc.edu/______mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api>

    <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>>


    <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>

    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>>>



     _____________________________________________________
             Dyninst-api mailing list
    Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxedu>
    <mailto:Dyninst-api@xxxxxxxx__edu <mailto:Dyninst-api@xxxxxxxxedu>>
        Â<mailto:Dyninst-api@xxxxxxx.
    <mailto:Dyninst-api@xxxxxxx.>____edu
    <mailto:Dyninst-api@xxxxxxxx__edu <mailto:Dyninst-api@xxxxxxxxedu>>>
    https://lists.cs.wisc.edu/______mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api>

    <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>>


    <https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>

    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>>>





        Â___________________________________________________
        ÂDyninst-api mailing list
    Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxedu>
    <mailto:Dyninst-api@xxxxxxxx__edu <mailto:Dyninst-api@xxxxxxxxedu>>
    https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>

    <https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
    <https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>>



      Â--
      Â--bw

      ÂBill Williams
      ÂParadyn Project
    bill@xxxxxxxxxxx <mailto:bill@xxxxxxxxxxx>
    <mailto:bill@xxxxxxxxxxx <mailto:bill@xxxxxxxxxxx>>




  --
  --bw

  Bill Williams
  Paradyn Project
  bill@xxxxxxxxxxx <mailto:bill@xxxxxxxxxxx>

--
--bw

Bill Williams
Paradyn Project
bill@xxxxxxxxxxx