On 02/18/2015 01:37 PM, Gerard wrote:
Ah ok, I didn't know that.
About how reproducible is the error, I run it three times (without the
change you suggested) and every time stopped at around 32000 threads.
Now I added appProc->continueExecution() and it happened again after
creating 32322 threads, so it seems this is not the problem.
Then it's got to be that somewhere in here, we're messing up internal
stop/continue state without that propagating out to the user level.
Debug logs will tell me something eventually...sadly, they're verbose
and time-consuming.
Which kernel version/distribution are you using, by the way?
Gerard
2015-02-18 20:29 GMT+01:00 Bill Williams <bill@xxxxxxxxxxx
<mailto:bill@xxxxxxxxxxx>>:
On 02/18/2015 01:00 PM, Gerard wrote:
I'm not sure at which point it hangs, but I have set up an
example that
I think mimics what happens with my application. In this example the
mutatee hangs after creating 32294 threads. I'm using the latest
master
version.
Okay, so there's at least one obvious problem here (and this is at
least in part the fault of our manuals): if the mutatee ever becomes
stopped, we won't continue it. The standard logic should be
something more like:
do
{
appProc->continueExecution();
bpatch.waitForStatusChange();
} while(!appProc->isTerminated()__);
assuming that you're starting from a stopped state (e.g. create or
attach and then some instrumentation). While it's about 99% safe to
assume that if your mutator isn't stopping a process, it won't be
stopped during its execution, it's not actually guaranteed, and
waitForStatusChange will return whenever there exists a process
that's become stopped or terminated.
If there's no third-party source of SIGSTOP (and there shouldn't
be), there's still something weird (if rare) going on. I'm running
with debug logs (DYNINST_DEBUG_PROCCONTROL), which will take some
time (and some disk) but should tell me whether the above change is
a sufficient fix for you.
--bw
Thanks,
Gerard
2015-02-16 18:51 GMT+01:00 Bill Williams <bill@xxxxxxxxxxx
<mailto:bill@xxxxxxxxxxx>
<mailto:bill@xxxxxxxxxxx <mailto:bill@xxxxxxxxxxx>>>:
On 02/16/2015 10:35 AM, Gerard wrote:
Hi,
I'm having another problem. I increased the MAX_THREADS
variable to
10240 because I need to instrument a mutatee that
creates around
10000
threads. After increasing the variable I could
instrument the
mutatee
without problems, but I added more snippets and the
process hung
again,
this time after creating 3999 threads. Now, it doesn't
matter if I
increase even more the constant, it still hangs.
Is there any other reason why the mutatee could hang or
is there any
other limit somewhere?
I'm not aware of other hardcoded limits, and if you already had
tramp guards disabled, it seems likely that you're running
into some
form of bad behavior at scale that's independent of
MAX_THREADS.
Is it possible for you to put together a simple reproducer
(mutator/mutatee) and send that to me? Do you know whether the
process is hanging at a particular point (thread
creation/destruction, process exit)?
Thanks,
Gerard
2015-02-04 11:22 GMT+01:00 Gerard <nouboh@xxxxxxxxx
<mailto:nouboh@xxxxxxxxx>
<mailto:nouboh@xxxxxxxxx <mailto:nouboh@xxxxxxxxx>>
<mailto:nouboh@xxxxxxxxx <mailto:nouboh@xxxxxxxxx>
<mailto:nouboh@xxxxxxxxx <mailto:nouboh@xxxxxxxxx>>>>:
Thanks! That was exactly the problem. I have
increased the
constant
MAX_THREADS and now I don't have this problem
anymore. Is
there any
reason why the constant is set to 32 instead of a
higher value?
I also have tried to find how to dynamically change
the DYNINST_max_num_threads value but I haven't
found where
is it
implemented (version 8.2.1). And about disabling tramp
guards, I
already had them disabled so it seems that this
workaround
does not
solve the problem.
Gerard
2015-02-03 20:13 GMT+01:00 Barton Miller
<bart@xxxxxxxxxxx <mailto:bart@xxxxxxxxxxx>
<mailto:bart@xxxxxxxxxxx <mailto:bart@xxxxxxxxxxx>>
<mailto:bart@xxxxxxxxxxx <mailto:bart@xxxxxxxxxxx>
<mailto:bart@xxxxxxxxxxx <mailto:bart@xxxxxxxxxxx>>>>:
Disabling tramp guards certainly works if you
really
know that
you're not recursing. That can be subtle and
error
prone, which
is why tramp guards were invented. Proceed
cautiously
here.
--bart
On 2/3/2015 12:02 PM, Matthew LeGendre wrote:
Another possible fix may be to disable tramp
guards. Tramp
guards are used to prevent recursive
instrumentation. For
example, if you instrument malloc() with
instrumentation
that calls malloc(), then tramp guards
will prevent
you from
going into infinite recursion.
If you already know that your
instrumentation can't
infinitely recurse,
then disabling tramp guards will give a big
performance win
and may work around this hang. To disable
tramp
guards, put
a call to
BPatch::setTrampRecursive(______true)
before you
insert instrumentation.
-Matt
_____________________________________________________
Dyninst-api mailing list
Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxxxx>
<mailto:Dyninst-api@xxxxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxxxx>>
<mailto:Dyninst-api@xxxxxxxx
<mailto:Dyninst-api@xxxxxxxx>____edu
<mailto:Dyninst-api@xxxxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxxxx>>>
https://lists.cs.wisc.edu/______mailman/listinfo/dyninst-api
<https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api>
<https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
<https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>>
<https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
<https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>
<https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
<https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>>>
_____________________________________________________
Dyninst-api mailing list
Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxxxx>
<mailto:Dyninst-api@xxxxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxxxx>>
<mailto:Dyninst-api@xxxxxxxx
<mailto:Dyninst-api@xxxxxxxx>____edu
<mailto:Dyninst-api@xxxxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxxxx>>>
https://lists.cs.wisc.edu/______mailman/listinfo/dyninst-api
<https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api>
<https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
<https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>>
<https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
<https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>
<https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
<https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>>>
___________________________________________________
Dyninst-api mailing list
Dyninst-api@xxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxxxx>
<mailto:Dyninst-api@xxxxxxxxxxxxx <mailto:Dyninst-api@xxxxxxxxxxx>>
https://lists.cs.wisc.edu/____mailman/listinfo/dyninst-api
<https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api>
<https://lists.cs.wisc.edu/__mailman/listinfo/dyninst-api
<https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api>>
--
--bw
Bill Williams
Paradyn Project
bill@xxxxxxxxxxx <mailto:bill@xxxxxxxxxxx>
<mailto:bill@xxxxxxxxxxx <mailto:bill@xxxxxxxxxxx>>
--
--bw
Bill Williams
Paradyn Project
bill@xxxxxxxxxxx <mailto:bill@xxxxxxxxxxx>
--
--bw
Bill Williams
Paradyn Project
bill@xxxxxxxxxxx
|
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}