Branch: refs/heads/bbiiggppiigg/fix-test-thread-1
Home: https://github.com/dyninst/dyninst
Commit: 76323fba5c2b251fd4a78e87e3e30382602b523e
https://github.com/dyninst/dyninst/commit/76323fba5c2b251fd4a78e87e3e30382602b523e
Author: Hsuan-Heng Wu <bbiiggppiigg@xxxxxxxxx>
Date: 2026-07-07 (Tue, 07 Jul 2026)
Changed paths:
M proccontrol/src/event.C
M proccontrol/src/handler.C
M proccontrol/src/int_event.h
M proccontrol/src/int_thread_db.C
M proccontrol/src/process.C
Log Message:
-----------
proccontrol: don't cache raw int_thread* in events
int_eventBreakpoint::thrd and int_eventNewUserThread::thr held raw
int_thread* pointers. A breakpoint or thread-create event can outlive
the thread that triggered it: the thread exits while the event is still
queued (or sitting in the callback/muxer layer), the int_thread is
deleted, and the cached pointer dangles -- observed as sporadic SIGSEGVs
in lookupInstalledBreakpoint() and EventNewUserThread::getNewThread()
when running the testsuite's test_thread_1.
Store the reference-counted Thread wrapper instead: it survives thread
destruction and its accessors are exit-safe (llthrd() returns NULL,
getLWP() falls back to the exit state). Guard the dereference sites
that can now legitimately see an exited thread: lookupInstalledBreakpoint,
EventBreakpoint::getBreakpoints, the read-only breakpoint/single-step
handlers, ThreadDBCreateHandler, and int_process::getBreakpoint (whose
mem is NULL once the process is cleaned up).
Co-Authored-By: Claude Fable 5 <noreply@xxxxxxxxxxxxx>
Commit: 82cf6ac18f0ce9b819f28c32d78590613c702943
https://github.com/dyninst/dyninst/commit/82cf6ac18f0ce9b819f28c32d78590613c702943
Author: Hsuan-Heng Wu <bbiiggppiigg@xxxxxxxxx>
Date: 2026-07-07 (Tue, 07 Jul 2026)
Changed paths:
M proccontrol/src/process.C
Log Message:
-----------
proccontrol: fix debug prints on exited threads
The pthrd_printf calls in ProcStopEventManager::prepEvent/checkEvents
dereference ev->getThread()->llthrd()->getLWP() unconditionally. A
held proc-stopper event's thread can exit before the event is released,
making llthrd() NULL and crashing the process -- but only when
DYNINST_DEBUG_PROCCONTROL is enabled, which made the underlying races
in this area impossible to trace.
Co-Authored-By: Claude Fable 5 <noreply@xxxxxxxxxxxxx>
Commit: c9dc680ed1fca1cc1b00f766e271b3fa350db2ad
https://github.com/dyninst/dyninst/commit/c9dc680ed1fca1cc1b00f766e271b3fa350db2ad
Author: Hsuan-Heng Wu <bbiiggppiigg@xxxxxxxxx>
Date: 2026-07-07 (Tue, 07 Jul 2026)
Changed paths:
M proccontrol/src/handler.C
Log Message:
-----------
proccontrol: restore state desyncs on thread death
The breakpoint machinery brackets proc-wide stops in desyncStateProc /
restoreStateProc pairs on the BreakpointHold, Breakpoint and
BreakpointResume state layers. When the thread that owns the event
dies mid-sequence, the restoring half was skipped, leaving every
surviving thread force-stopped forever (new threads inherit the desync
too), which hangs the process.
Two shapes of the leak:
- HandleBreakpointContinue/Clear/Restore bail out when the event's
thread is gone, skipping their paired restore. The restore is a
process-wide walk that only needs a live thread as its handle, so
route it through any thread still in the pool (the pool only holds
live threads; initialThread() can be stale during teardown).
- A thread that dies while single-stepping over a cleared (suspended)
breakpoint never produces the single-step trap, so the decoder never
generates the BreakpointRestore event at all: the breakpoint stays
out of memory and the proc-wide BreakpointResume desyncs leak.
Perform that recovery in HandleThreadCleanup while the dying thread
is still in the pool, so one restore walk exactly balances the one
desync each thread received.
Co-Authored-By: Claude Fable 5 <noreply@xxxxxxxxxxxxx>
Commit: 8c840867185b340712271a95498c47f9e08967dc
https://github.com/dyninst/dyninst/commit/8c840867185b340712271a95498c47f9e08967dc
Author: Hsuan-Heng Wu <bbiiggppiigg@xxxxxxxxx>
Date: 2026-07-07 (Tue, 07 Jul 2026)
Changed paths:
M proccontrol/src/int_thread_db.C
M proccontrol/src/process.C
Log Message:
-----------
proccontrol: fix thread_db stopped-thread asserts
thread_db_process::getEventForThread() asserts all threads are
handler-stopped before reading thread_db state out of the mutatee.
Two races broke the assertion when mutatee threads are created and
destroyed rapidly:
- Exiting/exited threads can have transiently stale handler states (the
exit event is decoded but not yet handled). Such threads cannot
execute user code and so cannot race the memory reads; make
allHandlerStopped() skip them, mirroring the exiting-thread handling
in syncRunState().
- A thread created after the proc-stopping thread_db breakpoint event
was released, but before the dispatch handler ran, is still running.
Its ThreadCreate event is pending in the mailbox and will stop it
(HandleThreadCreate initializes new threads to the process's desync'd
states), so defer the dispatch with ret_again until that has
happened.
Notes for reviewers: allHandlerStopped() has one other caller, in the
FreeBSD startup path; skipping exiting threads is believed equally
correct there (a thread that cannot run cannot violate quiescence) but
was not tested on that platform. Handler::ret_again existed but had no
in-tree users before this change; this is its first use.
Co-Authored-By: Claude Fable 5 <noreply@xxxxxxxxxxxxx>
Commit: 6324663d0222287b4859ca39097227a434715060
https://github.com/dyninst/dyninst/commit/6324663d0222287b4859ca39097227a434715060
Author: Hsuan-Heng Wu <bbiiggppiigg@xxxxxxxxx>
Date: 2026-07-07 (Tue, 07 Jul 2026)
Changed paths:
M proccontrol/src/response.C
M proccontrol/src/response.h
Log Message:
-----------
proccontrol: never destroy the response id locks
The response.C id_lock mutexes were plain statics. Static destructor
order across translation units is unspecified, so they could be
destroyed before int_cleanup (generator.C) stops the handler thread;
the handler thread then locked a destroyed mutex while draining
teardown events (glibc marks a destroyed mutex __kind = -1 and
pthread_mutex_lock returns EINVAL, which boost turns into a lock_error
exception and std::terminate).
Heap-allocate and intentionally leak both locks, mirroring the existing
fix and comment for Counter::locks in process.C.
Co-Authored-By: Claude Fable 5 <noreply@xxxxxxxxxxxxx>
Commit: 5e6df583dfc8185a60bfa5298e3d8d987b52022c
https://github.com/dyninst/dyninst/commit/5e6df583dfc8185a60bfa5298e3d8d987b52022c
Author: Hsuan-Heng Wu <bbiiggppiigg@xxxxxxxxx>
Date: 2026-07-07 (Tue, 07 Jul 2026)
Changed paths:
M proccontrol/src/linux.C
Log Message:
-----------
proccontrol: bound generator eviction spin at exit
GeneratorLinux::evictFromWaitpid() sends SIGUSR2 to the generator
thread and spins until the signal handler sets a flag. The generator
can exit on its own initiative (no live processes left) concurrently
with the eviction; the signal is then queued to a dying thread and
discarded, the flag never fires, and the destructor spins forever at
100% CPU inside exit().
Bound the wait to a couple of seconds: the eviction is best-effort --
we are already exiting and exit_group() will reap the generator thread
regardless.
Co-Authored-By: Claude Fable 5 <noreply@xxxxxxxxxxxxx>
Commit: 7813afd4c0291e856701e8428803720853c18ba5
https://github.com/dyninst/dyninst/commit/7813afd4c0291e856701e8428803720853c18ba5
Author: Hsuan-Heng Wu <bbiiggppiigg@xxxxxxxxx>
Date: 2026-07-07 (Tue, 07 Jul 2026)
Changed paths:
M proccontrol/src/linux.C
M proccontrol/src/process.C
Log Message:
-----------
proccontrol: don't stop threads that are exiting
If intStop() sends SIGSTOP to a thread whose exit event is already in
flight (PTRACE_EVENT_EXIT observed), the stop can never be delivered:
the thread just dies. The PendingStop state layer then stays desync'd
to running forever, which (a) blocks isStopped() for every higher
layer, so held proc-stopper breakpoint events are never released, and
(b) forces a run target for a thread that can only die. The process
deadlocks with the mutatee ptrace-stopped, the generator idle, and an
unreaped zombie thread -- the dominant sporadic hang when running the
testsuite's test_thread_1.
LinuxHandleLWPDestroy was written for exactly this case (clear pending
stops on pre-LWPDestroy; its comment describes the bug) but was never
registered with the handler pool. Register it, guard it against
already-destroyed threads, and additionally skip intStop() for threads
already observed exiting. The two halves are intentionally redundant:
the intStop() guard covers stops requested after the exit was observed,
the handler covers stops requested before it (SIGSTOP landing on a
thread already inside the exit syscall).
Co-Authored-By: Claude Fable 5 <noreply@xxxxxxxxxxxxx>
Commit: 9e16472cbec49d949561e10e1f478d3005fc8523
https://github.com/dyninst/dyninst/commit/9e16472cbec49d949561e10e1f478d3005fc8523
Author: Hsuan-Heng Wu <bbiiggppiigg@xxxxxxxxx>
Date: 2026-07-07 (Tue, 07 Jul 2026)
Changed paths:
M proccontrol/src/int_process.h
M proccontrol/src/process.C
Log Message:
-----------
proccontrol: re-sync run state before parking
The event loop can park with an actionable run-state divergence left
unserviced: a thread sits handler-stopped but targeted running -- e.g.
the PendingStop state wants it continued so an in-flight SIGSTOP can be
delivered -- yet no event is in flight to drive another syncRunState
pass. waitAndHandleEvents then parks on an empty mailbox (its own
should_block check even counts the pending stop as a reason to block),
the generator sees nothing runnable and idles, and the process
deadlocks: mutatee ptrace-stopped, SIGSTOP pinned undelivered in
SigPnd, proc-stopper breakpoint events held forever.
Captured live three times with different micro-interleavings; in each
case a single additional syncRunState pass (via an injected nop event,
or calling intCont directly on the divergent thread) fully unwound the
deadlock. Make the event loop do that for itself: sweep for processes
with a divergent thread and re-run syncRunState on them at both park
points -- before a blocking dequeue (the user thread's park), and when
the mailbox drains (the handler thread services events with
block=false and returns there; a divergence formed during its handling
would otherwise be left for nobody, as the user thread may already be
blocked inside its own dequeue and can only wake on an enqueue that
will never come). The pass is idempotent (threads already in their
target state or with a stop in flight are skipped), any action it
takes produces the event that wakes the dequeue, and the sweep runs at
most once per park, so an unresolvable divergence parks exactly as
before.
Co-Authored-By: Claude Fable 5 <noreply@xxxxxxxxxxxxx>
Commit: 63a96cb9d907fbd34516464f93e744373ca6e773
https://github.com/dyninst/dyninst/commit/63a96cb9d907fbd34516464f93e744373ca6e773
Author: Hsuan-Heng Wu <bbiiggppiigg@xxxxxxxxx>
Date: 2026-07-07 (Tue, 07 Jul 2026)
Changed paths:
M proccontrol/src/process.C
Log Message:
-----------
proccontrol: don't SIGSTOP an already-stopped thread
plat_syncRunState decides to stop a thread from its handler-side state,
which can lag the generator by tens of microseconds: the thread has
already stopped (e.g. on a breakpoint) and its event is decoded and
queued, but not yet handled. intStop then sends a SIGSTOP to an
already-ptrace-stopped thread. That is both useless -- the signal
cannot be delivered until the thread resumes, pinning the PendingStop
state desync'd to running -- and actively harmful: on Linux, a STOP
signal sent to a ptrace-stopped thread can put it in a transitional
state in which PTRACE_CONT persistently fails with ESRCH even though
the thread is alive (the exact behavior already documented in
plat_handle_ghost_thread). plat_syncRunState suppressed that failure
silently, so every subsequent continue attempt -- including the
pre-park self-heal passes -- failed the same way until the event queue
drained and the process deadlocked: mutatee ptrace-stopped, SIGSTOP
pinned in SigPnd, proc-stopper events held forever. Captured end to
end at microsecond resolution with an in-memory trace ring.
Fix both sides:
- intStop treats a thread whose generator state is already stopped or
exited as satisfied: its stop event is in flight and the handler will
mark it stopped imminently, so no signal is needed (mirrors the
existing exiting-thread guard).
- The remaining race window (stop requested in the microseconds before
the generator's decode is visible) cannot be closed from the handler
side, so when a ptrace op fails with ESRCH on a live thread, schedule
a retry via a nop event instead of silently swallowing the failure:
the op succeeds once the kernel state settles, converting a permanent
deadlock into a sub-millisecond stall.
Co-Authored-By: Claude Fable 5 <noreply@xxxxxxxxxxxxx>
Compare: https://github.com/dyninst/dyninst/compare/36f4fd0e98e4...63a96cb9d907
To unsubscribe from these emails, change your notification settings at https://github.com/dyninst/dyninst/settings/notifications
|