tractor/ai/conc-anal/subint_cancel_delivery_hang_issue.md

subint backend: parent trio loop parks after subint teardown (Ctrl-C works; not a CPython-level issue)

Follow-up to the Phase B subint spawn-backend PR (see tractor.spawn._subint, issue #379). Distinct from the subint_sigint_starvation_issue.md (SIGINT-unresponsive starvation hang): this one is Ctrl-C-able, which means it’s not the shared-GIL-hostage class and is ours to fix from inside tractor rather than waiting on upstream CPython / msgspec progress.

TL;DR

After a stuck-subint subactor is torn down via the hard-kill path, a parent-side trio task parks on an orphaned resource (most likely a chan.recv() / process_messages loop on the now-dead subint’s IPC channel) and waits forever for bytes that can’t arrive — because the channel was torn down without emitting a clean EOF/BrokenResourceError to the waiting receiver.

Unlike subint_sigint_starvation_issue.md, the main trio loop is iterating normally — SIGINT delivers cleanly and the test unhangs. But absent Ctrl-C, the test suite wedges indefinitely.

Symptom

Running test_subint_non_checkpointing_child under --spawn-backend=subint (in tests/test_subint_cancellation.py):

1. Test spawns a subactor whose main task runs threading.Event.wait(1.0) in a loop — releases the GIL but never inserts a trio checkpoint.
2. Parent does an.cancel_scope.cancel(). Our subint_proc cancel path fires: soft-kill sends Portal.cancel_actor() over the live IPC channel → subint’s trio loop should process the cancel msg on its IPC dispatcher task (since the GIL releases are happening).
3. Expected: subint’s trio.run() unwinds, driver thread exits naturally, parent returns.
4. Actual: parent trio.run() never completes. Test hangs past its trio.fail_after() deadline.

Evidence

strace on the hung pytest process during SIGINT

--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
write(17, "\2", 1)                      = 1

Contrast with the SIGINT-starvation hang (see subint_sigint_starvation_issue.md) where that same write() returned EAGAIN. Here the SIGINT byte is written successfully → Python’s signal handler pipe is being drained → main trio loop is iterating → SIGINT gets turned into trio.Cancelled → the test unhangs (if the operator happens to be there to hit Ctrl-C).

Stack dump (via tractor.devx.dump_on_hang)

Single main thread visible, parked in trio._core._io_epoll.get_events inside trio.run at the test’s trio.run(...) call site. No subint driver thread (subint was destroyed successfully — this is after the hard-kill path, not during it).

Root cause hypothesis

Most consistent with the evidence: a parent-side trio task is awaiting a chan.recv() / process_messages loop on the dead subint’s IPC channel. The sequence:

1. Soft-kill in subint_proc sends Portal.cancel_actor() over the channel. The subint’s trio dispatcher may or may not have processed the cancel msg before the subint was destroyed — timing-dependent.
2. Hard-kill timeout fires (because the subint’s main task was in threading.Event.wait() with no trio checkpoint — cancel-msg processing couldn’t race the timeout).
3. Driver thread abandoned, _interpreters.destroy() runs. Subint is gone.
4. But the parent-side trio task holding a chan.recv() / process_messages loop against that channel was not explicitly cancelled. The channel’s underlying socket got torn down, but without a clean EOF delivered to the waiting recv, the task parks forever on trio.lowlevel.wait_readable (or similar).

This matches the “main loop fine, task parked on orphaned I/O” signature.

Why this is ours to fix (not CPython’s)

• Main trio loop iterates normally → GIL isn’t starved.
• SIGINT is deliverable → not a signal-pipe-full / wakeup-fd contention scenario.
• The hang is in our supervision code, specifically in how subint_proc tears down its side of the IPC when the subint is abandoned/destroyed.

Possible fix directions

1. Explicit parent-side channel abort on subint abandon. In subint_proc’s teardown block, after the hard-kill timeout fires, explicitly close the parent’s end of the IPC channel to the subint. Any waiting chan.recv() / process_messages task sees BrokenResourceError (or ClosedResourceError) and unwinds.
2. Cancel parent-side RPC tasks tied to the dead subint’s channel. The Actor._rpc_tasks / nursery machinery should have a handle on any process_messages loops bound to a specific peer channel. Iterate those and cancel explicitly.
3. Bound the top-level await actor_nursery ._join_procs.wait() shield in subint_proc (same pattern as the other bounded shields the hard-kill patch added). If the nursery never sets _join_procs because a child task is parked, the bound would at least let the teardown proceed.

Of these, (1) is the most surgical and directly addresses the root cause. (2) is a defense-in-depth companion. (3) is a band-aid but cheap to add.

Current workaround

None in-tree. The test’s trio.fail_after() bound currently fires and raises TooSlowError, so the test visibly fails rather than hangs — which is intentional (an unbounded cancellation-audit test would defeat itself). But in interactive test runs the operator has to hit Ctrl-C to move past the parked state before pytest reports the failure.

Reproducer

./py314/bin/python -m pytest \
  tests/test_subint_cancellation.py::test_subint_non_checkpointing_child \
  --spawn-backend=subint --tb=short --no-header -v

Expected: hangs until trio.fail_after(15) fires, or Ctrl-C unwedges it manually.

References

• tractor.spawn._subint.subint_proc — current subint teardown code; see the _HARD_KILL_TIMEOUT bounded shields + daemon=True driver-thread abandonment (commit b025c982).
• ai/conc-anal/subint_sigint_starvation_issue.md — the sibling CPython-level hang (GIL-starvation, SIGINT-unresponsive) which is not this issue.
• Phase B tracking: issue #379.