TUI pending tool responses leak: some entries persist until restart

[hoblin]

Problem to solve

In the TUI, tool-response PendingMessages occasionally accumulate as visible entries that never disappear. Once stuck, they persist for the lifetime of the TUI process — only a restart clears them. Real Message rows do exist in the DB for these tool calls; the duplicates are stale PendingMessage entries that never received their removal broadcast.

This is the second known papercut in the pending-message broadcast path. The first — pending entries being styled like user messages — was addressed in #366. The current behavior shares the same surface but has a different root cause: it's not a styling drift, it's missing/lost broadcasts for some PMs in a batch.

Hypothesis

The leak is correlated with batch tool calls — when one LLM response contains multiple tool_use blocks, the system creates one PendingMessage per tool, each is executed, and each is then promoted by DrainJob. The fact that some but not all PMs leak (and only when there are multiple in the same round) points at a per-broadcast issue rather than a per-session issue.

Two candidate failure modes worth examining:

1. Some broadcast_removed events don't reach the TUI. PendingMessage#after_destroy_commit :broadcast_removed fires ActionCable.server.broadcast directly (app/models/pending_message.rb:121, 422). If a render/decorator path raises during broadcast_payload for a specific PM, the broadcast for that one PM is lost while siblings go through. The TUI never learns to remove the stale entry.
2. The TUI's removal handling skips when the matching create broadcast was missed. Race: PM is created → broadcast fires → TUI subscriber not yet attached → broadcast_removed arrives later for an entry the TUI never saw create. Less likely given persistent ActionCable subscription, but possible during reconnects.

The "only some, only in batches" pattern strongly suggests (1) — a per-PM render path that fails for certain tool_response shapes.

Research approach

Wire up debug logging (dev environment only) on the broadcast path to find the answer:

• Log every PendingMessage#broadcast_created and #broadcast_removed invocation with PM id, message_type, source_name, and tool_use_id.
• Log every MessageBroadcaster#emit invocation with message id and tool_use_id.
• On the TUI side, log every received broadcast with action (create/remove/update) and id.

Then reproduce with a session that triggers a batch tool round (e.g. spawn 5 sub-agents in parallel — exactly what triggered today's session). Compare:

• N PMs created → N broadcast_created logged → N broadcasts received by TUI?
• N PMs destroyed → N broadcast_removed logged → N broadcasts received by TUI?
• N Messages created → N MessageBroadcaster#emit logged → N create broadcasts received by TUI?

Any mismatch identifies the leak point.

Suspected choke points (starting hints, not prescribed)

• app/models/pending_message.rb:301-424 — broadcast_payload, broadcast_created, broadcast_removed. The payload assembly calls decorate(...).render(mode) which can raise per-decorator. If one PM's decorator raises, the broadcast is lost — but DrainJob still proceeds because broadcast_* callbacks aren't gating the transaction.
• app/decorators/pending_tool_response_decorator.rb and the pending_from_*_decorator.rb family — different render paths per source_type. A decorator that crashes on a specific tool_response shape (e.g. error-typed responses, or specific MCP tool names) would leak only that subset.
• lib/events/subscribers/message_broadcaster.rb:24-30 — also calls message.decorate.render(session.view_mode). Same kind of failure mode for the create broadcast: if it fails, the TUI never knows to attach the new Message to the entry that the pending broadcast already created.
• app/jobs/drain_job.rb:80-90 — tool_responses.each(&:promote!) runs N independent session.transaction blocks. Each promote! creates a Message + destroys a PM. The broadcasts fire from after_*_commit callbacks, so they're outside the transaction — exceptions there are silent on the DB side but eat the broadcast.

Acceptance criteria

• Debug logs added to dev environment let us count creates/removes/messages per batch round and identify any per-PM mismatch.
• Root cause identified and recorded in the issue (or a follow-up).
• Follow-up implementation issue opened with the appropriate fix.

Why this matters

A stale pending entry is visually indistinguishable from a real one (since #366 removed the styling difference). The TUI grows visually noisier across a long session and the human can't tell which entries are real conversation vs. broadcast leaks. It's the kind of small degradation that erodes trust in the UI without ever throwing a visible error.

Related

• Drop pending message visual distinction in TUI #366 — Drop pending message visual distinction in TUI
• Unify async message insertion and promote pending messages during tool execution #384 — Unify async message insertion and promote pending messages during tool execution
• TUI: per-kind rendering for PendingMessage #452 — TUI: per-kind rendering for PendingMessage
• Pending message interleaves with tool_use/tool_result pair #357 — Pending message interleaves with tool_use/tool_result pair

[hoblin]

Update: actual stuck entries from the live session

Hob shared the visible stuck entries from the TUI:

↩ [toolu_01PR57jybVJjxjctPWr3yB9c] ✓ OK
↩ [toolu_01Vtfmu4NGerwzqeUhVtr2U7] ✓ OK
↩ [toolu_01TWQdfiFHRiimgEzmmbcVAS] ❌ {"error":"Tools::UnknownToolError: Unknown tool: from_melete_goal"}
↩ [toolu_014rrDgKGXTr1A5ncYpiB9ij] ✓ OK
↩ [toolu_011336gg6B9CxZFzoSYKQ93z] ✓   # For promotions that yield a single Message...
↩ [toolu_012bCdxfzBMzRJxhENa36UR7] ✓ 198:        self.promoted_message_id = promote_as_tool_response!.id

Critical observation: one of the stuck entries is the from_melete_goal unknown-tool error. The other five are normal tool responses (mostly read_file results, judging by the content) from the same session round.

Revised hypothesis: the leak may be triggered by the unknown-tool failure mode rather than batch broadcasting. When the LLM dispatches a tool that doesn't exist in the registry:

1. The dispatcher still creates a tool_response PM with the error message (so the LLM sees its failure).
2. But the tool_use_id may not be tracked the same way as a registered tool — no matching tool_call expected, possibly no proper accounting in Session#tool_round_complete?.
3. DrainJob waits for round completion. If the AASM guard never flips because one tool in the batch isn't being counted, none of that round's PMs get promoted.
4. All sibling tool_responses in the round get stranded as PMs forever — exactly the symptom seen here.

This connects directly to #480. The fix sequence should probably be:

1. Resolve Research spike: source of from_* tool calls in the main loop #480 first (filter or properly handle from_* tool calls).
2. Verify whether TUI pending tool responses leak: some entries persist until restart #481 still reproduces without the unknown-tool trigger.
3. If yes → investigate broadcasts as originally planned. If no → the original "only some, only in batches" symptom was actually "the whole round when one tool fails to register."

The diagnostic logging proposed in this issue is still the right next step, but should specifically capture: (a) which PMs are in the round, (b) what tool_round_complete? returns and why, (c) why promote! isn't being called for these specific PMs.

[hoblin]

Another evidence. The tool response was propagated to the message but pending message stayed visible in TUI.

│↩ ✓ Sub-agent ticket-verifier-pr-5141 spawned (session 26). Its messages will appear in your conversation. To address it, prefix its name with @ in your message.                │
│                                                                                                                                                                                 │                                                                                                                          │
│↩ [toolu_01NQK6TYvsZv3RD1zTxbeiKZ] ✓ Sub-agent ticket-verifier-pr-5141 spawned (session 26). Its messages will appear in your conversation. To address it, prefix its name with @│
│in your message.