Prevent RPC session crashes on concurrent multi-image traffic - #496
Conversation
Implementation PlanProblem analysisA dashboard session (RPC child process) dies when multiple images are in flight simultaneously. The death is always the 16 MiB stdout backpressure guard ( Legitimate multi-image traffic exceeds the cap because:
Secondary items called out in the issue: the "Fatal: stdout write queue exceeded..." diagnostic goes to child stderr only, while Deliverables
Acceptance criteria → how they are met
Files to create or modify
Test plan
Risks and open questions
Plan created by mach6 |
Vitest coverage
|
Progress UpdateImplementation of the plan is complete. All five levers are in, with tests and docs. What changed (18 files):
Tests: new multi-image survival regression test (real Docs: root README, coding-agent README, Verification: full workspace test suite green ( Commit: Progress tracked by mach6 |
Progress UpdateIgnore dashboard runtime data under Commit: Progress tracked by mach6 |
Unverified Review Candidates — Pending AssessmentReview round: 1
Critical(none) ImportantF1 — Child-side dedupe identity is looser than the dashboard's strict decode → dangling image references (confidence 88/100; corroborated by a second specialist, empirically verified)
Pre-PR such blocks were silently dropped at every occurrence (no element, no request); post-PR the transcript renders a broken-image slot per dangling reference, and each render triggers a full authoritative transcript round trip over the same pipe — re-triggering the multi-MiB re-fetch bursts this PR exists to eliminate (N corrupted uploads → N full-transcript fetches per view; no negative caching). Also contradicts the PR docs' claim that such blocks "stay inline and are rejected … exactly as before" — true only for occurrence 1. Fix direction: make the child's gate as strict as the dashboard's (round-trip + signature), or cache negative entries on the dashboard so references to dropped ids render absent. F2 — Partial drain (progress while still over cap) is untested, and the existing "slow consumer survives" test is provably masked (confidence 92/100, empirically verified) No test covers the over-cap + partial drain scenario: no-drain window armed, a drain fires but only some bytes are accepted so the guard queue stays over 16 MiB, and the window must be reset because progress was made. The PR's only over-cap survival test is a full drain (queue to zero). Verified by experiment: deleting F3 — AC1's "the session remains usable" is unasserted, and the child-stdin half of the prompt path is bypassed (confidence 90/100) The new survival test makes exactly one prompt, drains, and ends: no follow-up prompt is ever sent, so issue 495 acceptance criterion 1's "the session remains usable" is not exercised. A regression that leaves guard state dirty after a burst (residual queued bytes, a stuck backpressure flag, or a leftover armed timer) would kill or stall the next prompt — invisible to a test that stops at the drain. Also, the test mocks SuggestionsF4 — dashboard.md claims the child stdout queue "caps at 16 MiB" — it no longer does (confidence 90/100) — F5 — AC4's "images continue to be delivered to the model" has no assertion (confidence 90/100) — the survival test never inspects F6 — Boundary and defensive branches with zero coverage (confidence 88/100) — (a) queue exactly at the 16 MiB cap: a F7 — F8 — F9 — Unrelated F10 — Root README edited despite the plan's "No root README change" (confidence 90/100, scope note) — the added sentence is factually accurate against the code and the repo's development guide mandates root-README accuracy for feature changes; noted for completeness, no action needed. Strengths
Agents run: code-reviewer, error-auditor, test-reviewer, completeness-checker, simplifier Reviewed by mach6 |
Review AssessmentClassifications
Acceptance criteria (for the record)
Action PlanNone — no merge blockers. Useful follow-ups F1–F4 remain outside the action plan (candidate for a follow-up PR/issue: tighten Assessment by mach6 |
…rain guard test), F4 (16 MiB threshold wording)
Progress UpdateFixed findings 1, 2, and 4 from the review round. Finding 1 — child dedupe gate was looser than the dashboard's strict decode (dangling references). Finding 2 — partial drain over cap was untested (and the slow-consumer test provably masked). Finding 4 — dashboard.md claimed the child stdout queue "caps at 16 MiB". Verification: full workspace suite via Commit: Progress tracked by mach6 |
Unverified Review Candidates — Pending AssessmentReview round: 2
Critical(none) ImportantF11 — Strict dedupe gate: several MIME/base64 branches unpinned by tests (confidence 84/100)
A future edit (or partial revert) desyncing the child's mirrored gate from the dashboard's SuggestionsF12 — Unreachable F13 — Three structurally identical pipe-error tests (confidence 82/100) — F14 — New Strengths
Agents run: code-reviewer, error-auditor, test-reviewer, completeness-checker, simplifier Reviewed by mach6 |
Review AssessmentClassifications
Acceptance criteria (for the record)
Action PlanNone — no merge blockers. Deferred follow-ups (outside the action plan): F3 (route the survival test through the real JSONL ingress and send a post-burst prompt), F5 (assert prompt-image blocks reach the model context), F11 (gate branch fixtures: trailing-garbage round-trip, WebP valid+reject, GIF reject, JPEG claim assertion), F12 (delete the unreachable check). F13/F14 are optional cleanups the author may take or leave. Assessment by mach6 |
Closes #495
A dashboard-driven RPC session can be force-killed by the 16 MiB stdout backpressure guard when multiple images are in flight simultaneously — both 2+ image prompt uploads and turns where the agent reads 2-3+ images quickly. Image base64 crosses the child stdout multiple times per turn (prompt re-emission, 3× per read-tool result, full
agent_endtranscript), and the dashboard's single-threaded consume stalls long enough for the queue to pass the cap.Implementation plan posted as a comment below.