Skip to content

CL-6486: time out a reply that stalls mid-stream - #220

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-6486-stall
Aug 21, 2026
Merged

TheGreatAxios merged 2 commits into
mainfrom
cl-6486-stall

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

The 120s stale-backstop in useStreamingReply only armed while isPendingReply was true — phase "awaiting" and text === "". The instant one token streamed, the guard returned early and never re-armed. A reply that starts streaming and then dies (model OOM, dropped Ollama connection, sidecar crash — all routine with local models) froze mid-sentence behind a blinking cursor forever: no error, no retry, indistinguishable from "still thinking."

The no-token case was already handled honestly (120s -> "No reply arrived — the agent may be unavailable."). This closes the gap for the mid-stream case, reusing the same honest-failure path.

Fix

Arm the backstop for the whole "awaiting" phase instead of just its tokenless prefix. Each token produces a new streamingReply object (awaiting(text) always returns a fresh literal), so the effect's own [streamingReply, clearMs] dependency already tears down and re-arms the timer on every token, with no extra ref bookkeeping needed. That means the timer measures silence since the last token, not total elapsed time.

Window: kept at 120s (PENDING_REPLY_CLEAR_MS)

Reused the existing constant rather than introducing a second one:

  • It's already the app's established tolerance for "no signal from the agent," used identically for the no-token case today.
  • Round 1 measured a healthy local turn running up to ~216s end to end on qwen3.8:27b — but that's cumulative time across multiple inference rounds, tool calls, and thinking, not a single continuous gap with zero new tokens. Because the window now resets on every token, a turn producing tokens at any reasonable cadence never gets close to 120s of pure silence, no matter how long the whole turn runs.
  • 120 continuous seconds with not one new token, mid-generation, is well past any single decode step, tool round-trip, or SSE reconnect a client should have to ride out.

Tests (packages/chat-ui/test/use-streaming-reply.test.tsx)

  • Updated the test that asserted today's (buggy) behavior — "a token arriving before the backstop never marks it timed out" — into two tests for the corrected contract:
    • a token resets the window (doesn't just prevent the original deadline)
    • silence after a token still times out a mid-stream stall
  • The no-token backstop test is untouched and still passes.
  • Full suite: 650 pass / 0 fail in packages/chat-ui. tsc --noEmit clean. bun run lint from repo root: 0 errors (12 pre-existing warnings, none in touched files).

Not done — reporting as the next unit (audit finding 7)

useWorkbenchStream's connection state ("connecting"|"live"|"polling") is still discarded by its only caller in chat-workspace.tsx and never rendered. Surfacing it quietly when the transport is degraded (without turning it into chrome on a healthy connection) is a separate, judgment-heavy UI change and didn't fit this timebox alongside the backstop fix + tests. Flagging for a follow-up issue rather than rushing it in.

What I did not verify

  • No live/manual repro against a real Ollama/sidecar crash — verified via the unit test harness only (fake timers via settle()), per the no-stack-boot constraint on this lane.
  • Did not add a retry button to the existing timed-out notice — the ticket's "offer retry" language — the existing chat.reply-timed-out notice renders through the same honest-failure path as before (unchanged), but there's no retry affordance wired to it in the codebase today (unlike the pending-send retry/discard flow). Wiring a retry action for a stalled agent reply looked like its own feature (re-invoking a turn, not just re-sending a message) rather than a one-line addition, so I left it out rather than bolt on something half-considered under time pressure.

Fixes CL-6486

The existing test asserted that a single token permanently disarms the
120s backstop; that's the CL-6486 bug, not a contract worth keeping.
Replace it with tests for the corrected contract: a token resets the
silence window rather than clearing it, and silence after a token still
times out a mid-stream stall.
The 120s backstop only armed while the reply was still tokenless
(isPendingReply). The instant one token streamed it stopped re-arming,
so a turn that starts streaming and then dies (model OOM, dropped
Ollama connection, sidecar crash) froze mid-sentence behind a blinking
cursor forever, with no error and no retry.

Arm the backstop for the whole "awaiting" phase instead of just its
empty-text prefix. Each token produces a new streamingReply object, so
the effect's own dependency already tears down and re-arms the timer on
every token — this measures the gap since the last token, not total
elapsed time, so a healthy long-running local turn (round 1 measured
~216s on qwen3.8:27b) never trips it.
@TheGreatAxios
TheGreatAxios merged commit e51d988 into main Aug 21, 2026
2 checks passed
@TheGreatAxios
TheGreatAxios deleted the cl-6486-stall branch August 25, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant