Skip to content

Add mid-turn steering to ChatSession - #608

Open
aleroot wants to merge 1 commit into
ml-explore:mainfrom
aleroot:steering
Open

aleroot wants to merge 1 commit into
ml-explore:mainfrom
aleroot:steering

Conversation

@aleroot

@aleroot aleroot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Proposed changes

Adding this so that my app can add requirements or change direction while a response is running. The session queues the instruction, preserves output already emitted, and continues from its structured conversation using a verified cache prefix where possible.

The API supports:

  • session.steer(...) with immediate acceptance IDs and bounded pending input.
  • .nextSafeBoundary to end ordinary text decoding at a supported boundary, or .nextStepBoundary to let the current step finish.
  • Generation.steering(.applied(ids)) and .steering(.failed(failure)) for tracking accepted instructions.
  • Explicit response targeting through latestResponse, plus canSteer for UI state.
  • Ordered tool-result handling, speculative-cache finalization, cancellation cleanup, and caller isolation for actor-owned sessions.

Steering does not undo emitted output or completed tool effects. Declared reasoning models and framed protocols finish their current step, including when .nextSafeBoundary is requested. Generation limits apply separately to each model step.

Existing string consumers keep the same API. Exhaustive switches over Generation and GenerateStopReason must handle the new .steering and .steered cases. Steering requires structured conversation history; sessions restored from a raw cache cannot accept it once that state is known.

Checklist

  • I have read the CONTRIBUTING document
  • I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the necessary documentation (if needed)

AI usage

  • I have read this PR description in full and approve it as my own, and it accurately describes the code changes.
  • AI usage disclosure: GPT 6 assisted with API review, regression tests, validation, and drafting the documentation.

Queue instructions for active responses and apply them at supported text
or step boundaries while preserving emitted output and completed tools.
Expose steering outcomes through Generation.steering and reuse verified
cache prefixes for continuation.

Handle completion races, cancellation, and caller isolation. Add unit
and real-model coverage, and document scheduling and compatibility.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants