Skip to content

Hold newly promoted tools off the wire until a cache-safe boundary - #1027

Merged
TheGreatAxios merged 4 commits into
mainfrom
cl-7868-any-mid-session-tool-set-change-costs-a-full-prompt-cache
Sep 14, 2026
Merged

TheGreatAxios merged 4 commits into
mainfrom
cl-7868-any-mid-session-tool-set-change-costs-a-full-prompt-cache

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Collaborator

Summary

Mid-session tool promotion used to grow the advertised tool set on the next turn, re-prefilling the provider's cached prompt prefix. Promotion now opens only the call gate, so the model can invoke the tool at once, while the wire set stays byte-identical until rotation, compaction, or session start commits it. A debug digest logs genuine tool-set changes.

Verification

  • Red: promotion-turn test plus adapter byte-identity test failed before the fix, pass after.
  • Targeted suites green: tool-search, assemble-runtime, director, exec runner (src and tests), TUI exit, codex/openai/grok Responses providers, openai-responses adapter — 179 pass, 0 fail across 11 files.
  • bun run lint clean (oxfmt and oxlint, 0 warnings, 0 errors).
  • bun run check: lint and build pass; tsc --noEmit reports only pre-existing errors in files outside this diff (summarizer, url-links, inference-response-kind test, vendor drift).

Fixes CL-7868

The tools array is the head of the provider's cached prompt prefix, so
any change to it re-prefills the whole request. Whether that actually
happens in real sessions is unmeasured: it shows up only as a billing and
latency spike a turn later, with nothing tying it back to a mount.

Log a digest when the set changes. Hashed, not verbatim — MCP tool
descriptions are arbitrary-length server-supplied text and do not belong
in the log stream.

This replaces an earlier attempt that sorted the array by name. That was
wrong: advertisedTools already orders deterministically, so sorting added
no stability, and an alphabetical insert can land at index 0 and
invalidate more of the prefix than appending does. A gate run carrying it
measured cache-hit rate down 3-8 points across all four eval tiers.
Activating a tool mid-session used to grow the advertised set on the next turn, re-prefilling the provider's cached prompt prefix. Promotion now opens only the call gate, so the model can invoke the tool at once, while the wire set stays byte-identical until rotation, compaction, or session start commits it.
@linear-code

linear-code Bot commented Sep 14, 2026

Copy link
Copy Markdown

CL-7868

@TheGreatAxios

Copy link
Copy Markdown
Collaborator Author

Critic · Comment (no blockers)

Mid-session promotion opens only the call gate while the wire set stays byte-identical until a cache-safe boundary commits it via flushPromotions. Deferral is sound and the boundaries are complete.

Findings

  • src/session/assemble-runtime.ts — gate/wire split is airtight: computeAdvertised reads wireActivated, isAdvertised reads live activated, and the activated wrapper forwards exactly the four ActivatedToolTracker methods, so nothing is dropped. flushPromotions preserves activation order and dedupes via wireActivatedSet.
  • src/tui/runner/session.ts + src/exec/runner.ts — flush lands at session start/resume (activate + flushPromotions before first build) and at compaction onFolded in both TUI and exec. Rotation (newSessionclear()) resets gate and wire together with a fresh transcript, so nothing strands. No boundary is missing: every other mid-session recompute (interrupt rebuild, model-switch re-gate) reads wireActivated and stays byte-stable by construction.
  • Gate-only callability is honest, with test evidence: dispatch consults the live gate (dynamicRunner.setCallGateisAdvertised), never the wire array (src/tui/dynamic-tool-runner.ts:88, src/tui/runner/session.ts:491), and the suite asserts end-to-end dispatch of a promoted-while-off-wire tool (tool_search then MCP dispatch with the gate onallowed.content is "saved" while the director wire omits the tool).
  • src/agent/director.ts — digest log is debug-gated and change-gated (no noise), and hashing instead of verbatim is right for server-supplied MCP text. One gap: it hashes name:description only, so an inputSchema-only change reshapes the wire bytes with an identical digest — a silent cache bust with no log line. Worth folding the schema into the hash.
  • Strict-provider residual: the old grok-Responses "cannot call a tool that was never declared" comment is deleted without live-provider evidence either way. Nothing in-repo validates outbound calls against the wire set, and adapter tests are serialization-only, so this is a monitor-not-block item — suggest a live smoke note for grok strict mode before relying on gate-only there.
  • #980 supersession: commit 361aa3f8 is not in history (force-push dropped it), but its code and its full rationale (rejected sort-by-name alternative, gate-run measurements) are preserved verbatim in 57fd0482. No unrecovered work.
  • Typecheck drift: not reproducible. With a fresh bun install, bunx tsc --noEmit exits 0 on both origin/main and this branch. The six "pre-existing" errors (summarizer, url-links, inference-response-kind test, intx-types/intx-inference vendor) reproduce only with stale node_modules — I confirmed this by symlinking a stale tree (errors appear) vs fresh install (clean), and summarizer.ts is byte-identical across both refs. Recommend re-running bun run check fresh before merge; there is no typecheck debt from this diff.
  • Verified here: 186 pass / 0 fail across the 11 claimed suites plus the grok adapter (12 files), and tsc --noEmit exit 0 on both refs. Not re-verified: live-provider cache-hit numbers and the pre-fix red run (structurally inevitable — the new tests reference flushPromotions).

Notes

  • src/agent/tool-search.ts ToolSearchDeps.promote doc says wire follows at "(compaction fold)" — start/resume also flush, so the parenthetical is slightly narrow.

The digest gates the tool-set-changed log line, and the tools array heads the provider's cached prompt prefix: a schema-only change reshapes the wire bytes while leaving a name:description hash untouched, so the cache bust goes unlogged. Hash the schema too; the wire behavior is unchanged.
@TheGreatAxios

Copy link
Copy Markdown
Collaborator Author

Digest gap fixed in cb5db51da: toolSetDigest (src/agent/director.ts) now hashes name:description:inputSchema instead of name:description, so an inputSchema-only change moves the digest and the cache bust gets its log line. Wire behavior is unchanged — updateToolDefinitions still always assigns, the digest only gates the debug log. Test: src/agent/director.test.ts gains a toolSetDigest block proving identical sets share a digest and a schema-only change alters it (verified it fails pre-fix). Verification, all green after a fresh bun install: director suites 76 pass / 0 fail, tool-search + assemble-runtime 51 pass / 0 fail, full suite 7471 pass / 0 fail across 511 files, bun run lint / typecheck / build all exit 0. Not merging — leaving that to you after CI green.

@TheGreatAxios
TheGreatAxios merged commit f4256a4 into main Sep 14, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant