Hold newly promoted tools off the wire until a cache-safe boundary - #1027
TheGreatAxios merged 4 commits into
Conversation
The tools array is the head of the provider's cached prompt prefix, so any change to it re-prefills the whole request. Whether that actually happens in real sessions is unmeasured: it shows up only as a billing and latency spike a turn later, with nothing tying it back to a mount. Log a digest when the set changes. Hashed, not verbatim — MCP tool descriptions are arbitrary-length server-supplied text and do not belong in the log stream. This replaces an earlier attempt that sorted the array by name. That was wrong: advertisedTools already orders deterministically, so sorting added no stability, and an alphabetical insert can land at index 0 and invalidate more of the prefix than appending does. A gate run carrying it measured cache-hit rate down 3-8 points across all four eval tiers.
Activating a tool mid-session used to grow the advertised set on the next turn, re-prefilling the provider's cached prompt prefix. Promotion now opens only the call gate, so the model can invoke the tool at once, while the wire set stays byte-identical until rotation, compaction, or session start commits it.
Critic · Comment (no blockers)Mid-session promotion opens only the call gate while the wire set stays byte-identical until a cache-safe boundary commits it via Findings
Notes
|
The digest gates the tool-set-changed log line, and the tools array heads the provider's cached prompt prefix: a schema-only change reshapes the wire bytes while leaving a name:description hash untouched, so the cache bust goes unlogged. Hash the schema too; the wire behavior is unchanged.
|
Digest gap fixed in |
Summary
Mid-session tool promotion used to grow the advertised tool set on the next turn, re-prefilling the provider's cached prompt prefix. Promotion now opens only the call gate, so the model can invoke the tool at once, while the wire set stays byte-identical until rotation, compaction, or session start commits it. A debug digest logs genuine tool-set changes.
Verification
bun run lintclean (oxfmt and oxlint, 0 warnings, 0 errors).bun run check: lint and build pass;tsc --noEmitreports only pre-existing errors in files outside this diff (summarizer, url-links, inference-response-kind test, vendor drift).Fixes CL-7868