Skip to content

Give Muse Spark tool-discipline rules - #981

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-7869-muse-spark-runs-to-the-turn-budget-without-tool-discipline
Sep 14, 2026
Merged

TheGreatAxios merged 2 commits into
mainfrom
cl-7869-muse-spark-runs-to-the-turn-budget-without-tool-discipline

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Collaborator

Closes CL-7869.

At medium reasoning effort the model does not reliably stop a tool loop. Given a bounded task and a read_file / grep / edit_file tool set, it re-reads files it has already read and keeps calling until the harness turn budget stops it.

Measurement

Scripted agent loop, two-file fixture with a buffer-aliasing bug, deterministic mocked tool results, 8-turn ceiling, terminate on DONE. The two prompts are identical except for three appended rules:

prompt / effort turns tool calls reasoning tokens input tokens
plain / medium 8 — hit the ceiling, never finished 9 586 13,612
terse / medium 3 3 271 3,198
plain / minimal 3 3 37 2,858
terse / minimal 3 3 36 2,892

4.3x the input tokens and a non-terminating run, from three missing sentences.

Note the interaction with CL-7867: at minimal the model terminates either way. The rules earn their keep exactly at the rungs where each wasted turn is most expensive, so lowering the default effort and adding the rules are complementary — the rules are what keep a future medium/high run from degenerating.

The measurement is generous to the plain prompt: the harness re-issued each turn's request to capture raw output items, so the real cost of the 8-turn run is higher than the table shows, not lower.

Where the rules live

Appended at the tail of the system prompt. That position is prefix-safe — measured on the same endpoint, tail appends hold a 99.1% cache hit (even when the appended value changes every turn) while an edit at the head drops it to 9%.

Also pinned

The model batches independent tool calls on its own — three read_file calls in one turn — and the gateway default already allows it. Sending parallel_tool_calls: false collapses that to one call per turn. hostQuirks.parallelToolCalls is correctly unset today; the added test is what keeps it that way, since flipping it would silently triple turn counts.

Testing

bun run typecheck, bun run lint, full suite: 8890 pass. The 3 failures are pre-existing on main.

The 8-vs-3 result is one fixture and one task shape — it establishes the failure mode is real and that the rules address it, not the size of the effect across the tier cases. The capability suite with --repeats 5 is what sizes it.

@linear-code

linear-code Bot commented Sep 13, 2026

Copy link
Copy Markdown

CL-7869

@TheGreatAxios
TheGreatAxios force-pushed the cl-7869-muse-spark-runs-to-the-turn-budget-without-tool-discipline branch from bef3e48 to 20cbc8f Compare September 13, 2026 18:02
@TheGreatAxios TheGreatAxios changed the title feat(director): give Muse Spark tool-discipline rules Give Muse Spark tool-discipline rules Sep 13, 2026
@TheGreatAxios

Copy link
Copy Markdown
Collaborator Author

Critic review · Approve

Appends Muse-Spark-only tool-discipline rules at the system-prompt tail for main sessions and leaves, composed before super() so they reach the wire.

Findings

No blocking findings. Verified against the diff and main-branch sources:

  • src/agent/director.ts — composing before super() is required, not stylistic: createChatDirector resolves modelFamilyPolicy from options.provider and the base director replays its own copy each turn, so post-super() appends never send. The new on-the-wire tests in src/director.test.ts pin exactly this (an earlier cut was a silent no-op).
  • src/subagent/run.ts — the added 9th positional arg lines up with the new trailing toolDisciplineRules optional param on SubAgentDirector; it is the only product-code construction site, so leaf coverage is complete and existing positional test callers are unaffected.
  • src/subagent/provider-family.ts — family detection is model-driven (muse-spark prefix match), so the rules follow the model across provider names. No other family is touched.
  • Tests are real: wire-level prompt assertions, policy unit test, and a parallel_tool_calls-unset pin. Eval scope (one fixture, 8-vs-3 turns) is honestly caveated in the body. All PR checks pass.

Should-fix (non-blocking, follow-up)

  • src/agent/model-family-policy.ts — "Never re-read a file you have already read this session" is absolute; read-after-edit to verify a change is a legitimate re-read. Consider "unless the file changed since you last read it."
  • Composed prompt is fixed at construction, so a mid-session /model switch to/from muse-spark leaves stale rules. Rare path; acceptable as-is.

Notes

  • The 99.1%-vs-9% cache figures are stated as measured on the endpoint; taking the body's word for it, tail placement is still the right default regardless.

At medium reasoning effort the model does not reliably stop a tool loop:
on a two-file fixture with a bounded fix it re-read files it had already
read and ran out an 8-turn ceiling without finishing. The same run with
three rules appended finished in 3 turns on 4.3x fewer input tokens.

The rules ride at the tail of the system prompt, which is prefix-safe —
tail appends hold a 99.1% cache hit while a head edit drops it to 9%.

Adds a `muse` model family and pins the Responses quirks: the model
batches independent tool calls on its own, and sending
parallel_tool_calls: false would collapse that to one call per turn.

CL-7869
The rules were appended to the director's own copy of the system prompt
after calling super(). The base director keeps its own copy and sets
options.systemPrompt from it on every ordinary turn, so withCurrentTools'
`?? this._systemPrompt` fallback never fired and the composed prompt was
built but never sent. A Muse Spark session received the base prompt
verbatim: the whole change was a no-op, and every existing test passed.

Compose before super() so the base director holds the composed prompt,
and give the compaction governor the same text it will actually be
estimating.

Sub-agent leaves get the rules too. They are the case the ticket names —
a spawned worker running to its turn budget — and SubAgentDirector never
received them at all.

Adds the test that would have caught this: assert the prompt on the
emitted infer action, not the prompt the director was constructed with.

CL-7869
@TheGreatAxios
TheGreatAxios force-pushed the cl-7869-muse-spark-runs-to-the-turn-budget-without-tool-discipline branch from a45ec38 to cb0665f Compare September 14, 2026 05:05
@TheGreatAxios
TheGreatAxios merged commit 9e2774f into main Sep 14, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant