feat(llm): reasoning budget for local tool steps and progress-note replies (F49, F50) - #453
Merged
Merged
Conversation
Adds localModels.reasoningBudgetTokens (default 1500, 0 = unbounded, else [64, 32768]) to the user file and the runtime config. Config v67 -> v68: the upgrade is additive (an older file has no field and takes the default) and 67 joins the accepted input versions; the v67 step (tools.shell.defaultTimeoutMs, agent.readScope) is untouched. The field bounds the GBNF think prelude of a local reasoning model; the grammar and prompt changes land in the F49 commit.
…onoured on the built prompt
A grammar-served reasoning model thought without bound before every tool
call: the fusion bench measured 500–7,152 reasoning tokens per step at
4.5 tok/s (84–96 % of each completion, one step of 22 minutes), and
`localModels.thinking` only reached server-templated prompts.
Grammar: the base grammar's prelude body is `think-char{0,N}` (Gemma:
`channel-char`) with N = reasoningBudgetTokens × 4 characters, single-width
units so the bound is in characters; past N only the close sentinel is
admitted. Budget 0 keeps the unbounded `think-fragment*` text. The grammar
string stays ~5.3 KB — llama.cpp expands `{0,N}` server-side into N nested
optional rules (6,000 at the default). Two per-request variants in
`reasoning-prelude.ts`: `withUnboundedReasoningPrelude` on the forced final
step (a reply/finish is never cut mid-thought) and `withoutReasoningPrelude`
under thinking: off. `buildGrammarForTools` keeps its per-base cache; the
base slot count grows to 8 for the variants.
Thinking switch: `qwen-think` carries `promptThinkingDisabledMarker`
(`<think>\n\n</think>\n\n`, the template's own disabled rendering);
`thinkingDisabledOnBuiltPrompt` keys `off` on it. The built prompt ends with
the marker, the request grammar takes the plain root, the completion is
parsed as starting outside a think block, the repair prompt strips and
re-appends the marker, and the prompt invariant accepts it. Gemma 4's turn
framing has no marker and is left as is; on/auto unchanged; the template
path keeps its own switch.
Visibility: `llm_raw_completion` and the `llm_completion` trace event carry
`reasoningTokens` — the text before the close sentinel priced at the same
four characters per token, so a cut step reads as `>= budget`. The TUI has
no output-token field on the step line, so nothing was added there.
Tests: build-grammar (bound, language up to and past the bound, budget 0,
string size, both variants, tool-name rewrite on variants), profile-manager
threading, server-template-policy predicate, profile-invariants marker,
build-prompt tail for off/on/auto/gemma/suppressed, step-executor (base
grammar, final-step lift, reasoningTokens, thinking: off end to end, repair
marker, thinking: on), trace-recorder forwarding, reasoning-budget units.
…s only on a sole reply
plombeer31
force-pushed
the
harness/16-local-turn-shaping
branch
from
September 16, 2026 22:03
727baf2 to
4e45059
Compare
plombeer31
force-pushed
the
harness/15-session-boundaries
branch
from
September 16, 2026 22:03
40d52cd to
1ff5148
Compare
plombeer31
changed the base branch from
harness/15-session-boundaries
to
main
September 17, 2026 10:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two local-model turn-shaping fixes from re-running the bench-2 matrix with a second local model (Qwen 3.8 27B, findings 32–33):
localModels.thinking: offhonoured on the built prompt. Qwen-shaped profiles get a prefilled<think>and a grammar whose reasoning prelude was unbounded; at 4.5 tok/s the model spent 500–7,000 reasoning tokens (84–96 % of every completion, one step 22 minutes) before each one-line tool call. NowlocalModels.reasoningBudgetTokens(default 1,500;0= unbounded) bounds the prelude in the GBNF grammar by characters (think-char{0,N}, N = 4 × budget; same for the Gemma 4channelprelude) — past the budget the grammar admits only the close sentinel, so the model must close and emit the call; the forced final step keeps an unbounded prelude.localModels.thinking: off, which until now only reached server-templated prompts, also applies to the built prompt for prefilled profiles: the prompt ends with the template's own disabled marker (<think>\n\n</think>\n\n) and the grammar drops the prelude. Thellm_completiontrace event carriesreasoningTokens(an estimate from the text, so a cut reads as>= budget).replybatched with work tools is a progress note, not the end of the turn. The model batchedos.shell.run+reply "(collecting file contents…)"; the runtime ran both and ended the turn with nothing built, because anyreplywas terminal wherever it sat in the batch. Now areplythat shares a completion with non-terminal calls is taken out before validation, the other calls run, and the text is recorded as a progress note (assistant_replyturn flaggedprogressNote, tool resultdetails.progressNote: true, step summaryprogress note + N tools,step_finishedflagged); the model is told once per turn, through the notice channel, that a reply ends the turn only when it is the sole call. A solereply, the forced final step andfinishbehave exactly as before. On the OpenAI-compatible HTTP stream a note goes out as aprogress_noteevent (opt-in extensions), never as a content delta, and the final payload picks the last non-note reply; Telegram, Discord, the task runner and fusion worker results skip notes.Config version 67 → 68:
localModels.reasoningBudgetTokens.Why
A local model that thinks in the open pays for every reasoning token at local speed; without a bound one step can take longer than the whole task. And a status line to the user should not silently be the end of the task — the transcript now shows it as what it was.
The budget is deliberately generous (1,500 tokens ≈ 6,000 characters — above every Gemma step observed and three times Qwen's median) and the cut is visible in the trace;
0restores today's behaviour.How it was verified
npm run lintcleannpx vitest run src/config src/llm src/prompt src/agent src/http src/runtime src/tracing— 199 files / 2,856 tests greensend-message-concurrencybaseline andgit-remote-sync's 15 s timeout under machine load (src/tools/os/gitre-run alone with--no-file-parallelism: 16 files / 109 tests green)thinking: offon a qwen profile ends with the disabled marker and the grammar has the plain root; the trace field; existing grammar/prompt pins updated deliberately.[shell, reply]→ turn continues, note recorded, next step runs, a sole reply then ends the turn;[reply, shell]same;[reply]alone ends; forced final[reply]ends; notice once per turn; streaming[note, final]yields one content delta with only the final text plus oneprogress_noteevent; without the change 5 of the 9 new executor/loop tests fail.Stacked on #452 (
harness/15-session-boundaries); merge in order.Rebased with the whole stack onto
mainat v0.6.2 (2026-09-17). Full suite on the stack head (#459): 928/929 files, 10,815 passed, 1 failed — the pre-existingsend-message-concurrencybaseline.