feat(local-llm): slot pinning, task ceiling while waiting, measured context and throughput, SWA launch flags, first-token progress (F12–F16) - #436
Open
plombeer31 wants to merge 7 commits into
Conversation
…equest, pinned after A session's first request (and its first after a pool resize) is pending: id_slot -1 with cache_prompt true, so llama-server selects the slot by prefix similarity (LRU otherwise) and a resumed session lands on the slot that still holds its prompt. The step executor pins the id_slot the completion names; the in-step repair and every later request of the session ride that slot. A stable-prefix change no longer rotates the session to a cold slot. Side calls (reflection, link generation, voting, distillation, the query rewriter) resolve their slot per call via sideCallSlotId(): the reserved reflection slot when the pool has two or more slots, else -1. In managed mode the pool is one slot until the first /props, so the boot-time reservation was -1 forever and every side call let the server pick any idle slot, the main loop's included. The reservation now prefers a slot no session is pinned to.
agent.task.maxDurationMs was checked only when a step started; the request itself carried the user's abort signal alone, so a turn parked in a provider wait ran past its two-hour window and had to be closed by hand. Each step's completion request now gets a deadline signal — the user's signal composed with the task's remaining time. When it fires mid-request the loop reads it before any cancellation check, treats it as time_ceiling and runs the reserved summary step on a fresh five-minute deadline of its own (llama-server keeps decoding the abandoned request until it notices the closed connection, so the summary queues behind it). A summary that overruns that preserves the max_steps outcome. Tools never see the deadline signal.
The auto context size is costed from the model's own attention layout instead of a scale on the file size: estimateKvBytesPerToken sums, over the layers, 2 × KV heads × head dims × bits per value (turbo3 3.5, q8 8.5, f16 16), weights sliding-window layers by min(window, ctx) / ctx and charges recurrent layers nothing. On Gemma 4 31B's header (60 layers, 5:1 sliding pattern, 16/4 KV heads, 256/512 dims, window 1024) that is ~19 KB/token at 131,072 with turbo3 against 13.7 measured — within 2×, where the file-size scale was 4-40× off. The file-size scale stays as the fallback for a model whose header cannot be read. MAX_AUTO_CONTEXT is 262,144, clamped by the trained context as before. At managed daemon start, once /health is OK, one 64-token completion measures the single-stream decode speed; it is recorded next to the pid file, held on the model profile manager (read lazily and on every /props refresh), shown on the CLI/TUI ready line and stated in the ### fusion machine facts as "~N tok/s single stream". fusion.delegate: on a slot-affine (local) leg a call that names no maxWorkers runs one worker unless llm.runMode.fusion.workers is pinned (ResolvedRunMode.workersPinned); an explicit maxWorkers is honoured up to the pool; a cloud leg keeps the configured default.
…che reuse A pure-TS GGUF header reader (KV section only, 8 MiB budget, never the tensors) gives the managed launch the model's architecture, trained context, layer count and attention shape: per-layer KV head counts, the sliding window and its layer pattern, the _swa head dims, and the hybrid tells (ssm.* keys, full_attention_interval, known recurrent architectures). A model with sliding-window layers or a hybrid/recurrent architecture cannot roll its cache back to a divergence point, so its prompt cache is reusable only whole: classifyPrefixReuse says "none" and the profile manager stamps profile.prefixReuse from /props.model_path for the packer. localModels.managed.swaFull (auto | on | off) launches such a model with --swa-full — whole-context KV in the sliding layers, any prefix reusable again — when configured on, or on auto when the full-SWA estimate (per-token KV × total / non-sliding layers, capped at 8×, flagged as an estimate) fits the launch's KV budget; a hybrid gets no flag. The fitted context, the flag decision and the reuse verdict go to the daemon log and to a pid-stamped launch record the runtime reads. Read synchronously on the launch path so the timer-driven health wait still starts before any clock moves. An unreadable header launches exactly as before.
v65 → v66. Additive: an older file has no field and gets "auto".
While a stream waits for its first byte the client polls GET /slots every 15 s with a 3 s per-poll deadline. An answer that shows work — this session's slot processing, any other slot processing (the request is queued behind it), or any slot's state changed since the last answered poll — extends the wait up to localModels.firstTokenTimeoutMs. Only a server that keeps answering with every slot idle and nothing moving for a whole idle budget (requestTimeoutMs) ends the wait early, as a first-token-stall whose message names the retry (which reuses the session's pinned slot, F13). A poll that times out is no verdict: /slots hangs while a slot evaluates a large prompt, so a hang means busy and the first-token timer stays the backstop exactly as before. The watch stops at the first byte; progressWatch: false turns it off.
Bring the tree to the state the package has once the other packages are merged around it: shared seams (config version, prompt tail order, fusion facts) resolved the same way as in the integrated build.
plombeer31
force-pushed
the
harness/10-errors
branch
from
September 16, 2026 22:03
b9dc4b7 to
9ff9b89
Compare
plombeer31
force-pushed
the
harness/11-llama
branch
from
September 16, 2026 22:03
e57453c to
bf1fd11
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Before: slot assignments lived in memory, were cleared on every resize and never persisted, so a resumed TUI re-evaluated 38K tokens cold and a no-data abort moved the step to another slot;
agent.task.maxDurationMswas checked only when a step started, so a turn ran past its ceiling inside a provider wait; the KV-per-token estimate came from file size (40x off),MAX_AUTO_CONTEXTwas 32,768, and local fan-out width was guessed; sliding-window and hybrid models were launched with no cache-reuse flags and the packer did not know they reuse nothing; the first-token wait was a plain timer.After: a session's first request (and the first after a resize) sends
id_slot: -1withcache_prompt: trueso llama-server picks the slot by prefix similarity, the answer is pinned and retries stay on it; side calls (rewriter, reflection, links, consolidator) go to the reserved reflection slot; each completion request gets a deadline signal from the task ceiling and a ceiling mid-request runs the finalization step with its own 5-minute deadline;estimateKvBytesPerTokenreads the GGUF header (gguf-metadata.ts, pure TS) — within 2x of the 13.7 KB/token measured for Gemma 4 31B —MAX_AUTO_CONTEXTis min(trained, 262,144), the managed daemon runs a 64-token throughput probe at start and the machine facts say "~N tok/s single stream", and a slot-affine pool defaults to one worker unlessrunMode.fusion.workersis pinned; sliding-window and hybrid architectures getprefixReuse: "none"and--swa-fullunderlocalModels.managed.swaFull(auto/on/off, auto only when the estimate fits); while waiting for the first byte,/slotsis polled every 15 s and the wait extends while the slot shows progress, falling back to the timer when the poll itself times out. Config bump for the new field.Why
Local compute is the scarce resource: every cold re-read is minutes. Slot choice by similarity is never worse than cold; a partial match re-evaluates from the divergence point. Includes the reconcile commit that collapses a duplicated
prefixReusefield on the model profile onto thePrefixReusetype.How it was verified
npm run lintcleannpx vitest run --minWorkers=1 --maxWorkers=3 src/agent src/cli src/config src/llm src/llm/run-mode src/local-llm src/memory/consolidator src/memory/links src/memory/reflection src/memory/retrieve src/memory/voting src/prompt src/runtime/bootstrap.test.ts src/runtime/llm-link-attempt.test.ts src/tools/fusion src/tui/local-models/local-models-orchestrator.test.ts— 247 files / 3626 tests green (baseline failures: none)Stacked on #435; merge in order.