t{r["turn"]} · {tok_s} · '
+ f'{_fmt_tok(r["completion_tokens"])} tok{cancel}{badge}
{"".join(out)}
-Forge takes a Hugging Face repo and turns it into an MTPLX-ready MTP model: convert to MLX, train the MTP adapter, verify that the result is actually faster and still exact, and publish back to the Hub if you want to share it. The honest part matters: Forge measures before and after on your hardware and shows you the verdict ("Depth 1 is fastest: 227.1 to 296.1, 1.30x") rather than assuming the adapter helped. Available in the app and as `mtplx forge`.
+Forge takes a Hugging Face repo and turns it into an MTPLX-ready MTP model: convert to MLX, train the MTP adapter, verify that the result is actually faster and still exact, and publish back to the Hub if you want to share it. The honest part matters: Forge measures before and after on your hardware and shows you the verdict ("Depth 1 is fastest: 227.1 to 296.1, 1.30x") rather than assuming the adapter helped. Available in the app and as `mtplx forge` subcommands.
-The official catalog lives on Hugging Face under [Youssofal](https://huggingface.co/Youssofal): Qwen 3.5 (4B, 9B), Qwen 3.6 (27B, 35B MoE) in speed, balance, and quality builds, plus Gemma 4. The app recommends from these based on your hardware.
+MTPLX does not support attaching a separately supplied MTP sidecar to an arbitrary MLX trunk. Matching architecture fields, tensor shapes, or provenance labels cannot prove that the head was trained against those exact trunk weights. Use a complete model that already includes its matching MTP weights, or use Forge to build and verify an artifact from its original source checkpoint.
+
+The official catalog lives on Hugging Face under [Youssofal](https://huggingface.co/Youssofal): Qwen 3.8 27B (Bare Speed, Optimized Speed, Optimized Quality, each with an FP16 build for M1 and M2), Qwen 3.6 (27B, 35B MoE) in speed and quality builds (the 35B MoE adds a balance build), Qwen 3.5 (4B, 9B), plus Gemma 4. The app and the CLI recommend from these based on your hardware.
## The server
-`mtplx start` (or the app's play button) serves an OpenAI-compatible API on `127.0.0.1:8000`: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, plus an Anthropic-compatible `/v1/messages` with streaming, tool calls in both styles, `/health`, and `/metrics`. Claude Code, Cline, Continue, Open WebUI, curl, the openai and anthropic Python clients: if it speaks the API, it works. The app and CLI share one server, so `mtplx start` attaches to the app's running model instead of loading a second copy.
+`mtplx start` (or the app's play button) serves an OpenAI-compatible API on `127.0.0.1:8000`: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, the optional `/v1/embeddings` and `/v1/rerank` (see below), plus an Anthropic-compatible `/v1/messages` with streaming, tool calls in both styles, `/health`, and `/metrics`. Claude Code, Cline, Continue, Open WebUI, curl, the openai and anthropic Python clients: if it speaks the API, it works. The app and CLI share one server, so `mtplx start` attaches to the app's running model instead of loading a second copy.
```bash
curl http://127.0.0.1:8000/v1/chat/completions \
@@ -69,7 +84,36 @@ curl http://127.0.0.1:8000/v1/chat/completions \
-d '{"model":"mtplx","messages":[{"role":"user","content":"hi"}],"stream":true}'
```
-Sessions survive: a warm-prefix session bank keeps multi-turn chats fast, and an optional SSD cache restores sessions near-instantly across restarts.
+Sessions survive: a warm-prefix session bank keeps multi-turn chats fast, and a default-on SSD session cache restores sessions near-instantly across restarts (disable with `--ssd-session-cache off`).
+
+### Embeddings and reranking
+
+The same daemon can serve retrieval models, so a RAG or agent-memory setup does not need a second inference server beside MTPLX. Point it at any MLX embedding or reranker model — Hugging Face id or local path, optionally with a `REF=served-id` alias:
+
+```bash
+mtplx serve \
+ --embedding-model mlx-community/Qwen3-Embedding-8B-4bit-DWQ \
+ --reranker-model vserifsaglam/Qwen3-Reranker-4B-4bit-MLX
+```
+
+```bash
+curl http://127.0.0.1:8000/v1/embeddings \
+ -H 'Content-Type: application/json' \
+ -d '{"model":"Qwen3-Embedding-8B-4bit-DWQ","input":["hello","world"]}'
+
+curl http://127.0.0.1:8000/v1/rerank \
+ -H 'Content-Type: application/json' \
+ -d '{"query":"where is the cache?","documents":["the cache lives in ~/.mtplx","unrelated text"]}'
+```
+
+Both flags repeat, so several models can be served at once and picked per request via `"model"`. Listing the same reference as both an embedder and a reranker loads **one** copy of the weights and serves both roles from it. Retrieval models load on first request and are capped by `--retrieval-max-resident` (default 2), which unloads the least recently used one beyond the cap — an unused endpoint costs nothing. `/v1/models` stays chat-only by default so chat clients that enumerate models never offer an embedder as a conversation target; list retrieval models with `?capability=embedding` or `?capability=rerank` (every entry carries its `capability`), and a chat completion that requests a retrieval id gets a clear 400 rather than a silent answer from the chat model.
+
+These models do not go through the MTP path, and that is deliberate: multi-token prediction makes *next-token* decoding cheaper, which means nothing for a model that returns a vector instead of a token stream. Configure them in the app under Settings → Retrieval endpoints, or persist them in `~/.mtplx/config.toml` as `embedding_models` and `reranker_models`. With nothing configured the endpoints answer 404 and chat behaves exactly as before. One safety gate: checkpoints that bundle their own Python inference code (the jina embedding/reranker MLX releases do) are refused with a 403 until you opt in with `--retrieval-trust-remote-code` (or `retrieval_trust_remote_code = true` in the config file) — a model download never gains code execution just by being pointed at.
+
+Sampler controls cover `temperature`, `top_p`, `top_k`, and the OpenAI penalty pair `presence_penalty` / `frequency_penalty` — per request, as server defaults (`--default-presence-penalty` / `--default-frequency-penalty` on `start`/`serve`/`quickstart`), or live via `mtplx settings set` and the app's Presence Penalty dial. Penalties default to 0, which is an exact no-op that preserves MTP exactness. Qwen's guidance: leave them at 0 for coding and agent work; ~0.5–1.5 presence penalty helps creative writing or when a model loops on itself.
+
+Concurrent scheduler modes, ownership guarantees, and backend-specific
+implementations are documented in [Concurrency modes](docs/concurrency.md).
## CLI quick reference
@@ -81,20 +125,21 @@ mtplx pull for in-cell line breaks in + // markdown tables. cmark hands raw HTML back as literal tag text, so + // the cell would render "
" verbatim AND keep the whole cell as one + // unbreakable token that overflows the fixed column frame into its + // neighbour (the "pile of overlapping text"). Convert
variants to + // real newlines at this single choke point so cells wrap. Comparison + // operators ("a < b") are untouched: the pattern requires "br". + // Deliberately
-only — other raw tags (, ) still render + // raw; strip-all would mangle "<"/">", so grow an allowlist if needed. + let readable = text.replacingOccurrences( + of: #"
` variants inside markdown table cells (PR #273 by @El-Patronum). +- Capture tooling persists exact completion token ids on all three lanes (PR #330 by @CharliePetch). + +## Experimental, off by default + +- `MTPLX_FUSE_PROJ`: load-time projection fusion for GDN, attention, and MLP (port of PR #316 by @grzracz). +- `MTPLX_VK_CROSSROW`: crossrow wide-verify kernel. +- Draft-confidence tracing and confidence-gated draft width, groundwork for head calibration. +- Marathon postcommit protection for very long sessions. + +## Docs + +- New: `HISTORY.md`, the dated record of putting native MTP on Apple Silicon. + +## Updating + +- App: Sparkle offers 2.9.2 automatically, or download the DMG at mtplx.com. +- CLI: `pip install -U mtplx` or `brew upgrade mtplx`. diff --git a/docs/research/native-mtp-on-mlx.md b/docs/research/native-mtp-on-mlx.md index 570316734..6d2ddca78 100644 --- a/docs/research/native-mtp-on-mlx.md +++ b/docs/research/native-mtp-on-mlx.md @@ -1,3 +1,5 @@ +*(v0.1-era research snapshot — see docs/releases/ for the current state.)* + # Native MTP On MLX MTPLX explores the built-in MTP heads in Qwen3-Next models on Apple Silicon. diff --git a/docs/runtime-contract.md b/docs/runtime-contract.md index 32226c224..18277ba7f 100644 --- a/docs/runtime-contract.md +++ b/docs/runtime-contract.md @@ -4,7 +4,7 @@ Verified models include `mtplx_runtime.json`. ```json { - "mtplx_version": "0.3.0", + "mtplx_version": "2.4.2", "arch_id": "qwen3-next-mtp", "mtp_depth_max": 3, "recommended_profile": "stable", @@ -20,4 +20,8 @@ Verified models include `mtplx_runtime.json`. } ``` +`mtplx_version` is stamped with the runtime's real version at build time, and +`recommended_profile` above is illustrative — `turbo` is the shipped default +profile for the quantized flagships. + Architecture-compatible models without this contract are not supported by default. diff --git a/docs/server.md b/docs/server.md index 71af5e179..ce310f180 100644 --- a/docs/server.md +++ b/docs/server.md @@ -1,12 +1,15 @@ # Server -The v0.1 server target is OpenAI-compatible local serving, with Anthropic +The server target is OpenAI-compatible local serving, with Anthropic Messages compatibility available for coding harness smoke tests. ```bash mtplx serve --host 127.0.0.1 --port 8000 --no-stats-footer ``` +See [Concurrency modes](concurrency.md) for scheduler selection, ownership +rules, and model/backend-specific implementations. + Endpoints: - `GET /health` @@ -18,7 +21,24 @@ Endpoints: - `GET /admin/sessions` - `POST /admin/cache/clear` -Binding to a non-localhost host requires an API key: +## Sharing on your network (other devices, Parallels/VM guests) + +The default bind is `127.0.0.1`: only this Mac can connect. To reach MTPLX +from other devices — or from a Windows VM in Parallels/VMware/UTM on the same +Mac, which arrives over the virtual network rather than loopback — bind all +interfaces. Non-localhost binds require an API key; if the key file doesn't +exist yet it is created with a fresh key and printed once: + +```bash +mtplx serve --host 0.0.0.0 --port 8000 --api-key-file ~/.mtplx/api-key +``` + +Startup prints a `Network OpenAI API Base URL` (your Mac's LAN address, e.g. +`http://192.168.1.20:8000/v1`). On the other machine, point any +OpenAI-compatible client at that base URL with the printed key as the API +key (sent as a Bearer token). Parallels shared networking reaches the Mac's +LAN address directly; macOS may ask once to allow incoming connections — +click Allow. To pass the key inline instead of a file: ```bash mtplx serve --host 0.0.0.0 --port 8000 --api-key "$MTPLX_API_KEY" @@ -40,12 +60,15 @@ That helper disables Open WebUI's Ollama probe and background task generations so MTPLX only serves visible chat turns by default. For Anthropic Messages-compatible clients, point the client base URL at the -same local server root: +bare server root — no `/v1` suffix: ```text -http://127.0.0.1:8000/v1 +http://127.0.0.1:8000 ``` +The Anthropic SDK appends `/v1/messages` itself; a `/v1` base would request +`/v1/v1/messages`, which is not a registered route. + ## Android Studio Android Studio's external model provider should use the OpenAI-compatible URL @@ -66,6 +89,9 @@ setup, run: mtplx doctor android-studio --port 8008 ``` -Use `--no-stats-footer` for Open WebUI, Claude Code, OpenCode, and other -clients that treat assistant content as the only user-visible answer. Metrics +Since 2.5.3 the stats footer only appears on MTPLX-owned surfaces (the app +and the built-in browser chat); API clients such as Open WebUI, Claude Code, +and OpenCode never receive it, so no flag is needed for them. +`--no-stats-footer` still turns it off everywhere, and +`MTPLX_STATS_FOOTER_SCOPE=all` restores the pre-2.5.3 behavior. Metrics remain available at `/metrics`. diff --git a/docs/specs/2026-08-08-qwen35b-eight-way-mtp-design.md b/docs/specs/2026-08-08-qwen35b-eight-way-mtp-design.md new file mode 100644 index 000000000..44410488d --- /dev/null +++ b/docs/specs/2026-08-08-qwen35b-eight-way-mtp-design.md @@ -0,0 +1,249 @@ +# Qwen 35B eight-way MTP serving + +Status: approved design for PR #245 + +## Goal + +Serve up to eight independent Qwen3.6-35B-A3B requests through one fixed-width +speculative MTP cohort. Each request keeps its own prompt, sampling state, +logical KV rows, recurrent state, stop state, and cancellation state. The target +and MTP draft work execute in lockstep across the cohort so the model weights are +amortized across requests. + +MTP is the default. The server must never change a concurrent request to AR as +an automatic fallback. + +## Scope + +This change extends the existing PR #245 branch. It includes: + +1. A fixed-width A3B MTP driver for the real 35B Speed model. +2. Exact per-request sampling and speculative acceptance at MTP depth 1. +3. Server admission, streaming, cancellation, and error propagation for up to + eight concurrent requests. +4. A persistent Qwen launcher that starts in MTP mode and installs the width-8 + lane at construction time. +5. Unit, parity, live concurrency, kernel-engagement, and performance evidence. + +The implementation is split into reviewable commits in this same pull request. +No second pull request is created. + +## Non-goals + +- Reusing the dense Qwen 27B width-2 K2 cohort. Its route table and kernels do + not match the 35B MoE model. +- Forcing MTP depth 2 or 3. The served 35B Speed model uses its promoted depth-1 + contract. +- Changing model weights, quantization, expert layout, or published artifacts. +- Claiming a throughput win from configuration or microbenchmarks alone. +- Adding automatic AR fallback when a cohort cannot be formed or installed. +- Enabling the lane for other model families without their own construction + contract and measurements. + +## Public behavior + +The new scheduler mode is `mtp_batch`. Its required service settings are: + +- `generation_mode=mtp` +- `load_mtp=true` +- `depth=1` +- `max_active_requests=8` +- `decode_batch_max=8` +- a fixed cohort capacity of eight slots + +One request uses the existing optimized solo MTP route. When two or more +compatible requests are ready within the bounded gather window, they enter one +fixed eight-slot MTP cohort. Unused slots remain inert padding rows. Requests +that arrive after a cohort starts wait for the next cohort; mid-run refill is +not part of the first production version. + +AR remains available only through an explicit request or explicit service +configuration. The scheduler never selects it because request count increased. + +## Construction boundary + +The width-8 route is installed once during server construction. Installation +validates: + +- Qwen3.6-35B-A3B model and backend identity; +- MTP availability and depth-1 draft-head topology; +- dtype, quantization, group size, expert layout, hidden width, layer count, + vocabulary width, and target/draft tensor shapes; +- the exact fixed target verify shape `[B=8, T=2]` and its flattened `M=16` + projection and MoE shapes; +- required optimized target, draft, attention, and MoE callables; +- a numerical self-check against the unchanged route at the real shapes. + +The installer returns a typed, immutable lane containing prebound callables and +fixed geometry. An invalid contract fails startup clearly. The enabled hot path +does not re-read environment variables, revalidate model metadata, or try an +optimized route and silently fall back. + +## Request ownership + +Each admitted request owns one slot for the life of its cohort run. A slot owns: + +- prompt tokens and true prompt length; +- row-specific KV offsets and KV contents; +- row-specific GDN/recurrent state; +- latest target logits and hidden state; +- MTP draft state for the current speculative cycle; +- target and draft sampler configurations; +- an independent seeded RNG stream; +- committed tokens, stop matcher, token budget, and finish reason; +- cancellation event and output queue. + +The cohort may store these rows in shared batched allocations. Sharing an +allocation is not sharing context: all reads, writes, offsets, masks, rewinds, +and commits remain row-owned. + +## Decode cycle + +The production lane uses MTP depth 1. For each active row: + +1. Sample the next target token `x0` from that row's current target + distribution. +2. Produce one draft token `d` from the MTP head using that row's hidden state + and draft sampler. +3. Run one target verify forward over the fixed `[8, 2]` tensor containing + `[x0, d]` for each slot. +4. Compute speculative acceptance independently per row using the same target + and draft probability contract as `generate_mtpk`. +5. Apply the same acceptance, target-minus-draft residual sampling, bonus-token + policy, and commit ordering as `generate_mtpk`, independently per row. A + rejecting row replays only its own correction through the existing ragged + fold-in state machine. +6. Keep inactive, cancelled, and completed rows masked and pinned without + letting them affect another row's decisions. + +The cohort follows the installed speculative-bonus policy. When the bonus is +enabled, an accepting row samples it from the matching target distribution with +that request's RNG. When it is explicitly omitted, the cohort omits it too. The +batched path must match `generate_mtpk`; it does not invent a different MTP +algorithm. + +All sampler values that may differ by request are frozen in the request state. +The cycle may batch probability materialization, but it must not replace eight +independent RNG streams with one shared RNG sequence. + +## Admission and compatibility + +The service groups requests only when their construction-time execution +contract is compatible: model, depth, target strategy, tool/constraint route, +and other values that change the target or draft graph. Sampling values and +seeds are row state and may differ within one cohort. + +The admission window is bounded. A single request does not wait for seven peers; +it takes the existing solo MTP route. Two through eight ready requests use the +fixed width-8 lane. Waiting requests form the next cohort. + +## Streaming, cancellation, and completion + +Committed tokens are emitted to the owning request queue only. Every request +future reaches exactly one terminal state: completed, cancelled, or failed. + +Cancellation marks the row inactive and completes its future even when cache +filtering or cleanup fails. A partial cache mutation fails the whole cohort +closed through the existing PR #245 boundary; the mutated cohort is never used +again. + +Normal batched MTP does not promise a reusable generation-final cache state. +The stream uses PR #245's direct compatibility classification and releases the +client without waiting behind later foreground model work. Any safe history +rebuild remains bounded idle work. + +## Telemetry + +Telemetry is collected at admission and cycle boundaries without adding +per-layer or per-dispatch proof work to the model hot path. Health exposes: + +- scheduler mode `mtp_batch`; +- active lane `mtp_batch_width_8` while a cohort is decoding; +- real request count and fixed cohort capacity; +- a real-width histogram, including width 8; +- target verify cycles, accepted draft tokens, rejected draft tokens, and + acceptance rate; +- cancellations and last cohort error; +- installed optimized-kernel route identity from construction. + +Kernel engagement is also verified outside the measured path with the existing +dispatch census/profiler tools. + +## Failure handling + +- Construction mismatch: fail server startup. Do not install the lane. +- Cohort forward, draft, sampler, or cache error: fail every unfinished request + in that cohort, close the cohort, and build a fresh cohort for later work. +- Per-request cancellation: close that request promptly and mask its row. +- Client disconnect: signal the same cancellation event used by explicit + cancellation. +- Postcommit incompatibility: release the stream and schedule only the existing + bounded idle history route. +- Insufficient peers: use solo MTP for one request; never substitute AR. + +## Verification + +### CPU and construction tests + +- Red tests first for scheduler selection, no automatic AR fallback, fixed + width, request ownership, terminal future completion, and startup failure. +- Batched-vs-solo token parity for greedy requests at fixed width 8. +- Exact sampled parity with fixed per-request seeds and mixed sampler settings. +- Mixed prompt lengths, budgets, stops, cancellation points, and completion + order. +- A rejecting row must not alter an accepting row's tokens, RNG state, cache + offsets, or recurrent state. +- Construction tests pin all real 35B geometry and optimized callable routes. + +### Live guarded gates + +All model work acquires `/tmp/mtplx-gpu-exclusive.lock`. Only Qwen is loaded. +DeepSeek stays disabled. + +1. Establish unchanged solo MTP baselines three times. +2. Run the width-8 cohort with unique per-lane markers and fixed seeds. +3. Require eight separate request/session IDs, zero foreign markers, all + terminal events, and a real width-8 histogram. +4. Cancel two rows while six continue; require both cancellation acknowledgments + and clean completion of the survivors. +5. Cross the prior MLX resource-limit boundary with a long width-8 run. +6. Capture profiler or dispatch-census evidence for the installed M=16 target + verify and matching draft/MoE routes outside the timed run. +7. Compare aggregate and per-request throughput against unchanged solo MTP. + +Promotion requires correctness plus a measured aggregate MTP throughput gain. +The initial target is at least 1.20x aggregate throughput over serving the same +eight requests through unchanged solo MTP, with no context leak, no sampler +drift, no resource-limit error, and no silent stock or AR fallback. If the real +shape misses that target, the lane remains experimental and the service stays +on solo MTP while the receipt is reported honestly. + +## Rollout + +1. Land core driver and parity tests in PR #245. +2. Land server admission, streaming, cancellation, and telemetry in PR #245. +3. Run the guarded real-model gate without changing the persistent service. +4. Only after the gate passes, change the Qwen launcher default to `mtp_batch` + and restart it while holding the GPU lock. +5. Verify exact model identity, MTP availability, optimized route identity, + width-8 behavior, gateway health, DeepSeek disabled state, and lock release. + +Rollback points the launcher back to the last known-good solo-MTP runner. It +does not restore `ar_batch` as the default. + +## Adversarial review + +1. **Critical: one row can corrupt another through a shared cache or RNG.** + The design requires row-owned offsets/state, independent RNG streams, fixed + seed sampled parity, and rejecting-neighbour tests before any live rollout. +2. **Critical: the fixed M=16 call silently misses the optimized kernels.** + Construction pins and self-checks the callable route; external dispatch + evidence and end-to-end A/B are promotion gates. There is no enabled-path + fallback. +3. **Critical: 8-way MTP is correct but slower than solo MTP.** + The persistent launcher changes only after the guarded 1.20x aggregate gate. + A miss leaves the implementation experimental and restores solo MTP. +4. **Minor: requests arriving during a long cohort wait for the next cohort.** + Mid-run arbitrary-length refill is intentionally deferred. Adding it later + requires a separate parity and latency gate; it cannot be smuggled into this + implementation without evidence. diff --git a/docs/specs/2026-08-09-qwen35b-mtp-batch-numerics-profiles-design.md b/docs/specs/2026-08-09-qwen35b-mtp-batch-numerics-profiles-design.md new file mode 100644 index 000000000..b51de0cf5 --- /dev/null +++ b/docs/specs/2026-08-09-qwen35b-mtp-batch-numerics-profiles-design.md @@ -0,0 +1,508 @@ +# Qwen 35B MTP-batch numerics profiles + +Status: implemented; throughput remains the default; measured balanced candidates +remain explicit and are not promoted + +## Goal + +Increase the coding accuracy of the fixed Qwen3.6-35B-A3B B8/T2 MTP lane +while retaining most of its aggregate throughput gain. Operators can select a +fully constructed numerics route with one startup flag. Selection never adds a +per-token eligibility check, automatic fallback, or request-time route switch. + +The production target for the balanced route is at least 300 aggregate output +tokens per second under the existing greedy eight-request benchmark while +improving both HumanEval+ and MBPP+ over the current throughput route. + +## Baselines + +The unchanged PR #245 route is the throughput control: + +- fixed target shape `[B=8, T=2]`, flattened projection shape `M=16`; +- fixed draft shape `[B=8, T=1]`, flattened projection shape `M=8`; +- greedy median: 321.070 aggregate output tokens per second; +- default-sampler median: 161.500 aggregate output tokens per second; +- HumanEval: 151/164; +- HumanEval+: 144/164; +- MBPP: 335/378; +- MBPP+: 285/378; +- combined plus score: 429/542. + +The unchanged serialized solo-MTP B1 quality control scored: + +- HumanEval: 151/164; +- HumanEval+: 145/164; +- MBPP: 338/378; +- MBPP+: 289/378; +- combined plus score: 434/542. + +The objective is not to relabel B8 arithmetic as the reference. B1 remains the +quality control. + +The original PR workload was rerun unchanged: eight marker prompts, seeds +4200-4207, 256 maximum tokens, serial B1 followed by concurrent requests, and +three rounds. The throughput rerun reproduced the old 321 TPS result. The first +balanced candidate changed only the GDN A/B projections. It kept more than 97% +of greedy throughput and exceeded throughput under the default sampler in this +bracket: + +| Profile | Greedy B1 | Greedy concurrent | Default B1 | Default concurrent | +| --- | ---: | ---: | ---: | ---: | +| throughput | 159.745 | 323.570 | 141.187 | 170.406 | +| balanced v1: stock A/B | 158.899 | 315.777 | 141.053 | 172.891 | + +All values are median aggregate output tokens per second. Every marker remained +row-local. Neither B8 profile reproduced complete B1 response hashes; all three +rounds matched 0/8 responses. That is why balanced is an accuracy profile, not +an exactness claim. + +The strict B8-only EvalPlus audit rejected balanced v1. The completed generation +contained 68 B8 cohorts, one B7 cohort, and one B1 request, so the audit is +directional rather than promotable. It scored HumanEval 152/164 and +HumanEval+ 145/164, but MBPP 335/378 and MBPP+ 282/378. The eight tasks from +the non-B8 cohort had the same pass/fail outcomes under B1, throughput B8, and +balanced v1. The MBPP+ regression therefore was not caused by that cohort. +Balanced v1 was removed from consideration instead of being promoted from its +throughput result. + +Receipt SHA-256 values: throughput greedy +`8620050108f4016b412fd34fa933be24b2e6c3c2a504758d849a4f5835894517`, +balanced greedy +`6745ad8ac53104ffbbcfe01519495b3e85a7f50979ea5783c0cebe2243f694dc`, +throughput default +`7c4c3446a518a558133fe5f7e3bdeb84263aaba97fb55f6c8fa24a276382e298`, +and balanced default +`0255aaa3182950b6adc3508c1a55209889d7fbe855c62bd3aa89920808c580bc`. + +### Corrected B1 control and device-resident greedy route + +The `159.745` value in the paired table is not the PR #174 optimized-B1 +decode result. It is aggregate HTTP throughput for eight separate B1 requests, +including eight prefills and request overhead. The exact PR #174 K1 harness was +rerun unchanged before comparing B8: + +| Control | Repeat 1 | Repeat 2 | +| --- | ---: | ---: | +| long-code natural stop | 197.321 TPS | 198.883 TPS | +| short coding prompt, 143 tokens under the current tokenizer | 207.207 TPS | 207.044 TPS | + +The current branch reported the compiled target-prefix route, device draft +input, whole-MoE, GDN post-convolution, packed projections, row-owned router, +and combine-tail as installed. The historical PR #174 checkout reproduced +193.231-196.275 TPS on the same machine. The B1 harness and served aggregate +B8 harness measure different things and remain separate controls. + +Auditing that stack found one real B8 omission: greedy B8 materialized full +`[8,V]` draft logits and `[8,2,V]` verify logits on the CPU every cycle. The +new construction-selected greedy sampling route keeps draft IDs on device, +feeds them directly into the B8/T2 verify input, and transfers only small token +ID arrays for primary and verify decisions. Default stochastic sampling keeps +the existing exact batched sparse route. Requests with frequency or presence +penalties keep the dense compatibility route. + +The served A/B results are: + +| Workload | Previous B8 | Device-ID B8 | Change | Output parity | +| --- | ---: | ---: | ---: | --- | +| coding, 8 x 192 tokens | 414.852 TPS | 452.413 TPS | +9.05% | all 8 hashes exact | +| legacy greedy | 324.019 TPS | 349.064 TPS | +7.73% | all 8 hashes exact | +| legacy default sampler | 170.406 TPS | 166.743 TPS | -2.15% | all 8 hashes exact | + +The default-sampler path does not select the new greedy route; its small timing +change is recorded as run-to-run drift, not an optimization claim. Receipt +SHA-256 values are +`b0683bab11fabab4f544f3d34292d92ceb559cf034b5adf5c2cc72fc7a40ef49` +for the coding run and +`8f64743bc745a61f198dc370103d175e6c4b713c6d0573d863323aa69f6a8e47` +for the legacy greedy run. + +Two adjacent candidates were rejected: + +- forcing the existing unsorted MoE gather route reduced legacy greedy B8 from + 324.019 to 301.040 TPS (-7.09%) and did not change the eight measured B8 + output hashes; +- a construction-only M16 whole-MoE probe improved one real target block from + 0.5308 to 0.5112 ms (1.038x), but changed BF16 output (`max_abs=0.0491`). + The projected gain did not justify adding a new 40-layer arithmetic route. + +## Public interface + +The server accepts one construction-time option: + +```text +--mtp-batch-numerics throughput|balanced|b1-exact +``` + +The corresponding config key is `mtp_batch_numerics`. The value is resolved +once while server arguments are constructed and passed explicitly to the lane +installer. Generation does not read an environment variable or inspect the +option again. + +The option applies only to Qwen 35B `--scheduler-mode mtp_batch`. Selecting +`balanced` or `b1-exact` with another scheduler fails startup with a clear +configuration error. A singleton request continues to use unchanged solo MTP. +`throughput` and `balanced` govern physical B8 cohorts at real widths two +through eight. `b1-exact` deliberately serializes every sealed request through +that same unchanged B1 runner. + +The initial default remains `throughput`. `balanced` may become the persistent +Qwen launcher default only after all promotion gates pass. `b1-exact` is always +explicit and has no throughput promise. + +## Profiles + +### `throughput` + +This is the default PR #245 implementation. Its target verify work uses the +installed B8/T2/M16 route, and its draft work uses B8/T1/M8. Greedy sampling +uses the device-ID route described above. Default stochastic sampling and +penalty-bearing compatibility requests retain their separately bound routes. +Target/draft model arithmetic, cache ownership, and scheduling do not change. + +Route identity: + +```text +qwen35b_a3b_mtp_batch_b8_t2_m16_throughput +``` + +### `balanced` + +This route keeps B8 scheduling, row-owned caches, batched attention, batched +sampling, and shared weight access. Construction attribution found the first +divergence in layer 0's GDN projections: B8 flattens `[8,2,H]` to M16 while B1 +uses M2, changing BF16 accumulation order. The current candidate binds eight +unchanged B1/T2 calls for layer 0's QKV, Z, and B projections. The recurrent A +gate remains on the B8/T2 route, and every projection in layers 1 through 29 +also keeps the throughput route. The 24 B1 calls are traced into one fixed +B8/T2 compiled target graph; there is no request-time loop, eligibility check, +or fallback. + +The real layer-0 QKV shape is `[8,2,2048] -> [8,2,8192]`, BF16 with 4-bit +affine group-64 weights. Two ten-repeat construction probes measured: + +| QKV route | B1 max abs | Median latency, probe 1 | Median latency, probe 2 | +| --- | ---: | ---: | ---: | +| throughput M16 | 0.03125 | 0.1715 ms | 0.1666 ms | +| stock M16 | 0.03125 | 0.2646 ms | 0.2558 ms | +| eight unchanged B1/T2 calls | 0 | 0.2616 ms | 0.2646 ms | +| one batched-weight logical-M2 QMV | 0 | 0.6482 ms | 0.5826 ms | + +The existing flattened stock-like QMV candidate was also rejected: it was not +exact (`max_abs=0.046875`) and its median was 0.6130 ms. These measurements +select the eight-call route before any new EvalPlus run. Probe receipt SHA-256 +values are +`3685ff3c4cd4d8266e79f0963ca18fb5bd813c199c15e4f92605a431ed68556d` +and +`4a39f2bb96c1c35bdcaa1bcac9741c5f8604014c71ea97b99ef0b559342ace2a`. + +The second candidate applied the exact QKV call to all 30 GDN layers and kept +stock B/A. Its installation proved the QKV boundary bitwise, exact layer-0 +recurrent output, exact argmax, exact offsets, and exact row isolation. It ran +true B8 in all four concurrent cohorts, but reached only 261.641 aggregate TPS, +or 1.652 times its paired B1 median. This is below the 300 TPS floor, so the +candidate was rejected without running EvalPlus. Its receipt SHA-256 is +`25493b02922894c12998517a787e66198f397151942e25eb49ba9c9a062a9bb1`. + +The all-layer candidate needed a 12/128 compiled/eager full-graph BF16 bound +because its measured relative error was 0.09277. The smaller layer-zero route +restores the existing 9/128 bound. The post-convolution numerical, argmax, +ownership, and candidate-boundary checks remain unchanged. + +A QKV-only layer-zero candidate was rejected before timing. Its replacement +boundary was bitwise identical to eight B1/T2 calls and its full-graph relative +errors remained within 9/128, but the real construction corpus changed the +compiled/eager, heterogeneous B1, and same-geometry B8 argmax decisions. The +gate failed closed. + +The next candidate made all four layer-zero projections B1-exact. It passed the +unchanged construction argmax and ownership gates and reached 322.234 greedy +aggregate TPS, or 99.6% of the throughput control. It was still rejected: +default-sampler throughput fell to 140.654 TPS, below the 153.425 floor. The +same workload used 434 fixed B8 verify cycles versus throughput's 341, showing +that reduced target/draft acceptance—not projection latency—caused the loss. +Its greedy and default receipt SHA-256 values are +`721a44abbfa5e71b1e66dfaa15a17e6494fe3f092c5128d963047ae9bcc6affa` +and +`5506894ff2865626b9533b922b36e5be19ce7d2619b44e321d836c7fd4226b9c`. + +The next QKV/B/A candidate restored Z to B8/T2. It failed construction before +timing: compiled/eager relative attention error was 0.09159 and same-B8 +optimized/stock attention error was 0.10849, both above the fixed 9/128 +(0.0703125) bound. Its QKV/B/A replacement boundaries and argmax decisions +were exact, so the failure specifically shows that Z must remain coherent with +the corrected QKV path. The numerical bound was not relaxed. + +The QKV/Z candidate made every construction argmax equal and made layer-zero +convolution state exact, but still failed the unchanged numerical gate. Its +compiled/eager relative errors were 0.08462 for logits and 0.06958 for +attention. Its same-B8 errors were 0.07968 for hidden and 0.07390 for attention. +The limit was 0.0703125. It was rejected before timing. + +The QKV/Z/A candidate disproved A as that missing correction. Its +compiled/eager attention error rose to 0.10142, and its same-B8 errors rose to +0.11155 for hidden and 0.10063 for attention. Heterogeneous numerical parity +also failed. It was rejected before timing. + +The final candidate added B while leaving A batched. It passed construction but +reached only 259.650 greedy aggregate TPS, below the 300 TPS floor, and was +rejected without promotion. Its receipt SHA-256 is +`2e822ed75011ded21a583ae1495c88a5c23b35ccd156fc2e0dfcca46869f4ca8`. +This exhausted the coherent rowwise-B1 layer-zero projection subsets without +relaxing the construction bound. + +The balanced operator set is fixed in source after measurement. It is not +chosen dynamically from a logit margin, row count, or runtime probe. Candidate +operators are promoted one at a time, and each candidate must improve numerical +parity and pass the end-to-end throughput and quality gates before another is +added. It is closer to B1 numerically, but it is not token-exact with B1: later +B8 BF16 reductions can still change a near-tie token and subsequent context. + +Route identity: + +```text +qwen35b_a3b_mtp_batch_b8_t2_l0_b1_qkv_z_b_balanced +``` + +### `b1-exact` + +This route keeps the same request queue and model-owner thread, but never calls +the B8 driver. Every sealed request is run once through the unchanged +request-local B1 MTP implementation. Failures remain request-local unless +model-owner cleanup fails, which poisons the service before another row runs. + +The label `b1-exact` is therefore contractual by construction: the adapter does +not reproduce B1 arithmetic in a new kernel; it invokes the B1 implementation +itself. This preserves token and cache behavior at the cost of serial aggregate +throughput. Health and completion metadata say `serial_b1_exact` and never +claim physical B8 execution. + +Route identity: + +```text +qwen35b_mtp_batch_b1_exact_serial +``` + +## Attribution before kernel work + +The first implementation step is an offline, construction-shape attribution +receipt. It runs the same prompts, token inputs, cache contents, and committed +positions through unchanged B1, throughput B8, and same-geometry B8 references. +It records the first divergent operation and per-layer maximum absolute and ULP +errors for: + +- projection outputs; +- MoE router scores and expert IDs; +- expert outputs and combine results; +- GDN convolution output and recurrent state; +- attention K/V and offsets; +- hidden states; +- target and draft logits; +- argmax and speculative accept/reject decisions. + +This attribution runs only in a dedicated construction/benchmark command. It +does not add counters, comparisons, synchronization, environment reads, or +fallback accounting to production generation. + +No operator is changed merely because its B1 and B8 shapes differ. The first +candidate must address the first measured material divergence on the real +4-bit affine Qwen 35B shapes. + +## Construction architecture + +The installer parses the profile into a closed enum and selects one immutable +route specification. Each B8 specification owns: + +- its exact target, capture, draft, and MTP-update callables; +- its exact GDN, attention, router, expert, combine, and projection routes; +- its construction self-check; +- its route ID and config fingerprint; +- its session/cache compatibility identity. + +The exact specification instead binds the service's multi-request executor to +the unchanged solo runner. Its B8 callable table is retained only for the +startup construction receipt and is never dispatched. + +The installer validates the model, dtype, quantization, geometry, callable +table, and profile-specific receipt once. It returns an immutable +`InstalledA3BMTPBatchLane` with the selected callables already bound. The decode +loop invokes those callables directly. + +There is no enabled-path `if profile == ...`, no custom-then-stock exception +handler, and no automatic downgrade. Stock or B1 arithmetic used by a profile +is an explicit member of its construction route table. + +## Cache and session identity + +The numerics profile and route fingerprint are part of every reusable decode +state identity. A target/MTP cache or session-bank entry created under one +profile cannot be restored under another profile. Changing the flag requires a +server restart and creates a new compatibility domain. + +Prompt prefill remains the unchanged request-local B1 prefill contract. For the +two B8 profiles, the profile boundary begins where requests enter the fixed B8 +decode lane. Exact mode never crosses that boundary. B8 profile tests still +prove target offsets, MTP-history offsets, recurrent ownership, inactive-row +freezing, and cancellation isolation. + +## Health and receipts + +Health and completion metadata report: + +- requested and effective `mtp_batch_numerics` profile; +- exact profile route ID; +- profile config fingerprint; +- construction receipt verdict; +- real cohort width and physical fixed width. + +For exact mode, physical width is truthfully reported as one even when eight +requests were sealed together. These are existing request/cohort-boundary +statistics. The change does not add per-token, per-layer, per-cycle, or +per-dispatch engagement counters. + +## Error handling + +Startup fails before serving when: + +- the profile name is unknown; +- a non-throughput profile is selected outside Qwen 35B `mtp_batch`; +- a required profile callable is absent; +- a profile receipt fails its declared parity contract; +- the cache/session compatibility fingerprint is incomplete; +- the installed route ID does not match the selected profile. + +An installed profile never switches route after a request has been admitted. +Existing cohort cleanup failure behavior remains fail-closed. + +## Correctness gates + +Both B8 profiles must retain the existing PR #245 ownership and serving gates: + +- eight distinct request IDs and markers with no foreign text; +- exact unaffected-row isolation when another row changes; +- exact row-permutation parity for equivalent B8 inputs; +- exact cache offsets and commit ownership; +- inert padding, completed, and cancelled rows; +- active cancellation with one model-owner cleanup; +- the 13,239-token-per-row long-context gate; +- no AR or stock fallback. + +`throughput` retains its existing bounded BF16 and exact argmax construction +contract. + +`balanced` must improve B1 parity at every replaced boundary, preserve exact +token decisions in the construction corpus, and pass profile-specific numerical +bounds chosen before the end-to-end quality run. + +`b1-exact` must reproduce unchanged B1 output hashes for identical prompts, +seeds, sampling parameters, and token limits. Its construction receipt must say +`unchanged_solo_runner`, and the B8 driver dispatch count must remain zero. + +## Quality gates + +Quality is measured with EvalPlus 0.3.1, one greedy completion per task, using +the same prompts, hashes, maximum-token budget, and scoring commands already +recorded in PR #245. + +The balanced route may be called an accuracy improvement only when: + +- HumanEval base is at least 151/164; +- HumanEval+ is at least 145/164; +- MBPP base is at least 335/378; +- MBPP+ is at least 286/378; +- both plus suites improve over the throughput route; +- no response contains another row's task or marker; +- every scored response has an audited physical B8 route. + +Matching the full B1 combined score of 434/542 is the goal, but not the minimum +balanced promotion threshold. The PR reports the exact point estimate, paired +swap counts, and exact McNemar p-value without claiming significance that the +sample does not establish. + +The `b1-exact` route must reproduce the unchanged B1 program hashes on the +fixed deterministic corpus before its flag is documented as available. + +## Performance gates + +All model work acquires `/tmp/mtplx-gpu-exclusive.lock`; only Qwen is loaded. +Timing uses an isolated server port with no unrelated request traffic. +Profiler and dispatch-census runs are separate from timed runs. + +Each candidate is measured against unchanged `throughput` with the same eight +prompts, seeds, token budgets, stop settings, service configuration, and thermal +conditions. Three paired rounds are required. + +Balanced promotion requires: + +- greedy median aggregate output throughput at least 300 tokens per second; +- default-sampler median at least 153.425 tokens per second, or 95% of the + current 161.500 control; +- no regression in request isolation, cancellation, long-context behavior, or + peak-memory safety; +- the real route ID and B8/M16-or-declared-hybrid dispatch geometry in the + untimed census. + +The `b1-exact` route is measured by the same protocol, but its result is +reported without a throughput floor. It remains explicit and never becomes the +default automatically. + +## Rollout + +1. Add the startup enum, config key, validation, health fields, and immutable + route-table plumbing without changing throughput arithmetic. +2. Add the offline first-divergence attribution receipt. +3. Implement and measure one balanced operator candidate at a time. +4. Freeze the smallest balanced operator set that clears all quality and + throughput gates. +5. Bind `b1-exact` to the unchanged request-local B1 runner and expose it only + after paired deterministic hashes pass. +6. Run the full changed-area and repository test suites. +7. Add code, benchmark receipts, and the profile table to the existing PR #245. + Do not create another PR. +8. Change the persistent launcher default to `balanced` only if its gates pass; + otherwise leave `throughput` as default and report the miss honestly. + +## Failure-mode check + +### The balanced route overfits the published coding tasks + +Severity: critical if task results are used to choose individual arithmetic +branches. Mitigation: operator selection is based on construction-shape +attribution and parity before EvalPlus is run. EvalPlus is a final promotion +gate, not a per-task tuning loop. Every attempted candidate and rejected result +is recorded. + +### A profile flag creates hidden hot-path branching or fallback + +Severity: critical because it can erase the measured gain and violate the lane +contract. Mitigation: the enum selects a complete route specification during +installation. Source tests inspect the installed driver call graph, and a +dispatch census confirms the selected route outside timing. + +### Reusable state crosses numerics profiles + +Severity: critical because a request can begin from cache state produced by a +different arithmetic contract. Mitigation: include the profile and route +fingerprint in reusable cache/session identities and reject mismatches before +restore. + +### Exact B1 arithmetic erases the B8 speedup + +Severity: expected risk, not a correctness failure. Mitigation: `b1-exact` +remains explicit with no throughput promise. Balanced changes one attributed +operator at a time and cannot ship below the declared throughput floors. + +### Greedy clears 300 TPS while default sampling regresses materially + +Severity: critical for the default product experience. Mitigation: balanced +promotion includes the separate 95%-of-control default-sampler floor. + +## Non-goals + +- Making B8 the quality reference by routing all singleton requests through + padded B8 arithmetic. +- Request-level or per-row switching between numerics profiles. +- Logit-margin fallback or replay of selected rows from an active B8 cohort. +- Changing model weights, quantization, MTP depth, sampler semantics, or + published artifacts. +- Generalizing the profiles to other models without their own geometry, + construction receipt, and end-to-end measurements. diff --git a/docs/superpowers/specs/2026-07-29-retrieval-idle-standby-design.md b/docs/superpowers/specs/2026-07-29-retrieval-idle-standby-design.md new file mode 100644 index 000000000..6b62f792b --- /dev/null +++ b/docs/superpowers/specs/2026-07-29-retrieval-idle-standby-design.md @@ -0,0 +1,86 @@ +# Retrieval idle standby + +## Problem + +Retrieval models load on first request and then stay resident until the +`--retrieval-max-resident` cap is exceeded. With two configured models and a cap +of two, the cap can never be exceeded, so nothing is ever unloaded: several GB +stay held for the lifetime of the daemon even when no retrieval request has +arrived for hours. + +The session bank (KV cache) has the same shape — it grows with use and is only +shed under system memory pressure. + +Neither is a defect in the eviction logic. Idle release simply does not exist: +there is no idle timeout anywhere in the runtime. + +## Non-goals + +Unloading the **chat model** is out of scope. No unload path exists for it +anywhere in the core; adding one touches `ServerState`, warmup, the MTP contract +and session prefixes, and makes every subsequent request pay a cold start. It +deserves its own spec. + +"Partially unloading" weights is not a thing — a transformer is resident or it +is not. What can be released separately is the MLX buffer pool, the session +bank, and whole model weights. + +## Design + +A watcher task runs beside the existing memory-pressure loop, every 30 s, and is +started only when an idle timeout is configured. Without configuration the +daemon behaves exactly as before. + +Two triggers: + +1. **Idle timeout** — retrieval backends whose last use is older than the + threshold are unloaded; when every retrieval model is idle the session bank + is archived to the existing SSD cold tier rather than dropped, so a resumed + conversation restores its prefix from disk instead of re-prefilling. +2. **Memory pressure** — the existing guard gains a hook that unloads idle + retrieval backends on the rising edge into CRITICAL. + +Pinned backends — those inside an in-flight request — are never touched. The +watcher reuses the pin counting already used by the resident cap. + +## Components + +| Unit | Responsibility | +| --- | --- | +| `RetrievalRegistry.unload_idle(older_than_s)` | Unload unpinned backends past the threshold; return freed bytes and the ids affected. Pure registry logic, unit-testable without weights. | +| `_retrieval_idle_loop(state, interval_s)` | Call `unload_idle`, then archive the session bank when nothing is resident. Never raises into the server. | +| `--retrieval-idle-timeout` | Seconds; `0` disables. Threaded through the `serve`/`quickstart` parsers, the child argv builder, and the server parser — the three points the retrieval flags already traverse. | +| `config.toml: retrieval_idle_timeout` | Persisted default. | +| Snapshot `retrieval.idle_timeout_s` + per-model `idleSeconds` | Lets the app show "unloads in N min" rather than only "loaded". | +| App setting + Activity display | Timeout field beside the resident cap; countdown in the retrieval card. | + +## Data flow + +``` +watcher (30 s) + ├─ registry.unload_idle(timeout) → frees weights, updates residency + └─ sessions.archive_cold_tier() → session bank to SSD, when idle + ↓ + next snapshot reports freed memory +``` + +## Error handling + +The watcher must never take the server down: exceptions are logged and +swallowed, and a failed archive does not prevent the weight unload. Unloading a +backend that is concurrently acquired is impossible by construction — the pin +count is checked under the registry lock. + +## Testing + +- `unload_idle` respects the threshold, skips pinned backends, reports freed + bytes, and is a no-op when nothing is stale. +- The watcher is not started when no timeout is configured. +- Idle seconds appear in the snapshot and decode in the app, including the + absent-field case for older daemons. + +## Measured baseline + +On the machine this was designed against: chat weights 18.4 GB, session bank +6.9 GB, retrieval 6.1 GB when both models are resident. The scope above targets +the latter two, roughly 13 GB. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 651e475e8..a5700875e 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -14,9 +14,10 @@ Expected production failures should be actionable, not tracebacks: |---|---| | MLX missing | `python3 -m pip install mlx` from native arm64 Python | | Rosetta Python | switch to native arm64 Python and rerun `mtplx doctor` | -| default model missing | `mtplx pull Youssofal/Qwen3.6-27B-MTPLX-Optimized-Speed` | +| default model missing | `mtplx pull Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed` | | Open WebUI cannot connect | use `http://127.0.0.1:8000/v1` on the host, or `http://host.docker.internal:8000/v1` inside Docker | | Docker daemon stopped | start Docker Desktop | -| low disk/RAM | change `MTPLX_MODEL_DIR`, free storage, lower context/profile, or use a smaller model | +| low disk/RAM | app: choose another drive under Settings → Model Storage; CLI: change `MTPLX_MODEL_DIR`; or free storage, lower context/profile, or use a smaller model | +| huggingface.co unreachable / blocked | CLI: `HF_ENDPOINT=https://hf-mirror.com mtplx pull
' + "".join(
+ f'{_esc(t)}' for c, t in items) + "
"
+
+
+def _card(title: str, body: str) -> str:
+ return f"{_esc(title)}
{body}{}
' + + +# --------------------------------------------------------------------------- +# per-turn record (receipt extras layered over trace._turn_row) + + +def _enrich(turn: dict) -> dict: + row = _turn_row(turn) + receipt = turn.get("receipt") or {} + row["decode_elapsed_s"] = receipt.get("decode_elapsed_s") + row["sliding"] = [(lbl, receipt.get(f"sliding_decode_tok_s_{key}")) for lbl, key in _SLIDING] + row["accepted_by_depth"] = receipt.get("accepted_by_depth") or [] + row["drafted_by_depth"] = receipt.get("drafted_by_depth") or [] + row["mean_accept_p"] = receipt.get("mean_accept_probability_by_depth") or [] + row["verify_calls"] = receipt.get("verify_calls") + row["draft_time_s"] = receipt.get("draft_time_s") + row["verify_time_s"] = receipt.get("verify_time_s") + row["accept_time_s"] = receipt.get("accept_time_s") + row["samples"] = [e for e in turn.get("flight", []) if e.get("ev") == "s"] + return row + + +def _row_dur(r: dict) -> float: + return ((r["ttft_s"] or 0.0) + (r["decode_elapsed_s"] or 0.0)) or r["wall_s"] or 1.0 + + +def _share(r: dict) -> str: + think = r["client_reasoning_tokens"] or 0 + denom = r["completion_tokens"] or (think + (r["client_output_tokens"] or 0)) + return f"{think / denom * 100:.0f}% think" if denom else "" + + +# --------------------------------------------------------------------------- +# sections + + +def _sec_timeline(rows: list[dict]) -> str: + rows = [r for r in rows if r["start"]] + if not rows: + return _card("Session timeline", _QUIET.format("no dated turns")) + t0 = min(r["start"] for r in rows) + t1 = max(r["start"] + _row_dur(r) for r in rows) + x0, x1, rh = 44.0, 856.0, 26.0 + h = len(rows) * rh + 42 + + def sx(ts: float) -> float: + return x0 + (ts - t0) / max(t1 - t0, 1e-9) * (x1 - x0) + + out = [f'") + + +def _sec_cache(rows: list[dict]) -> str: + rows = [r for r in rows if (r["prompt_tokens"] or 0) > 0] + if not rows: + return _card("Cache waterfall", _QUIET.format("no receipts with prompt tokens")) + ticks = _num_ticks(max(float(r["prompt_tokens"]) for r in rows)) + xmax = ticks[-1] or 1.0 + x0, x1, rh = 44.0, 812.0, 24.0 + h = len(rows) * rh + 42 + + def sx(v: float) -> float: + return x0 + v / xmax * (x1 - x0) + + out = [f'") + + +def _accept_overlay(samples: list[dict]) -> tuple[list[tuple[float, float]], float | None]: + """FUTURE flight fields: per-second overall acceptance from cumulative + acc/drf per-depth arrays (rate = delta(sum(acc)) / delta(sum(drf)) between + consecutive samples), plus verify-share of wall from cumulative vt seconds. + Today's recorder emits neither — returns ([], None) so nothing is drawn.""" + seq: list[tuple[float, float, float]] = [] + for s in samples: + acc, drf = s.get("acc"), s.get("drf") + if isinstance(acc, list) and isinstance(drf, list) and acc and drf: + try: + seq.append((float(s.get("ts") or 0), sum(map(float, acc)), sum(map(float, drf)))) + except (TypeError, ValueError): + continue + pts: list[tuple[float, float]] = [] + for (_t0, a0, d0), (t1, a1, d1) in zip(seq, seq[1:]): + dd = d1 - d0 + if dd > 0: # zero drafted this second -> no rate point, never a fabricated one + pts.append((t1, max(0.0, min((a1 - a0) / dd, 1.0)))) + share = None + vts = [(float(s.get("ts") or 0), float(s["vt"])) for s in samples + if isinstance(s.get("vt"), (int, float))] + if len(vts) >= 2 and vts[-1][0] > vts[0][0]: + share = max(0.0, min((vts[-1][1] - vts[0][1]) / (vts[-1][0] - vts[0][0]), 1.0)) + return pts, share + + +def _tps_cell(r: dict) -> str | None: + sliding = [(lbl, float(v)) for lbl, v in r["sliding"] if v is not None] + samples = r["samples"] + if not samples and not sliding: + return None + w, px0, px1, py0, py1 = 252, 30, 240, 10, 86 + approx = not samples + if samples: + ts0 = float(samples[0].get("ts") or 0) + pts = [(float(s.get("ts") or 0) - ts0, float(s.get("tps") or 0)) for s in samples] + span = max(pts[-1][0], 1e-9) + else: + pts = [(float(i), v) for i, (_, v) in enumerate(sliding)] + span = max(len(pts) - 1.0, 1.0) + ticks = _num_ticks(max([v for _, v in pts] or [1.0]), 2) + ymax = ticks[-1] or 1.0 + + def spt(p: float, v: float) -> tuple[float, float]: + return px0 + p / span * (px1 - px0), py1 - v / ymax * (py1 - py0) + + out = [f'') + tok_s = f"{r['decode_tok_s']:.1f} tok/s" if r["decode_tok_s"] else "-" + cancel = ' · cancelled' if r["status"] != "ok" else "" + badge = f'{"approx" if approx else "flight"}' + return (f'')
+
+
+def _sec_tps(rows: list[dict]) -> str:
+ cells = [c for c in (_tps_cell(r) for r in rows) if c]
+ if not cells:
+ return _card("Per-request TPS", _QUIET.format("no flight samples or receipt sliding windows"))
+ note = _QUIET.format(
+ "solid blue line — flight recorder per-second samples · dashed line with markers — sketch "
+ "from receipt sliding windows (approximation; no flight data recorded for these requests) · "
+ "thin orange overlay (only when flight samples carry cumulative accept/draft counters) — "
+ "per-second MTP acceptance rate on a 0–100% band of the cell height · per-turn acceptance "
+ "and verify-time detail lives in the MTP section below")
+ return _card("Per-request TPS", note + f'
{"".join(cells)}
')
+
+
+def _pct_grid(x0: float, x1: float, h: float) -> str:
+ """Hairline grid + axis for a 0..100% horizontal scale (ticks every 25%)."""
+ parts = []
+ for f in (0.0, 0.25, 0.5, 0.75, 1.0):
+ tx = x0 + f * (x1 - x0)
+ parts.append(f'Acceptance rate by draft depth
" + _QUIET.format( + "no MTP acceptance counters in any receipt for this session") + depth_n = max(len(r["drafted_by_depth"]) for r in with_mtp) + ramp = _DEPTH_RAMP.get(depth_n, _DEPTH_RAMP[4]) + body, y = [], 8.0 + for r in rows: + drafted, accepted = r["drafted_by_depth"], r["accepted_by_depth"] + if not drafted: + reason = "no server receipt" if r["receipt_missing"] else "no MTP data" + if r["status"] != "ok": + reason += " · cancelled" + body.append(f'Acceptance rate by draft depth
" + note + legend + + f'") + + +def _mtp_time_panel(rows: list[dict]) -> str: + """100%-normalized stacked bar per turn splitting decode_elapsed_s into + draft / verify / accept / other, absolute seconds always visible.""" + x0, x1, rh = 44.0, 640.0, 26.0 + have, skipped = [], [] + for r in rows: + comps = (r["draft_time_s"], r["verify_time_s"], r["accept_time_s"]) + if r["decode_elapsed_s"] and any(c is not None for c in comps): + have.append(r) + else: + skipped.append(f"t{r['turn']}" + (" (cancelled)" if r["status"] != "ok" else "")) + if not have: + return "Decode time split
" + _QUIET.format( + "no draft/verify/accept timing in any receipt for this session") + h = len(have) * rh + 42 + body = [] + for i, r in enumerate(have): + y = i * rh + 8 + total = float(r["decode_elapsed_s"]) + d = float(r["draft_time_s"] or 0.0) + v = float(r["verify_time_s"] or 0.0) + a = float(r["accept_time_s"] or 0.0) + other = max(total - (d + v + a), 0.0) + denom = max(total, d + v + a) or 1.0 + segs = [("draft", d, _BLUE), ("verify", v, _ORANGE), ("accept", a, _AQUA), ("other", other, _YELLOW)] + body.append(f'Decode time split (draft / verify / accept / other)
" + note + legend + + f'" + tail) + + +def _sec_mtp(rows: list[dict]) -> str: + if not rows: + return _card("MTP acceptance & verify time", _QUIET.format("no assistant turns")) + return _card("MTP acceptance & verify time", _mtp_accept_panel(rows) + _mtp_time_panel(rows)) + + +def _sec_scatter(receipts: list[dict], session_ids: set[int], port: int) -> str: + pts: list[tuple[float, float, bool, dict]] = [] + for rec in receipts: + y = rec.get("decode_tok_s") + x = rec.get("context_len") + if x is None: + x = ((rec.get("prompt_tokens") or 0) + (rec.get("completion_tokens") or 0)) or None + if x and y: + pts.append((float(x), float(y), id(rec) in session_ids, rec)) + if not pts: + return _card("Context vs decode speed", _QUIET.format("no receipts with decode speed")) + pts.sort(key=lambda p: p[2]) # history first, session points painted on top + h, x0, x1, y0, y1 = 336, 56.0, 1092.0, 14.0, 284.0 + xticks, yticks = _num_ticks(max(p[0] for p in pts)), _num_ticks(max(p[1] for p in pts), 4) + xmax, ymax = xticks[-1] or 1.0, yticks[-1] or 1.0 + out = [f'") + + +def _sec_digest(digest: list[dict]) -> str: + if not digest: + return _card("Turn digest", _QUIET.format("no assistant turns")) + head = ('mtplx trace report
'
+ + "".join(f''
+ for k, v in cards) + "
")
+ pathology = _card("Pathology flags", '{_esc(k)}
{_esc(v)}
- ' + "".join(
+ f'
- !!{_esc(f)} ' for f in flags) + "