Environment
Nativ 0.3.3 (bundled mlx-vlm 0.6.15), macOS 27.0, M5 Max / 128 GB, model mlx-community/Qwen3.8-27B-nvfp4.
Observations
prefixCachingEnabled defaults to false and does not appear to be surfaced anywhere in the UI (we flipped it in ~/Library/Application Support/Nativ/Settings.plist).- Even when enabled, the default pool is 2048 blocks × 16 tokens = 32,768 tokens — smaller than a single Claude Code system prompt (~100k tokens), so for coding-agent traffic the cache never engages at all (
/v1/cache/stats showed zero pool activity during a real agent session).
Measured impact (same machine, same model, same task)
- A real headless Claude Code refactor via
/v1/messages: timed out at 40 min with caching at defaults — 6 requests averaging 682 s, 597k prompt tokens re-ingested, because every agent turn re-paid the full prefill of the growing conversation. (Ollama, same weights/quant, finishes the identical task in 744 s.)
- With
prefixCachingEnabled=true and prefixCacheBlocks=9600 (~154k tokens): a 100k-token grown turn dropped 233 s → 1.66 s (cache_read_input_tokens: 81869), and a 16-turn growing conversation held a 0.84→0.95 cached fraction with flat ~1.6–2.0 s TTFT. RSS stayed ~15.5 GB.
Suggestion
Consider (a) enabling prefix caching by default, or at least surfacing the setting in the Developer/Settings UI, and (b) scaling the default pool with the loaded model's context rather than a fixed 32k. Aware of #272 (eviction hang with prefix caching) — that may be the blocker for on-by-default, but the current defaults make Nativ time out on workloads it otherwise handles well.
Environment
Nativ 0.3.3 (bundled mlx-vlm 0.6.15), macOS 27.0, M5 Max / 128 GB, model
mlx-community/Qwen3.8-27B-nvfp4.Observations
prefixCachingEnableddefaults tofalseand does not appear to be surfaced anywhere in the UI (we flipped it in~/Library/Application Support/Nativ/Settings.plist)./v1/cache/statsshowed zero pool activity during a real agent session).Measured impact (same machine, same model, same task)
/v1/messages: timed out at 40 min with caching at defaults — 6 requests averaging 682 s, 597k prompt tokens re-ingested, because every agent turn re-paid the full prefill of the growing conversation. (Ollama, same weights/quant, finishes the identical task in 744 s.)prefixCachingEnabled=trueandprefixCacheBlocks=9600(~154k tokens): a 100k-token grown turn dropped 233 s → 1.66 s (cache_read_input_tokens: 81869), and a 16-turn growing conversation held a 0.84→0.95 cached fraction with flat ~1.6–2.0 s TTFT. RSS stayed ~15.5 GB.Suggestion
Consider (a) enabling prefix caching by default, or at least surfacing the setting in the Developer/Settings UI, and (b) scaling the default pool with the loaded model's context rather than a fixed 32k. Aware of #272 (eviction hang with prefix caching) — that may be the blocker for on-by-default, but the current defaults make Nativ time out on workloads it otherwise handles well.