feat(llm): native messages, prompt caching and per-model params for cloud providers (F26, F27, F28) - #433
Open
plombeer31 wants to merge 4 commits into
Open
feat(llm): native messages, prompt caching and per-model params for cloud providers (F26, F27, F28)#433plombeer31 wants to merge 4 commits into
plombeer31 wants to merge 4 commits into
Conversation
…ms for every OpenAI-compatible kind
Wire `userModels[].reasoningFormat` into the provider (it was parsed and
never read); the default becomes `auto`, which reads `reasoning`,
`reasoning_content` or `thinking` on both the streamed and the unary
path, so a DeepSeek-style stream no longer loses its reasoning.
Decode `<tool_call>` blocks on every kind, not only Qwen's: the Hermes
JSON form (`{"name", "arguments"}`) joins the Qwen XML-ish form, and a
reply is a tool call only when the whole trimmed content is such
blocks. The reasoning-channel fallback (#105) stays Qwen-only.
A streamed chunk carrying both text and a tool-call delta keeps its
text. OpenAI reasoning models (`o1`…`o9`, `gpt-5`, with or without a
vendor prefix) get no `temperature` and `max_completion_tokens` instead
of `max_tokens` — the rejection `request-size-rejection.ts` quotes.
`userModels[].params` merges per-model wire parameters over the body
and `extraBody`; `CompletionRequest.reasoningEffort` is spelled per
kind (`reasoning: {effort}` on OpenRouter, `reasoning_effort` on
OpenAI-compatible, omitted elsewhere).
…le cache routes, cached tokens on the record
Anthropic caches nothing without `cache_control` markers, and Claude
through OpenRouter had none, so every step of a long turn paid the full
input price. `applyAnthropicCacheControl` places two ephemeral
breakpoints — the system message and the last history message before
the tail — on an OpenAI-shaped `messages` array; the provider turns it
on for `anthropic/…` / `claude…` models and Anthropic's own host, and
the entry's `promptCache` policy (parsed since v24, never read) now
means something: `off` sends none, `explicit-markers` always does.
Google models via OpenRouter cached 83 % of input on Google AI Studio
and 1–4 % on Vertex: an `openrouter` entry with no `providerPreferences`
of its own now sends `{order: ["Google AI Studio", "Google"],
allow_fallbacks: true}` for `google/…` models, behind the new
`llm.openrouter.preferCacheRoutes` (default on).
`prompt_tokens_details.cached_tokens` is read on both paths into
`usage.cachedTokens` and `cacheHitTokens` (the trace's field); the cost
accumulator and the turn usage meter price cached tokens at the model's
`cacheRead` rate and the turn snapshot reports `cachedTokens`.
Every cloud request was one `user` message holding the stable prefix and the whole history as `assistant_tool_call:` / `tool_result[…]` text, and Gemini Flash kept continuing that text instead of calling tools (84,931 characters in one completion; three worker streams in one run). `buildPrompt` now also returns the prompt as structure — `BuiltPrompt.messages`: the stable prefix, the packed turns with each tool-result body capped exactly as the text form caps it (one shared walk in `build-prompt-world-conversation.ts`), and the tail without `### conversation`. The flat `text` is byte-identical to before. The step executor puts it on `LlmStreamParams.messages` for a native-tools primary only (the repair retry re-shapes the tail with the notice), and `llm-link-attempt` forwards it to a native link alone, so a grammar fallback link still gets the flat prompt and its GBNF. `buildOpenAiChatBody` lays a structured request out as `system`, then assistant `tool_calls` (ids `call_<row index>`, stable between packer cuts) answered by `tool` messages, then one final `user` message; a call with no recorded result is answered `(no result recorded)`, a result whose call was cut away rides as its flat line, and the packer's recap opens the history. `llm.providers[].messageShape: "flat"` keeps the single message; a 400 about roles / `tool_call_id` / `messages` flips the session to flat and resends once, logged. Sub-calls and grammar / subscription-CLI kinds are unchanged. The fabricated-transcript detector and stream watcher (patches 3 and 5) stay as the net.
…t auto, openrouter.preferCacheRoutes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Before: every cloud request was one
usermessage with the history flattened toassistant_tool_call:/tool_result[…]text and no system role, which is what invited cloud models to write that syntax as text instead of calling tools; nocache_controlanywhere, so Anthropic models via OpenRouter were never cached and Google models cached 1–4 % on some routes;reasoningFormatnever reached the provider, tagged<tool_call>text was parsed only for the Qwen kind, andtemperature/max_tokenswere sent to models that reject them.After: per-provider
messageShape: "native" | "flat"(native by default for openrouter / openai-compatible / aimlapi / gemini; flat for subscription CLIs and llama-server): a system message, the packed conversation as realuser/assistant(withtool_calls) /toolturns and the tail as one final user message, with a one-time fallback to flat when a 400 mentions roles;cache_controlbreakpoints on the system message and the last history message for Anthropic models, a default OpenRouter provider preference toward cache-capable routes for Google models (llm.openrouter.preferCacheRoutes), andcached_tokenson the usage record;reasoning/reasoning_content/thinkingdeltas accepted by default, Hermes and Qwen tagged calls parsed for every OpenAI-compatible kind when the whole content is the tagged block, a chunk carrying both text and tool calls keeps the text, and per-model params (o*/gpt-5ids droptemperatureand usemax_completion_tokens;llm.models[].paramsfor the rest). Config bump for the new fields.Why
Design call D5: native-tool providers get native multi-turn messages; the llama-server grammar path keeps the flat text, which is the KV-friendly layout. The three are one provider-body change — a native body without breakpoints or reasoning fields is half a change. The fabrication detectors from PR 2 stay as the safety net and are no-ops on native messages.
How it was verified
npm run lintcleannpx vitest run --minWorkers=1 --maxWorkers=3 src/agent src/analytics src/config src/llm/provider src/llm/provider/openai src/llm/provider/openrouter src/llm/provider/registry src/prompt src/runtime/llm-link-attempt.test.ts src/session— 129 files / 1977 tests green (baseline failures: none)max_tokenssentStacked on #432; merge in order.