diff --git a/AGENTS.md b/AGENTS.md index 1f38c9df..a540b2ef 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -569,6 +569,8 @@ Text completion, vision, embeddings, and sub-calls route through plugin-register - **Name escape** — qualified tool names use `__` for dots (`os.fs.read` → `os__fs__read`) in [openai-tool-call-adapter.ts](src/llm/provider/openai/openai-tool-call-adapter.ts). `reply` / `finish` are synthetic OpenAI functions alongside registry tools. - **Vendor presets** ([src/tui/providers/provider-presets.ts](src/tui/providers/provider-presets.ts)) — 19 named cloud/local endpoints (Anthropic, Groq, Moonshot, Perplexity, Qwen/DashScope, SambaNova, …) that all resolve to the existing `openai-compatible` kind with `baseUrl` prefilled. Adding a vendor is a preset entry, not a provider kind. The one documented exception is `subscription-cli` — a subprocess backend has no baseUrl, no key and no HTTP path, so a preset cannot express it; within that kind the preset philosophy re-applies one level down (a new vendor CLI is a descriptor entry, never a new kind). Vendors that do not authenticate with `Authorization: Bearer` set `apiKeyHeader` (Anthropic: `x-api-key`) plus any mandatory static `headers` (Anthropic: `anthropic-version`); both are copied onto the saved config entry by [providers-wizard-build-entry.ts](src/tui/providers/providers-wizard-build-entry.ts) and applied to **both** request paths by the single [openai-auth-headers.ts](src/llm/provider/openai/openai-auth-headers.ts) builder, so discovery and chat cannot disagree. The bar for a new entry: probe `/v1/models` **with the headers the preset will actually send** and get either 200 with a `data` array, or a 401/403 that rejects the *credential* — a 401 whose body names a header the preset does not send (`x-api-key header is required`, `Invalid bearer token` for what is an API key) is a **failing** probe, not a passing one. Either way the same host must answer 404 for a bogus sibling path; a gateway that rejects everything before routing proves nothing. - **OpenRouter provider routing** — `llm.providers[].providerPreferences` is sent verbatim as the body's `provider` object by `buildOpenAiChatBody` (turns and sub-calls, streaming and unary) and `describeImageViaOpenAi` (vision). Only the `openrouter` factory forwards it: no other kind documents a `provider` field. It is set *before* the `extraBody` merge, so an explicit `extraBody.provider` — the old workaround — still wins. Deliberately **not** sent by `verifyProviderKey` (it probes the cheapest paid model, which a host pinned for the operator's model may not serve, and would misreport a good key as `model_unavailable`), the contract probe (built from wizard state, which carries no entry passthroughs — `extraBody` is absent there too), the catalog fetch (`GET /models`), or OpenRouter embeddings (a pin chosen for a chat model's hosts would strand an embedding model). Pinned by [openrouter-provider-routing.test.ts](src/llm/provider/openrouter/openrouter-provider-routing.test.ts) and [register-built-in-providers.test.ts](src/llm/provider/registry/register-built-in-providers.test.ts). +- **Prompt caching (F27)** — Anthropic caches nothing without `cache_control` markers. For `anthropic/…` / `claude…` models and for Anthropic's own host, [prompt-cache-control.ts](src/llm/provider/openai/prompt-cache-control.ts) places two ephemeral breakpoints on the native layout: the system message and the last history message before the tail (never inside the changing tail, which would pay the cache-write premium for nothing; never on a `tool_calls`-only assistant message, which has no text part). `llm.providers[].promptCache` decides: `off` sends none, `explicit-markers` always sends them, `auto` (default) sends them for Anthropic only. Google models on OpenRouter cached 83 % of input on Google AI Studio and 1–4 % on Vertex, so an `openrouter` entry with no `providerPreferences` of its own sends `{order: ["Google AI Studio", "Google"], allow_fallbacks: true}` for `google/…` models, behind `llm.openrouter.preferCacheRoutes` (default on). `prompt_tokens_details.cached_tokens` is read on both paths into `usage.cachedTokens` and `cacheHitTokens` (the trace's field); the cost accumulator and the turn usage meter price cached tokens at the model's `cacheRead` rate ([usage-cost.ts](src/llm/provider/usage-cost.ts)). Pinned by [prompt-cache-control.test.ts](src/llm/provider/openai/prompt-cache-control.test.ts), the cache-route cases in [openrouter-provider-routing.test.ts](src/llm/provider/openrouter/openrouter-provider-routing.test.ts) and [usage-cost.test.ts](src/llm/provider/usage-cost.test.ts). +- **Reasoning fields, tagged calls and per-model parameters (F28)** — the provider's `reasoningFormat` defaults to `auto`, which reads `reasoning`, `reasoning_content` or `thinking` on both the streamed and the unary path; `userModels[].reasoningFormat` pins one field. A reply that is nothing but `` blocks — Qwen's `` form or the Hermes `{"name", "arguments"}` form — is decoded as tool calls on every OpenAI-compatible kind ([qwen-tagged-tool-response-adapter.ts](src/llm/provider/openai/qwen-tagged-tool-response-adapter.ts)); prose around a block keeps it a reply, and only the Qwen kind also reads the reasoning channel (#105). A streamed chunk carrying both text and a tool-call delta keeps its text. OpenAI reasoning models (`o1`…`o9`, `gpt-5`, with or without a vendor prefix) get no `temperature` and `max_completion_tokens` instead of `max_tokens` ([model-params.ts](src/llm/provider/openai/model-params.ts)); `userModels[].params` merges per-model wire parameters over the body and `extraBody` (reserved keys still win); `CompletionRequest.reasoningEffort` is spelled `reasoning: {effort}` on OpenRouter, `reasoning_effort` on OpenAI-compatible kinds and omitted elsewhere. - **Bundled catalogs** — `OPENROUTER_MODELS_CATALOG` (split across `openrouter-frontier-chat-models.ts` / `openrouter-open-weight-chat-models.ts`) and `AIMLAPI_MODELS_CATALOG` are offline snapshots regenerated from each vendor's public `/models` endpoint; the shared row builders live in [model-catalog-entry.ts](src/llm/provider/model-catalog-entry.ts). Refresh = re-pull the endpoint, remap (`context_length`, `input_modalities` → vision, `supported_parameters` → tools, price × 1e6 → USD/1M) and update the date in each file header. `scoreChat` in the OpenRouter fetcher **ranks** vendors; it must not gate them — the Anthropic/Gemini exclusions it used to carry hid ~40 served models from the picker. - **Model search** ([src/llm/provider/model-search.ts](src/llm/provider/model-search.ts)) — one ranked, multi-term scorer over model ids plus catalog metadata (vendor, `vision`/`text`, `tools`, `cache`, context shorthand like `1m`, `free`/`cheap`/`routed`). Tag matching is exact equality, so a context window is tagged three ways — as displayed (`1.0m`), floored to the whole unit (`1m`, the bucket a window falls in rather than a `>=` filter: 1_310_720 answers to both `1m` and `1.3m`, a 2M window only to `2m`), and, when the window is an exact multiple of 1024, in binary (131_072 answers to `128k`). Add a tag rather than changing [format-model-details.ts](src/llm/provider/format-model-details.ts): the display string is what the rows render. Terms are ANDed, matches are ranked (exact id > id prefix > vendor > word start > substring > subsequence) and equal ranks keep input order so the picker does not jitter per keystroke. Used by `filterModelIds` (TUI modal picker + Cloud pane) and by `atomic-agent models search`. Row rendering is shared through [format-model-details.ts](src/llm/provider/format-model-details.ts) — do not re-implement the price/context/capability strings in a frontend. @@ -606,7 +608,7 @@ Pinned by [find-strict-schema-violations.test.ts](src/llm/provider/openai/find-s ### Locked invariants 1. **Local llama-server path unchanged when no cloud provider is active.** Grammar, slots, and GBNF tests remain the reference behaviour. -2. **Stable prefix untouched.** Cloud providers receive the same monolithic prompt string; no chat-message refactor in v1. +2. **Stable prefix untouched; native-tools links get native messages (D5).** The stable prefix is still built once, byte for byte, and `buildPrompt` still returns the monolithic `text` every grammar link (llama-server, its GBNF prelude and KV layout) and every subscription CLI sends. Beside it, `BuiltPrompt.messages` exposes the same prompt as structure — `system` (the prefix), the packed turns with tool-result bodies capped exactly as the text form caps them, and the tail without `### conversation` — and the step executor puts it on `LlmStreamParams.messages` for a native-tools primary only. `llm-link-attempt` forwards it to a native link alone; `buildOpenAiChatBody` lays it out as `system` + assistant `tool_calls` (ids `call_`) answered by `tool` messages + one final `user` message ([openai-native-messages.ts](src/llm/provider/openai/openai-native-messages.ts)). Why: every cloud request used to be one `user` message holding the history as `assistant_tool_call:` / `tool_result[…]` text, and Gemini Flash kept continuing that text instead of calling tools (84,931 characters in one completion). `llm.providers[].messageShape: "flat"` restores the single message for a server that rejects tool-role messages, and a 400 about roles / `tool_call_id` / `messages` flips a session to flat and resends once (logged). The fabricated-transcript detector and stream watcher stay as the net for both shapes. Pinned by [openai-native-messages.test.ts](src/llm/provider/openai/openai-native-messages.test.ts), the message-shape cases in [openai-provider.test.ts](src/llm/provider/openai/openai-provider.test.ts) and [openai-build-body.test.ts](src/llm/provider/openai/openai-build-body.test.ts), [llm-link-attempt.test.ts](src/runtime/llm-link-attempt.test.ts) and the structured-prompt cases in [build-prompt.test.ts](src/prompt/build-prompt.test.ts) / [step-executor.test.ts](src/agent/step-executor.test.ts). 3. **One inference per step survives.** `toolCallsToBatch` produces the same `ToolCallBatch` shape as `parseToolCalls`. 4. **`atomic-agent serve` never proxies upstream.** HTTP `/v1/chat/completions` always funnels into `runtime.runTurn`. 5. **Plugin registration only.** New provider kinds call `registerProviderKind`; no central switch statements. diff --git a/src/agent/step-executor.test.ts b/src/agent/step-executor.test.ts index b99ebc6c..033fbff9 100644 --- a/src/agent/step-executor.test.ts +++ b/src/agent/step-executor.test.ts @@ -4548,3 +4548,90 @@ describe("executeStep tool roles (F18)", () => { expect(b.params.grammar).toBe(b.baseGrammar); }); }); + +describe("executeStep — the structured prompt on a native-tools link", () => { + const completion = (toolCalls?: CompletionResult["toolCalls"]): CompletionResult => ({ + content: "", + reasoningContent: toolCalls ? "" : "thinking only", + stop: true, + truncated: false, + timing: { promptMs: 1, predictedMs: 1, promptTokens: 20, predictedTokens: 5 }, + cacheHitTokens: 0, + slotId: -1, + modelId: "openai/gpt-5.5", + ...(toolCalls ? { toolCalls } : {}), + }); + const replyCall: CompletionResult["toolCalls"] = [ + { id: "c", type: "function", function: { name: "reply", arguments: JSON.stringify({ text: "ok" }) } }, + ]; + + async function run(toolTransport: ToolCallTransport, answers: CompletionResult[]) { + const registry = new ToolRegistry(); + registry.register(replyTool); + const base = createEmptySessionState({ id: `s-messages-${toolTransport}`, workingDir: "/w" }); + const session = { ...base, turns: [{ kind: "user" as const, text: "hi", at: 1 }] }; + const seen: Array[1]["llmComplete"]>>[0]> = []; + let call = 0; + const grammar = + toolTransport === "grammar" + ? await buildGrammar(PLAIN_INSTRUCT_PROFILE, join(process.cwd(), "grammars")) + : ""; + await executeStep( + { + session, + toolDescriptors: DEFAULT_TOOL_DESCRIPTORS, + capabilities: CAPS, + skillCatalog: SKILLS, + stepIndex: 0, + signal: new AbortController().signal, + userMessage: "hi", + }, + { + registry, + slotManager: new SlotManager(2), + async llmComplete(params) { + seen.push(params); + const answer = answers[Math.min(call, answers.length - 1)]!; + call += 1; + return answer; + }, + grammar, + profile: PLAIN_INSTRUCT_PROFILE, + toolTransport, + toolCallAdapter: null, + supportsSlotAffinity: toolTransport === "grammar", + }, + ); + return seen; + } + + it("carries `messages` beside the flat prompt, built from the same packed conversation", async () => { + const [params] = await run("native_tools", [completion(replyCall)]); + expect(params?.messages).toBeDefined(); + expect(params?.messages?.system).toBe(params?.prompt.slice(0, params.messages.system.length)); + expect(params?.messages?.turns).toEqual([{ kind: "user", text: "hi" }]); + expect(params?.messages?.tail).not.toContain("### conversation"); + expect(params?.prompt).toContain("### conversation\nuser: hi"); + }); + + it("re-shapes the structured tail for the one-shot repair, notice included", async () => { + const seen = await run("native_tools", [completion(), completion(replyCall)]); + expect(seen).toHaveLength(2); + const repair = seen[1]!; + expect(repair.prompt).toContain("### tool-call-repair"); + expect(repair.messages?.tail).toContain("### tool-call-repair"); + expect(repair.messages?.tail).toContain("native function-calling interface"); + expect(repair.messages?.system).toBe(seen[0]!.messages?.system); + expect(repair.messages?.turns).toEqual(seen[0]!.messages?.turns); + }); + + it("sends none on the grammar transport", async () => { + const grammarAnswer: CompletionResult = { + ...completion(), + reasoningContent: "", + content: JSON.stringify([{ tool: "reply", args: { text: "ok" } }]), + }; + const [params] = await run("grammar", [grammarAnswer]); + expect(params).not.toHaveProperty("messages"); + }); +}); diff --git a/src/agent/step-executor.ts b/src/agent/step-executor.ts index d894532f..aa63a400 100644 --- a/src/agent/step-executor.ts +++ b/src/agent/step-executor.ts @@ -99,6 +99,7 @@ import { type ModelProfile, } from "../llm/model-profile.js"; import type { + PromptMessages, ResponseFormatJsonSchema, ToolCallTransport, } from "../llm/provider/completion-types.js"; @@ -115,6 +116,16 @@ export type { PromptCapturedTokens, StepEvent } from "./step-events.js"; export interface LlmStreamParams { prompt: string; + /** + * The same prompt as structure — stable prefix, packed turns, tail — + * for a native-tools link that lays history out as real chat messages + * instead of one user message of transcript text (which Gemini Flash + * kept continuing as text instead of calling tools). Set only when the + * primary transport is `native_tools`; the seam forwards it on that + * transport alone, so a grammar fallback link still gets `prompt` / + * `grammarPrompt`. + */ + messages?: PromptMessages; /** * Lazy grammar-transport variant of `prompt`. Set when `prompt` was * built prefill-suppressed for a native-tools primary while the @@ -588,6 +599,7 @@ async function executeStepInner( const llmParams: LlmStreamParams = { ...buildLlmStreamParams({ promptText: prompt.text, + promptMessages: prompt.messages, deps, grammar: stepGrammar, slotId: slot.slotId, @@ -975,6 +987,24 @@ async function executeStepInner( deps.toolTransport, promptCarriesPrefill, ), + // The structured prompt must be repair-shaped too, or a native + // link would replay the stale tail without the notice. The notice + // lands at the end of the final user message; the chat form never + // carried a prefill, so there is nothing to strip. + ...(llmParams.messages + ? { + messages: { + ...llmParams.messages, + tail: buildToolCallRepairPrompt( + llmParams.messages.tail, + repairError, + deps.profile, + deps.toolTransport, + false, + ), + }, + } + : {}), // The grammar-link variant must be repair-shaped too — spreading // `llmParams` alone would hand a grammar fallback link the STALE // base prompt without the repair notice. It is repair-shaped for @@ -1956,6 +1986,7 @@ function resolveStepGrammar( function buildLlmStreamParams(args: { promptText: string; + promptMessages?: PromptMessages; deps: Pick< StepDependencies, | "toolTransport" @@ -1990,6 +2021,9 @@ function buildLlmStreamParams(args: { }); return { ...base, + // The structured prompt rides only on the native path; the seam + // forwards it only to a native link (`llm-link-attempt.ts`). + ...(args.promptMessages ? { messages: args.promptMessages } : {}), // Keep `grammar` populated (not blanked) even on the native path: the // provider fallback chain may hand this request to a grammar-only // llama-server link, which needs the GBNF. Native (cloud) providers diff --git a/src/analytics/turn-usage-meter.test.ts b/src/analytics/turn-usage-meter.test.ts index 981f50f0..6a5c7718 100644 --- a/src/analytics/turn-usage-meter.test.ts +++ b/src/analytics/turn-usage-meter.test.ts @@ -211,3 +211,35 @@ describe("TurnUsageMeter", () => { }); }); }); + +describe("TurnUsageMeter — cached prompt tokens", () => { + it("sums cachedTokens across calls and prices them at cacheRead", () => { + const meter = new TurnUsageMeter(); + meter.begin(S); + const model: ResolvedModel = { + ...pricedModel(3, 15), + pricing: { input: 3, output: 15, cacheRead: 0.3 }, + }; + meter.record({ + sessionId: S, + usage: { ...usage(1_000_000, 0), cachedTokens: 800_000 }, + model, + }); + meter.record({ + sessionId: S, + usage: { ...usage(1_000_000, 0), cachedTokens: 200_000 }, + model, + }); + const snap = meter.snapshot(S); + expect(snap.cachedTokens).toBe(1_000_000); + // (200k × 3 + 800k × 0.3) + (800k × 3 + 200k × 0.3), per million. + expect(snap.costUsd).toBeCloseTo(0.84 + 2.46, 9); + }); + + it("omits cachedTokens when no call reported the figure", () => { + const meter = new TurnUsageMeter(); + meter.begin(S); + meter.record({ sessionId: S, usage: usage(10, 5), model: pricedModel(3, 15) }); + expect(meter.snapshot(S)).not.toHaveProperty("cachedTokens"); + }); +}); diff --git a/src/analytics/turn-usage-meter.ts b/src/analytics/turn-usage-meter.ts index d0893cf0..3e50f74f 100644 --- a/src/analytics/turn-usage-meter.ts +++ b/src/analytics/turn-usage-meter.ts @@ -1,5 +1,6 @@ import type { CompletionUsage } from "../llm/provider/completion-types.js"; import type { ResolvedModel } from "../llm/provider/model-resolver.js"; +import { estimateUsageCostUsd } from "../llm/provider/usage-cost.js"; /** * Token + spend totals accumulated for a single turn. @@ -18,14 +19,23 @@ import type { ResolvedModel } from "../llm/provider/model-resolver.js"; export interface TurnUsageSnapshot { promptTokens?: number; completionTokens?: number; + /** + * Prompt tokens the services served from a prompt cache, summed over + * the turn's calls. Present only when at least one call reported the + * figure: a provider that says nothing about caching yields no key, + * which is different from one that reports zero hits. + */ + cachedTokens?: number; costUsd?: number; } interface TurnBucket { promptTokens: number; completionTokens: number; + cachedTokens: number; costUsd: number; sawUsage: boolean; + sawCachedUsage: boolean; sawPricedUsage: boolean; } @@ -57,8 +67,10 @@ export class TurnUsageMeter { this.buckets.set(sessionId, { promptTokens: 0, completionTokens: 0, + cachedTokens: 0, costUsd: 0, sawUsage: false, + sawCachedUsage: false, sawPricedUsage: false, }); } @@ -82,13 +94,15 @@ export class TurnUsageMeter { bucket.sawUsage = true; bucket.promptTokens += usage.promptTokens; bucket.completionTokens += usage.completionTokens; + if (usage.cachedTokens !== undefined) { + bucket.sawCachedUsage = true; + bucket.cachedTokens += usage.cachedTokens; + } const pricing = params.model?.pricing; if (!pricing) return; bucket.sawPricedUsage = true; - bucket.costUsd += - (usage.promptTokens / 1_000_000) * pricing.input + - (usage.completionTokens / 1_000_000) * pricing.output; + bucket.costUsd += estimateUsageCostUsd(usage, pricing); } /** @@ -107,6 +121,7 @@ export class TurnUsageMeter { return { promptTokens: bucket.promptTokens, completionTokens: bucket.completionTokens, + ...(bucket.sawCachedUsage ? { cachedTokens: bucket.cachedTokens } : {}), ...(bucket.sawPricedUsage ? { costUsd: bucket.costUsd } : {}), }; } diff --git a/src/config/config-schema.ts b/src/config/config-schema.ts index d6049a02..d1e9a2e9 100644 --- a/src/config/config-schema.ts +++ b/src/config/config-schema.ts @@ -971,6 +971,12 @@ export interface AtomicAgentConfig { supportsVision?: boolean; requestTimeoutMs?: number; promptCache?: "auto" | "off" | "explicit-markers"; + /** + * Native-tools request layout: `native` (default) sends a system + * message plus the history as assistant `tool_calls` / `tool` + * results; `flat` sends the one user message of transcript text. + */ + messageShape?: "native" | "flat"; /** * OpenRouter provider routing (`order`, `only`, `ignore`, * `allow_fallbacks`, `require_parameters`, `sort`, @@ -1015,6 +1021,7 @@ export interface AtomicAgentConfig { supportsTools?: "none" | "basic" | "parallel" | "strict"; supportsPromptCache?: boolean; reasoningFormat?: + | "auto" | "none" | "delta_reasoning" | "delta_thinking" @@ -1025,6 +1032,11 @@ export interface AtomicAgentConfig { cacheRead?: number; cacheWrite?: number; }; + /** + * Wire parameters for this model, merged into every chat body + * after the provider's `extraBody`. Reserved keys still win. + */ + params?: Record; }>; }>; toolTransport: "auto" | "grammar" | "native_tools"; @@ -1056,6 +1068,15 @@ export interface AtomicAgentConfig { * `src/llm/run-mode/resolve-run-mode.ts`. */ runMode?: UserLlmRunModeConfig; + /** + * Settings for every `openrouter` entry at once. + * `preferCacheRoutes` (default `true`) pins `google/…` models to the + * routes that honour prompt caching unless the entry configured + * `providerPreferences` itself. + */ + openrouter?: { + preferCacheRoutes?: boolean; + }; }; } @@ -2175,6 +2196,13 @@ export interface UserConfigFile { // transcript keeps after a cut (default 0.65), so the cut holds and the // prompt only grows at its end between cuts. Additive: an older file has // no field and takes the default. +// v66: provider entries accept `messageShape` (`native` | `flat`, the +// layout of a native-tools request), `userModels[].params` (per-model +// wire parameters merged over `extraBody`) and `userModels[].reasoningFormat` +// accepts `auto`; a new `llm.openrouter` block carries `preferCacheRoutes` +// (default `true`). All additive: an older file parses with every field +// absent, which is the native layout, no extra parameters, `auto` +// reasoning and the cache-capable routes for Google models. export const USER_CONFIG_VERSION = 66; /** diff --git a/src/config/llm-config.test.ts b/src/config/llm-config.test.ts index 1d29596a..05c624dd 100644 --- a/src/config/llm-config.test.ts +++ b/src/config/llm-config.test.ts @@ -688,3 +688,82 @@ describe("provider strictTools", () => { } }); }); + +describe("userModels[].params and reasoningFormat auto", () => { + const withRow = (row: Record) => ({ + version: USER_CONFIG_VERSION, + llm: { + activeTextProvider: "cloud", + activeEmbeddingProvider: "local-llama", + toolTransport: "auto" as const, + providers: [ + { id: "local-llama", kind: "llama-server", url: "http://127.0.0.1:19091" }, + { + id: "cloud", + kind: "openai-compatible", + baseUrl: "https://example.invalid", + defaultChatModel: "m", + userModels: [{ id: "m", kind: "chat", ...row }], + }, + ], + }, + }); + const row = (file: ReturnType) => + file.llm?.providers[1]?.userModels?.[0]; + + it("round-trips params as a plain object", () => { + expect( + row(parseUserConfigFile(withRow({ params: { top_p: 0.9, reasoning_effort: "low" } }))) + ?.params, + ).toEqual({ top_p: 0.9, reasoning_effort: "low" }); + }); + + it("is absent by default", () => { + expect(row(parseUserConfigFile(withRow({})))?.params).toBeUndefined(); + }); + + it("rejects params that are not an object", () => { + for (const bad of ["x", 1, ["a"]]) { + expect(() => parseUserConfigFile(withRow({ params: bad }))).toThrow( + /userModels\[0\]\.params/, + ); + } + }); + + it("accepts `auto` as a reasoning format", () => { + expect( + row(parseUserConfigFile(withRow({ reasoningFormat: "auto" })))?.reasoningFormat, + ).toBe("auto"); + }); +}); + +describe("llm.openrouter", () => { + const withOpenRouter = (openrouter: unknown) => ({ + version: USER_CONFIG_VERSION, + llm: { + activeTextProvider: "local-llama", + activeEmbeddingProvider: "local-llama", + toolTransport: "auto" as const, + providers: [{ id: "local-llama", kind: "llama-server", url: "http://127.0.0.1:19091" }], + ...(openrouter === undefined ? {} : { openrouter }), + }, + }); + + it("round-trips preferCacheRoutes", () => { + expect( + parseUserConfigFile(withOpenRouter({ preferCacheRoutes: false })).llm?.openrouter, + ).toEqual({ preferCacheRoutes: false }); + }); + + it("is absent by default, and an empty block stays empty", () => { + expect(parseUserConfigFile(withOpenRouter(undefined)).llm?.openrouter).toBeUndefined(); + expect(parseUserConfigFile(withOpenRouter({})).llm?.openrouter).toEqual({}); + }); + + it("rejects a non-boolean preferCacheRoutes and a non-object block", () => { + expect(() => parseUserConfigFile(withOpenRouter({ preferCacheRoutes: "yes" }))).toThrow( + /llm\.openrouter\.preferCacheRoutes/, + ); + expect(() => parseUserConfigFile(withOpenRouter(["x"]))).toThrow(/llm\.openrouter/); + }); +}); diff --git a/src/config/llm-config.ts b/src/config/llm-config.ts index d8f1ad11..bffdb24b 100644 --- a/src/config/llm-config.ts +++ b/src/config/llm-config.ts @@ -70,11 +70,22 @@ export type UserLlmProviderEntry = { */ maxOutputTokens?: number; /** - * Prompt-caching policy for this provider. Declared in the config - * schema and on `LlmProviderConfigEntry`; no provider reads it yet, - * so today it only has to survive the round-trip through config. + * Prompt-caching policy for this provider. `off` sends no cache + * markers; `explicit-markers` always sends Anthropic breakpoints; + * `auto` (the default) sends them when the model or host is + * Anthropic's. See `openai/prompt-cache-control.ts`. */ promptCache?: "auto" | "off" | "explicit-markers"; + /** + * How a native-tools request lays the prompt out: `native` (the + * default) as a system message plus the history as assistant + * `tool_calls` and `tool` results, `flat` as the single user message + * of transcript text. Set `flat` for a server that rejects tool-role + * messages (older vLLM, llama.cpp shims); the provider also learns it + * from a 400 about roles for the rest of a session. Grammar and + * subscription-CLI kinds are flat by construction. + */ + messageShape?: "native" | "flat"; /** * OpenRouter provider routing — `order`, `only`, `ignore`, * `allow_fallbacks`, `require_parameters`, `sort`, `data_collection`, @@ -146,14 +157,32 @@ export type UserModelEntry = { supportsVision?: boolean; supportsTools?: "none" | "basic" | "parallel" | "strict"; supportsPromptCache?: boolean; + /** + * Which response field carries the model's reasoning. `auto` (the + * provider default when unset) reads `reasoning`, `reasoning_content` + * or `thinking`, whichever is present; a named format pins one. + */ reasoningFormat?: - "none" | "delta_reasoning" | "delta_thinking" | "delta_reasoning_content"; + | "auto" + | "none" + | "delta_reasoning" + | "delta_thinking" + | "delta_reasoning_content"; pricing?: { input: number; output: number; cacheRead?: number; cacheWrite?: number; }; + /** + * Wire parameters for this model, merged into every OpenAI-compatible + * chat body after the provider's `extraBody` (a model-level setting is + * the more specific one). The way to hand a model a `temperature`, + * `top_p`, `reasoning_effort` or any vendor field the runtime does not + * model; reserved keys (`model`, `messages`, `stream`, `tools`) still + * cannot be overridden. + */ + params?: Record; }; export type UserLlmFallbackConfig = { @@ -165,6 +194,21 @@ export type UserLlmFallbackConfig = { failureWindowMs?: number; }; +/** + * OpenRouter-wide settings, as opposed to a single `openrouter` entry's. + */ +export type UserLlmOpenRouterConfig = { + /** + * Steer models whose caching depends on the route toward the routes + * that cache — today `google/…` models to Google AI Studio, which + * cached 83 % of input where Vertex cached 1–4 % — unless the entry + * configured `providerPreferences` of its own. Default `true`; a + * pinned route can raise latency or lose availability while it is + * down, so it is a default an operator can turn off. + */ + preferCacheRoutes?: boolean; +}; + export type UserLlmFileConfig = { activeTextProvider: string; activeEmbeddingProvider: string; @@ -173,6 +217,8 @@ export type UserLlmFileConfig = { fallback?: UserLlmFallbackConfig; /** Run mode (local | cloud | fusion) and the fusion legs. See `llm-run-mode-config.ts`. */ runMode?: UserLlmRunModeConfig; + /** Settings for every `openrouter` entry at once. */ + openrouter?: UserLlmOpenRouterConfig; }; const PROVIDER_ID_RE = /^[a-z][a-z0-9-]{0,31}$/; @@ -325,6 +371,9 @@ export function parseLlmProviderEntry( promptCache: parseOptionalEnum< NonNullable >(obj.promptCache, `${field}.promptCache`, PROMPT_CACHE_MODES), + messageShape: parseOptionalEnum< + NonNullable + >(obj.messageShape, `${field}.messageShape`, MESSAGE_SHAPES), providerPreferences: parseOptionalPlainObject( obj.providerPreferences, `${field}.providerPreferences`, @@ -422,8 +471,10 @@ function parseOptionalPlainObject( } const PROMPT_CACHE_MODES = new Set(["auto", "off", "explicit-markers"]); +const MESSAGE_SHAPES = new Set(["native", "flat"]); const TOOLS_SUPPORT_LEVELS = new Set(["none", "basic", "parallel", "strict"]); const REASONING_FORMATS = new Set([ + "auto", "none", "delta_reasoning", "delta_thinking", @@ -527,6 +578,7 @@ function parseUserModelEntry(raw: unknown, field: string): UserModelEntry { NonNullable >(obj.reasoningFormat, `${field}.reasoningFormat`, REASONING_FORMATS), pricing: parseUserModelPricing(obj.pricing, `${field}.pricing`), + params: parseOptionalPlainObject(obj.params, `${field}.params`), }; } @@ -732,6 +784,8 @@ export function parseUserLlmFileConfig( ? undefined : parseLlmRunModeConfig(obj.runMode, providers, "llm.runMode"); + const openrouter = parseLlmOpenRouterConfig(obj.openrouter, "llm.openrouter"); + return { activeTextProvider, activeEmbeddingProvider, @@ -739,5 +793,22 @@ export function parseUserLlmFileConfig( providers, ...(fallback ? { fallback } : {}), ...(runMode ? { runMode } : {}), + ...(openrouter ? { openrouter } : {}), }; } + +export function parseLlmOpenRouterConfig( + raw: unknown, + field: string, +): UserLlmOpenRouterConfig | undefined { + if (raw === undefined || raw === null) return undefined; + if (typeof raw !== "object" || Array.isArray(raw)) { + throw new ConfigValidationError(field, "expected object"); + } + const obj = raw as Record; + const preferCacheRoutes = parseOptionalBoolean( + obj.preferCacheRoutes, + `${field}.preferCacheRoutes`, + ); + return preferCacheRoutes === undefined ? {} : { preferCacheRoutes }; +} diff --git a/src/llm/provider/completion-types.ts b/src/llm/provider/completion-types.ts index 7f0ba685..bfde9e88 100644 --- a/src/llm/provider/completion-types.ts +++ b/src/llm/provider/completion-types.ts @@ -10,10 +10,62 @@ export interface CompletionUsage { promptTokens: number; completionTokens: number; totalTokens: number; + /** + * Prompt tokens the service served from its prompt cache + * (`prompt_tokens_details.cached_tokens` on OpenAI, OpenRouter and + * Gemini's compatibility layer). A subset of `promptTokens`, priced at + * the model's `cacheRead` rate when one is known. Absent — not zero — + * when the service did not report it. + */ + cachedTokens?: number; +} + +/** + * One turn of the packed conversation, as the prompt builder renders it: + * the same rows `### conversation` carries as text, with each + * tool-result body already capped exactly as the text form caps it. + * Provider-neutral on purpose — a native-tools provider lays these out as + * real chat messages, a grammar provider never reads them. + */ +export type PromptTurn = + | { kind: "user"; text: string } + | { kind: "assistant_reply"; text: string } + | { kind: "assistant_tool_call"; tool: string; args: Record } + | { + kind: "tool_result"; + tool: string; + status: "ok" | "error"; + body: string; + truncated: boolean; + }; + +/** + * The prompt as structure instead of as one string: the three zones a + * native-message request lays out as `system`, history and a final + * `user` message. `prompt` (the flat text) is always present beside it + * and is what every other transport sends; the two are built from the + * same packed conversation, so they cannot disagree about what the + * model sees. + */ +export interface PromptMessages { + /** The stable prefix, byte for byte. */ + system: string; + /** The packer's one-line recap of dropped turns, or `null`. */ + droppedSummary: string | null; + /** The visible conversation, oldest first. */ + turns: ReadonlyArray; + /** The variable tail without `### conversation`: the final user message. */ + tail: string; } export interface CompletionRequest { prompt: string; + /** + * The same prompt as structure. Set only for main-turn requests built + * for a native-tools link; providers that lay history out as real + * messages read it, everything else ignores it and sends `prompt`. + */ + messages?: PromptMessages; grammar?: string; slotId?: number; cachePrompt?: boolean; @@ -43,6 +95,14 @@ export interface CompletionRequest { * they rely on `grammar` instead. */ responseFormat?: ResponseFormatJsonSchema; + /** + * How hard a reasoning model should think on this completion. Spelled + * per vendor by the body builder (`reasoning: { effort }` on + * OpenRouter, `reasoning_effort` on OpenAI-compatible services) and + * omitted for kinds that document neither. Ignored by grammar-only + * providers. Set by the fusion fan-out for its workers. + */ + reasoningEffort?: "low" | "medium" | "high"; } /** diff --git a/src/llm/provider/cost-accumulator.ts b/src/llm/provider/cost-accumulator.ts index 3ac8c1de..36dd567f 100644 --- a/src/llm/provider/cost-accumulator.ts +++ b/src/llm/provider/cost-accumulator.ts @@ -1,5 +1,6 @@ import type { CompletionUsage } from "./completion-types.js"; import type { ResolvedModel } from "./model-resolver.js"; +import { estimateUsageCostUsd } from "./usage-cost.js"; export type CostAccumulatorSnapshot = { sessionUsd: number; @@ -66,10 +67,7 @@ function estimateCost( model: ResolvedModel | undefined, ): number { if (!usage || !model?.pricing) return 0; - const { input, output } = model.pricing; - const prompt = usage.promptTokens / 1_000_000; - const completion = usage.completionTokens / 1_000_000; - return prompt * input + completion * output; + return estimateUsageCostUsd(usage, model.pricing); } function utcDayKey(now: Date, resetHourUtc: number): string { diff --git a/src/llm/provider/llm-provider.ts b/src/llm/provider/llm-provider.ts index a8a32bd2..41f5a592 100644 --- a/src/llm/provider/llm-provider.ts +++ b/src/llm/provider/llm-provider.ts @@ -34,8 +34,22 @@ export interface ProviderHealthResult { export type ToolsSupportLevel = "none" | "basic" | "parallel" | "strict"; +/** + * Which field of an OpenAI-compatible message/delta carries the model's + * reasoning. `auto` (the provider default) reads whichever of + * `reasoning`, `reasoning_content` and `thinking` is present — OpenRouter, + * DeepSeek-style servers and Anthropic-compatible shims each use a + * different one, and a stream whose field is not the configured one used + * to lose its reasoning silently. The named formats pin one field for a + * model whose server also writes a *different* field with something that + * is not reasoning. + */ export type ReasoningFormat = - "none" | "delta_reasoning" | "delta_thinking" | "delta_reasoning_content"; + | "auto" + | "none" + | "delta_reasoning" + | "delta_thinking" + | "delta_reasoning_content"; /** * Snapshot of what a provider can do. `toolTransport` drives whether diff --git a/src/llm/provider/openai/model-params.ts b/src/llm/provider/openai/model-params.ts new file mode 100644 index 00000000..ea25ebe8 --- /dev/null +++ b/src/llm/provider/openai/model-params.ts @@ -0,0 +1,57 @@ +/** + * Per-model-family wire parameters for OpenAI-compatible chat bodies. + * + * Two facts the builder needs about a model id, kept next to each other + * so the rule that reads the id lives in one place: + * + * - OpenAI's reasoning models (`o1`, `o3`, `o4-mini`, `gpt-5`, …) reject + * `temperature` and reject `max_tokens` in favour of + * `max_completion_tokens`. atag's own `request-size-rejection.ts` + * quotes the second rejection verbatim and used to rely on the + * fallback chain to survive it; the first simply failed the turn. + * - Everything else keeps the historical body byte for byte. + * + * The id is read after any vendor prefix (`openai/o3` on OpenRouter is + * the same model as `o3` on OpenAI), and the prefix test is anchored so + * a name that merely starts with the letter `o` (`olmo`) is not a + * reasoning model. + */ +export interface ModelParamProfile { + /** Whether the model accepts `temperature` at all. */ + temperature: boolean; + /** The field the output cap is spelled in. */ + capField: "max_tokens" | "max_completion_tokens"; +} + +const OPENAI_REASONING_MODEL_RE = /^(?:o[1-9]|gpt-5)(?![a-z])/i; + +export function modelParamProfile(modelId: string): ModelParamProfile { + const bare = modelId.includes("/") + ? modelId.slice(modelId.lastIndexOf("/") + 1) + : modelId; + if (OPENAI_REASONING_MODEL_RE.test(bare)) { + return { temperature: false, capField: "max_completion_tokens" }; + } + return { temperature: true, capField: "max_tokens" }; +} + +/** + * How `reasoningEffort` is spelled for a provider kind. OpenRouter takes a + * `reasoning` object; OpenAI-compatible services take the flat + * `reasoning_effort` OpenAI documents; the rest document neither, and a + * field a service does not know is at best ignored and at worst a 400. + */ +export function reasoningEffortField( + providerKind: string | undefined, + effort: "low" | "medium" | "high", +): Record { + switch (providerKind) { + case "openrouter": + return { reasoning: { effort } }; + case "openai-compatible": + case "qwen-openai-compatible": + return { reasoning_effort: effort }; + default: + return {}; + } +} diff --git a/src/llm/provider/openai/openai-build-body.test.ts b/src/llm/provider/openai/openai-build-body.test.ts index eb593b80..9c684f71 100644 --- a/src/llm/provider/openai/openai-build-body.test.ts +++ b/src/llm/provider/openai/openai-build-body.test.ts @@ -479,3 +479,166 @@ describe("buildOpenAiChatBody — strict function tools", () => { expect("tools" in body).toBe(false); }); }); + +describe("buildOpenAiChatBody — per-model parameters", () => { + // OpenAI's reasoning models reject `temperature` and answer + // `max_tokens` with "Use 'max_completion_tokens' instead." — the + // rejection `request-size-rejection.ts` quotes. Read after any + // vendor prefix, so the same model through OpenRouter gets the same + // body. + it.each(["o3", "o4-mini", "o1-preview", "gpt-5", "gpt-5.2-mini", "openai/o3", "openai/gpt-5-codex"])( + "sends no temperature and max_completion_tokens for %s", + (model) => { + const body = buildOpenAiChatBody( + { prompt: "hi", temperature: 0.7, maxTokens: 512 }, + model, + false, + ); + expect(body).not.toHaveProperty("temperature"); + expect(body).not.toHaveProperty("max_tokens"); + expect(body.max_completion_tokens).toBe(512); + }, + ); + + it.each(["olmo-3", "gpt-4.1", "google/gemini-3.8-flash", "qwen/qwen3.8-27b", "o-mega"])( + "keeps the historical body for %s", + (model) => { + const body = buildOpenAiChatBody({ prompt: "hi", maxTokens: 512 }, model, false); + expect(body.temperature).toBe(0.2); + expect(body.max_tokens).toBe(512); + expect(body).not.toHaveProperty("max_completion_tokens"); + }, + ); + + it("merges userModels[].params over the body and over extraBody, reserved keys excepted", () => { + const body = buildOpenAiChatBody( + { prompt: "hi" }, + "m", + false, + { top_p: 0.5, chat_template_kwargs: { enable_thinking: false } }, + undefined, + undefined, + undefined, + { + modelParams: { + top_p: 0.9, + temperature: 1, + model: "other", + messages: [], + }, + }, + ); + expect(body.top_p).toBe(0.9); + expect(body.temperature).toBe(1); + expect(body.chat_template_kwargs).toEqual({ enable_thinking: false }); + expect(body.model).toBe("m"); + expect(body.messages).toEqual([{ role: "user", content: "hi" }]); + }); + + it("spells reasoningEffort the way each kind documents it, and omits it elsewhere", () => { + const request = { prompt: "hi", reasoningEffort: "low" as const }; + expect( + buildOpenAiChatBody(request, "m", false, undefined, undefined, undefined, undefined, { + providerKind: "openrouter", + }).reasoning, + ).toEqual({ effort: "low" }); + expect( + buildOpenAiChatBody(request, "m", false, undefined, undefined, undefined, undefined, { + providerKind: "openai-compatible", + }).reasoning_effort, + ).toBe("low"); + for (const providerKind of ["gemini", "aimlapi", undefined]) { + const body = buildOpenAiChatBody( + request, + "m", + false, + undefined, + undefined, + undefined, + undefined, + { ...(providerKind ? { providerKind } : {}) }, + ); + expect(body).not.toHaveProperty("reasoning"); + expect(body).not.toHaveProperty("reasoning_effort"); + } + // Absent, nothing is sent whatever the kind. + const plain = buildOpenAiChatBody({ prompt: "hi" }, "m", false, undefined, undefined, undefined, undefined, { + providerKind: "openrouter", + }); + expect(plain).not.toHaveProperty("reasoning"); + }); +}); + +describe("buildOpenAiChatBody — the native message layout", () => { + const tools = [{ type: "function", function: { name: "os__fs__read" } }]; + const messages = { + system: "### system\nprefix", + droppedSummary: null, + turns: [ + { kind: "user" as const, text: "read a" }, + { kind: "assistant_tool_call" as const, tool: "os.fs.read", args: { path: "a" } }, + { kind: "tool_result" as const, tool: "os.fs.read", status: "ok" as const, body: "A", truncated: false }, + ], + tail: "### respond\nRespond now.\n", + }; + + it("sends system, history and a final user message when the request is structured", () => { + const body = buildOpenAiChatBody({ prompt: "flat", messages, tools }, "m", true); + expect(body.messages).toEqual([ + { role: "system", content: "### system\nprefix" }, + { role: "user", content: "read a" }, + { + role: "assistant", + content: null, + tool_calls: [ + { id: "call_1", type: "function", function: { name: "os__fs__read", arguments: '{"path":"a"}' } }, + ], + }, + { role: "tool", tool_call_id: "call_1", content: "A" }, + { role: "user", content: "### respond\nRespond now.\n" }, + ]); + expect(body.tools).toEqual(tools); + }); + + it("sends the flat prompt when the shape is flat, when there are no tools, or when nothing is structured", () => { + const flat = [{ role: "user", content: "flat" }]; + expect( + buildOpenAiChatBody({ prompt: "flat", messages, tools }, "m", true, undefined, undefined, undefined, undefined, { + messageShape: "flat", + }).messages, + ).toEqual(flat); + expect(buildOpenAiChatBody({ prompt: "flat", messages }, "m", true).messages).toEqual(flat); + expect(buildOpenAiChatBody({ prompt: "flat", tools }, "m", true).messages).toEqual(flat); + }); + + it("places the Anthropic breakpoints on the native layout", () => { + const body = buildOpenAiChatBody( + { prompt: "flat", messages, tools }, + "anthropic/claude-sonnet-4.5", + true, + undefined, + undefined, + undefined, + undefined, + { anthropicCacheControl: true }, + ); + const sent = body.messages as Array>; + expect(sent[0]?.content).toEqual([ + { type: "text", text: "### system\nprefix", cache_control: { type: "ephemeral" } }, + ]); + expect(sent[3]?.content).toEqual([ + { type: "text", text: "A", cache_control: { type: "ephemeral" } }, + ]); + expect(sent[4]).toEqual({ role: "user", content: "### respond\nRespond now.\n" }); + }); + + it("uses the adapter's own name escape when one is given", () => { + const body = buildOpenAiChatBody({ prompt: "flat", messages, tools }, "m", true, undefined, undefined, undefined, undefined, { + nameEscape: (name) => name.toUpperCase(), + }); + const assistant = (body.messages as Array>)[2] as { + tool_calls: Array<{ function: { name: string } }>; + }; + expect(assistant.tool_calls[0]?.function.name).toBe("OS.FS.READ"); + }); +}); diff --git a/src/llm/provider/openai/openai-build-body.ts b/src/llm/provider/openai/openai-build-body.ts index cbb6a2d0..0701fa41 100644 --- a/src/llm/provider/openai/openai-build-body.ts +++ b/src/llm/provider/openai/openai-build-body.ts @@ -2,7 +2,11 @@ import type { CompletionRequest } from "../completion-types.js"; import { hasStrictFunctionTools } from "../adapters/tool-call-adapter.js"; import { ensureJsonMention } from "./ensure-json-mention.js"; import { filterCloudCompletionRequest } from "./sampling-filter.js"; +import { modelParamProfile, reasoningEffortField } from "./model-params.js"; +import { buildNativeMessages } from "./openai-native-messages.js"; import { toStrictOpenAiTools } from "./openai-strict-tools.js"; +import { nameEscape } from "./openai-tool-call-adapter.js"; +import { applyAnthropicCacheControl } from "./prompt-cache-control.js"; /** * Fields the caller owns unconditionally. `extraBody` is merged *under* @@ -12,6 +16,51 @@ import { toStrictOpenAiTools } from "./openai-strict-tools.js"; */ const RESERVED_BODY_KEYS = ["model", "messages", "stream", "tools"] as const; +/** + * What the builder knows about the provider and model beyond the request + * itself. Every field is optional and absent leaves the body exactly as + * it was before the field existed. + */ +export interface OpenAiBodyOptions { + /** + * Per-model wire parameters from `userModels[].params`, merged over the + * body *and* over `extraBody` — a model-level setting is more specific + * than a provider-level one. Reserved keys still win. + */ + modelParams?: Record; + /** + * The registered kind sending this body, for fields whose spelling is + * the vendor's (`reasoningEffort`). Absent, those fields are omitted. + */ + providerKind?: string; + /** + * Place Anthropic prompt-cache breakpoints on the messages (see + * `prompt-cache-control.ts`). Decided by the provider from the model + * id, the host and the entry's `promptCache` policy. + */ + anthropicCacheControl?: boolean; + /** + * How a request that carries `messages` (the structured prompt) is + * laid out: `native` as `system` + history + final `user` + * (`openai-native-messages.ts`), `flat` as the one `user` message of + * text every request used to be. A request without `messages` is + * always flat. Default `native`. + */ + messageShape?: "native" | "flat"; + /** The adapter's tool-name escape, for the history's `tool_calls`. */ + nameEscape?: (qualifiedName: string) => string; +} + +/** The wire layout `buildOpenAiChatBody` chose for a request. */ +export function resolveMessageShape( + request: Pick, + options: Pick, +): "native" | "flat" { + if (!request.messages) return "flat"; + if (!request.tools || request.tools.length === 0) return "flat"; + return options.messageShape ?? "native"; +} + export function buildOpenAiChatBody( request: CompletionRequest, defaultChatModel: string, @@ -20,8 +69,10 @@ export function buildOpenAiChatBody( maxOutputTokens?: number, strictTools?: boolean, providerPreferences?: Record, + options: OpenAiBodyOptions = {}, ): Record { const filtered = filterCloudCompletionRequest(request); + const profile = modelParamProfile(defaultChatModel); // Settled before the body exists because it also decides the prompt: // a request that sends `response_format` must mention JSON (see // `ensureJsonMention`). The tools guard is explained where @@ -30,17 +81,33 @@ export function buildOpenAiChatBody( filtered.tools && filtered.tools.length > 0 ? undefined : filtered.responseFormat; + // The structured prompt rides only on a main turn (it needs `tools` + // to answer with), and only when the provider takes the native + // layout; a sub-call, or a service that refused the layout, sends the + // flat text — which `messages` was built beside, from the same packed + // conversation, so both say the same thing. + const messages = + resolveMessageShape(filtered, options) === "native" && filtered.messages + ? buildNativeMessages(filtered.messages, { + nameEscape: options.nameEscape ?? nameEscape, + }) + : [ + { + role: "user", + content: responseFormat + ? ensureJsonMention(filtered.prompt) + : filtered.prompt, + }, + ]; const body: Record = { model: defaultChatModel, - messages: [ - { - role: "user", - content: responseFormat - ? ensureJsonMention(filtered.prompt) - : filtered.prompt, - }, - ], - temperature: filtered.temperature ?? 0.2, + messages, + // OpenAI's reasoning models reject the field outright (`Unsupported + // parameter: 'temperature'`), so for them it is not sent at all — + // not even a caller's own value. See `model-params.ts`. + ...(profile.temperature + ? { temperature: filtered.temperature ?? 0.2 } + : {}), stream, }; // `max_tokens` only when somebody actually asked for a bound. @@ -61,9 +128,17 @@ export function buildOpenAiChatBody( // through the entry's `extraBody` — `max_tokens` is deliberately not // in `RESERVED_BODY_KEYS`, so that passthrough wins. // Order: what this call asked for, else the provider's configured - // ceiling, else nothing at all. + // ceiling, else nothing at all. The field is the model family's own: + // OpenAI's reasoning models answer `max_tokens` with "'max_tokens' is + // not supported with this model. Use 'max_completion_tokens' instead." const cap = filtered.maxTokens ?? maxOutputTokens; - if (typeof cap === "number") body.max_tokens = cap; + if (typeof cap === "number") body[profile.capField] = cap; + if (filtered.reasoningEffort !== undefined) { + Object.assign( + body, + reasoningEffortField(options.providerKind, filtered.reasoningEffort), + ); + } if (stream) { // Ask for the usage block on the stream's last chunk. Without it // most servers send none — OpenAI, llama.cpp and everything built on @@ -132,10 +207,21 @@ export function buildOpenAiChatBody( // `extraBody.provider` is the older way to say the same thing, and it // keeps winning. Absent, the body is byte-identical to what it was. if (providerPreferences) body.provider = providerPreferences; - if (!extraBody) return body; - // Vendor passthrough. Merged last so it can reach fields this builder - // does not model, then reserved keys are restored on top. - const merged: Record = { ...body, ...extraBody }; + if (options.anthropicCacheControl) { + body.messages = applyAnthropicCacheControl( + body.messages as ReadonlyArray>, + ); + } + const modelParams = options.modelParams; + if (!extraBody && !modelParams) return body; + // Vendor passthrough, then the model's own parameters. Merged last so + // they can reach fields this builder does not model, then reserved + // keys are restored on top. + const merged: Record = { + ...body, + ...extraBody, + ...modelParams, + }; for (const key of RESERVED_BODY_KEYS) { if (key in body) merged[key] = body[key]; else delete merged[key]; diff --git a/src/llm/provider/openai/openai-native-messages.test.ts b/src/llm/provider/openai/openai-native-messages.test.ts new file mode 100644 index 00000000..f498f495 --- /dev/null +++ b/src/llm/provider/openai/openai-native-messages.test.ts @@ -0,0 +1,173 @@ +import { describe, expect, it } from "vitest"; + +import type { PromptMessages } from "../completion-types.js"; +import { OpenAiHttpError } from "./openai-http.js"; +import { + NO_RESULT_RECORDED, + buildNativeMessages, + isNativeShapeRejection, +} from "./openai-native-messages.js"; +import { nameEscape } from "./openai-tool-call-adapter.js"; + +const options = { nameEscape }; + +function prompt(overrides: Partial): PromptMessages { + return { + system: "### system\nprefix", + droppedSummary: null, + turns: [], + tail: "### world\n(none)\n\n### respond\nRespond now.\n", + ...overrides, + }; +} + +describe("buildNativeMessages", () => { + it("lays the packed turns out as system, real turns and one final user message", () => { + const messages = buildNativeMessages( + prompt({ + turns: [ + { kind: "user", text: "read a.txt" }, + { kind: "assistant_tool_call", tool: "os.fs.read", args: { path: "a.txt" } }, + { kind: "tool_result", tool: "os.fs.read", status: "ok", body: "hello", truncated: false }, + { kind: "assistant_reply", text: "it says hello" }, + { kind: "user", text: "now b.txt" }, + ], + }), + options, + ); + expect(messages).toEqual([ + { role: "system", content: "### system\nprefix" }, + { role: "user", content: "read a.txt" }, + { + role: "assistant", + content: null, + tool_calls: [ + { + id: "call_1", + type: "function", + function: { name: "os__fs__read", arguments: '{"path":"a.txt"}' }, + }, + ], + }, + { role: "tool", tool_call_id: "call_1", content: "hello" }, + { role: "assistant", content: "it says hello" }, + { role: "user", content: "now b.txt" }, + { role: "user", content: "### world\n(none)\n\n### respond\nRespond now.\n" }, + ]); + }); + + it("ids calls by their row index, so an append-only history keeps its ids", () => { + const turns: PromptMessages["turns"] = [ + { kind: "user", text: "go" }, + { kind: "assistant_tool_call", tool: "a", args: {} }, + { kind: "tool_result", tool: "a", status: "ok", body: "1", truncated: false }, + ]; + const before = buildNativeMessages(prompt({ turns }), options); + const after = buildNativeMessages( + prompt({ + turns: [ + ...turns, + { kind: "assistant_tool_call", tool: "b", args: {} }, + { kind: "tool_result", tool: "b", status: "ok", body: "2", truncated: false }, + ], + }), + options, + ); + // Everything but the final user message is a prefix of the next request. + expect(after.slice(0, before.length - 1)).toEqual(before.slice(0, -1)); + expect((after[4] as { tool_calls: Array<{ id: string }> }).tool_calls[0]?.id).toBe("call_3"); + }); + + it("carries the error status and the truncation note on a tool message", () => { + const messages = buildNativeMessages( + prompt({ + turns: [ + { kind: "assistant_tool_call", tool: "os.shell.run", args: { cmd: "x" } }, + { kind: "tool_result", tool: "os.shell.run", status: "error", body: "exit 1", truncated: true }, + ], + }), + options, + ); + expect(messages[2]).toEqual({ + role: "tool", + tool_call_id: "call_0", + content: "error: exit 1 (truncated)", + }); + }); + + it("answers a call the history has no result for before anything follows it", () => { + const messages = buildNativeMessages( + prompt({ + turns: [ + { kind: "assistant_tool_call", tool: "a", args: {} }, + { kind: "user", text: "steer" }, + { kind: "assistant_tool_call", tool: "b", args: {} }, + ], + }), + options, + ); + expect(messages.map((m) => m.role)).toEqual([ + "system", + "assistant", + "tool", + "user", + "assistant", + "tool", + "user", + ]); + expect(messages[2]).toEqual({ role: "tool", tool_call_id: "call_0", content: NO_RESULT_RECORDED }); + expect(messages[5]).toEqual({ role: "tool", tool_call_id: "call_2", content: NO_RESULT_RECORDED }); + }); + + it("carries a result whose call was cut away as the flat line, in a user message", () => { + const messages = buildNativeMessages( + prompt({ + droppedSummary: "summary: 3 older turns dropped", + turns: [ + { kind: "tool_result", tool: "os.fs.read", status: "ok", body: "tail of a read", truncated: true }, + { kind: "tool_result", tool: "os.fs.list", status: "error", body: "nope", truncated: false }, + { kind: "assistant_reply", text: "done" }, + ], + }), + options, + ); + expect(messages).toEqual([ + { role: "system", content: "### system\nprefix" }, + { role: "user", content: "summary: 3 older turns dropped" }, + { + role: "user", + content: + "tool_result[os.fs.read ok]: tail of a read (truncated)\ntool_result[os.fs.list error]: nope", + }, + { role: "assistant", content: "done" }, + { role: "user", content: "### world\n(none)\n\n### respond\nRespond now.\n" }, + ]); + }); + + it("sends system and the tail alone for an empty conversation", () => { + expect(buildNativeMessages(prompt({}), options)).toEqual([ + { role: "system", content: "### system\nprefix" }, + { role: "user", content: "### world\n(none)\n\n### respond\nRespond now.\n" }, + ]); + }); +}); + +describe("isNativeShapeRejection", () => { + const http = (status: number | null, message: string) => + new OpenAiHttpError(message, status, "http://x", false, null, "p"); + + it("names a 400 about roles, tool_call_id or messages", () => { + expect(isNativeShapeRejection(http(400, 'openai provider 400: {"error":"Unknown role: tool"}'))).toBe(true); + expect(isNativeShapeRejection(http(400, "openai provider 400: invalid tool_call_id"))).toBe(true); + expect(isNativeShapeRejection(http(400, "openai provider 400: messages[2] must be a user message"))).toBe(true); + expect(isNativeShapeRejection(http(400, "openai provider 400: tool_calls is not supported"))).toBe(true); + }); + + it("leaves every other failure to its own handling", () => { + expect(isNativeShapeRejection(http(400, "openai provider 400: maximum context length is 8192 tokens"))).toBe(false); + expect(isNativeShapeRejection(http(422, "openai provider 422: role"))).toBe(false); + expect(isNativeShapeRejection(http(null, "fetch failed: roles"))).toBe(false); + expect(isNativeShapeRejection(new Error("role"))).toBe(false); + expect(isNativeShapeRejection("role")).toBe(false); + }); +}); diff --git a/src/llm/provider/openai/openai-native-messages.ts b/src/llm/provider/openai/openai-native-messages.ts new file mode 100644 index 00000000..afe55829 --- /dev/null +++ b/src/llm/provider/openai/openai-native-messages.ts @@ -0,0 +1,155 @@ +import type { PromptMessages, PromptTurn } from "../completion-types.js"; + +/** + * The prompt as real chat messages for a native-tools provider. + * + * Every cloud request used to be one `user` message holding the stable + * prefix and the whole history as text — `assistant_tool_call: …` and + * `tool_result[…]: …` lines. A model that loses the thread of native + * function calling keeps writing that text instead of calling tools + * (Gemini Flash did, repeatedly: 84,931 characters in one completion, + * three worker streams in one run). Laid out as the messages the API + * defines — `system`, then assistant `tool_calls` answered by `tool` + * results, then one final `user` message — the history is what the + * model was trained to continue with a call, and it caches as a prefix + * that grows by one step per request. + * + * Shape rules, all of which exist because the API rejects the request + * otherwise: + * - a `tool` message must answer a `tool_calls` entry by id. Ids are + * `call_` with `n` the row's index in the packed turns, so the + * same history renders the same ids on every step between cuts; + * - a call the history has no result for gets a synthesised + * `(no result recorded)` answer before anything else follows it; + * - a result whose call the packer cut away has nothing to answer, so + * it is carried as the text line the flat form would show, in a + * `user` message; + * - the packer's dropped-turns recap opens the history as a `user` + * message, since it is what stood in for those turns. + */ + +export type OpenAiChatMessage = Record; + +export interface NativeMessageOptions { + /** The adapter's escape for tool names (`os.fs.read` → `os__fs__read`). */ + nameEscape: (qualifiedName: string) => string; +} + +export const NO_RESULT_RECORDED = "(no result recorded)"; + +export function buildNativeMessages( + prompt: PromptMessages, + options: NativeMessageOptions, +): OpenAiChatMessage[] { + const out: OpenAiChatMessage[] = [{ role: "system", content: prompt.system }]; + if (prompt.droppedSummary) { + out.push({ role: "user", content: prompt.droppedSummary }); + } + // Calls emitted and still waiting for their `tool` answer, in order. + let pending: string[] = []; + // Consecutive orphan result lines, folded into one user message. + let orphanLines: string[] = []; + + const flushOrphans = (): void => { + if (orphanLines.length === 0) return; + out.push({ role: "user", content: orphanLines.join("\n") }); + orphanLines = []; + }; + const answerPending = (): void => { + for (const id of pending) { + out.push({ role: "tool", tool_call_id: id, content: NO_RESULT_RECORDED }); + } + pending = []; + }; + + prompt.turns.forEach((turn, index) => { + switch (turn.kind) { + case "user": + answerPending(); + flushOrphans(); + out.push({ role: "user", content: turn.text }); + break; + case "assistant_reply": + answerPending(); + flushOrphans(); + out.push({ role: "assistant", content: turn.text }); + break; + case "assistant_tool_call": { + answerPending(); + flushOrphans(); + const id = `call_${index}`; + out.push({ + role: "assistant", + content: null, + tool_calls: [ + { + id, + type: "function", + function: { + name: options.nameEscape(turn.tool), + arguments: JSON.stringify(turn.args), + }, + }, + ], + }); + pending.push(id); + break; + } + case "tool_result": { + const id = pending.shift(); + if (id === undefined) { + orphanLines.push(renderOrphanResultLine(turn)); + break; + } + flushOrphans(); + out.push({ + role: "tool", + tool_call_id: id, + content: renderToolMessageContent(turn), + }); + break; + } + } + }); + answerPending(); + flushOrphans(); + out.push({ role: "user", content: prompt.tail }); + return out; +} + +/** + * What a `tool` message says: the capped body, prefixed with the status + * when the call failed (the flat line carries it in its header) and + * suffixed with the truncation note the flat line also carries. + */ +function renderToolMessageContent( + turn: Extract, +): string { + const body = turn.status === "error" ? `error: ${turn.body}` : turn.body; + return turn.truncated ? `${body} (truncated)` : body; +} + +/** The flat form's line, for a result whose call is out of view. */ +function renderOrphanResultLine( + turn: Extract, +): string { + return `tool_result[${turn.tool} ${turn.status}]: ${turn.body}${ + turn.truncated ? " (truncated)" : "" + }`; +} + +/** + * A 400 that rejects the message layout itself — roles the server does + * not accept, an unknown `tool_call_id`, a `messages` array it will not + * validate — as opposed to a 400 about anything else in the request. + * Older vLLM builds and llama.cpp shims answer this way; the provider + * then sends the flat form for the rest of the session. + */ +export function isNativeShapeRejection(err: unknown): boolean { + if (!(err instanceof Error)) return false; + const status = (err as { status?: unknown }).status; + if (status !== 400) return false; + return /\brole\b|\broles\b|tool_call_id|tool_calls|\bmessages\b/i.test( + err.message, + ); +} diff --git a/src/llm/provider/openai/openai-normalise-response.ts b/src/llm/provider/openai/openai-normalise-response.ts index fd6f6924..229c79df 100644 --- a/src/llm/provider/openai/openai-normalise-response.ts +++ b/src/llm/provider/openai/openai-normalise-response.ts @@ -1,22 +1,28 @@ -import type { CompletionResult } from "../completion-types.js"; +import type { CompletionResult, CompletionUsage } from "../completion-types.js"; +import type { ReasoningFormat } from "../llm-provider.js"; +import { createReasoningExtractor } from "./reasoning-extractor.js"; export function normaliseOpenAiChatResponse( json: Record, defaultChatModel: string, + reasoningFormat: ReasoningFormat = "auto", ): CompletionResult { const choice = (json.choices as Array> | undefined)?.[0] ?? {}; const message = (choice.message as Record | undefined) ?? {}; - const usage = (json.usage as Record | undefined) ?? {}; const toolCalls = message.tool_calls as CompletionResult["toolCalls"]; const content = normaliseMessageContent(message.content); - // Reasoning models served over OpenAI-compatible APIs (Qwen3.8 with - // preserve_thinking, DeepSeek-R1) return their CoT in a dedicated - // `reasoning_content` field alongside `content`. - const reasoningContent = - typeof message.reasoning_content === "string" - ? message.reasoning_content - : ""; + // Reasoning models served over OpenAI-compatible APIs return their CoT + // in a dedicated field alongside `content` — `reasoning_content` + // (Qwen with preserve_thinking, DeepSeek-R1), `reasoning` (OpenRouter) + // or `thinking`. Same extractor as the stream consumer, so the unary + // and streamed paths cannot disagree about where reasoning lives. + const reasoningContent = createReasoningExtractor( + reasoningFormat, + ).extractFromMessage(message); + const usage = normaliseOpenAiUsage( + json.usage as Record | undefined, + ); return { content, reasoningContent, @@ -25,23 +31,50 @@ export function normaliseOpenAiChatResponse( timing: { promptMs: 0, predictedMs: 0, - promptTokens: Number(usage.prompt_tokens ?? 0), - predictedTokens: Number(usage.completion_tokens ?? 0), + promptTokens: usage.promptTokens, + predictedTokens: usage.completionTokens, }, - cacheHitTokens: 0, + cacheHitTokens: usage.cachedTokens ?? 0, slotId: -1, modelId: typeof json.model === "string" ? json.model : defaultChatModel, - usage: { - promptTokens: Number(usage.prompt_tokens ?? 0), - completionTokens: Number(usage.completion_tokens ?? 0), - totalTokens: Number(usage.total_tokens ?? 0), - }, + usage, toolCalls, finishReason: typeof choice.finish_reason === "string" ? choice.finish_reason : null, }; } +/** + * One reading of an OpenAI-shaped `usage` block for both paths. + * + * `prompt_tokens_details.cached_tokens` is where OpenAI, OpenRouter and + * Gemini's compatibility layer report the prompt tokens served from a + * prompt cache (Anthropic through OpenRouter lands there too; a shim + * that speaks Anthropic's own `cache_read_input_tokens` is read as a + * fallback). It is the only evidence a turn has that its caching + * arrangement is working, so it rides on `usage` as `cachedTokens` — + * absent, not zero, when the service did not say — and on + * `cacheHitTokens`, which the trace already records per completion. + */ +export function normaliseOpenAiUsage( + raw: Record | undefined, +): CompletionUsage { + const usage = raw ?? {}; + const details = usage.prompt_tokens_details; + const cached = + details !== null && typeof details === "object" + ? (details as Record).cached_tokens + : usage.cache_read_input_tokens; + return { + promptTokens: Number(usage.prompt_tokens ?? 0), + completionTokens: Number(usage.completion_tokens ?? 0), + totalTokens: Number(usage.total_tokens ?? 0), + ...(typeof cached === "number" && Number.isFinite(cached) && cached >= 0 + ? { cachedTokens: cached } + : {}), + }; +} + /** * `message.content` is a plain string on most servers, but multimodal * responses may carry an array of content parts. Join the text parts so diff --git a/src/llm/provider/openai/openai-provider.test.ts b/src/llm/provider/openai/openai-provider.test.ts index b4b6afda..259a97c8 100644 --- a/src/llm/provider/openai/openai-provider.test.ts +++ b/src/llm/provider/openai/openai-provider.test.ts @@ -97,20 +97,68 @@ describe("OpenAiProvider qwen tagged-tool compatibility", () => { ]); }); - it("leaves the existing OpenAI provider path unchanged by default", async () => { - const tagged = - "/tmp/a"; - const result = await provider( + it("decodes a reply that is only tagged blocks on the default kind too", async () => { + // Hermes fine-tunes and Qwen-derived models write `` text + // over any OpenAI-compatible server, not only the Qwen kind; a reply + // that is nothing but such blocks is a tool call wherever it came + // from. Both dialects: Qwen's XML-ish form and the Hermes JSON form. + for (const tagged of [ + "/tmp/a", + '{"name": "os.fs.read", "arguments": {"path": "/tmp/a"}}', + ]) { + const result = await provider( + fakeFetch({ role: "assistant", content: tagged }) as unknown as typeof fetch, + undefined, + ).complete({ prompt: "read", tools }); + expect(result.content).toBe(""); + expect(result.finishReason).toBe("tool_calls"); + expect(result.toolCalls).toMatchObject([ + { + type: "function", + function: { name: "os__fs__read", arguments: '{"path":"/tmp/a"}' }, + }, + ]); + } + }); + + it("keeps a reply that merely quotes the syntax, and never reads reasoning on the default kind", async () => { + const quoted = + "Use this shape:\n/tmp/a"; + const prose = await provider( + fakeFetch({ role: "assistant", content: quoted }) as unknown as typeof fetch, + undefined, + ).complete({ prompt: "read", tools }); + expect(prose.content).toBe(quoted); + expect(prose.toolCalls).toBeUndefined(); + expect(prose.finishReason).toBe("stop"); + + // #105's reasoning-channel fallback is Qwen's: elsewhere the channel + // is scratch space, and a call thought about is not a call made. + const thought = await provider( fakeFetch({ role: "assistant", - content: tagged, + content: "", + reasoning_content: + "/tmp/a", }) as unknown as typeof fetch, undefined, ).complete({ prompt: "read", tools }); + expect(thought.toolCalls).toBeUndefined(); + expect(thought.reasoningContent).toContain(""); + }); - expect(result.content).toBe(tagged); - expect(result.toolCalls).toBeUndefined(); - expect(result.finishReason).toBe("stop"); + it("reads reasoning from whichever field the service writes, by default", async () => { + for (const field of ["reasoning", "reasoning_content", "thinking"]) { + const result = await provider( + fakeFetch({ + role: "assistant", + content: "ok", + [field]: "because", + }) as unknown as typeof fetch, + undefined, + ).complete({ prompt: "hi" }); + expect(result.reasoningContent).toBe("because"); + } }); it("streams deltas, then adapts the buffered tagged calls (buffer-then-adapt)", async () => { @@ -262,3 +310,224 @@ describe("OpenAiProvider strictTools wiring", () => { }); }); }); + +describe("OpenAiProvider — cached prompt tokens", () => { + it("reads prompt_tokens_details.cached_tokens on the unary path", async () => { + const fetchImpl = vi.fn( + async () => + new Response( + JSON.stringify({ + model: "m", + choices: [{ message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + usage: { + prompt_tokens: 1000, + completion_tokens: 5, + total_tokens: 1005, + prompt_tokens_details: { cached_tokens: 900 }, + }, + }), + { status: 200, headers: { "content-type": "application/json" } }, + ), + ); + const result = await provider(fetchImpl as unknown as typeof fetch, undefined).complete({ + prompt: "hi", + }); + expect(result.usage?.cachedTokens).toBe(900); + expect(result.cacheHitTokens).toBe(900); + }); + + it("reads it on the streamed path, and leaves it absent when the service says nothing", async () => { + const frame = (obj: Record) => `data: ${JSON.stringify(obj)}\n\n`; + const streamWith = (usage: Record) => + vi.fn( + async () => + new Response( + frame({ choices: [{ index: 0, delta: { content: "ok" }, finish_reason: null }] }) + + frame({ choices: [{ index: 0, delta: {}, finish_reason: "stop" }], usage }) + + "data: [DONE]\n\n", + { status: 200, headers: { "content-type": "text/event-stream" } }, + ), + ); + const drain = async (fetchImpl: typeof fetch) => { + const stream = provider(fetchImpl, undefined).completeStream({ prompt: "hi" }); + for (;;) { + const next = await stream.next(); + if (next.done) return next.value; + } + }; + const cached = await drain( + streamWith({ + prompt_tokens: 1000, + completion_tokens: 5, + total_tokens: 1005, + prompt_tokens_details: { cached_tokens: 640 }, + }) as unknown as typeof fetch, + ); + expect(cached.usage?.cachedTokens).toBe(640); + expect(cached.cacheHitTokens).toBe(640); + + const silent = await drain( + streamWith({ prompt_tokens: 1000, completion_tokens: 5, total_tokens: 1005 }) as unknown as typeof fetch, + ); + expect(silent.usage).not.toHaveProperty("cachedTokens"); + expect(silent.cacheHitTokens).toBe(0); + }); +}); + +describe("OpenAiProvider — message shape on the wire", () => { + const messages = { + system: "prefix", + droppedSummary: null, + turns: [ + { kind: "user" as const, text: "hi" }, + { kind: "assistant_tool_call" as const, tool: "os.fs.read", args: { path: "a" } }, + { kind: "tool_result" as const, tool: "os.fs.read", status: "ok" as const, body: "A", truncated: false }, + ], + tail: "tail", + }; + const okReply = () => + new Response( + JSON.stringify({ + model: "m", + choices: [{ message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + }), + { status: 200, headers: { "content-type": "application/json" } }, + ); + const okStream = () => + new Response( + `data: ${JSON.stringify({ choices: [{ index: 0, delta: { content: "ok" }, finish_reason: null }] })}\n\n` + + `data: ${JSON.stringify({ choices: [{ index: 0, delta: {}, finish_reason: "stop" }] })}\n\ndata: [DONE]\n\n`, + { status: 200, headers: { "content-type": "text/event-stream" } }, + ); + const roleRejection = () => + new Response(JSON.stringify({ error: { message: "Unknown role: tool", type: "invalid_request_error" } }), { + status: 400, + headers: { "content-type": "application/json" }, + }); + const otherRejection = () => + new Response(JSON.stringify({ error: { message: "maximum context length is 8192 tokens, requested 9000" } }), { + status: 400, + headers: { "content-type": "application/json" }, + }); + + function capturing(replies: Array<() => Response>) { + const bodies: Record[] = []; + let call = 0; + const fetchImpl = vi.fn(async (_url: string, init?: RequestInit) => { + bodies.push(JSON.parse(String(init?.body)) as Record); + const reply = replies[Math.min(call, replies.length - 1)]!; + call += 1; + return reply(); + }); + return { bodies, fetchImpl: fetchImpl as unknown as typeof fetch }; + } + const roles = (body: Record | undefined) => + (body?.messages as Array<{ role: string }>).map((m) => m.role); + const logger = () => ({ warn: vi.fn() }); + + function shaped(fetchImpl: typeof fetch, extra: Partial[0]> = {}) { + return new OpenAiProvider({ + id: "test", + baseUrl: "https://example.invalid", + apiKey: "", + defaultChatModel: "m", + fetchImpl, + ...extra, + }); + } + const drain = async (stream: AsyncGenerator) => { + for (;;) if ((await stream.next()).done) return; + }; + + it("sends the native layout by default, unary and streamed", async () => { + const unary = capturing([okReply]); + await shaped(unary.fetchImpl).complete({ prompt: "flat", messages, tools, sessionId: "s" }); + expect(roles(unary.bodies[0])).toEqual(["system", "user", "assistant", "tool", "user"]); + + const streamed = capturing([okStream]); + await drain(shaped(streamed.fetchImpl).completeStream({ prompt: "flat", messages, tools, sessionId: "s" })); + expect(roles(streamed.bodies[0])).toEqual(["system", "user", "assistant", "tool", "user"]); + }); + + it("sends the flat layout when the entry says so", async () => { + const { bodies, fetchImpl } = capturing([okReply]); + await shaped(fetchImpl, { messageShape: "flat" }).complete({ prompt: "flat", messages, tools }); + expect(bodies[0]?.messages).toEqual([{ role: "user", content: "flat" }]); + }); + + it("falls back to flat once on a 400 about roles, logs it, and keeps the session flat", async () => { + const { bodies, fetchImpl } = capturing([roleRejection, okReply, okReply, okReply]); + const log = logger(); + const provider = shaped(fetchImpl, { logger: log }); + const result = await provider.complete({ prompt: "flat", messages, tools, sessionId: "s1" }); + expect(result.content).toBe("ok"); + expect(roles(bodies[0])).toEqual(["system", "user", "assistant", "tool", "user"]); + expect(bodies[1]?.messages).toEqual([{ role: "user", content: "flat" }]); + expect(log.warn).toHaveBeenCalledTimes(1); + expect(log.warn.mock.calls[0]?.[0]).toMatch(/rejected native chat messages/); + + // The same session goes out flat up front; another session is still native. + await provider.complete({ prompt: "flat", messages, tools, sessionId: "s1" }); + expect(bodies[2]?.messages).toEqual([{ role: "user", content: "flat" }]); + await provider.complete({ prompt: "flat", messages, tools, sessionId: "s2" }); + expect(roles(bodies[3])).toEqual(["system", "user", "assistant", "tool", "user"]); + expect(bodies).toHaveLength(4); + }); + + it("falls back on the streamed path too, before any chunk exists", async () => { + const { bodies, fetchImpl } = capturing([roleRejection, okStream]); + const log = logger(); + const provider = shaped(fetchImpl, { logger: log }); + const stream = provider.completeStream({ prompt: "flat", messages, tools, sessionId: "s" }); + const deltas: string[] = []; + for (;;) { + const next = await stream.next(); + if (next.done) { + expect(next.value.content).toBe("ok"); + break; + } + deltas.push(next.value.delta); + } + expect(deltas.join("")).toBe("ok"); + expect(bodies).toHaveLength(2); + expect(bodies[1]?.messages).toEqual([{ role: "user", content: "flat" }]); + expect(log.warn).toHaveBeenCalledTimes(1); + }); + + it("does not resend on a 400 about anything else, nor when already flat", async () => { + const other = capturing([otherRejection]); + await expect( + shaped(other.fetchImpl).complete({ prompt: "flat", messages, tools, sessionId: "s" }), + ).rejects.toThrow(/maximum context length/); + expect(other.bodies).toHaveLength(1); + + const flat = capturing([roleRejection]); + await expect( + shaped(flat.fetchImpl, { messageShape: "flat" }).complete({ prompt: "flat", messages, tools }), + ).rejects.toThrow(/Unknown role/); + expect(flat.bodies).toHaveLength(1); + }); + + it("marks Anthropic breakpoints for a Claude model, and none for another", async () => { + const claude = capturing([okReply]); + await shaped(claude.fetchImpl, { defaultChatModel: "anthropic/claude-sonnet-4.5" }).complete({ + prompt: "flat", + messages, + tools, + }); + const sent = claude.bodies[0]?.messages as Array>; + expect(sent[0]?.content).toEqual([{ type: "text", text: "prefix", cache_control: { type: "ephemeral" } }]); + + const off = capturing([okReply]); + await shaped(off.fetchImpl, { defaultChatModel: "anthropic/claude-sonnet-4.5", promptCache: "off" }).complete({ + prompt: "flat", + messages, + tools, + }); + expect(JSON.stringify(off.bodies[0])).not.toContain("ephemeral"); + + const gpt = capturing([okReply]); + await shaped(gpt.fetchImpl, { defaultChatModel: "openai/gpt-4.1" }).complete({ prompt: "flat", messages, tools }); + expect(JSON.stringify(gpt.bodies[0])).not.toContain("ephemeral"); + }); +}); diff --git a/src/llm/provider/openai/openai-provider.ts b/src/llm/provider/openai/openai-provider.ts index d12434ce..08883545 100644 --- a/src/llm/provider/openai/openai-provider.ts +++ b/src/llm/provider/openai/openai-provider.ts @@ -20,7 +20,12 @@ import { withStrictNullArgumentDrop, } from "./openai-tool-call-adapter.js"; import { createOpenAiStreamConsumer } from "./openai-stream-consumer.js"; -import { buildOpenAiChatBody } from "./openai-build-body.js"; +import { + buildOpenAiChatBody, + resolveMessageShape, + type OpenAiBodyOptions, +} from "./openai-build-body.js"; +import { isNativeShapeRejection } from "./openai-native-messages.js"; import { buildOpenAiHeaders, createOpenAiAttemptBudget, @@ -37,9 +42,11 @@ import { isNetworkError } from "../../reliability/network-error.js"; import { normaliseOpenAiChatResponse } from "./openai-normalise-response.js"; import { normalizeOpenAiBaseUrl } from "./normalize-openai-base-url.js"; import { describeImageViaOpenAi } from "./openai-describe-image.js"; +import { isAnthropicHost, isAnthropicModel } from "./prompt-cache-control.js"; import { adaptQwenCompletionResult, adaptQwenTaggedToolResponse, + type TaggedToolAdaptOptions, } from "./qwen-tagged-tool-response-adapter.js"; import type { CreditLimitLogger } from "./plan-credit-limit-retry.js"; import { sendWithStructuredOutputFallback } from "./structured-output-fallback.js"; @@ -64,7 +71,39 @@ export interface OpenAiProviderOptions { toolCallAdapter?: ToolCallAdapter; streamConsumer?: StreamConsumer; apiPathPrefix?: string; + /** + * Tagged text tool calls (``, Qwen's XML-ish + * form or the Hermes JSON form) are decoded for every kind when the + * whole reply is such blocks. `"qwen"` additionally reads a call out of + * `reasoning_content` when `content` holds none (#105). + */ taggedToolCompatibility?: "qwen"; + /** + * The registered kind this client serves (`openrouter`, + * `openai-compatible`, …), for body fields whose spelling is the + * vendor's. See `OpenAiBodyOptions.providerKind`. + */ + providerKind?: string; + /** + * Wire parameters of the default chat model from `userModels[].params`, + * merged over every chat body after `extraBody`. + */ + modelParams?: Record; + /** + * How the structured prompt is laid out on the wire: `native` (the + * default) as `system` + history messages + final `user`; `flat` as the + * one `user` message of text every request used to be. A 400 about + * roles / `tool_call_id` / `messages` flips a session to `flat` and + * retries once. See `openai-native-messages.ts`. + */ + messageShape?: "native" | "flat"; + /** + * The entry's prompt-caching policy. `off` sends no cache markers; + * `explicit-markers` always sends Anthropic breakpoints; `auto` (and + * absent) sends them when the model or the host is Anthropic's. See + * `prompt-cache-control.ts`. + */ + promptCache?: "auto" | "off" | "explicit-markers"; /** * Vendor-specific fields merged into every chat completion body. * See `RESERVED_BODY_KEYS` in `openai-build-body.ts` for the keys @@ -111,6 +150,17 @@ export class OpenAiProvider implements LlmProvider { private readonly maxOutputTokens: number | undefined; private readonly strictTools: boolean; private readonly providerPreferences: Record | undefined; + private readonly bodyOptions: OpenAiBodyOptions; + private readonly reasoningFormat: ReasoningFormat; + private readonly messageShape: "native" | "flat"; + /** + * Sessions whose requests go out flat after the service rejected the + * native layout (`"*"` for requests without a session id). Per + * session rather than per provider so one shim-backed session cannot + * downgrade another's caching; per instance because a provider is + * rebuilt on every config write, which is the cheapest re-probe. + */ + private readonly flatSessions = new Set(); constructor(options: OpenAiProviderOptions) { this.id = options.id; @@ -124,9 +174,12 @@ export class OpenAiProvider implements LlmProvider { this.toolCallAdapter = options.strictTools ? withStrictNullArgumentDrop(baseToolCallAdapter) : baseToolCallAdapter; + // `auto` reads whichever reasoning field the service writes; a + // configured format (`userModels[].reasoningFormat`) pins one. + this.reasoningFormat = options.reasoningFormat ?? "auto"; this.streamConsumer = options.streamConsumer ?? - createOpenAiStreamConsumer(options.reasoningFormat ?? "delta_reasoning"); + createOpenAiStreamConsumer(this.reasoningFormat); this.capabilities = { vision: options.supportsVision ?? true, visionSource: options.supportsVision ? "modalities.vision" : "absent", @@ -135,7 +188,7 @@ export class OpenAiProvider implements LlmProvider { supportsParallelTools: options.supportsParallelTools ?? true, supportsSlotAffinity: false, supportsPromptCache: options.supportsPromptCache ?? true, - reasoningFormat: options.reasoningFormat ?? "delta_reasoning", + reasoningFormat: this.reasoningFormat, }; this.defaultChatModel = options.defaultChatModel; this.apiPathPrefix = normalizeApiPathPrefix(options.apiPathPrefix ?? "/v1"); @@ -144,6 +197,15 @@ export class OpenAiProvider implements LlmProvider { this.maxOutputTokens = options.maxOutputTokens; this.strictTools = options.strictTools ?? false; this.providerPreferences = options.providerPreferences; + this.messageShape = options.messageShape ?? "native"; + this.bodyOptions = { + nameEscape: (name) => this.toolCallAdapter.nameEscape(name), + ...(options.providerKind ? { providerKind: options.providerKind } : {}), + ...(options.modelParams ? { modelParams: options.modelParams } : {}), + ...(resolveAnthropicCacheControl(options) + ? { anthropicCacheControl: true } + : {}), + }; this.http = { baseUrl: normalizeOpenAiBaseUrl(options.baseUrl), apiKey: options.apiKey, @@ -159,57 +221,120 @@ export class OpenAiProvider implements LlmProvider { async complete(request: CompletionRequest): Promise { // The body the response actually came from; see `OnOpenAiRequestBody`. let sentBody: Record | undefined; - // Unary only: sub-calls carry `response_format`, streamed turns never do. - const json = await sendWithStructuredOutputFallback( - { - providerId: this.id, - model: this.defaultChatModel, - logger: this.http.logger, - }, + const send = (shape: "native" | "flat") => + // Unary only: sub-calls carry `response_format`, streamed turns never do. + sendWithStructuredOutputFallback( + { + providerId: this.id, + model: this.defaultChatModel, + logger: this.http.logger, + }, + request, + (req) => this.buildBody(req, false, shape), + (body) => + openAiPostJson( + this.http, + `${this.apiPathPrefix}/chat/completions`, + body, + request, + (sent) => { + sentBody = sent; + }, + ), + ); + const shape = this.shapeFor(request); + let json: Record; + try { + json = await send(shape); + } catch (err) { + if (!this.shouldFallBackToFlat(request, shape, err)) throw err; + this.markFlat(request, err); + json = await send("flat"); + } + const adapted = adaptQwenTaggedToolResponse( + json, request, - (req) => - buildOpenAiChatBody( - req, - this.defaultChatModel, - false, - this.extraBody, - this.maxOutputTokens, - this.strictTools, - this.providerPreferences, - ), - (body) => - openAiPostJson( - this.http, - `${this.apiPathPrefix}/chat/completions`, - body, - request, - (sent) => { - sentBody = sent; - }, - ), + this.taggedToolOptions(), ); - const adapted = - this.taggedToolCompatibility === "qwen" - ? adaptQwenTaggedToolResponse(json, request) - : json; return withSentMaxTokens( - normaliseOpenAiChatResponse(adapted, this.defaultChatModel), + normaliseOpenAiChatResponse( + adapted, + this.defaultChatModel, + this.reasoningFormat, + ), sentBody, ); } - async *completeStream( + /** + * A reply that is nothing but `` blocks is a tool call on + * every kind (Hermes fine-tunes and Qwen-derived models write one over + * any OpenAI-compatible server); only the Qwen kind also looks inside + * the reasoning channel. See `TaggedToolAdaptOptions`. + */ + private taggedToolOptions(): TaggedToolAdaptOptions { + return { fromReasoning: this.taggedToolCompatibility === "qwen" }; + } + + private buildBody( request: CompletionRequest, - ): AsyncGenerator { - const body = buildOpenAiChatBody( + stream: boolean, + shape: "native" | "flat", + ): Record { + return buildOpenAiChatBody( request, this.defaultChatModel, - true, + stream, this.extraBody, this.maxOutputTokens, this.strictTools, this.providerPreferences, + { ...this.bodyOptions, messageShape: shape }, + ); + } + + /** The layout this request goes out in, given what the session learned. */ + private shapeFor(request: CompletionRequest): "native" | "flat" { + if (this.messageShape === "flat") return "flat"; + return this.flatSessions.has(sessionKey(request)) ? "flat" : "native"; + } + + /** + * Whether a failed send is the service refusing the native layout — + * which only a request that actually went out native can be — rather + * than anything else, and whether a flat resend is still wanted. + */ + private shouldFallBackToFlat( + request: CompletionRequest, + shape: "native" | "flat", + err: unknown, + ): boolean { + if (request.signal?.aborted) return false; + if (resolveMessageShape(request, { messageShape: shape }) !== "native") { + return false; + } + return isNativeShapeRejection(err); + } + + private markFlat(request: CompletionRequest, err: unknown): void { + this.flatSessions.add(sessionKey(request)); + this.http.logger?.warn( + `llm: "${this.id}" rejected native chat messages (roles / tool_call_id); sending the flat prompt for the rest of this session`, + { + provider: this.id, + model: this.defaultChatModel, + sessionId: request.sessionId ?? null, + status: 400, + detail: err instanceof Error ? err.message.slice(0, 240) : String(err), + }, ); + } + + async *completeStream( + request: CompletionRequest, + ): AsyncGenerator { + let shape = this.shapeFor(request); + let body = this.buildBody(request, true, shape); const path = `${this.apiPathPrefix}/chat/completions`; let accumulated = ""; let accumulatedReasoning = ""; @@ -310,6 +435,17 @@ export class OpenAiProvider implements LlmProvider { // stopped. `signal.reason` is abort-shaped by construction. if (request.signal?.aborted) throw cancellationError(request.signal, err); + // The service refused the message layout, before any byte of + // output existed (a 400 comes from the open): the same request + // goes out once more in the flat form, and the session stays + // flat. Only ever one such resend — the second attempt is flat + // by construction and cannot match again. + if (!committed && this.shouldFallBackToFlat(request, shape, err)) { + this.markFlat(request, err); + shape = "flat"; + body = this.buildBody(request, true, shape); + continue; + } if (!canReopenStream(err, committed, budget)) throw err; // No `res.body.cancel()` here, on purpose. The only way to reach // this line with a response in hand is `isNetworkError(err)` on @@ -365,10 +501,11 @@ export class OpenAiProvider implements LlmProvider { // so native and tagged calls are judged from the same final dispatchable // tool-call set. A synthetic `finishReason: "tool_calls"` from the // adapter is not evidence that the provider actually terminated cleanly. - const adaptedFinal = - this.taggedToolCompatibility === "qwen" - ? adaptQwenCompletionResult(final, request) - : final; + const adaptedFinal = adaptQwenCompletionResult( + final, + request, + this.taggedToolOptions(), + ); return withSentMaxTokens( applyToolCallTerminationSafety( adaptedFinal, @@ -435,6 +572,31 @@ export class OpenAiProvider implements LlmProvider { } } +/** + * Whether this client places Anthropic cache breakpoints. The policy + * word decides when it is explicit; otherwise the model id or the host + * has to be Anthropic's, because every other service ignores the marker + * at best and rejects the request at worst. + */ +function resolveAnthropicCacheControl(options: OpenAiProviderOptions): boolean { + switch (options.promptCache) { + case "off": + return false; + case "explicit-markers": + return true; + default: + return ( + isAnthropicModel(options.defaultChatModel) || + isAnthropicHost(options.baseUrl) + ); + } +} + +/** The key a session's learned layout is remembered under. */ +function sessionKey(request: Pick): string { + return request.sessionId ?? "*"; +} + function normalizeApiPathPrefix(prefix: string): string { const trimmed = prefix.trim().replace(/\/+$/, ""); return trimmed.startsWith("/") ? trimmed : `/${trimmed}`; @@ -463,7 +625,7 @@ function completionFromStreamFinal( promptTokens: usage.promptTokens, predictedTokens: usage.completionTokens, }, - cacheHitTokens: 0, + cacheHitTokens: usage.cachedTokens ?? 0, slotId: -1, modelId: streamFinal?.modelId ?? defaultChatModel, usage, diff --git a/src/llm/provider/openai/openai-stream-consumer.test.ts b/src/llm/provider/openai/openai-stream-consumer.test.ts index d27d5cb1..6d53f074 100644 --- a/src/llm/provider/openai/openai-stream-consumer.test.ts +++ b/src/llm/provider/openai/openai-stream-consumer.test.ts @@ -239,3 +239,75 @@ describe("openai stream consumer tool-call assembly", () => { ]); }); }); + +describe("openai stream consumer: text beside a tool-call delta", () => { + it("keeps the text of a chunk that also carries a tool-call delta", async () => { + // Gemini's compatibility layer and Anthropic shims put the model's + // prose and its call in one event; the prose used to be dropped. + const consumer = createOpenAiStreamConsumer("auto"); + const iterator = consumer.consume( + bodyOf( + sseFrame({ + model: "test-model", + choices: [ + { + index: 0, + delta: { + content: "Reading the file. ", + tool_calls: [ + { + index: 0, + id: "call_1", + type: "function", + function: { name: "os__fs__read", arguments: '{"path":"a"}' }, + }, + ], + }, + finish_reason: null, + }, + ], + }) + DONE, + ), + undefined, + ); + const deltas: string[] = []; + let final: StreamFinalResult | undefined; + for (;;) { + const step = await iterator.next(); + if (step.done) { + final = step.value as StreamFinalResult; + break; + } + deltas.push(step.value.delta); + } + expect(deltas.join("")).toBe("Reading the file. "); + expect(final?.content).toBe("Reading the file. "); + expect(final?.toolCalls).toMatchObject([ + { function: { name: "os__fs__read", arguments: '{"path":"a"}' } }, + ]); + }); + + it("reads reasoning from any of the three fields under `auto`", async () => { + for (const field of ["reasoning", "reasoning_content", "thinking"]) { + const consumer = createOpenAiStreamConsumer("auto"); + const iterator = consumer.consume( + bodyOf( + sseFrame({ + model: "test-model", + choices: [{ index: 0, delta: { [field]: "hmm" }, finish_reason: null }], + }) + DONE, + ), + undefined, + ); + let final: StreamFinalResult | undefined; + for (;;) { + const step = await iterator.next(); + if (step.done) { + final = step.value as StreamFinalResult; + break; + } + } + expect(final?.reasoningContent).toBe("hmm"); + } + }); +}); diff --git a/src/llm/provider/openai/openai-stream-consumer.ts b/src/llm/provider/openai/openai-stream-consumer.ts index 4445a2c5..4eae156a 100644 --- a/src/llm/provider/openai/openai-stream-consumer.ts +++ b/src/llm/provider/openai/openai-stream-consumer.ts @@ -7,6 +7,7 @@ import type { } from "../completion-types.js"; import type { ReasoningFormat } from "../llm-provider.js"; import { createFabricatedTranscriptWatcher } from "../../reliability/fabricated-tool-transcript.js"; +import { normaliseOpenAiUsage } from "./openai-normalise-response.js"; import { createReasoningExtractor } from "./reasoning-extractor.js"; import { parseOpenAiSseEvent, @@ -127,15 +128,21 @@ export function createOpenAiStreamConsumer( toolArgsBuffer = chunk.toolArgsBuffer; const replyText = extractPartialReplyTextFromToolArguments(toolArgsBuffer); - if (replyText.length > 0) { + // Text the event carried beside its tool-call delta is + // shown as it is; reply text still streaming inside the + // call's arguments is shown as it becomes readable. + const textDelta = + chunk.delta + + (replyText.length > 0 + ? replyText.slice(chunk.emittedReplyLength) + : ""); + if ( + replyText.length > 0 || + textDelta.length > 0 || + chunk.reasoningDelta.length > 0 + ) { yield { - delta: replyText.slice(chunk.emittedReplyLength), - reasoningDelta: chunk.reasoningDelta, - done: false, - }; - } else if (chunk.reasoningDelta.length > 0) { - yield { - delta: "", + delta: textDelta, reasoningDelta: chunk.reasoningDelta, done: false, }; @@ -356,9 +363,5 @@ function normaliseUsage( raw: Record | null, ): CompletionUsage | undefined { if (!raw) return undefined; - return { - promptTokens: Number(raw.prompt_tokens ?? 0), - completionTokens: Number(raw.completion_tokens ?? 0), - totalTokens: Number(raw.total_tokens ?? 0), - }; + return normaliseOpenAiUsage(raw); } diff --git a/src/llm/provider/openai/parse-sse-chunk.ts b/src/llm/provider/openai/parse-sse-chunk.ts index fd1f9735..6082dc21 100644 --- a/src/llm/provider/openai/parse-sse-chunk.ts +++ b/src/llm/provider/openai/parse-sse-chunk.ts @@ -95,8 +95,12 @@ export function parseOpenAiSseEvent( if (toolCalls && toolCalls.length > 0) { const frag = toolCalls[0]?.function?.arguments ?? ""; const nextBuffer = toolArgsBuffer + frag; + // A chunk that carries both text and a tool-call delta keeps its + // text: some services (Gemini's compatibility layer, Anthropic + // shims) put the model's prose and its call in one event, and + // dropping the prose here lost the reply that went with the call. return { - delta: "", + delta: content, reasoningDelta, toolArgsBuffer: nextBuffer, toolArgsDelta: true, diff --git a/src/llm/provider/openai/prompt-cache-control.test.ts b/src/llm/provider/openai/prompt-cache-control.test.ts new file mode 100644 index 00000000..0c859095 --- /dev/null +++ b/src/llm/provider/openai/prompt-cache-control.test.ts @@ -0,0 +1,131 @@ +import { describe, expect, it } from "vitest"; + +import { + GOOGLE_CACHE_ROUTE_PREFERENCES, + applyAnthropicCacheControl, + isAnthropicHost, + isAnthropicModel, + isGoogleModel, +} from "./prompt-cache-control.js"; + +const EPHEMERAL = { type: "ephemeral" }; + +describe("model and host detection", () => { + it("names Anthropic models through a router and directly", () => { + expect(isAnthropicModel("anthropic/claude-sonnet-4.5")).toBe(true); + expect(isAnthropicModel("claude-opus-4-1")).toBe(true); + expect(isAnthropicModel("Anthropic/Claude-3.7")).toBe(true); + expect(isAnthropicModel("openai/gpt-5")).toBe(false); + expect(isAnthropicModel("google/gemini-3.8-flash")).toBe(false); + }); + + it("names Anthropic's own host and nothing else", () => { + expect(isAnthropicHost("https://api.anthropic.com/v1")).toBe(true); + expect(isAnthropicHost("https://api.anthropic.com")).toBe(true); + expect(isAnthropicHost("https://openrouter.ai/api")).toBe(false); + expect(isAnthropicHost("not a url")).toBe(false); + expect(isAnthropicHost(undefined)).toBe(false); + }); + + it("names Google models on OpenRouter", () => { + expect(isGoogleModel("google/gemini-3.8-flash")).toBe(true); + expect(isGoogleModel("Google/gemma-4-31b")).toBe(true); + expect(isGoogleModel("gemini-3.8-flash")).toBe(false); + expect(GOOGLE_CACHE_ROUTE_PREFERENCES).toEqual({ + order: ["Google AI Studio", "Google"], + allow_fallbacks: true, + }); + }); +}); + +describe("applyAnthropicCacheControl", () => { + it("marks the system message and the last history message before the tail", () => { + const messages = [ + { role: "system", content: "prefix" }, + { role: "user", content: "build it" }, + { + role: "assistant", + content: null, + tool_calls: [{ id: "call_1", type: "function", function: { name: "f", arguments: "{}" } }], + }, + { role: "tool", tool_call_id: "call_1", content: "ok" }, + { role: "user", content: "### respond\nRespond now." }, + ]; + const marked = applyAnthropicCacheControl(messages); + expect(marked[0]).toEqual({ + role: "system", + content: [{ type: "text", text: "prefix", cache_control: EPHEMERAL }], + }); + expect(marked[3]).toEqual({ + role: "tool", + tool_call_id: "call_1", + content: [{ type: "text", text: "ok", cache_control: EPHEMERAL }], + }); + // Never inside the changing tail, never on the user turn in between. + expect(marked[1]).toEqual(messages[1]); + expect(marked[2]).toEqual(messages[2]); + expect(marked[4]).toEqual(messages[4]); + // Two breakpoints, well under Anthropic's four. + expect(JSON.stringify(marked).split("ephemeral")).toHaveLength(3); + }); + + it("skips back over a tool-calls-only assistant message to the nearest text", () => { + const marked = applyAnthropicCacheControl([ + { role: "system", content: "prefix" }, + { role: "user", content: "hi" }, + { role: "assistant", content: null, tool_calls: [] }, + { role: "user", content: "tail" }, + ]); + expect(marked[1]).toEqual({ + role: "user", + content: [{ type: "text", text: "hi", cache_control: EPHEMERAL }], + }); + expect(marked[2]).toEqual({ role: "assistant", content: null, tool_calls: [] }); + }); + + it("marks only the system message when there is no history", () => { + const marked = applyAnthropicCacheControl([ + { role: "system", content: "prefix" }, + { role: "user", content: "tail" }, + ]); + expect(marked[0]?.content).toEqual([ + { type: "text", text: "prefix", cache_control: EPHEMERAL }, + ]); + expect(marked[1]).toEqual({ role: "user", content: "tail" }); + }); + + it("leaves the flat single-message form untouched", () => { + const flat = [{ role: "user", content: "everything" }]; + expect(applyAnthropicCacheControl(flat)).toEqual(flat); + }); + + it("marks the last text part of a message already split into parts", () => { + const marked = applyAnthropicCacheControl([ + { + role: "system", + content: [ + { type: "text", text: "a" }, + { type: "image_url", image_url: { url: "data:," } }, + { type: "text", text: "b" }, + ], + }, + { role: "user", content: "tail" }, + ]); + expect(marked[0]?.content).toEqual([ + { type: "text", text: "a" }, + { type: "image_url", image_url: { url: "data:," } }, + { type: "text", text: "b", cache_control: EPHEMERAL }, + ]); + }); + + it("does not mutate its input", () => { + const messages = [ + { role: "system", content: "prefix" }, + { role: "user", content: "hi" }, + { role: "user", content: "tail" }, + ]; + const snapshot = JSON.stringify(messages); + applyAnthropicCacheControl(messages); + expect(JSON.stringify(messages)).toBe(snapshot); + }); +}); diff --git a/src/llm/provider/openai/prompt-cache-control.ts b/src/llm/provider/openai/prompt-cache-control.ts new file mode 100644 index 00000000..4e1fc940 --- /dev/null +++ b/src/llm/provider/openai/prompt-cache-control.ts @@ -0,0 +1,118 @@ +/** + * Anthropic prompt-cache breakpoints on an OpenAI-shaped `messages` + * array. + * + * Anthropic caches nothing unless the request says where the cacheable + * prefix ends: a `cache_control: {type: "ephemeral"}` on a content part. + * OpenRouter forwards the marker for `anthropic/…` models, and an + * Anthropic-compatible endpoint reads it directly. Without it every step + * of a long turn pays the full input price for a prefix that has not + * changed since the previous step. + * + * Two breakpoints, out of the four Anthropic allows: + * - the system message — the stable prefix, identical for the whole + * session and across sessions on the same tool set; + * - the last history message before the tail — the prefix grows by one + * step's worth of messages each step, and a breakpoint at its end + * lets the next request read everything up to here from the cache. + * + * A breakpoint inside the changing tail would cache nothing and pay the + * cache-write premium, so the final message is never marked. Marks land + * on text parts only: an assistant message that is nothing but + * `tool_calls` has no part to carry one, so the walk skips back to the + * nearest message with text. + */ + +const EPHEMERAL = { type: "ephemeral" } as const; + +/** + * Whether the model id names an Anthropic model, on any service: + * `anthropic/claude-…` through a router, `claude-…` directly. + */ +export function isAnthropicModel(modelId: string): boolean { + const lower = modelId.toLowerCase(); + return lower.startsWith("anthropic/") || lower.includes("claude"); +} + +/** Whether the base URL is Anthropic's own API. */ +export function isAnthropicHost(baseUrl: string | undefined): boolean { + if (!baseUrl) return false; + try { + return new URL(baseUrl).hostname.endsWith("api.anthropic.com"); + } catch { + return false; + } +} + +/** + * Whether Google models routed through OpenRouter should be steered to + * the routes that honour prompt caching (Google AI Studio cached 83 % of + * input in the benchmark; Vertex 1–4 %). + */ +export function isGoogleModel(modelId: string): boolean { + return modelId.toLowerCase().startsWith("google/"); +} + +/** + * OpenRouter `provider` preferences that keep a Google model on its + * cache-capable routes while still allowing a fallback when both are + * down. Applied only when the operator configured no preferences of + * their own and `llm.openrouter.preferCacheRoutes` is not off. + */ +export const GOOGLE_CACHE_ROUTE_PREFERENCES: Readonly> = + Object.freeze({ + order: ["Google AI Studio", "Google"], + allow_fallbacks: true, + }); + +type Message = Record; + +/** + * Return a copy of `messages` with the breakpoints applied. The input + * array and its messages are not mutated. Messages with a non-string + * `content` (already parts, or `null`) are left as they are except that a + * parts array gets the marker on its last text part. + */ +export function applyAnthropicCacheControl( + messages: ReadonlyArray, +): Message[] { + const out = messages.map((message) => ({ ...message })); + if (out.length === 0) return out; + const first = out[0]!; + if (first.role === "system") { + out[0] = withCacheControl(first) ?? first; + } + // The last history message: everything before the final (tail) + // message, and after the system message. Walk back over messages that + // cannot carry a marker. + for (let i = out.length - 2; i >= 1; i -= 1) { + const marked = withCacheControl(out[i]!); + if (marked !== null) { + out[i] = marked; + break; + } + } + return out; +} + +function withCacheControl(message: Message): Message | null { + const content = message.content; + if (typeof content === "string") { + if (content.length === 0) return null; + return { + ...message, + content: [{ type: "text", text: content, cache_control: EPHEMERAL }], + }; + } + if (Array.isArray(content)) { + for (let i = content.length - 1; i >= 0; i -= 1) { + const part = content[i] as Record | null; + if (part && part.type === "text" && typeof part.text === "string") { + const parts = [...content]; + parts[i] = { ...part, cache_control: EPHEMERAL }; + return { ...message, content: parts }; + } + } + } + return null; +} diff --git a/src/llm/provider/openai/qwen-tagged-tool-response-adapter.test.ts b/src/llm/provider/openai/qwen-tagged-tool-response-adapter.test.ts index b32657bb..20d28534 100644 --- a/src/llm/provider/openai/qwen-tagged-tool-response-adapter.test.ts +++ b/src/llm/provider/openai/qwen-tagged-tool-response-adapter.test.ts @@ -635,3 +635,94 @@ describe("adaptQwenTaggedToolResponse with a strict tools payload", () => { expect(adaptQwenTaggedToolResponse(response, { tools })).toBe(response); }); }); + +describe("adaptQwenTaggedToolResponse — the Hermes JSON dialect", () => { + const request = { prompt: "x", tools: offeredTools }; + + it("converts `{json}` blocks, typed arguments kept as they are", () => { + const adapted = adaptQwenTaggedToolResponse( + responseWith({ + role: "assistant", + content: [ + '{"name": "os.fs.read", "arguments": {"count": 2, "enabled": true, "paths": ["a"]}}', + '\n{"name": "reply", "arguments": {"text": "done"}}\n', + ].join(""), + }), + request, + ); + const message = firstMessage(adapted); + expect(message.content).toBeNull(); + expect( + (message.tool_calls as Array<{ function: { name: string; arguments: string } }>).map( + (call) => [call.function.name, JSON.parse(call.function.arguments)], + ), + ).toEqual([ + ["os__fs__read", { count: 2, enabled: true, paths: ["a"] }], + ["reply", { text: "done" }], + ]); + }); + + it("accepts `parameters` for the arguments object and a call with none", () => { + const adapted = adaptQwenTaggedToolResponse( + responseWith({ + role: "assistant", + content: + '{"name": "os__fs__read", "parameters": {"text": "t"}}{"name": "os.fs.read"}', + }), + request, + ); + expect( + (firstMessage(adapted).tool_calls as Array<{ function: { arguments: string } }>).map( + (call) => JSON.parse(call.function.arguments), + ), + ).toEqual([{ text: "t" }, {}]); + }); + + it("coerces a string standing where the schema wants another type", () => { + const adapted = adaptQwenTaggedToolResponse( + responseWith({ + role: "assistant", + content: + '{"name": "os.fs.read", "arguments": {"count": "3", "ratio": "0.5", "enabled": "false"}}', + }), + request, + ); + const call = (firstMessage(adapted).tool_calls as Array<{ function: { arguments: string } }>)[0]!; + expect(JSON.parse(call.function.arguments)).toEqual({ + count: 3, + ratio: 0.5, + enabled: false, + }); + }); + + it("rejects an unknown tool, an undeclared or mistyped argument, malformed JSON, and prose around the block", () => { + const inputs = [ + '{"name": "not.offered", "arguments": {}}', + '{"name": "os.fs.read", "arguments": {"nope": 1}}', + '{"name": "os.fs.read", "arguments": {"count": 1.5}}', + '{"name": "reply", "arguments": {}}', + '{"name": "reply", "arguments": {"text": "x"}', + '{"arguments": {"text": "x"}}', + 'Call it like this: {"name": "reply", "arguments": {"text": "x"}}', + ]; + for (const content of inputs) { + const original = responseWith({ role: "assistant", content }); + expect(adaptQwenTaggedToolResponse(original, request)).toBe(original); + } + }); + + it("ignores reasoning_content when told to", () => { + const original = responseWith({ + role: "assistant", + content: "", + reasoning_content: + '{"name": "reply", "arguments": {"text": "x"}}', + }); + expect( + adaptQwenTaggedToolResponse(original, request, { fromReasoning: false }), + ).toBe(original); + expect( + firstMessage(adaptQwenTaggedToolResponse(original, request)).tool_calls, + ).toHaveLength(1); + }); +}); diff --git a/src/llm/provider/openai/qwen-tagged-tool-response-adapter.ts b/src/llm/provider/openai/qwen-tagged-tool-response-adapter.ts index f3335b99..eb530334 100644 --- a/src/llm/provider/openai/qwen-tagged-tool-response-adapter.ts +++ b/src/llm/provider/openai/qwen-tagged-tool-response-adapter.ts @@ -15,18 +15,40 @@ type OfferedTool = { required: ReadonlySet; }; -type TaggedCall = { - name: string; - parameters: Array<{ name: string; value: string }>; -}; +/** + * One `` block, in either dialect the parser accepts: + * - Qwen's XML-ish form, `V…`, + * whose values are strings the schema coerces (`parameters`); + * - the Hermes / ChatML form, `{"name": NAME, "arguments": {…}}`, whose + * values are already typed JSON (`args`). + */ +type TaggedCall = + | { + name: string; + parameters: Array<{ name: string; value: string }>; + } + | { name: string; args: Record }; -const TOOL_CALL_RE = - /\s*\s*\n]+)>([\s\S]*?)<\/function>\s*<\/tool_call>/gy; +const TOOL_CALL_BLOCK_RE = /\s*([\s\S]*?)<\/tool_call>/gy; +const QWEN_FUNCTION_RE = /^\s*\n]+)>([\s\S]*?)<\/function>\s*$/; const PARAMETER_RE = /\s*\n]+)>([\s\S]*?)<\/parameter>/gy; +export interface TaggedToolAdaptOptions { + /** + * Also read a tagged call out of `reasoning_content` when `content` + * holds none (#105 — Qwen thinking models put the call there). The + * adapter's own default, and what the Qwen kind sends; every other + * kind passes `false`, because on those services the reasoning channel + * is scratch space and a call quoted while the model thinks is not a + * call. + */ + fromReasoning?: boolean; +} + export function adaptQwenTaggedToolResponse( response: Record, request: Pick, + options: TaggedToolAdaptOptions = {}, ): Record { const choices = response.choices as Array> | undefined; @@ -49,7 +71,9 @@ export function adaptQwenTaggedToolResponse( // (`[]`) takes the same reasoning path, so `fromReasoning` covers both. const fromReasoning = contentCalls === null || contentCalls.length === 0; const toolCalls = fromReasoning - ? parseSource(message.reasoning_content, offered) + ? options.fromReasoning === false + ? null + : parseSource(message.reasoning_content, offered) : contentCalls; if (!toolCalls || toolCalls.length === 0) return response; @@ -79,6 +103,7 @@ export function adaptQwenTaggedToolResponse( export function adaptQwenCompletionResult( result: CompletionResult, request: Pick, + options: TaggedToolAdaptOptions = {}, ): CompletionResult { const wire = { choices: [ @@ -92,7 +117,10 @@ export function adaptQwenCompletionResult( }, ], }; - const adapted = adaptQwenTaggedToolResponse(wire, request); + const adapted = adaptQwenTaggedToolResponse(wire, request, options); + // Declined: the completion is exactly what the provider returned, and + // rebuilding it would turn an absent `toolCalls` into an empty array. + if (adapted === wire) return result; const choice = (adapted.choices as Array>)[0]; const message = choice?.message as Record | undefined; if (!message) return result; @@ -181,7 +209,10 @@ function parseSource( for (const taggedCall of tagged) { const tool = offered.get(taggedCall.name.trim()); if (!tool) return null; - const args = coerceArguments(taggedCall.parameters, tool); + const args = + "args" in taggedCall + ? coerceTypedArguments(taggedCall.args, tool) + : coerceArguments(taggedCall.parameters, tool); if (!args) return null; calls.push({ id: `call_qwen_tagged_${calls.length}`, @@ -195,21 +226,109 @@ function parseSource( return calls; } +/** + * Every `` block in `source`, which must consist of nothing + * else: text before, between or after the blocks makes the whole thing a + * reply that quotes the syntax, and `null` says so. + */ function parseTaggedCalls(source: string): TaggedCall[] | null { const calls: TaggedCall[] = []; let offset = 0; while (offset < source.length) { - TOOL_CALL_RE.lastIndex = offset; - const match = TOOL_CALL_RE.exec(source); + TOOL_CALL_BLOCK_RE.lastIndex = offset; + const match = TOOL_CALL_BLOCK_RE.exec(source); if (!match) return source.slice(offset).trim().length === 0 ? calls : null; - const parameters = parseParameters(match[2] ?? ""); - if (!parameters) return null; - calls.push({ name: match[1] ?? "", parameters }); - offset = TOOL_CALL_RE.lastIndex; + const call = parseTaggedCallBody(match[1] ?? ""); + if (!call) return null; + calls.push(call); + offset = TOOL_CALL_BLOCK_RE.lastIndex; } return calls; } +/** The inside of one block, in whichever dialect it is written. */ +function parseTaggedCallBody(body: string): TaggedCall | null { + const trimmed = body.trim(); + if (trimmed.length === 0) return null; + const qwen = QWEN_FUNCTION_RE.exec(trimmed); + if (qwen) { + const parameters = parseParameters(qwen[2] ?? ""); + if (!parameters) return null; + return { name: qwen[1] ?? "", parameters }; + } + if (!trimmed.startsWith("{")) return null; + // Hermes / ChatML: `{"name": …, "arguments": {…}}`. Some fine-tunes + // write `parameters` for the arguments object; a bare name with no + // arguments object at all is a call with none. + let parsed: unknown; + try { + parsed = JSON.parse(trimmed); + } catch { + return null; + } + const record = asRecord(parsed); + if (!record || typeof record.name !== "string") return null; + const args = asRecord(record.arguments) ?? asRecord(record.parameters); + if ( + args === null && + record.arguments !== undefined && + record.parameters !== undefined + ) { + return null; + } + return { name: record.name, args: args ?? {} }; +} + +/** + * Hermes arguments arrive typed, so only a string standing where the + * schema wants something else is coerced — the same reading the Qwen + * form gets for every value — and the result is validated the same way. + */ +function coerceTypedArguments( + raw: Record, + tool: OfferedTool, +): Record | null { + const args = Object.create(null) as Record; + try { + for (const [name, value] of Object.entries(raw)) { + if (!Object.hasOwn(tool.properties, name)) { + throw new Error("invalid parameter"); + } + const schema = tool.properties[name] ?? {}; + args[name] = + typeof value === "string" && !admitsString(schema) + ? coerceJsonSchemaValue(value, schema) + : value; + } + for (const name of tool.required) { + if (!Object.hasOwn(args, name)) + throw new Error("missing required parameter"); + } + if (!validateJsonSchemaValue(args, tool.schema)) { + throw new Error("arguments do not match offered schema"); + } + return args; + } catch { + return null; + } +} + +/** Whether a property schema takes a string as it is. */ +function admitsString(schema: Record): boolean { + const type = schema.type; + if (type === undefined) return true; + if (type === "string") return true; + if (Array.isArray(type) && type.includes("string")) return true; + const anyOf = schema.anyOf; + if (Array.isArray(anyOf)) { + return anyOf.some((branch) => { + const record = asRecord(branch); + return record !== null && admitsString(record); + }); + } + return false; +} + function parseParameters( source: string, ): Array<{ name: string; value: string }> | null { @@ -230,7 +349,7 @@ function parseParameters( } function coerceArguments( - parameters: TaggedCall["parameters"], + parameters: Array<{ name: string; value: string }>, tool: OfferedTool, ): Record | null { const args = Object.create(null) as Record; diff --git a/src/llm/provider/openai/reasoning-extractor.ts b/src/llm/provider/openai/reasoning-extractor.ts index bba4c0c2..d0028956 100644 --- a/src/llm/provider/openai/reasoning-extractor.ts +++ b/src/llm/provider/openai/reasoning-extractor.ts @@ -1,37 +1,60 @@ import type { ReasoningFormat } from "../llm-provider.js"; export interface ReasoningExtractor { + /** Reasoning text carried by one streamed `{ delta }` event. */ extractDelta(payload: Record): string; + /** Reasoning text carried by a unary `choices[0].message`. */ + extractFromMessage(message: Record): string; } -export function createReasoningExtractor( - format: ReasoningFormat, -): ReasoningExtractor { +/** + * The reasoning fields OpenAI-compatible services have settled on, in the + * order `auto` consults them. A message never legitimately carries two of + * them, so the first non-empty one wins. + */ +const REASONING_FIELDS: readonly string[] = [ + "reasoning", + "reasoning_content", + "thinking", +]; + +function fieldFor(format: ReasoningFormat): readonly string[] { switch (format) { case "delta_thinking": - return { - extractDelta: (payload) => { - const delta = payload.delta as Record | undefined; - return typeof delta?.thinking === "string" ? delta.thinking : ""; - }, - }; + return ["thinking"]; case "delta_reasoning_content": - return { - extractDelta: (payload) => { - const delta = payload.delta as Record | undefined; - return typeof delta?.reasoning_content === "string" - ? delta.reasoning_content - : ""; - }, - }; + return ["reasoning_content"]; case "delta_reasoning": - return { - extractDelta: (payload) => { - const delta = payload.delta as Record | undefined; - return typeof delta?.reasoning === "string" ? delta.reasoning : ""; - }, - }; + return ["reasoning"]; + case "auto": + return REASONING_FIELDS; default: - return { extractDelta: () => "" }; + return []; + } +} + +function firstString( + record: Record | undefined, + fields: readonly string[], +): string { + if (!record) return ""; + for (const field of fields) { + const value = record[field]; + if (typeof value === "string" && value.length > 0) return value; } + return ""; +} + +export function createReasoningExtractor( + format: ReasoningFormat, +): ReasoningExtractor { + const fields = fieldFor(format); + return { + extractDelta: (payload) => + firstString( + payload.delta as Record | undefined, + fields, + ), + extractFromMessage: (message) => firstString(message, fields), + }; } diff --git a/src/llm/provider/openrouter/openrouter-provider-routing.test.ts b/src/llm/provider/openrouter/openrouter-provider-routing.test.ts index d5bd5778..9f86028c 100644 --- a/src/llm/provider/openrouter/openrouter-provider-routing.test.ts +++ b/src/llm/provider/openrouter/openrouter-provider-routing.test.ts @@ -152,3 +152,46 @@ describe("OpenRouterProvider — providerPreferences", () => { expect(vision.bodies[0]).not.toHaveProperty("provider"); }); }); + +describe("OpenRouterProvider — cache-capable routes for Google models", () => { + const GOOGLE = "google/gemini-3.8-flash"; + const CACHE_ROUTES = { + order: ["Google AI Studio", "Google"], + allow_fallbacks: true, + }; + + it("pins a Google model to its caching routes by default, unary and streamed", async () => { + const { bodies, fetchImpl } = capture(unaryReply); + const provider = openRouter(fetchImpl, { defaultChatModel: GOOGLE }); + await provider.complete(request); + expect(bodies[0]?.provider).toEqual(CACHE_ROUTES); + + const streamed = capture(streamReply); + await drain( + openRouter(streamed.fetchImpl, { defaultChatModel: GOOGLE }).completeStream(request), + ); + expect(streamed.bodies[0]?.provider).toEqual(CACHE_ROUTES); + }); + + it("lets the operator's own providerPreferences win", async () => { + const { bodies, fetchImpl } = capture(unaryReply); + await openRouter(fetchImpl, { + defaultChatModel: GOOGLE, + providerPreferences: PREFERENCES, + }).complete(request); + expect(bodies[0]?.provider).toEqual(PREFERENCES); + }); + + it("sends nothing when preferCacheRoutes is off, or the model is not Google's", async () => { + const off = capture(unaryReply); + await openRouter(off.fetchImpl, { + defaultChatModel: GOOGLE, + preferCacheRoutes: false, + }).complete(request); + expect(off.bodies[0]).not.toHaveProperty("provider"); + + const other = capture(unaryReply); + await openRouter(other.fetchImpl).complete(request); + expect(other.bodies[0]).not.toHaveProperty("provider"); + }); +}); diff --git a/src/llm/provider/openrouter/openrouter-provider.ts b/src/llm/provider/openrouter/openrouter-provider.ts index 01394ad7..526d7be3 100644 --- a/src/llm/provider/openrouter/openrouter-provider.ts +++ b/src/llm/provider/openrouter/openrouter-provider.ts @@ -2,6 +2,10 @@ import { OpenAiProvider, type OpenAiProviderOptions, } from "../openai/openai-provider.js"; +import { + GOOGLE_CACHE_ROUTE_PREFERENCES, + isGoogleModel, +} from "../openai/prompt-cache-control.js"; /** Root without `/v1` — {@link OpenAiProvider} appends `/v1/chat/completions`. */ export const DEFAULT_OPENROUTER_BASE = "https://openrouter.ai/api"; @@ -30,6 +34,12 @@ export type OpenRouterProviderOptions = Omit< xTitle?: string; /** Comma-separated marketplace categories, e.g. `cli-agent,personal-agent`. */ categories?: string; + /** + * `llm.openrouter.preferCacheRoutes`: when no `providerPreferences` + * are configured, pin `google/…` models to the routes that honour + * prompt caching (`GOOGLE_CACHE_ROUTE_PREFERENCES`). Default `true`. + */ + preferCacheRoutes?: boolean; }; /** @@ -48,13 +58,33 @@ export class OpenRouterProvider extends OpenAiProvider { if (options.categories) { headers["X-OpenRouter-Categories"] = options.categories; } + const defaultChatModel = options.defaultChatModel ?? "openrouter/auto"; super({ ...options, id: options.id, // OpenAiProvider normalizes the base URL. baseUrl: options.baseUrl ?? DEFAULT_OPENROUTER_BASE, headers: { ...headers, ...options.headers }, - defaultChatModel: options.defaultChatModel ?? "openrouter/auto", + defaultChatModel, + providerPreferences: resolveProviderPreferences( + options, + defaultChatModel, + ), }); } } + +/** + * The operator's own routing wins whenever it exists. Absent, a Google + * model gets the cache-capable routes unless the operator switched that + * default off; every other model keeps the body exactly as it was. + */ +function resolveProviderPreferences( + options: OpenRouterProviderOptions, + modelId: string, +): Record | undefined { + if (options.providerPreferences) return options.providerPreferences; + if (options.preferCacheRoutes === false) return undefined; + if (!isGoogleModel(modelId)) return undefined; + return { ...GOOGLE_CACHE_ROUTE_PREFERENCES }; +} diff --git a/src/llm/provider/registry/provider-types.ts b/src/llm/provider/registry/provider-types.ts index b184a0b9..3c65e0f6 100644 --- a/src/llm/provider/registry/provider-types.ts +++ b/src/llm/provider/registry/provider-types.ts @@ -39,6 +39,8 @@ export type LlmProviderConfigEntry = { supportsVision?: boolean; requestTimeoutMs?: number; promptCache?: "auto" | "off" | "explicit-markers"; + /** Native-tools request layout; see `UserLlmProviderEntry.messageShape`. */ + messageShape?: "native" | "flat"; /** * OpenRouter provider routing, sent as the chat body's `provider` * object. Only the `openrouter` factory forwards it; an explicit @@ -92,6 +94,8 @@ export type UserModelConfigEntry = { cacheRead?: number; cacheWrite?: number; }; + /** Per-model wire parameters; see `UserModelEntry.params`. */ + params?: Record; }; export type LlmFallbackConfig = { diff --git a/src/llm/provider/registry/register-built-in-providers.test.ts b/src/llm/provider/registry/register-built-in-providers.test.ts index 63990fea..eb205b80 100644 --- a/src/llm/provider/registry/register-built-in-providers.test.ts +++ b/src/llm/provider/registry/register-built-in-providers.test.ts @@ -87,3 +87,132 @@ describe("providerPreferences through the built-in factories", () => { }, ); }); + +describe("userModels[] wire options through the built-in factories", () => { + // `reasoningFormat` and `params` on the entry's row for the model it + // serves reach the provider; a row for another model does not. + it.each(["openai-compatible", "qwen-openai-compatible", "openrouter", "aimlapi", "gemini"])( + "%s applies the served model's params and reasoning format", + async (kind) => { + registerBuiltInProviderKinds(); + const factory = getProviderFactory(kind); + if (!factory) throw new Error(`${kind} is not registered`); + const bodies: Record[] = []; + vi.stubGlobal( + "fetch", + vi.fn(async (_url: string, init?: RequestInit) => { + bodies.push(JSON.parse(String(init?.body)) as Record); + return new Response( + JSON.stringify({ + choices: [ + { message: { role: "assistant", content: "ok" }, finish_reason: "stop" }, + ], + }), + { status: 200, headers: { "content-type": "application/json" } }, + ); + }), + ); + const provider = await factory({ + config: {} as AtomicAgentConfig, + entry: { + id: kind, + kind, + apiKey: "test-key", + baseUrl: "https://example.invalid", + defaultChatModel: "served-model", + userModels: [ + { + id: "other-model", + kind: "chat", + reasoningFormat: "delta_thinking", + params: { top_p: 0.1 }, + }, + { + id: "served-model", + kind: "chat", + reasoningFormat: "delta_reasoning_content", + params: { top_p: 0.9, presence_penalty: 0.5 }, + }, + ], + }, + logger: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() } as never, + }); + expect(provider.capabilities.reasoningFormat).toBe("delta_reasoning_content"); + await provider.complete({ prompt: "hi" }); + expect(bodies[0]).toMatchObject({ top_p: 0.9, presence_penalty: 0.5 }); + }, + ); + + it("defaults to `auto` reasoning and no extra parameters without a row", async () => { + registerBuiltInProviderKinds(); + const factory = getProviderFactory("openai-compatible"); + if (!factory) throw new Error("openai-compatible is not registered"); + const provider = await factory({ + config: {} as AtomicAgentConfig, + entry: { + id: "plain", + kind: "openai-compatible", + apiKey: "test-key", + baseUrl: "https://example.invalid", + defaultChatModel: "some-model", + }, + logger: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() } as never, + }); + expect(provider.capabilities.reasoningFormat).toBe("auto"); + }); +}); + +describe("llm.openrouter.preferCacheRoutes through the openrouter factory", () => { + async function openRouterBody( + config: Partial, + entry: Partial, + ): Promise> { + registerBuiltInProviderKinds(); + const factory = getProviderFactory("openrouter"); + if (!factory) throw new Error("openrouter is not registered"); + const bodies: Record[] = []; + vi.stubGlobal( + "fetch", + vi.fn(async (_url: string, init?: RequestInit) => { + bodies.push(JSON.parse(String(init?.body)) as Record); + return new Response( + JSON.stringify({ + choices: [{ message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + }), + { status: 200, headers: { "content-type": "application/json" } }, + ); + }), + ); + const provider = await factory({ + config: config as AtomicAgentConfig, + entry: { + id: "openrouter", + kind: "openrouter", + apiKey: "test-key", + defaultChatModel: "google/gemini-3.8-flash", + ...entry, + }, + logger: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() } as never, + }); + await provider.complete({ prompt: "hi" }); + return bodies[0]!; + } + + it("is on when the file says nothing", async () => { + const body = await openRouterBody({}, {}); + expect(body.provider).toEqual({ + order: ["Google AI Studio", "Google"], + allow_fallbacks: true, + }); + }); + + it("is off when the file says so, and yields to the entry's own preferences", async () => { + const off = await openRouterBody( + { llm: { openrouter: { preferCacheRoutes: false } } } as Partial, + {}, + ); + expect(off).not.toHaveProperty("provider"); + const own = await openRouterBody({}, { providerPreferences: PREFERENCES }); + expect(own.provider).toEqual(PREFERENCES); + }); +}); diff --git a/src/llm/provider/registry/register-built-in-providers.ts b/src/llm/provider/registry/register-built-in-providers.ts index 1ef3d125..8f4cb912 100644 --- a/src/llm/provider/registry/register-built-in-providers.ts +++ b/src/llm/provider/registry/register-built-in-providers.ts @@ -20,10 +20,35 @@ import { resolveCliAdapter, SubscriptionCliProvider, } from "../subscription-cli/index.js"; -import { registerProviderKind } from "./provider-types.js"; +import { + registerProviderKind, + type LlmProviderConfigEntry, +} from "./provider-types.js"; let registered = false; +/** + * What the entry's `userModels[]` row for the model it serves says about + * the wire: a pinned reasoning field (`reasoningFormat`) and per-model + * request parameters (`params`). Read here, at construction, because the + * provider is rebuilt whenever the entry changes; a row for some other + * model on the same entry says nothing about this one. Absent, the + * provider keeps its defaults (`auto` reasoning, no extra parameters). + */ +function modelWireOptions( + entry: LlmProviderConfigEntry, + modelId: string, +): { + reasoningFormat?: NonNullable[number]["reasoningFormat"]; + modelParams?: Record; +} { + const row = entry.userModels?.find((model) => model.id === modelId); + return { + ...(row?.reasoningFormat ? { reasoningFormat: row.reasoningFormat } : {}), + ...(row?.params ? { modelParams: row.params } : {}), + }; +} + export function registerBuiltInProviderKinds(): void { if (registered) return; registered = true; @@ -56,6 +81,7 @@ export function registerBuiltInProviderKinds(): void { } return new OpenAiProvider({ id: entry.id, + providerKind: "openai-compatible", baseUrl: entry.baseUrl, apiKey: entry.apiKey ?? "", defaultChatModel: entry.defaultChatModel, @@ -67,6 +93,9 @@ export function registerBuiltInProviderKinds(): void { extraBody: entry.extraBody, maxOutputTokens: entry.maxOutputTokens, strictTools: entry.strictTools, + promptCache: entry.promptCache, + messageShape: entry.messageShape, + ...modelWireOptions(entry, entry.defaultChatModel), logger: ctx.logger, }); }); @@ -80,6 +109,7 @@ export function registerBuiltInProviderKinds(): void { } return new OpenAiProvider({ id: entry.id, + providerKind: "qwen-openai-compatible", baseUrl: entry.baseUrl, apiKey: entry.apiKey ?? "", defaultChatModel: entry.defaultChatModel, @@ -92,17 +122,22 @@ export function registerBuiltInProviderKinds(): void { extraBody: entry.extraBody, maxOutputTokens: entry.maxOutputTokens, strictTools: entry.strictTools, + promptCache: entry.promptCache, + messageShape: entry.messageShape, + ...modelWireOptions(entry, entry.defaultChatModel), logger: ctx.logger, }); }); registerProviderKind("openrouter", (ctx) => { const entry = ctx.entry; + const model = entry.defaultChatModel ?? "openrouter/auto"; return new OpenRouterProvider({ id: entry.id, + providerKind: "openrouter", baseUrl: entry.baseUrl, apiKey: entry.apiKey ?? "", - defaultChatModel: entry.defaultChatModel ?? "openrouter/auto", + defaultChatModel: model, headers: entry.headers, supportsVision: entry.supportsVision ?? true, supportsParallelTools: entry.supportsTools ?? true, @@ -110,10 +145,17 @@ export function registerBuiltInProviderKinds(): void { extraBody: entry.extraBody, maxOutputTokens: entry.maxOutputTokens, strictTools: entry.strictTools, + promptCache: entry.promptCache, + messageShape: entry.messageShape, + ...modelWireOptions(entry, model), // OpenRouter's own `provider` routing block. Deliberately wired on // this kind alone: it is not part of the OpenAI schema, and no // other kind here documents a field by that name. providerPreferences: entry.providerPreferences, + // `llm.openrouter.preferCacheRoutes`, the llm-wide default that + // steers a Google model to its cache-capable routes when the entry + // pinned none itself. Absent in the file means on. + preferCacheRoutes: ctx.config.llm?.openrouter?.preferCacheRoutes ?? true, logger: ctx.logger, httpReferer: OPENROUTER_APP_REFERER, xTitle: OPENROUTER_APP_TITLE, @@ -123,14 +165,19 @@ export function registerBuiltInProviderKinds(): void { registerProviderKind("aimlapi", (ctx) => { const entry = ctx.entry; + const model = entry.defaultChatModel ?? AIMLAPI_DEFAULT_CHAT_MODEL; return new AimlapiProvider({ id: entry.id, + providerKind: "aimlapi", baseUrl: entry.baseUrl, apiKey: entry.apiKey ?? "", - defaultChatModel: entry.defaultChatModel ?? AIMLAPI_DEFAULT_CHAT_MODEL, + defaultChatModel: model, extraBody: entry.extraBody, maxOutputTokens: entry.maxOutputTokens, strictTools: entry.strictTools, + promptCache: entry.promptCache, + messageShape: entry.messageShape, + ...modelWireOptions(entry, model), headers: entry.headers, supportsVision: entry.supportsVision ?? true, supportsParallelTools: entry.supportsTools ?? true, @@ -141,14 +188,19 @@ export function registerBuiltInProviderKinds(): void { registerProviderKind("gemini", (ctx) => { const entry = ctx.entry; + const model = entry.defaultChatModel ?? GEMINI_DEFAULT_CHAT_MODEL; return new GeminiProvider({ id: entry.id, + providerKind: "gemini", baseUrl: entry.baseUrl, apiKey: entry.apiKey ?? "", - defaultChatModel: entry.defaultChatModel ?? GEMINI_DEFAULT_CHAT_MODEL, + defaultChatModel: model, extraBody: entry.extraBody, maxOutputTokens: entry.maxOutputTokens, strictTools: entry.strictTools, + promptCache: entry.promptCache, + messageShape: entry.messageShape, + ...modelWireOptions(entry, model), headers: entry.headers, supportsVision: entry.supportsVision ?? true, supportsParallelTools: entry.supportsTools ?? true, diff --git a/src/llm/provider/usage-cost.test.ts b/src/llm/provider/usage-cost.test.ts new file mode 100644 index 00000000..6a4608c4 --- /dev/null +++ b/src/llm/provider/usage-cost.test.ts @@ -0,0 +1,38 @@ +import { describe, expect, it } from "vitest"; + +import { estimateUsageCostUsd } from "./usage-cost.js"; + +describe("estimateUsageCostUsd", () => { + it("prices cached prompt tokens at the cacheRead rate", () => { + const usd = estimateUsageCostUsd( + { promptTokens: 1_000_000, completionTokens: 0, totalTokens: 1_000_000, cachedTokens: 800_000 }, + { input: 3, output: 15, cacheRead: 0.3 }, + ); + // 200k uncached at $3/M + 800k cached at $0.30/M. + expect(usd).toBeCloseTo(0.6 + 0.24, 9); + }); + + it("charges the plain input rate when no cacheRead rate is known", () => { + const usd = estimateUsageCostUsd( + { promptTokens: 1_000_000, completionTokens: 0, totalTokens: 1_000_000, cachedTokens: 800_000 }, + { input: 3, output: 15 }, + ); + expect(usd).toBeCloseTo(3, 9); + }); + + it("is the historical formula when nothing was cached", () => { + const usd = estimateUsageCostUsd( + { promptTokens: 500_000, completionTokens: 100_000, totalTokens: 600_000 }, + { input: 2, output: 10, cacheRead: 0.2 }, + ); + expect(usd).toBeCloseTo(1 + 1, 9); + }); + + it("never lets a cached count exceed the prompt it is part of", () => { + const usd = estimateUsageCostUsd( + { promptTokens: 1_000, completionTokens: 0, totalTokens: 1_000, cachedTokens: 5_000 }, + { input: 1_000_000, output: 0, cacheRead: 0 }, + ); + expect(usd).toBe(0); + }); +}); diff --git a/src/llm/provider/usage-cost.ts b/src/llm/provider/usage-cost.ts new file mode 100644 index 00000000..54f24a95 --- /dev/null +++ b/src/llm/provider/usage-cost.ts @@ -0,0 +1,29 @@ +import type { CompletionUsage } from "./completion-types.js"; +import type { ResolvedModel } from "./model-resolver.js"; + +/** + * Estimated spend of one completion, in USD, from its usage and the + * model's per-million-token prices. + * + * Cached prompt tokens are a subset of `promptTokens`: they are billed at + * the model's `cacheRead` rate when the catalog or the operator names + * one, and at the plain input rate otherwise (an unknown discount is not + * assumed). A cache-write premium is not modelled — services do not + * report which tokens were written, only which were read. + */ +export function estimateUsageCostUsd( + usage: CompletionUsage, + pricing: NonNullable, +): number { + const cached = Math.min( + Math.max(0, usage.cachedTokens ?? 0), + Math.max(0, usage.promptTokens), + ); + const cacheReadRate = pricing.cacheRead ?? pricing.input; + const uncached = usage.promptTokens - cached; + return ( + (uncached / 1_000_000) * pricing.input + + (cached / 1_000_000) * cacheReadRate + + (usage.completionTokens / 1_000_000) * pricing.output + ); +} diff --git a/src/prompt/build-prompt-types.ts b/src/prompt/build-prompt-types.ts index 2f565c0a..4bcea07a 100644 --- a/src/prompt/build-prompt-types.ts +++ b/src/prompt/build-prompt-types.ts @@ -1,5 +1,8 @@ import type { ModelProfile } from "../llm/model-profile.js"; -import type { ToolCallTransport } from "../llm/provider/completion-types.js"; +import type { + PromptMessages, + ToolCallTransport, +} from "../llm/provider/completion-types.js"; import type { ProfileFact } from "../memory/profile-store.js"; import type { ProfileClipStats } from "./clip-profile-section.js"; import type { ConversationPackStart } from "../session/conversation-turn.js"; @@ -119,6 +122,13 @@ export interface BuiltPrompt { text: string; stablePrefix: string; tail: string; + /** + * The same prompt as structure — stable prefix, the packed turns, and + * the tail without its `### conversation` section — for a provider + * that sends history as real chat messages. Built from the same packed + * conversation as `text`, so the two never disagree. + */ + messages: PromptMessages; tokens: { stablePrefix: number; loadedSkills: number; diff --git a/src/prompt/build-prompt-world-conversation.test.ts b/src/prompt/build-prompt-world-conversation.test.ts index fcb54f10..c4994c63 100644 --- a/src/prompt/build-prompt-world-conversation.test.ts +++ b/src/prompt/build-prompt-world-conversation.test.ts @@ -5,7 +5,10 @@ import { toolResultTurn, userTurn, } from "../session/conversation-turn.js"; -import { renderPackedConversation } from "./build-prompt-world-conversation.js"; +import { + packedConversationTurns, + renderPackedConversation, +} from "./build-prompt-world-conversation.js"; const bigFile = Array.from( { length: 300 }, @@ -68,3 +71,58 @@ describe("renderPackedConversation", () => { ]); }); }); + +describe("packedConversationTurns", () => { + it("caps each tool-result body exactly as the flat rendering does", () => { + const packed = { + visibleTurns: [ + userTurn("review the game", 1), + readCall({ path: "js/main.js" }, 2), + readCall({ path: "js/ship.js", offset: 101, limit: 300 }, 3), + readResult(4), + readResult(5), + assistantReplyTurn("done", 6), + ], + droppedSummary: "summary: older turns", + }; + const flat = renderPackedConversation(packed).split("\n"); + const turns = packedConversationTurns(packed); + expect(turns.map((t) => t.kind)).toEqual([ + "user", + "assistant_tool_call", + "assistant_tool_call", + "tool_result", + "tool_result", + "assistant_reply", + ]); + // Every structured body appears verbatim behind its flat header: + // the same cap, the same paging hint, the same offset. + const rendered = renderPackedConversation(packed); + const structuredBodies = turns + .filter((t): t is Extract => t.kind === "tool_result") + .map((t) => t.body); + expect(structuredBodies).toHaveLength(2); + for (const body of structuredBodies) { + expect(body.length).toBeLessThan(bigFile.length); + expect(rendered).toContain(`tool_result[os.fs.read ok]: ${body}`); + } + expect(structuredBodies[0]).toContain("offset: "); + expect(structuredBodies[1]).toContain("offset: "); + expect(structuredBodies[0]).not.toBe(structuredBodies[1]); + expect(flat[0]).toBe("summary: older turns"); + }); + + it("carries a reply's attachment note the way the flat line does", () => { + const turns = packedConversationTurns({ + visibleTurns: [ + assistantReplyTurn("here", { at: 1, attachments: ["/tmp/a.png"] }), + toolResultTurn({ tool: "os.shell.run", status: "error", summary: "boom", truncated: true, at: 2 }), + ], + droppedSummary: null, + }); + expect(turns).toEqual([ + { kind: "assistant_reply", text: "here (attached: /tmp/a.png)" }, + { kind: "tool_result", tool: "os.shell.run", status: "error", body: "boom", truncated: true }, + ]); + }); +}); diff --git a/src/prompt/build-prompt-world-conversation.ts b/src/prompt/build-prompt-world-conversation.ts index 7f8d6b92..b1642de4 100644 --- a/src/prompt/build-prompt-world-conversation.ts +++ b/src/prompt/build-prompt-world-conversation.ts @@ -1,7 +1,9 @@ +import type { PromptTurn } from "../llm/provider/completion-types.js"; import type { SessionState } from "../session/session-state.js"; import { findCurrentMacroTurnStart, readStartLineOf, + renderToolResultBody, renderTurnForPrompt, type ConversationTurn, type RenderTurnOptions, @@ -15,20 +17,81 @@ export function renderWorldSnapshotSection(session: SessionState): string { ); } +type PackedTurns = { + visibleTurns: readonly ConversationTurn[]; + droppedSummary: string | null; +}; + /** * Render the packed conversation section. When `packConversation` folded * older turns into a summary, that summary is emitted as the first line * so the model can tell the transcript was compressed. */ -export function renderPackedConversation(packed: { - visibleTurns: readonly ConversationTurn[]; - droppedSummary: string | null; -}): string { +export function renderPackedConversation(packed: PackedTurns): string { if (packed.visibleTurns.length === 0 && packed.droppedSummary === null) { return "(no messages yet)"; } const lines: string[] = []; if (packed.droppedSummary) lines.push(packed.droppedSummary); + for (const [turn, options] of packedTurnRenderOptions(packed)) { + lines.push(renderTurnForPrompt(turn, options)); + } + return lines.join("\n"); +} + +/** + * The same packed conversation as structure, for a provider that lays + * history out as real chat messages. Each row carries what its text line + * carries — a tool-result body capped by the same `RenderTurnOptions` the + * text form applied, a reply with its attachment note — so the two forms + * describe one transcript. + */ +export function packedConversationTurns(packed: PackedTurns): PromptTurn[] { + const out: PromptTurn[] = []; + for (const [turn, options] of packedTurnRenderOptions(packed)) { + switch (turn.kind) { + case "user": + out.push({ kind: "user", text: turn.text }); + break; + case "assistant_tool_call": + out.push({ + kind: "assistant_tool_call", + tool: turn.tool, + args: turn.args, + }); + break; + case "tool_result": + out.push({ + kind: "tool_result", + tool: turn.tool, + status: turn.status, + body: renderToolResultBody(turn, options), + truncated: turn.truncated === true, + }); + break; + case "assistant_reply": + out.push({ + kind: "assistant_reply", + text: + turn.attachments !== undefined && turn.attachments.length > 0 + ? `${turn.text} (attached: ${turn.attachments.join(", ")})` + : turn.text, + }); + break; + } + } + return out; +} + +/** + * Every visible turn paired with the render options the text form uses + * for it. One walk for both renderers, so a cap decided here (fresh + * `os.http.request` bodies, `os.fs.read` paging hints) cannot differ + * between the flat and the structured prompt. + */ +function* packedTurnRenderOptions( + packed: PackedTurns, +): Generator<[ConversationTurn, RenderTurnOptions]> { // Index of the first turn that belongs to the current (un-replied) macro // turn. Tools listed in `TOOLS_FULL_BODY_WHEN_FRESH` (see conversation-turn.ts) // render their full payload only while inside this slice; older @@ -51,7 +114,6 @@ export function renderPackedConversation(packed: { const readStartLine = pendingReadStarts.shift(); if (readStartLine !== undefined) options.readStartLine = readStartLine; } - lines.push(renderTurnForPrompt(turn, options)); + yield [turn, options]; } - return lines.join("\n"); } diff --git a/src/prompt/build-prompt.test.ts b/src/prompt/build-prompt.test.ts index d8dd0742..79ef2820 100644 --- a/src/prompt/build-prompt.test.ts +++ b/src/prompt/build-prompt.test.ts @@ -1780,3 +1780,80 @@ describe("buildPrompt tool roles (F18)", () => { expect(full.tail).toContain("os.git.show"); }); }); + +describe("buildPrompt structured form (`messages`)", () => { + it("exposes the stable prefix, the packed turns and the tail without `### conversation`", () => { + const base = mkSession(); + const prompt = buildPrompt({ + session: { + ...base, + turns: [ + ...base.turns, + { kind: "assistant_tool_call", tool: "browser.read_aria", args: { x: 1 }, at: 1 }, + { + kind: "tool_result", + tool: "browser.read_aria", + status: "error", + summary: "timed out waiting for page", + truncated: true, + at: 2, + }, + ], + }, + toolDescriptors: TOOLS, + capabilities: CAPS, + skillCatalog: SKILLS, + transientNotice: "be brief", + currentDate: "2026-09-14", + toolTransport: "native_tools", + suppressReasoningPrefill: true, + }); + expect(prompt.messages.system).toBe(prompt.stablePrefix); + expect(prompt.messages.droppedSummary).toBeNull(); + expect(prompt.messages.turns).toEqual([ + { kind: "user", text: "Check inbox" }, + { kind: "assistant_tool_call", tool: "browser.read_aria", args: { x: 1 } }, + { + kind: "tool_result", + tool: "browser.read_aria", + status: "error", + body: "timed out waiting for page", + truncated: true, + }, + ]); + const { tail } = prompt.messages; + expect(tail).not.toContain("### conversation"); + expect(tail).not.toContain("assistant_tool_call:"); + expect(tail).toContain("### world"); + expect(tail).toContain("### notice\nbe brief"); + expect(tail).toContain("CURRENT DATE: 2026-09-14"); + expect(tail.trimEnd().endsWith("### respond\nRespond now.")).toBe(true); + // The flat tail is the same halves with the conversation between them. + const [before, after] = tail.split("### task-policy").length > 1 + ? [tail.slice(0, tail.indexOf("### task-policy")), tail.slice(tail.indexOf("### task-policy"))] + : [tail.slice(0, tail.indexOf("### notice")), tail.slice(tail.indexOf("### notice"))]; + expect(prompt.tail.startsWith(before)).toBe(true); + expect(prompt.tail.endsWith(after)).toBe(true); + expect(prompt.tail).toContain("### conversation"); + }); + + it("carries the dropped-turns recap separately from the turns", () => { + const turns = Array.from({ length: 40 }, (_, i) => + i % 2 === 0 + ? { kind: "user" as const, text: `message ${i} ${"x".repeat(200)}`, at: i } + : { kind: "assistant_reply" as const, text: `reply ${i} ${"y".repeat(200)}`, at: i }, + ); + const prompt = buildPrompt({ + session: mkSession({ turns }), + toolDescriptors: TOOLS, + capabilities: CAPS, + skillCatalog: SKILLS, + conversationMaxTokens: 400, + }); + expect(prompt.droppedTurns).toBeGreaterThan(0); + expect(prompt.messages.droppedSummary).toBe( + prompt.tail.slice(prompt.tail.indexOf("### conversation\n") + "### conversation\n".length).split("\n")[0], + ); + expect(prompt.messages.turns.length).toBe(turns.length - prompt.droppedTurns); + }); +}); diff --git a/src/prompt/build-prompt.ts b/src/prompt/build-prompt.ts index c186fdba..a787d78d 100644 --- a/src/prompt/build-prompt.ts +++ b/src/prompt/build-prompt.ts @@ -11,7 +11,9 @@ import { packConversation, pairTokenCosts, } from "../session/conversation-turn.js"; +import type { PromptMessages } from "../llm/provider/completion-types.js"; import { + packedConversationTurns, renderPackedConversation, renderWorldSnapshotSection, } from "./build-prompt-world-conversation.js"; @@ -318,44 +320,43 @@ export function buildPrompt(input: BuildPromptInput): BuiltPrompt { // it, so a change lands in the part of the prompt that is re-read // anyway rather than ahead of a transcript the model would otherwise // have reused from its KV cache. - const tailParts: string[] = []; + // The tail is assembled in two halves around `### conversation`. The + // flat text joins all three; the structured form (`messages`) sends the + // conversation as real chat messages and the two halves as one final + // user message. + const tailBefore: string[] = []; if (memoryIndex !== null) { - tailParts.push("### memory-index", memoryIndex, ``); + tailBefore.push("### memory-index", memoryIndex, ``); } if (factsForTail !== null) { - tailParts.push("### session-facts", factsForTail, ``); + tailBefore.push("### session-facts", factsForTail, ``); } if (recalled !== null) { - tailParts.push("### recalled", recalled, ``); + tailBefore.push("### recalled", recalled, ``); } - tailParts.push( - `### world`, - worldSnapshot, - ``, - `### conversation`, - conversation, - ``, - ); + tailBefore.push(`### world`, worldSnapshot, ``); + const conversationParts = [`### conversation`, conversation, ``]; + const tailAfter: string[] = []; if (profile !== null) { - tailParts.push("### profile", profile, ``); + tailAfter.push("### profile", profile, ``); } if (lessons !== null) { - tailParts.push("### lessons", lessons, ``); + tailAfter.push("### lessons", lessons, ``); } if (procedures !== null) { - tailParts.push("### procedures", procedures, ``); + tailAfter.push("### procedures", procedures, ``); } if (loadedForTail !== null) { - tailParts.push("### loaded-skills", loadedForTail, ``); + tailAfter.push("### loaded-skills", loadedForTail, ``); } if (loadedToolsRendered.body !== null) { - tailParts.push("### loaded-tools", loadedToolsRendered.body, ``); + tailAfter.push("### loaded-tools", loadedToolsRendered.body, ``); } if (taskPolicy !== null) { - tailParts.push(`### task-policy`, taskPolicy.body, ``); + tailAfter.push(`### task-policy`, taskPolicy.body, ``); } if (input.transientNotice && input.transientNotice.length > 0) { - tailParts.push(`### notice`, input.transientNotice, ``); + tailAfter.push(`### notice`, input.transientNotice, ``); } // Current date lives in the variable tail (not the stable prefix) so it // sits close to the generation point where the model actually attends to @@ -363,7 +364,7 @@ export function buildPrompt(input: BuildPromptInput): BuiltPrompt { // the model kept anchoring on its training-era year. Rendered as a bold // standalone line so it stands out. Omitted when not provided. if (input.currentDate) { - tailParts.push( + tailAfter.push( `CURRENT DATE: ${input.currentDate} — this is today. Use it for any time-relative reasoning; never assume an earlier year.`, ``, ); @@ -374,7 +375,17 @@ export function buildPrompt(input: BuildPromptInput): BuiltPrompt { // loops (e.g. "I will write the response. I will check the response." // observed when the only trailing directive lived ~13k tokens upstream). // Byte-stable and short, so it does not meaningfully hurt cache reuse. - tailParts.push(`### respond`, `Respond now.`, ``); + tailAfter.push(`### respond`, `Respond now.`, ``); + // The structured form stops here: the framing and prefill below are + // text-completion artifacts a chat transport never sees (they are + // suppressed for it anyway — `suppressReasoningPrefill`). + const messages: PromptMessages = { + system: stablePrefix, + droppedSummary: packed.droppedSummary, + turns: packedConversationTurns(packed), + tail: [...tailBefore, ...tailAfter].join("\n"), + }; + const tailParts: string[] = [...tailBefore, ...conversationParts, ...tailAfter]; if (turnFraming !== undefined) { // Gemma 4 turn-framing: close the system turn and open the model turn. // The model emits its own `<|channel>thought` block — we do NOT prefill @@ -422,6 +433,7 @@ export function buildPrompt(input: BuildPromptInput): BuiltPrompt { text, stablePrefix, tail, + messages, tokens: { stablePrefix: budgetResult.perSection.stablePrefix, loadedSkills: budgetResult.perSection.loadedSkills, diff --git a/src/runtime/llm-link-attempt.test.ts b/src/runtime/llm-link-attempt.test.ts new file mode 100644 index 00000000..475631c1 --- /dev/null +++ b/src/runtime/llm-link-attempt.test.ts @@ -0,0 +1,65 @@ +import { describe, expect, it } from "vitest"; + +import type { LlmStreamParams } from "../agent/step-executor.js"; +import type { + CompletionRequest, + PromptMessages, +} from "../llm/provider/completion-types.js"; +import type { LlmProvider } from "../llm/provider/llm-provider.js"; +import { + fakeAnswer, + fakeProvider, +} from "../llm/provider/fake-provider.fixture.js"; +import { completeOnLink, openStreamOnLink } from "./llm-link-attempt.js"; + +const messages: PromptMessages = { + system: "prefix", + droppedSummary: null, + turns: [{ kind: "user", text: "hi" }], + tail: "tail", +}; + +const params: LlmStreamParams = { + prompt: "flat", + messages, + grammar: 'root ::= "ok"', + slotId: 0, + sessionId: "s1", + tools: [{ type: "function", function: { name: "reply" } }], +}; + +function recording(transport: "native_tools" | "grammar") { + const requests: CompletionRequest[] = []; + const provider: LlmProvider = fakeProvider( + transport, + transport, + async (request) => { + requests.push(request); + return fakeAnswer(transport); + }, + ); + return { requests, provider }; +} + +describe("llm-link-attempt — the structured prompt", () => { + it("reaches a native-tools link, unary and streamed", async () => { + const { requests, provider } = recording("native_tools"); + const deps = { resolveSlice: () => ({ provider, transport: "native_tools" as const }) }; + await completeOnLink(deps, params, "native_tools"); + expect(requests[0]?.messages).toBe(messages); + expect(requests[0]?.tools).toEqual(params.tools); + + const { primed } = await openStreamOnLink(deps, params, "native_tools"); + for (;;) if ((await primed.rest.next()).done) break; + expect(requests[1]?.messages).toBe(messages); + }); + + it("never reaches a grammar link, which gets the flat prompt and the GBNF", async () => { + const { requests, provider } = recording("grammar"); + const deps = { resolveSlice: () => ({ provider, transport: "grammar" as const }) }; + await completeOnLink(deps, params, "grammar"); + expect(requests[0]).not.toHaveProperty("messages"); + expect(requests[0]?.grammar).toBe(params.grammar); + expect(requests[0]?.prompt).toBe("flat"); + }); +}); diff --git a/src/runtime/llm-link-attempt.ts b/src/runtime/llm-link-attempt.ts index 206b0d91..6a662347 100644 --- a/src/runtime/llm-link-attempt.ts +++ b/src/runtime/llm-link-attempt.ts @@ -50,6 +50,9 @@ function promptFor( function nativeRequestFields(params: LlmStreamParams) { return { + // The structured prompt goes to a native link only: a grammar link + // renders the flat text through its own template and GBNF prelude. + ...(params.messages ? { messages: params.messages } : {}), ...(params.tools ? { tools: params.tools } : {}), ...(params.toolChoice !== undefined ? { toolChoice: params.toolChoice } diff --git a/src/session/conversation-turn.ts b/src/session/conversation-turn.ts index 836ddebe..f077e9b2 100644 --- a/src/session/conversation-turn.ts +++ b/src/session/conversation-turn.ts @@ -184,7 +184,12 @@ export function renderTurnForPrompt( } } -function renderToolResultBody( +/** + * The body of a tool result as the prompt shows it — the same caps + * whether it lands on a `tool_result[…]:` text line or in a native + * `tool` message, which is why it is exported rather than inlined. + */ +export function renderToolResultBody( turn: Extract, options: RenderTurnOptions, ): string {