Per-model context-window defaults for chat agents (stop the silent 4096 fallback) - #247
Merged
TheGreatAxios merged 4 commits intoAug 21, 2026
Merged
Conversation
Chat agents served through the Ollama openai-compatible adapter silently fall back to @intx/inference's built-in 4096-token default (and Ollama's own small built-in context window) whenever nothing sets a per-model ceiling. These tests pin the fix: a curated model's quirks resolve its real advertised context window, a smaller-ceiling model gets its own limit rather than the larger model's, a caller override wins, and an unlisted model gets no guessed override at all -- verified against the adapter's actual built request body, not just the resolved config.
@corbits/ollama-adapter's OllamaAdapterConfig mechanism (numCtx, maxOutputTokens, reasoningEffort) had no real data behind it, so every Ollama-served chat agent still ran at the built-in 4096-token default. quirksForDeployment resolves each curated model's advertised native context window at offering-creation time, the same way capabilitiesForDeployment resolves capabilities -- gpt-oss:20b and llama3.1:8b get their real 128K ceiling, Qwen3's 27B/30B-class models get their smaller 32K native ceiling rather than an over-requested num_ctx that risks allocation failure or heavy swap on the inference host. A model outside this table gets no override, and a caller-supplied override always wins.
seedCatalog computed capabilities for a new offering but never its quirks, so nothing ever reached the model_offering row an Ollama chat agent actually launches against. This test seeds Ollama's curated catalog and asserts each offering's POST body carries the per-model quirks quirksForDeployment resolves -- gpt-oss:20b's real 128K ceiling, Qwen3's smaller real ceiling -- rather than an empty quirks column.
ensureCatalogOffering now threads a quirks value through to the offering POST body -- @intx/hub-api already persists it on model_offering.quirks, and the platform's own InferenceSource resolution already spreads a non-null quirks column into the request-build path, so this is the one place workbench needed to populate it. seedCatalog computes it with quirksForDeployment right alongside capabilitiesForDeployment, scoped to the ollama provider only; every other provider's offerings are unaffected, and a model with no vetted default still gets none.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@intx/inference's built-inoptions.maxTokens ?? 4096fallback, and at whatever smallnum_ctxOllama itself defaults to, whenever nothing set a per-model ceiling -- the model catalog seeded no context/token data at all.@corbits/ollama-adapter(this branch's base, CL-6204: Ollama adapter overrides via the inference adapter manifest #236) built the mechanism to overridenum_ctx/max output tokens/reasoning effort per model, but nothing populated it with real values.quirksForDeployment(packages/inference-catalog/src/ollama-context-defaults.ts), resolved the same waycapabilitiesForDeploymentresolves capabilities:gpt-oss:20bandllama3.1:8bget their real 128K native context window, Qwen3's 27B/30B-class models get their real, smaller 32K native ceiling (not an over-requestednum_ctxthat risks allocation failure or heavy swap on the inference host). A model outside the table gets no override, and a caller-supplied override always wins.seedCatalog(packages/hub-client/src/seed.ts) right alongsidecapabilitiesForDeployment, so a seeded Ollama offering'squirkscolumn actually carries it --@intx/hub-apialready persistsquirksonmodel_offeringand the platform's ownInferenceSourceresolution already spreads a non-nullquirkscolumn into the request-build path, so this is the one place workbench needed to close the gap. Scoped to theollamaprovider only; every other provider is untouched.Verification
Confirmed the value actually reaches the built request body, not just the resolved config: a test builds a real request through
createOllamaAdapterwith the resolved quirks and assertsbody.options.num_ctxandbody.max_tokens-- provinggpt-oss:20bcarries131072/32768(not4096) andqwen3.8:27bcarries its own smaller32768ceiling.Test plan
bun testinpackages/inference-catalog(75 tests, incl. newollama-context-defaults.test.ts)bun testinpackages/hub-client(159 tests, incl. new seeding-wiring case inseed.test.ts)tsc --noEmitin both packagesprettier --writeon every changed file