Per-model Ollama context-window and token defaults - #251
Merged
Merged
Conversation
Chat agents served through the Ollama openai-compatible adapter silently fall back to @intx/inference's built-in 4096-token default (and Ollama's own small built-in context window) whenever nothing sets a per-model ceiling. These tests pin the fix: a curated model's quirks resolve its real advertised context window, a smaller-ceiling model gets its own limit rather than the larger model's, a caller override wins, and an unlisted model gets no guessed override at all -- verified against the adapter's actual built request body, not just the resolved config.
@corbits/ollama-adapter's OllamaAdapterConfig mechanism (numCtx, maxOutputTokens, reasoningEffort) had no real data behind it, so every Ollama-served chat agent still ran at the built-in 4096-token default. quirksForDeployment resolves each curated model's advertised native context window at offering-creation time, the same way capabilitiesForDeployment resolves capabilities -- gpt-oss:20b and llama3.1:8b get their real 128K ceiling, Qwen3's 27B/30B-class models get their smaller 32K native ceiling rather than an over-requested num_ctx that risks allocation failure or heavy swap on the inference host. A model outside this table gets no override, and a caller-supplied override always wins.
seedCatalog computed capabilities for a new offering but never its quirks, so nothing ever reached the model_offering row an Ollama chat agent actually launches against. This test seeds Ollama's curated catalog and asserts each offering's POST body carries the per-model quirks quirksForDeployment resolves -- gpt-oss:20b's real 128K ceiling, Qwen3's smaller real ceiling -- rather than an empty quirks column.
ensureCatalogOffering now threads a quirks value through to the offering POST body -- @intx/hub-api already persists it on model_offering.quirks, and the platform's own InferenceSource resolution already spreads a non-null quirks column into the request-build path, so this is the one place workbench needed to populate it. seedCatalog computes it with quirksForDeployment right alongside capabilitiesForDeployment, scoped to the ollama provider only; every other provider's offerings are unaffected, and a model with no vetted default still gets none.
Per-model context-window defaults for chat agents (stop the silent 4096 fallback)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carries the work from #247, which landed on this branch after its parent PR (#236) had already merged to main, leaving it stranded.
Chat agents served by Ollama were silently running at a 4096-token cap:
@intx/inference's adapters fall back tooptions.maxTokens ?? 4096when a caller supplies nothing, and the model catalog seeded no context or token data for any model. Ollama'snum_ctx— the real context window, distinct from the output cap — rides only through the endpoint'soptionspassthrough and was likewise unset.quirksForDeploymentnow resolves those per model, the same way capabilities are already resolved, and catalog seeding passes them through. The rest of the chain was already wired:model_offering.quirksis persisted and spread intoInferenceSource.quirks, which is what the Ollama adapter reads.Defaults follow each model's real advertised ceiling rather than a single number:
gpt-oss:20bandllama3.1:8bget 128K, while the Qwen3 27B/30B-class models get their genuine 32K native ceiling — 131072 there needs YaRN scaling, and an over-largenum_ctxrisks allocation failure or heavy swap on the inference host. Unlisted models get no override rather than a guess, and a caller-supplied value always wins.Scoped strictly to the
ollamaprovider; other providers' offerings are untouched.Tests assert the value reaches the built request body, not just the resolved config —
gpt-oss:20bcarriesoptions: {num_ctx: 131072}andmax_tokens: 32768(explicitly asserted!== 4096), and the catalog seed's POST body carries the per-model quirks. That assertion is the point: a silently-dropped setting is the exact failure mode here.