fix(models): cap local context for responsive chats - #6296
Open
RafaelPupio wants to merge 2 commits into
Open
RafaelPupio wants to merge 2 commits into
RafaelPupio wants to merge 2 commits into
Conversation
3 tasks
Author
|
CI status note: the three |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
ODYSSEUS_LOCAL_CONTEXT_CAPas an optional local-only upper bound for model context windows so large local allocations do not make interactive chats appear hung during prompt prefill. Docker Compose defaults the ceiling to 8192 tokens across base, NVIDIA, and AMD configurations. Configured API and proxy endpoints retain their advertised context. This is complementary to runtime allocation discovery in #5800 and does not replace it.Target branch
dev, notmain. All PRs land indev;mainis curated by the maintainer at each release. If your PR is onmainby accident, click "Edit" on this PR and change the base.Linked Issue
Part of #5193
Type of Change
Checklist
devdocker compose uporuvicorn app:app) and verified the change works end-to-end. Type-checks and unit tests are not enough.How to Test
pytest tests/test_model_context.py tests/test_gpu_compose_standalone.py -q. The submitted branch passes all 57 focused tests.docker compose config -q,docker compose -f docker-compose.gpu-nvidia.yml config -q, anddocker compose -f docker-compose.gpu-amd.yml config -q.ODYSSEUS_LOCAL_CONTEXT_CAP=8192, create a session against a local model with a larger advertised window, then request/api/session/{id}/context_info. The response should report 8192. Configured API and proxy endpoints must remain unchanged.Runtime validation completed using the current branch source in a disposable container bound to
127.0.0.1:17001; the active local deployment on port 7001 was untouched.GET /api/healthreturned 200. A temporary session against local Ollama modelqwen3-coder:30breturned{"context_length":8192}; application logs showed the known 131072-token window being capped to 8192.Additional automated verification completed with Python compilation, all three Compose validations, and 5,914 broader-suite passes with one independently reproduced upstream browser smoke-test failure deselected. The upstream CI run at the exact base commit was green.
Visual / UI changes — REQUIRED if you touched anything that renders
Not applicable. This PR changes model-context policy, Compose configuration, documentation, and tests only. No rendered UI files changed.
Screenshots / clips
Not applicable.