Skip to content

fix(models): cap local context for responsive chats - #6296

Open
RafaelPupio wants to merge 2 commits into
odysseus-dev:devfrom
RafaelPupio:fix/local-context-cap-mcp-compat
Open

RafaelPupio wants to merge 2 commits into
odysseus-dev:devfrom
RafaelPupio:fix/local-context-cap-mcp-compat

Conversation

@RafaelPupio

@RafaelPupio RafaelPupio commented Sep 14, 2026

Copy link
Copy Markdown

Summary

Adds ODYSSEUS_LOCAL_CONTEXT_CAP as an optional local-only upper bound for model context windows so large local allocations do not make interactive chats appear hung during prompt prefill. Docker Compose defaults the ceiling to 8192 tokens across base, NVIDIA, and AMD configurations. Configured API and proxy endpoints retain their advertised context. This is complementary to runtime allocation discovery in #5800 and does not replace it.

Target branch

  • This PR targets dev, not main. All PRs land in dev; main is curated by the maintainer at each release. If your PR is on main by accident, click "Edit" on this PR and change the base.

Linked Issue

Part of #5193

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)
  • New feature (non-breaking — adds new behaviour)
  • Breaking change (changes or removes existing behaviour)
  • Refactor / cleanup (behaviour unchanged)
  • Documentation only
  • CI / tooling / configuration

Checklist

  • I searched open issues and open PRs — this is not a duplicate.
  • This PR targets dev
  • My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
  • I actually ran the app (docker compose up or uvicorn app:app) and verified the change works end-to-end. Type-checks and unit tests are not enough.
  • I did not run the app/runtime validation and stated that gap in How to Test. Leave this unchecked when the app-run box above is checked.

How to Test

  1. Run pytest tests/test_model_context.py tests/test_gpu_compose_standalone.py -q. The submitted branch passes all 57 focused tests.
  2. Run docker compose config -q, docker compose -f docker-compose.gpu-nvidia.yml config -q, and docker compose -f docker-compose.gpu-amd.yml config -q.
  3. Set ODYSSEUS_LOCAL_CONTEXT_CAP=8192, create a session against a local model with a larger advertised window, then request /api/session/{id}/context_info. The response should report 8192. Configured API and proxy endpoints must remain unchanged.

Runtime validation completed using the current branch source in a disposable container bound to 127.0.0.1:17001; the active local deployment on port 7001 was untouched. GET /api/health returned 200. A temporary session against local Ollama model qwen3-coder:30b returned {"context_length":8192}; application logs showed the known 131072-token window being capped to 8192.

Additional automated verification completed with Python compilation, all three Compose validations, and 5,914 broader-suite passes with one independently reproduced upstream browser smoke-test failure deselected. The upstream CI run at the exact base commit was green.

Visual / UI changes — REQUIRED if you touched anything that renders

Not applicable. This PR changes model-context policy, Compose configuration, documentation, and tests only. No rendered UI files changed.

Screenshots / clips

Not applicable.

@github-actions github-actions Bot added the needs runtime validation Runtime validation not attested — tick the app-run box after running it, or state the gap label Sep 14, 2026
@github-actions github-actions Bot added ready for review Description complete — ready for maintainer review and removed needs runtime validation Runtime validation not attested — tick the app-run box after running it, or state the gap labels Sep 14, 2026
@RafaelPupio

Copy link
Copy Markdown
Author

CI status note: the three pull_request_target validation checks pass. The full CI, workflow-security, container-scan, CodeQL, secret-scan, and dependency-review runs are currently marked action_required; GitHub is waiting for an upstream maintainer to approve workflows from this first-time fork contribution. Local focused, broad-suite, Compose, compilation, and disposable runtime evidence is recorded in the PR description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant