Skip to content

fix/codex broker session cache - #234

Merged
merlerm merged 1 commit into
mainfrom
fix/codex-broker-session-cache
Sep 19, 2026
Merged

merlerm merged 1 commit into
mainfrom
fix/codex-broker-session-cache

Conversation

@merlerm

@merlerm merlerm commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Codex requests through the isolated inference broker lost the session-id header used by the ChatGPT endpoint for cache affinity. This caused repeated conversation history to be charged as uncached input and exhausted experiment budgets prematurely.

Preserve a single UUID-shaped session-id for the ChatGPT transport. Reject malformed or duplicate identifiers before contacting upstream, and retain the existing fixed destination, host credentials, and request/tool restrictions. Add regression tests for session forwarding, hostile headers, invalid identifiers, and the separate API-key transport.

Validation:

  • 54 broker tests passed, including all existing adversarial cases; formatting, import ordering, and targeted mypy checks passed.
  • Repeated synthetic prompts through the broker: 0% cache reuse without the header versus 98.7% with it, reproduced for Astra and Sol.
  • A matched ClutteredRetrieval2D seed-42 Astra run at the same $20 budget improved cache reuse from 28.95% to 97.86% and generated 57,126 output tokens versus 19,742 before the fix.

@merlerm
merlerm requested a review from tomsilver September 19, 2026 21:14
@merlerm
merlerm enabled auto-merge (squash) September 19, 2026 21:14
@merlerm
merlerm merged commit f4e31ac into main Sep 19, 2026
4 checks passed
@merlerm
merlerm deleted the fix/codex-broker-session-cache branch September 19, 2026 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants