feat(openai): support manually placed prompt cache breakpoints in the Responses API#2171
Open
dgrijalva wants to merge 1 commit into
Open
feat(openai): support manually placed prompt cache breakpoints in the Responses API#2171dgrijalva wants to merge 1 commit into
dgrijalva wants to merge 1 commit into
Conversation
… Responses API OpenAI added explicit prompt-cache breakpoints for GPT-5.6+: a prompt_cache_breakpoint field on input_text/input_image/input_file content parts, and a request-level prompt_cache_options object. A breakpoint is placed at a chosen position in the history by setting the documented OPENAI_PROMPT_CACHE_BREAKPOINT_KEY in a text, image, or document block's additional_params; both message conversion paths extract the marker onto the corresponding content part. Request-level prompt_cache_options passes through additional_params, which previously dropped it as an unknown AdditionalParameters field.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2170
Implementation
PromptCacheBreakpoint/PromptCacheBreakpointMode::ExplicitandPromptCacheOptions/PromptCacheModewire types;prompt_cache_breakpointonInputText/InputImage/InputFile(skipped whenNone, so existing wire output is unchanged);prompt_cache_optionsonAdditionalParameters(previously silently dropped as an unknown key).pubconstOPENAI_PROMPT_CACHE_BREAKPOINT_KEY(with doctest); the marker is extracted from blockadditional_paramsin both conversion paths (TryFrom<crate::completion::Message> for Vec<InputItem>andresponses_user_content) across all five input-block arms. Invalid payloads fail conversion. Assistant content is not scanned — OpenAI documents breakpoints on input blocks only.Testing
gpt-5.6: the marked ~6.7k-token block produced a real cache hit (cached_tokens: 6739of 6766 in the recorded usage) and the fixture assertion pins the request body (breakpoint on the marked block only;prompt_cache_options/prompt_cache_keyat top level). Full OpenAI cassette suite replays green. (One recorder quirk:prompt_cache_keyvalues get redacted as identifiers, so the fixture assertion checks presence, not value.)cargo clippy --all-features --all-targetsandcargo fmt -- --checkclean.Note on API design: the
additional_paramscarrier and public key const are our best guess at the rig-native shape (mirroring the Anthropic documenttitle/citationspattern). Happy to rework if you'd prefer a different mechanism.Related: #2168 (Anthropic counterpart)