fix(cost): record cache reads and bill them at the cached rate - #38
Merged
Conversation
Two bugs, one root: cache metrics were captured and then dropped. 1. cached_tokens never reached the persisted execution. capture_response read metadata.cached_tokens into the context's generic metadata bag, but Pipeline::Context had no first-class accessor for it and build_result never copied it into the Result. Since the reliability middleware passes context.output (the Result) to AttemptTracker#complete_attempt, every attempt recorded cached_tokens: 0, which aggregated to 0 in the executions column. Across ~1M production rows the value was 0 every time, while the provider was in fact reporting substantial cache hits (a live probe returned cached=6016 of input=7112). Context now carries cached_tokens/cache_creation_tokens as real accessors and build_result passes them through. 2. Cache reads were billed at the full input rate. calculate_costs charged (input_tokens * input_price) unconditionally. That is correct on Anthropic, whose input_tokens is reported NET of cache_read_input_tokens — the cache is additive there and extra_token_costs already prices it, so that path is deliberately untouched. It is wrong on OpenAI and Gemini, whose prompt_tokens/promptTokenCount INCLUDE the cache reads. Those tokens were charged at 4x their real price, overstating spend ~2.7x on the workload caching exists for (a long stable system prompt replayed every turn): 7112 input with 6016 cached costs $0.0052, not the $0.0142 that was being recorded. Worse, once the provider also reports a cache_read cost component, extra_token_costs would add the same reads a second time — a real double-charge, latent only because RubyLLM 1.14's Message has no #cost. input_cost now splits uncached/cached for the include-providers and sets context[:cache_read_priced] so extra_token_costs skips those reads. It falls back to the full rate whenever the split can't be made safely (no cache reported, unknown cached price, non-standard pricing shape), so no model is ever under-billed. The two existing prompt-caching examples asserted only that a Result came back — their own comments noted they could not see the cache tokens. They now assert the values that bug 1 was swallowing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two bugs, one root: cache metrics were captured and then dropped.
cached_tokensnever reached the persisted execution.capture_responsereadmetadata.cached_tokensinto the context's generic metadata bag, butPipeline::Contexthad no first-class accessor for it andbuild_resultnever copied it into the Result — so every attempt recordedcached_tokens: 0. Context now carriescached_tokens/cache_creation_tokensas real accessors andbuild_resultpasses them through.Cache reads were billed at the full input rate.
calculate_costschargedinput_tokens * input_priceunconditionally. Correct on Anthropic (input is reported net of cache reads — that path is untouched), wrong on OpenAI/Gemini whoseprompt_tokens/promptTokenCountinclude cache reads. Cached tokens were charged at ~4x their real price, overstating spend ~2.7x on cached workloads (7112 input with 6016 cached costs $0.0052, not the $0.0142 being recorded). Cache reads are now split out and billed at the provider's cached rate.Test plan
spec/agents/cached_token_pricing_spec.rbcovering per-provider billing (Anthropic net vs OpenAI/Gemini gross) and cached-token propagation into the Resultspec/agents/prompt_caching_spec.rbbundle exec rspec+standardrbgreen🤖 Generated with Claude Code