Enforce Cloud key budgets and rate limits; add usage export and budget reads - #124
Merged
Merged
Conversation
Route resolution may now carry the calling key's identity and limits
(`key: {id, subject, labels}`, `limits: {monthly_budget_usd, rate_per_min,
burst}`). They are parsed strictly; absent means no key limits, malformed
makes the route unavailable (fail closed).
Ingress, after the plan rate bucket:
- subject token bucket "{caller}|{subject}" -> 429 key_rate_limit
- atomic PostgreSQL budget reservation per (tenant, subject, UTC month)
-> 402 key_budget_exhausted; store outage -> 503 key_budget_unavailable
- the reservation is settled synchronously in _finish on every exit path
(success, error, exception, capacity rejection, stream end/disconnect)
with the actual cost; dead holders expire after 15 minutes
- routing_summary records key_id and subject
Internal API: /internal/usage gains until_ts (exclusive), group_by
(key|model|route|day|hour) and watermark_ts; new /internal/usage/export
(id cursor, held back by LEDGER_WATERMARK_LAG_S so the cursor never skips
a late-committing lower id) and /internal/budgets.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A synchronous settle blocked every request on the worker for up to its 2-second store timeout. Settles now run on a small thread pool; until one lands its reservation still counts, which only errs toward rejecting, and a failed settle still expires with the reservation TTL. Shutdown drains them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ed settles A stream the client closed before its usage chunk, or an unpriced model, made a successful call free against the key's budget. Such calls now book the reservation amount; failed calls still book only what they report. A settle that hits a store outage is retried with backoff instead of losing the cost. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (6)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Control-plane route resolution now carries each application key's subject (its key group, or the first key of its rotation chain) and limits. The ingress enforces them on every SaaS call:
key_rate_limit.key_budget_exhausted. The actual cost is booked when the call ends, on every exit path, off the event loop and retried on store errors. A successful call of unknown cost books the reservation; expired reservations are swept.key/limitsdata fails closed (503route_unavailable). Absent fields keep today's behaviour for older control planes.Reconciliation reads for the Cloud:
/internal/usagegainsuntil_ts,group_by=key|model|route|day|hourandwatermark_ts./internal/usage/export(id cursor, rows held back untilLEDGER_WATERMARK_LAG_Sso late commits on other replicas are not skipped)./internal/budgets.New env:
CLOUD_BUDGET_RESERVATION_USD(0.05),LEDGER_WATERMARK_LAG_S(120). Schema changes are additive (CREATE TABLE/ALTER … IF NOT EXISTS).Contract:
docs/MACHINE-API.mdin unhardcoded-cloud. Companion Cloud PR follows.Validation: full suite 1197 passed, 2 skipped (baseline 1142). 55 new tests in
tests/test_key_subject_limits.py. The Cloud two-process bridge test exercises group budgets across key rotation and key rate limits against this branch.Overlaps with open #106 (
auth_proxy.py,host_store.py); whichever merges second needs a rebase.🤖 Generated with Claude Code
Summary by CodeRabbit