fix(api): raise the body limit on the push routes, and only those - #3
Merged
Conversation
axum applies a 2 MiB `DefaultBodyLimit` to every route unless a layer says otherwise, and nothing here ever did. That default was therefore the real ceiling on everything a caller can hand this service, and it was invisible: the refusal reads `Failed to buffer the request body: length limit exceeded`, which names neither the limit nor the fact that it belongs to us and not to the vendor behind the graph. A caller spent a week failing against it. Measured from their side on 2026-09-04, from both sides of the wall: pushes of 11,408 / 10,387 / 8,976 records went through, and pushes of 12,000 and 16,096 did not. Divide and 2 MiB is exactly where those cross — their payloads run about 130 to 175 bytes a record depending on the vendor. Worse than the refusal was what it cost them: a size refusal is deterministic, so their retry budget was spent re-sending identical bytes and the work was then parked, silently, for 2,683 coordinates. 8 MiB, as GATE_MAX_PUSH_BODY_BYTES. The ceiling on the ceiling is memory rather than taste: a body limit is a per-request buffer and this service runs with a 512 MiB limit, so four times the old value keeps a large caller comfortably inside it while a burst of ten concurrent pushes still costs under a sixth of the pod. The env override is floored at axum's own default so that a typo cannot make the service refuse bodies it accepted before anybody set the variable. PUSH ROUTES ONLY, applied per route rather than as one layer on the Router. A push is a batch — one request stands for as many items as the caller grouped — and the honest bound on it is memory. A graph declaration, a breaker poke and a console read are documents, none has ever come near 2 MiB, and raising their ceiling would buy nothing while letting anybody hand a 512 MiB pod a body per request. The tests go through the ROUTER rather than reading the knob back, because the knob was never the thing that was wrong: a test asserting `knobs().max_push_body` would have passed just as happily on the day a caller was being refused. Mutation-checked: dropping the layer from a push route fails the case that a 3 MiB push is accepted, and moving it to the whole Router fails the case that a document route still refuses one. No broker is needed and none is reached — the extractor rejects before the handler runs. Claude-Session: https://claude.ai/code/session_01RJHziF1EPc5fH7Kd7WRdgk
`env_u32` accepts anything up to `u32::MAX`, so `GATE_MAX_PUSH_BODY_BYTES` could ask each push to buffer about 4 GiB. The value is floored at axum's default and now capped as well. The floor was reasoned about and the ceiling was not, and the ceiling is the side that matters: the limit is a per-request memory RESERVATION, and the wire body is only one of four copies alive at once — the buffered bytes, the `serde_json::Value` they parse into, the copy the envelope is built on and the body sent to the broker. Nothing in front of the router limits concurrency, so the multiplier is the number of callers, not a constant. Claude-Session: https://claude.ai/code/session_012K8u7BEJyd6nDNMCQAgH3z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Il difetto
axum applica un
DefaultBodyLimitdi 2 MiB a ogni rotta se nessun layer dice altro, e qui non lo diceva nessuno. Quel default era quindi il vero tetto su tutto ciò che un chiamante può consegnare a questo servizio, ed era invisibile: il rifiuto diceche non nomina né il limite né il fatto che è nostro e non del vendor dietro il graph.
La misura
Un chiamante ci ha sbattuto contro per una settimana. Misurato dal suo lato il 04/09/2026, dai due lati del muro:
Dividendo, 2 MiB è esattamente dove quei numeri si incrociano: i loro payload stanno fra 130 e 175 byte per record a seconda del vendor.
Peggio del rifiuto è stato il costo. Un rifiuto per dimensione è deterministico, quindi il loro retry budget è stato speso a rispedire byte identici, e poi il lavoro è stato parcheggiato — in silenzio — per 2.683 coordinate.
Il cambio
8 MiB, configurabile con
GATE_MAX_PUSH_BODY_BYTES.Il tetto sul tetto è la memoria, non il gusto: un body limit è un buffer per richiesta e questo servizio gira con un limite di 512 MiB. Quattro volte il valore vecchio tiene un chiamante grosso comodamente dentro, e una raffica di dieci push concorrenti costa comunque meno di un sesto del pod.
L'override da environment è floored al default di axum, così un errore di battitura non può far rifiutare al servizio body che accettava prima che qualcuno toccasse la variabile.
Solo le rotte di push, e per rotta
Applicato sulle quattro rotte di push invece che con un
.layer()sul Router intero.Un push è un batch: una richiesta vale per quanti item il chiamante è riuscito a raggruppare, e il limite onesto è la memoria. Una dichiarazione di graph, un poke al breaker e una lettura di console sono documenti, nessuno si è mai avvicinato a 2 MiB, e alzare il loro tetto non comprerebbe niente mentre lascerebbe consegnare a un pod da 512 MiB un body per richiesta che nessuna dichiarazione ha mai richiesto.
Test
Passano dal Router, non rileggono il knob — ed è il punto: il knob non è mai stato la cosa sbagliata, e un test su
knobs().max_push_bodysarebbe passato allegramente anche il giorno in cui un chiamante veniva rifiutato.Mutation-check: togliere il layer da una rotta di push fa cadere il primo, spostarlo sul Router intero fa cadere il terzo.
Nessun broker serve e nessuno viene raggiunto: l'extractor rifiuta prima che l'handler parta.
cargo test,cargo clippy --all-targets --all-featuresecargo fmt --checkverdi.https://claude.ai/code/session_01RJHziF1EPc5fH7Kd7WRdgk