fix(submit): refuse AT the body ceiling, and pin the contract #585 published - #602
fix(submit): refuse AT the body ceiling, and pin the contract #585 published#602ZacxDev wants to merge 2 commits into
Conversation
…blished Follow-up to #585, carrying two rounds of audit findings. The headline is an off-by-one that a test asserted was unobservable. R1-F1 — the ceiling guard was off by one, and its "unreachable" claim was false. `len(body) > MaxSubmitBodyBytes` let a body of EXACTLY 10485760 upload. submit_ceiling_value_test.go asserted in prose that `>` vs `>=` could not be observed — "the envelope is 19 bytes and base64 steps by 4 … there is no input that distinguishes the two operators". That measured ONE provenance and spoke for all four. Measured through the real SubmitBodySize: provenance envelope mod 4 zipLen hitting the ceiling exactly none 19 3 unreachable commit only 77 1 unreachable commit + Dirty=false 97 1 unreachable commit + Dirty=true 96 0 7864246 base64 output is always a multiple of 4, so only an envelope that is itself 0 mod 4 can land on the ceiling — and that is exactly what `--allow-dirty` sends. A 7,864,246-byte zip therefore produced a 10,485,760-byte body that the guard waved through. Changed to `>=`. The comment and README say plainly that this is a choice under UNCERTAINTY, not a settled fact: Next.js reads `bytesRead > bodySizeLimit` (would accept it) while an end-to-end measurement answered 413 at exactly the cap. Neither can be re-measured here. The asymmetry decides it — refusing one byte early costs a documented flag, accepting one byte too many costs the whole upload and an error naming nothing about size. R1-F2 — MaxSubmitBodyBytes shipped with no godoc; ErrBundleTooLarge wore its evidence. A missing blank line glued the two blocks, so `go doc MaxSubmitBodyBytes` printed the bare const line. Each declaration now carries its own doc, verified with `go doc`; the orphaned derivation block moved onto SubmitBodySize, which it was always about. R1-F3 — the published exit-code contract had no guard. exitcodes_doc.go publishes that the refusal exits 1 and that "a script branching on 2 for that case must branch on 1", and appapi called it "now assertable". It was assertable and not asserted: exit 1 was reached only by falling through exitCode's default. Added the real-command test plus two ledger rows and their floor entries. R0-F1 — a comment named a guard that does not enforce what it claimed. The constant said pkgzip/caps_claim_test.go is "the reason this constant lives here". Measured with both controls: that test bans four literal phrases from pkgzip.go, and a fully server-attributed constant using none of them PASSES. The placement is architectural and held by review; the comment now says so. R0-F2 — three user-facing strings contradicted shipped behaviour, all saying the CLI does not know any server ceiling while it vendors one and refuses on it. The one with teeth is printSubmitSizeDiagnosis: it runs only when the vendored ceiling did NOT fire, i.e. precisely when that number may be too high, and it told the author no local cap existed. It now names the ceiling and the direction it can be wrong in. pkgzip's wording is narrowed to the compressed ARCHIVE, which genuinely is unbounded here. R1-F4/F5/F6 — `Submitting …` went to stdout before a refusal that went to stderr, so the CLI announced an upload it then declined; the spinner is now skipped for a body it can predict will be refused (appapi still makes the decision — this only chooses whether to spin). The refusal's trailing colon promised "--allow-oversize submits anyway" and printed --package-only, which never submits; both commands are now offered under their own sentence. The ceiling-staleness regex exempted spaced "10 MiB" unconditionally while leaving unspaced "10MiB" reportable — dropped, because MiB spellings in those files are pkgzip's caps, not this ceiling. Mutation results, each watched to fail for its own reason: `>=` -> `>` RED TestSubmitBodyExactlyAtCeilingIsRefused ceiling shifted one quantum RED same test's negative control refusal tagged ErrBadRequest RED TestOversizeBundleExitsGeneric (SURVIVES the full suite at origin/main) code-2 ledger row deleted RED the floor each of 4 ledger phrases reworded RED independently Red at origin/main, green here: TestSubmitBodyExactlyAtCeilingIsRefused, the new stdout assertion in TestAppSubmitOversizeEndToEnd, and both halves of TestExportedSubmitCeilingDeclsCarryTheirOwnDoc. A first draft of the code-2 ledger phrases used the bare "must branch on", which another bullet in the same code already contains — rewording the claim out of the doc left the row GREEN. Measured, then respelled with the 1/2 the claim is about. go vet ./... clean; go test ./... 21/21 packages, 0 failures, full clone.
Both are in this PR's own new test files; CI's `lint` job found them and it was right. Local golangci-lint now reports 0 issues. ST1008 — realOversizeRefusal returned (error, string, string). Reordered to (string, string, error). SA1019 — parser.ParseDir is deprecated as of Go 1.25. The godoc guard now reads the package directory itself and runs parser.ParseFile over each non-test .go file, taking each declaration's doc comment straight off the AST. go/doc and go/packages are both avoidable here: the only thing needed is the comment attached to a declaration, and that is on the AST already. The positive control moved with it — it now asserts a non-zero parsed-FILE count as well as the three declarations being found, so a walk that matched nothing cannot pass vacuously. Re-verified after the rewrite, not assumed: the guard is still RED at origin/main, with both halves firing for their own reasons and naming MaxSubmitBodyBytes as the thief of ErrBundleTooLarge's doc block. go vet clean; go test ./... 21/21 packages, 0 failures; golangci-lint 0 issues.
|
Heads-up from the session that just merged #601 (
Cheap re-check before merging, since a clean git merge is not a clean merge: git fetch origin
git merge-tree --write-tree origin/main <602-head>; echo "rc=$?" # non-zero = conflict
# then materialise it and run the suite, and diff for silently-dropped rows:
diff <(git show origin/main:internal/cmd/exitcodes_claims_test.go) internal/cmd/exitcodes_claims_test.goThe last one matters most here: No action needed from #601's side — flagging it so the second merge doesn't repeat that. |
Follow-up to #585, carrying findings from a round-0 and a round-1 audit of it. Base is
main@7467c62. No file overlap with #596 or #600.The headline is an off-by-one that a shipped test asserted was unobservable.
1. The ceiling guard was off by one, and its "unreachable" claim was false
internal/appapi/appblocks.gorefused onlen(body) > MaxSubmitBodyBytes, so a body of exactly 10485760 uploaded in full.submit_ceiling_value_test.gosaid in prose that this could not matter — "the envelope is 19 bytes and base64 steps by 4, so body sizes go … 10485759, 10485763 — nothing lands on 10485760 at all. There is no input that distinguishes the two operators." That measured one provenance and spoke for all four. Measured through the realSubmitBodySize:Dirty:falseDirty:truebase64 output is always a multiple of 4, so only an envelope that is itself ≡ 0 (mod 4) can land on the ceiling — and that is exactly what
civitai app submit --allow-dirtysends. A 7,864,246-byte zip therefore produced a 10,485,760-byte body the guard waved through: the precise body #585 exists to stop.Changed to
>=.The reasoning is an asymmetry, not a claim about the server
There is a real unresolved contradiction here and the code, the README and this PR all say so rather than inventing a mechanism:
bytesRead > bodySizeLimit, which would accept a body of exactly 10485760.10485759 -> 401(the body reached auth),10485760 -> 413,12000000 -> 400.One of those is wrong. Neither can be re-measured from a unit test — it needs credentials and a real submit — so nobody should write down which. The decision does not depend on resolving it: refusing one byte early costs an author a flag they are already told about (
--allow-oversize), while accepting one byte too many costs the whole ~10 MB upload and returns an error naming nothing about size. The consequences are wildly unequal, so the guard takes the cheap side.Pinned by
TestSubmitBodyExactlyAtCeilingIsRefused, which drives the realSubmitVersionat the one input that can tell the operators apart, with a negative control one base64 quantum below (so the assertion means>=and not "refuses everything") and two CONTROL checks that the fixture really sits on the boundary and that the provenance survivessanitised().2.
MaxSubmitBodyBytesshipped with no godocA missing blank line glued its doc block to
ErrBundleTooLarge's. Measured before:The constant three published surfaces quote literally had no godoc; the sentinel wore the constant's evidence. Each declaration now carries its own doc (verified with
go doc, not by reading), and the derivation block that documented nothing moved ontoSubmitBodySize, which it was always about.Added
TestExportedSubmitCeilingDeclsCarryTheirOwnDoc, which asserts each doc begins with its own identifier rather than merely being non-empty — non-emptiness cannot see the failure that happened, sinceErrBundleTooLarge's doc was long, detailed, and about something else.3. The published exit-code contract had no guard
exitcodes_doc.gopublishes that the refusal exits1, and — under code 2 — that "a script branching on2for that case must branch on1", a documented breaking change.appblocks.gocalled the code "now assertable". It was assertable and not asserted: no ledger row, no floor entry, no reference toErrBundleTooLargeundercmd/civitai/. Behaviour was correct only by falling throughexitCode'sdefault.Added
TestOversizeBundleExitsGeneric(the real error from the real command, not a hand-tagged fixture) +TestOversizeSentinelIsNotAnAPIKind, mirroringTestDirtyWorkTreeExitsGeneric, plus twocontractClaimrows and theirexitCodeClaimsFloorentries.4. A comment named a guard that does not enforce what it claimed
appblocks.gosaidpkgzip/caps_claim_test.gois "exactly the claim … that test exists to keep OUT of that file, and the reason this constant lives here instead." Measured with both controls:🔴 This IS the server's number+ the whole evidence chain) using none of the four banned phrases: PASSES.caps mirror the server: FAILS with the test's own message.So the guard bans four literal spellings; it does not force the placement. The placement is still correct on architectural grounds (
appapibuilds the body, so the body's limit belongs besideSubmitBodySize) — the constant did not move, only the stated reason.claudedocs/decisions/31was already accurate on this and is untouched.5. Three user-facing strings contradicted shipped behaviour
All three said the CLI does not know any server ceiling, while it vendors 10485760 and refuses on it.
The one with teeth is
printSubmitSizeDiagnosis: it runs only when the vendored ceiling did not fire, which is precisely the situation where that number may be too high. If the platform ever lowersproxyClientMaxBodySize, a body under the vendored value clears the guard, is truncated, returns400: Invalid JSON— #423 exactly — and the diagnosis told the author no local cap existed, pointing them away from the only number that could explain it. It now names the ceiling and the direction it can be wrong in.pkgzip's wording is narrowed to the compressed archive, which genuinely has no known platform ceiling — as distinct from the request body, which does.caps_claim_test.gore-run and green; no banned phrase introduced.6. Three cheap ones
Submitting …goes to stdout and the refusal to stderr, so the CLI announced an upload it then declined. The spinner is now skipped for a body it can predict will be refused. It predicts, it does not decide:appapi.SubmitVersionremains the only place the refusal is made, so a disagreement could only show or withhold a spinner.app_submit_oversize_test.goread only stderr and was blind to this; it now asserts stdout, with a positive control that it is reading a non-empty stream.--allow-oversizesubmits anyway" and then printedcivitai app submit --package-only, which never submits. Both commands are now offered under their own sentence.10 ?MiB|10 ?MBwere listed as suspicious then exempted via!strings.HasPrefix(m, "10 "), so spaced forms could never be reported while unspaced10MiBcould. Dropped rather than repaired: enumerated over both files,10 MiBappears in each and in both it is pkgzip's per-file cap, not this ceiling (alongside2 MiB,50 MiB,200 MiB,32 MB). A rule reporting them would be wrong; one exempting them by prefix only looks like a rule.Mutation results
Each mutant applied in a
cp -acopy withrm -f <copy>/.gitfirst, and watched to fail for its own reason:>=→>TestSubmitBodyExactlyAtCeilingIsRefused: "a submit body of EXACTLY 10485760 bytes was uploaded"civitai.ErrBadRequestTestOversizeBundleExitsGeneric: "exitCode(...) = 2, want 1". SURVIVES the full suite atmain(rc=0, 21/21 packages)Red at
origin/main, green here:TestSubmitBodyExactlyAtCeilingIsRefused; the new stdout assertion inTestAppSubmitOversizeEndToEnd; both halves ofTestExportedSubmitCeilingDeclsCarryTheirOwnDoc(which namesMaxSubmitBodyBytesas the thief correctly).TestOversizeBundleExitsGenericpasses atmainunmutated — it is an invariant guard on the published contract, not a regression test, and is labelled as such. The mutation above is what shows it has teeth.One finding this PR made against itself
A first draft of the code-2 ledger phrases used the bare
"must branch on". That is walkable: code 2's--jsonbullet already contains "must branch on the exit code first", so rewording the migration notice to "Scripts may need updating." left the row GREEN. Caught by mutating, then respelled with the1/2the claim is actually about, and re-mutated to confirm both phrases now kill independently.Gate
Makefiletargetci=tidy vet test build. All four run:go vet ./...— cleango test ./... -count=1— rc=0, 21/21 packages ok, 0 failuresgo build ./...— okgo mod tidy— no changeBoth tiers, stated separately. The above is the full-clone tier. The depth-1 tier (
scripts/ci-shallow.sh, which is also what CI'sbuild-testjob runs —actions/checkout@v4defaults tofetch-depth: 1) was run too:ok=21/21 failing-tests=0 failing-packages=0 timeouts=0 filtered-empty=0.golangci-lint2.13.2 locally: 0 issues. CI'slintjob (2.12.2) initially caught two staticcheck findings in this PR's own new test files —ST1008(error not last) andSA1019(parser.ParseDirdeprecated in Go 1.25). Both fixed in the second commit; the godoc guard was re-verified RED atmainafter the rewrite rather than assumed. All 13 checks now green.Not verified
proxyClientMaxBodySizemoves). It makes the staleness legible —printSubmitSizeDiagnosisnow names the vendored number and the direction it can be wrong in — but nothing here detects the change.