Skip to content

test(retry): pin the 429 header-before-message ordering the contract publishes - #601

Merged
ZacxDev merged 2 commits into
mainfrom
test/pin-429-header-before-message
Sep 14, 2026
Merged

ZacxDev merged 2 commits into
mainfrom
test/pin-429-header-before-message

Conversation

@ZacxDev

@ZacxDev ZacxDev commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

What

internal/cmd/exitcodes_doc.go and README publish a 🔴 bullet on exit code 6:

THE HEADER IS CONSULTED BEFORE THE MESSAGE, so a cap-worded 429 that carries Retry-After exits 5, NOT 2.

Nothing in the repo guarded it. This adds the guard and corrects the claim row that said none was possible.

The gap, measured

Consult the message first — return terminal on cap wording, whatever the header — and the published bullet inverts from 5 to 2 while the full suite stays green. retry_test.go's two 429 neighbours each hold one half of the input fixed:

existing test body Retry-After can see the ordering?
TestRead429WithRetryAfterExhaustsGeneric empty present no
TestRead429WithoutRetryAfterIsTerminalWithCursorHint cap-worded absent no

It takes both at once, which nothing sent.

The guard

TestCapWorded429WithRetryAfterIsRetriedNotReclassified (pkg/civitai) serves a cap-worded body (isDeepPagingCap matches) plus Retry-After: 0 on every response, and asserts:

  • the header decided — readMaxAttempts (4) requests, not 1;
  • the classification by errors.Is per AGENTS.md item 7, since the sentinels carry no visible text: ErrNetwork (exit 5), not ErrBadRequest (exit 2), not ErrRateLimited (exit 6);
  • the exit-5 message, not readError's. The cap wording still reaches the string via snippet(raw), so the discriminating absences are rate limited (429) and --cursorreadError's own text, absent from the body.

Verification matrix

tree result
HEAD make ci rc=0, 21 ok / 0 FAIL; -race clean; gofmt clean
mutant (message before header) 20 ok packages, exactly one --- FAIL — this test, failing on all five of its own assertions incl. both errors.Is ones

The mutant was written down as a diff and applied from it; the anchor was asserted to match exactly once first; retry.go was backed up with cp -a and restored by copy (never git checkout --), and git diff confirms it is byte-identical to HEAD. That single-FAIL reading confirms round 4's 21 ok / 0 FAIL from the other side: nothing that existed before could see the swap.

Linter: golangci-lint run ./...0 issues, and it was validated with a negative control (a deliberate unused variable → rc=1, 1 issue). Scope caveat: that was v2.13.2 via nix-shell; CI pins v2.12.2, so the zero is a claim about 2.13.2, not about the pinned job.

The pinnedBy correction

The claim row read:

pinnedBy: "nothing local — the assumption is about the server. That is the point of publishing it"

That conflated two claims. The reachability — whether the server ever attaches Retry-After to a cap 429 — is genuinely unguardable and is exactly why the bullet is published. The CLI's own ordering is local, trivially observable, and was unguarded. As written the field told the next maintainer no guard was possible, which is the description-reads-as-coverage failure that table exists to stop.

The published bullets are unchanged: "that assumption is vendored and has no local guard" refers to the server half, which is still true. No generated region is touched.

Residual, not fixed here

pinnedBy is free prose — exitcodes_claims_test.go only prints it in a failure message, so nothing checks that a named test exists. The sibling ledger in app_newest_submission_test.go does check this (testFuncBody(...)"the ledger names coverage that does not exist"), but several rows here name tests in other packages, so an existence check is a cross-package change and out of scope for this PR.

Closes the rank-22 item in claudedocs/handoff-external-issue-513-numeric-username.md (cli#591 round-4 F2).

…publishes

`exitcodes_doc.go` and README publish a 🔴 bullet: THE HEADER IS CONSULTED
BEFORE THE MESSAGE, so a cap-worded 429 carrying `Retry-After` exits 5, NOT 2.
Nothing guarded it. Consulting the message first — return terminal on cap
wording, whatever the header — inverts that bullet from 5 to 2 with the whole
suite green, because retry_test.go's two 429 neighbours each hold one half of
the input fixed: TestRead429WithRetryAfterExhaustsGeneric sends an EMPTY body,
TestRead429WithoutRetryAfterIsTerminalWithCursorHint sends no header. Neither
can distinguish the two orderings; it takes both at once.

TestCapWorded429WithRetryAfterIsRetriedNotReclassified sends both: a cap-worded
body (isDeepPagingCap matches) plus `Retry-After` on every response. It asserts
the header decided (4 requests, not 1) and, per AGENTS.md item 7, asserts the
classification by errors.Is — ErrNetwork (5), not ErrBadRequest (2), not
ErrRateLimited (6) — since the sentinels carry no visible text.

Matrix: green at HEAD (21 ok / 0 FAIL); under the mutant, 20 ok packages and
exactly one `--- FAIL`, this test. So every pre-existing test stays green under
it, confirming round 4's 21 ok / 0 FAIL reading from the other side.

Also corrects the claim row's `pinnedBy`, which read "nothing local — the
assumption is about the server". That conflated two claims: the REACHABILITY
(does the server ever attach Retry-After to a cap 429) is genuinely unguardable
and is why the bullet is published; the CLI's own ORDERING is local and now
pinned. As written the field told the next maintainer no guard was possible —
the description-reads-as-coverage failure that table exists to stop.

The published bullets are unchanged: "that assumption is vendored and has no
local guard" refers to the server half, which is still true.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JQ4B1VHew33RNX5Ck7ELS
@ZacxDev

ZacxDev commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

CI settled: 13/13 checks terminal, all SUCCESS, MERGEABLE/CLEAN.

That closes the linter version caveat in the body from the other side — the lint job ran the pinned v2.12.2 and is green, so the zero no longer rests on my local v2.13.2 reading. (Per AGENTS.md item 11, lint reports but does not gate, which is why the local run happened at all.)

…of only pinning it

Round 0 of the audit on this PR returned `requirement questioned` with two 🟡
and two 🟢. All four taken.

F1 (🟡, payload) — the guard pinned the SENTINEL; the requirement of record and
the published bullet are about an EXIT CODE. cmd/civitai's
TestCapWorded429WithRetryAfterExitsFiveNotTwo now composes cap-body +
Retry-After -> exitCode() == 5, using the idiom the adjacent 429 guard already
states: "pkg/civitai owns the sentinel; this owns the number a script reads from
`$?`." Nothing composed those two inputs at the rc level before, so the seam was
unowned for the one 429 case the contract marks 🔴. Carries the same
instrument positive control (exitCode must be able to return 2).

F2 (🟡, payload) — the precedence was described in three surfaces and argued in
none, so a maintainer reading the bullet call it "the exact hazard the 2
reclassification exists to prevent" would reasonably invert it, and be blocked
by a test asserting a contract without saying why. retry.go's 429 branch now
argues it: a Retry-After header is a STRUCTURED signal the server sent
deliberately; isDeepPagingCap is a three-phrase substring match over prose its
own doc comment calls "deliberately narrow" and which any proxy or copy-edit can
produce or destroy. Letting the heuristic outrank the structured signal trades
this hazard for a worse one — a genuine throttle refused as a usage error, which
a backoff loop never retries. It also names the four surfaces that move if the
precedence ever changes, and corrects "there is no local guard on it": there are
now two on the ORDERING; the REACHABILITY is the half that stays unguardable.

F3 (🟢, scaffolding) — errors.Is(err, ErrRateLimited) is an INVARIANT guard and
is now labelled one. Measured: it does not fire under the ordering mutant, since
errkind.go attaches exactly one sentinel and the cap matcher still matches, so
ErrRateLimited is unreachable by construction on this input. Kept, not counted
as regression coverage.

F4 (🟢, scaffolding) — dropped the restated 20-ok/1-FAIL matrix from the
`pinnedBy` comment. It was a fourth in-tree copy, hardcoded a package count
nothing checks, and RULES puts the measurement in the commit and the claim in
the comment.

🔴 Correcting a claim in the previous commit message and the PR body: they say
the mutant fails "all five of its own assertions incl. both `errors.Is` ones".
The count 5 is right but the phrasing is wrong twice — the test has SIX
assertions and THREE `errors.Is` ones, and the one that does not fire is F3's.
Corrected publicly on the PR rather than silently.

Matrix, re-measured on this tree: `make ci` rc=0, 21 ok / 0 FAIL; gofmt 0 of
487 files; golangci-lint 0 issues; `-race` clean on both guards. Under the
message-before-header mutant: 19 ok packages and exactly TWO `--- FAIL`s, both
new guards — the sentinel one on 5 of its 6 assertions, the rc one on "got 1
request(s)" and "exits 2, want 5 (exitNetwork)", with its CONTROL assertion not
firing. Mutant written as a diff, anchor asserted to occur exactly once,
retry.go restored from a `cp -a` copy and confirmed byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JQ4B1VHew33RNX5Ck7ELS
@ZacxDev

ZacxDev commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

🔴 Correction to the PR body, plus round 0's findings — all four taken (af35fbf)

First, a claim in the body above and in cc24058's commit message is wrong. It says the mutant fails "all five of its own assertions incl. both errors.Is ones". The count 5 is right; the phrasing is wrong twice — the test has six assertions and three errors.Is ones, and the one that does not fire is the ErrRateLimited check. Correcting it here rather than editing the body silently, since a reviewer may already have read it.

Round 0 — requirements: 5 (unattributed: 2) · deletion candidates: 3 · verdict requirement questioned

It re-measured every number in the body — all confirmed — and independently reproduced round 4's 21 ok / 0 FAIL by applying the mutant with both of this PR's files reverted to 7467c62. It explicitly rejected the deletion candidate (closing this PR by making a cap 429 terminal at exit 2 regardless of the header), on the grounds that deferring a three-phrase substring match over an explicit structured server signal is the worse trade. Findings:

🟡 F1 — the guard pinned the SENTINEL; the requirement and the bullet are about an EXIT CODE. The rank-22 ask was "pinning rc-5-for-cap-body-with-Retry-After" and round 4's own control was rc-level. Nothing composed cap-body + Retry-AfterexitCode() anywhere. cmd/civitai/read_error_stderr_test.go now does, in the idiom the adjacent 429 guard already states — "pkg/civitai owns the sentinel; this owns the number a script reads from $?" — with the same instrument positive control (exitCode must be able to return 2).

🟡 F2 — the precedence was described in three surfaces and argued in none. A maintainer reading the bullet call this "the exact hazard the 2 reclassification exists to prevent" would reasonably invert it, and hit a test asserting a contract without saying why. retry.go's 429 branch now argues it: Retry-After is a structured signal the server sent deliberately; isDeepPagingCap is a three-phrase substring match over prose its own doc comment calls "deliberately narrow", which any proxy or copy-edit can produce or destroy. Inverting trades this hazard for a worse one — a genuine throttle refused as a usage error, which a backoff loop never retries. It also names the four surfaces that move if the precedence changes, and corrects "there is no local guard on it": there are now two on the ordering; the reachability is the half that stays unguardable.

🟢 F3errors.Is(err, ErrRateLimited) is an invariant guard and is now labelled one. It does not fire under the mutant: errkind.go attaches exactly one sentinel and the cap matcher still matches, so ErrRateLimited is unreachable by construction on this input. Kept, not counted as regression coverage.

🟢 F4 — dropped the restated 20-ok/1-FAIL matrix from the pinnedBy comment: a fourth in-tree copy, hardcoding a package count nothing checks. The claim stays, the measurement lives in the commit.

Matrix on af35fbf

make ci rc=0, 21 ok / 0 FAIL
gofmt 0 flagged of 487 .go files
golangci-lint 0 issues (v2.13.2 local; CI's pinned v2.12.2 job is the authority and was green on cc24058)
-race clean on both guards
mutant (message before header) 19 ok packages, exactly TWO --- FAILs — both new guards. Sentinel guard: 5 of 6 assertions. rc guard: "got 1 request(s)" and "exits 2, want 5 (exitNetwork)", with its CONTROL assertion not firing

Mutant written down as a diff, anchor asserted to occur exactly once, retry.go restored from a cp -a copy and confirmed byte-identical. A mid-battery timeout did leave the mutant in the tree once; it was caught by an explicit re-check before the commit, not by luck.

Overlap with #602

Both edit internal/cmd/exitcodes_claims_test.go. The auditor test-merged them (git merge-tree --write-tree, branching on the exit code, not a marker grep) → exit 0, merged tree 21 ok / 0 FAIL, hunks disjoint (#602 appends floor entries and prepends rows; this edits the code-6 row's pinnedBy). That measurement is against cc24058 + f02c2003 only — whoever merges second should re-run it, since my later commit af35fbf touches that file again.

Round-0 trial ledger for this PR: ran: 1 · changed the outcome: 1

It ran before the merge decision (the audit-pr-nudge.py routing working as intended) and changed what shipped: two payload findings, one of which closes the requirement of record.

Round 1 (the nine correctness axes) has not been run. These four fixes carry my own verification, not an auditor's.

@ZacxDev
ZacxDev merged commit b727a83 into main Sep 14, 2026
13 checks passed
@ZacxDev
ZacxDev deleted the test/pin-429-header-before-message branch September 14, 2026 05:31
ZacxDev added a commit that referenced this pull request Sep 14, 2026
…not just routed

rank 22 — DONE. cli#601 merged (b727a83). The published header-before-message
ordering is now pinned TWICE: the sentinel in pkg/civitai/retry_test.go and the
exit code in cmd/civitai/read_error_stderr_test.go. Round 0 of the audit ran
before the merge decision and found the first guard was one hop short — the
requirement of record and the 🔴 bullet are about an `rc`, and nothing composed
cap-body + Retry-After -> exitCode(). That is the durable finding: a
classification test and an exit-code test are two claims.

rank 9 — DONE, and closed in the terms it was stated in. The streak's end was
INFERRED from a workflow_dispatch run; it is now OBSERVED in scheduled run
34848144324 (2026-09-14T13:15 UTC): drift=success, 0 failing steps and 15
EXECUTED, notify=success. The step count is the load-bearing half — a run that
skipped its steps reports success too.

rank 3 — no longer "unrouted". An agent is dispatched against civitai/civitai to
fix the Meilisearch-side coercion and open a PR; the brief's constraint is
recorded here because it is the easy mistake: a boundary String(...) cast is NOT
the fix, since "0222" is already 222 by the time the read path sees it. New rank
25 carries the outcome check so the next session does not re-dispatch it.

rank 26 — cli#602's merged-tree re-run is recorded as DELIBERATELY SKIPPED
(operator's call), not absent. The mechanical check is posted on that PR.

State now: main moved 7467c62 -> 426288f (#601, #596, #603). Flags that this
repo has a SECOND live handoff doc maintained by a concurrent session, with its
own ranked list — "rank 23 closes" in a commit subject may not mean this doc's.

Four new lessons, all from this session's own mistakes: a timed-out mutation
battery leaves the mutant in the tree (the restore line is the one that dies); a
grep for "the old text is gone" cannot distinguish REMOVED from QUOTED-IN-THE-
RETRACTION; `| head` ate grep's exit status again; and the merge drop-check has a
premise — it reported 103 "dropped" lines that were this PR's own replacements,
because main had not touched that path at all.

Doc: 64,635 B of the 65,536 B ceiling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JQ4B1VHew33RNX5Ck7ELS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant