fix(stale-base-triage): INHERITED is backwards for census guards — measured wrong 2 of 4 - #1629
Merged
Merged
Conversation
…reen them out
MEASURED, on this tool's own first live sweep. It named five PRs
`INHERITED — likely cured by rebase`. That is a falsifiable claim, so each was
tested by rebuilding the merged tree (PR head + current origin/main) and running
ONLY the named failing test:
#1450 1 passed -> INHERITED was RIGHT
#1286 1 passed -> INHERITED was RIGHT
#1603 1 failed -> INHERITED was FALSE
#1194 1 failed -> INHERITED was FALSE
#1038 merge conflict -> untestable, excluded from the rate
2 of 4 testable verdicts were FALSE. `#1600`'s own unit block calls ONE false
INHERITED disqualifying for arming, so this blocks arming until it is fixed.
THE MECHANISM. The INHERITED test is "the failing test's own FILE is
byte-identical at the head and the merge-base, and main has moved it". That is
sound for a test which exercises code it NAMES, and systematically wrong for a
repo-wide census/scanner guard, which inspects OTHER files: there an unchanged
guard file is the NORMAL state of a genuine, self-inflicted breakage, because
the PR never touches the guard — it adds a file the guard scans. Both false
verdicts were the same census guard, and in both the offender was a NEW FILE the
PR itself adds (#1194's was its own scripts/tests/test_break_glass_merge.py,
confirmed absent from main, so a rebase would have carried the offender along
with the red).
THE FIX reuses the derivation `ledger-check.sh` already runs (14daa42) rather
than growing a second one: `testlib.census_scan.analyze()` computes, from the
AST, every test whose verdict depends on the repo's FILE SET. A hand-kept list
of "the census guards" would itself be a ledger and would rot exactly the way
the ledgers it describes did. Membership => demote to NOT EXPLAINED BY
STALENESS, which is the conservative arm this tool already has.
FAIL-SAFE DIRECTION, and the bug my own test found. `census_scan.analyze()` on a
mis-rooted path RETURNS AN EMPTY RESULT rather than raising. The first version
trusted that and would have answered "not a census guard" for every test in the
repo — failing OPEN into precisely the bug being fixed. The trip is
`parsed == 0`; an unbuildable index RAISES and the caller records COULD NOT
MEASURE. A count floor was tried first and was wrong: it turned every
end-to-end fixture repo into COULD NOT MEASURE, because a small repo with no
census guards is a TRUE answer, not a broken scan. The production-strength
claim lives in the suite instead, as a positive control.
Placed LAST, after the evidence is computed, so a demoted verdict still prints
the candidates that WOULD have justified INHERITED — that evidence is what made
the original false verdicts falsifiable in one command. It can only ever demote;
it can never manufacture an INHERITED.
COST. The derivation is ~28s against a sweep that ran in 35s, so the index is
built on FIRST ASK and reused: a sweep that reaches no INHERITED branch pays
nothing. Pinned by a test that asserts the screen is not consulted for a test
the blob comparison already rejected.
Tests: 106 passed (was 93 before this change). The new block is red at
origin/main — 9 failed there, all on the absent symbol. The BEHAVIOURAL delta is
pinned on both sides with the identical fixture: base asserts ALPHA is
INHERITED (test_an_INHERITED_red_names_the_commit_that_already_fixed_it, still
green here), and the new test asserts the same fixture demotes when the oracle
says census guard — the two differ ONLY in the oracle's answer, so the screen is
isolated and nothing else moves.
Also widens the sys.path guard from "exactly one append" to "every mutation is
an append, and there are two". A count was the wrong pin: it goes red on an
addition that OBEYS the rule while saying nothing about one that breaks it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01298sLVSsAeZJpGNJTftdWv
Claude-Session-Id: 9ca6b87d-3d18-4866-9247-6ea38a486603
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes rank 13 of
claudedocs/handoff-gate-speed-and-ci-signal.md. This is what currently blocks arming the triage bot.The measurement
The bot's
INHERITED — likely cured by rebaseis falsifiable, so its own first live sweep was tested rather than argued about: rebuild each merged tree (PR head + currentorigin/main), run only the named failing test.test_a_partial_run_is_declared_…test_agent_without_any_tab_is_untouched2 of 4 testable verdicts were false.
#1600's own unit block calls ONE disqualifying.The mechanism
"The failing test's file is byte-identical at head and merge-base, and main moved it" is sound for a test that exercises code it names, and backwards for a repo-wide census guard that inspects other files — there, an unchanged guard file is the normal state of a genuine breakage, because the PR never touches the guard; it adds a file the guard scans. In both false cases the offender was a new file the PR itself adds (#1194's was its own
test_break_glass_merge.py, absent frommain, so a rebase carries the offender along with the red).The fix
Reuses the derivation
ledger-check.shalready runs (14daa42a) rather than growing a second one —census_scan.analyze()derives from the AST every test whose verdict depends on the repo's file set. A hand-kept list would be a ledger and would rot exactly the way the ledgers it describes did.It is placed last, so a demoted verdict still prints the candidates that would have justified INHERITED — that evidence is what made the false verdicts falsifiable in one command. It can only demote, never manufacture an INHERITED.
The bug my own test found
census_scan.analyze()on a mis-rooted path returns empty rather than raising. The first version trusted that and would have answered "not a census guard" for every test — failing open into the exact bug being fixed. The trip isparsed == 0; unbuildable RAISES and the caller records COULD NOT MEASURE. A count floor was tried first and was wrong (it turned every fixture repo into COULD NOT MEASURE); the production-strength claim is a positive control in the suite instead.Verification
origin/main, all on the absent symbol. ⚠ That is a structural red — so the behavioural delta is pinned on both sides with the identical fixture: base asserts ALPHA is INHERITED (test_an_INHERITED_red_names_the_commit_that_already_fixed_it, still green here), the new test asserts the same fixture demotes. They differ only in the oracle's answer.parsed == 0trip → mis-rooted red.⚠ The bot stays inert (
--comment-mode dry-runpinned by value). This does not arm it.🤖 Generated with Claude Code
https://claude.ai/code/session_01298sLVSsAeZJpGNJTftdWv