fix(tests): site the 18 remaining store roots off the contended disk, and census the set - #1458
Conversation
Round 1 audit — 6 findings, all fixedRound 0 (requirements & deletion) and round 1 (the nine axes) were run blind of each other, read-only, in separate worktrees. Round 1 verdict: safe to merge, nothing deploy-blocking. Every finding was in the scaffolding, and all six were one shape: the guard's description is wider than the guard — the same class this PR exists to fix, recurring inside the fix. What round 1 re-verified and found sound: 18 → 0 by AST (total Round 0 ledger: Queued follow-ups, deliberately NOT in this PR
Not measuredNothing here ran in the gate's own environment. If the CI container has no usable tmpfs, |
Round 2 delta re-audit — all 8 round-1 claims hold; 3 new findings, all fixedRound 2 ran blind of the fix author, against Claim 4 was verified at both points of the tmpfs dimension, which is what makes it a claim rather than an assertion: with tmpfs forced unusable, both behavioural arms SKIP and the structural ledger still fails. Authoritative tier at the head, read by content: The finding that matteredThe operand-type widening shipped with no regression test — restoring the exact pre-round-1 Now pinned by Ledger
Classification declared at round 1 and unchanged: Still queued, still not in this PR
|
Round 3 delta re-audit — and the ladder STOPS HEREAll seven round-2 claims verified. Two were checked structurally rather than by eye: with docstrings stripped, the payload file's AST is byte-identical across the range (100659 → 100659 lines, against a positive control that can see a one-token change), and the ledger's is The new guard came out stronger than it claimed. Eight mutants: each arm is independently covered (restoring either type gate alone names exactly that arm's probes); the negative half is genuinely reached, not short-circuited; and it goes red — not green — when the flow gate dies or The fifth-shape hunt came back clean, and that is a result: 27 concrete shapes measured against the census's five failure mechanisms — sibling-method via Round 3's four findings — all fixed in
|
…asure a SITING fix
The shim intercepts fsync(2) in libc, so it stalls whatever the fd is backed
by — including tmpfs. That makes it structurally unable to measure the fix it
was built to motivate: with the default mode, both arms of "store on disk" vs
"store on tmpfs" go red, and the red on the fixed arm is the shim's, not the
code's. Measured, and it is the reason this mode exists:
default mode, fd on ext4 -> stalls 65.0s (fs magic=0xef53)
default mode, fd on tmpfs -> stalls 65.0s (fs magic=0x1021994)
`SLOWFSYNC_SKIP_TMPFS=1` models the mechanism the README documents — node-local
*device* contention, which an fsync with no backing device does not wait for —
by passing a TMPFS_MAGIC fd straight through. Two properties are deliberate:
the pass-through does NOT consume the one-shot latch (else "the store moved to
tmpfs" becomes "the shim ran out of ammunition", a green that means nothing),
and a failing fstatfs() stalls rather than skips (a reproducer that quietly
stops firing reports a pass). The stall line now prints the fd's fs magic, so
which filesystem was stalled is readable rather than inferred.
Controls, all watched:
SKIP_TMPFS=1, fd on ext4 -> stalls 65.0s
SKIP_TMPFS=1, fd on tmpfs -> pass-through, 0s
SKIP_TMPFS=1, tmpfs THEN ext4 -> pass-through, then the full 65.0s stall
Default behaviour is unchanged, so every measurement already recorded in
scripts/ci-repro/README.md still describes what the shim does with the env var
unset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
… and census the set devrc#1211 sited `store`; a later round sited `scoped_store`. Both times the tests that build their OWN root inline were left on disk — 18 of them, spelled `tmp_path / "store"` — and one of those is `TestARefusedWriteIsIndistinguishableFromAnAbsentOne`, whose `test_POSITIVE_CONTROL_the_APPEND_comparison_CAN_see_the_difference` kept failing `tekton/devrc-pytests` on PRs whose diff cannot reach it. It is the only test in that class that gets a `200 appended` — its siblings all assert a 404, answered before any write — so it is the only one that reaches `server.py:_replace_bytes` and executes the two in-request fsyncs. That is why it, specifically, recurred. WHAT CHANGED * a `sited_root` fixture: `store_siting.store_root` with a test's lifetime. All 18 sites take their root from it. A fixture rather than 30 inline `with` blocks because several of the sites build the root inside a HELPER whose return value — a path, or a pair of `present()`/`absent()` closures — is used throughout the test body, so a `with` around the helper CALL would tear the store down before an assertion ran; and because a 30-block re-indentation is the shape of the scripted conversion this repo already reverted once. The one behaviour difference from an inline `with` (pytest does not throw a failure into a fixture generator, so the budget check runs even for a failing test) is written down in the fixture rather than glossed — it is already true of `store` and `scoped_store`. * `_DISK_ROOTED_SITES = 33` (a COUNT) becomes `_DISK_ROOTED_ALLOWLIST` (a SET, keyed `<Class.function> :: <unparsed expression>`, each entry carrying its reason), asserted in BOTH directions. A count could not tell "one site migrated, one written" from "nothing happened"; the set names the site in the failure message. 15 entries remain — the population that was never spelled "store" — every one verified to issue no write verb, so no request reaches `_replace_bytes`. Still disk-backed, recorded rather than hidden. * `_SITED_STORE_ROOT_CALLERS`, the other side of the relationship, so a deleted siting is red even in the shape the census cannot see. * `TestTheStoreIsSitedOffTheContendedDisk`'s docstring said a fixture falling back "EVERYWHERE" would be caught while its positive control took ONE fixture. Corrected, and `sited_root` gets its own behavioural control. RED BEFORE GREEN, on the exact failing test, via `scripts/ci-repro/slowfsync.c` | tree | shim mode | store on | result | |---|---|---|---| | origin/main | none | ext4 | class `5 passed in 8.76s` | | origin/main | default | ext4 | `1 failed in 65.52s`, TimeoutError, MECHANISM = SERVER_BLOCKED_IN_FSYNC | | origin/main | SKIP_TMPFS=1 | ext4 | `1 failed in 63.96s`, stall `fs magic=0xef53` | | branch | SKIP_TMPFS=1 | tmpfs | `1 passed in 3.67s`, 2 pass-throughs `magic=0x1021994` | | branch | default | tmpfs | `1 failed in 63.69s` — the shim can STILL kill it | | branch | SKIP_TMPFS=1, siting forced to fall back | ext4 | `1 failed in 64.29s` | The last two rows are what make the fourth mean something: row 5 that the green is not an inert reproducer, row 6 that it is the SITING and not another edit. MUTATION, `__pycache__` cleared between mutants * a migrated test reverted to `tmp_path / "store"` -> census RED naming that test, sited-ledger green, 22 others passed. * `sited_root` de-sited to `tmp_path / "store"` -> BOTH red. * `sited_root` de-sited to `tmp_path / "holder"` -> census GREEN, sited-ledger RED. This is the case that makes the second guard load-bearing, and it corrected a comment that had claimed the blind spot was "a fixture" when it is "a fixture under a name outside _ROOT_NAMES". Also re-measured: `_ROOT_NAMES` now contributes ZERO sites on this file (15 with it, 15 with it empty, identical keys). Its comment named seven sites it was propping up; all seven were among the 18 migrated. Kept — two probe tests depend on the on-sight arm — with the measurement written down instead of the stale one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…red false in the same file The previous commit replaced a stale measurement about `_ROOT_NAMES` with a fresh one (15 sites with the set, 15 with it empty, identical keys — it now catches nothing on `test_subsystem_store_api.py`, because all seven sites its old comment said it was propping up were among the 18 migrated). It then justified keeping the set by asserting that two of the ledger's own probe tests depend on the on-sight arm, and named `test_the_site_index_does_not_key_on_the_DIRECTORY_being_spelled_store`. That is FALSE, and it is the same defect the paragraph it replaced had: a claim about a guard's coverage, written from intent rather than from running it. Every probe in that test hands its path to `running(...)`, so the FLOW arm counts it and the directory's spelling never decides anything. MEASURED: `_ROOT_NAMES` set to the empty set, then every argument-free test in `test_store_siting_ledger.py` executed — ZERO behaviour changes. The set is inert everywhere that is measured today, not just on the file it was written for. It is still kept, but the comment now says plainly that the reason is an ARGUMENT and not a measurement (`_ROOT_CONSUMERS` closes renames but not growth into a new consumer name, and the on-sight arm is the only path that does not go through that set), and it names the closing condition for deleting it: a measurement showing the flow arm covers a store built under a name the consumer set does not know. Nothing here demonstrates that today. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…g it reported a pass
`getenv("SLOWFSYNC_SKIP_TMPFS") != NULL` tests PRESENCE, not value. The spelling an
operator reaches for to switch the filesystem-aware mode OFF switched it on, and the
file header and README both document the mode as `=1` — so the operator got exactly
what the header's own paragraph warns about: "a shim that quietly stops firing reports
a pass".
`skip_tmpfs_enabled()` compares the value. ON for 1/true/yes/on (case-insensitive);
OFF for 0/false/no/off, the empty string, and the variable being unset; an
unrecognised value resolves in the FIRING direction with a one-shot line on stderr
naming it, because silently picking either mode re-creates this defect one spelling
over and picking the non-firing one hides it behind a pass.
Measured, both values watched, on
TestARefusedWriteIsIndistinguishableFromAnAbsentOne::
test_POSITIVE_CONTROL_the_APPEND_comparison_CAN_see_the_difference, whose store this
branch sites on tmpfs:
before SKIP_TMPFS=0 two pass-through lines, 1 passed in 3.18s <- the bug
before SKIP_TMPFS=1 two pass-through lines, 1 passed in 3.26s
before unset stall magic=0x1021994, 1 failed in 64.38s
after SKIP_TMPFS=0 stall magic=0x1021994, 1 failed in 65.91s <- now == unset
after SKIP_TMPFS=1 two pass-through lines, 1 passed in 5.10s
Sixteen spellings watched on a one-fsync tmpfs probe: `1 true TRUE yes YES on On`
pass through in 0.00s; `0 false no off Off`, the empty string and the unset variable
take the 65s stall; `maybe` and `2` print the warning and then stall.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
… file of three
Four findings from the round-8 audit, all in the census and its own descriptions.
F1 — `_is_disk_rooted_store_expr` required the right operand to be an `ast.Name` or
an `ast.Constant` before consulting the flow gate. That is a guard on the SHAPE of a
spelling: `tmp_path / f"store-{k}"` is a `JoinedStr`, `tmp_path / ("store" + k)` is a
`BinOp`, and neither counted no matter where it flowed. Measured as a LIVE hole, not
imagined — swapping
`TestTheRENAMEIsFSYNCedToo.test_an_append_fsyncs_BOTH_the_file_and_its_DIRECTORY` back
to `_build_store(tmp_path / f"store-{ALLOW_SCOPE}", …)`, a write-path store that
reaches `api.append_bullet` -> `_replace_bytes` and its two in-request fsyncs:
before ledger 23 passed, api 760 passed -> the mutant SURVIVED
after ledger 1 failed / 22 passed, on the census's own message, naming the test
The type gate is deleted; `_used_as_a_store_root` does the discriminating, and it
already ruled out round 3's `(tmp_path / name).write_text(body)` false positive
(`test_a_path_that_is_never_used_as_a_store_still_does_NOT_count`).
No-op on today's tree, swept rather than assumed: old and widened predicates run over
all 184 `test_*.py` files in `scripts/tests` give **221 keys each, byte-identical key
sets**, no file changed. Shape matrix, old -> new: f-string 0->1, `"store" + k` 0->1,
`k.lower()` 0->1, `names[0]` 0->1, two-hop f-string 0->1, `joinpath(f"…")` 0->1;
`Path(tmp_path)/"store"`, `base = tmp_path`, `os.path.join`, `str(tmp_path)+"/store"`,
`tmp_path_factory.mktemp` and a fixture-returned holder stay 0->0, as does the
scratch-file false accusation. The residual is written into the docstring as the
residual; the widening does NOT close the helper-return, `Path(tmp_path)` or alias
shapes.
F5 — the census hardcoded `TESTS / "test_subsystem_store_api.py"` two functions under
`EXPECTED_SERVER_TESTS`, which names three; so did the sited ledger. Both now loop the
frozenset, keys carry the file, and each keeps its `Class.function` form with no line
numbers. Measured at the moment of the fix: api 15, `test_cairn_write.py` 0,
`test_cairn_cli.py` 0 — no live defect, and the two zeros are the point, since a file
never opened reports the same zero as a file that is clean. The sited ledger gains
`test_cairn_write.py :: store` and `test_cairn_cli.py :: source_store`; the api file's
15 allowlist entries are unchanged in content, only nested under their file. Both
ledgers now assert their file set equals `EXPECTED_SERVER_TESTS`, so a new ledgered
file cannot arrive as a silent absence.
F6 — `test_the_census_can_actually_SEE_a_disk_rooted_site` probed only
`tmp_path / 'store'`, which `_ROOT_NAMES` answers ON SIGHT: the flow arm could have
been dead and the control still green. A second probe under `holder`, outside
`_ROOT_NAMES`, now exercises it, with an assertion that the name really is outside.
F2 — `TestTheStoreIsSitedOffTheContendedDisk`'s docstring said the census and the
allowlist "enumerate EVERY store site in this file", ten lines under the retraction of
the previous "EVERYWHERE" overclaim. Corrected to what the predicate reads, with the
whole measured residual — the fixture-scope hole, the not-spelled-off-`tmp_path`
shapes, and `_ROOT_CONSUMERS` growth — rather than a fresh justification. The census's
own docstring recorded only the fixture hole and now records all three.
F3 — the sited ledger's failure message said it was "the ONLY guard that can see"
a fixture de-sited under a name outside `_ROOT_NAMES`. That is false wherever a usable
tmpfs exists: `TestTheStoreIsSitedOffTheContendedDisk::test_the_sited_root_fixture_
ACTUALLY_lands_on_tmpfs_when_one_exists` reads the fstype of `sited_root.parent` and
fails alongside it. The "22 others passed" behind the original claim was a one-FILE run
of this 23-test module. The arm stays — the behavioural control SKIPS where there is
no usable tmpfs — but the claim is now scoped to "on a machine with no usable tmpfs",
in the message and in the comment above the set.
Ledger: 23 passed on the restored tree, census key set identical to the pre-change 15.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…ft 23 passed
Round 1 deleted the `isinstance(right, (ast.Name, ast.Constant))` gates from both
arms of `_is_disk_rooted_store_expr` so the flow gate alone decides, and measured
the widening BY HAND in a commit message. In the module whose entire premise is
that a hand measurement is not coverage.
MEASURED: restoring both gates verbatim — the exact pre-round-1 code — leaves this
file at 23 passed. Nothing in the repo goes red. Every existing probe uses an
`ast.Constant` operand: both probes in `test_the_census_can_actually_SEE_a_disk_
rooted_site` ('store', 'holder'), and every probe in `test_a_path_that_is_never_
used_as_a_store_still_does_NOT_count`. No test anywhere exercised a JoinedStr,
BinOp, Call or Subscript operand in either direction.
`test_the_operand_NODE_TYPE_is_not_what_decides_either` is the coverage: four
non-Constant spellings (f-string, concatenation, call, subscript) across both the
`/` and `.joinpath` arms, each appearing twice — once flowing into `_build_store`
and once flowing nowhere. Named individually rather than totalled, so three cannot
regress behind one that still works.
Watched red, both halves, `__pycache__` cleared between mutants:
* restore both `isinstance` gates -> the four COUNTED probes report 0 and this
test is the ONLY failure, on its own message. 1 failed, 23 passed.
* `_used_as_a_store_root` defaults to True (the narrowest expression that makes
the flow gate accept everything) -> the four UNCOUNTED probes report 1 and this
test fails on its own message. 5 failed, 19 passed.
* unmutated -> 24 passed.
Also corrects the predicate's own docstring, which cited `test_a_path_that_is_
never_used_as_a_store_still_does_NOT_count` as the control for the widening. That
test cannot see it — its probes are Constants. A comment is a claim too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…cope, no fixture
The stated reach — a `tmp_path / X` that flows into a store consumer "WITHIN ONE
FUNCTION SCOPE, and nothing else" — is wider than the code. RE-MEASURED here, not
inherited from the audit: all in one function scope, all reaching `running` (which
IS in `_ROOT_CONSUMERS`), all spelled `tmp_path / 'served'`, no fixture anywhere:
inline (control) -> 1
stores = {'a': tmp_path / 'served'}; running(stores['a']) -> 0
stores = [tmp_path / 'served']; running(stores[0]) -> 0
for served in (tmp_path / 'served',): running(served) -> 0
roots = [p for p in (tmp_path / 'served',)]; running(roots[0]) -> 0
served = tmp_path / 'served'; def go(): running(served); go() -> 0
The list and comprehension rows are mine; the other four reproduce the audit's.
None of the three enumerated residual bullets covers these — bullet 1 is a
`@pytest.fixture` scope boundary, bullet 2 is "not spelled `tmp_path / X`", bullet
3 is `_ROOT_CONSUMERS` growth — so the enumeration read as complete while missing a
whole family. Root cause: `_assignments` resolves `Assign`/`AnnAssign`/`NamedExpr`/
`withitem` and nothing else, so a `for` and a comprehension target bind invisibly;
`_path_base` returns None for an `ast.Subscript`, so a container element is not a
path expression it can name a base for; and `_walk_scope` stops at EVERY nested
function, `def go()` included — it is not only pytest's injection that crosses a
boundary.
Fixes the ENUMERATION at all five places that state it, not just the reported one:
the census docstring (fourth bullet, with the numbers), the "at least once" test's
docstring, the predicate's own "READ THE RESIDUAL" paragraph, `_assignments`'
docstring, and `TestTheStoreIsSitedOffTheContendedDisk` in the api file. Widening
`_assignments` is deliberately NOT done — it is written down rather than guarded,
the same deal the other three bullets get.
The api paragraph also loses its "each pinned by a named guard over there" clause,
which is false for two of the three holes; the positive replacement is the next
commit.
24 passed on the ledger, unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…ayer — the fix already existed and had never reached the failing site The SERVER_BLOCKED_IN_FSYNC diagnosis stands. What was wrong was the remedy set: (a) bound the write path was recommended as "the only one that makes the SERVER correct", but it is a production change to crash-durability semantics made to close a test-harness siting gap. The tmpfs fix from #1211/#1219/#1239 sited 5 store roots and left 18 open-coded on disk, and the test that keeps reddening the gate is one of the 18 — it is also the only test in its class that gets a 200, so it is the only one that executes the two in-request fsyncs. Records what #1458 ships, the independently re-run census mutation, that slowfsync.c could not measure a siting fix without the new opt-in, and the two residuals: nothing is measured in CI, and a green gate on #1458 is not the verifier (handoff-gate-flake-store-api.md rank 1 is). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…her three are prose
"The holes, each measured and each pinned by a named guard over there rather than
left to be rediscovered as news" was true of exactly one hole. Verified here, not
inherited:
* hole 1 (a root bound in a `@pytest.fixture`) — PINNED.
`test_a_store_root_bound_in_a_pytest_FIXTURE_is_NOT_counted` is a real test that
runs every time, asserts 0 for both the plain and the tuple-returning fixture
form against an inline control of 1. Its cited live instances exist:
`served = tmp_path / "ordered-served"` in `shuffled_pair` (:3990/:3993) and
`served = tmp_path / "ambig-served"` in `ambiguous_pair` (:4339/:4342).
* hole 2 (a root not spelled off the NAME `tmp_path`) — NO TEST ANYWHERE. Every
occurrence of `Path(tmp_path)`, `base = tmp_path`, `os.path.join`,
`str(tmp_path) + `, `tmp_path_factory` in `test_store_siting_ledger.py` is
inside a docstring (:486-488, :894-895); the only code hits in the whole tests
directory are unrelated files. Nothing re-measures the claimed 0.
* hole 3 (`_ROOT_CONSUMERS` growth) — its own clause already said "that set's own
comment", i.e. a comment, contradicting "named guard" mid-sentence. The
`serve_store(served)` -> 0 control it cites appears once in the file, in that
comment.
* hole 4 (the one the previous commit added) — also only written down.
So the label now comes first on every bullet, PINNED or ONLY WRITTEN DOWN, at both
places the enumeration appears. That distinction is the sentence's whole stated
purpose — "so it cannot be rediscovered as news" is a property of a guard that
re-measures, and three of these four can silently close or widen with the suite
green.
24 passed on the ledger, unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
… has three
The sentence defining what the disk-rooted census reads — "flows into a store
consumer within one function scope AND THROUGH A BINDING FORM `_assignments`
RESOLVES" — is satisfied by a shape that measures 0:
def test_probe(tmp_path):
stores = {'a': tmp_path / 'served'} # an ast.Assign; _assignments DOES
running(stores['a']) # yield ('stores', <Dict>)
Measured 0 against an inline control of 1; the list form likewise. The blocker
is a THIRD condition the sentence never stated: `_path_base` returns None for
the `ast.Subscript` handed to the consumer, so the bound name never enters
`flowing_roots` and the binding is never reached. Teaching `_path_base` to see
through a Subscript takes both shapes 0 -> 1 while widening `_assignments`
leaves both at 0, which is what attributes the mechanism.
Corrected at all three sites that state the definition.
Same commit, same underlying defect: two of those sites grouped the CLOSURE
under "one function scope" / "a binding the census does not resolve". Measured,
the closure fails none of the three conditions — `served = tmp_path / 'served'`
is a plain assignment and `_path_base(served)` names a base — and stays 0 under
a widened `_assignments` AND a widened `_path_base`. It is the scope arm,
`_walk_scope` stopping at the nested `def`. The census bullet already said this;
these two copies had lost the correction.
Also: the `serve_store(served)` -> 0 control is cited as appearing "once in that
file, in the comment itself". It appears twice (`_ROOT_CONSUMERS`' comment and
the fourth residual bullet). The count is replaced by the property that actually
matters — both occurrences are prose, and no test runs it.
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
The docstring said "Widening this function would close them", where *them* is
the three shapes named one sentence earlier — one of which the SAME sentence
attributes to `_path_base`. Measured by actually widening it (adding `ast.For`
/ `ast.AsyncFor` / `ast.comprehension` targets to the yields) and re-running the
fourth residual bullet's five probes against an inline control of 1:
for target 0 -> 1
comprehension target (isolated) 0 -> 1
dict element 0 -> 0
list element 0 -> 0
closure 0 -> 0
Two of five. The two element rows go 0 -> 1 under a widened `_path_base` with
`_assignments` untouched, so they are that function's; the closure stays 0 under
both widenings, so it is `_walk_scope`'s.
This is not a nit, because the same docstring frames the widening as THE
deferred fix and tells the maintainer to re-run the census and record what it
starts seeing. Following that instruction produces a census that moves, which
reads as the hole closing, while the element and closure halves stay open and
the suite stays green.
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…trated none
The bullet's opening named one mechanism for all five rows ("not through a
binding form `_assignments` resolves"). Widening one thing at a time and
re-measuring separates them:
* `bind` — `for` target, comprehension target. Adding `ast.For` /
`ast.comprehension` targets to `_assignments` takes both 0 -> 1 and moves
no other row.
* `base` — dict element, list element. Unmoved by a widened `_assignments`;
0 -> 1 when `_path_base` is taught to see through an `ast.Subscript`.
* `scope` — the closure. Unmoved by either widening; `_walk_scope` stops at
the nested `def`.
Each row now carries its mechanism.
Row 4 is replaced. `roots = [p for p in (tmp_path / 'served',)];
running(roots[0])` was filed under the comprehension target and does not
demonstrate it: 0 under a widened `_assignments`, 1 under a widened `_path_base`
alone, so the SUBSCRIPT is the blocker and the comprehension target is not
load-bearing — `mark()` walks the whole bound value, so once `roots` is reached
the comprehension is transparent. `out = [running(p) for p in (tmp_path /
'served',)]` isolates the target: 0 as shipped, 1 under a widened
`_assignments`.
It is replaced rather than relabelled "doubly-blocked", because it is not:
one widening is sufficient to close it.
Docstrings only; no executable change. 784 passed across both files.
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…s three Self-audit of the three commits above, and the same defect they exist to fix. Two of them say the closure "fails NONE of the three" / "passes all three" conditions and measures 0 anyway. That reads as a contradiction, and it is one: "within one function scope" means `_walk_scope`'s scope, and the closure's binding and its consumer are two of those. It fails the FIRST condition — on the code's meaning of scope, not the reader's — and neither of the other two. Both sites now say that, and the census bullet's own preamble says "within one TEST FUNCTION" rather than "scope" for exactly the reason that one of its rows is where those two words come apart. Also: the api file's bullet announced "TWO mechanisms" and then listed three (bind / base / scope), and said "neither of the two arms above" where three bullets precede it. Docstrings only; the executable AST of both files is identical to 0618eea with docstrings stripped. 784 passed across both files. Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
30aa044 to
a6dd11e
Compare
…is doc kept saying they were Two independent surfaces read the same minute disagree with the claim this item and its PR commentary were built on: classic branch protection on main returns no required status checks and enforce_admins=false, and the repo has no rulesets and no rules applying to main. The gates are advisory. A red gate is still a hazard — the other one, that nobody is forced to look at — but "nobody can merge" was false and it inflated the urgency of every gate item here. The line now says to re-read the setting rather than cite the doc. Also records the SECOND, distinct timeout flake now reddening #1458: test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_N, added hours earlier by #1445's own ladder, SIGKILLed at its 120s subprocess bound. Not an assertion, unreachable from #1458's diff, and green on #1462 minutes earlier on the same base. Wall times CI-to-CI say the node was not inflated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
Gated on the MERGED tree — both tiers green, and the red CI is attributedPicked this up from rank 1 of
Merged tree —
⚠ The dev-host tier ( The red
|
| tree | store-building classes | raw and writing |
|---|---|---|
origin/main |
sited 1, raw 6 | 2 classes / 16 test methods |
| merged with this PR | sited 7, raw 0 | 0 |
And the guard was watched to go red. In a cp -a copy with .git removed and
PYTHONDONTWRITEBYTECODE=1:
- control, unmutated:
24 passed - mutant — one sited site reverted to
tmp_path / "store"
(TestRefusedIsIndistinguishableFromAbsent.test_RECALL_…): KILLED,1 failed, 23 passed,
bytest_the_disk_rooted_census_matches_the_allowlist_EXACTLYwith its own message naming
the exact site and line — not by a different guard's error, and not by the happy path resolving.
That is the half a per-file ledger could not do: before this PR, the same mutation passes, because
test_every_ledgered_file_IMPORTS_AND_CALLS_the_shared_siting_at_least_once only asks whether the
file uses store_siting anywhere.
Why this mattered — the concrete payout
devrc#1435's red CI was TestARefusedWriteIsIndistinguishableFromAnAbsentOne::test_POSITIVE_CONTROL_the_APPEND_comparison_CAN_see_the_difference
— one of the 16 above, and the handoff's leading hypothesis was the _free_port() TOCTOU. That
hypothesis is wrong: the test drives the server through running(), which builds it in-process
with port=0 and never calls _free_port(). It was the fsync mechanism this PR removes. The same
test also failed CI on #1434 and #1447.
⚠ Scope note on the reason strings: the allowlist's "read-only" entries claim no write verb reaches
_replace_bytes, which is the fsync mechanism specifically — not that a disk-backed root is
harmless. The PR says this itself; repeating it because it is the sentence most likely to be
compressed into "covered".
…ayer — the fix already existed and had never reached the failing site The SERVER_BLOCKED_IN_FSYNC diagnosis stands. What was wrong was the remedy set: (a) bound the write path was recommended as "the only one that makes the SERVER correct", but it is a production change to crash-durability semantics made to close a test-harness siting gap. The tmpfs fix from #1211/#1219/#1239 sited 5 store roots and left 18 open-coded on disk, and the test that keeps reddening the gate is one of the 18 — it is also the only test in its class that gets a 200, so it is the only one that executes the two in-request fsyncs. Records what #1458 ships, the independently re-run census mutation, that slowfsync.c could not measure a siting fix without the new opt-in, and the two residuals: nothing is measured in CI, and a green gate on #1458 is not the verifier (handoff-gate-flake-store-api.md rank 1 is). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…is doc kept saying they were Two independent surfaces read the same minute disagree with the claim this item and its PR commentary were built on: classic branch protection on main returns no required status checks and enforce_admins=false, and the repo has no rulesets and no rules applying to main. The gates are advisory. A red gate is still a hazard — the other one, that nobody is forced to look at — but "nobody can merge" was false and it inflated the urgency of every gate item here. The line now says to re-read the setting rather than cite the doc. Also records the SECOND, distinct timeout flake now reddening #1458: test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_N, added hours earlier by #1445's own ladder, SIGKILLed at its 120s subprocess bound. Not an assertion, unreachable from #1458's diff, and green on #1462 minutes earlier on the same base. Wall times CI-to-CI say the node was not inflated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…ged RED on the second flake This doc's rank 22 called #1458 open; it landed while the doc sat in review, which is the stale-claim class this ladder kept finding — here staled by an external event rather than a sibling commit. Verified by content on origin/main, not by ancestry: a squash makes merge-base --is-ancestor false forever. sited_root, _DISK_ROOTED_ALLOWLIST, the operand-type guard and skip_tmpfs_enabled are all present; the single remaining tmp_path / "store" is a docstring at :367. Also records that it merged with pytests RED on the unrelated #1445 timeout flake, because "merged" and "merged green" are different claims. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
… — the fix existed and had never reached the failing site (#1462) * docs(handoff): rank 22's own three remedies were aimed at the wrong layer — the fix already existed and had never reached the failing site The SERVER_BLOCKED_IN_FSYNC diagnosis stands. What was wrong was the remedy set: (a) bound the write path was recommended as "the only one that makes the SERVER correct", but it is a production change to crash-durability semantics made to close a test-harness siting gap. The tmpfs fix from #1211/#1219/#1239 sited 5 store roots and left 18 open-coded on disk, and the test that keeps reddening the gate is one of the 18 — it is also the only test in its class that gets a 200, so it is the only one that executes the two in-request fsyncs. Records what #1458 ships, the independently re-run census mutation, that slowfsync.c could not measure a siting fix without the new opt-in, and the two residuals: nothing is measured in CI, and a green gate on #1458 is not the verifier (handoff-gate-flake-store-api.md rank 1 is). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726 * docs(handoff): RETRACTED — the Tekton checks are not required, and this doc kept saying they were Two independent surfaces read the same minute disagree with the claim this item and its PR commentary were built on: classic branch protection on main returns no required status checks and enforce_admins=false, and the repo has no rulesets and no rules applying to main. The gates are advisory. A red gate is still a hazard — the other one, that nobody is forced to look at — but "nobody can merge" was false and it inflated the urgency of every gate item here. The line now says to re-read the setting rather than cite the doc. Also records the SECOND, distinct timeout flake now reddening #1458: test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_N, added hours earlier by #1445's own ladder, SIGKILLed at its 120s subprocess bound. Not an assertion, unreachable from #1458's diff, and green on #1462 minutes earlier on the same base. Wall times CI-to-CI say the node was not inflated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726 * docs(handoff): #1458 MERGED as ce9b55c — verified by content, and merged RED on the second flake This doc's rank 22 called #1458 open; it landed while the doc sat in review, which is the stale-claim class this ladder kept finding — here staled by an external event rather than a sibling commit. Verified by content on origin/main, not by ancestry: a squash makes merge-base --is-ancestor false forever. sited_root, _DISK_ROOTED_ALLOWLIST, the operand-type guard and skip_tmpfs_enabled are all present; the single remaining tmp_path / "store" is a docstring at :367. Also records that it merged with pytests RED on the unrelated #1445 timeout flake, because "merged" and "merged green" are different claims. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pe and nothing else test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_N spawns a nested full run-tests.sh under a hard timeout=120 and is SIGKILLed at the bound. It shares the observable with the store-api flake — a check red on a diff that cannot reach it — and no mechanism: this is subprocess.TimeoutExpired with rc -9, not an assertion, and grepping MECHANISM = finds nothing because no store server is involved. Added by #1445's own ladder hours before it first went red. Ruled out with values: any diff (#1458's four files cannot reach that file); determinism (#1462 passed minutes earlier on the same base); general node load (failing vs passing run 1030.34s vs 1065.64s and 155.60s vs 154.73s — the failing run was faster). NOT established: what the nested run blocked on; the pipelinerun was pruned within the hour. Rank 1 gains two corrections it cannot be measured without: the anchor moved to ce9b55c (#1458 sited 18 roots that #1211/#1219/#1239 had left on disk), so a rate against 65f7325 spans two interventions; and the population is no longer one mechanism, so classify by failing test before counting. Also retracts "required check" there — measured: no required checks, no rulesets, enforce_admins=false. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…d it was never the worst one (#1512) Rank 3 of the gate-speed handoff read "needs a diagnosis". It did not: the suite's own classifier had already printed MECHANISM = SERVER_BLOCKED_IN_FSYNC on devrc-ci-86zxj, and #1458 (ce9b55c) then removed that mechanism by siting the store roots on tmpfs. Nothing recorded that the fix had LANDED, so the next session would have re-derived the whole thing. Measured over tekton/devrc-pytests, newest verdict per PR head, split on ce9b55c's timestamp: window verdicts genuine failures this test pre-fix 125 29 4 post-fix 45 6 0 Two things stated rather than glossed: * THE POWER IS WEAK. At the pre-fix per-verdict rate (3.2%) the expected count in 45 verdicts is ~1.4, so P(0) is about 0.23. The zero is CONSISTENT with the fix and does not establish it; the mechanism's removal is what does. Written into the comment as not-proven so it cannot be upgraded by a reader. * IT WAS NEVER THE WORST FLAKE. In the same pre-fix window test_every_decrypt_family_VERDICT_is_pinned_WHOLE failed 8 times to this test's 4, and is also at 0 post-fix. This one was ranked and worked because it had a long diagnosis attached, not because it was frequent. Also corrects the port-race docstring, which still read as though that intermittent were live. Its "whether closing this race moves that rate is UNKNOWN" was right and is why the classes stayed separate: had they been merged into one story, the tmpfs fix would have been credited to the port-race retry and the real mechanism would still be in the request path. Comment-only; no behaviour change. Verified: the full file is 760 passed. Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…pe and nothing else (#1477) * docs(gate-flake): rank 7 — a SECOND gate flake sharing this doc's shape and nothing else test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_N spawns a nested full run-tests.sh under a hard timeout=120 and is SIGKILLed at the bound. It shares the observable with the store-api flake — a check red on a diff that cannot reach it — and no mechanism: this is subprocess.TimeoutExpired with rc -9, not an assertion, and grepping MECHANISM = finds nothing because no store server is involved. Added by #1445's own ladder hours before it first went red. Ruled out with values: any diff (#1458's four files cannot reach that file); determinism (#1462 passed minutes earlier on the same base); general node load (failing vs passing run 1030.34s vs 1065.64s and 155.60s vs 154.73s — the failing run was faster). NOT established: what the nested run blocked on; the pipelinerun was pruned within the hour. Rank 1 gains two corrections it cannot be measured without: the anchor moved to ce9b55c (#1458 sited 18 roots that #1211/#1219/#1239 had left on disk), so a rate against 65f7325 spans two interventions; and the population is no longer one mechanism, so classify by failing test before counting. Also retracts "required check" there — measured: no required checks, no rulesets, enforce_admins=false. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726 * docs(gate-flake): rank 7 is keyed to the FILE — a second test hit the same bound, and my provenance claim was wrong A third occurrence landed while this PR was open, on a DIFFERENT test in the same file: test_the_SUMMARY_BANNER_names_the_real_selection_source, #1462 at dc97239, with the identical TimeoutExpired — and its spawned command carries no --targets, so it is an entire suite run inside one test at a 120s bound. The unit is therefore _run's shared timeout at :102, not either test name, and the closing condition now keys on the file. Keyed to one test name, a second exposed test walks straight past it. Retracts this item's own provenance claim: "added by #1445's ladder" is true of the subset-note test and false of the file and of _run, both from #1073 (809486f). #1445 added one more caller to a bound that already existed. Also corrects the determinism evidence: #1462 passed this file at 8df1117 and failed it at dc97239 — two docs-only commits on the same base, different test failing — which is stronger than the cross-PR comparison it replaces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726 * docs(gate-flake): rank 7 rewritten from measurements — the diagnosis was wrong and so were three of its rule-outs An adversarial round on this PR returned four 🔴 and I could re-derive all of them. Replacing the item rather than patching it, because most of what it said was not true: - The unit was `_run`'s bound. There are SIX bound sites; `_run_env:862` is the one behind 2 of the 3 occurrences, so the old closing condition was walkable by fixing `_run` alone. - "Ruled out: general node load" is now the LEADING hypothesis. Measured: the flaking spawns take 57.20s and 47.10s against a 120s bound, and the same file on the same tree measures 205s and 428.40s — 2.09x ambient variance. - "Ruled out: determinism" rested on a same-base comparison that is false; the branch was rebased across ce9b55c between the green and the red. - "A nested FULL run" is false for both tests — argv or DEVRC_TARGETS narrows each to one target, so half the Next probe had nothing to do. Dropped. - The provenance sentence has now been wrong three times (#1445 never touched the file; #289 created it, #1073 added both flaking tests). Deleted rather than corrected a fourth time — nothing depended on it. - A third occurrence, #1454, was uncounted, and its diff DOES touch run-tests.sh, so the "cannot reach it" rule-out never covered it. Adds what the item lacked: the detector argument (raise the bound, do not remove it — a full set is ~28x a one-target run), and a closing condition with a positive control, since the old one was satisfiable by a rename or a skip. Rank 1: the anchor sentence named the wrong contaminated window, and the branch-protection note duplicated the Gotchas entry minus its qualifiers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726 * docs(gate-flake): rank 7 asserts NO cause — three drafts asserted one, and a fourth round refuted the newest Round 2 returned four more findings and I re-derived every one. The pattern is now unambiguous: each rewrite of this item invented a cause it had not measured. This version states none. Refuted MYSELF, against the auditor's own leading rival: #1429 (a0839ec) did NOT double CI parallelism. It changed min(nproc,4) to min(nproc,cgroup quota,8), and devrc-ci-gate's pytests step sets limits.cpu: "4" — both formulas yield 4. It doubled only on an unquotaed host. Re-measured the headroom myself rather than inheriting it, and the numbers do not reproduce: :418 43.48s (audit said 70.94s), :988 40.51s (51.49s), :723 38.89s (84.22s), :582 37.78s (69.70s); the file 137.69s and 164.27s against 205s and 428.40s. Four measurements, 3.11x spread, same tree. THAT is the finding — no point wall time here is stable, so the item now states the range and the spread and tells the reader not to quote a point value. Fixed from round 2: "or the nested runs made concurrent" is forbidden (run-tests.sh:3889 NESTED RUNS MUST BE SERIAL, enforced :4043, with a live guard) and is removed; the kubectl recipe regains KUBECONFIG=$KC_HOMELAB and drops the false "nowhere else"; "~28x" as a target COUNT becomes the measured 1194s serial full set; :418 and :582 are named for the first time; the date predicate becomes ancestry; and the tekton SKILL.md cross-reference is replaced by the live measurement plus a warning that SKILL.md is stale and says the opposite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726 * docs(gate-flake): restore why rank 7's key is the six sites — the retracted draft, not just the conclusion Closes the last open finding from round 2: the rewrite prevented the hazard structurally but deleted the record of why. CLAUDE.md calls a retracted theory the most valuable content in the codebase precisely because the next reader re-derives it otherwise — and here the narrowing is tempting twice over, since keying to one test name OR to _run alone both look sufficient and both miss :988 via _run_env, which is 2 of the 3 observed occurrences. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
… not a flake, and #1458 never touched it (#1531) `#1512` left a comment in `test_subsystem_store_api.py` saying `test_every_decrypt_family_VERDICT_is_pinned_WHOLE` failed 8 times to this test's 4 — "twice as often" — and concluding: COUNT them before choosing which flake to chase. The count was right. The conclusion was wrong, and counting is what produced it. That test is not intermittent. Measured 2026-09-11 over the 200 most-recently-updated PR heads (`tekton/devrc-pytests`, 2026-09-05..09-11, 197 terminal verdicts): * MECHANISM — it is not on #1458's path at all. The escrow file imports no store server, no `store_siting`, no `build_server`; it drives `escrow-verify.py` against an in-memory `FakeDownloader`, so `server.py:_replace_bytes`'s in-request fsync — the one thing `ce9b55c3` removes — is never reached. `ce9b55c3`'s diff names the file zero times, and the test runs in 1.71 s. * TIME — all 8 failures fall inside ONE 14-hour window on 2026-09-08 (05:44Z..19:44Z) across 8 distinct heads, and every one reports `failed=7` or `failed=8`: a whole-suite red hitting every open PR at once. * CAUSE — nixpkgs moved `age` to 1.3.2, changing its tamper classification, and that test pins those verdicts by exact string equality. Re-keyed by #1392 (`94f82796`, 09-08T18:26Z) and #1403 (`4f49f5dc`, 20:32Z) — two days before `ce9b55c3` (09-10T20:56Z) existed. The two failures after 18:26Z are stale-base heads that had not picked up #1392. Contrast, same instrument: the test #1458 actually fixed failed 5 times across 4 separate days (09-06, 09-07, 09-08, 09-09 x2) — scattered, which is what contention looks like. So the comment now carries a discriminator instead of "count them": a flake scatters across days and reports `failed=1`; an environment red clusters in one window and reports the same N>1 on every head. Count only what survives both. Also records the blind spot both counts inherit: GitHub truncates a status description at 138 characters, so only the FIRST failing test is ever named and every count here is a lower bound. Bounded by arithmetic on the same rows — of 14 post-fix failures, 12 derive `failed=1`, one derives 2, one is unparseable. The escrow test gains its own delisting note, so the next flake hunt does not re-derive this: a red there is a claim about the `age` binary, not about CI load. Comment-only, two files, no behaviour change. Claude-Session: https://claude.ai/code/session_01LWAGG1mTfkozfKa5MMjXuC Claude-Session-Id: 9b6235dd-2c40-4230-b6d3-5fa6d24a74fd Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
MEASURED, and it is the reason this exists: of the SIX genuine `tekton/devrc-pytests` failures in the whole post-#1458 window, THREE — half — were the same test on PRs 17, 40 and 40 commits behind `main`, all since merged, and that test passes on current `main`. An earlier sample had 8 of 8 failing open PRs 5-42 commits behind, with a rebase curing 4 outright. Nothing blocks a merge here, so the only value the gate has is whether a human believes a red — and half the genuine reds are already-fixed failures a human then debugs against a diff that cannot reach them. `scripts/stale-base-triage.py` answers that deterministically, running no tests: it reads the NEWEST `tekton/devrc-pytests` status per context, maps each failing test named in the description to the ONE file defining it, and emits the evidence pair — the PR head's copy of that file is byte-identical to the merge-base's (so this change never touched it) while `main` HAS moved it, in commits absent from the head. Verdict per PR: INHERITED / NOT EXPLAINED BY STALENESS / COULD NOT MEASURE. Measured live, read-only, over all 51 open PRs: 20 red heads, 3 INHERITED (#1286, #1194, #1038), 2 NOT EXPLAINED, 15 COULD NOT MEASURE, 4 broken-gate `error` rows counted on their own line. That non-zero 3 is the positive control for the count: a zero here would otherwise be indistinguishable from an instrument wired to nothing. 🔴 REPORT ONLY BY DEFAULT. `COMMENT_MODE_DEFAULT` is the literal `"off"`, pinned by `test_the_comment_mode_default_is_the_LITERAL_off` both as a value and as source text, so arming the PR-comment writer costs a visible line in the arming commit. Nothing was posted to GitHub at any point. Traps this repo has already paid for, each handled and each pinned by a test: * `/commits/{sha}/status` (SINGULAR) maps `error` onto `failure` — never read. Only the plural list endpoint, which carries no roll-up field at all. * the list is NEWEST-FIRST — folded on `max(created_at)`, order-independent. * `error` is a broken gate, not a code failure — its own summary line, never in the red total. * a SQUASH merge never makes a head an ancestor of its base; ancestry is asked only about the PR head, and every "this landed" claim is made by blob OID. * `git diff --quiet <ref> -- <path>` exits 0 when the path exists on NEITHER side; existence is proved with `git cat-file -e` first, pinned by an AST scan of the git invocations rather than a string search that its own comment would satisfy. * the 140-char description cap means a description can prove "at least one test failed and here is its name" but never "these are all of them" — so an INHERITED verdict requires `failed=N` to have survived AND to equal the number of names. That is conservative on purpose: dismissing a real red is the one error this tool must not make. * the test-name -> file mapping is searched, never guessed, with a distinct outcome for not-found, ambiguous, class-qualified, parametrised and cap-truncated names. Testing. 56 tests; RED at base `60194765` (the module cannot collect — the script does not exist there), GREEN at HEAD. Three enumerated mutation rounds under PYTHONDONTWRITEBYTECODE=1, mutants derived from the AST plus targeted textual ones: round 1 74 mutants / 15 survivors AND THE POSITIVE CONTROL SURVIVED, which is what found that `CONTEXT` was pinned nowhere; round 2 82 / 3 with the control killed; round 3 81 mutants, 78 killed, 0 survived, control killed. Round 1's survivors also found a real defect — the `on_main` half of the evidence pair was a guard that can never run (`commits_touching` already walks `merge-base..main`), so it is now measured and printed for verification rather than gated on, and `not in_head` moved into `prove_candidates` where a test can reach it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0
MEASURED, and it is the reason this exists: of the SIX genuine `tekton/devrc-pytests` failures in the whole post-#1458 window, THREE — half — were the same test on PRs 17, 40 and 40 commits behind `main`, all since merged, and that test passes on current `main`. An earlier sample had 8 of 8 failing open PRs 5-42 commits behind, with a rebase curing 4 outright. Nothing blocks a merge here, so the only value the gate has is whether a human believes a red — and half the genuine reds are already-fixed failures a human then debugs against a diff that cannot reach them. `scripts/stale-base-triage.py` answers that deterministically, running no tests: it reads the NEWEST `tekton/devrc-pytests` status per context, maps each failing test named in the description to the ONE file defining it, and emits the evidence pair — the PR head's copy of that file is byte-identical to the merge-base's (so this change never touched it) while `main` HAS moved it, in commits absent from the head. Verdict per PR: INHERITED / NOT EXPLAINED BY STALENESS / COULD NOT MEASURE. Measured live, read-only, over all 51 open PRs: 20 red heads, 3 INHERITED (#1286, #1194, #1038), 2 NOT EXPLAINED, 15 COULD NOT MEASURE, 4 broken-gate `error` rows counted on their own line. That non-zero 3 is the positive control for the count: a zero here would otherwise be indistinguishable from an instrument wired to nothing. 🔴 REPORT ONLY BY DEFAULT. `COMMENT_MODE_DEFAULT` is the literal `"off"`, pinned by `test_the_comment_mode_default_is_the_LITERAL_off` both as a value and as source text, so arming the PR-comment writer costs a visible line in the arming commit. Nothing was posted to GitHub at any point. Traps this repo has already paid for, each handled and each pinned by a test: * `/commits/{sha}/status` (SINGULAR) maps `error` onto `failure` — never read. Only the plural list endpoint, which carries no roll-up field at all. * the list is NEWEST-FIRST — folded on `max(created_at)`, order-independent. * `error` is a broken gate, not a code failure — its own summary line, never in the red total. * a SQUASH merge never makes a head an ancestor of its base; ancestry is asked only about the PR head, and every "this landed" claim is made by blob OID. * `git diff --quiet <ref> -- <path>` exits 0 when the path exists on NEITHER side; existence is proved with `git cat-file -e` first, pinned by an AST scan of the git invocations rather than a string search that its own comment would satisfy. * the 140-char description cap means a description can prove "at least one test failed and here is its name" but never "these are all of them" — so an INHERITED verdict requires `failed=N` to have survived AND to equal the number of names. That is conservative on purpose: dismissing a real red is the one error this tool must not make. * the test-name -> file mapping is searched, never guessed, with a distinct outcome for not-found, ambiguous, class-qualified, parametrised and cap-truncated names. Testing. 56 tests; RED at base `60194765` (the module cannot collect — the script does not exist there), GREEN at HEAD. Three enumerated mutation rounds under PYTHONDONTWRITEBYTECODE=1, mutants derived from the AST plus targeted textual ones: round 1 74 mutants / 15 survivors AND THE POSITIVE CONTROL SURVIVED, which is what found that `CONTEXT` was pinned nowhere; round 2 82 / 3 with the control killed; round 3 81 mutants, 78 killed, 0 survived, control killed. Round 1's survivors also found a real defect — the `on_main` half of the evidence pair was a guard that can never run (`commits_touching` already walks `merge-base..main`), so it is now measured and printed for verification rather than gated on, and `not in_head` moved into `prove_candidates` where a test can reach it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0
…red (#1524) * feat(ci-signal): name the commit that already fixed a PR's inherited red MEASURED, and it is the reason this exists: of the SIX genuine `tekton/devrc-pytests` failures in the whole post-#1458 window, THREE — half — were the same test on PRs 17, 40 and 40 commits behind `main`, all since merged, and that test passes on current `main`. An earlier sample had 8 of 8 failing open PRs 5-42 commits behind, with a rebase curing 4 outright. Nothing blocks a merge here, so the only value the gate has is whether a human believes a red — and half the genuine reds are already-fixed failures a human then debugs against a diff that cannot reach them. `scripts/stale-base-triage.py` answers that deterministically, running no tests: it reads the NEWEST `tekton/devrc-pytests` status per context, maps each failing test named in the description to the ONE file defining it, and emits the evidence pair — the PR head's copy of that file is byte-identical to the merge-base's (so this change never touched it) while `main` HAS moved it, in commits absent from the head. Verdict per PR: INHERITED / NOT EXPLAINED BY STALENESS / COULD NOT MEASURE. Measured live, read-only, over all 51 open PRs: 20 red heads, 3 INHERITED (#1286, #1194, #1038), 2 NOT EXPLAINED, 15 COULD NOT MEASURE, 4 broken-gate `error` rows counted on their own line. That non-zero 3 is the positive control for the count: a zero here would otherwise be indistinguishable from an instrument wired to nothing. 🔴 REPORT ONLY BY DEFAULT. `COMMENT_MODE_DEFAULT` is the literal `"off"`, pinned by `test_the_comment_mode_default_is_the_LITERAL_off` both as a value and as source text, so arming the PR-comment writer costs a visible line in the arming commit. Nothing was posted to GitHub at any point. Traps this repo has already paid for, each handled and each pinned by a test: * `/commits/{sha}/status` (SINGULAR) maps `error` onto `failure` — never read. Only the plural list endpoint, which carries no roll-up field at all. * the list is NEWEST-FIRST — folded on `max(created_at)`, order-independent. * `error` is a broken gate, not a code failure — its own summary line, never in the red total. * a SQUASH merge never makes a head an ancestor of its base; ancestry is asked only about the PR head, and every "this landed" claim is made by blob OID. * `git diff --quiet <ref> -- <path>` exits 0 when the path exists on NEITHER side; existence is proved with `git cat-file -e` first, pinned by an AST scan of the git invocations rather than a string search that its own comment would satisfy. * the 140-char description cap means a description can prove "at least one test failed and here is its name" but never "these are all of them" — so an INHERITED verdict requires `failed=N` to have survived AND to equal the number of names. That is conservative on purpose: dismissing a real red is the one error this tool must not make. * the test-name -> file mapping is searched, never guessed, with a distinct outcome for not-found, ambiguous, class-qualified, parametrised and cap-truncated names. Testing. 56 tests; RED at base `60194765` (the module cannot collect — the script does not exist there), GREEN at HEAD. Three enumerated mutation rounds under PYTHONDONTWRITEBYTECODE=1, mutants derived from the AST plus targeted textual ones: round 1 74 mutants / 15 survivors AND THE POSITIVE CONTROL SURVIVED, which is what found that `CONTEXT` was pinned nowhere; round 2 82 / 3 with the control killed; round 3 81 mutants, 78 killed, 0 survived, control killed. Round 1's survivors also found a real defect — the `on_main` half of the evidence pair was a guard that can never run (`commits_touching` already walks `merge-base..main`), so it is now measured and printed for verification rather than gated on, and `not in_head` moved into `prove_candidates` where a test can reach it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0 * fix(stale-base-triage): the verdict was a function of the failing test's NAME LENGTH (F2) `names_provably_complete` could only prove completeness by reading `failed=N` out of the status description. `failed=` is the LAST field in the gate's banner and GitHub caps a description at 140 BYTES, so whether it survives is decided by how long the failing test's name is — and on the three PRs this tool's requirement was measured on (#1454, #1462, #1499) it did not survive. All three resolved to COULD NOT MEASURE with the correct answer already in the row. The tool fired on 0 of the 3 cases that justified it. The fix is a SECOND route, and it is a proof rather than a heuristic. `scripts/run-tests.sh` GUARD 4 defines collected = passed + skipped + failed + errors + xfailed + xpassed TOT_FAILED += failed + errors so `collected - passed - skipped == failed + xfailed + xpassed >= failed`, and names are a subset of the failures, so `len(names) <= failed <= derived`. When `derived == len(names)` the inequality is squeezed shut and completeness is PROVEN. `collected=`, `passed=` and `skipped=` all precede `failed=`, so they are exactly the fields that survive the cut. The direct `failed=N` route is kept and still tried first, because it is the stronger claim. 🔴 THE BOUND IS AN OVER-COUNT, NEVER AN UNDER-COUNT, and that direction is the whole safety argument: an under-count would certify a completeness the row does not have and dismiss a real red. `xfailed`/`xpassed` can only inflate it, and `_TOTALS_RE` demands the three fields ADJACENT AND IN ORDER so that a number the cap cut SHORT cannot shrink it — `collected=` and `passed=` are each proven intact by having matched banner text after them, and `skipped=`, the only one that can be short, subtracts LESS and so errs toward withholding. MEASURED, read-only, 2026-09-11 (provably-complete rows / red heads): population old new 100 most-recent closed PRs 5/28 18/28 - the three justifying PRs #1454 #1462 #1499 0/3 3/3 58 open PRs 4/26 7/26 - #1518, #1515 (9 commits behind, worked) 0/2 2/2 SOUNDNESS CONTROL on the same real rows: 17 carried BOTH a visible `failed=N` and a derivable bound; the two agreed EXACTLY on all 17, with zero under-counts. Genuine multi-failure reds stay withheld — #1177 derives 39 against one name, #1280 failed=5, #1440 failed=3, all still COULD NOT MEASURE. The coupling to `run-tests.sh` is pinned mechanically rather than asserted: `test_the_run_tests_collected_arithmetic_this_derivation_rests_on_is_pinned` reads the `collected=$((…))` and `TOT_FAILED=$((…))` expressions out of the shell source and checks the CONTAINMENT relation, so renaming the shell locals is fine and changing WHAT IS SUMMED is not. What it cannot check — that `xfailed`/`xpassed` are non-negative — is stated in the test, not asserted. Prior art is now cited in the header: `main-status-watch.py`'s `screen_all_known_flakes` is the same completeness gate in 15 lines, and its "would have fired ZERO times on 100 commits" is the same `failed=N` route being eaten by the same cap. This file's contribution is the derived route and the blob/commit evidence half; the gate itself is prior art and says so. Red/green matrix: 13 of the 14 new tests fail at 5ad08e5 and pass at HEAD; the 14th is labelled in its own docstring as an invariant guard, not regression coverage. Suite: 56 -> 70 tests, all green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0 * test(stale-base-triage): close the three gaps the F2 mutation sweep found Round 1 of the sweep on the new completeness logic killed 10 of 15. Four of the five survivors were real gaps in the tests, not in the code: * M14 — reordering `passed=`/`skipped=` in `run-tests.sh`'s TOTAL banner SURVIVED, because the pin searched the whole FILE and `run-tests.sh` prints TWO banners (a full-run one and a SCOPED one). Mutating one was satisfied by the other. The pin now requires EVERY banner line to carry the ordered fields, and asserts it found at least two. * M11 — spelling the `skipped` group `(\d*)` SURVIVED: no fixture fed a row cut exactly on the `=`, which is what a cap landing one character early produces and which makes `int("")` RAISE on the reporting path. Added as a fixture. * M10 — replacing the underivable-row refusal with a bound of 0 SURVIVED, because `names` is non-empty by then so 0 can never equal it and the VERDICT is identical. Only the operator-facing reason distinguishes them, and the two refusals call for different next steps, so both reasons are now pinned as whole strings. * M5b — the naive three-separate-searches spelling of the totals regex is now driven explicitly and is killed by the existing out-of-order fixture. Two survivors are argued EQUIVALENT rather than fixed, and are labelled as such in the source: * M5 — matching the three fields with `.*` between them instead of adjacently. `.*` still requires the ORDER, and a single-banner description contains exactly one of each field, so no input this parser can receive distinguishes them. The adjacency is kept for the truncation argument it encodes. * M13 — `run-tests.sh` SHRINKING what it sums into `failed=` keeps the derived value an UPPER bound, so the derivation stays sound. The pin checks CONTAINMENT on purpose; the added comment says so, and M15 (a term LEAVING `collected` — the change that does break it) is killed by the same pin. Round 2 over the fixed tree: 6/8 killed, the two above surviving as argued, no new finding — so the ladder ends there. Both controls held in both rounds (unmutated tree green; `COMMENT_MODE_DEFAULT = "on"` killed by four tests). Run under PYTHONDONTWRITEBYTECODE=1, each edit verified to have changed the file, verdicts read from the runner's own result lines. Suite: 70 -> 71 tests, all green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0 * refactor(ci-signal): one rule, one place — and the two copies had already diverged (D6) `classify` in `stale-base-triage.py` was BYTE-IDENTICAL to the one in `main-status-watch.py`. `_FAILING_RE`, `_FAILED_COUNT_RE`, `parse_failing_names` and `parse_failed_count` were duplicated AND already divergent — only the newer copy stripped the `TOTAL` truncation fragment, and only the newer copy folded statuses by timestamp instead of by array position. The disagreement is the finding; consolidating is what made it audible. All five now live in `scripts/lib/ci_status.py`, with `derived_failure_upper_bound` beside them. Both files import it by path. What stays in each consumer is POLICY — which context may speak, which names are known flakes, what completeness means, what to do about a red — because the two answer those differently. 🔴 `main-status-watch.py` IS LIVE ON BOTH HOSTS, every 10 minutes, so each divergence was reconciled deliberately rather than by taking whichever copy was newer: * THE FOLD. The live copy kept the FIRST row per context, which is correct only because GitHub returns the array newest-first — an assumption about someone else's response ordering, load-bearing, unstated and untested. (⚠ It is NOT the "yields the OLDEST post" bug: first-wins over a newest-first array yields the NEWEST. The defect is order-DEPENDENCE, not a wrong row today.) It now folds on `max(created_at)`, which agrees with first-wins on GitHub's current ordering and stays right if that changes. With no `created_at` anywhere every stamp compares equal and the FIRST row wins — the old rule, as the fallback. `test_newest_per_context_is_ORDER_INDEPENDENT_which_first_wins_was_not` reimplements the old rule and shows it returning the `pending` row on the order it does not expect, beside the new one right on both. * THE CONTEXT FILTER moved out of the fold and into `commit_verdict`, where the policy belongs, and BEFORE the emptiness check — a commit carrying only a foreign pipeline's rows must have NO verdict, never green, or an unrelated green would close an open red episode. Pinned by a new unit test beside the existing end-to-end one. * THE isinstance SKIP. The live copy deliberately had none, arguing the arm was unreachable (its walk validates every row is a dict and names the commit if not) and that silently dropping a row could drop the red. The shared function keeps the triage tool's skip: still unreachable for the watcher, unchanged behaviour for the triage tool. Neither contract moves; the module says why. * THE FRAGMENT STRIP is now shared, and is provably INERT for the watcher: a row cut at `| TOTA` was necessarily cut before `failed=N` too, so its screen — which also demands the count — refuses either way. * THE DERIVED COMPLETENESS ROUTE is deliberately NOT adopted by the watcher, and the file now says why: there, proving completeness makes the tool SPEAK; here it makes it stay SILENT about a red. Widening a silence over main's only automated detector is a change to make on purpose with its own measurement, not a free win inherited from a sibling. Duplication is now refused mechanically: `test_the_shared_predicates_are_NOT_re_declared_in_either_consumer` fails on a local `def` of any of the five, in either file, and carries a positive control so the scan cannot pass vacuously. TWO PRE-EXISTING TESTS BROKE, both for the right reason, and both were fixed rather than weakened: one used `def classify(...)` as its positive control for the docstring-stripper (the function is simply elsewhere now — it uses `commit_verdict` instead), and one copies the script to a tmp dir and runs it, which has no sibling `lib/` (it now passes PYTHONPATH, and says why). DEPLOYMENT. The unit runs the script straight out of the checkout (`ExecStart=… %h/workspace/devrc/scripts/main-status-watch.py`), so a `git pull` delivers both files and no home-manager switch is involved. A run landing in the instant between the two files arriving fails to import; the unit is a timer-driven oneshot with no `OnFailure=` toast and the next poll is 10 minutes away, so that window costs one skipped poll. ⚠ `X-Restart-Triggers` in `nix/home.nix` still names only `main-status-watch.py` and was deliberately NOT edited — for a timer-driven oneshot that re-reads the file every run it is cosmetic, and touching `home.nix` here would widen this change's blast radius for no behavioural gain. `test_main_status_watch.py`: 96 -> 99 tests, all green. `test_stale_base_triage.py`: 71 tests, all green. Both suites: 170 green. No live-system change was made: the deployed copy is the base clone on `main`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0 * refactor(stale-base-triage): delete --fetch and --json; keep --sweep, --help and the env seams (D1-D5) The six deletion candidates were assessed against the BROKEN tool, so each was re-measured after F2 rather than acted on. Decisions, with the evidence: DELETE `--fetch`. It was the only git subcommand in this file that WRITES, in a tool whose stated safety property is that it writes nothing, and `refs/` lives in the COMMON git dir — a worktree gives zero isolation there, so "namespaced under refs/stale-base-triage/*" was a weaker claim than the one the header makes. MEASURED: this repo's PRs are same-repo branches and `origin`'s refspec is `+refs/heads/*:refs/remotes/origin/*`, so an ordinary `git fetch origin` already brings every head — 58 of 58 open PR heads resolved in the base clone with no `--fetch` anywhere. It bought nothing and cost the tool its absolute safety claim. The UNMEASURED message now says `git fetch origin`. The guard got STRONGER, not weaker: `test_only_refs_under_its_own_namespace_are_ ever_written` (a relative claim) is replaced by `test_no_git_subcommand_that_ WRITES_is_ever_invoked`, which enumerates the read-only subcommands as an ALLOWLIST off the AST — so a subcommand nobody enumerated is a write by default, and `remote` is admitted only in its `get-url` form. DELETE `--json`. No consumer: a whole-repo search for `stale-base-triage` / `STALE_BASE_TRIAGE` returns this script and its test file, nothing else — no unit, no skill, no nix, no caller. F2 does not change that; it makes the RENDERED sweep useful, not a machine payload. It cost a module-level `_JSON_MODE`, a `say()` indirection on every human line in the file, and a second exit path computing `n_inherited` its own way. KEEP `--sweep`, and it is now the primary mode. Its rationale is exactly what F2 reversed: MEASURED read-only over 58 open PRs, the sweep surfaces 7 provably complete reds where it surfaced 4, and the three it gained (#1518, #1515, #1450) are 9-94 commits behind and actively worked — not the 259/357/508-behind abandoned ones the old gate happened to let through. KEEP the self-parsing `--help`. What it prints is the `#` header — the env ledger, the exit codes, the measured rationale — which argparse cannot see. The alternative is copying the ledger into an epilog, i.e. reintroducing the duplication the previous commit just removed, and the ledger is two-way pinned against the code that reads it. KEEP all four env seams and the ledger test. `STALE_BASE_TRIAGE_GH` and `_REPO` are the seams the test harness points at a stub — they are why no test in this file can reach real GitHub. `_BUDGET` is the only way to drive the budget guard to zero. `_COMMENT_MODE` is the arming seam and FAILS CLOSED across seven spellings. Each has a concrete in-repo consumer; none is speculative. `COMMENT_MODE_DEFAULT` is untouched: still the literal `"off"`, still pinned both ways (value and source text). Nothing was posted to GitHub; every live run in this work was read-only with mode=off. VERIFIED AGAINST THE ORIGINAL SYMPTOM, on live data, not inferred from the code: old code (5ad08e5), `--pr 1518`: VERDICT: COULD NOT MEASURE, rc 0 this tree, `--pr 1518`: VERDICT: INHERITED, rc 10 and a four-PR read-only sweep returns INHERITED for #1518/#1515/#1450 with the fix commits named, while #1177 (derived 39 against one name) stays COULD NOT MEASURE with the HINT line. ⚠ The deleted-flag guard was SPELLED on its first draft and failed on the header paragraph that explains the deletion — the same trap `_git_subcommands` is AST-based to avoid. It now reads `add_argument` literals, module-level assignments and string constants off the AST, so prose can neither satisfy nor break it. Suite: 71 tests, all green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0 * docs(stale-base-triage): the cap is 140 BYTES, and the two files disagreed about it `scripts/lib/ci_status.py` said BYTES and `stale-base-triage.py` said CHARACTERS, in prose describing the same boundary. Bytes is what was measured: every real row sampled on this repo is 138 CHARACTERS, which is 140 bytes once the `—` in `FAILED: pytests —` is counted as the three UTF-8 bytes it is. A 140-CHARACTER cap would have produced 138-byte, 140-character rows, and none was seen. ⚠ WHAT THAT MEASUREMENT DOES NOT SETTLE, now said in the file rather than implied: whether the byte cut is GitHub's or the posting pipeline's own truncation. Every row sampled carries exactly one multi-byte character, so the two hypotheses were never separated — a row with two em-dashes would do it, and none exists. The boundary is measured; its owner is not. It matters because a fixture built on a character cut lands two bytes late and quietly leaves `failed=` readable — i.e. it would exercise the DIRECT route while claiming to test the derived one, which is a green proving nothing. `cut_like_github` in the test file cuts on bytes for exactly that reason and `test_CONTROL_the_synthetic_rows_land_on_the_real_140_BYTE_boundary` asserts both lengths. No behaviour change. Suite: 71 tests, all green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0 * test(no-real-launchers): the consolidation made main-status-watch.py a `home-manager` hit — acknowledged and PINNED `test_every_hazardous_binary_the_scripts_reach_is_stubbed_or_acknowledged` went red on the D6 commit, and it was right to. `launcher_scan.hazard_hits` is a TEXT scan over `scripts/` and says so in its own docstring — a prose mention counts, deliberately and fail-closed. The shared-module import comment explains that the systemd unit runs this file straight out of the checkout, so a `git pull` delivers both files and no home-manager switch is involved, and that sentence is exactly why importing a sibling module is safe here. It is the FIFTH file of this shape. 🔴 RE-JUSTIFIED, NOT REWORDED, which is this repo's stated convention and is written into the entry above the table — rewording to dodge the scanner removes the sentence and keeps the risk. 🔴 AND PINNED, because that table entry says in its own words that an acknowledgement "would otherwise blind the guard" — and it has been blinded before: adding a real `home-manager switch` to a script whose name was already acknowledged for other reasons produced a real launch with 54 guard tests green. So two new tests carry this row: test_main_status_watch_SPAWNS_these_argv0_AND_NOTHING_ELSE — an AST walk asserting the spawn argv[0] set is exactly {git, <computed>, <not-a-list>}, GROWS-OR-SHRINKS. Both opaque entries are named rather than waved at: `<computed>` is `[gh, "api", path]` behind the MAIN_STATUS_WATCH_GH stub seam, `<not-a-list>` is `trigger_deadman`'s `cmd`, whose production literal is pinned exactly — and refused a `--force` — by an existing test. The comment says outright that this pin is not the whole story on its own. test_home_manager_is_MENTIONED_but_never_SPAWNED — both halves of the acknowledgement's claim: the mention must still EXIST (or the row has outlived its sentence) and it must remain a MENTION, asserted against the argv[0] set AND against `_code_only()`, which strips comments and docstrings by AST. BOTH CONTROLS WATCHED, in both directions: * injecting `subprocess.run(["home-manager", "switch"])` fails BOTH new tests, each with its own message — not a different guard's; * rewording the mention to "no HM switch" fails the mention-must-exist half AND the ledger's own file-set pin, because the row then over-claims. ⚠ FOUND BY RUNNING THE WHOLE `scripts/tests` TARGET, not by the change-scoped mapping: `scoped-tests.sh` selected only the two suites named in the diff, and this guard reads every file under `scripts/` regardless of what changed. A change that only adds a COMMENT can go red here. scripts/tests/{test_no_real_launchers,test_main_status_watch,test_stale_base_ triage}.py together: 252 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0 * fix(ci-signal): the consolidation moved code out from under the citation guard, and the first citation written into the new module dangled ROUND 1 audit findings on #1524. Six filed, six addressed, plus one the same fixture exposed in the sibling parser. F1 — `scripts/lib/ci_status.py` cited `test_newest_per_context_skips_a_malformed_row_rather_than_raising`, which existed nowhere, and the arm it claimed was guarded was unguarded: deleting `if not isinstance(row, dict): continue` SURVIVED all 172 tests. The deeper cause is the guard, not the citation — both copies of `test_every_test_this_script_names_actually_exists` are scoped to ONE script file, so neither scanned the shared module the consolidation created. The stale-base-triage copy now FOLLOWS its script's `scripts/lib` imports (derived, not enumerated, so a second shared module is covered the day it appears) with a positive control that fails if the import-following reaches nothing; the main-status-watch copy names the owner rather than implying coverage it lacks. The cited test is now written, and it kills the mutant. F2 — `_git_subcommands` silently DROPPED what it could not constant-fold, so its stated rule ("an unknown git subcommand is a write by default") was wider than what it enforced: a computed head was ABSENT, not DENIED. It now emits `<computed>` / `<not-a-list>` / `<empty-argv>` / `<no-argv>` sentinels — the shape `test_main_status_watch.py`'s `_spawn_argv0_literals` already used 200 lines away in this same PR — reads argv from the SECOND POSITIONAL rather than "whichever argument is a list", and the allowlist is a predicate a test drives over synthetic source. The bare-`subprocess.run(["git", "push"])` hole the finding also names is closed separately: every git spawn must be lexically inside the one `_git` helper, and the spawn argv0 set is pinned GROWS-OR-SHRINKS. F3 — the `gh` write path was pinned by the literal word POST, which `gh api -X PATCH`, `-X PUT`, `--method DELETE`, `gh pr comment` and `gh api -f` (a POST with no method flag at all) each walk straight past. It now asserts the STATE: an AST ledger of the `gh` verbs the source can reach (`("api",)` and the one `("api", "-X", "POST")`), plus a receipt classifier that decides read-vs-write from the method and the field flags. The harness stub also had to start recording ONE LINE PER INVOCATION — a comment body contains newlines, so `calls()` was returning prose fragments as if they were argvs, which was invisible while the only question asked of a line was whether it contained POST. F4 — `TOTALS_RE` was unanchored and `re.search` takes the FIRST match, so a description carrying an earlier per-target row derives a bound of 2 where the run's own `failed=` is 27. That is the UNDER-count direction, the one error the module explicitly forbids, and `run-tests.sh` really does print such rows (`FAIL <dir> (collected=… passed=… skipped=… failed=… errors=…)`) before the TOTAL banner. Both parsers are now anchored on `TOTAL` — `FAILED_COUNT_RE` had the identical defect and is reachable from the same string, which the audit did not file; fixing one and not the other would have left the same certification hazard on the direct route. Both anchors err toward REFUSING. No reachable exploit is claimed: what assembles the description is the posting pipeline in homelab-talos and cannot be pinned from here. F5 — the "140 BYTES" reconciliation (02bc6f8) had not finished; `main-status-watch.py` said "140 characters" nineteen lines above its own "THE 140-BYTE CAP". Swept all five sites including the test NAME, and replaced the prose with a control: the two real truncated rows are 138 chars / 140 bytes, asserted. The honest unknown is kept intact — whether the cut is GitHub's or the pipeline's is still undetermined, because every sampled row carries exactly one multi-byte character. F6 — `sys.path.insert(0, …/scripts/lib)` prepended the whole shared-module directory ahead of the standard library for every later import, including the lazy `import traceback` on the unattended-crash path. `append` removes the class. M2 (deleting `if not ctx: continue`) is NOT filed: an `""` key cannot match either caller's context, so it is an equivalent mutant. Evidence. Suites: test_stale_base_triage 71 -> 78, test_main_status_watch 101 -> 102. 395 passed across the four affected files (the only files in the repo that reference any changed path; nothing else imports `ci_status`). ⚠ scoped-tests.sh is NOT evidence here and its PASS is withdrawn: it printed RESULT: PASS against the PRE-REBASE base, and on current main it correctly REFUSES (exit 4) because the diff touches `scripts/lib/**` — scoping a shared module selects the files that NAME it and drops every target reaching it by import. `scripts/gate.sh --tier both` is what covers that gap. RED at pr1524-head: the F4 test fails with derived=2 against the real 27. Mutation: 19 mutants, 18 KILLED, 1 SURVIVED — and the survivor is M4 (first-match -> last-match), which is EQUIVALENT once anchored because `TOTAL collected=` occurs at most once on any banner this repo can post. That is asserted in the test rather than left unexplained, and reachability was confirmed separately: breaking the same statement kills 39 tests. Positive control KILLED in the batch; swept under PYTHONDONTWRITEBYTECODE=1 with __pycache__ removed between mutants. COMMENT_MODE_DEFAULT is unchanged ("off"), both pins intact, nothing posted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0 * test(stale-base-triage): the write screen read a comment BODY as flags, and three guards were never exercised ROUND 2's five findings. No payload change — every one of these is a weakness in a guard this PR itself wrote. F-R2-1 `gh_write_calls` parsed argument VALUES as flags, last-wins. Measured on the previous version, same call shape as the real `post_comment`: -X POST /…/comments -f body=hello -> WRITE (control) -X POST /…/comments -f body=switch to --method GET now -> read (WRONG) -X POST /…/comments -f body=use -X GET here -> read (WRONG) /repos/o/r/x --input=payload.json -> read (WRONG) /repos/o/r/x -fbody=hello -> read (WRONG) All five now classify as writes. The method is FIRST-wins and is scanned only up to the first field flag — everything after one is body text — while field flags are scanned to the end, in both the separated and the ATTACHED spelling `gh` accepts. It does NOT stop at the first `/`-prefixed positional the way `_gh_argv_shapes` does: `gh api /repos/o/r/x -X POST` is legal and would read as clean, which is the one direction that matters. Reachable, so it is pinned end to end as well: `comment_body` interpolates a `main` commit SUBJECT, and the new test drives a hostile one through the real script. F-R2-2 the spawn pin folded every computed argv0 into one set member, so `binary = "git"; subprocess.run([binary, "push"])` appended to the real source passed all four guards unchanged. Computed spawn heads are now asserted by NAME (every one is the `gh` seam) and by COUNT (cross-checked against `_gh_argv_shapes`), and the violating source is driven over synthetically. F-R2-3 `_spawn_argv0`'s `<not-a-list>` arm — the net for `os.system("git push")` — was never exercised; the mutant replacing it with `pass` SURVIVED. Driven over synthetic source now, like `_git_subcommands`' sentinels. F-R2-4 both scripts' `sys.path.append` carried a 🔴 comment and no guard; reverting either to `sys.path.insert(0, …)` SURVIVED. Pinned in each file's own test, structurally. F-R2-5 five sites named GitHub as the cutter of the 140-byte description cap while the new control says the measurement cannot distinguish GitHub's cut from the pipeline's. The measured half (BYTES, not characters) is unchanged everywhere; only the attribution is dropped, and each site now points at `test_CONTROL_the_real_truncated_rows_land_on_the_BYTE_cap_not_the_CHAR_cap`. `GITHUB_DESCRIPTION_BYTE_CAP`/`cut_like_github` renamed to `DESCRIPTION_BYTE_CAP` /`cut_at_the_byte_cap` for the same reason. The unknown is NOT resolved in either direction. MUTATION TABLE (isolated `cp -a` copy, `PYTHONDONTWRITEBYTECODE=1`, `__pycache__` cleared per mutant, verdicts counted from pytest's own result lines; baseline in the copy 187 passed / 0 failed): mutant verdict killed by CONTEXT literal -> devrc-NOPE (control) KILLED test_the_context_is_PINNED_… method last-wins again KILLED …reads_an_ARGUMENT_VALUE_as_DATA… body-stop removed KILLED …reads_an_ARGUMENT_VALUE_as_DATA… `--input=` leaves the prefix tuple KILLED test_EVERY_field_flag_… attached short forms leave the tuple KILLED test_EVERY_field_flag_… M5 `<not-a-list>` arm -> pass KILLED …ARGV_IS_NOT_A_LIST_is_a_SENTINEL… M6 sbt sys.path.insert(0, …) KILLED …APPENDED_to_sys_path… (sbt) M8 msw sys.path.insert(0, …) KILLED …APPENDED_to_sys_path… (msw) second computed spawn in the real file KILLED …SECOND_computed_spawn… + the pin non-Name head folded into "gh" KILLED …SECOND_computed_spawn… Each died on its own guard's assertion, naming the row or expression mutated — not on a neighbouring guard's error. `method last-wins` SURVIVED on the first sweep: with the body-stop in place the two rules only differ on a value carried by a NON-body flag, so a `-H` row was added and it dies there. Suites: `test_stale_base_triage.py` 84 + `test_main_status_watch.py` 103 = 187 passed (180 at 3aec90b, +7 new tests). `test_no_real_launchers.py` + `test_audit_dispatch.py`, the other two files naming these scripts, 215 passed. `COMMENT_MODE_DEFAULT = "off"` and both pins untouched; nothing posted anywhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Zachary Lowden <dev@vetr.com>
The siting gap
tekton/devrc-pytestsintermittently failstest_subsystem_store_api.py::TestARefusedWriteIsIndistinguishableFromAnAbsentOne::test_POSITIVE_CONTROL_the_APPEND_comparison_CAN_see_the_difference,including on docs-only PRs. The mechanism is already diagnosed in
scripts/ci-repro/README.md:server.py:_replace_bytesfsyncs the file and then itsparent directory inside the request, before the response is written; under disk
contention on the single node
devrc-ciis pinned to, one fsync exceedsHANG_TIMEOUT(60.0) and the client raises
TimeoutError. The suite's own classifier says sounprompted:
MECHANISM = SERVER_BLOCKED_IN_FSYNC.A fix for that shipped in #1211 / #1219 / #1239 —
scripts/testlib/store_siting.py'sstore_root(tmp_path), which sites the store on tmpfs when one is usable and falls backto
tmp_pathotherwise. It was never applied to the site that fails.At
origin/mainthis file had 5 call sites ofstore_siting.store_rootand 18that open-coded
tmp_path / "store". The failing class was one of the 18: its_phaseshelper built
root = tmp_path / "store", unsited, on disk unconditionally.Why that one test and not its siblings: within the class,
test_POSITIVE_CONTROL_…isthe only test that gets a
200 appended. The others all assert a 404 — refused orabsent — which is answered before any write and never reaches
_replace_bytes. So it isthe only test in the class that executes the two in-request fsyncs.
Corroborating, from the CI traceback already recorded in
ci-repro/README.md: thestalling write targets
/tmp/nix-build-devrc-pytests.drv-0/pytest-of-nixbld13/pytest-0/popen-gw3/…/store— atmp_path-derived path, not adevrc-store-*tmpfs holder.What this changes
1. All 18 open-coded sites now take their root from a
sited_rootfixture —store_siting.store_rootwith a test's lifetime.A fixture rather than 18 inline
withblocks, and the reason is lifetime, notconvenience. Several of the sites build the root inside a helper whose return value
is used throughout the test body:
_phasesreturns a pair ofpresent()/absent()closures,
_entry_pathand_twin_heading_entryreturn a path. Awithwrapped aroundthe helper call tears the store down before a single assertion runs. A
withwrappedaround each whole test body is 30 blocks of re-indentation — which is the exact shape of
the scripted conversion this repo already reverted once (it "silently skipped 9 of 19
signature edits while its own assertion still passed", per the ledger's own comment).
Taking the context manager as a fixture makes its lifetime the test's, per site, with
nothing to get wrong per site.
The one behaviour difference from an inline
withis written into the fixture ratherthan glossed:
store_rootskips its budget check when the body is raising, and pytestdoes not throw a test's failure into a fixture generator, so on this path the check runs
even for a failing test. That is already true of
storeandscoped_store— it is thestanding cost of the idiom here, not something introduced by this change.
2. The guard is widened from one fixture to a census.
test_store_siting_ledger.pyalready carried an AST ratchet, but as a count:_DISK_ROOTED_SITES = 33. A count cannot distinguish "one site was migrated and one newone was written" from "nothing happened", and its own history is a string of arguments
about whether the number moving meant new debt or a wider predicate.
It is now an enumerated set:
_DISK_ROOTED_ALLOWLIST, keyed<Class.function> :: <unparsed expression>(never a line number — that moves when anunrelated docstring gains a sentence), each entry carrying its reason.
test_the_disk_rooted_census_matches_the_allowlist_EXACTLYasserts set equality in bothdirections, and names the offending site in the failure message.
15 entries remain. They are the population that was never spelled "store" —
served,served-elsewhere,stage,absent,big,at-the-cap,unambiguous— and every onewas checked to issue no write verb (no
post_bullet, nocreate_put, nomethod="PUT"/"POST"), so no request reaches_replace_bytesand nothing fsyncs insidea request. They are still disk-backed, and that is recorded rather than hidden: siting
them is the obvious follow-up, deliberately not folded into a change scoped to the
write-path sites that were failing the gate.
A second arm,
_SITED_STORE_ROOT_CALLERS, pins the other side of the relationship — theset of
store_siting.store_root(...)callers — so a deleted siting is red too. Themutation matrix below shows that arm is not redundant.
The existing behavioural positive control is kept, and its class docstring is corrected.
It claimed a fixture falling back to disk "EVERYWHERE" would be caught, while its control
took one fixture (
store). That is one site wide; 18 further roots sat behind thatgreen, including the failing one.
sited_rootnow has its own behavioural control.Red before green
scripts/ci-repro/slowfsync.cgains an opt-inSLOWFSYNC_SKIP_TMPFS=1mode, becausethe default mode cannot measure a siting fix: it interposes on
fsync(2)in libc, soit stalls tmpfs too, and both arms of the comparison go red. Measured:
The new mode passes a
TMPFS_MAGICfd through without consuming the one-shot latch(a pass-through that spent it would turn "the store moved to tmpfs" into "the shim ran
out of ammunition"), and a failing
fstatfs()stalls rather than skips. Defaultbehaviour is unchanged, so every measurement already in
ci-repro/README.mdstill holds.All rows below are the same single test,
-sso the shim's own stderr is visible:origin/main5 passed in 8.76sorigin/main1 failed in 65.52s,TimeoutErroratsocket.py:720,MECHANISM = SERVER_BLOCKED_IN_FSYNC, stall at_replace_bytes:2073 os.fsync(fh.fileno())origin/mainSKIP_TMPFS=11 failed in 63.96s, same, stall linefs magic=0xef53SKIP_TMPFS=11 passed in 3.67s, exactly twopass-through … magic=0x1021994lines (the file and its parent dir —_replace_bytes's pair), latch untouched1 failed in 63.69s, stall linefs magic=0x1021994SKIP_TMPFS=1, siting forced to fall back1 failed in 64.29s, stall linefs magic=0xef53The last two rows are what make the fourth mean anything, and they are different claims.
Row 5: the shim can still kill this test on the branch, so the green is not "the
reproducer no longer reaches this code". Row 6: forcing
store_siting.tmpfs_dir()toNonereproducesmain's red exactly, so the green is the siting and not some otheredit on the branch.
The
origin/mainrows were taken withtest_subsystem_store_api.py,testlib/store_siting.py,test_store_siting_ledger.pyandserver.pybyte-identicalto
origin/main(onlyslowfsync.cdiffered). Those four files are also identical atfbc4dd39and at the current base50b384c0, so the baseline holds for both.Mutation of the census guard
Run with
__pycache__cleared between mutants (a same-length edit inside one second isinvisible to CPython's mtime-in-seconds + size cache and scores SURVIVED without running):
tmp_path / "store"sited_rootde-sited totmp_path / "store"sited_rootde-sited totmp_path / "holder"Each died on its own message, not on some other test's error. The census message
reads:
Mutant 3 is the one that earns the second arm: a de-siting inside a fixture, under a
directory name outside
_ROOT_NAMES, is invisible to the census — its flow arm stops atthe fixture's scope boundary, its on-sight arm only recognises
store/src— so_SITED_STORE_ROOT_CALLERSis what catches it. It also falsified my own first draft ofthat comment, which had claimed the blind spot was "a fixture" — mutant 2 shows it is
narrower than that.
🔴 CORRECTED: this said
_SITED_STORE_ROOT_CALLERScatches mutant 3 "only", and thatis false wherever a usable tmpfs exists. The
otherscolumn above is a one-FILErun of the 23-test ledger module, so "every other guard in both files green" was
extrapolated from a measurement that never opened the other file. Re-run over BOTH
files on a tmpfs host, mutant 3 gives 2 failed, 781 passed:
The behavioural control reads the fstype of
sited_root.parentand finds the disk. Thecensus stayed green, so the blind spot itself is re-confirmed by the same run.
The arm still earns its place, for a narrower reason than "only": the behavioural
control SKIPS where there is no usable tmpfs — absent, not tmpfs, under
_MIN_FREE_BYTESfree, or unwritable — which this PR says the gate may well be. So thetrue claim is the only guard that can see it on a machine with no usable tmpfs, and
that is now what the failure message and the comment above the set say.
Also re-measured — and a claim of mine that the measurement falsified
_ROOT_NAMES's comment said seven sites were counted only by the on-sightdirectory-name arm, and named them. All seven were among the 18 migrated. Re-measured by
running the census with
_ROOT_NAMES = {"store", "src"}and again with it empty:15 both times, identical keys — it contributes nothing on this file now.
My first replacement comment then said it was kept because "two of this file's own probe
tests depend on the on-sight arm", naming
test_the_site_index_does_not_key_on_the_DIRECTORY_being_spelled_store. That wasfalse, and running it is what showed it: every probe in that test hands its path to
running(...), so the flow arm counts it and the spelling never decides anything.Measured by setting
_ROOT_NAMESto empty and running every argument-free test in themodule — zero behaviour changes.
So the comment now says the true thing: the set is inert everywhere that is measured
today, it is kept on an argument (the flow arm's
_ROOT_CONSUMERScloses renames butnot growth into a new consumer name, and the on-sight arm is the only path that does not
go through that set), and the closing condition for deleting it is named. The
contribution is now measurable in one command instead of asserted in prose, which is what
caught the false sentence.
Gate
Authoritative tier — the one Tekton runs,
nix build .#checks.x86_64-linux.pytests,on this branch rebased onto
4e26ec9a:Counted from the runner's own per-target lines, not from an exit code. The 2 skips are
the pre-existing pinned real-Postgres ones;
scripts/testsskipped 0, which meansboth tmpfs behavioural controls — the existing
storeone and the newsited_rootone —ran rather than skipping.
/dev/shmis usable in the nix sandbox.Node tier,
nix build .#checks.x86_64-linux.nodetests, built separately (neverconcurrently with the pytest one — a combined invocation contends on the nix store and
produces false failures):
RESULT: PASS (exit=0),NIXBUILD_RC=0,# fail 0on everysuite. This diff touches no JS; the run is here because "both tiers" is the standing rule,
not because it could have been reached.
Dev-host tier,
scripts/gate.sh --tier pytest: inconclusive, and I am not countingit as a pass. It got through
scripts/tests(13519 passed, no failures) and then therunner was SIGTERMed at ~65 min, so it printed
RESULT: FAIL (exit=143)— a killed run,not a test failure. That run also showed one failure in
scripts/dl-router/tests/test_store.py::test_six_writers_with_a_tiny_busy_timeout_still_land_every_row(
OperationalError('database is locked')against a deliberate 5 msbusy_timeout).That test is unreachable from this diff, and the discriminating control is above: the
same file, same test, 1020/1020 passed in the sandbox tier. A second agent was
running a full
run-tests.shin~/workspace/devrc-gate-baseconcurrently, which isexactly the load-contention flake
CLAUDE.mdalready documents for dl-router SQLite.Not verified
/tmpis ext4 and/dev/shmis tmpfs. The gate's sandbox may have no usable tmpfs, in which casestore_rootfalls back to disk by design and this changes nothing there.store_root's docstring enumerates the five ways that happens.write verb reaches them), not from a runtime trace.