Skip to content

fix(tests): site the 18 remaining store roots off the contended disk, and census the set - #1458

Merged
ZacxDev merged 12 commits into
mainfrom
fix/site-every-store-off-the-contended-disk
Sep 10, 2026
Merged

ZacxDev merged 12 commits into
mainfrom
fix/site-every-store-off-the-contended-disk

Conversation

@ZacxDev

@ZacxDev ZacxDev commented Sep 9, 2026

Copy link
Copy Markdown
Member

The siting gap

tekton/devrc-pytests intermittently fails
test_subsystem_store_api.py::TestARefusedWriteIsIndistinguishableFromAnAbsentOne::test_POSITIVE_CONTROL_the_APPEND_comparison_CAN_see_the_difference,
including on docs-only PRs. The mechanism is already diagnosed in
scripts/ci-repro/README.md: server.py:_replace_bytes fsyncs the file and then its
parent directory inside the request, before the response is written; under disk
contention on the single node devrc-ci is pinned to, one fsync exceeds HANG_TIMEOUT
(60.0) and the client raises TimeoutError. The suite's own classifier says so
unprompted: MECHANISM = SERVER_BLOCKED_IN_FSYNC.

A fix for that shipped in #1211 / #1219 / #1239scripts/testlib/store_siting.py's
store_root(tmp_path), which sites the store on tmpfs when one is usable and falls back
to tmp_path otherwise. It was never applied to the site that fails.

At origin/main this file had 5 call sites of store_siting.store_root and 18
that open-coded tmp_path / "store". The failing class was one of the 18: its _phases
helper built root = tmp_path / "store", unsited, on disk unconditionally.

Why that one test and not its siblings: within the class, test_POSITIVE_CONTROL_… is
the only test that gets a 200 appended. The others all assert a 404 — refused or
absent — which is answered before any write and never reaches _replace_bytes. So it is
the only test in the class that executes the two in-request fsyncs.

Corroborating, from the CI traceback already recorded in ci-repro/README.md: the
stalling write targets
/tmp/nix-build-devrc-pytests.drv-0/pytest-of-nixbld13/pytest-0/popen-gw3/…/store — a
tmp_path-derived path, not a devrc-store-* tmpfs holder.

What this changes

1. All 18 open-coded sites now take their root from a sited_root fixture
store_siting.store_root with a test's lifetime.

A fixture rather than 18 inline with blocks, and the reason is lifetime, not
convenience. Several of the sites build the root inside a helper whose return value
is used throughout the test body: _phases returns a pair of present()/absent()
closures, _entry_path and _twin_heading_entry return a path. A with wrapped around
the helper call tears the store down before a single assertion runs. A with wrapped
around each whole test body is 30 blocks of re-indentation — which is the exact shape of
the scripted conversion this repo already reverted once (it "silently skipped 9 of 19
signature edits while its own assertion still passed", per the ledger's own comment).
Taking the context manager as a fixture makes its lifetime the test's, per site, with
nothing to get wrong per site.

The one behaviour difference from an inline with is written into the fixture rather
than glossed: store_root skips its budget check when the body is raising, and pytest
does not throw a test's failure into a fixture generator, so on this path the check runs
even for a failing test. That is already true of store and scoped_store — it is the
standing cost of the idiom here, not something introduced by this change.

2. The guard is widened from one fixture to a census.

test_store_siting_ledger.py already carried an AST ratchet, but as a count:
_DISK_ROOTED_SITES = 33. A count cannot distinguish "one site was migrated and one new
one was written" from "nothing happened", and its own history is a string of arguments
about whether the number moving meant new debt or a wider predicate.

It is now an enumerated set: _DISK_ROOTED_ALLOWLIST, keyed
<Class.function> :: <unparsed expression> (never a line number — that moves when an
unrelated docstring gains a sentence), each entry carrying its reason.
test_the_disk_rooted_census_matches_the_allowlist_EXACTLY asserts set equality in both
directions, and names the offending site in the failure message.

15 entries remain. They are the population that was never spelled "store" — served,
served-elsewhere, stage, absent, big, at-the-cap, unambiguous — and every one
was checked to issue no write verb (no post_bullet, no create_put, no
method="PUT"/"POST"), so no request reaches _replace_bytes and nothing fsyncs inside
a request. They are still disk-backed, and that is recorded rather than hidden: siting
them is the obvious follow-up, deliberately not folded into a change scoped to the
write-path sites that were failing the gate.

A second arm, _SITED_STORE_ROOT_CALLERS, pins the other side of the relationship — the
set of store_siting.store_root(...) callers — so a deleted siting is red too. The
mutation matrix below shows that arm is not redundant.

The existing behavioural positive control is kept, and its class docstring is corrected.
It claimed a fixture falling back to disk "EVERYWHERE" would be caught, while its control
took one fixture (store). That is one site wide; 18 further roots sat behind that
green, including the failing one. sited_root now has its own behavioural control.

Red before green

scripts/ci-repro/slowfsync.c gains an opt-in SLOWFSYNC_SKIP_TMPFS=1 mode, because
the default mode cannot measure a siting fix: it interposes on fsync(2) in libc, so
it stalls tmpfs too, and both arms of the comparison go red. Measured:

default mode, fd on ext4   -> stalls 65.0s (fs magic=0xef53)
default mode, fd on tmpfs  -> stalls 65.0s (fs magic=0x1021994)

The new mode passes a TMPFS_MAGIC fd through without consuming the one-shot latch
(a pass-through that spent it would turn "the store moved to tmpfs" into "the shim ran
out of ammunition"), and a failing fstatfs() stalls rather than skips. Default
behaviour is unchanged, so every measurement already in ci-repro/README.md still holds.

All rows below are the same single test, -s so the shim's own stderr is visible:

tree shim mode store lands on result
origin/main none (control) ext4 whole class 5 passed in 8.76s
origin/main default ext4 1 failed in 65.52s, TimeoutError at socket.py:720, MECHANISM = SERVER_BLOCKED_IN_FSYNC, stall at _replace_bytes:2073 os.fsync(fh.fileno())
origin/main SKIP_TMPFS=1 ext4 1 failed in 63.96s, same, stall line fs magic=0xef53
this branch SKIP_TMPFS=1 tmpfs 1 passed in 3.67s, exactly two pass-through … magic=0x1021994 lines (the file and its parent dir — _replace_bytes's pair), latch untouched
this branch default tmpfs 1 failed in 63.69s, stall line fs magic=0x1021994
this branch SKIP_TMPFS=1, siting forced to fall back ext4 1 failed in 64.29s, stall line fs magic=0xef53

The last two rows are what make the fourth mean anything, and they are different claims.
Row 5: the shim can still kill this test on the branch, so the green is not "the
reproducer no longer reaches this code". Row 6: forcing store_siting.tmpfs_dir() to
None reproduces main's red exactly, so the green is the siting and not some other
edit on the branch.

The origin/main rows were taken with test_subsystem_store_api.py,
testlib/store_siting.py, test_store_siting_ledger.py and server.py byte-identical
to origin/main (only slowfsync.c differed). Those four files are also identical at
fbc4dd39 and at the current base 50b384c0, so the baseline holds for both.

Mutation of the census guard

Run with __pycache__ cleared between mutants (a same-length edit inside one second is
invisible to CPython's mtime-in-seconds + size cache and scores SURVIVED without running):

mutant census sited ledger others, THIS FILE ONLY
a migrated test reverted to tmp_path / "store" RED, naming that exact test green 22 passed
sited_root de-sited to tmp_path / "store" RED RED 21 passed
sited_root de-sited to tmp_path / "holder" green RED 22 passed

Each died on its own message, not on some other test's error. The census message
reads:

AssertionError: these disk-backed store roots are NOT in _DISK_ROOTED_ALLOWLIST:
  TestTheLoaderRefusesHostileEntriesByKind.test_a_CLEAN_store_is_UNCHANGED_by_the_guard :: tmp_path / 'store'   (line 13749)

Mutant 3 is the one that earns the second arm: a de-siting inside a fixture, under a
directory name outside _ROOT_NAMES, is invisible to the census — its flow arm stops at
the fixture's scope boundary, its on-sight arm only recognises store/src — so
_SITED_STORE_ROOT_CALLERS is what catches it. It also falsified my own first draft of
that comment, which had claimed the blind spot was "a fixture" — mutant 2 shows it is
narrower than that.

🔴 CORRECTED: this said _SITED_STORE_ROOT_CALLERS catches mutant 3 "only", and that
is false wherever a usable tmpfs exists.
The others column above is a one-FILE
run of the 23-test ledger module, so "every other guard in both files green" was
extrapolated from a measurement that never opened the other file. Re-run over BOTH
files on a tmpfs host, mutant 3 gives 2 failed, 781 passed:

FAILED test_subsystem_store_api.py::TestTheStoreIsSitedOffTheContendedDisk::test_the_sited_root_fixture_ACTUALLY_lands_on_tmpfs_when_one_exists
FAILED test_store_siting_ledger.py::test_the_SITED_store_roots_are_a_pinned_ledger_too

The behavioural control reads the fstype of sited_root.parent and finds the disk. The
census stayed green, so the blind spot itself is re-confirmed by the same run.

The arm still earns its place, for a narrower reason than "only": the behavioural
control SKIPS where there is no usable tmpfs — absent, not tmpfs, under
_MIN_FREE_BYTES free, or unwritable — which this PR says the gate may well be. So the
true claim is the only guard that can see it on a machine with no usable tmpfs, and
that is now what the failure message and the comment above the set say.

Also re-measured — and a claim of mine that the measurement falsified

_ROOT_NAMES's comment said seven sites were counted only by the on-sight
directory-name arm, and named them. All seven were among the 18 migrated. Re-measured by
running the census with _ROOT_NAMES = {"store", "src"} and again with it empty:
15 both times, identical keys — it contributes nothing on this file now.

My first replacement comment then said it was kept because "two of this file's own probe
tests depend on the on-sight arm", naming
test_the_site_index_does_not_key_on_the_DIRECTORY_being_spelled_store. That was
false
, and running it is what showed it: every probe in that test hands its path to
running(...), so the flow arm counts it and the spelling never decides anything.
Measured by setting _ROOT_NAMES to empty and running every argument-free test in the
module — zero behaviour changes.

So the comment now says the true thing: the set is inert everywhere that is measured
today, it is kept on an argument (the flow arm's _ROOT_CONSUMERS closes renames but
not growth into a new consumer name, and the on-sight arm is the only path that does not
go through that set), and the closing condition for deleting it is named. The
contribution is now measurable in one command instead of asserted in prose, which is what
caught the false sentence.

Gate

Authoritative tier — the one Tekton runs, nix build .#checks.x86_64-linux.pytests,
on this branch rebased onto 4e26ec9a:

RESULT: PASS (exit=0)      NIXBUILD_RC=0
86 per-target result lines, 0 of them FAIL
  PASS  scripts/tests             (collected=13587 passed=13587 skipped=0 floor=13026)
  PASS  scripts/dl-router/tests   (collected=1020  passed=1020  skipped=0 floor=942)
TOTAL collected=21716  passed=21714  skipped=2  failed=0  (floor 20441 = sum of 28 per-target floors)
grep 'panic: test timed out' -> 0

Counted from the runner's own per-target lines, not from an exit code. The 2 skips are
the pre-existing pinned real-Postgres ones; scripts/tests skipped 0, which means
both tmpfs behavioural controls — the existing store one and the new sited_root one —
ran rather than skipping. /dev/shm is usable in the nix sandbox.

Node tier, nix build .#checks.x86_64-linux.nodetests, built separately (never
concurrently with the pytest one — a combined invocation contends on the nix store and
produces false failures): RESULT: PASS (exit=0), NIXBUILD_RC=0, # fail 0 on every
suite. This diff touches no JS; the run is here because "both tiers" is the standing rule,
not because it could have been reached.

Dev-host tier, scripts/gate.sh --tier pytest: inconclusive, and I am not counting
it as a pass.
It got through scripts/tests (13519 passed, no failures) and then the
runner was SIGTERMed at ~65 min, so it printed RESULT: FAIL (exit=143) — a killed run,
not a test failure. That run also showed one failure in
scripts/dl-router/tests/test_store.py::test_six_writers_with_a_tiny_busy_timeout_still_land_every_row
(OperationalError('database is locked') against a deliberate 5 ms busy_timeout).
That test is unreachable from this diff, and the discriminating control is above: the
same file, same test, 1020/1020 passed in the sandbox tier. A second agent was
running a full run-tests.sh in ~/workspace/devrc-gate-base concurrently, which is
exactly the load-contention flake CLAUDE.md already documents for dl-router SQLite.

Not verified

  • CI itself. Everything above is measured on the dev host, where /tmp is ext4 and
    /dev/shm is tmpfs. The gate's sandbox may have no usable tmpfs, in which case
    store_root falls back to disk by design and this changes nothing there.
    store_root's docstring enumerates the five ways that happens.
  • The 15 allowlisted sites are argued to be write-free from their call graph (no
    write verb reaches them), not from a runtime trace.

@ZacxDev

ZacxDev commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Round 1 audit — 6 findings, all fixed

Round 0 (requirements & deletion) and round 1 (the nine axes) were run blind of each other, read-only, in separate worktrees.

Round 1 verdict: safe to merge, nothing deploy-blocking. Every finding was in the scaffolding, and all six were one shape: the guard's description is wider than the guard — the same class this PR exists to fix, recurring inside the fix.

What round 1 re-verified and found sound: 18 → 0 by AST (total tmp_path / X 142 → 124, an exact 18 drop, so nothing re-added under another name); all 15 allowlisted sites read individually and confirmed write-free; row 6 of the matrix load-bearing (forcing tmpfs_dir() to None reproduces main's red, so the green is the siting); teardown does not mask a real failure.

Round 0 ledger: round 0 · requirements: 9 (unattributed: 3) · deletion candidates: 3.

Queued follow-ups, deliberately NOT in this PR

  1. Delete _ROOT_NAMES — measured inert three times independently (emptied set → identical pass counts and an identical census key set), including under the widened predicate. Kept on an argument, not a measurement.
  2. Site the 15 allowlisted entries and drop the allowlist — a 33-site conversion is the shape that got reverted before, so not folded in here.
  3. Question the layer (round 0, R1). The failing write lands on the step container's ephemeral /tmp; the CI task sets no TMPDIR and mounts nothing there. A medium: Memory emptyDir at /tmp would site every tmp_path in the suite in one infra change and retire this scaffolding layer as a flake defence. Not asserted to work — where the daemon build's /tmp resolves was not measured. Infra repo, operator call.

Not measured

Nothing here ran in the gate's own environment. If the CI container has no usable tmpfs, store_root falls back to disk by design. Checkable from any failure log: devrc-store-* = sited, pytest-of-* = fell back.

1. F1: `_is_disk_rooted_store_expr` no longer gates on the right operand being `ast.Name`/`ast.Constant`; `_used_as_a_store_root` alone decides, so an f-string/concat/method-call spelling of a write-path store root is now censused.
2. F1: claimed no-op on today's tree — old vs widened predicate over all 184 `test_*.py` in `scripts/tests` yields 221 identical keys, and the ledgered-file census is unchanged at 15.
3. F2: the "enumerate EVERY store site in this file" claim is removed; the bullet and the census docstring now state the predicate's actual reach and enumerate all three residual classes.
4. F3: "the ONLY guard that can see it" is rescoped to "on a machine with no usable tmpfs", in both the failure message and the comment, and the PR body's mutation table is relabelled `others, THIS FILE ONLY` with a CORRECTED block.
5. F4: `SLOWFSYNC_SKIP_TMPFS` compares its VALUE rather than testing presence, so `=0` disables the mode instead of enabling it.
6. F5: both the census and the sited ledger loop `EXPECTED_SERVER_TESTS` instead of hardcoding one file; keys gained a file component and remain qualname-based, never line numbers.
7. F6: the census positive control gained a second probe under a directory name outside `_ROOT_NAMES`, so the flow arm is exercised by its own control.
8. `_ROOT_NAMES` was NOT deleted and the 15 allowlist entries were NOT touched — both deliberately out of scope for this round.

@ZacxDev

ZacxDev commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Round 2 delta re-audit — all 8 round-1 claims hold; 3 new findings, all fixed

Round 2 ran blind of the fix author, against d4e6afaf..157cd02e. All eight claims verified, four re-derived to the digit (221/221 keys over 184 files; census 15/0/0; mutant 3 → 2 failed, 781 passed; 16 SLOWFSYNC_SKIP_TMPFS spellings). Claim 8 — a negative claim — checked byte-for-byte: keys and reason strings identical.

Claim 4 was verified at both points of the tmpfs dimension, which is what makes it a claim rather than an assertion: with tmpfs forced unusable, both behavioural arms SKIP and the structural ledger still fails.

Authoritative tier at the head, read by content: scripts/tests collected=13587 passed=13587 skipped=0, TOTAL collected=21716 passed=21714 skipped=2 failed=0, RESULT: PASS.

The finding that mattered

The operand-type widening shipped with no regression test — restoring the exact pre-round-1 isinstance gates left the ledger at 23 passed. Every existing probe used an ast.Constant operand, so nothing exercised a JoinedStr/BinOp/Call/Subscript operand in either direction. A hand measurement in a commit message, in the module whose premise is that a hand measurement is not coverage.

Now pinned by test_the_operand_NODE_TYPE_is_not_what_decides_either — four non-Constant spellings across both arms, each twice (flowing into _build_store → must count 1; flowing nowhere → must stay 0). Watched red three ways: both gates restored → 1 failed, 23 passed; flow gate defaulted to True → the negative half fails on its own message; unmutated → 24 passed. Re-verified independently with a narrower mutation (one arm's gate only, isolating the guard from its sibling condition) — still the only failure, still its own message.

Ledger

round 2 · payload lines changed THIS round: 24 (since round 1: 24) · elapsed: 36m

Classification declared at round 1 and unchanged: test_subsystem_store_api.py PAYLOAD (revert test), test_store_siting_ledger.py and ci-repro/** SCAFFOLDING. Round 2 correctly notes all 24 sit inside a class docstring, so by executable payload the round changed 0 — recorded, but the measure is not being switched mid-ladder.

Still queued, still not in this PR

_ROOT_NAMES deletion (measured inert four times, including under the widened predicate) · siting the 15 allowlisted entries · round 0's R1, the CI /tmp tmpfs mount that would retire this scaffolding layer entirely (infra repo, operator call) · widening _assignmentsnot the trivial follow-up it looks like: the container-element half has no statically resolvable base for stores['a'], and closing either half changes the census over the real files, forcing a re-record of the allowlist in the same commit.

1. F7: `test_the_operand_NODE_TYPE_is_not_what_decides_either` pins the operand-type widening with four non-Constant spellings across both the `/` and `.joinpath` arms, each present twice — counted when it flows into a consumer, 0 when it flows nowhere.
2. F7: the predicate docstring no longer cites `test_a_path_that_is_never_used_as_a_store_still_does_NOT_count` as the control for the widening; that test's probes are Constants and cannot see it.
3. F8: the residual enumeration gained a fourth hole — a root reaching a consumer in one scope through a container element, a `for`/comprehension target, or a closure — with the measured numbers, applied at five sites rather than the one reported.
4. F8: `_assignments` was NOT widened; the enumeration was corrected instead.
5. F9: every residual bullet now leads with `PINNED` or `ONLY WRITTEN DOWN`, at both places the enumeration appears; one of four holes is guarded.
6. F9: the false "each pinned by a named guard over there" clause is removed from `test_subsystem_store_api.py`.
7. `_ROOT_NAMES`, the 15 `_DISK_ROOTED_ALLOWLIST` entries, and `_assignments` were all left untouched this round, deliberately.

@ZacxDev

ZacxDev commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Round 3 delta re-audit — and the ladder STOPS HERE

All seven round-2 claims verified. Two were checked structurally rather than by eye: with docstrings stripped, the payload file's AST is byte-identical across the range (100659 → 100659 lines, against a positive control that can see a one-token change), and the ledger's is +169/−0 with the whole addition being the new test function. So "_assignments not widened" and "_ROOT_NAMES/allowlist untouched" are proven, not asserted.

The new guard came out stronger than it claimed. Eight mutants: each arm is independently covered (restoring either type gate alone names exactly that arm's probes); the negative half is genuinely reached, not short-circuited; and it goes red — not green — when the flow gate dies or _ROOT_CONSUMERS is narrowed. Emptying _ROOT_NAMES leaves it at 24 passed, which removes the one confound in the fixture design.

The fifth-shape hunt came back clean, and that is a result: 27 concrete shapes measured against the census's five failure mechanisms — sibling-method via self, pass-through helper param, functools.partial, .append-then-subscript, star-unpack, with-as, module-scope global, two-hop container, and the rest. No fifth hole. The enumeration is complete at the mechanism level; what was wrong was which mechanism the prose blamed.

Round 3's four findings — all fixed in 784d2651..30aa044b

One defect at several reader-facing sites: residual bullet 4 mis-attributed the mechanism behind a measured zero. The fix round re-derived the attribution one-at-a-time (monkeypatching _assignments and _path_base separately — reading the code cannot attribute a mechanism):

probe                          shipped  +assign  +base  +both
inline (control)                     1        1      1      1
dict element                         0        0      1      1
list element                         0        0      1      1
for target                           0        1      -      -
comprehension target (isolated)      0        1      -      -
closure                              0        0      0      0

So: widening _assignments closes 2 of 5, not "them" — the element rows are _path_base's and the closure is _walk_scope's. The definition sentence named two conditions where the code has three (_path_base must be able to name a base). And one of my own framings was wrong and the fix round refused it with measurement: row 5 is not "doubly-blocked" — +base alone takes it 0→1, so the subscript is sufficient and the comprehension target was never load-bearing there. The row was replaced with one that isolates the target rather than described as something it is not.

The fix round also caught and corrected its own error in commits 1–3 (it had written the reader's sense of "one function scope" into a sentence defining the code's), and deleted rather than reworded a count it could not verify from the tree.

🔴 Why the ladder stops here, stated explicitly rather than left implicit

The findings-keyed rule would run a round 4. I am stopping on the stated criterion instead, and naming why the rounds will not stop on their own:

  • No 🔴 in any round. Zero deploy-blocking findings across three rounds.
  • The entire remaining blast radius is "the document mis-states which line of code causes a measured zero." Nothing outside docstrings.
  • The recurring shape was swept at every site, not the one reported — F10 at 3 sites, F13 at 2, F11/F12 at their own, plus the serve_store count.
  • The attribution gate is structurally inert here. Rounds 2 and 3 each changed zero executable payload lines by AST measurement, while the by-file count reads 78 — because docstring lines in a payload file still count as payload. The gate cannot fire on a ladder that has moved entirely into prose.
  • Each round's replacement sentence has become the next round's finding, repeatedly. That is the non-terminating shape, and the round-3 auditor reached the same conclusion independently: "neither is worth a round 4, and I would not run one."

What is deliberately left UNFIXED — open, not absent

  1. _ROOT_NAMES is inert — measured four times independently, including under the widened predicate. Kept on an argument, not a measurement. Deletion is a queued follow-up.
  2. The 15 _DISK_ROOTED_ALLOWLIST entries stay on disk. All 15 were read individually and confirmed write-free, so none reaches _replace_bytes. Siting them retires the allowlist entirely — a 33-site conversion, which is the shape that got reverted once already.
  3. _assignments is not widened. Not the cheap follow-up it looks like: the container-element half has no statically resolvable base for stores['a'], and closing either half moves the census over the real files, forcing a re-record of the allowlist in the same commit.
  4. Round 0's R1 — the layer question. The failing write lands on the step container's ephemeral /tmp; the CI task sets no TMPDIR and mounts nothing there. A medium: Memory emptyDir at /tmp would site every tmp_path in the suite in one infra change and retire this entire scaffolding layer as a flake defence. Not asserted to work — where the daemon build's /tmp resolves was never measured. Infra repo, operator call.
  5. scripts/ci-repro/slowfsync.c has no automated build or lint anywhere. Its evidence is hand-run and does not recur; a future edit breaking it is undetectable until someone next needs it.
  6. AsyncFor appears in the _assignments docstring's description of what widening would add, but only for and comprehension targets were measured.

🔴 And the thing a green gate here does NOT establish

A green gate on this PR is not the verifier. The gate validating a gate fix is not independent evidence, and one green cannot separate "the fix worked" from "this run would not have flaked". The verifier is the flake rate against a fresh baseline whose PR heads postdate the merge — claudedocs/handoff-gate-flake-store-api.md rank 1.

Nothing in three rounds was measured in the gate's own environment. If the CI container has no usable tmpfs, store_root falls back to disk by design and this changes nothing there. That is checkable from any future failure log without re-deriving anything: devrc-store-* = sited, pytest-of-* = fell back.

Round 0 trial record: ran: 1 · changed the outcome: 1 — weakly. It did not change the merge decision; it produced two items no correctness axis generates (an inert guard, and a requirement that belongs in another repo).

1. F10: the census definition sentence names a third condition — the expression must be one `_path_base` can name a base for — at all three sites that state it.
2. F11: `_assignments`' docstring carries the per-row table and says widening closes the `for` and comprehension-target rows only, leaving the two element rows and the closure open.
3. F12: residual bullet 4's row 4 is now an isolated comprehension target that moves 0→1 under a widened `_assignments`; the replaced row and the reason are recorded in place.
4. F13: the closure is attributed to the scope walker, not to a binding form, at both sites that had it under "one function scope".
5. The `serve_store` occurrence count is replaced by the property that matters — both occurrences are prose and no test runs it.
6. The census bullet's preamble says "within one TEST FUNCTION", naming why the reader's sense of "one function" and the code's come apart.
7. `_ROOT_NAMES`, the 15 allowlist entries and `_assignments` itself were left untouched, again and deliberately.
8. LADDER CLOSED on the stated criterion, not on a clean round — rationale above.

vetrdev and others added 7 commits September 10, 2026 09:42
…asure a SITING fix

The shim intercepts fsync(2) in libc, so it stalls whatever the fd is backed
by — including tmpfs. That makes it structurally unable to measure the fix it
was built to motivate: with the default mode, both arms of "store on disk" vs
"store on tmpfs" go red, and the red on the fixed arm is the shim's, not the
code's. Measured, and it is the reason this mode exists:

    default mode, fd on ext4  -> stalls 65.0s (fs magic=0xef53)
    default mode, fd on tmpfs -> stalls 65.0s (fs magic=0x1021994)

`SLOWFSYNC_SKIP_TMPFS=1` models the mechanism the README documents — node-local
*device* contention, which an fsync with no backing device does not wait for —
by passing a TMPFS_MAGIC fd straight through. Two properties are deliberate:
the pass-through does NOT consume the one-shot latch (else "the store moved to
tmpfs" becomes "the shim ran out of ammunition", a green that means nothing),
and a failing fstatfs() stalls rather than skips (a reproducer that quietly
stops firing reports a pass). The stall line now prints the fd's fs magic, so
which filesystem was stalled is readable rather than inferred.

Controls, all watched:

    SKIP_TMPFS=1, fd on ext4        -> stalls 65.0s
    SKIP_TMPFS=1, fd on tmpfs       -> pass-through, 0s
    SKIP_TMPFS=1, tmpfs THEN ext4   -> pass-through, then the full 65.0s stall

Default behaviour is unchanged, so every measurement already recorded in
scripts/ci-repro/README.md still describes what the shim does with the env var
unset.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
… and census the set

devrc#1211 sited `store`; a later round sited `scoped_store`. Both times the
tests that build their OWN root inline were left on disk — 18 of them, spelled
`tmp_path / "store"` — and one of those is
`TestARefusedWriteIsIndistinguishableFromAnAbsentOne`, whose
`test_POSITIVE_CONTROL_the_APPEND_comparison_CAN_see_the_difference` kept
failing `tekton/devrc-pytests` on PRs whose diff cannot reach it. It is the only
test in that class that gets a `200 appended` — its siblings all assert a 404,
answered before any write — so it is the only one that reaches
`server.py:_replace_bytes` and executes the two in-request fsyncs. That is why
it, specifically, recurred.

WHAT CHANGED

* a `sited_root` fixture: `store_siting.store_root` with a test's lifetime. All
  18 sites take their root from it. A fixture rather than 30 inline `with`
  blocks because several of the sites build the root inside a HELPER whose
  return value — a path, or a pair of `present()`/`absent()` closures — is used
  throughout the test body, so a `with` around the helper CALL would tear the
  store down before an assertion ran; and because a 30-block re-indentation is
  the shape of the scripted conversion this repo already reverted once. The one
  behaviour difference from an inline `with` (pytest does not throw a failure
  into a fixture generator, so the budget check runs even for a failing test) is
  written down in the fixture rather than glossed — it is already true of
  `store` and `scoped_store`.
* `_DISK_ROOTED_SITES = 33` (a COUNT) becomes `_DISK_ROOTED_ALLOWLIST` (a SET,
  keyed `<Class.function> :: <unparsed expression>`, each entry carrying its
  reason), asserted in BOTH directions. A count could not tell "one site
  migrated, one written" from "nothing happened"; the set names the site in the
  failure message. 15 entries remain — the population that was never spelled
  "store" — every one verified to issue no write verb, so no request reaches
  `_replace_bytes`. Still disk-backed, recorded rather than hidden.
* `_SITED_STORE_ROOT_CALLERS`, the other side of the relationship, so a deleted
  siting is red even in the shape the census cannot see.
* `TestTheStoreIsSitedOffTheContendedDisk`'s docstring said a fixture falling
  back "EVERYWHERE" would be caught while its positive control took ONE fixture.
  Corrected, and `sited_root` gets its own behavioural control.

RED BEFORE GREEN, on the exact failing test, via `scripts/ci-repro/slowfsync.c`

| tree | shim mode | store on | result |
|---|---|---|---|
| origin/main | none | ext4 | class `5 passed in 8.76s` |
| origin/main | default | ext4 | `1 failed in 65.52s`, TimeoutError, MECHANISM = SERVER_BLOCKED_IN_FSYNC |
| origin/main | SKIP_TMPFS=1 | ext4 | `1 failed in 63.96s`, stall `fs magic=0xef53` |
| branch | SKIP_TMPFS=1 | tmpfs | `1 passed in 3.67s`, 2 pass-throughs `magic=0x1021994` |
| branch | default | tmpfs | `1 failed in 63.69s` — the shim can STILL kill it |
| branch | SKIP_TMPFS=1, siting forced to fall back | ext4 | `1 failed in 64.29s` |

The last two rows are what make the fourth mean something: row 5 that the green
is not an inert reproducer, row 6 that it is the SITING and not another edit.

MUTATION, `__pycache__` cleared between mutants

* a migrated test reverted to `tmp_path / "store"` -> census RED naming that
  test, sited-ledger green, 22 others passed.
* `sited_root` de-sited to `tmp_path / "store"` -> BOTH red.
* `sited_root` de-sited to `tmp_path / "holder"` -> census GREEN, sited-ledger
  RED. This is the case that makes the second guard load-bearing, and it
  corrected a comment that had claimed the blind spot was "a fixture" when it is
  "a fixture under a name outside _ROOT_NAMES".

Also re-measured: `_ROOT_NAMES` now contributes ZERO sites on this file (15 with
it, 15 with it empty, identical keys). Its comment named seven sites it was
propping up; all seven were among the 18 migrated. Kept — two probe tests depend
on the on-sight arm — with the measurement written down instead of the stale one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…red false in the same file

The previous commit replaced a stale measurement about `_ROOT_NAMES` with a
fresh one (15 sites with the set, 15 with it empty, identical keys — it now
catches nothing on `test_subsystem_store_api.py`, because all seven sites its
old comment said it was propping up were among the 18 migrated). It then
justified keeping the set by asserting that two of the ledger's own probe tests
depend on the on-sight arm, and named
`test_the_site_index_does_not_key_on_the_DIRECTORY_being_spelled_store`.

That is FALSE, and it is the same defect the paragraph it replaced had: a claim
about a guard's coverage, written from intent rather than from running it. Every
probe in that test hands its path to `running(...)`, so the FLOW arm counts it
and the directory's spelling never decides anything.

MEASURED: `_ROOT_NAMES` set to the empty set, then every argument-free test in
`test_store_siting_ledger.py` executed — ZERO behaviour changes. The set is inert
everywhere that is measured today, not just on the file it was written for.

It is still kept, but the comment now says plainly that the reason is an ARGUMENT
and not a measurement (`_ROOT_CONSUMERS` closes renames but not growth into a new
consumer name, and the on-sight arm is the only path that does not go through
that set), and it names the closing condition for deleting it: a measurement
showing the flow arm covers a store built under a name the consumer set does not
know. Nothing here demonstrates that today.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…g it reported a pass

`getenv("SLOWFSYNC_SKIP_TMPFS") != NULL` tests PRESENCE, not value. The spelling an
operator reaches for to switch the filesystem-aware mode OFF switched it on, and the
file header and README both document the mode as `=1` — so the operator got exactly
what the header's own paragraph warns about: "a shim that quietly stops firing reports
a pass".

`skip_tmpfs_enabled()` compares the value. ON for 1/true/yes/on (case-insensitive);
OFF for 0/false/no/off, the empty string, and the variable being unset; an
unrecognised value resolves in the FIRING direction with a one-shot line on stderr
naming it, because silently picking either mode re-creates this defect one spelling
over and picking the non-firing one hides it behind a pass.

Measured, both values watched, on
TestARefusedWriteIsIndistinguishableFromAnAbsentOne::
test_POSITIVE_CONTROL_the_APPEND_comparison_CAN_see_the_difference, whose store this
branch sites on tmpfs:

  before  SKIP_TMPFS=0   two pass-through lines, 1 passed in  3.18s   <- the bug
  before  SKIP_TMPFS=1   two pass-through lines, 1 passed in  3.26s
  before  unset          stall magic=0x1021994,  1 failed in 64.38s
  after   SKIP_TMPFS=0   stall magic=0x1021994,  1 failed in 65.91s   <- now == unset
  after   SKIP_TMPFS=1   two pass-through lines, 1 passed in  5.10s

Sixteen spellings watched on a one-fsync tmpfs probe: `1 true TRUE yes YES on On`
pass through in 0.00s; `0 false no off Off`, the empty string and the unset variable
take the 65s stall; `maybe` and `2` print the warning and then stall.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
… file of three

Four findings from the round-8 audit, all in the census and its own descriptions.

F1 — `_is_disk_rooted_store_expr` required the right operand to be an `ast.Name` or
an `ast.Constant` before consulting the flow gate. That is a guard on the SHAPE of a
spelling: `tmp_path / f"store-{k}"` is a `JoinedStr`, `tmp_path / ("store" + k)` is a
`BinOp`, and neither counted no matter where it flowed. Measured as a LIVE hole, not
imagined — swapping
`TestTheRENAMEIsFSYNCedToo.test_an_append_fsyncs_BOTH_the_file_and_its_DIRECTORY` back
to `_build_store(tmp_path / f"store-{ALLOW_SCOPE}", …)`, a write-path store that
reaches `api.append_bullet` -> `_replace_bytes` and its two in-request fsyncs:

  before  ledger 23 passed, api 760 passed          -> the mutant SURVIVED
  after   ledger 1 failed / 22 passed, on the census's own message, naming the test

The type gate is deleted; `_used_as_a_store_root` does the discriminating, and it
already ruled out round 3's `(tmp_path / name).write_text(body)` false positive
(`test_a_path_that_is_never_used_as_a_store_still_does_NOT_count`).

No-op on today's tree, swept rather than assumed: old and widened predicates run over
all 184 `test_*.py` files in `scripts/tests` give **221 keys each, byte-identical key
sets**, no file changed. Shape matrix, old -> new: f-string 0->1, `"store" + k` 0->1,
`k.lower()` 0->1, `names[0]` 0->1, two-hop f-string 0->1, `joinpath(f"…")` 0->1;
`Path(tmp_path)/"store"`, `base = tmp_path`, `os.path.join`, `str(tmp_path)+"/store"`,
`tmp_path_factory.mktemp` and a fixture-returned holder stay 0->0, as does the
scratch-file false accusation. The residual is written into the docstring as the
residual; the widening does NOT close the helper-return, `Path(tmp_path)` or alias
shapes.

F5 — the census hardcoded `TESTS / "test_subsystem_store_api.py"` two functions under
`EXPECTED_SERVER_TESTS`, which names three; so did the sited ledger. Both now loop the
frozenset, keys carry the file, and each keeps its `Class.function` form with no line
numbers. Measured at the moment of the fix: api 15, `test_cairn_write.py` 0,
`test_cairn_cli.py` 0 — no live defect, and the two zeros are the point, since a file
never opened reports the same zero as a file that is clean. The sited ledger gains
`test_cairn_write.py :: store` and `test_cairn_cli.py :: source_store`; the api file's
15 allowlist entries are unchanged in content, only nested under their file. Both
ledgers now assert their file set equals `EXPECTED_SERVER_TESTS`, so a new ledgered
file cannot arrive as a silent absence.

F6 — `test_the_census_can_actually_SEE_a_disk_rooted_site` probed only
`tmp_path / 'store'`, which `_ROOT_NAMES` answers ON SIGHT: the flow arm could have
been dead and the control still green. A second probe under `holder`, outside
`_ROOT_NAMES`, now exercises it, with an assertion that the name really is outside.

F2 — `TestTheStoreIsSitedOffTheContendedDisk`'s docstring said the census and the
allowlist "enumerate EVERY store site in this file", ten lines under the retraction of
the previous "EVERYWHERE" overclaim. Corrected to what the predicate reads, with the
whole measured residual — the fixture-scope hole, the not-spelled-off-`tmp_path`
shapes, and `_ROOT_CONSUMERS` growth — rather than a fresh justification. The census's
own docstring recorded only the fixture hole and now records all three.

F3 — the sited ledger's failure message said it was "the ONLY guard that can see"
a fixture de-sited under a name outside `_ROOT_NAMES`. That is false wherever a usable
tmpfs exists: `TestTheStoreIsSitedOffTheContendedDisk::test_the_sited_root_fixture_
ACTUALLY_lands_on_tmpfs_when_one_exists` reads the fstype of `sited_root.parent` and
fails alongside it. The "22 others passed" behind the original claim was a one-FILE run
of this 23-test module. The arm stays — the behavioural control SKIPS where there is
no usable tmpfs — but the claim is now scoped to "on a machine with no usable tmpfs",
in the message and in the comment above the set.

Ledger: 23 passed on the restored tree, census key set identical to the pre-change 15.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…ft 23 passed

Round 1 deleted the `isinstance(right, (ast.Name, ast.Constant))` gates from both
arms of `_is_disk_rooted_store_expr` so the flow gate alone decides, and measured
the widening BY HAND in a commit message. In the module whose entire premise is
that a hand measurement is not coverage.

MEASURED: restoring both gates verbatim — the exact pre-round-1 code — leaves this
file at 23 passed. Nothing in the repo goes red. Every existing probe uses an
`ast.Constant` operand: both probes in `test_the_census_can_actually_SEE_a_disk_
rooted_site` ('store', 'holder'), and every probe in `test_a_path_that_is_never_
used_as_a_store_still_does_NOT_count`. No test anywhere exercised a JoinedStr,
BinOp, Call or Subscript operand in either direction.

`test_the_operand_NODE_TYPE_is_not_what_decides_either` is the coverage: four
non-Constant spellings (f-string, concatenation, call, subscript) across both the
`/` and `.joinpath` arms, each appearing twice — once flowing into `_build_store`
and once flowing nowhere. Named individually rather than totalled, so three cannot
regress behind one that still works.

Watched red, both halves, `__pycache__` cleared between mutants:

  * restore both `isinstance` gates -> the four COUNTED probes report 0 and this
    test is the ONLY failure, on its own message. 1 failed, 23 passed.
  * `_used_as_a_store_root` defaults to True (the narrowest expression that makes
    the flow gate accept everything) -> the four UNCOUNTED probes report 1 and this
    test fails on its own message. 5 failed, 19 passed.
  * unmutated -> 24 passed.

Also corrects the predicate's own docstring, which cited `test_a_path_that_is_
never_used_as_a_store_still_does_NOT_count` as the control for the widening. That
test cannot see it — its probes are Constants. A comment is a claim too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…cope, no fixture

The stated reach — a `tmp_path / X` that flows into a store consumer "WITHIN ONE
FUNCTION SCOPE, and nothing else" — is wider than the code. RE-MEASURED here, not
inherited from the audit: all in one function scope, all reaching `running` (which
IS in `_ROOT_CONSUMERS`), all spelled `tmp_path / 'served'`, no fixture anywhere:

    inline (control)                                              -> 1
    stores = {'a': tmp_path / 'served'}; running(stores['a'])      -> 0
    stores = [tmp_path / 'served']; running(stores[0])             -> 0
    for served in (tmp_path / 'served',): running(served)          -> 0
    roots = [p for p in (tmp_path / 'served',)]; running(roots[0]) -> 0
    served = tmp_path / 'served'; def go(): running(served); go()  -> 0

The list and comprehension rows are mine; the other four reproduce the audit's.

None of the three enumerated residual bullets covers these — bullet 1 is a
`@pytest.fixture` scope boundary, bullet 2 is "not spelled `tmp_path / X`", bullet
3 is `_ROOT_CONSUMERS` growth — so the enumeration read as complete while missing a
whole family. Root cause: `_assignments` resolves `Assign`/`AnnAssign`/`NamedExpr`/
`withitem` and nothing else, so a `for` and a comprehension target bind invisibly;
`_path_base` returns None for an `ast.Subscript`, so a container element is not a
path expression it can name a base for; and `_walk_scope` stops at EVERY nested
function, `def go()` included — it is not only pytest's injection that crosses a
boundary.

Fixes the ENUMERATION at all five places that state it, not just the reported one:
the census docstring (fourth bullet, with the numbers), the "at least once" test's
docstring, the predicate's own "READ THE RESIDUAL" paragraph, `_assignments`'
docstring, and `TestTheStoreIsSitedOffTheContendedDisk` in the api file. Widening
`_assignments` is deliberately NOT done — it is written down rather than guarded,
the same deal the other three bullets get.

The api paragraph also loses its "each pinned by a named guard over there" clause,
which is false for two of the three holes; the positive replacement is the next
commit.

24 passed on the ledger, unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
ZacxDev added a commit that referenced this pull request Sep 10, 2026
…ayer — the fix already existed and had never reached the failing site

The SERVER_BLOCKED_IN_FSYNC diagnosis stands. What was wrong was the remedy set:
(a) bound the write path was recommended as "the only one that makes the SERVER
correct", but it is a production change to crash-durability semantics made to
close a test-harness siting gap. The tmpfs fix from #1211/#1219/#1239 sited 5
store roots and left 18 open-coded on disk, and the test that keeps reddening
the gate is one of the 18 — it is also the only test in its class that gets a
200, so it is the only one that executes the two in-request fsyncs.

Records what #1458 ships, the independently re-run census mutation, that
slowfsync.c could not measure a siting fix without the new opt-in, and the two
residuals: nothing is measured in CI, and a green gate on #1458 is not the
verifier (handoff-gate-flake-store-api.md rank 1 is).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
ZacxDev and others added 5 commits September 10, 2026 09:42
…her three are prose

"The holes, each measured and each pinned by a named guard over there rather than
left to be rediscovered as news" was true of exactly one hole. Verified here, not
inherited:

  * hole 1 (a root bound in a `@pytest.fixture`) — PINNED.
    `test_a_store_root_bound_in_a_pytest_FIXTURE_is_NOT_counted` is a real test that
    runs every time, asserts 0 for both the plain and the tuple-returning fixture
    form against an inline control of 1. Its cited live instances exist:
    `served = tmp_path / "ordered-served"` in `shuffled_pair` (:3990/:3993) and
    `served = tmp_path / "ambig-served"` in `ambiguous_pair` (:4339/:4342).
  * hole 2 (a root not spelled off the NAME `tmp_path`) — NO TEST ANYWHERE. Every
    occurrence of `Path(tmp_path)`, `base = tmp_path`, `os.path.join`,
    `str(tmp_path) + `, `tmp_path_factory` in `test_store_siting_ledger.py` is
    inside a docstring (:486-488, :894-895); the only code hits in the whole tests
    directory are unrelated files. Nothing re-measures the claimed 0.
  * hole 3 (`_ROOT_CONSUMERS` growth) — its own clause already said "that set's own
    comment", i.e. a comment, contradicting "named guard" mid-sentence. The
    `serve_store(served)` -> 0 control it cites appears once in the file, in that
    comment.
  * hole 4 (the one the previous commit added) — also only written down.

So the label now comes first on every bullet, PINNED or ONLY WRITTEN DOWN, at both
places the enumeration appears. That distinction is the sentence's whole stated
purpose — "so it cannot be rediscovered as news" is a property of a guard that
re-measures, and three of these four can silently close or widen with the suite
green.

24 passed on the ledger, unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
… has three

The sentence defining what the disk-rooted census reads — "flows into a store
consumer within one function scope AND THROUGH A BINDING FORM `_assignments`
RESOLVES" — is satisfied by a shape that measures 0:

    def test_probe(tmp_path):
        stores = {'a': tmp_path / 'served'}   # an ast.Assign; _assignments DOES
        running(stores['a'])                  # yield ('stores', <Dict>)

Measured 0 against an inline control of 1; the list form likewise. The blocker
is a THIRD condition the sentence never stated: `_path_base` returns None for
the `ast.Subscript` handed to the consumer, so the bound name never enters
`flowing_roots` and the binding is never reached. Teaching `_path_base` to see
through a Subscript takes both shapes 0 -> 1 while widening `_assignments`
leaves both at 0, which is what attributes the mechanism.

Corrected at all three sites that state the definition.

Same commit, same underlying defect: two of those sites grouped the CLOSURE
under "one function scope" / "a binding the census does not resolve". Measured,
the closure fails none of the three conditions — `served = tmp_path / 'served'`
is a plain assignment and `_path_base(served)` names a base — and stays 0 under
a widened `_assignments` AND a widened `_path_base`. It is the scope arm,
`_walk_scope` stopping at the nested `def`. The census bullet already said this;
these two copies had lost the correction.

Also: the `serve_store(served)` -> 0 control is cited as appearing "once in that
file, in the comment itself". It appears twice (`_ROOT_CONSUMERS`' comment and
the fourth residual bullet). The count is replaced by the property that actually
matters — both occurrences are prose, and no test runs it.

Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
The docstring said "Widening this function would close them", where *them* is
the three shapes named one sentence earlier — one of which the SAME sentence
attributes to `_path_base`. Measured by actually widening it (adding `ast.For`
/ `ast.AsyncFor` / `ast.comprehension` targets to the yields) and re-running the
fourth residual bullet's five probes against an inline control of 1:

    for target                         0 -> 1
    comprehension target (isolated)    0 -> 1
    dict element                       0 -> 0
    list element                       0 -> 0
    closure                            0 -> 0

Two of five. The two element rows go 0 -> 1 under a widened `_path_base` with
`_assignments` untouched, so they are that function's; the closure stays 0 under
both widenings, so it is `_walk_scope`'s.

This is not a nit, because the same docstring frames the widening as THE
deferred fix and tells the maintainer to re-run the census and record what it
starts seeing. Following that instruction produces a census that moves, which
reads as the hole closing, while the element and closure halves stay open and
the suite stays green.

Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…trated none

The bullet's opening named one mechanism for all five rows ("not through a
binding form `_assignments` resolves"). Widening one thing at a time and
re-measuring separates them:

  * `bind`  — `for` target, comprehension target. Adding `ast.For` /
    `ast.comprehension` targets to `_assignments` takes both 0 -> 1 and moves
    no other row.
  * `base`  — dict element, list element. Unmoved by a widened `_assignments`;
    0 -> 1 when `_path_base` is taught to see through an `ast.Subscript`.
  * `scope` — the closure. Unmoved by either widening; `_walk_scope` stops at
    the nested `def`.

Each row now carries its mechanism.

Row 4 is replaced. `roots = [p for p in (tmp_path / 'served',)];
running(roots[0])` was filed under the comprehension target and does not
demonstrate it: 0 under a widened `_assignments`, 1 under a widened `_path_base`
alone, so the SUBSCRIPT is the blocker and the comprehension target is not
load-bearing — `mark()` walks the whole bound value, so once `roots` is reached
the comprehension is transparent. `out = [running(p) for p in (tmp_path /
'served',)]` isolates the target: 0 as shipped, 1 under a widened
`_assignments`.

It is replaced rather than relabelled "doubly-blocked", because it is not:
one widening is sufficient to close it.

Docstrings only; no executable change. 784 passed across both files.

Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
…s three

Self-audit of the three commits above, and the same defect they exist to fix.

Two of them say the closure "fails NONE of the three" / "passes all three"
conditions and measures 0 anyway. That reads as a contradiction, and it is one:
"within one function scope" means `_walk_scope`'s scope, and the closure's
binding and its consumer are two of those. It fails the FIRST condition — on
the code's meaning of scope, not the reader's — and neither of the other two.
Both sites now say that, and the census bullet's own preamble says "within one
TEST FUNCTION" rather than "scope" for exactly the reason that one of its rows
is where those two words come apart.

Also: the api file's bullet announced "TWO mechanisms" and then listed three
(bind / base / scope), and said "neither of the two arms above" where three
bullets precede it.

Docstrings only; the executable AST of both files is identical to 0618eea with
docstrings stripped. 784 passed across both files.

Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
@ZacxDev
ZacxDev force-pushed the fix/site-every-store-off-the-contended-disk branch from 30aa044 to a6dd11e Compare September 10, 2026 14:43
ZacxDev added a commit that referenced this pull request Sep 10, 2026
…is doc kept saying they were

Two independent surfaces read the same minute disagree with the claim this
item and its PR commentary were built on: classic branch protection on main
returns no required status checks and enforce_admins=false, and the repo has
no rulesets and no rules applying to main. The gates are advisory. A red gate
is still a hazard — the other one, that nobody is forced to look at — but
"nobody can merge" was false and it inflated the urgency of every gate item
here. The line now says to re-read the setting rather than cite the doc.

Also records the SECOND, distinct timeout flake now reddening #1458:
test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_N, added hours
earlier by #1445's own ladder, SIGKILLed at its 120s subprocess bound. Not an
assertion, unreachable from #1458's diff, and green on #1462 minutes earlier
on the same base. Wall times CI-to-CI say the node was not inflated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
@ZacxDev

ZacxDev commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Gated on the MERGED tree — both tiers green, and the red CI is attributed

Picked this up from rank 1 of handoff-alert-recall-and-skill-consumers (talos-infra#1488). The
rank asked for exactly this work; the open-PR sweep found it already done here, so this is a gate

  • verification pass rather than a second implementation.

Merged treeorigin/main 2a4fe326 + this PR, clean ort merge. Sandbox tier (the one
Tekton runs), derivations built ONE AT A TIME:

  • pytests PASScollected=21958 passed=21956 skipped=2 failed=0, SCOPE: FULL (28 of 28 hermetic target(s)), RESULT: PASS (exit=0), NIXBUILD_RC=0
  • nodetests PASSTOTAL suites=5 files=41 tests=1449 pass=1449 fail=0 (floor 1367),
    RESULT: PASS (exit=0), NIXBUILD_RC=0

⚠ The dev-host tier (scripts/gate.sh) was not run on the merged tree.

The red tekton/devrc-pytests on this PR head

test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_Nnot reachable from this diff. It
lives in test_run_tests_targets.py and spawns run-tests.sh as a subprocess against
scripts/collector/i3/tests; this PR touches test_subsystem_store_api.py,
test_store_siting_ledger.py and two ci-repro/ files. It passes on current main on the dev
host (1 passed), and it passed on the merged tree above.

It is a distinct flake family from the fsync one this PR fixes, and the population is
measurable: surveying the 45 most-recent PRs, four failed tekton/devrc-pytests on a test in
test_run_tests_targets.py, each on an unrelated diff — #1429
(test_the_SUMMARY_BANNER_names_the_real_selection_source), #1450 and #1466 (both
test_a_partial_run_is_declared_where_gate_sh_actually_LOOKS), and this one. Every member of that
family shells out to a real run-tests.sh run, which is the shape that gets load-sensitive in the
sandbox. Worth its own guard eventually; it is not this PR's.

The closing condition, checked with an instrument independent of this PR's own census

An AST scan at class scope over test_subsystem_store_api.py, counting classes that build a store
and issue a write verb (so they fsync inside the request):

tree store-building classes raw and writing
origin/main sited 1, raw 6 2 classes / 16 test methods
merged with this PR sited 7, raw 0 0

And the guard was watched to go red. In a cp -a copy with .git removed and
PYTHONDONTWRITEBYTECODE=1:

  • control, unmutated: 24 passed
  • mutant — one sited site reverted to tmp_path / "store"
    (TestRefusedIsIndistinguishableFromAbsent.test_RECALL_…): KILLED, 1 failed, 23 passed,
    by test_the_disk_rooted_census_matches_the_allowlist_EXACTLY with its own message naming
    the exact site and line — not by a different guard's error, and not by the happy path resolving.

That is the half a per-file ledger could not do: before this PR, the same mutation passes, because
test_every_ledgered_file_IMPORTS_AND_CALLS_the_shared_siting_at_least_once only asks whether the
file uses store_siting anywhere.

Why this mattered — the concrete payout

devrc#1435's red CI was TestARefusedWriteIsIndistinguishableFromAnAbsentOne::test_POSITIVE_CONTROL_the_APPEND_comparison_CAN_see_the_difference
— one of the 16 above, and the handoff's leading hypothesis was the _free_port() TOCTOU. That
hypothesis is wrong: the test drives the server through running(), which builds it in-process
with port=0 and never calls _free_port(). It was the fsync mechanism this PR removes. The same
test also failed CI on #1434 and #1447.

⚠ Scope note on the reason strings: the allowlist's "read-only" entries claim no write verb reaches
_replace_bytes, which is the fsync mechanism specifically — not that a disk-backed root is
harmless. The PR says this itself; repeating it because it is the sentence most likely to be
compressed into "covered".

@ZacxDev
ZacxDev merged commit ce9b55c into main Sep 10, 2026
2 of 3 checks passed
@ZacxDev
ZacxDev deleted the fix/site-every-store-off-the-contended-disk branch September 10, 2026 20:56
ZacxDev added a commit that referenced this pull request Sep 11, 2026
…ayer — the fix already existed and had never reached the failing site

The SERVER_BLOCKED_IN_FSYNC diagnosis stands. What was wrong was the remedy set:
(a) bound the write path was recommended as "the only one that makes the SERVER
correct", but it is a production change to crash-durability semantics made to
close a test-harness siting gap. The tmpfs fix from #1211/#1219/#1239 sited 5
store roots and left 18 open-coded on disk, and the test that keeps reddening
the gate is one of the 18 — it is also the only test in its class that gets a
200, so it is the only one that executes the two in-request fsyncs.

Records what #1458 ships, the independently re-run census mutation, that
slowfsync.c could not measure a siting fix without the new opt-in, and the two
residuals: nothing is measured in CI, and a green gate on #1458 is not the
verifier (handoff-gate-flake-store-api.md rank 1 is).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
ZacxDev added a commit that referenced this pull request Sep 11, 2026
…is doc kept saying they were

Two independent surfaces read the same minute disagree with the claim this
item and its PR commentary were built on: classic branch protection on main
returns no required status checks and enforce_admins=false, and the repo has
no rulesets and no rules applying to main. The gates are advisory. A red gate
is still a hazard — the other one, that nobody is forced to look at — but
"nobody can merge" was false and it inflated the urgency of every gate item
here. The line now says to re-read the setting rather than cite the doc.

Also records the SECOND, distinct timeout flake now reddening #1458:
test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_N, added hours
earlier by #1445's own ladder, SIGKILLed at its 120s subprocess bound. Not an
assertion, unreachable from #1458's diff, and green on #1462 minutes earlier
on the same base. Wall times CI-to-CI say the node was not inflated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
ZacxDev added a commit that referenced this pull request Sep 11, 2026
…ged RED on the second flake

This doc's rank 22 called #1458 open; it landed while the doc sat in review,
which is the stale-claim class this ladder kept finding — here staled by an
external event rather than a sibling commit.

Verified by content on origin/main, not by ancestry: a squash makes
merge-base --is-ancestor false forever. sited_root, _DISK_ROOTED_ALLOWLIST,
the operand-type guard and skip_tmpfs_enabled are all present; the single
remaining tmp_path / "store" is a docstring at :367.

Also records that it merged with pytests RED on the unrelated #1445 timeout
flake, because "merged" and "merged green" are different claims.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
ZacxDev added a commit that referenced this pull request Sep 11, 2026
… — the fix existed and had never reached the failing site (#1462)

* docs(handoff): rank 22's own three remedies were aimed at the wrong layer — the fix already existed and had never reached the failing site

The SERVER_BLOCKED_IN_FSYNC diagnosis stands. What was wrong was the remedy set:
(a) bound the write path was recommended as "the only one that makes the SERVER
correct", but it is a production change to crash-durability semantics made to
close a test-harness siting gap. The tmpfs fix from #1211/#1219/#1239 sited 5
store roots and left 18 open-coded on disk, and the test that keeps reddening
the gate is one of the 18 — it is also the only test in its class that gets a
200, so it is the only one that executes the two in-request fsyncs.

Records what #1458 ships, the independently re-run census mutation, that
slowfsync.c could not measure a siting fix without the new opt-in, and the two
residuals: nothing is measured in CI, and a green gate on #1458 is not the
verifier (handoff-gate-flake-store-api.md rank 1 is).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726

* docs(handoff): RETRACTED — the Tekton checks are not required, and this doc kept saying they were

Two independent surfaces read the same minute disagree with the claim this
item and its PR commentary were built on: classic branch protection on main
returns no required status checks and enforce_admins=false, and the repo has
no rulesets and no rules applying to main. The gates are advisory. A red gate
is still a hazard — the other one, that nobody is forced to look at — but
"nobody can merge" was false and it inflated the urgency of every gate item
here. The line now says to re-read the setting rather than cite the doc.

Also records the SECOND, distinct timeout flake now reddening #1458:
test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_N, added hours
earlier by #1445's own ladder, SIGKILLed at its 120s subprocess bound. Not an
assertion, unreachable from #1458's diff, and green on #1462 minutes earlier
on the same base. Wall times CI-to-CI say the node was not inflated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726

* docs(handoff): #1458 MERGED as ce9b55c — verified by content, and merged RED on the second flake

This doc's rank 22 called #1458 open; it landed while the doc sat in review,
which is the stale-claim class this ladder kept finding — here staled by an
external event rather than a sibling commit.

Verified by content on origin/main, not by ancestry: a squash makes
merge-base --is-ancestor false forever. sited_root, _DISK_ROOTED_ALLOWLIST,
the operand-type guard and skip_tmpfs_enabled are all present; the single
remaining tmp_path / "store" is a docstring at :367.

Also records that it merged with pytests RED on the unrelated #1445 timeout
flake, because "merged" and "merged green" are different claims.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev added a commit that referenced this pull request Sep 11, 2026
…pe and nothing else

test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_N spawns a nested full
run-tests.sh under a hard timeout=120 and is SIGKILLed at the bound. It shares
the observable with the store-api flake — a check red on a diff that cannot
reach it — and no mechanism: this is subprocess.TimeoutExpired with rc -9, not
an assertion, and grepping MECHANISM = finds nothing because no store server
is involved. Added by #1445's own ladder hours before it first went red.

Ruled out with values: any diff (#1458's four files cannot reach that file);
determinism (#1462 passed minutes earlier on the same base); general node load
(failing vs passing run 1030.34s vs 1065.64s and 155.60s vs 154.73s — the
failing run was faster). NOT established: what the nested run blocked on; the
pipelinerun was pruned within the hour.

Rank 1 gains two corrections it cannot be measured without: the anchor moved
to ce9b55c (#1458 sited 18 roots that #1211/#1219/#1239 had left on disk), so
a rate against 65f7325 spans two interventions; and the population is no
longer one mechanism, so classify by failing test before counting. Also
retracts "required check" there — measured: no required checks, no rulesets,
enforce_admins=false.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
ZacxDev added a commit that referenced this pull request Sep 11, 2026
…d it was never the worst one (#1512)

Rank 3 of the gate-speed handoff read "needs a diagnosis". It did not: the
suite's own classifier had already printed MECHANISM = SERVER_BLOCKED_IN_FSYNC
on devrc-ci-86zxj, and #1458 (ce9b55c) then removed that mechanism by siting
the store roots on tmpfs. Nothing recorded that the fix had LANDED, so the next
session would have re-derived the whole thing.

Measured over tekton/devrc-pytests, newest verdict per PR head, split on
ce9b55c's timestamp:

    window     verdicts  genuine failures  this test
    pre-fix         125                29          4
    post-fix         45                 6          0

Two things stated rather than glossed:

* THE POWER IS WEAK. At the pre-fix per-verdict rate (3.2%) the expected count
  in 45 verdicts is ~1.4, so P(0) is about 0.23. The zero is CONSISTENT with
  the fix and does not establish it; the mechanism's removal is what does.
  Written into the comment as not-proven so it cannot be upgraded by a reader.

* IT WAS NEVER THE WORST FLAKE. In the same pre-fix window
  test_every_decrypt_family_VERDICT_is_pinned_WHOLE failed 8 times to this
  test's 4, and is also at 0 post-fix. This one was ranked and worked because
  it had a long diagnosis attached, not because it was frequent.

Also corrects the port-race docstring, which still read as though that
intermittent were live. Its "whether closing this race moves that rate is
UNKNOWN" was right and is why the classes stayed separate: had they been merged
into one story, the tmpfs fix would have been credited to the port-race retry
and the real mechanism would still be in the request path.

Comment-only; no behaviour change. Verified: the full file is 760 passed.


Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
ZacxDev added a commit that referenced this pull request Sep 11, 2026
…pe and nothing else (#1477)

* docs(gate-flake): rank 7 — a SECOND gate flake sharing this doc's shape and nothing else

test_the_subset_note_reports_N_of_the_FULL_set_not_N_of_N spawns a nested full
run-tests.sh under a hard timeout=120 and is SIGKILLed at the bound. It shares
the observable with the store-api flake — a check red on a diff that cannot
reach it — and no mechanism: this is subprocess.TimeoutExpired with rc -9, not
an assertion, and grepping MECHANISM = finds nothing because no store server
is involved. Added by #1445's own ladder hours before it first went red.

Ruled out with values: any diff (#1458's four files cannot reach that file);
determinism (#1462 passed minutes earlier on the same base); general node load
(failing vs passing run 1030.34s vs 1065.64s and 155.60s vs 154.73s — the
failing run was faster). NOT established: what the nested run blocked on; the
pipelinerun was pruned within the hour.

Rank 1 gains two corrections it cannot be measured without: the anchor moved
to ce9b55c (#1458 sited 18 roots that #1211/#1219/#1239 had left on disk), so
a rate against 65f7325 spans two interventions; and the population is no
longer one mechanism, so classify by failing test before counting. Also
retracts "required check" there — measured: no required checks, no rulesets,
enforce_admins=false.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726

* docs(gate-flake): rank 7 is keyed to the FILE — a second test hit the same bound, and my provenance claim was wrong

A third occurrence landed while this PR was open, on a DIFFERENT test in the
same file: test_the_SUMMARY_BANNER_names_the_real_selection_source, #1462 at
dc97239, with the identical TimeoutExpired — and its spawned command carries
no --targets, so it is an entire suite run inside one test at a 120s bound.
The unit is therefore _run's shared timeout at :102, not either test name, and
the closing condition now keys on the file. Keyed to one test name, a second
exposed test walks straight past it.

Retracts this item's own provenance claim: "added by #1445's ladder" is true
of the subset-note test and false of the file and of _run, both from #1073
(809486f). #1445 added one more caller to a bound that already existed.

Also corrects the determinism evidence: #1462 passed this file at 8df1117 and
failed it at dc97239 — two docs-only commits on the same base, different test
failing — which is stronger than the cross-PR comparison it replaces.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726

* docs(gate-flake): rank 7 rewritten from measurements — the diagnosis was wrong and so were three of its rule-outs

An adversarial round on this PR returned four 🔴 and I could re-derive all of
them. Replacing the item rather than patching it, because most of what it said
was not true:

- The unit was `_run`'s bound. There are SIX bound sites; `_run_env:862` is the
  one behind 2 of the 3 occurrences, so the old closing condition was walkable
  by fixing `_run` alone.
- "Ruled out: general node load" is now the LEADING hypothesis. Measured: the
  flaking spawns take 57.20s and 47.10s against a 120s bound, and the same file
  on the same tree measures 205s and 428.40s — 2.09x ambient variance.
- "Ruled out: determinism" rested on a same-base comparison that is false; the
  branch was rebased across ce9b55c between the green and the red.
- "A nested FULL run" is false for both tests — argv or DEVRC_TARGETS narrows
  each to one target, so half the Next probe had nothing to do. Dropped.
- The provenance sentence has now been wrong three times (#1445 never touched
  the file; #289 created it, #1073 added both flaking tests). Deleted rather
  than corrected a fourth time — nothing depended on it.
- A third occurrence, #1454, was uncounted, and its diff DOES touch
  run-tests.sh, so the "cannot reach it" rule-out never covered it.

Adds what the item lacked: the detector argument (raise the bound, do not
remove it — a full set is ~28x a one-target run), and a closing condition with
a positive control, since the old one was satisfiable by a rename or a skip.

Rank 1: the anchor sentence named the wrong contaminated window, and the
branch-protection note duplicated the Gotchas entry minus its qualifiers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726

* docs(gate-flake): rank 7 asserts NO cause — three drafts asserted one, and a fourth round refuted the newest

Round 2 returned four more findings and I re-derived every one. The pattern is
now unambiguous: each rewrite of this item invented a cause it had not measured.
This version states none.

Refuted MYSELF, against the auditor's own leading rival: #1429 (a0839ec) did
NOT double CI parallelism. It changed min(nproc,4) to min(nproc,cgroup quota,8),
and devrc-ci-gate's pytests step sets limits.cpu: "4" — both formulas yield 4.
It doubled only on an unquotaed host.

Re-measured the headroom myself rather than inheriting it, and the numbers do
not reproduce: :418 43.48s (audit said 70.94s), :988 40.51s (51.49s), :723
38.89s (84.22s), :582 37.78s (69.70s); the file 137.69s and 164.27s against
205s and 428.40s. Four measurements, 3.11x spread, same tree. THAT is the
finding — no point wall time here is stable, so the item now states the range
and the spread and tells the reader not to quote a point value.

Fixed from round 2: "or the nested runs made concurrent" is forbidden
(run-tests.sh:3889 NESTED RUNS MUST BE SERIAL, enforced :4043, with a live
guard) and is removed; the kubectl recipe regains KUBECONFIG=$KC_HOMELAB and
drops the false "nowhere else"; "~28x" as a target COUNT becomes the measured
1194s serial full set; :418 and :582 are named for the first time; the date
predicate becomes ancestry; and the tekton SKILL.md cross-reference is replaced
by the live measurement plus a warning that SKILL.md is stale and says the
opposite.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v
Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726

* docs(gate-flake): restore why rank 7's key is the six sites — the retracted draft, not just the conclusion

Closes the last open finding from round 2: the rewrite prevented the hazard
structurally but deleted the record of why. CLAUDE.md calls a retracted theory
the most valuable content in the codebase precisely because the next reader
re-derives it otherwise — and here the narrowing is tempting twice over, since
keying to one test name OR to _run alone both look sufficient and both miss
:988 via _run_env, which is 2 of the 3 observed occurrences.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4p3xnvfqhRRjAUGKLK96v

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev added a commit that referenced this pull request Sep 11, 2026
… not a flake, and #1458 never touched it (#1531)

`#1512` left a comment in `test_subsystem_store_api.py` saying
`test_every_decrypt_family_VERDICT_is_pinned_WHOLE` failed 8 times to this
test's 4 — "twice as often" — and concluding: COUNT them before choosing which
flake to chase. The count was right. The conclusion was wrong, and counting is
what produced it.

That test is not intermittent. Measured 2026-09-11 over the 200
most-recently-updated PR heads (`tekton/devrc-pytests`, 2026-09-05..09-11, 197
terminal verdicts):

  * MECHANISM — it is not on #1458's path at all. The escrow file imports no
    store server, no `store_siting`, no `build_server`; it drives
    `escrow-verify.py` against an in-memory `FakeDownloader`, so
    `server.py:_replace_bytes`'s in-request fsync — the one thing `ce9b55c3`
    removes — is never reached. `ce9b55c3`'s diff names the file zero times,
    and the test runs in 1.71 s.
  * TIME — all 8 failures fall inside ONE 14-hour window on 2026-09-08
    (05:44Z..19:44Z) across 8 distinct heads, and every one reports `failed=7`
    or `failed=8`: a whole-suite red hitting every open PR at once.
  * CAUSE — nixpkgs moved `age` to 1.3.2, changing its tamper classification,
    and that test pins those verdicts by exact string equality. Re-keyed by
    #1392 (`94f82796`, 09-08T18:26Z) and #1403 (`4f49f5dc`, 20:32Z) — two days
    before `ce9b55c3` (09-10T20:56Z) existed. The two failures after 18:26Z are
    stale-base heads that had not picked up #1392.

Contrast, same instrument: the test #1458 actually fixed failed 5 times across
4 separate days (09-06, 09-07, 09-08, 09-09 x2) — scattered, which is what
contention looks like.

So the comment now carries a discriminator instead of "count them": a flake
scatters across days and reports `failed=1`; an environment red clusters in one
window and reports the same N>1 on every head. Count only what survives both.

Also records the blind spot both counts inherit: GitHub truncates a status
description at 138 characters, so only the FIRST failing test is ever named and
every count here is a lower bound. Bounded by arithmetic on the same rows — of
14 post-fix failures, 12 derive `failed=1`, one derives 2, one is unparseable.

The escrow test gains its own delisting note, so the next flake hunt does not
re-derive this: a red there is a claim about the `age` binary, not about CI load.

Comment-only, two files, no behaviour change.


Claude-Session: https://claude.ai/code/session_01LWAGG1mTfkozfKa5MMjXuC
Claude-Session-Id: 9b6235dd-2c40-4230-b6d3-5fa6d24a74fd

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZacxDev added a commit that referenced this pull request Sep 12, 2026
…1462), corrected (#1525); th (#1548)

Claude-Session-Id: d775cf17-db66-4e6e-a90e-f67e3ca5f726
ZacxDev added a commit that referenced this pull request Sep 12, 2026
MEASURED, and it is the reason this exists: of the SIX genuine
`tekton/devrc-pytests` failures in the whole post-#1458 window, THREE — half —
were the same test on PRs 17, 40 and 40 commits behind `main`, all since merged,
and that test passes on current `main`. An earlier sample had 8 of 8 failing
open PRs 5-42 commits behind, with a rebase curing 4 outright. Nothing blocks a
merge here, so the only value the gate has is whether a human believes a red —
and half the genuine reds are already-fixed failures a human then debugs against
a diff that cannot reach them.

`scripts/stale-base-triage.py` answers that deterministically, running no tests:
it reads the NEWEST `tekton/devrc-pytests` status per context, maps each failing
test named in the description to the ONE file defining it, and emits the evidence
pair — the PR head's copy of that file is byte-identical to the merge-base's (so
this change never touched it) while `main` HAS moved it, in commits absent from
the head. Verdict per PR: INHERITED / NOT EXPLAINED BY STALENESS / COULD NOT
MEASURE.

Measured live, read-only, over all 51 open PRs: 20 red heads, 3 INHERITED
(#1286, #1194, #1038), 2 NOT EXPLAINED, 15 COULD NOT MEASURE, 4 broken-gate
`error` rows counted on their own line. That non-zero 3 is the positive control
for the count: a zero here would otherwise be indistinguishable from an
instrument wired to nothing.

🔴 REPORT ONLY BY DEFAULT. `COMMENT_MODE_DEFAULT` is the literal `"off"`, pinned
by `test_the_comment_mode_default_is_the_LITERAL_off` both as a value and as
source text, so arming the PR-comment writer costs a visible line in the arming
commit. Nothing was posted to GitHub at any point.

Traps this repo has already paid for, each handled and each pinned by a test:
* `/commits/{sha}/status` (SINGULAR) maps `error` onto `failure` — never read.
  Only the plural list endpoint, which carries no roll-up field at all.
* the list is NEWEST-FIRST — folded on `max(created_at)`, order-independent.
* `error` is a broken gate, not a code failure — its own summary line, never in
  the red total.
* a SQUASH merge never makes a head an ancestor of its base; ancestry is asked
  only about the PR head, and every "this landed" claim is made by blob OID.
* `git diff --quiet <ref> -- <path>` exits 0 when the path exists on NEITHER
  side; existence is proved with `git cat-file -e` first, pinned by an AST scan
  of the git invocations rather than a string search that its own comment would
  satisfy.
* the 140-char description cap means a description can prove "at least one test
  failed and here is its name" but never "these are all of them" — so an
  INHERITED verdict requires `failed=N` to have survived AND to equal the number
  of names. That is conservative on purpose: dismissing a real red is the one
  error this tool must not make.
* the test-name -> file mapping is searched, never guessed, with a distinct
  outcome for not-found, ambiguous, class-qualified, parametrised and
  cap-truncated names.

Testing. 56 tests; RED at base `60194765` (the module cannot collect — the
script does not exist there), GREEN at HEAD. Three enumerated mutation rounds
under PYTHONDONTWRITEBYTECODE=1, mutants derived from the AST plus targeted
textual ones: round 1 74 mutants / 15 survivors AND THE POSITIVE CONTROL
SURVIVED, which is what found that `CONTEXT` was pinned nowhere; round 2 82 / 3
with the control killed; round 3 81 mutants, 78 killed, 0 survived, control
killed. Round 1's survivors also found a real defect — the `on_main` half of the
evidence pair was a guard that can never run (`commits_touching` already walks
`merge-base..main`), so it is now measured and printed for verification rather
than gated on, and `not in_head` moved into `prove_candidates` where a test can
reach it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0
ZacxDev added a commit that referenced this pull request Sep 12, 2026
MEASURED, and it is the reason this exists: of the SIX genuine
`tekton/devrc-pytests` failures in the whole post-#1458 window, THREE — half —
were the same test on PRs 17, 40 and 40 commits behind `main`, all since merged,
and that test passes on current `main`. An earlier sample had 8 of 8 failing
open PRs 5-42 commits behind, with a rebase curing 4 outright. Nothing blocks a
merge here, so the only value the gate has is whether a human believes a red —
and half the genuine reds are already-fixed failures a human then debugs against
a diff that cannot reach them.

`scripts/stale-base-triage.py` answers that deterministically, running no tests:
it reads the NEWEST `tekton/devrc-pytests` status per context, maps each failing
test named in the description to the ONE file defining it, and emits the evidence
pair — the PR head's copy of that file is byte-identical to the merge-base's (so
this change never touched it) while `main` HAS moved it, in commits absent from
the head. Verdict per PR: INHERITED / NOT EXPLAINED BY STALENESS / COULD NOT
MEASURE.

Measured live, read-only, over all 51 open PRs: 20 red heads, 3 INHERITED
(#1286, #1194, #1038), 2 NOT EXPLAINED, 15 COULD NOT MEASURE, 4 broken-gate
`error` rows counted on their own line. That non-zero 3 is the positive control
for the count: a zero here would otherwise be indistinguishable from an
instrument wired to nothing.

🔴 REPORT ONLY BY DEFAULT. `COMMENT_MODE_DEFAULT` is the literal `"off"`, pinned
by `test_the_comment_mode_default_is_the_LITERAL_off` both as a value and as
source text, so arming the PR-comment writer costs a visible line in the arming
commit. Nothing was posted to GitHub at any point.

Traps this repo has already paid for, each handled and each pinned by a test:
* `/commits/{sha}/status` (SINGULAR) maps `error` onto `failure` — never read.
  Only the plural list endpoint, which carries no roll-up field at all.
* the list is NEWEST-FIRST — folded on `max(created_at)`, order-independent.
* `error` is a broken gate, not a code failure — its own summary line, never in
  the red total.
* a SQUASH merge never makes a head an ancestor of its base; ancestry is asked
  only about the PR head, and every "this landed" claim is made by blob OID.
* `git diff --quiet <ref> -- <path>` exits 0 when the path exists on NEITHER
  side; existence is proved with `git cat-file -e` first, pinned by an AST scan
  of the git invocations rather than a string search that its own comment would
  satisfy.
* the 140-char description cap means a description can prove "at least one test
  failed and here is its name" but never "these are all of them" — so an
  INHERITED verdict requires `failed=N` to have survived AND to equal the number
  of names. That is conservative on purpose: dismissing a real red is the one
  error this tool must not make.
* the test-name -> file mapping is searched, never guessed, with a distinct
  outcome for not-found, ambiguous, class-qualified, parametrised and
  cap-truncated names.

Testing. 56 tests; RED at base `60194765` (the module cannot collect — the
script does not exist there), GREEN at HEAD. Three enumerated mutation rounds
under PYTHONDONTWRITEBYTECODE=1, mutants derived from the AST plus targeted
textual ones: round 1 74 mutants / 15 survivors AND THE POSITIVE CONTROL
SURVIVED, which is what found that `CONTEXT` was pinned nowhere; round 2 82 / 3
with the control killed; round 3 81 mutants, 78 killed, 0 survived, control
killed. Round 1's survivors also found a real defect — the `on_main` half of the
evidence pair was a guard that can never run (`commits_touching` already walks
`merge-base..main`), so it is now measured and printed for verification rather
than gated on, and `not in_head` moved into `prove_candidates` where a test can
reach it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0
ZacxDev added a commit that referenced this pull request Sep 12, 2026
…red (#1524)

* feat(ci-signal): name the commit that already fixed a PR's inherited red

MEASURED, and it is the reason this exists: of the SIX genuine
`tekton/devrc-pytests` failures in the whole post-#1458 window, THREE — half —
were the same test on PRs 17, 40 and 40 commits behind `main`, all since merged,
and that test passes on current `main`. An earlier sample had 8 of 8 failing
open PRs 5-42 commits behind, with a rebase curing 4 outright. Nothing blocks a
merge here, so the only value the gate has is whether a human believes a red —
and half the genuine reds are already-fixed failures a human then debugs against
a diff that cannot reach them.

`scripts/stale-base-triage.py` answers that deterministically, running no tests:
it reads the NEWEST `tekton/devrc-pytests` status per context, maps each failing
test named in the description to the ONE file defining it, and emits the evidence
pair — the PR head's copy of that file is byte-identical to the merge-base's (so
this change never touched it) while `main` HAS moved it, in commits absent from
the head. Verdict per PR: INHERITED / NOT EXPLAINED BY STALENESS / COULD NOT
MEASURE.

Measured live, read-only, over all 51 open PRs: 20 red heads, 3 INHERITED
(#1286, #1194, #1038), 2 NOT EXPLAINED, 15 COULD NOT MEASURE, 4 broken-gate
`error` rows counted on their own line. That non-zero 3 is the positive control
for the count: a zero here would otherwise be indistinguishable from an
instrument wired to nothing.

🔴 REPORT ONLY BY DEFAULT. `COMMENT_MODE_DEFAULT` is the literal `"off"`, pinned
by `test_the_comment_mode_default_is_the_LITERAL_off` both as a value and as
source text, so arming the PR-comment writer costs a visible line in the arming
commit. Nothing was posted to GitHub at any point.

Traps this repo has already paid for, each handled and each pinned by a test:
* `/commits/{sha}/status` (SINGULAR) maps `error` onto `failure` — never read.
  Only the plural list endpoint, which carries no roll-up field at all.
* the list is NEWEST-FIRST — folded on `max(created_at)`, order-independent.
* `error` is a broken gate, not a code failure — its own summary line, never in
  the red total.
* a SQUASH merge never makes a head an ancestor of its base; ancestry is asked
  only about the PR head, and every "this landed" claim is made by blob OID.
* `git diff --quiet <ref> -- <path>` exits 0 when the path exists on NEITHER
  side; existence is proved with `git cat-file -e` first, pinned by an AST scan
  of the git invocations rather than a string search that its own comment would
  satisfy.
* the 140-char description cap means a description can prove "at least one test
  failed and here is its name" but never "these are all of them" — so an
  INHERITED verdict requires `failed=N` to have survived AND to equal the number
  of names. That is conservative on purpose: dismissing a real red is the one
  error this tool must not make.
* the test-name -> file mapping is searched, never guessed, with a distinct
  outcome for not-found, ambiguous, class-qualified, parametrised and
  cap-truncated names.

Testing. 56 tests; RED at base `60194765` (the module cannot collect — the
script does not exist there), GREEN at HEAD. Three enumerated mutation rounds
under PYTHONDONTWRITEBYTECODE=1, mutants derived from the AST plus targeted
textual ones: round 1 74 mutants / 15 survivors AND THE POSITIVE CONTROL
SURVIVED, which is what found that `CONTEXT` was pinned nowhere; round 2 82 / 3
with the control killed; round 3 81 mutants, 78 killed, 0 survived, control
killed. Round 1's survivors also found a real defect — the `on_main` half of the
evidence pair was a guard that can never run (`commits_touching` already walks
`merge-base..main`), so it is now measured and printed for verification rather
than gated on, and `not in_head` moved into `prove_candidates` where a test can
reach it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0

* fix(stale-base-triage): the verdict was a function of the failing test's NAME LENGTH (F2)

`names_provably_complete` could only prove completeness by reading `failed=N`
out of the status description. `failed=` is the LAST field in the gate's banner
and GitHub caps a description at 140 BYTES, so whether it survives is decided by
how long the failing test's name is — and on the three PRs this tool's
requirement was measured on (#1454, #1462, #1499) it did not survive. All three
resolved to COULD NOT MEASURE with the correct answer already in the row. The
tool fired on 0 of the 3 cases that justified it.

The fix is a SECOND route, and it is a proof rather than a heuristic.
`scripts/run-tests.sh` GUARD 4 defines

    collected = passed + skipped + failed + errors + xfailed + xpassed
    TOT_FAILED += failed + errors

so `collected - passed - skipped == failed + xfailed + xpassed >= failed`, and
names are a subset of the failures, so `len(names) <= failed <= derived`. When
`derived == len(names)` the inequality is squeezed shut and completeness is
PROVEN. `collected=`, `passed=` and `skipped=` all precede `failed=`, so they
are exactly the fields that survive the cut. The direct `failed=N` route is kept
and still tried first, because it is the stronger claim.

🔴 THE BOUND IS AN OVER-COUNT, NEVER AN UNDER-COUNT, and that direction is the
whole safety argument: an under-count would certify a completeness the row does
not have and dismiss a real red. `xfailed`/`xpassed` can only inflate it, and
`_TOTALS_RE` demands the three fields ADJACENT AND IN ORDER so that a number the
cap cut SHORT cannot shrink it — `collected=` and `passed=` are each proven
intact by having matched banner text after them, and `skipped=`, the only one
that can be short, subtracts LESS and so errs toward withholding.

MEASURED, read-only, 2026-09-11 (provably-complete rows / red heads):

  population                                   old      new
  100 most-recent closed PRs                   5/28    18/28
  - the three justifying PRs #1454 #1462 #1499  0/3      3/3
  58 open PRs                                  4/26     7/26
  - #1518, #1515 (9 commits behind, worked)     0/2      2/2

SOUNDNESS CONTROL on the same real rows: 17 carried BOTH a visible `failed=N`
and a derivable bound; the two agreed EXACTLY on all 17, with zero under-counts.
Genuine multi-failure reds stay withheld — #1177 derives 39 against one name,
#1280 failed=5, #1440 failed=3, all still COULD NOT MEASURE.

The coupling to `run-tests.sh` is pinned mechanically rather than asserted:
`test_the_run_tests_collected_arithmetic_this_derivation_rests_on_is_pinned`
reads the `collected=$((…))` and `TOT_FAILED=$((…))` expressions out of the
shell source and checks the CONTAINMENT relation, so renaming the shell locals
is fine and changing WHAT IS SUMMED is not. What it cannot check — that
`xfailed`/`xpassed` are non-negative — is stated in the test, not asserted.

Prior art is now cited in the header: `main-status-watch.py`'s
`screen_all_known_flakes` is the same completeness gate in 15 lines, and its
"would have fired ZERO times on 100 commits" is the same `failed=N` route being
eaten by the same cap. This file's contribution is the derived route and the
blob/commit evidence half; the gate itself is prior art and says so.

Red/green matrix: 13 of the 14 new tests fail at 5ad08e5 and pass at HEAD; the
14th is labelled in its own docstring as an invariant guard, not regression
coverage. Suite: 56 -> 70 tests, all green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0

* test(stale-base-triage): close the three gaps the F2 mutation sweep found

Round 1 of the sweep on the new completeness logic killed 10 of 15. Four of the
five survivors were real gaps in the tests, not in the code:

* M14 — reordering `passed=`/`skipped=` in `run-tests.sh`'s TOTAL banner
  SURVIVED, because the pin searched the whole FILE and `run-tests.sh` prints
  TWO banners (a full-run one and a SCOPED one). Mutating one was satisfied by
  the other. The pin now requires EVERY banner line to carry the ordered
  fields, and asserts it found at least two.
* M11 — spelling the `skipped` group `(\d*)` SURVIVED: no fixture fed a row cut
  exactly on the `=`, which is what a cap landing one character early produces
  and which makes `int("")` RAISE on the reporting path. Added as a fixture.
* M10 — replacing the underivable-row refusal with a bound of 0 SURVIVED,
  because `names` is non-empty by then so 0 can never equal it and the VERDICT
  is identical. Only the operator-facing reason distinguishes them, and the two
  refusals call for different next steps, so both reasons are now pinned as
  whole strings.
* M5b — the naive three-separate-searches spelling of the totals regex is now
  driven explicitly and is killed by the existing out-of-order fixture.

Two survivors are argued EQUIVALENT rather than fixed, and are labelled as such
in the source:

* M5 — matching the three fields with `.*` between them instead of adjacently.
  `.*` still requires the ORDER, and a single-banner description contains
  exactly one of each field, so no input this parser can receive distinguishes
  them. The adjacency is kept for the truncation argument it encodes.
* M13 — `run-tests.sh` SHRINKING what it sums into `failed=` keeps the derived
  value an UPPER bound, so the derivation stays sound. The pin checks
  CONTAINMENT on purpose; the added comment says so, and M15 (a term LEAVING
  `collected` — the change that does break it) is killed by the same pin.

Round 2 over the fixed tree: 6/8 killed, the two above surviving as argued, no
new finding — so the ladder ends there. Both controls held in both rounds
(unmutated tree green; `COMMENT_MODE_DEFAULT = "on"` killed by four tests).
Run under PYTHONDONTWRITEBYTECODE=1, each edit verified to have changed the
file, verdicts read from the runner's own result lines.

Suite: 70 -> 71 tests, all green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0

* refactor(ci-signal): one rule, one place — and the two copies had already diverged (D6)

`classify` in `stale-base-triage.py` was BYTE-IDENTICAL to the one in
`main-status-watch.py`. `_FAILING_RE`, `_FAILED_COUNT_RE`, `parse_failing_names`
and `parse_failed_count` were duplicated AND already divergent — only the newer
copy stripped the `TOTAL` truncation fragment, and only the newer copy folded
statuses by timestamp instead of by array position. The disagreement is the
finding; consolidating is what made it audible.

All five now live in `scripts/lib/ci_status.py`, with
`derived_failure_upper_bound` beside them. Both files import it by path. What
stays in each consumer is POLICY — which context may speak, which names are
known flakes, what completeness means, what to do about a red — because the two
answer those differently.

🔴 `main-status-watch.py` IS LIVE ON BOTH HOSTS, every 10 minutes, so each
divergence was reconciled deliberately rather than by taking whichever copy was
newer:

* THE FOLD. The live copy kept the FIRST row per context, which is correct only
  because GitHub returns the array newest-first — an assumption about someone
  else's response ordering, load-bearing, unstated and untested. (⚠ It is NOT
  the "yields the OLDEST post" bug: first-wins over a newest-first array yields
  the NEWEST. The defect is order-DEPENDENCE, not a wrong row today.) It now
  folds on `max(created_at)`, which agrees with first-wins on GitHub's current
  ordering and stays right if that changes. With no `created_at` anywhere every
  stamp compares equal and the FIRST row wins — the old rule, as the fallback.
  `test_newest_per_context_is_ORDER_INDEPENDENT_which_first_wins_was_not`
  reimplements the old rule and shows it returning the `pending` row on the
  order it does not expect, beside the new one right on both.
* THE CONTEXT FILTER moved out of the fold and into `commit_verdict`, where the
  policy belongs, and BEFORE the emptiness check — a commit carrying only a
  foreign pipeline's rows must have NO verdict, never green, or an unrelated
  green would close an open red episode. Pinned by a new unit test beside the
  existing end-to-end one.
* THE isinstance SKIP. The live copy deliberately had none, arguing the arm was
  unreachable (its walk validates every row is a dict and names the commit if
  not) and that silently dropping a row could drop the red. The shared function
  keeps the triage tool's skip: still unreachable for the watcher, unchanged
  behaviour for the triage tool. Neither contract moves; the module says why.
* THE FRAGMENT STRIP is now shared, and is provably INERT for the watcher: a row
  cut at `| TOTA` was necessarily cut before `failed=N` too, so its screen —
  which also demands the count — refuses either way.
* THE DERIVED COMPLETENESS ROUTE is deliberately NOT adopted by the watcher, and
  the file now says why: there, proving completeness makes the tool SPEAK; here
  it makes it stay SILENT about a red. Widening a silence over main's only
  automated detector is a change to make on purpose with its own measurement,
  not a free win inherited from a sibling.

Duplication is now refused mechanically:
`test_the_shared_predicates_are_NOT_re_declared_in_either_consumer` fails on a
local `def` of any of the five, in either file, and carries a positive control
so the scan cannot pass vacuously.

TWO PRE-EXISTING TESTS BROKE, both for the right reason, and both were fixed
rather than weakened: one used `def classify(...)` as its positive control for
the docstring-stripper (the function is simply elsewhere now — it uses
`commit_verdict` instead), and one copies the script to a tmp dir and runs it,
which has no sibling `lib/` (it now passes PYTHONPATH, and says why).

DEPLOYMENT. The unit runs the script straight out of the checkout
(`ExecStart=… %h/workspace/devrc/scripts/main-status-watch.py`), so a `git pull`
delivers both files and no home-manager switch is involved. A run landing in the
instant between the two files arriving fails to import; the unit is a
timer-driven oneshot with no `OnFailure=` toast and the next poll is 10 minutes
away, so that window costs one skipped poll. ⚠ `X-Restart-Triggers` in
`nix/home.nix` still names only `main-status-watch.py` and was deliberately NOT
edited — for a timer-driven oneshot that re-reads the file every run it is
cosmetic, and touching `home.nix` here would widen this change's blast radius
for no behavioural gain.

`test_main_status_watch.py`: 96 -> 99 tests, all green.
`test_stale_base_triage.py`: 71 tests, all green. Both suites: 170 green.
No live-system change was made: the deployed copy is the base clone on `main`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0

* refactor(stale-base-triage): delete --fetch and --json; keep --sweep, --help and the env seams (D1-D5)

The six deletion candidates were assessed against the BROKEN tool, so each was
re-measured after F2 rather than acted on. Decisions, with the evidence:

DELETE `--fetch`. It was the only git subcommand in this file that WRITES, in a
tool whose stated safety property is that it writes nothing, and `refs/` lives
in the COMMON git dir — a worktree gives zero isolation there, so "namespaced
under refs/stale-base-triage/*" was a weaker claim than the one the header
makes. MEASURED: this repo's PRs are same-repo branches and `origin`'s refspec
is `+refs/heads/*:refs/remotes/origin/*`, so an ordinary `git fetch origin`
already brings every head — 58 of 58 open PR heads resolved in the base clone
with no `--fetch` anywhere. It bought nothing and cost the tool its absolute
safety claim. The UNMEASURED message now says `git fetch origin`.

The guard got STRONGER, not weaker: `test_only_refs_under_its_own_namespace_are_
ever_written` (a relative claim) is replaced by `test_no_git_subcommand_that_
WRITES_is_ever_invoked`, which enumerates the read-only subcommands as an
ALLOWLIST off the AST — so a subcommand nobody enumerated is a write by default,
and `remote` is admitted only in its `get-url` form.

DELETE `--json`. No consumer: a whole-repo search for `stale-base-triage` /
`STALE_BASE_TRIAGE` returns this script and its test file, nothing else — no
unit, no skill, no nix, no caller. F2 does not change that; it makes the
RENDERED sweep useful, not a machine payload. It cost a module-level
`_JSON_MODE`, a `say()` indirection on every human line in the file, and a
second exit path computing `n_inherited` its own way.

KEEP `--sweep`, and it is now the primary mode. Its rationale is exactly what F2
reversed: MEASURED read-only over 58 open PRs, the sweep surfaces 7 provably
complete reds where it surfaced 4, and the three it gained (#1518, #1515, #1450)
are 9-94 commits behind and actively worked — not the 259/357/508-behind
abandoned ones the old gate happened to let through.

KEEP the self-parsing `--help`. What it prints is the `#` header — the env
ledger, the exit codes, the measured rationale — which argparse cannot see. The
alternative is copying the ledger into an epilog, i.e. reintroducing the
duplication the previous commit just removed, and the ledger is two-way pinned
against the code that reads it.

KEEP all four env seams and the ledger test. `STALE_BASE_TRIAGE_GH` and
`_REPO` are the seams the test harness points at a stub — they are why no test
in this file can reach real GitHub. `_BUDGET` is the only way to drive the
budget guard to zero. `_COMMENT_MODE` is the arming seam and FAILS CLOSED
across seven spellings. Each has a concrete in-repo consumer; none is
speculative.

`COMMENT_MODE_DEFAULT` is untouched: still the literal `"off"`, still pinned
both ways (value and source text). Nothing was posted to GitHub; every live run
in this work was read-only with mode=off.

VERIFIED AGAINST THE ORIGINAL SYMPTOM, on live data, not inferred from the code:
  old code (5ad08e5), `--pr 1518`:  VERDICT: COULD NOT MEASURE,  rc 0
  this tree,           `--pr 1518`:  VERDICT: INHERITED,          rc 10
and a four-PR read-only sweep returns INHERITED for #1518/#1515/#1450 with the
fix commits named, while #1177 (derived 39 against one name) stays COULD NOT
MEASURE with the HINT line.

⚠ The deleted-flag guard was SPELLED on its first draft and failed on the header
paragraph that explains the deletion — the same trap `_git_subcommands` is
AST-based to avoid. It now reads `add_argument` literals, module-level
assignments and string constants off the AST, so prose can neither satisfy nor
break it.

Suite: 71 tests, all green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0

* docs(stale-base-triage): the cap is 140 BYTES, and the two files disagreed about it

`scripts/lib/ci_status.py` said BYTES and `stale-base-triage.py` said
CHARACTERS, in prose describing the same boundary. Bytes is what was measured:
every real row sampled on this repo is 138 CHARACTERS, which is 140 bytes once
the `—` in `FAILED: pytests —` is counted as the three UTF-8 bytes it is. A
140-CHARACTER cap would have produced 138-byte, 140-character rows, and none
was seen.

⚠ WHAT THAT MEASUREMENT DOES NOT SETTLE, now said in the file rather than
implied: whether the byte cut is GitHub's or the posting pipeline's own
truncation. Every row sampled carries exactly one multi-byte character, so the
two hypotheses were never separated — a row with two em-dashes would do it, and
none exists. The boundary is measured; its owner is not.

It matters because a fixture built on a character cut lands two bytes late and
quietly leaves `failed=` readable — i.e. it would exercise the DIRECT route
while claiming to test the derived one, which is a green proving nothing.
`cut_like_github` in the test file cuts on bytes for exactly that reason and
`test_CONTROL_the_synthetic_rows_land_on_the_real_140_BYTE_boundary` asserts
both lengths.

No behaviour change. Suite: 71 tests, all green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0

* test(no-real-launchers): the consolidation made main-status-watch.py a `home-manager` hit — acknowledged and PINNED

`test_every_hazardous_binary_the_scripts_reach_is_stubbed_or_acknowledged` went
red on the D6 commit, and it was right to. `launcher_scan.hazard_hits` is a TEXT
scan over `scripts/` and says so in its own docstring — a prose mention counts,
deliberately and fail-closed. The shared-module import comment explains that the
systemd unit runs this file straight out of the checkout, so a `git pull`
delivers both files and no home-manager switch is involved, and that sentence is
exactly why importing a sibling module is safe here. It is the FIFTH file of
this shape.

🔴 RE-JUSTIFIED, NOT REWORDED, which is this repo's stated convention and is
written into the entry above the table — rewording to dodge the scanner removes
the sentence and keeps the risk.

🔴 AND PINNED, because that table entry says in its own words that an
acknowledgement "would otherwise blind the guard" — and it has been blinded
before: adding a real `home-manager switch` to a script whose name was already
acknowledged for other reasons produced a real launch with 54 guard tests green.
So two new tests carry this row:

  test_main_status_watch_SPAWNS_these_argv0_AND_NOTHING_ELSE — an AST walk
  asserting the spawn argv[0] set is exactly {git, <computed>, <not-a-list>},
  GROWS-OR-SHRINKS. Both opaque entries are named rather than waved at:
  `<computed>` is `[gh, "api", path]` behind the MAIN_STATUS_WATCH_GH stub seam,
  `<not-a-list>` is `trigger_deadman`'s `cmd`, whose production literal is
  pinned exactly — and refused a `--force` — by an existing test. The comment
  says outright that this pin is not the whole story on its own.

  test_home_manager_is_MENTIONED_but_never_SPAWNED — both halves of the
  acknowledgement's claim: the mention must still EXIST (or the row has outlived
  its sentence) and it must remain a MENTION, asserted against the argv[0] set
  AND against `_code_only()`, which strips comments and docstrings by AST.

BOTH CONTROLS WATCHED, in both directions:
  * injecting `subprocess.run(["home-manager", "switch"])` fails BOTH new tests,
    each with its own message — not a different guard's;
  * rewording the mention to "no HM switch" fails the mention-must-exist half
    AND the ledger's own file-set pin, because the row then over-claims.

⚠ FOUND BY RUNNING THE WHOLE `scripts/tests` TARGET, not by the change-scoped
mapping: `scoped-tests.sh` selected only the two suites named in the diff, and
this guard reads every file under `scripts/` regardless of what changed. A
change that only adds a COMMENT can go red here.

scripts/tests/{test_no_real_launchers,test_main_status_watch,test_stale_base_
triage}.py together: 252 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0

* fix(ci-signal): the consolidation moved code out from under the citation guard, and the first citation written into the new module dangled

ROUND 1 audit findings on #1524. Six filed, six addressed, plus one the same
fixture exposed in the sibling parser.

F1 — `scripts/lib/ci_status.py` cited
`test_newest_per_context_skips_a_malformed_row_rather_than_raising`, which
existed nowhere, and the arm it claimed was guarded was unguarded: deleting
`if not isinstance(row, dict): continue` SURVIVED all 172 tests. The deeper
cause is the guard, not the citation — both copies of
`test_every_test_this_script_names_actually_exists` are scoped to ONE script
file, so neither scanned the shared module the consolidation created. The
stale-base-triage copy now FOLLOWS its script's `scripts/lib` imports (derived,
not enumerated, so a second shared module is covered the day it appears) with a
positive control that fails if the import-following reaches nothing; the
main-status-watch copy names the owner rather than implying coverage it lacks.
The cited test is now written, and it kills the mutant.

F2 — `_git_subcommands` silently DROPPED what it could not constant-fold, so
its stated rule ("an unknown git subcommand is a write by default") was wider
than what it enforced: a computed head was ABSENT, not DENIED. It now emits
`<computed>` / `<not-a-list>` / `<empty-argv>` / `<no-argv>` sentinels — the
shape `test_main_status_watch.py`'s `_spawn_argv0_literals` already used 200
lines away in this same PR — reads argv from the SECOND POSITIONAL rather than
"whichever argument is a list", and the allowlist is a predicate a test drives
over synthetic source. The bare-`subprocess.run(["git", "push"])` hole the
finding also names is closed separately: every git spawn must be lexically
inside the one `_git` helper, and the spawn argv0 set is pinned
GROWS-OR-SHRINKS.

F3 — the `gh` write path was pinned by the literal word POST, which
`gh api -X PATCH`, `-X PUT`, `--method DELETE`, `gh pr comment` and
`gh api -f` (a POST with no method flag at all) each walk straight past. It now
asserts the STATE: an AST ledger of the `gh` verbs the source can reach
(`("api",)` and the one `("api", "-X", "POST")`), plus a receipt classifier that
decides read-vs-write from the method and the field flags. The harness stub also
had to start recording ONE LINE PER INVOCATION — a comment body contains
newlines, so `calls()` was returning prose fragments as if they were argvs,
which was invisible while the only question asked of a line was whether it
contained POST.

F4 — `TOTALS_RE` was unanchored and `re.search` takes the FIRST match, so a
description carrying an earlier per-target row derives a bound of 2 where the
run's own `failed=` is 27. That is the UNDER-count direction, the one error the
module explicitly forbids, and `run-tests.sh` really does print such rows
(`FAIL <dir> (collected=… passed=… skipped=… failed=… errors=…)`) before the
TOTAL banner. Both parsers are now anchored on `TOTAL` — `FAILED_COUNT_RE` had
the identical defect and is reachable from the same string, which the audit did
not file; fixing one and not the other would have left the same certification
hazard on the direct route. Both anchors err toward REFUSING. No reachable
exploit is claimed: what assembles the description is the posting pipeline in
homelab-talos and cannot be pinned from here.

F5 — the "140 BYTES" reconciliation (02bc6f8) had not finished;
`main-status-watch.py` said "140 characters" nineteen lines above its own
"THE 140-BYTE CAP". Swept all five sites including the test NAME, and replaced
the prose with a control: the two real truncated rows are 138 chars / 140 bytes,
asserted. The honest unknown is kept intact — whether the cut is GitHub's or the
pipeline's is still undetermined, because every sampled row carries exactly one
multi-byte character.

F6 — `sys.path.insert(0, …/scripts/lib)` prepended the whole shared-module
directory ahead of the standard library for every later import, including the
lazy `import traceback` on the unattended-crash path. `append` removes the class.

M2 (deleting `if not ctx: continue`) is NOT filed: an `""` key cannot match
either caller's context, so it is an equivalent mutant.

Evidence.
  Suites: test_stale_base_triage 71 -> 78, test_main_status_watch 101 -> 102.
  395 passed across the four affected files (the only files in the repo that
  reference any changed path; nothing else imports `ci_status`).
  ⚠ scoped-tests.sh is NOT evidence here and its PASS is withdrawn: it printed
  RESULT: PASS against the PRE-REBASE base, and on current main it correctly
  REFUSES (exit 4) because the diff touches `scripts/lib/**` — scoping a shared
  module selects the files that NAME it and drops every target reaching it by
  import. `scripts/gate.sh --tier both` is what covers that gap.
  RED at pr1524-head: the F4 test fails with derived=2 against the real 27.
  Mutation: 19 mutants, 18 KILLED, 1 SURVIVED — and the survivor is M4
  (first-match -> last-match), which is EQUIVALENT once anchored because
  `TOTAL collected=` occurs at most once on any banner this repo can post. That
  is asserted in the test rather than left unexplained, and reachability was
  confirmed separately: breaking the same statement kills 39 tests. Positive
  control KILLED in the batch; swept under PYTHONDONTWRITEBYTECODE=1 with
  __pycache__ removed between mutants.

COMMENT_MODE_DEFAULT is unchanged ("off"), both pins intact, nothing posted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQx16RMr8YQYBfsXaBEsSm
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0

* test(stale-base-triage): the write screen read a comment BODY as flags, and three guards were never exercised

ROUND 2's five findings. No payload change — every one of these is a weakness in
a guard this PR itself wrote.

F-R2-1 `gh_write_calls` parsed argument VALUES as flags, last-wins. Measured on
the previous version, same call shape as the real `post_comment`:

    -X POST /…/comments -f body=hello                       -> WRITE (control)
    -X POST /…/comments -f body=switch to --method GET now  -> read  (WRONG)
    -X POST /…/comments -f body=use -X GET here             -> read  (WRONG)
    /repos/o/r/x --input=payload.json                       -> read  (WRONG)
    /repos/o/r/x -fbody=hello                               -> read  (WRONG)

All five now classify as writes. The method is FIRST-wins and is scanned only up
to the first field flag — everything after one is body text — while field flags
are scanned to the end, in both the separated and the ATTACHED spelling `gh`
accepts. It does NOT stop at the first `/`-prefixed positional the way
`_gh_argv_shapes` does: `gh api /repos/o/r/x -X POST` is legal and would read as
clean, which is the one direction that matters. Reachable, so it is pinned end to
end as well: `comment_body` interpolates a `main` commit SUBJECT, and the new
test drives a hostile one through the real script.

F-R2-2 the spawn pin folded every computed argv0 into one set member, so
`binary = "git"; subprocess.run([binary, "push"])` appended to the real source
passed all four guards unchanged. Computed spawn heads are now asserted by NAME
(every one is the `gh` seam) and by COUNT (cross-checked against
`_gh_argv_shapes`), and the violating source is driven over synthetically.

F-R2-3 `_spawn_argv0`'s `<not-a-list>` arm — the net for `os.system("git push")`
— was never exercised; the mutant replacing it with `pass` SURVIVED. Driven over
synthetic source now, like `_git_subcommands`' sentinels.

F-R2-4 both scripts' `sys.path.append` carried a 🔴 comment and no guard;
reverting either to `sys.path.insert(0, …)` SURVIVED. Pinned in each file's own
test, structurally.

F-R2-5 five sites named GitHub as the cutter of the 140-byte description cap
while the new control says the measurement cannot distinguish GitHub's cut from
the pipeline's. The measured half (BYTES, not characters) is unchanged
everywhere; only the attribution is dropped, and each site now points at
`test_CONTROL_the_real_truncated_rows_land_on_the_BYTE_cap_not_the_CHAR_cap`.
`GITHUB_DESCRIPTION_BYTE_CAP`/`cut_like_github` renamed to `DESCRIPTION_BYTE_CAP`
/`cut_at_the_byte_cap` for the same reason. The unknown is NOT resolved in either
direction.

MUTATION TABLE (isolated `cp -a` copy, `PYTHONDONTWRITEBYTECODE=1`,
`__pycache__` cleared per mutant, verdicts counted from pytest's own result
lines; baseline in the copy 187 passed / 0 failed):

  mutant                                    verdict  killed by
  CONTEXT literal -> devrc-NOPE (control)   KILLED   test_the_context_is_PINNED_…
  method last-wins again                    KILLED   …reads_an_ARGUMENT_VALUE_as_DATA…
  body-stop removed                         KILLED   …reads_an_ARGUMENT_VALUE_as_DATA…
  `--input=` leaves the prefix tuple        KILLED   test_EVERY_field_flag_…
  attached short forms leave the tuple      KILLED   test_EVERY_field_flag_…
  M5 `<not-a-list>` arm -> pass             KILLED   …ARGV_IS_NOT_A_LIST_is_a_SENTINEL…
  M6 sbt sys.path.insert(0, …)              KILLED   …APPENDED_to_sys_path… (sbt)
  M8 msw sys.path.insert(0, …)              KILLED   …APPENDED_to_sys_path… (msw)
  second computed spawn in the real file    KILLED   …SECOND_computed_spawn… + the pin
  non-Name head folded into "gh"            KILLED   …SECOND_computed_spawn…

Each died on its own guard's assertion, naming the row or expression mutated —
not on a neighbouring guard's error. `method last-wins` SURVIVED on the first
sweep: with the body-stop in place the two rules only differ on a value carried
by a NON-body flag, so a `-H` row was added and it dies there.

Suites: `test_stale_base_triage.py` 84 + `test_main_status_watch.py` 103 = 187
passed (180 at 3aec90b, +7 new tests). `test_no_real_launchers.py` +
`test_audit_dispatch.py`, the other two files naming these scripts, 215 passed.
`COMMENT_MODE_DEFAULT = "off"` and both pins untouched; nothing posted anywhere.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session-Id: a7f5b63b-6de5-4fdc-9814-c8a8fc7b2ba0

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Zachary Lowden <dev@vetr.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants