fix(tests): one bound, one place β the 120s subprocess cap reddened the gate on diffs that could not reach it - #1493
Conversation
β¦he gate on diffs that could not reach it `test_run_tests_targets.py` spawns nested `run-tests.sh` runs under a hard subprocess timeout. That bound was 120s, open-coded at SIX call sites, and it has been failing PRs whose diff cannot touch this file β devrc#1458 and #1462 both merged with `tekton/devrc-pytests` RED because of it. MEASURED, not inferred: - Mechanism reproduced first-hand by lowering the constant: `subprocess. TimeoutExpired`, returncode -9, traceback ending in `_check_timeout`. It is NOT an assertion failure, which is exactly why it sent readers to debug their own diff. - One nested run of `scripts/collector/i3/tests` β the SMALLEST target, 13 tests β took 47s on the dev host at load ~52, i.e. 39% of the old bound, and almost all of it is the runner's fixed preflight rather than the tests. CI is documented at 27-50 concurrent full-suite runs on one node. So the bound was breached by CONTENTION, not by anything a diff did. Three changes: 1. `_RUNNER_TIMEOUT_S = 600` β one named constant carrying the measurement and the reasoning. ~12x the measured figure, still bounding a genuine hang well inside the gate's budget. Deliberately not env-overridable. 2. `_spawn()` β the single place a nested run is bounded. All six sites route through it. It TRANSLATES `TimeoutExpired` into a failure that states the classification ("this is not an assertion failure and probably not your diff"), because a bare TimeoutExpired names a command and a number and tells the reader nothing about which is wrong. 3. `test_no_call_site_open_codes_its_own_subprocess_bound` β pins the RELATIONSHIP, parsed via AST rather than grepped. Guard mutation-tested; control green (32 passed), each hazard killed by the assertion that claims it, naming line and function: A 7th raw subprocess.run in _run() KILLED "without going through `_spawn`" B aliased `from subprocess import run as _r` KILLED same (a regex walks past this) C _spawn bound respelled literal 120 KILLED "timeout is Constant(value=120)" D _spawn bound deleted entirely KILLED "timeout is ABSENT" D matters most: a check that only inspected the `timeout=` kwargs it FOUND would pass hardest on the unbounded hang this bound exists to prevent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SCmdWjiRn4PtLGTJrgmiX1 Claude-Session-Id: 460124dc-bb3d-4afb-b9e5-0687bec64e44
Merged-tree gate β both tiers green
β The dev-host tier ( β What this green does NOT prove. The flake is contention-triggered, so a passing run is The thing that is proven is the regression direction: a seventh open-coded bound, or a |
β¦, and the cost is not "preflight" Round-0 audit findings, all fixed. 1. The AST predicate was NARROWER THAN ITS OWN DOCSTRING β the defect class it exists to catch. It resolved `from subprocess import run as r` but NOT `import subprocess as sp; sp.run(...)`, a sibling spelling of the same aliasing the docstring claims to close; and it matched only `run`, so `Popen(...).communicate()` with no bound was invisible while the assertion beside it called exactly that "worse than the red it replaces". Both were measured SURVIVING. Now an enumerated `_SPAWNING_CALLABLES` plus resolution of module aliases and from-imports. 2. The constant's own comment said the 47s is "almost all the runner's fixed preflight rather than the tests". FALSE, and it is the load-bearing half: a `--targets` run still executes the hook-test and shell-test families, which `--targets` does not narrow. Of ~54s under `bash -x`, the SELECTED target's pytest is 2.9s and ~46s is those families. Discriminator, same box, same load ~52, minutes apart: `--targets <dir>` = 47s vs `--files <one file>` = 4s, because `--files` sets SCOPED_MODE and run-tests.sh:4518 drops the families. The comment now says so, names the root cause, and says this bound is the SYMPTOM fix. 3. The worst-case budget was asserted, not measured. Recorded: four real nested runs, so 4x600s = 40 min against a measured timeouts.tasks of 1h10m β 57% of the task budget, and a run that hits it posts NOTHING (checks pending forever). Stated as the trade this value makes. 4. Occurrence count was under-stated. #1450 is a third clean occurrence β a skills-doc-only diff red on a real-nested-run test in this file. #1429 is genuinely ambiguous (its diff touches run-tests.sh) and #1466 was a broken gate; both correctly excluded. 5. Dead code the previous commit propagated into new code: the `cwd` parameter no caller has ever passed, and an unused `import shutil`. Mutation battery re-run on the widened predicate, control green (32 passed), reading ONLY the `E ` lines: E import subprocess as sp; sp.run(...) KILLED (SURVIVED before) F Popen(...).communicate(), no bound KILLED (SURVIVED before) A 7th raw subprocess.run KILLED B from subprocess import run as _r KILLED C bound respelled literal 120 KILLED "timeout is Constant(value=120)" D bound deleted entirely KILLED "timeout is ABSENT" Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SCmdWjiRn4PtLGTJrgmiX1 Claude-Session-Id: 460124dc-bb3d-4afb-b9e5-0687bec64e44
Round 0 β ran, and it changed the outcome
π΄ Correction to this PR's description β posting rather than silently editing the bodyThe body says the 47 s is "almost all β¦ the runner's fixed preflight rather than the tests." It is not preflight. A I re-measured the discriminator myself rather than taking the audit's word β same box,
~12Γ. So this PR is the symptom fix, and the constant now says so. The root cause is that a π΄ And the guard was narrower than its own docstringThe defect class the guard exists to catch. Measured SURVIVING against the first version:
Both now killed. Full battery re-run on the widened predicate, control green (32 passed),
π‘ The worst-case budget was asserted, not measured β now it isThe body claimed 600 s "still bounds a genuine hang well inside the gate's own budget". I had Occurrence count was under-stated#1450 is a third clean occurrence β diff is Also deletedThe Still open, deliberatelyThe root-cause fix (a third runner mode: skip the families for Also open and larger than this PR: 25 other files under |
β¦nnot select β the root cause, not the bound
Operator chose to fix the cause in this PR rather than file it.
THE CAUSE. `run-tests.sh` drops the hook-test and shell-test families on a
`--files` run, keyed on SCOPED_MODE β which only `--files` sets. A `--targets`
run therefore executed BOTH families in full, and no `--targets` value can name
either. Re-keyed on `SCOPE_STATE != FULL`, so any narrowed run drops them.
MEASURED, dev host, same box:
--targets scripts/collector/i3/tests 47 s (before, load ~52)
--targets scripts/collector/i3/tests 9 s (after, load ~46)
Under `bash -x`, of ~54 s the SELECTED target's pytest was 2.9 s and ~46 s was
those families. That ~12x is what breached the nested-subprocess bound in
test_run_tests_targets.py under CI contention.
π΄ The skip is ANNOUNCED on the PARTIAL surface too. Widening the skip without
widening the announcement would be a run silently executing two families fewer
than its own SUMMARY claims β the #276 shape. Both moved together.
β It does NOT bring the rest of SCOPED_MODE: a PARTIAL run still enforces every
per-target floor, GUARD 7 REQUIRED and GUARD 2 TOTAL, and still reports
SCOPE: PARTIAL. Asserted, because reusing SCOPED_MODE wholesale would have
passed the skip tests and silently suspended the floors.
Regression matrix for the new test: RED at origin/main's runner (AssertionError;
its output shows `PASS scripts/claude-hooks/... (script)` β the family running
inside a --targets run), GREEN at HEAD. The pre-existing
test_a_full_run_still_runs_the_hook_and_shell_families is the positive control
that a FULL run is unaffected.
ROUND-1 AUDIT FIXES:
1. The bound now lives in testlib/scoped_harness.py, which ALREADY had its own
`timeout=600` default used by five test files. A constant private to
test_run_tests_targets.py was a FOURTH copy while claiming "one rule, one
place". Both read RUNNER_TIMEOUT_S now. Still unconsolidated and said so:
test_run_tests_preconditions.py (300) and
test_devshell_satisfies_required_tools.py (120 x2).
2. Value re-derived after the runner fix: 600 -> 300. 600 was sized against a
47 s run that no longer exists.
3. The cost paragraph named the WRONG CAP and inverted the consequence. The
binding cap is the gate TASK's `timeout: 60m`, below the pipeline's
`timeouts.tasks: 70m`, so a slow test can never reach the latter. Per that
pipeline's own measured probe: task-level timeout -> Failed, finally RAN ->
a posted RED; timeouts.tasks -> finally NEVER RAN -> the unclearable pending
the old comment wrongly warned about.
4. The guard passed VACUOUSLY with no spawn call in the file β mutant M11
(delegate _spawn's body to a testlib helper) SURVIVED. Now asserts >= 1
bounded spawn inside _spawn, killed by its own message.
5. `_SPAWNING_CALLABLES` claimed "every subprocess entry point that can start a
child" and was not. Added getoutput/getstatusoutput (the sharp case: no
timeout parameter exists, so unbounded by construction) and replaced the
claim with an enumeration naming what is deliberately NOT covered.
6. "all three merged" was wrong: #1450 is still OPEN.
Mutation battery, control green, each killed by the assertion that claims it:
M13 getoutput KILLED M9 Popen().communicate() KILLED
M14 getstatusoutput KILLED M11 vacuous pass KILLED (was SURVIVED)
M2 import as sp KILLED
87 passed across test_run_tests_targets + the five scoped_harness consumers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SCmdWjiRn4PtLGTJrgmiX1
Claude-Session-Id: 460124dc-bb3d-4afb-b9e5-0687bec64e44
Round 1 β findings fixed, and the operator chose to fix the ROOT CAUSE in this PRRound 1 was dispatched blind (diff + checklist, no prior conclusions). It returned three π΄ The root cause is now fixed β
|
| invocation | before | after |
|---|---|---|
--targets scripts/collector/i3/tests |
47 s (load ~52) | 9 s (load ~46) |
π΄ The skip is announced on the PARTIAL surface too. Widening the skip without widening
the announcement would be a run silently executing two families fewer than its own SUMMARY
claims β the #276 shape this runner exists to refuse. Both moved together.
β It does not drag the rest of SCOPED_MODE along: a PARTIAL run still enforces every
per-target floor, GUARD 7's REQUIRED direction and GUARD 2's skip TOTAL, and still reports
SCOPE: PARTIAL. That is asserted, because reusing SCOPED_MODE wholesale would have passed
the skip assertions while silently suspending the floors.
Regression matrix: RED at origin/main's runner β and its failure output is the proof,
showing PASS scripts/claude-hooks/tests/test_claude_notify.py (script) inside a --targets
run β GREEN at HEAD. The pre-existing test_a_full_run_still_runs_the_hook_and_shell_families
is the positive control that FULL is unaffected.
Round-1 findings
π‘ 1 β my cost paragraph named the wrong cap AND inverted the consequence. Verified
independently before adopting: the gate task's timeout: "60m" is below the pipeline's
timeouts.tasks: "70m", so a slow test can never reach the latter. And per that pipeline's
own measured three-way probe (Tekton v1.12.0): timeouts.tasks β finally NEVER RAN β
posts nothing, checks pending forever; task-level timeout β Failed, finally RAN β a
posted red. So overrunning yields a legible red, not the unclearable pending I warned
about. Corrected, with the ~22.5m-queue caveat that makes the real post-queue budget ~37m.
π‘ 2 β the guard passed VACUOUSLY. Both lists accumulate over calls the walk found, so a
file with no spawn call satisfies them by finding nothing. Mutant M11 β delegate _spawn's
body to a testlib/ helper, the natural shape of "move the bound somewhere shared" β
SURVIVED. Now asserts β₯1 bounded spawn inside _spawn; M11 is killed by that assertion's
own message. This is the same "passes hardest on the worst outcome" shape as the
bound-ABSENT check, one level up.
π‘ 3 β _SPAWNING_CALLABLES' comment claimed "every subprocess entry point that can start a
child" and was not. Added getoutput/getstatusoutput β the sharp case, since they take
no timeout parameter at all and are unbounded by construction. Replaced the claim with
an enumeration that names what is deliberately not covered (os.system, from subprocess import *, assignment aliasing) rather than implying closure.
π’ 6 β "one rule, one place" was file-local, and I had added a fourth copy.
scripts/testlib/scoped_harness.py:48 already carried run(args, timeout: int = 600, β¦),
used by five test files. The constant now lives there as RUNNER_TIMEOUT_S and this file
imports it. Still unconsolidated, and stated in the code rather than left implicit:
test_run_tests_preconditions.py (300) and test_devshell_satisfies_required_tools.py
(120 Γ2).
π’ 5 β "all three merged" was wrong. #1450 is still OPEN (mergedAt: null);
#1458 and #1462 merged over the red. Corrected in the comment.
π’ 4 β the PR body. Its "fixed preflight" sentence is retracted; see my earlier comment
and the body, now updated.
The bound was re-derived, not kept
600 was sized against a 47 s run that no longer exists. With the cause fixed the same run
is 9 s, so the value is now 300 β ~33Γ headroom, and a worst case of 4Γ300 s = 20 min
against the 60m task cap instead of 40 min.
Mutation battery β control green, each killed by the assertion that claims it
| mutant | outcome |
|---|---|
M13 subprocess.getoutput (no timeout param exists) |
KILLED |
M14 subprocess.getstatusoutput |
KILLED |
M2 import subprocess as sp |
KILLED |
M9 Popen(...).communicate(), unbounded |
KILLED |
| M11 spawn delegated out of the file (vacuous pass) | KILLED β was SURVIVED |
87 passed across test_run_tests_targets.py and the five scoped_harness consumers.
β Round 1's verified-not-findings are worth keeping: all six moved spawn sites are
byte-equivalent pre/post on argv/cwd/env/capture_output/text; TimeoutExpired.stdout
is genuinely bytes under text=True, so the .decode() is correct; and "four real nested
runs" is exact β instrumented, 39 spawns of which 35 are --check-* early exits at
0.00β0.09 s.
1. Finding 1 (the cost paragraph named timeouts.tasks 70m when the binding cap is the gate TASK's own 60m, and inverted the consequence): claimed corrected after independently verifying both halves against the live pipeline definitions and that pipeline's own measured three-way timeout probe; claimed to now state that a task-level overrun posts a legible RED, and to carry the ~22.5m queue caveat leaving ~37m real execution budget.
2. Finding 2 (the AST guard passed vacuously when the file contains no spawn call; mutant M11 measured SURVIVING): claimed addressed by asserting at least one bounded spawn inside `_spawn`, and claimed re-verified by M11 now being KILLED by that assertion's own distinct message rather than by a neighbouring one.
3. Finding 3 (`_SPAWNING_CALLABLES`' comment claimed every subprocess entry point and did not): claimed addressed by adding getoutput/getstatusoutput and by replacing the universal claim with an enumeration that names os.system, star-import and assignment aliasing as deliberately NOT covered.
4. Finding 6 (one-rule-one-place was file-local; scoped_harness.py already held a fourth copy of the same 600 default used by five files): claimed addressed by moving the constant into testlib/scoped_harness.py as RUNNER_TIMEOUT_S and importing it here; claimed the two remaining unconsolidated sites are named in the code rather than left implicit.
5. Finding 5 ("all three merged" false for #1450): claimed corrected after re-checking all three via gh; #1450 OPEN, #1458 and #1462 merged.
6. SCOPE CHANGE, operator-directed: the root cause is fixed in run-tests.sh (SCOPED_MODE -> SCOPE_STATE != FULL), the skip announced on the PARTIAL surface, and the bound re-derived 600 -> 300 against the post-fix 9s measurement. Claimed red/green matrix for the new test: red at origin/main's runner, green at HEAD, with the pre-existing full-run test as the positive control.
β¦s on the evidence of a sixth Round-2 delta audit, one finding, and it is a defect the round-1 fix introduced. Moving the constant into testlib/scoped_harness.py was right; changing its VALUE in the same edit was not. `RUNNER_TIMEOUT_S` is the default for `run()`, which test_scoped_runs.py, test_scoped_mapper.py, test_scoped_scope_marker.py, test_scoped_gate_contract.py and test_scoped_ledgers.py use across ~47 call sites. Setting it to 300 halved the bound for all five, justified entirely by a 9s measurement taken in the sixth. Their runs are not 9s. MEASURED on the dev host: 28.9s at load ~57 for test_the_node_runner_reports_FULL_on_a_REAL_run; the audit observed 70.5s for a sibling under different load. At 300 that is ~4-10x headroom, against the >2.55x contention inflation that caused this PR β i.e. the narrowing would have re-created here the exact failure class the runner fix removes elsewhere: a red on tekton/devrc-pytests naming a test the diff cannot reach. So 600 stands, unchanged from what this module already had. What the PR removes is the six open-coded `timeout=120` copies, not the bound's value. The rejected 300 and the measurements that rejected it are recorded beside the constant so nobody re-derives it from the 9s figure. The worst-case-budget argument that motivated 300 is kept as information and corrected: 4x600s = 40 min requires all four nested runs to HANG β after the runner fix each measures 2-9s β and a genuine hang is a defect you want surfaced, not absorbed. Also from round 2's non-findings: - `bounded_in_spawn` counted spawn calls sited in `_spawn`, not ones checked to carry a bound, while its message said "NO bounded spawn". Renamed to `spawns_in_spawn` and the message narrowed to what the expression measures. - "a LEGIBLE RED" overstated it: the posted status is `error` carrying `COULD NOT RUN`, which CLAUDE.md tells readers is a broken gate rather than a bad change. The substantive claim (posts something vs pending forever) stands. - Cited line numbers corrected: test_devshell_satisfies_required_tools.py is 110,438 not 105,436. Re-verified all three cited sites by grep. Mutation battery re-run, control green, M11 now killed by the renamed assertion's own message. 105 passed across test_run_tests_targets, the five scoped_harness consumers and test_run_tests_preconditions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SCmdWjiRn4PtLGTJrgmiX1 Claude-Session-Id: 460124dc-bb3d-4afb-b9e5-0687bec64e44
Round 2 (delta) β one finding, and it was a defect the round-1 fix introducedRange π‘ The one finding: I re-sized a shared bound on one consumer's evidenceMoving the constant into
At 300 that is ~4β10Γ headroom, against the >2.55Γ contention inflation that caused this So 600 stands β unchanged from what the module already had. What this PR removes is the The worst-case-budget argument that motivated 300 is kept as information and corrected: This is the fix-round prose failure the ladder rules name explicitly: wider on one axis, Round 2's non-findings, acted on anyway
Mutation battery re-run, control green, M11 killed by the renamed assertion's own message. β Round 2 ran the dev-host tier only and says so; it is not a gate claim. Merged-tree sandbox |
β¦emselves Round-3 delta audit. Every finding is in prose THIS ladder's own fix commits added β the recorded failure mode, and none of it is a code defect. A. "What this PR removes is the six open-coded timeout=120 copies, NOT the bound's value" β false for the scope that matters. Those six sites go 120 -> 600, a 5x widening, and that widening is the PR's headline fix. What is unchanged is the SHARED CONSTANT, for the five files already reading it. As written it would tell a maintainer no timeout was raised anywhere. Both copies of the sentence now name the scope. B. The comment on `spawns_in_spawn` said the `unbounded` assertion "above β¦ has already run". It is BELOW β line 355 against the comment at 337 and the assertion at 348. The comment's whole argument was "this needn't check boundedness, that is already proven", which would have justified deleting the check it leaned on. Corrected, and the two are now stated as independent: one refuses an EMPTY set, the other a BOUNDLESS call. C. "an `error` carrying COULD NOT RUN" β wrong string. Traced the reporter's branch selection in devrc-ci-pipeline.yaml: the COULD NOT RUN arm is guarded by BUILD_STATUS = "Succeeded", which a timed-out task does not satisfy; a SIGKILLed step leaves no steprc and lands on `KILLED: <leg> β the gate pod died at or after step <phase> (preempted/evicted/OOM/timeout)`. The pipeline states it itself: "A task-level timeout: expiring while a step is EXECUTING also SIGKILLs it, so it reads as KILLED". The conclusion (error, not a code failure, posts something) survives; the citation did not. D. A four-line HANG BOUND paragraph was duplicated verbatim inside the docstring whose thesis is "one rule, one place". Deduplicated. E. The revert deleted the one sentence quantifying a cost of 600 while keeping every sentence quantifying a cost of 300 β sweeping the arguments against the rejected value harder than against the adopted one. 4x600s = 40 min against a ~37 min post-queue budget is now stated plainly as the strongest argument AGAINST 600, with why it is accepted anyway. Also: "~47 call sites" re-counted by AST β 45, of which 44 take the default. Verified rather than trusted: 600 is the module's pre-existing default at b315cdd; the six 120s sites; the five importing files; 110/438 and preconditions:68; the gate task's 60m vs timeouts.tasks 70m; the three-way probe table; no stale 300 claims to be live anywhere. Guard battery re-run, control green, all five mutants killed by their own messages. 105 passed across the seven affected files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SCmdWjiRn4PtLGTJrgmiX1 Claude-Session-Id: 460124dc-bb3d-4afb-b9e5-0687bec64e44
Round 3 (delta) β four findings, all of them sentences this ladder wrote about itselfRange π‘ A β "not the bound's value" was false for the scope that matters. Both the constant's π‘ B β I asserted an execution order that is backwards. The comment on π‘ C β wrong string, traced rather than assumed. I wrote "an π‘ D β a four-line paragraph duplicated verbatim inside the docstring whose thesis is "one π’ E β I swept the arguments against the value I rejected harder than against the one I Also re-counted by AST: "~47 call sites" is 45, of which 44 take the default. Verified rather than trusted600 as the base default ( GateMerged-tree sandbox, previous tip: |
β¦OPS HERE Round-4 delta audit. Two findings, both prose, both defects the round-3 fix introduced while fixing round 2's. 1. Round 3 replaced one false claim with its OPPOSITE. It had said the PR changes "WHERE the bound is written, not WHAT it is" (false for the six consolidated sites); the replacement called the 120 -> 600 rise "the PR's headline fix" / "the whole point of the change" β which oversells 600 as a measured value and is contradicted twenty lines below by "120 WAS NEVER THE REAL DEFECT EITHER", and by commit 409b1d3's own subject, "the root cause, not the bound". Both copies now say the rise is a CONSEQUENCE of the consolidation, accepted; the headline fix is the runner scoping; 600 remains an unpinned hang bound. The two dead drafts are recorded so a third is not derived. 2. Round 3's fix for finding E deleted the 70.5s observation β the single worst recorded, i.e. the one most hostile to 300 β while adding a smaller new one, so the stated headroom improved from ~4-10x to ~10-18x with no note that evidence had been removed. That is the selective-sweep mechanism finding E was about, performed inside the commit fixing E. The 70.5s is restored and LABELLED (second-hand, load unrecorded) rather than deleted, with the range stated honestly as ~4x to ~18x depending which observation you take. π΄ THE LADDER STOPS HERE, ON THE ATTRIBUTION GATE, NOT ON A CLEAN ROUND. Measured, not asserted: round 3's fix (beb227e) and this one both change 100% comment lines β every added/removed line matches `^[+-]\s*#`. Two consecutive rounds whose fixes changed ZERO payload lines means the ladder has left the PR: rounds 2, 3 and 4 each found only the previous round's own sentences, while the executable payload has been untouched since round 1. The code was settled four rounds ago: the runner fix is 14 non-comment lines in run-tests.sh and 4 in scoped_harness.py, green in both sandbox tiers on the merged tree, with the guard mutation-killable by two independent mutants each dying on its own message. Nothing is filed as follow-up β round 4's two findings are fixed here, and it reported no others. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SCmdWjiRn4PtLGTJrgmiX1 Claude-Session-Id: 460124dc-bb3d-4afb-b9e5-0687bec64e44
Round 4 β two findings, both fixed, and π΄ THE LADDER STOPS HERE ON THE ATTRIBUTION GATERound 4 confirmed all five of round 3's fixes landed β and found that two of them had introduced π‘ 1 β I replaced one false claim with its OPPOSITE. Round 3 corrected "this PR changes π‘ 2 β the fix for finding E used the very mechanism finding E was about. E was "you swept Why the ladder ends here, and it is not a clean roundMeasured, not asserted: round 3's fix ( That matches what the rounds found. Rounds 2, 3 and 4 each found only the previous round's own
The independent auditor reached the same conclusion unprompted: "stop the prose ladder rather π΄ Stating what I am NOT doing, so this is distinguishable from convergence: this is the What actually shipsThe code settled four rounds ago: 14 non-comment lines in Verified on the merged tree, sandbox tier, one derivation at a time: |
Final merged-tree gate β both tiers green on the closing tip
π΄ And the property the whole runner change had to preserve: 10 β The dev-host tier ( |
Closes the second gate-flake family, documented as rank 7 in
handoff-gate-flake-store-api.md(talos-infra#1477). That doc diagnosed it; this fixes it βat the cause, not only at the symptom.
The defect
run-tests.shskips the hook-test and shell-test families on a narrowed run β but the skipwas keyed on
SCOPED_MODE, which only--filessets. So a--targetsrun executed bothfamilies in full, and no
--targetsvalue can name either.scripts/tests/test_run_tests_targets.pyspawns nested--targetsruns under a subprocessbound. Those runs were ~12Γ more expensive than the work they selected, and under CI
contention they breached the bound β failing
tekton/devrc-pytestson PRs whose diff cannotreach either file. Three occurrences: #1450 (a
SKILL.md), #1458, #1462 (a handoffdoc). #1458 and #1462 merged over the red; #1450 is still open.
Measured
--targets scripts/collector/i3/tests(13 tests)--files <one file in it>Under
bash -x, of ~54 s: the selected target's pytest was 2.9 s; ~46 s was the twofamilies (
test_base_clone_staleness.sh19.3 s,test_diagnose_disk_accounting.sh13.2 s,test_bash_guard.py8.7 s).Mechanism reproduced first-hand, by lowering the bound rather than waiting for CI:
subprocess.TimeoutExpired, returncode-9, traceback ending in_check_timeoutβ andtherefore not an assertion failure, which is why it sent readers to debug their own diff.
The change
scripts/run-tests.shβ the family skip is re-keyed fromSCOPED_MODEtoSCOPE_STATE != FULL, so any narrowed run drops them.π΄ The skip is announced on the PARTIAL surface too. Widening the skip without widening the
announcement would be a run silently executing two families fewer than its own SUMMARY claims
β the #276 shape this runner exists to refuse.
β It does not drag the rest of
SCOPED_MODEalong. A PARTIAL run still enforces everyper-target floor, GUARD 7's REQUIRED direction and GUARD 2's skip TOTAL, and still reports
SCOPE: PARTIAL. That is asserted β reusingSCOPED_MODEwholesale would have passed theskip assertions while silently suspending the floors.
scripts/testlib/scoped_harness.pyβRUNNER_TIMEOUT_S = 300now lives here, the modulewhose
run()already carried its owntimeout=600default across five test files. A constantprivate to one test file would have been a fourth copy of the predicate.
scripts/tests/test_run_tests_targets.pyβ six open-codedtimeout=120sites collapseinto one
_spawn(), which translatesTimeoutExpiredinto a failure stating theclassification ("not an assertion failure, probably not your diff"). Plus an AST guard
against a seventh site.
Regression matrix
RED at
origin/main's runner β the failure output is itself the proof, showingPASS scripts/claude-hooks/tests/test_claude_notify.py (script)inside a--targetsrun βGREEN at HEAD. The pre-existing
test_a_full_run_still_runs_the_hook_and_shell_familiesis the positive control that a FULL run is unaffected.
Guard mutation-tested
Control green. Each mutant killed by the assertion that claims it, read from the
Elinesonly:
subprocess.runimport subprocess as sp/from subprocess import run as _rPopen(...).communicate(), unboundedgetoutput/getstatusoutput(notimeoutparam exists)The last two matter most: a check that only inspected the
timeout=kwargs it found, orthat accumulated over calls it found, passes hardest on the worst outcome.
Audit ladder
Round 0 (requirements/deletion) β challenged this PR's own premise and is why the root
cause is fixed here rather than filed. Round 1 (blind, nine axes) β three π‘, three π’, all
fixed; it caught that my cost paragraph named the wrong CI cap and inverted the consequence,
that the guard could pass vacuously, and that "one rule, one place" was file-local.
Known-incomplete, stated rather than implied
test_run_tests_preconditions.py(300) andtest_devshell_satisfies_required_tools.py(120 Γ2) still carry their own bounds. They drive fast precondition/PATH-stub aborts, so
none is near its bound β this is about the thesis, not a live defect.
os.system,from subprocess import *and assignment aliasing are deliberately notcovered; the comment enumerates rather than claiming closure.