fix(testplan): every plan ends by naming what it cannot know, and two of its own sentences were measured false (#588) - #589
Merged
Conversation
… of its own sentences were measured false (#588) `mise run testplan` was wrong five times in one week. Four erred on the expensive side, which is survivable. The fifth did not: on #521 it named four runs and 27 specs, every one of them green, while the leg that reproduces the defect was in none of them. Taken as a ceiling, that plan said the work was done. The defect is not the table. It is what the tool says of itself: "a floor, not a ceiling" lived once, in prose, in CONTRIBUTING.md, and every brief of that week had to repeat it by hand. That is this repository's own rule about comments, applied to a tool's documentation. THREE MECHANISMS, IN THE ORDER THEY MATTER 1 The output says it, every time. plan.go now ends every plan — runs, nothing-to-run, and un-triaged alike — with what no table can answer: What this plan cannot know is which population the defect lives in. It routes on the paths a diff touched: on #521 it named four runs and 27 specs, all green, while the leg that reproduces the defect was in none of them. So it is a floor — add whatever you know that a path cannot say. Not "this plan is a floor": a clause of that shape is a manner of speaking and gets skimmed. What does not get skimmed is the name of the thing the reader is about to get wrong, so the first sentence names it, the second is the dated measurement, and "floor" arrives last as their conclusion. 2 An `Unproven` sentence that names an artefact is read against it (claims.go). A rule now carries `Cites`, each citation quoting the fragment it holds up, naming the file that decides it, and the tokens that must be present or absent. A sentence naming a file or a mise task and citing nothing is itself a failure. 3 A rule may not prescribe a run that cannot drive what it governs. leg.sh declares both halves — which legs it refuses when FEINT_VM is off, and which suites each leg runs — so the test reads leg.sh rather than a list kept beside the table. WHAT THE SECOND MECHANISM FOUND, MEASURED RATHER THAN ARGUED internal/core/resource/ said "a change to the stored shape is proved across a restart by `mise run conformance:environment`". That suite is tools/conformance/environment/up.sh: zero occurrences of `snapshot`, zero of `--state`, and its fixture says of itself that it declares no infrastructure. It stores nothing, saves nothing, reloads nothing. internal/core/store/ prescribed that same suite for the JSON snapshot, and admitted only the cross-version gap. Nothing in tools/conformance drives `--state` at all. The run is gone from its Runs and the sentence says so. internal/core/machine/ said "a bridge is a different verdict for isolation alone" (#574's note). Three declared capabilities turn on the mode — Isolation, Balancing and PrivateFromHost, all `d.OVN` in capabilities.go — and the firewall is built differently under a bridge with no capability saying so. No mechanism found this one: prose naming no artefact is not greppable. It was corrected by reading the driver. WHAT THE THIRD MECHANISM FOUND The four dataplane suites — scaleway/network.sh, outscale/network.sh, exoscale/network.sh, outscale/balancer.sh — were governed by their provider directory and earned a client leg that does not run them at all. Each has its own rule now, naming the runtime leg. The three ssh chains and sshlogin.sh were in the same state and were found by hand while measuring the sshkey rule's own sentence; they earn `conformance:ssh`. gates mise run prepush (green, 38 s); mise run testplan on this diff, which prescribed docs:check, falsify:lint and falsify -- tools/falsify/specs/a-plan-that-cannot-rot.json; all three run. 21 of 21 mutations of that spec red, including the seven added here — among them "the stack gate is routed to conformance:leg -- fields again", which is #588's second error verbatim. WHAT THIS STILL DOES NOT SEE, and it is the honest limit A rule that names a real directory, prescribes a leg that runs, and is simply wrong about which population the defect lives in passes all five properties. #521 is exactly that. Nothing static sees it, which is why mechanism 1 is the one that had to ship. Four scripts under tools/conformance are still routed to a leg that does not run them (crash.sh, parity.sh, stacks.sh, witness.sh); they are named in rules.go rather than left to be discovered. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #588.
mise run testplanwas wrong five times in one week. Four erred expensive, whichis survivable. The fifth did not: on #521, read as a ceiling, every run it
named was green while the nightly leg that reproduces the defect was in none of
them.
The defect is not the table. A rule that names a real directory and
prescribes a leg that runs, and is simply wrong about which population holds the
defect, passes every static check. It is what the tool says about itself: its
documentation writes "a floor, not a ceiling" once, in
CONTRIBUTING.md, andthat sentence had to be repeated by hand in every brief this week — one agent
took it as a ceiling anyway until its brief overrode it.
A sentence that must be repeated in every brief is not a property of the tool.
What every plan now ends with
"This plan is a floor" is a clause, and a clause of that shape reads as a
manner of speaking. So the first sentence names the thing the reader is about
to get wrong rather than the tool's genre; the second is a dated measurement,
not a caution; and "floor" arrives last, as the conclusion of the two rather than
as a label.
It prints in all three branches of
String()— runs prescribed, nothing to run,and paths un-triaged. The third matters most: "Nothing beyond
mise run prepush"is the output that reads most like an all-clear.
The five errors, caught or missed
conformance:environment"up.sh,Absent: snapshot, --state— fails. Proven twice: a planted absence, and a mutation pointing a real claim atleg.sh(which does contain "snapshot").functional.shrouted toconformance:leg -- fieldstools/runtime-mode.sh) only exists since #574's extraction, so it catches the routing as the tree stands today.conformance:environmentearned by a test-only diff_test.goplus atestdata/file prints "Nothing beyondmise run prepush" — now with the floor line under it.capabilities.godeclares three capabilities that vary —Isolation,Balancing,PrivateFromHost— anddocs/limits.mdrecords that the bridge builds its firewall differently with no capability saying so. The corrected sentence now carries a claim on those three tokens, so the next drift of that fact reddens.rules.go, inclaims.go, and in the printed output itself.Two caught, one already covered, two missed and said so. Naming a miss beats
widening a mechanism until it fits.
The mechanisms forced corrections to the table itself
internal/core/resource/andinternal/core/store/both namedconformance:environmentfor state that suite never touches:up.shhas zerosnapshot, zero--state, and its fixture declares no infrastructure. The runis gone from the store's
Runs; both sentences now cite the artefact.*/network.sh,outscale/balancer.sh) earned aclient leg that does not run them — one rule each now names the runtime leg.
The three
ssh.shchains andsshlogin.shwere in the same state, found byhand while measuring the sshkey rule's sentence.
Unprovensentences now carry citations read againstleg.sh,mise.toml,capabilities.go,runtime-proof.yml,up.shandlimits.md.Guards and falsification
TestEveryPlanSaysWhatItCannotKnow,TestEveryUnprovenClaimHoldsAgainstTheArtefactItNames,TestTheClaimCheckerFindsAClaimThatHasGoneFalse(six planted defects plus theaccepting half), and
TestARuleMayNotPrescribeARunThatCannotDriveWhatItGoverns— both halves readleg.sh, with a control assertingscw-cli.shis not dragged into the 590 sleg.
a-plan-that-cannot-rot.json: 7 new mutations and 1 retargeted — its oldfindquoted the very sentence #588 measured false. 21 of 21 red, green after
restoration.
falsify:lintgreen over 923 mutations.No mutation had to be excused, so no comment had to be rewritten — the first
lot this week where that was true.
Gates
mise run prepushgreen in 38 s. The nine pre-existing guards verified by name.mise run testplanon its own diff prescribeddocs:check,falsify:lintand
falsify -- a-plan-that-cannot-rot.json; all three run and green.What this will never see, written into the code rather than left to be found
tools/conformance/. The general property — a prescribed run must invoke thefile it was prescribed for — is not enforced, and four scripts still fail
it (
crash.sh,parity.sh,stacks.sh,witness.sh). They are named inrules.gorather than left to be discovered; they err on the useless side, notthe dangerous one.
leg.shnames a script doesnot prove it sits in the arm the sentence means. Where it mattered the token
carries its context (
FEINT_FIELD_GATE=1 tools/conformance/score.sh).every file contains would pass. What stops that is that writing the claim means
opening the artefact — a discipline, not a control, and
claims.gosays so.did not prescribe one.