fix(testplan): every plan ends by naming what it cannot know, and two of its own sentences were measured false (#588)

CONTRIBUTING.fr.md

@@ -97,10 +97,19 @@ de noms de répertoires, qui est la moitié qu'une table se tromperait en premie
 CI.** Tous les autres gates d'ici ajoutent de la preuve ; celui-ci en retire, et
 cela inverse le mode de défaillance : une règle fausse est silencieuse. La CI
 inchangée, le pire cas d'une règle fausse est une pull request rouge, ce pour
-quoi la CI existe. Ce qu'aucun test n'attrape, c'est une règle qui nomme un vrai
-répertoire et prescrit la mauvaise jambe : les deux pourritures mécaniques sont
-gardées dans `tools/testplan/plan_test.go`, la sémantique ne l'est pas, et c'est
-écrit plutôt que dissimulé.
+quoi la CI existe.
+
+**Chaque plan imprime sa propre limite, et ce n'est pas une politesse.** L'outil
+s'est trompé cinq fois en une semaine (#588). Quatre fois du côté cher, ce qui
+est survivable ; la cinquième, il a nommé quatre exécutions et 27 specs, toutes
+vertes, alors que la jambe qui reproduit le défaut n'était dans aucune. Pris
+pour un plafond, ce plan disait que le travail était fait. Un plan se termine
+donc en nommant ce qu'aucune table ne peut savoir : **dans quelle population
+vit le défaut**. Quatre pourritures mécaniques sont gardées dans
+`tools/testplan/plan_test.go` : un chemin non trié, une règle qui ne correspond
+à rien, une phrase `Unproven` que l'artefact qu'elle nomme contredit, et une
+règle qui prescrit une jambe incapable de piloter ce qu'elle trie. L'erreur de
+population n'est gardée par rien, et c'est écrit plutôt que dissimulé.
 
 La table qu'il lit, sous forme courte :
 

CONTRIBUTING.md

@@ -90,10 +90,19 @@ names, which is the half of it a table would get wrong first.
 **The plan replaces the full pass locally. It never replaces the CI matrix.**
 Every other gate here adds proof; this one subtracts it, and that inverts the
 failure mode — a wrong rule is silent. With CI untouched, the worst case of a
-wrong rule is a red pull request, which is exactly what CI is for. What no test
-can catch is a rule that names a real directory and prescribes the wrong leg:
-the two mechanical rots are guarded in `tools/testplan/plan_test.go`, the
-semantic one is not, and it is written down rather than pretended away.
+wrong rule is a red pull request, which is exactly what CI is for.
+
+**Every plan prints its own limit, and that is not a courtesy.** The tool was
+wrong five times in one week (#588). Four erred on the expensive side, which is
+survivable; the fifth named four runs and 27 specs, every one of them green,
+while the leg that reproduces the defect was in none of them — read as a
+ceiling, that plan said the work was done. So a plan now ends by naming what no
+table can answer: **which population the defect lives in**. Four mechanical rots
+are guarded in `tools/testplan/plan_test.go` — an un-triaged path, a rule
+matching nothing, an `Unproven` sentence contradicted by the artefact it names,
+and a rule prescribing a leg that cannot drive what it governs. The population
+error is not guarded by anything, and it is written down rather than pretended
+away.
 
 The table it reads, in short form:
 

tools/falsify/specs/a-plan-that-cannot-rot.json

@@ -66,8 +66,9 @@
     },
     {
       "label": "a rule that runs several commands stops having to say what it leaves unproven, and the plan becomes a green describing a smaller world than its reader believes (#564)",
+      "note": "The sentence this used to mutate was the one #588 measured false — `mise run conformance:environment` proving a stored shape across a restart, against a suite that saves no state — so it is gone and this points at another rule with two runs.",
       "file": "tools/testplan/rules.go",
-      "find": "\t\tUnproven: \"a change to the stored shape is proved across a restart by `mise run conformance:environment`\",",
+      "find": "\t\tUnproven: \"octl reshapes its answers unless a call pins `-o raw`; the leg proves the wire, not the CLI's rendering\",",
       "replace": "\t\tUnproven: prepushIsTheWholeGate,",
       "test": "TestEveryRuleWithRunsSaysWhatItDoesNotProve"
     },
@@ -98,6 +99,55 @@
       "find": "\t\t\tif changed[m.File] {",
       "replace": "\t\t\tif changed[m.File] || true {",
       "test": "TestAChangedFileEarnsTheSpecsThatMutateIt"
+    },
+    {
+      "label": "the plan stops saying what it cannot know, so it reads as a ceiling again — which is how #521 was called done while the nightly leg kept failing (#588)",
+      "file": "tools/testplan/plan.go",
+      "find": "\tb.WriteString(cannotKnow)\n\treturn b.String()",
+      "replace": "\tif false {\n\t\tb.WriteString(cannotKnow)\n\t}\n\treturn b.String()",
+      "test": "TestEveryPlanSaysWhatItCannotKnow"
+    },
+    {
+      "label": "an Unproven sentence may name an artefact and cite nothing, which is the state the #567 sentence lived in: it claimed a suite proved a stored shape across a restart, and nothing read the suite (#588)",
+      "file": "tools/testplan/claims.go",
+      "find": "\t\tif named := artefactsNamed(r.Unproven); len(named) > 0 && len(r.Cites) == 0 {",
+      "replace": "\t\tif named := artefactsNamed(r.Unproven); len(named) > 0 && len(r.Cites) == 0 && false {",
+      "test": "TestTheClaimCheckerFindsAClaimThatHasGoneFalse"
+    },
+    {
+      "label": "a claim stops being read against the artefact it names, so a sentence resting on a token that has been deleted keeps reading as proof (#588)",
+      "file": "tools/testplan/claims.go",
+      "find": "\t\tif !strings.Contains(text, collapse(token)) {",
+      "replace": "\t\tif !strings.Contains(text, collapse(token)) && false {",
+      "test": "TestTheClaimCheckerFindsAClaimThatHasGoneFalse"
+    },
+    {
+      "label": "a claim of absence stops being checked — the exact half that would have caught #567, where up.sh contains neither `snapshot` nor `--state` and a rule said it proved a restart (#588)",
+      "file": "tools/testplan/claims.go",
+      "find": "\t\tif strings.Contains(text, collapse(token)) {",
+      "replace": "\t\tif strings.Contains(text, collapse(token)) && false {",
+      "test": "TestTheClaimCheckerFindsAClaimThatHasGoneFalse"
+    },
+    {
+      "label": "a real claim of this table is pointed at the wrong artefact: leg.sh does contain the word `snapshot`, so the sentence about the environment suite becomes false and must be caught against the committed table rather than against a fixture (#588)",
+      "file": "tools/testplan/rules.go",
+      "find": "\t\t\tIn:     \"tools/conformance/environment/up.sh\",\n\t\t\tAbsent: []string{\"--state\", \"snapshot\"},",
+      "replace": "\t\t\tIn:     \"tools/conformance/leg.sh\",\n\t\t\tAbsent: []string{\"--state\", \"snapshot\"},",
+      "test": "TestEveryUnprovenClaimHoldsAgainstTheArtefactItNames"
+    },
+    {
+      "label": "the stack gate is routed to `conformance:leg -- fields` again, which is #588's second error verbatim: a real leg, which runs, and which has no machine runtime — the one population that gate refuses to work in (#566, #477)",
+      "file": "tools/testplan/rules.go",
+      "find": "\t\tRuns: []string{\"FEINT_VM=incus-ovn mise run conformance:functional\"},\n\t\tUnproven: \"three passes on one host",
+      "replace": "\t\tRuns: []string{\"conformance:leg -- fields\"},\n\t\tUnproven: \"three passes on one host",
+      "test": "TestARuleMayNotPrescribeARunThatCannotDriveWhatItGoverns"
+    },
+    {
+      "label": "the runtime leg is prescribed without a runtime, so every rule naming it sends the reader to a leg leg.sh refuses at second zero (#459, #588)",
+      "file": "tools/testplan/plan.go",
+      "find": "const runtimeLeg = \"FEINT_VM=incus-ovn mise run conformance:leg -- runtime\"",
+      "replace": "const runtimeLeg = \"mise run conformance:leg -- runtime\"",
+      "test": "TestARuleMayNotPrescribeARunThatCannotDriveWhatItGoverns"
     }
   ]
 }

tools/testplan/claims.go

@@ -0,0 +1,160 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+)
+
+// What an `Unproven` sentence rests on, so that a sentence which has gone false
+// reddens instead of being believed.
+//
+// THE DEFECT THIS IS. On #567 a rule said, in its own `Unproven`, that "a change
+// to the stored shape is proved across a restart by `mise run
+// conformance:environment`". That suite is tools/conformance/environment/up.sh,
+// and it contains **zero** occurrences of `snapshot` and of `--state`: it starts
+// an emulator, asserts its ready conditions, stops it, and its fixture declares
+// no infrastructure at all. The sentence was false on the day it was written and
+// it was greppable on that day — which is the whole argument for this file. Two
+// of the five errors #588 counts are contradicted by a rule's own prose, and
+// prose is the one part of this table nothing was reading.
+//
+// So a sentence that names an artefact of this repository must say what makes it
+// true, and that "what" is read off the artefact rather than believed.
+//
+// WHAT THIS DOES NOT DO, because a mechanism that oversells itself is the defect
+// it exists to catch:
+//
+//   - It cannot judge a sentence that names no artefact. "a bridge is a
+//     different verdict for isolation alone" (#574, wrong: three declared
+//     capabilities turn on the mode) names nothing greppable, and no arrangement
+//     of this file would have caught it. It was corrected by hand.
+//   - A claim can be made vacuous by choosing a token every file contains. What
+//     stops that is not this code: it is that writing the claim at all means
+//     opening the artefact, which is exactly the step whose absence produced the
+//     false sentence above.
+//   - Containment is whole-file. A claim that leg.sh names a script does not
+//     prove the script sits in the arm the sentence means. Where that mattered,
+//     the token was chosen to include its context (`FEINT_FIELD_GATE=1
+//     tools/conformance/score.sh`).
+//   - One claim discharges a sentence, even one naming two artefacts. The
+//     forcing rule is a prompt, not a proof.
+type claim struct {
+	// About is the fragment of the rule's `Unproven` this holds up. It must be
+	// a substring of it, so that editing the sentence out from under a claim is
+	// itself a failure rather than a silent orphan.
+	About string
+	// In is the repository-relative artefact whose text decides the claim.
+	In string
+	// Shows are tokens that must appear in In. Whitespace is collapsed on both
+	// sides before matching, so gofmt realigning a struct literal does not turn
+	// a true claim red.
+	Shows []string
+	// Absent are tokens that must not appear in In. This is the half that
+	// catches the #567 sentence, and the half that will redden the day somebody
+	// makes it true — at which point the sentence is what needs rewriting.
+	Absent []string
+}
+
+// The artefacts a sentence names, extracted syntactically.
+//
+// Backticks are not required. They were the first shape of this and they hand
+// out a dodge: drop the backticks and the sentence names nothing this can see.
+// A path is a path in prose too.
+var (
+	artefactPath = regexp.MustCompile(`[A-Za-z0-9_][A-Za-z0-9_./-]*\.(?:sh|go|py|json|ya?ml|toml|md)\b`)
+	artefactTask = regexp.MustCompile(`mise run ([a-z0-9][a-z0-9:_-]*)`)
+	artefactBare = regexp.MustCompile("`([a-z0-9][a-z0-9-]*:[a-z0-9][a-z0-9-]*)`")
+)
+
+// artefactsNamed lists the files and mise tasks a sentence names.
+func artefactsNamed(text string) []string {
+	var named []string
+	seen := map[string]bool{}
+	add := func(s string) {
+		if s != "" && !seen[s] {
+			seen[s] = true
+			named = append(named, s)
+		}
+	}
+	for _, m := range artefactPath.FindAllString(text, -1) {
+		add(m)
+	}
+	for _, re := range []*regexp.Regexp{artefactTask, artefactBare} {
+		for _, m := range re.FindAllStringSubmatch(text, -1) {
+			add(m[1])
+		}
+	}
+	return named
+}
+
+// collapse folds every run of whitespace to one space, so a token may be
+// written the way a reader would say it rather than the way gofmt aligned it.
+func collapse(s string) string { return strings.Join(strings.Fields(s), " ") }
+
+// checkClaims reads every claim against the artefact it names and returns what
+// no longer holds. An empty result is the only acceptable one; the caller is
+// TestEveryUnprovenClaimHoldsAgainstTheArtefactItNames, and
+// TestTheClaimCheckerFindsAClaimThatHasGoneFalse plants each kind of defect to
+// prove this can find them at all.
+func checkClaims(root string, rs []rule) []string {
+	var problems []string
+	for _, r := range rs {
+		if r.Unproven == prepushIsTheWholeGate {
+			if len(r.Cites) > 0 {
+				problems = append(problems, fmt.Sprintf(
+					"rule %q leaves Unproven empty and still cites %d artefact(s): a claim with no "+
+						"sentence holds nothing up", r.Path, len(r.Cites)))
+			}
+			continue
+		}
+		if named := artefactsNamed(r.Unproven); len(named) > 0 && len(r.Cites) == 0 {
+			problems = append(problems, fmt.Sprintf(
+				"rule %q says something about %s and cites nothing. An Unproven sentence that names "+
+					"an artefact must say what makes it true: the one that did not (#567) claimed a "+
+					"suite proved a stored shape across a restart, and that suite saves no state",
+				r.Path, strings.Join(named, ", ")))
+		}
+		for _, c := range r.Cites {
+			problems = append(problems, checkClaim(root, r, c)...)
+		}
+	}
+	return problems
+}
+
+func checkClaim(root string, r rule, c claim) []string {
+	var problems []string
+	if !strings.Contains(r.Unproven, c.About) {
+		return []string{fmt.Sprintf(
+			"rule %q cites %q as holding up %q, which is not a fragment of its Unproven any more: "+
+				"the sentence moved and the claim stayed", r.Path, c.In, c.About)}
+	}
+	if len(c.Shows)+len(c.Absent) == 0 {
+		return []string{fmt.Sprintf(
+			"rule %q cites %s and asks nothing of it; a citation that reads nothing is a comment",
+			r.Path, c.In)}
+	}
+	body, err := os.ReadFile(filepath.Join(root, c.In)) //nolint:gosec // a path from this repository's own table
+	if err != nil {
+		return []string{fmt.Sprintf(
+			"rule %q rests on %s, which cannot be read: %v", r.Path, c.In, err)}
+	}
+	text := collapse(string(body))
+	for _, token := range c.Shows {
+		if !strings.Contains(text, collapse(token)) {
+			problems = append(problems, fmt.Sprintf(
+				"rule %q says %q, and %s does not contain %q — the sentence rests on something that "+
+					"is no longer there", r.Path, c.About, c.In, token))
+		}
+	}
+	for _, token := range c.Absent {
+		if strings.Contains(text, collapse(token)) {
+			problems = append(problems, fmt.Sprintf(
+				"rule %q says %q, and %s now contains %q — the artefact grew what the sentence says "+
+					"it lacks, so the sentence is what needs rewriting", r.Path, c.About, c.In, token))
+		}
+	}
+	return problems
+}

tools/testplan/plan.go

@@ -251,9 +251,41 @@ func (p plan) String() string {
 			fmt.Fprintf(&b, "  · %s\n", u)
 		}
 	}
+	b.WriteString(cannotKnow)
 	return b.String()
 }
 
+// What every plan ends on, whatever it contains.
+//
+// This tool was wrong five times in one week (#588). Four erred on the
+// expensive side, which is survivable. The fifth was #521: it named `probe`,
+// `fields`, `runtime`, `environment` and 27 specs, every one of them green,
+// while the defect lived in a population none of them carried — the ssh suites
+// are in no leg, and `-- runtime` runs the network suites, which end on their
+// own `feint clean` and sweep the leak before any closing doorstep can see it.
+// Read as a ceiling, that plan said the work was done.
+//
+// Nothing mechanical sees that: the rule named a real directory and prescribed
+// a leg that runs. So the property has to live in the output. It said "a floor,
+// not a ceiling" once, in prose, in CONTRIBUTING.md, and every brief of that
+// week had to repeat it by hand — which is this repository's own rule about
+// comments, applied to a tool's documentation.
+//
+// The wording is deliberate and it is not "this plan is a floor". A clause of
+// that shape is a manner of speaking and gets skimmed; what does not get
+// skimmed is the name of the thing the reader is about to get wrong. So the
+// first sentence names what no table can answer — which population the defect
+// lives in — the second is the dated measurement rather than a caution, and
+// "floor" arrives last, as the conclusion of the two.
+//
+// TestEveryPlanSaysWhatItCannotKnow fails without this line, in all three of a
+// plan's shapes: runs, nothing-to-run, and un-triaged.
+const cannotKnow = "\n" +
+	"What this plan cannot know is which population the defect lives in. It routes\n" +
+	"on the paths a diff touched: on #521 it named four runs and 27 specs, all\n" +
+	"green, while the leg that reproduces the defect was in none of them. So it is\n" +
+	"a floor — add whatever you know that a path cannot say.\n"
+
 // resolve turns the requested base into one this clone actually has.
 //
 // `origin/main` is the right default and it is not always present: a fresh