Skip to content

fix(testplan): every plan ends by naming what it cannot know, and two of its own sentences were measured false (#588) - #589

Merged
stephrobert merged 1 commit into
mainfrom
fix/588-the-plan-says-what-it-cannot-know
Aug 28, 2026
Merged

fix(testplan): every plan ends by naming what it cannot know, and two of its own sentences were measured false (#588)#589
stephrobert merged 1 commit into
mainfrom
fix/588-the-plan-says-what-it-cannot-know

Conversation

@stephrobert

Copy link
Copy Markdown
Owner

Closes #588.

mise run testplan was wrong five times in one week. Four erred expensive, which
is survivable. The fifth did not: on #521, read as a ceiling, every run it
named was green while the nightly leg that reproduces the defect was in none of
them.

The defect is not the table. A rule that names a real directory and
prescribes a leg that runs, and is simply wrong about which population holds the
defect, passes every static check. It is what the tool says about itself: its
documentation writes "a floor, not a ceiling" once, in CONTRIBUTING.md, and
that sentence had to be repeated by hand in every brief this week — one agent
took it as a ceiling anyway until its brief overrode it.

A sentence that must be repeated in every brief is not a property of the tool.

What every plan now ends with

What this plan cannot know is which population the defect lives in. It routes
on the paths a diff touched: on #521 it named four runs and 27 specs, all
green, while the leg that reproduces the defect was in none of them. So it is
a floor — add whatever you know that a path cannot say.

"This plan is a floor" is a clause, and a clause of that shape reads as a
manner of speaking. So the first sentence names the thing the reader is about
to get wrong
rather than the tool's genre; the second is a dated measurement,
not a caution; and "floor" arrives last, as the conclusion of the two rather than
as a label.

It prints in all three branches of String() — runs prescribed, nothing to run,
and paths un-triaged. The third matters most: "Nothing beyond mise run prepush"
is the output that reads most like an all-clear.

The five errors, caught or missed

# error verdict
#567 "a stored shape is proved across a restart by conformance:environment" caught. The sentence names a task, so a claim is forced, and the only honest claim — up.sh, Absent: snapshot, --state — fails. Proven twice: a planted absence, and a mutation pointing a real claim at leg.sh (which does contain "snapshot").
#566/#477 functional.sh routed to conformance:leg -- fields caught, and demonstrated by a mutation that re-routes it. Honest caveat written at the site: the marker the predicate reads (tools/runtime-mode.sh) only exists since #574's extraction, so it catches the routing as the tree stands today.
#548/#570 conformance:environment earned by a test-only diff missed here, already caught by #569. Measured: a _test.go plus a testdata/ file prints "Nothing beyond mise run prepush" — now with the floor line under it.
#574 "a bridge is a different verdict for isolation alone" missed. The clause names no artefact, so there is nothing for the claim checker to read. Measured by hand instead: capabilities.go declares three capabilities that vary — Isolation, Balancing, PrivateFromHost — and docs/limits.md records that the bridge builds its firewall differently with no capability saying so. The corrected sentence now carries a claim on those three tokens, so the next drift of that fact reddens.
#521 right rules, wrong population missed by construction, and named as such — in rules.go, in claims.go, and in the printed output itself.

Two caught, one already covered, two missed and said so. Naming a miss beats
widening a mechanism until it fits.

The mechanisms forced corrections to the table itself

  • internal/core/resource/ and internal/core/store/ both named
    conformance:environment for state that suite never touches: up.sh has zero
    snapshot, zero --state, and its fixture declares no infrastructure. The run
    is gone from the store's Runs; both sentences now cite the artefact.
  • Four dataplane suites (*/network.sh, outscale/balancer.sh) earned a
    client leg that does not run them — one rule each now names the runtime leg.
    The three ssh.sh chains and sshlogin.sh were in the same state, found by
    hand while measuring the sshkey rule's sentence.
  • Fifteen Unproven sentences now carry citations read against leg.sh,
    mise.toml, capabilities.go, runtime-proof.yml, up.sh and limits.md.

Guards and falsification

TestEveryPlanSaysWhatItCannotKnow,
TestEveryUnprovenClaimHoldsAgainstTheArtefactItNames,
TestTheClaimCheckerFindsAClaimThatHasGoneFalse (six planted defects plus the
accepting half), and
TestARuleMayNotPrescribeARunThatCannotDriveWhatItGoverns — both halves read
leg.sh, with a control asserting scw-cli.sh is not dragged into the 590 s
leg.

a-plan-that-cannot-rot.json: 7 new mutations and 1 retargeted — its old find
quoted the very sentence #588 measured false. 21 of 21 red, green after
restoration. falsify:lint green over 923 mutations.

No mutation had to be excused, so no comment had to be rewritten — the first
lot this week where that was true.

Gates

mise run prepush green in 38 s. The nine pre-existing guards verified by name.
mise run testplan on its own diff prescribed docs:check, falsify:lint
and falsify -- a-plan-that-cannot-rot.json; all three run and green.

What this will never see, written into the code rather than left to be found

  • Mechanism 3's population is five files, reading two markers only, scoped to
    tools/conformance/. The general property — a prescribed run must invoke the
    file it was prescribed for
    — is not enforced, and four scripts still fail
    it (crash.sh, parity.sh, stacks.sh, witness.sh). They are named in
    rules.go rather than left to be discovered; they err on the useless side, not
    the dangerous one.
  • Claim containment is whole-file: a claim that leg.sh names a script does
    not prove it sits in the arm the sentence means. Where it mattered the token
    carries its context (FEINT_FIELD_GATE=1 tools/conformance/score.sh).
  • One claim discharges a sentence, even one naming two artefacts, and a token
    every file contains would pass. What stops that is that writing the claim means
    opening the artefact — a discipline, not a control, and claims.go says so.
  • No conformance leg was run: nothing here changes a served answer, and the plan
    did not prescribe one.

… of its own sentences were measured false (#588)

`mise run testplan` was wrong five times in one week. Four erred on the
expensive side, which is survivable. The fifth did not: on #521 it named four
runs and 27 specs, every one of them green, while the leg that reproduces the
defect was in none of them. Taken as a ceiling, that plan said the work was
done.

The defect is not the table. It is what the tool says of itself: "a floor, not
a ceiling" lived once, in prose, in CONTRIBUTING.md, and every brief of that
week had to repeat it by hand. That is this repository's own rule about
comments, applied to a tool's documentation.

THREE MECHANISMS, IN THE ORDER THEY MATTER

  1  The output says it, every time. plan.go now ends every plan — runs,
     nothing-to-run, and un-triaged alike — with what no table can answer:

         What this plan cannot know is which population the defect lives in. It
         routes on the paths a diff touched: on #521 it named four runs and 27
         specs, all green, while the leg that reproduces the defect was in none
         of them. So it is a floor — add whatever you know that a path cannot
         say.

     Not "this plan is a floor": a clause of that shape is a manner of speaking
     and gets skimmed. What does not get skimmed is the name of the thing the
     reader is about to get wrong, so the first sentence names it, the second is
     the dated measurement, and "floor" arrives last as their conclusion.

  2  An `Unproven` sentence that names an artefact is read against it
     (claims.go). A rule now carries `Cites`, each citation quoting the fragment
     it holds up, naming the file that decides it, and the tokens that must be
     present or absent. A sentence naming a file or a mise task and citing
     nothing is itself a failure.

  3  A rule may not prescribe a run that cannot drive what it governs. leg.sh
     declares both halves — which legs it refuses when FEINT_VM is off, and
     which suites each leg runs — so the test reads leg.sh rather than a list
     kept beside the table.

WHAT THE SECOND MECHANISM FOUND, MEASURED RATHER THAN ARGUED

  internal/core/resource/  said "a change to the stored shape is proved across a
                           restart by `mise run conformance:environment`". That
                           suite is tools/conformance/environment/up.sh: zero
                           occurrences of `snapshot`, zero of `--state`, and its
                           fixture says of itself that it declares no
                           infrastructure. It stores nothing, saves nothing,
                           reloads nothing.
  internal/core/store/     prescribed that same suite for the JSON snapshot, and
                           admitted only the cross-version gap. Nothing in
                           tools/conformance drives `--state` at all. The run is
                           gone from its Runs and the sentence says so.
  internal/core/machine/   said "a bridge is a different verdict for isolation
                           alone" (#574's note). Three declared capabilities
                           turn on the mode — Isolation, Balancing and
                           PrivateFromHost, all `d.OVN` in capabilities.go — and
                           the firewall is built differently under a bridge with
                           no capability saying so. No mechanism found this one:
                           prose naming no artefact is not greppable. It was
                           corrected by reading the driver.

WHAT THE THIRD MECHANISM FOUND

  The four dataplane suites — scaleway/network.sh, outscale/network.sh,
  exoscale/network.sh, outscale/balancer.sh — were governed by their provider
  directory and earned a client leg that does not run them at all. Each has its
  own rule now, naming the runtime leg. The three ssh chains and sshlogin.sh
  were in the same state and were found by hand while measuring the sshkey
  rule's own sentence; they earn `conformance:ssh`.

  gates   mise run prepush (green, 38 s); mise run testplan on this diff, which
          prescribed docs:check, falsify:lint and
          falsify -- tools/falsify/specs/a-plan-that-cannot-rot.json; all three
          run. 21 of 21 mutations of that spec red, including the seven added
          here — among them "the stack gate is routed to conformance:leg --
          fields again", which is #588's second error verbatim.

WHAT THIS STILL DOES NOT SEE, and it is the honest limit

A rule that names a real directory, prescribes a leg that runs, and is simply
wrong about which population the defect lives in passes all five properties.
#521 is exactly that. Nothing static sees it, which is why mechanism 1 is the
one that had to ship. Four scripts under tools/conformance are still routed to
a leg that does not run them (crash.sh, parity.sh, stacks.sh, witness.sh);
they are named in rules.go rather than left to be discovered.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@stephrobert
stephrobert merged commit 8467c11 into main Aug 28, 2026
30 checks passed
@stephrobert
stephrobert deleted the fix/588-the-plan-says-what-it-cannot-know branch August 28, 2026 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant