feat(conformance, ci): the example stacks are applied every night, and the gate that does it names its runtime and its budget (#504)

[stephrobert]

Closes #504.

#520 is not closed here — it is disproved, measured, and recorded. See the
last section.

conformance:functional is the only gate that boots real machines and applies the
example stacks, and nothing in CI ran it. Yesterday it earned that sentence:
it is what surfaced the isolation-detach race, finding that cost three pull
requests, and the defect was older than the entire range bisected.

Where the gate runs, and why not where it seemed obvious

A stacks job of its own, parallel to the two mode legs on
runtime-proof.yml, incus-ovn only. Not a step on the existing incus-ovn leg,
and both reasons are measurements:

• that leg was red on four of the last seven scheduled nights, and a step
  queued behind a red is a gate that never runs — which is The only gate that boots real machines never plays the example stacks, where all four dataplane defects were found by hand #504's own complaint;
• it would inherit that leg's host. Reproduced locally: the three ssh suites
  each exit 0 and each leave something standing (scw-* and exo-* rule
  sets no client call can delete, fnt-default, feint-uplink), and feint stop
  sweeps none of them;
• three passes is half an hour, which would push a 45-minute job onto its own
  timeout, and a job that times out is a verdict nobody reads.

No job renamed, so .github/rulesets/main.json is untouched — this workflow
is schedule/dispatch only and none of its jobs is a required check. streak and
report now needs: [runtime, stacks], so a red stack gate cannot be
summarised or closed unseen, and the header says out loud that #125's promotion
criterion is widened on purpose.

The setup steps are duplicated from the incus-ovn leg deliberately:
extracting them while that leg is red would make any new red ambiguous.
TestBothRuntimeJobsPinTheSameHost holds the two copies to the same pins.
actionlint, zizmor (--persona regular --min-severity low) and poutine clean. No
secret, no cross-repository token.

The budget: three passes, and the arithmetic that chose it

The class this gate hunts struck 9 times in 13 runs. So one pass calls it
absent 31 % of the time; three passes bring that to 3 %, against 9.6 % for
two. At 295 s a pass, measured, three are about fifteen minutes and fit a
75-minute job; five would double the cost for a factor of ten and sit on the
timeout.

Sequential on one host, so pass k+1's doorstep is pass k's closing verdict.

Proof the gate bites

A gate that has never refused anything is the thing this repository spends its
time removing. So: #475 planted (ApplyFirewall returns nil), rebuilt, one pass —
red in 69 s, naming stack, machine, address and port:

scaleway: a rule of platform-app-worker-a's group opens 8080 and
platform-web-0 does not reach 10.30.2.5:8080

Restored, rebuilt, green.

Two defects found inside the gate itself, both "a comment is not a control"

The closing doorstep was a comment. functional.sh ended with
guard_leftovers_for "$RUNTIME" "the end of the run" under a paragraph stating
the doorstep question was asked again on the way out. It was not:
guard_leftovers_for arms --doorstep on the literal word doorstep, so it
never asked what machines and networks were left standing. Fixed — and the test
reads the scope the file passes and then executes the guard with it.

The runtime resolution was #574's line, one directory over.
RUNTIME="${FEINT_FUNCTIONAL_RUNTIME:-incus-ovn}" silently overrode what the
caller asked for, exactly as evidence:update did before yesterday. Extracted to
tools/runtime-mode.sh and shared with evidence:update — whose unchanged
tests are what prove the move changed nothing.

Falsification

the-stack-gate-runs-somewhere.json — 8 of 8 bite.
evidence-leg-mode.json retargeted onto the moved file (falsify --lint caught
its four dead fragments) — 7 of 7 bite.

One mutation was dropped rather than kept: neutralising || exit 1 on the
resolver call does not bite, because the resolver still prints its refusal. It
was removed instead of being wrapped in a comment that would have lied about it.

#520 — out of the pilotable, and its stated cause disproved

Terraform is refused for that pack since #525, so the pack's own API was driven
instead: 100 back-to-back by-id reads of one balancer, 100 fresh
create-then-read pairs, one elastic IP, 50 instance pools — zero empty
names, addresses or sizes.

There is no window: createLoadBalancer writes name and ip before the
operation that already says state: success. And the stack's third by-id door
read back fine in the same apply, so whatever produced the nulls discriminates
between the three doors and this pack does not.

Recorded in docs/limits.md with its acknowledgement re-dated. What would lift it
is upstream exoscale/terraform-provider-exoscale#573; no change to this pack
would. The issue is left open for the maintainer to act on rather than closed
from here.

The nightly red of runtime-proof — the brief's attribution was wrong

Seven consecutive red scheduled nights, last green 2026-08-21, and four
distinct causes, of which the witness doorstep is one:

night	incus	incus-ovn
08-22 … 08-25	Scaleway ssh suite	Scaleway ssh suite
08-26	Scaleway ssh suite	Outscale network suite
08-27	green	dataplane witness gate (doorstep)
08-28	feint images (alpine CDN)	same

The witness gate landed 2026-08-26 18:15, so 08-27 is the only night it has
ever been reached. Not fixed here: the honest repair is #521's shape — ask the
doorstep after the stop, so the leak reddens the run that leaked — and that
needs the leaking suites fixed too. A bare feint clean there would hide a real
leak. The stacks job is immune by construction.

Gates

prepush green · docs:check green · falsify:lint 899 mutations over 145
specs · two spec replays 15/15 · actionlint, zizmor, poutine clean ·
FEINT_VM=incus-ovn mise run conformance:functional green 3/3, plus six
extra single-pass runs green · host clean at the end.

What was not obtained

• One pass of the first three-pass run went red on the Scaleway firewall
  pair, the same verdict shape as the planted defect. This diff touches no
  emulator code, so it is on main. Not reproduced: 6/6 green single passes
  and a clean 3/3 green run (884 s). That pass ran while the station was
  compiling and running go test; the perturbation is a candidate that could not
  be separated. It is in the commit body rather than swept away.
• mise run limits:check is red on main before this branch — the
  "13 Scaleway list operations read their query and still drop 24 declared parameters (the per-parameter half of #271) #277 per-parameter half" section cites Scaleway renamed the VPC object-storage field on 2026-08-25 and the response body still answers the old name: the query parameter was fixed, the answer was not #570, closed today. Not this lot's, and
  no section was re-dated that had not been audited.
• An internal poller's machine filter was wrong (scw- against
  feint-scw-), so half its evidence never existed. Nothing went red under it —
  but it would have.
• golangci-lint reported 32 issues in files under a deleted sibling
  worktree: a stale cache, not code. Cleared.

Where testplan was wrong

It sent a change to tools/conformance/functional.sh to
conformance:leg -- fields — a leg with no machine runtime, the one population
this gate refuses to run in — and never named the gate itself. Fixed in the same
lot: rules.go now triages functional.sh, functionallib.sh and
runtime-mode.sh by name onto FEINT_VM=incus-ovn mise run conformance:functional,
each with what it still does not prove.