Skip to content

fix(machine, ci): the exit gives back what no client can delete, and the leg that leaks is the one that reddens (#521) - #586

Merged
stephrobert merged 1 commit into
mainfrom
fix/runtime-proof-closing-doorstep
Aug 28, 2026
Merged

fix(machine, ci): the exit gives back what no client can delete, and the leg that leaks is the one that reddens (#521)#586
stephrobert merged 1 commit into
mainfrom
fix/runtime-proof-closing-doorstep

Conversation

@stephrobert

Copy link
Copy Markdown
Owner

Closes #521.

The incus-ovn leg of runtime-proof.yml has been red on its doorstep, and
since #585 it is the only cause left on that workflow. Every suite passes —
the three network suites, the three ssh suites, the report, the stop — and then:

a previous run left 0 machine(s) and 2 network(s) on this host.
  network feint-uplink, network fnt-default,
  rule-set exo-11e594f4819, rule-set scw-aa3f1f56ec5

There is no previous run. It is a fresh runner, and the suites of the same job
left those objects
. The doorstep of the next step found them and blamed
somebody else.

Who creates each object, and why nothing removed it

Measured by replaying the leg on the station, reading the host at every frontier.
The reproduction is byte-for-byte the CI failure — same four objects, same
counts, only the Scaleway digest differs because that one is minted per run.

object who makes it why nothing removed it
fnt-default the Outscale ssh suite — its Vm lives outside a Net, so it has no attachment of its own and boots on DefaultMachineNetwork no emulated resource owns it, so no client delete can reach it
feint-uplink the driver, when fnt-default is created under OVN already released since #521 — it stayed because fnt-default still drew from it (used_by=1, read on the host)
scw-<digest> the Scaleway ssh suite — the project's default security group, permissive, attached to nothing the API refuses to delete a project default group, and the id is minted per run: one host ACL accumulates per session
exo-<digest> the Exoscale ssh suite — the account's default group, DefaultIngress: drop, attached while the instance lives same refusal; fixed id, so it survives rather than piles up

No osc- rule set appears, and that is not luck: an Outscale Vm outside a Net
wears no security group.

The "shared session resource" hypothesis was measured and rejected

The brief offered it: perhaps feint-uplink and fnt-default are session
resources the gate is wrong to count.

They are shared within one emulator — but the doorstep is asked when that
emulator is gone
, so the session is over and the gate is right. Exempting them
would blind it to the leftovers of a killed run, which is exactly what
crash.sh asserts must survive a SIGKILL and be named.

So the fix belongs on the emulator side, not the gate's, and that is where it
went.

No feint clean was added anywhere

That shortcut would have made the night pass and removed the only thing that
detects a real leak. The release is targeted per object, and it asks both
questions before any destructive command — is it ours (the label, and the
description EnsureFirewall writes) and is anything still drawing from it
(used_by, read first rather than the delete being attempted blind).

A feint stop without --cleanup therefore still keeps the firewall on the
machines it deliberately leaves running.

machine.UplinkReleaser becomes machine.PlumbingReleaser; ReleasePlumbing
gives back rule sets → default network → uplink, in the only order the runtime
accepts
— reversed, every step answers "in use", which is the measured state.

The gate, at the right place, saying the right thing

feint clean --check --closing and guard.sh leftovers-after: the same refusal,
with the culprit named as this run. mise.toml and leg.sh move to it too —
they carried #521's form but the doorstep's sentence. runtime-proof.yml gains
one step, What this leg left on the host, after its own stop, and deliberately
not if: always(): a leg that already died left wreckage, not a leak. No job
renamed.

And the doorstep now names the rule sets it does not refuse on, with the
reason. Until now they were printed only when some other object had already
made it refuse — so the artefact that accumulates once per session was invisible
exactly where it was all that was left.

An instrument that lied, found on the way

The ledger's attribution column recorded networks as name-prefix:fnt-, while
Survey selects them by label — and the first network the failing leg named,
feint-uplink, carries no such prefix. Corrected, with a test; the old test
asserted the lie and was rewritten.

Reproduction

BEFORE  ssh suites (3× exit 0) → feint stop → doorstep
        exit 1: feint-uplink (used_by=1), fnt-default (used_by=0),
                scw-45e6b24f03f, exo-11e594f4819

AFTER   same sequence → the exit logs
        "released the plumbing this run held: scw-…, exo-…, fnt-default, feint-uplink"
        host: no network of ours, no rule set of ours, the operator's acltest untouched
        closing doorstep exit 0, gate doorstep exit 0

Falsification

22 mutations of a-run-ends-where-it-could-start.json — 15 new, one per
guard added — plus 3 retargeted in run-leaves-nothing.json and
unkillable-dhcp-orphan.json. Every one red.

Replayed by hand against a warm build cache: mise run falsify copies the
repository and recompiles the whole module per mutation, which is minutes for
internal/core/machine. All are declared in the specs so falsify:all replays
them nightly; falsify:lint is green.

Gates

prepush ✅ · mise run conformance ✅ (267 s) · conformance:leg -- probe ✅ ·
-- fields ✅ · conformance:environment ✅ · the full ssh-then-doorstep
sequence on the station ✅.

What was not obtained

Where testplan was wrong, on the load-bearing point

It named probe, fields, runtime, environment and 27 falsify specs — and
not the population the defect lives in. The ssh suites are in no leg, and
-- runtime runs the network suites, which end on their own feint clean and
therefore sweep the leak away before any closing doorstep sees it.

Taken as a ceiling, everything would have been green and the leg would still
fail. Floor, not ceiling — the fifth time this week.

…he leg that leaks is the one that reddens (#521)

The incus-ovn leg of runtime-proof.yml failed at the doorstep of its own
witness gate, on a GitHub runner nothing had ever touched:

    a previous run left 0 machine(s) and 2 network(s) on this host.
      network feint-uplink: not-attempted-yet, reported-only
      network fnt-default: not-attempted-yet, reported-only
      rule-set exo-11e594f4819: not-attempted-yet, reported-only
      rule-set scw-aa3f1f56ec5: not-attempted-yet, reported-only

There was no previous run. The leg's own three ssh suites had made all four,
each exiting 0, and the station reproduced it exactly on 2026-08-28: same four
objects, same counts, only the Scaleway digest differing because that one is
minted per run.

WHO MAKES EACH, AND WHY NOTHING REMOVED IT

  fnt-default    DefaultMachineNetwork, created the first time a machine boots
                 with no attachment of its own. Here that is the Outscale ssh
                 suite's Vm, which lives outside a Net. No emulated resource
                 owns it, so no client's delete can ever reach it.
  feint-uplink   host plumbing, released since #521 — and it stayed, because
                 fnt-default still drew from it. One release short, and both
                 networks survive.
  scw-<digest>   the rule set of the Scaleway project's default security group.
                 Permissive, therefore attached to nothing (EnforcesNothing),
                 and undeletable by a client: the API answers "the default
                 security group of a project cannot be deleted". Its identifier
                 is minted per run, so one host ACL accumulated per session.
  exo-<digest>   the same for the Exoscale account's default group, whose
                 identifier is fixed, so it survives rather than piles up.

One property joins them: no client call can remove any of them, so leaving them
measured nothing about the suites. That is #521's own sentence about the uplink,
and it is why this is not --cleanup: a machine or a network a client was
supposed to delete and did not is a leak, it stays, and the closing doorstep
fails the run that leaked it.

WHAT CHANGED

machine.UplinkReleaser becomes machine.PlumbingReleaser, and Incus.ReleasePlumbing
gives back all three kinds, in the only order the runtime accepts: rule sets,
then the default network, then the uplink. Reversed, every step answers "in use"
and the exit gives back nothing, which is exactly the measured state. Each
release answers both questions before a destructive command — is it ours, and is
anything still drawing from it — so a `feint stop` without --cleanup leaves the
firewall on the machines it deliberately leaves running. An operator's rule set,
and an unlabelled network under fnt-default's short and ordinary name, are never
touched.

The second half is the sentence, and it was as wrong as the position. `feint
clean --check --closing` and `tools/conformance/guard.sh leftovers-after` ask the
identical question and name *this* run; runtime-proof.yml asks it after its own
stop instead of letting the next step meet the residue, and not on the failure
path, where the leftovers are a crash's wreckage rather than a leak. The doorstep
now names the rule sets it does *not* refuse on, with the reason, instead of
passing over them in silence — until now they were printed only when some other
object had already made the check refuse.

The ledger's attribution column went with them: a network is found by the label
Survey reads, not by the `fnt-` prefix the column claimed, and the first network
the failing leg named — feint-uplink — carries no such prefix. A column that can
name a mark the object does not carry cannot decide what may be touched.

MEASURED

  before  the three ssh suites under FEINT_VM=incus-ovn, then feint stop, then
          the gate's doorstep: exit 1, the four objects above, byte for byte
          the CI failure
  after   the same sequence: the exit logs "released the plumbing this run held:
          scw-…, exo-…, fnt-default, feint-uplink", the host holds nothing of
          ours, and both the closing and the doorstep forms exit 0
  gates   mise run prepush; mise run conformance (green, 267 s);
          conformance:leg -- probe, -- fields, and conformance:environment;
          22 mutations of a-run-ends-where-it-could-start.json replayed in
          place, plus the three retargeted in run-leaves-nothing.json and
          unkillable-dhcp-orphan.json, every one red

Not proved here: FEINT_VM=incus-ovn conformance:leg -- runtime is red on this
station at the Outscale "two Nets, a machine in each" step, and it is red the
same way on 2f90c42 with the same address — a pre-existing station verdict, not
this change. The leg that carries the defect this commit fixes is the ssh
sequence above, which that leg does not run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@stephrobert
stephrobert merged commit 379e1f8 into main Aug 28, 2026
30 checks passed
@stephrobert
stephrobert deleted the fix/runtime-proof-closing-doorstep branch August 28, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A green conformance run leaves the uplink and one detached rule set behind, and its own doorstep then refuses the next run

1 participant