fix(machine, ci): the exit gives back what no client can delete, and the leg that leaks is the one that reddens (#521) - #586
Merged
Conversation
…he leg that leaks is the one that reddens (#521) The incus-ovn leg of runtime-proof.yml failed at the doorstep of its own witness gate, on a GitHub runner nothing had ever touched: a previous run left 0 machine(s) and 2 network(s) on this host. network feint-uplink: not-attempted-yet, reported-only network fnt-default: not-attempted-yet, reported-only rule-set exo-11e594f4819: not-attempted-yet, reported-only rule-set scw-aa3f1f56ec5: not-attempted-yet, reported-only There was no previous run. The leg's own three ssh suites had made all four, each exiting 0, and the station reproduced it exactly on 2026-08-28: same four objects, same counts, only the Scaleway digest differing because that one is minted per run. WHO MAKES EACH, AND WHY NOTHING REMOVED IT fnt-default DefaultMachineNetwork, created the first time a machine boots with no attachment of its own. Here that is the Outscale ssh suite's Vm, which lives outside a Net. No emulated resource owns it, so no client's delete can ever reach it. feint-uplink host plumbing, released since #521 — and it stayed, because fnt-default still drew from it. One release short, and both networks survive. scw-<digest> the rule set of the Scaleway project's default security group. Permissive, therefore attached to nothing (EnforcesNothing), and undeletable by a client: the API answers "the default security group of a project cannot be deleted". Its identifier is minted per run, so one host ACL accumulated per session. exo-<digest> the same for the Exoscale account's default group, whose identifier is fixed, so it survives rather than piles up. One property joins them: no client call can remove any of them, so leaving them measured nothing about the suites. That is #521's own sentence about the uplink, and it is why this is not --cleanup: a machine or a network a client was supposed to delete and did not is a leak, it stays, and the closing doorstep fails the run that leaked it. WHAT CHANGED machine.UplinkReleaser becomes machine.PlumbingReleaser, and Incus.ReleasePlumbing gives back all three kinds, in the only order the runtime accepts: rule sets, then the default network, then the uplink. Reversed, every step answers "in use" and the exit gives back nothing, which is exactly the measured state. Each release answers both questions before a destructive command — is it ours, and is anything still drawing from it — so a `feint stop` without --cleanup leaves the firewall on the machines it deliberately leaves running. An operator's rule set, and an unlabelled network under fnt-default's short and ordinary name, are never touched. The second half is the sentence, and it was as wrong as the position. `feint clean --check --closing` and `tools/conformance/guard.sh leftovers-after` ask the identical question and name *this* run; runtime-proof.yml asks it after its own stop instead of letting the next step meet the residue, and not on the failure path, where the leftovers are a crash's wreckage rather than a leak. The doorstep now names the rule sets it does *not* refuse on, with the reason, instead of passing over them in silence — until now they were printed only when some other object had already made the check refuse. The ledger's attribution column went with them: a network is found by the label Survey reads, not by the `fnt-` prefix the column claimed, and the first network the failing leg named — feint-uplink — carries no such prefix. A column that can name a mark the object does not carry cannot decide what may be touched. MEASURED before the three ssh suites under FEINT_VM=incus-ovn, then feint stop, then the gate's doorstep: exit 1, the four objects above, byte for byte the CI failure after the same sequence: the exit logs "released the plumbing this run held: scw-…, exo-…, fnt-default, feint-uplink", the host holds nothing of ours, and both the closing and the doorstep forms exit 0 gates mise run prepush; mise run conformance (green, 267 s); conformance:leg -- probe, -- fields, and conformance:environment; 22 mutations of a-run-ends-where-it-could-start.json replayed in place, plus the three retargeted in run-leaves-nothing.json and unkillable-dhcp-orphan.json, every one red Not proved here: FEINT_VM=incus-ovn conformance:leg -- runtime is red on this station at the Outscale "two Nets, a machine in each" step, and it is red the same way on 2f90c42 with the same address — a pre-existing station verdict, not this change. The leg that carries the defect this commit fixes is the ssh sequence above, which that leg does not run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #521.
The
incus-ovnleg ofruntime-proof.ymlhas been red on its doorstep, andsince #585 it is the only cause left on that workflow. Every suite passes —
the three network suites, the three ssh suites, the report, the stop — and then:
There is no previous run. It is a fresh runner, and the suites of the same job
left those objects. The doorstep of the next step found them and blamed
somebody else.
Who creates each object, and why nothing removed it
Measured by replaying the leg on the station, reading the host at every frontier.
The reproduction is byte-for-byte the CI failure — same four objects, same
counts, only the Scaleway digest differs because that one is minted per run.
fnt-defaultDefaultMachineNetworkfeint-uplinkfnt-defaultis created under OVNfnt-defaultstill drew from it (used_by=1, read on the host)scw-<digest>exo-<digest>DefaultIngress: drop, attached while the instance livesNo
osc-rule set appears, and that is not luck: an Outscale Vm outside a Netwears no security group.
The "shared session resource" hypothesis was measured and rejected
The brief offered it: perhaps
feint-uplinkandfnt-defaultare sessionresources the gate is wrong to count.
They are shared within one emulator — but the doorstep is asked when that
emulator is gone, so the session is over and the gate is right. Exempting them
would blind it to the leftovers of a killed run, which is exactly what
crash.shasserts must survive a SIGKILL and be named.So the fix belongs on the emulator side, not the gate's, and that is where it
went.
No
feint cleanwas added anywhereThat shortcut would have made the night pass and removed the only thing that
detects a real leak. The release is targeted per object, and it asks both
questions before any destructive command — is it ours (the label, and the
description
EnsureFirewallwrites) and is anything still drawing from it(
used_by, read first rather than the delete being attempted blind).A
feint stopwithout--cleanuptherefore still keeps the firewall on themachines it deliberately leaves running.
machine.UplinkReleaserbecomesmachine.PlumbingReleaser;ReleasePlumbinggives back rule sets → default network → uplink, in the only order the runtime
accepts — reversed, every step answers "in use", which is the measured state.
The gate, at the right place, saying the right thing
feint clean --check --closingandguard.sh leftovers-after: the same refusal,with the culprit named as this run.
mise.tomlandleg.shmove to it too —they carried #521's form but the doorstep's sentence.
runtime-proof.ymlgainsone step, What this leg left on the host, after its own stop, and deliberately
not
if: always(): a leg that already died left wreckage, not a leak. No jobrenamed.
And the doorstep now names the rule sets it does not refuse on, with the
reason. Until now they were printed only when some other object had already
made it refuse — so the artefact that accumulates once per session was invisible
exactly where it was all that was left.
An instrument that lied, found on the way
The ledger's attribution column recorded networks as
name-prefix:fnt-, whileSurveyselects them by label — and the first network the failing leg named,feint-uplink, carries no such prefix. Corrected, with a test; the old testasserted the lie and was rewritten.
Reproduction
Falsification
22 mutations of
a-run-ends-where-it-could-start.json— 15 new, one perguard added — plus 3 retargeted in
run-leaves-nothing.jsonandunkillable-dhcp-orphan.json. Every one red.Replayed by hand against a warm build cache:
mise run falsifycopies therepository and recompiles the whole module per mutation, which is minutes for
internal/core/machine. All are declared in the specs sofalsify:allreplaysthem nightly;
falsify:lintis green.Gates
prepush✅ ·mise run conformance✅ (267 s) ·conformance:leg -- probe✅ ·-- fields✅ ·conformance:environment✅ · the full ssh-then-doorstepsequence on the station ✅.
What was not obtained
conformance:leg -- runtimeis red on this station, twice, at the Outscale"two Nets, a machine in each" step:
feint-osc-… does not carry 10.184.1.4.Not assumed to be this branch's:
2f90c42was exported to a scratch tree,built, and the same two suites run in the same order — identical failure,
same address. Pre-existing and station-specific (CI's run 33182105527 passed
that suite). It deserves its own issue: the wait is
wait_until 24, twenty-fourseconds for a machine to carry its address under OVN.
feint imagescannot build on this station — thedefaultIncus profilecarries no NIC, so fix(machine, ci): the build instance is waited for on the address its next command needs, and the runner is proved innocent before the build blames it (#583) #585's new refusal fires. That is
feint images buildsilently depends on the operator's Incus default profile, and a station where feint has run may not have a NIC there #584. A NIC was added forthe build and removed again, profile verified restored.
conformance:witnesswas not run end to end; its doorstep was reproducedexactly, which is the line that failed.
stacksjob's own closing state was not touched.Where
testplanwas wrong, on the load-bearing pointIt named
probe,fields,runtime,environmentand 27 falsify specs — andnot the population the defect lives in. The ssh suites are in no leg, and
-- runtimeruns the network suites, which end on their ownfeint cleanandtherefore sweep the leak away before any closing doorstep sees it.
Taken as a ceiling, everything would have been green and the leg would still
fail. Floor, not ceiling — the fifth time this week.