What the leftovers gate caught, twice
Run 33182105527, 2026-08-28, job incus-ovn, at the dataplane witness gate:
network feint-uplink: not-attempted-yet, reported-only
network fnt-default: not-attempted-yet, reported-only
rule-set exo-11e594f4819: not-attempted-yet, reported-only
rule-set scw-aa3f1f56ec5: not-attempted-yet, reported-only
a previous run left 0 machine(s) and 2 network(s) on this host.
FAIL: this host still holds what an earlier run left.
Same shape on run 33039017046, 2026-08-27.
The runner is ephemeral, so "an earlier run" is this run
runtime-proof.yml runs on ubuntu-24.04 — a fresh host every time. Nothing can
survive from the night before. The objects the gate names were created by an
earlier leg of the same job and never removed.
The gate's wording says "an earlier run" because it was written for the
maintainer's station, where that is the common case. On a hosted runner the same
sentence points at a leak, and reads like someone else's mess. Worth fixing in
the message too, since a report that misnames its own subject costs a diagnosis.
Two of the nine red nights are this
The scheduled Runtime proof has been red every night since 2026-08-21. The
causes are not one defect — measured, one per night:
| night |
cause |
| 08-26 |
the peering is active and i-219d7119 still cannot reach 10.184.1.4 |
| 08-27 |
this host still holds what an earlier run left |
| 08-28 05:16 |
the emulator did not answer /_feint/conformance |
| 08-28 14:49 |
this host still holds what an earlier run left |
| 08-29 |
platform-web-0 is not listening on 443 (see the ready: issue) |
This issue is the second column's rows 2 and 4 only. The others are their own
findings, and #538 aggregating all of them under one title is why none of the
five has been worked: a red night that means five different things converges on
nothing.
What leaks, and why it is not harmless
feint-uplink and fnt-default are the infrastructure networks of OVN mode,
not per-resource ones — the uplink carrying the routed block and the default
bridge. A leaked network is not idle plumbing (#426): it holds its address block,
the next run asks for that block under a fresh name, and the runtime refuses at
the DHCP bind, minutes in, with Address already in use. The gate exists to turn
that late confusing failure into an early named one, and it did.
The rule sets exo-… and scw-… leaking alongside says the sweep misses more
than one family.
What to establish before fixing
Whether these two networks are meant to survive a leg. The witness gate
brings one emulator per stack up on its own port after the main one stopped; if
the uplink is deliberately shared and durable, the defect is that the gate counts
it, not that the leg leaks it. Those are opposite fixes, and the comment in
runtime-proof.yml around the leftovers-after step does not settle it.
Read that first. Then either the leg sweeps what it made, or the gate stops
counting what it was never meant to own — and whichever it is gets a test that
fails without it.
Not established
- Whether the leak is specific to the witness gate's per-stack emulators or
happens on any incus-ovn leg. Both red nights had the gate enabled.
- Whether
feint clean --vm incus-ovn removes all four objects. The command is
printed by the guard, and nothing in CI runs it to find out.
What the leftovers gate caught, twice
Run 33182105527, 2026-08-28, job
incus-ovn, at the dataplane witness gate:Same shape on run 33039017046, 2026-08-27.
The runner is ephemeral, so "an earlier run" is this run
runtime-proof.ymlruns onubuntu-24.04— a fresh host every time. Nothing cansurvive from the night before. The objects the gate names were created by an
earlier leg of the same job and never removed.
The gate's wording says "an earlier run" because it was written for the
maintainer's station, where that is the common case. On a hosted runner the same
sentence points at a leak, and reads like someone else's mess. Worth fixing in
the message too, since a report that misnames its own subject costs a diagnosis.
Two of the nine red nights are this
The scheduled
Runtime proofhas been red every night since 2026-08-21. Thecauses are not one defect — measured, one per night:
i-219d7119still cannot reach10.184.1.4/_feint/conformanceplatform-web-0is not listening on 443 (see theready:issue)This issue is the second column's rows 2 and 4 only. The others are their own
findings, and #538 aggregating all of them under one title is why none of the
five has been worked: a red night that means five different things converges on
nothing.
What leaks, and why it is not harmless
feint-uplinkandfnt-defaultare the infrastructure networks of OVN mode,not per-resource ones — the uplink carrying the routed block and the default
bridge. A leaked network is not idle plumbing (#426): it holds its address block,
the next run asks for that block under a fresh name, and the runtime refuses at
the DHCP bind, minutes in, with
Address already in use. The gate exists to turnthat late confusing failure into an early named one, and it did.
The rule sets
exo-…andscw-…leaking alongside says the sweep misses morethan one family.
What to establish before fixing
Whether these two networks are meant to survive a leg. The witness gate
brings one emulator per stack up on its own port after the main one stopped; if
the uplink is deliberately shared and durable, the defect is that the gate counts
it, not that the leg leaks it. Those are opposite fixes, and the comment in
runtime-proof.ymlaround theleftovers-afterstep does not settle it.Read that first. Then either the leg sweeps what it made, or the gate stops
counting what it was never meant to own — and whichever it is gets a test that
fails without it.
Not established
happens on any
incus-ovnleg. Both red nights had the gate enabled.feint clean --vm incus-ovnremoves all four objects. The command isprinted by the guard, and nothing in CI runs it to find out.