fix(machine): the published address moves onto the interface that wears the rule sets, and what a machine answers on stops being a guess (#548) - #590
Merged
stephrobert merged 6 commits intoAug 28, 2026
Conversation
…rs the rule sets, and what a machine answers on stops being a guess (#548) A Scaleway server created *with* its flexible IP boots carrying only that address, so the driver gives it a routed NIC (#202), and Incus accepts no security option on one at all (#337). The private NIC arrived afterwards, took the rule set, and left the published address on the bare interface. Measured 2026-08-27 on examples/stacks/scaleway, and reproduced from the API alone on 2026-08-28 in both driver modes: a port the group's drop default never opened answered from the station, with a listener proved inside the machine and a bare port as the negative control. RouteAddress releases the address from the routed device without removing the device, which is the only one of three remedies Incus 7.2 accepts on a running instance, then hands it to the interface the pack named. Read on the NIC after the move, in both modes: eth0 routed and addressless, eth1 carrying the address and security.acls. From the station: 443 open, the port no rule names closed with its listener still running, the bare port as the control. Two things had to be settled first, and neither is a cost: - Inspect answered one address, the first of the lowest-named interface, and the layer above then decided its *kind* from the pack's declared block. The two agreed only while a routed NIC sorted before a managed one. After the move both share eth1, so the old reading would have published the private address where it published the public one, for three packs at once. The driver reports every address and settles nothing; PublicAddressOf and PrivateAddressOf pick out of that set by the block. - A NIC attached to a running machine is configured inside the guest by this driver, and nothing in the guest remembers that across a reboot: the machine came back with no address on it, the ninety-second wait gave up on a lease nobody offers, and the published address stopped answering. The start path restores what the device reserves, in both modes, before it waits. tools/conformance/functional.sh stops skipping the public half by naming capabilities.firewall_public_only and this issue: it asserts the same pair the firewall family does, on the address the API publishes, gated on the runtime declaring capabilities.firewall_public_when_joined (health schema 7). Both example stacks gained a listener on a port no rule of their web group names, which is what a closed half needs to be told from a dead service. Assisted-by: Claude Code (claude-opus-5)
…e routed NIC (#548) The move is a real change to one of our own machines, and the mode branches below it refuse a network the emulator did not create. Releasing first and refusing after would leave the machine having lost the address it answered on, for a route it never got — and the network name reaching here comes from the pack's Plan.RouteVia, built from stored values a restored snapshot controls. TestAMigrationIsNotStartedForANetworkTheEmulatorDoesNotOwn fails without it, in both driver modes, and asserts that no device edit was issued before the refusal. Also re-dates the two docs/limits.md sections this lot audited: the firewall bounds and the packs' hand-off. Only those two — `tools/docs/limits-acks.py refresh` re-dates all fifty, and nobody looked at the other forty-eight. Assisted-by: Claude Code (claude-opus-5)
… lot added (#548) The first mutation of lifecycle-tells-the-truth.json was retargeted at the release when #548 moved that branch, and pointed at the error handling rather than at the call: a mutation that only swallows the error still emits the release, so the test it names would have stayed green and the spec would have measured nothing. It skips the branch now, keeping `routed` evaluated. And the migration spec gains the mutation for the network's ownership check, which the guard earned when it was added: without it the address is taken off the machine before anyone asks whether the network it is moving to is ours. Assisted-by: Claude Code (claude-opus-5)
…at answers and carries nothing (#548) Moving the pinned-address restore in front of the wait changed what TestAGuestThatNeverConfiguresItsInterfaceIsReported measures without changing its verdict: its fixture refuses every `exec`, so the restore failed first and the error the test read was no longer the wait's. A falsification found it — neutralising the wait's error return left the test green, which is the shape this repository calls a comment rather than a control. The fixture now takes every command and reports no address, so the restore succeeds and only the wait can fail. The assertion names the wait's own wording as well as the machine and the device. Assisted-by: Claude Code (claude-opus-5)
…d what the plan promised (#548) The record a boot leaves is written when Start answers, and the addresses the plan promised are installed after that, by the replay. So the record was older than the machine by exactly those: measured 2026-08-28 under `--vm incus-ovn`, a server rebooted through the API once its public address had moved onto its private NIC came back recorded as `10.199.0.2` alone, while the station reached 203.0.113.2 on it in the same pass. No pack reads that combination today — Scaleway declares the route and publishes from its own store, Exoscale reads the layer and declares no route — but it is the same lie #541 closed, waiting for a fourth pack. Reconciler.PowerOn re-reads at the end of the replay, and Binding.Rescan overwrites only when the runtime has something to say, so a virtual machine still booting never erases what an earlier read learned. The pack's Settle hook moved with it, from straight after the start to after that read. It was too early by exactly the same addresses: the one pack that implements it copies the machine's private address into an API field, and internal/providers/outscale's TestAStoppedVmKeepsItsPrivateAddress went red the day the re-read was added and the hook stayed where it was. Assisted-by: Claude Code (claude-opus-5)
Assisted-by: Claude Code (claude-opus-5)
stephrobert
deleted the
fix/548-the-published-address-moves-onto-the-filtered-nic
branch
August 28, 2026 19:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #548.
A server created with a public IP kept an unfiltered routed NIC beside its
private one: the group covered one and not the other. #573 made the emulator
honest about it, #581 measured a third path that works — and did not ship it,
because
Inspectdecided which address a machine "answers on" by taking thefirst global IPv4 of the lowest-named interface, so moving the address would
have made
Binding.Addressreport the private one.That inference was the defect. Interface order is a convention, and what
varies here becomes a field: the pack already declares which block is public,
and #541 gave the layer
PublicAddressOf/PrivateAddressOffor exactly thisquestion.
The reproduction, per mode, read on the NIC
From the API alone: group
dropplus one rule on 443, server created with itsflexible IP, private NIC afterwards, listeners on 443 and 80, and 8080 bare as
the negative control.
--vm incus-ovn--vm incuseth0routedipv4.address=203.0.113.2, nosecurity.acls;eth1managed with the groupeth1also carrying the drop defaulteth0routed, no address at all;eth1ipv4.routes.external=203.0.113.2/32andsecurity.acls=scw-…eth1ipv4.routes=203.0.113.2/32andsecurity.aclseth1carries10.199.0.2/24againThe bridge had never been measured on this shape. It reproduced the escape
exactly, and the remedy holds there too — which is #574's lesson repaid: the
poorest population is the one that bites.
Step 1 was necessary and not sufficient, and that was measured rather than assumed
Machine.IPis gone;Machine.Addressesis the whole set, andPublicAddressOf/PrivateAddressOfpick from it by the pack's declared block,so interface order can no longer decide a kind.
That alone was not enough. After a reboot of a migrated machine:
10.199.0.2— the private address only,because the boot records before the replay installs the addresses the plan
promised.
PublicAddressOfwould have answered nothing.Binding.Rescan, at the end ofReconciler.PowerOn):10.199.0.2,203.0.113.2.So the re-read is part of the delivery, and the pack's
Settlehook moved afterit. That ordering was named by a test, not guessed: Outscale's
TestAStoppedVmKeepsItsPrivateAddresswent red the moment the re-read landed andthe hook stayed.
A second regression the measurement caught: after a restart the guest had no
address on the hot-attached NIC — a pre-existing #549-family gap that until now
only cost the peered routes, because the public address rode its own NIC.
restoreGuestNetworknow restores what a device reserves, in both modes,before the wait.
The skip is gone, and what replaced it is an assertion
functional.shno longer skips the public half unconditionally. The assertion ithid passes on both stacks, on every pass:
Both stacks gained a 9090 listener, declared as
service.closed_portinproof.json— and a stack that declares none fails, rather than quietlyproving less. What remains is a skip gated on a declared capability, new in
health schema 7:
capabilities.firewall_public_when_joined.firewall_public_onlystays and stays false — still true of a machine withnowhere to move an address.
Falsification
public-address-migration.json, 9 new mutations, plus retargets acrosslifecycle-tells-the-truth,address-kind,routed-nic,interface-plan,pack-firewall-handoffandbalancer-dataplane—falsify:lintwas carrying8 dead mutations before this lot; 932 fragments apply now.
Every mutation compiled and reddened its test: migration ownership (instance
and network), the escape-only-if-it-carries condition, the hot-attach replay,
Inspect's full set, the stored-address parse, the pinned-address restore, there-read, and its refusal to overwrite with nothing.
And one falsification found a test that had stopped measuring its subject:
moving the restore in front of the wait made
TestAGuestThatNeverConfiguresItsInterfaceIsReportedpass for the wrong reason.Its fixture now answers every command and carries nothing.
Gates
prepush·conformance:leg -- probe·-- fields·FEINT_VM=incus mise run conformance:leg -- runtime(all four dataplanesuites) ·
FEINT_VM=incus-ovn mise run conformance:functional(3 passes, 112oks, run twice — once mid-way, once on the final code) · the four named contract
tests and the fourth pack — all green.
notInTheDriverSurfacestill empty.What was not obtained
FEINT_VM=incus-ovn conformance:leg -- runtimeis red, and not from thisbranch.
outscale/network.shfails at "two Nets, a machine in each":feint-osc-… does not carry 10.184.1.4.feintwas rebuilt from8467c11andthe same suite run on the same station: identical failure. Pre-existing,
OVN-only — the bridge leg passes it — and filed as The runtime leg fails on the maintainer's station and passes in CI, at a wait of 24 seconds for an address under OVN: a verdict that depends on the host #587. The leg stops there,
so the exoscale and balancer suites did not run under OVN; they did under the
bridge.
pinganswersnothing), the same before and after, station as the control.
docs/limits.mdnow says so rather than repeating fix(store, scaleway): what a pack stores is what a snapshot gives back, and the VPC body answers the name Scaleway publishes today (#567, #570) #581's "outbound survives".
mise run limits:checkis red onmaintoo, on the#277section. Only thetwo sections actually audited were re-dated; re-dating fifty would claim a pass
nobody made.
conformance:stacksand the 27 other specs the plan named were not played;lint proves their fragments still apply and the nightly replays them.
Where
testplanwas wrongIt never named
FEINT_VM=incus mise run conformance:leg -- runtime— only theOVN one. The bridge is where the escape also lived, where the restart regression
also lived, and it is the only leg that got through the Outscale suite.
Its own closing paragraph, shipped an hour earlier in #588, says why: it routes
on paths, and the population is the half it cannot know.