Skip to content

fix(machine): the published address moves onto the interface that wears the rule sets, and what a machine answers on stops being a guess (#548) - #590

Merged
stephrobert merged 6 commits into
mainfrom
fix/548-the-published-address-moves-onto-the-filtered-nic
Aug 28, 2026
Merged

fix(machine): the published address moves onto the interface that wears the rule sets, and what a machine answers on stops being a guess (#548)#590
stephrobert merged 6 commits into
mainfrom
fix/548-the-published-address-moves-onto-the-filtered-nic

Conversation

@stephrobert

Copy link
Copy Markdown
Owner

Closes #548.

A server created with a public IP kept an unfiltered routed NIC beside its
private one: the group covered one and not the other. #573 made the emulator
honest about it, #581 measured a third path that works — and did not ship it,
because Inspect decided which address a machine "answers on" by taking the
first global IPv4 of the lowest-named interface
, so moving the address would
have made Binding.Address report the private one.

That inference was the defect. Interface order is a convention, and what
varies here becomes a field: the pack already declares which block is public,
and #541 gave the layer PublicAddressOf / PrivateAddressOf for exactly this
question.

The reproduction, per mode, read on the NIC

From the API alone: group drop plus one rule on 443, server created with its
flexible IP, private NIC afterwards, listeners on 443 and 80, and 8080 bare as
the negative control.

--vm incus-ovn --vm incus
before eth0 routed ipv4.address=203.0.113.2, no security.acls; eth1 managed with the group identical, eth1 also carrying the drop default
443 OPEN · 80 OPEN ← the escape · 8080 refused 443 OPEN · 80 OPEN · 8080 refused
after eth0 routed, no address at all; eth1 ipv4.routes.external=203.0.113.2/32 and security.acls=scw-… eth1 ipv4.routes=203.0.113.2/32 and security.acls
443 OPEN · 80 refused · 8080 refused 443 OPEN · 80 timed out · 8080 timed out
after a restart unchanged; eth1 carries 10.199.0.2/24 again unchanged

The bridge had never been measured on this shape. It reproduced the escape
exactly, and the remedy holds there too — which is #574's lesson repaid: the
poorest population is the one that bites.

Step 1 was necessary and not sufficient, and that was measured rather than assumed

Machine.IP is gone; Machine.Addresses is the whole set, and
PublicAddressOf / PrivateAddressOf pick from it by the pack's declared block,
so interface order can no longer decide a kind.

That alone was not enough. After a reboot of a migrated machine:

  • step 1 alone: the store recorded 10.199.0.2 — the private address only,
    because the boot records before the replay installs the addresses the plan
    promised
    . PublicAddressOf would have answered nothing.
  • with the re-read (Binding.Rescan, at the end of Reconciler.PowerOn):
    10.199.0.2,203.0.113.2.

So the re-read is part of the delivery, and the pack's Settle hook moved after
it. That ordering was named by a test, not guessed: Outscale's
TestAStoppedVmKeepsItsPrivateAddress went red the moment the re-read landed and
the hook stayed.

A second regression the measurement caught: after a restart the guest had no
address on the hot-attached NIC — a pre-existing #549-family gap that until now
only cost the peered routes, because the public address rode its own NIC.
restoreGuestNetwork now restores what a device reserves, in both modes,
before the wait.

The skip is gone, and what replaced it is an assertion

functional.sh no longer skips the public half unconditionally. The assertion it
hid passes on both stacks, on every pass:

the station reaches platform-web-0 on 443 and is refused on 9090,
both listening inside

Both stacks gained a 9090 listener, declared as service.closed_port in
proof.json — and a stack that declares none fails, rather than quietly
proving less. What remains is a skip gated on a declared capability, new in
health schema 7: capabilities.firewall_public_when_joined.
firewall_public_only stays and stays false — still true of a machine with
nowhere to move an address.

Falsification

public-address-migration.json, 9 new mutations, plus retargets across
lifecycle-tells-the-truth, address-kind, routed-nic, interface-plan,
pack-firewall-handoff and balancer-dataplanefalsify:lint was carrying
8 dead mutations before this lot; 932 fragments apply now.

Every mutation compiled and reddened its test: migration ownership (instance
and network), the escape-only-if-it-carries condition, the hot-attach replay,
Inspect's full set, the stored-address parse, the pinned-address restore, the
re-read, and its refusal to overwrite with nothing.

And one falsification found a test that had stopped measuring its subject:
moving the restore in front of the wait made
TestAGuestThatNeverConfiguresItsInterfaceIsReported pass for the wrong reason.
Its fixture now answers every command and carries nothing.

Gates

prepush · conformance:leg -- probe · -- fields ·
FEINT_VM=incus mise run conformance:leg -- runtime (all four dataplane
suites) · FEINT_VM=incus-ovn mise run conformance:functional (3 passes, 112
oks, run twice — once mid-way, once on the final code) · the four named contract
tests and the fourth pack — all green. notInTheDriverSurface still empty.

What was not obtained

Where testplan was wrong

It never named FEINT_VM=incus mise run conformance:leg -- runtime — only the
OVN one. The bridge is where the escape also lived, where the restart regression
also lived, and it is the only leg that got through the Outscale suite.

Its own closing paragraph, shipped an hour earlier in #588, says why: it routes
on paths, and the population is the half it cannot know.

…rs the rule sets, and what a machine answers on stops being a guess (#548)

A Scaleway server created *with* its flexible IP boots carrying only that
address, so the driver gives it a routed NIC (#202), and Incus accepts no
security option on one at all (#337). The private NIC arrived afterwards, took
the rule set, and left the published address on the bare interface. Measured
2026-08-27 on examples/stacks/scaleway, and reproduced from the API alone on
2026-08-28 in both driver modes: a port the group's drop default never opened
answered from the station, with a listener proved inside the machine and a bare
port as the negative control.

RouteAddress releases the address from the routed device without removing the
device, which is the only one of three remedies Incus 7.2 accepts on a running
instance, then hands it to the interface the pack named. Read on the NIC after
the move, in both modes: eth0 routed and addressless, eth1 carrying the address
and security.acls. From the station: 443 open, the port no rule names closed
with its listener still running, the bare port as the control.

Two things had to be settled first, and neither is a cost:

  - Inspect answered one address, the first of the lowest-named interface, and
    the layer above then decided its *kind* from the pack's declared block. The
    two agreed only while a routed NIC sorted before a managed one. After the
    move both share eth1, so the old reading would have published the private
    address where it published the public one, for three packs at once. The
    driver reports every address and settles nothing; PublicAddressOf and
    PrivateAddressOf pick out of that set by the block.
  - A NIC attached to a running machine is configured inside the guest by this
    driver, and nothing in the guest remembers that across a reboot: the
    machine came back with no address on it, the ninety-second wait gave up on
    a lease nobody offers, and the published address stopped answering. The
    start path restores what the device reserves, in both modes, before it
    waits.

tools/conformance/functional.sh stops skipping the public half by naming
capabilities.firewall_public_only and this issue: it asserts the same pair the
firewall family does, on the address the API publishes, gated on the runtime
declaring capabilities.firewall_public_when_joined (health schema 7). Both
example stacks gained a listener on a port no rule of their web group names,
which is what a closed half needs to be told from a dead service.

Assisted-by: Claude Code (claude-opus-5)
…e routed NIC (#548)

The move is a real change to one of our own machines, and the mode branches
below it refuse a network the emulator did not create. Releasing first and
refusing after would leave the machine having lost the address it answered on,
for a route it never got — and the network name reaching here comes from the
pack's Plan.RouteVia, built from stored values a restored snapshot controls.

TestAMigrationIsNotStartedForANetworkTheEmulatorDoesNotOwn fails without it, in
both driver modes, and asserts that no device edit was issued before the
refusal.

Also re-dates the two docs/limits.md sections this lot audited: the firewall
bounds and the packs' hand-off. Only those two — `tools/docs/limits-acks.py
refresh` re-dates all fifty, and nobody looked at the other forty-eight.

Assisted-by: Claude Code (claude-opus-5)
… lot added (#548)

The first mutation of lifecycle-tells-the-truth.json was retargeted at the
release when #548 moved that branch, and pointed at the error handling rather
than at the call: a mutation that only swallows the error still emits the
release, so the test it names would have stayed green and the spec would have
measured nothing. It skips the branch now, keeping `routed` evaluated.

And the migration spec gains the mutation for the network's ownership check,
which the guard earned when it was added: without it the address is taken off
the machine before anyone asks whether the network it is moving to is ours.

Assisted-by: Claude Code (claude-opus-5)
…at answers and carries nothing (#548)

Moving the pinned-address restore in front of the wait changed what
TestAGuestThatNeverConfiguresItsInterfaceIsReported measures without changing
its verdict: its fixture refuses every `exec`, so the restore failed first and
the error the test read was no longer the wait's. A falsification found it —
neutralising the wait's error return left the test green, which is the shape
this repository calls a comment rather than a control.

The fixture now takes every command and reports no address, so the restore
succeeds and only the wait can fail. The assertion names the wait's own wording
as well as the machine and the device.

Assisted-by: Claude Code (claude-opus-5)
…d what the plan promised (#548)

The record a boot leaves is written when Start answers, and the addresses the
plan promised are installed after that, by the replay. So the record was older
than the machine by exactly those: measured 2026-08-28 under `--vm incus-ovn`,
a server rebooted through the API once its public address had moved onto its
private NIC came back recorded as `10.199.0.2` alone, while the station reached
203.0.113.2 on it in the same pass. No pack reads that combination today —
Scaleway declares the route and publishes from its own store, Exoscale reads
the layer and declares no route — but it is the same lie #541 closed, waiting
for a fourth pack.

Reconciler.PowerOn re-reads at the end of the replay, and Binding.Rescan
overwrites only when the runtime has something to say, so a virtual machine
still booting never erases what an earlier read learned.

The pack's Settle hook moved with it, from straight after the start to after
that read. It was too early by exactly the same addresses: the one pack that
implements it copies the machine's private address into an API field, and
internal/providers/outscale's TestAStoppedVmKeepsItsPrivateAddress went red the
day the re-read was added and the hook stayed where it was.

Assisted-by: Claude Code (claude-opus-5)
@stephrobert
stephrobert merged commit 3b00d23 into main Aug 28, 2026
30 checks passed
@stephrobert
stephrobert deleted the fix/548-the-published-address-moves-onto-the-filtered-nic branch August 28, 2026 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant