fix(machine): the hot attach takes turns too, and a transient port-group conflict stops disarming the firewall capability

[stephrobert]

Follows #577, which closed the same shape on three doors of Start. Two
neighbouring sites were measured in passing there and left alone. One turned
out to be worse than reported, and the other's remedy is the opposite of #577's
— with the measurement that says why.

Site 1 — the hot attach: the window is real, and there are two of them

Measured on a bare runtime (Incus 7.2, OVN), a network carrying an iso-* rule
set, a running container, and the detach (network unset then acl delete, the
order IsolateNetwork uses) fired during the operation:

branch	result
config device add — the hot attach	11 killed of 14, detach fired 25–200 ms into a ~400 ms add
device set ipv4.address — the address move	6 killed of 8
the same, detach at 0 ms or after 250 ms	clean

Word for word the same failure as #577: Cannot find security ACL ID for "iso-…".

The second branch was not in #577's report. ipv4.address is not a key an
OVN NIC updates in place: the daemon detaches and re-attaches the device — about
ten seconds on this station — and re-plugs the port. A far wider window than the
attach's, and nobody had looked at it.

Every mutating branch of Attach now holds its network's lock around its one
runtime call, the exact shape attachExtra already used. Lock order is one-way
(attach-lock then network-lock, never the reverse), so peerLock's multi-lock
rule is intact.

Consequence, taken deliberately and written at the site: two adds on one network
take turns — the same arbitration #577 made for two starts.
TestTwoMachinesAttachWithoutQueueing now drives two networks and still refuses
its named defect (#348).

One detail that is not cosmetic: the new locals are named unlock so that
start-vs-isolation-detach.json's fragments keep matching exactly once.
Without it, #577's own falsification spec would have become ambiguous silently.

Site 2 — the firewall write: the defect was not the conflict, it was the consequence

The original observation, recovered from the run that produced it: two PUT
failures at 08:58:35 and 08:58:44 on 2026-08-28, nine seconds apart, both
inserts colliding with the same row that predated them both:

write firewall osc-7b5ef90b888: Failed creating port group "incus_acl4448_net4613"
… constraint violation: Transaction causes multiple rows in "Port_Group" table

The second failure had no concurrent writer. The daemon's own existence check
can read stale for seconds under load, so no ordering of this process's calls
could have prevented it.

That is what decides retry versus ordering, and it decides it the opposite way
from #577 without contradicting it: there the missing object had been
deleted — no retry resurrects it, and ordering is the only remedy; here it is
created by the winner, and asking again is how the loser sees it.
TestANonTransientFailureIsNotRetried is unchanged.

The worst consequence was never the failed write. A transient conflict went
through firewallRefused, which withdrew capabilities.firewall for the rest
of the process's life — and every suite keyed on that capability then skips its
enforcement assertions. One timing collision silently disarmed the firewall proof
for a whole run: the instrument muzzled by the noise it was meant to ride out.
A genuine refusal from the host still withdraws the capability.

Three parts, each measured: the create and the PUT go through runUntilFree
(whose isTransientConflict already names this wording, #522); a lost create —
"The network ACL already exists", the wording Incus 7.2 answers the loser with,
measured on this station — is a success, not a refusal, because the object
the caller wanted is there; and a transient conflict no longer reaches
firewallRefused.

Gates

prepush 0 (after the rebase onto main) · FEINT_VM=incus-ovn conformance:leg -- runtime 0, 626 s, clean teardown · functional.sh outscale
green three times in a row — the suite that revealed this family judges the
fix.

Falsification: attach-vs-isolation-detach.json 2 mutations, and
ensurefirewall-conflict.json 3 (back to d.run, tolerance neutralised, the
condition falsified) — each compiles, each bites. The 16 neighbouring specs
testplan named replayed 16/16; falsify:lint 867 mutations over 141 specs, so
#577's fragments still match exactly once.

What was not obtained

• The port-group collision was never reproduced on demand, despite ~2000
  instrumented PUTs overlapping device sets and starts on a quiet host: zero
  collisions. It needs a loaded OVSDB, and shows up roughly once per session of
  functional runs. The measurement therefore rests on the two wild occurrences —
  whose UUID reading is the decisive part — and on a staged replay of the exact
  wording. If it reappears, the journal should now show it absorbed, with no pack
  ERROR and no capability withdrawal.
• No full LinkNic + AcceptNetPeering chain through the emulator for site 1.
  The measurement asked for was the bare runtime and it is done; fix(machine): a start takes turns with the isolation detach, so a peering acceptance no longer kills the machines booting beside it #577's full
  chain covers the family through the starts.