fix(machine, ci): the build instance is waited for on the address its next command needs, and the runner is proved innocent before the build blames it (#583) - #585
Merged
stephrobert merged 2 commits intoAug 28, 2026
Conversation
…epository before an image build depends on it (#583) Both legs died on the night of 2026-08-28, on every image, before a suite ran: apk add --no-cache openssh in the build instance: fetching https://dl-cdn.alpinelinux.org/alpine/v3.21/main: Permission denied `Permission denied` on a connect() is EACCES — what an nftables or iptables REJECT answers, not a missing route and not a CDN refusal. Nothing here moved: the action is pinned by SHA, egress-policy is audit and silent, no native egress firewall denial appears in the run, and the Alpine CDN answered 200 in 0.121 s from the maintainer's station the same morning. What moved is the runner image — 20260729.566 on the last night that cleared this step, 20260819.586 on the night that did not. Docker's FORWARD DROP policy and Incus's own accept rules share that filter, and an image that changes either takes egress away from containers in exactly this shape. So the step measures the filter, inserts accept rules for the Incus bridge, and then **proves the repair before the build depends on it**: a throwaway container fetches one real URL and the step refuses by name if it cannot. A prerequisite asserted rather than demonstrated is what this repository spends its time removing — and the night of the 28th reported `the emulator did not answer /_feint/conformance`, which was true and thirty milliseconds too late to help. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t command needs (#583) The nightly runtime proof died on both legs on 2026-08-28, on Alpine, before a suite ran: apk add --no-cache openssh in the build instance: fetching https://dl-cdn.alpinelinux.org/alpine/v3.21/main: Permission denied It is not the runner's filter and not the CDN. Measured the same day: a container on the same bridge, launched from the same image, fetched that exact URL — the new CI guard proves it before any build depends on it, and it passed. AlmaLinux built successfully in the same pass, from the same bridge, seconds earlier. The difference is when the fetch happens. `waitForBuilder` waited for `incus exec -- true` to answer and then for `cloud-init status --wait`, whose failure is deliberately ignored because some images carry no cloud-init. Alpine's cloud image is one of them, and Alpine boots in about a second — so both waits were satisfied while DHCP had not yet handed out a lease, and apk fetched with no address. AlmaLinux takes long enough with systemd and cloud-init that the lease is there by the time dnf runs. The race was always there. What made it visible is a runner image that jumped three weeks, from 20260729.566 on the last night that cleared this step to 20260819.586 on the night that did not. So the wait is on the observable condition rather than a proxy for it — the machine carries a global address, asked of the machine itself. That is #459's rule one layer down, and it is what the CI guard had to do by hand to prove the network was fine. TestTheBuilderIsNotDeclaredReadyWithoutAnAddress carries its own control: a builder that DOES hold an address must not be waited on, or the test would pass over a wait that always blocks. Two mutations, both measured red, green after restoration. The build recorder gains the address case for the same reason: it stands in for a real host, and a real builder has one. Without it every build in that file waited out its three-minute deadline, which is how this consequence was found. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
stephrobert
deleted the
fix/583-the-build-container-reaches-the-network
branch
August 28, 2026 15:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #583.
The nightly runtime proof died on both legs on 2026-08-28, on Alpine, before a
suite ran:
It is not the runner's filter, not harden-runner, and not the CDN. It is a
race in this repository, five years old, that a runner image made visible.
The three exclusions, each measured
egress-policy: auditat both call sites, its own log sayspre-stepwith the insights link, and no native-egress-firewall denial appearsanywhere in the run — GitHub's firewall surfaces refusals in the summary with
the offending command and the rule.
throwaway container on the same bridge, from the same image, and fetches the
exact URL apk fetches. On the run that validated this branch:
== the probe carries: 10.218.237.76/24, then== a container reaches the Alpine CDN. The network was always fine.APKINDEX.tar.gzanswered 200 in 0.121 s from themaintainer's station the same morning.
And AlmaLinux built successfully in the same pass, seconds earlier, from the
same bridge.
The cause
waitForBuilderwaited forincus exec -- trueto answer, then forcloud-init status --wait, whose failure is deliberately ignored because someimages carry no cloud-init.
Alpine's cloud image is one of them, and Alpine boots in about a second. Both
waits were satisfied while DHCP had not yet handed out a lease, so apk fetched
with no address. AlmaLinux takes long enough with systemd and cloud-init that the
lease is there by the time dnf runs.
What made it visible is the runner image jumping three weeks —
20260729.566onthe last night that cleared this step,
20260819.586on the night that did not.The race was always there.
The fix
The wait is on the observable condition rather than a proxy for it: the
machine carries a global address, asked of the machine itself. That is #459's
rule one layer down, and it is exactly what the CI guard had to do by hand to
prove the network was fine.
A build instance that never gets one is refused by name, because a build that
proceeds without an address fails later and blames the package repository — which
is what happened for a week.
Proof, on the runner and in a test
The validating run of this branch: the
incusleg is green, the first since2026-08-21, and Alpine builds:
TestTheBuilderIsNotDeclaredReadyWithoutAnAddresscarries its own control: abuilder that does hold an address must not be waited on, or the test would
pass over a wait that always blocks. Two mutations — the wait bypassed, and an
empty address list counted as an address — both measured red, green after
restoration.
The mutations were replayed by hand against a warm build cache rather than
through the falsification harness, which copies the repository and recompiles the
whole module per mutation: minutes for
internal/core/machine, seconds this way.The spec is declared in
tools/falsify/specs/sofalsify:allreplays it atnight, where that cost does not matter.
The CI guard, and why it stays
It proved the network before the build depended on it, and that is what let the
diagnosis land on our code instead of GitHub's. It stays for the next time: it
measures the filter, restores accept rules for the Incus bridge, and fails by
name if a container still cannot reach a package repository.
The night of the 28th reported
FAIL: the emulator did not answer /_feint/conformance— true, and thirty milliseconds too late to be useful,because the report step runs under
if: always()and met an emulator that neverstarted. That legibility half is recorded on #583 for its own change.
One consequence, found and fixed here
buildRecorderanswered nothing to the address query, so every build inincus_imagebuild_test.gowaited out its three-minute deadline —TestOneBuilderPerImageAndPerProcesstook 360 s and failed. The recorder standsin for a real host and a real builder has an address, so it now says so. 0.58 s
after.
Gates
mise run prepush0 ·actionlintandzizmor --persona regular --min-severity lowclean on the workflow · the two mutations bite · and the workflow itself wasdispatched twice on this branch to measure, which is how the guard's own output
became evidence.