Skip to content

fix(machine, ci): the build instance is waited for on the address its next command needs, and the runner is proved innocent before the build blames it (#583) - #585

Merged
stephrobert merged 2 commits into
mainfrom
fix/583-the-build-container-reaches-the-network
Aug 28, 2026
Merged

fix(machine, ci): the build instance is waited for on the address its next command needs, and the runner is proved innocent before the build blames it (#583)#585
stephrobert merged 2 commits into
mainfrom
fix/583-the-build-container-reaches-the-network

Conversation

@stephrobert

Copy link
Copy Markdown
Owner

Closes #583.

The nightly runtime proof died on both legs on 2026-08-28, on Alpine, before a
suite ran:

apk add --no-cache openssh in the build instance:
  fetching https://dl-cdn.alpinelinux.org/alpine/v3.21/main: Permission denied

It is not the runner's filter, not harden-runner, and not the CDN. It is a
race in this repository, five years old, that a runner image made visible.

The three exclusions, each measured

  • harden-runner: egress-policy: audit at both call sites, its own log says
    pre-step with the insights link, and no native-egress-firewall denial appears
    anywhere in the run — GitHub's firewall surfaces refusals in the summary with
    the offending command and the rule.
  • The runner's packet filter: the new guard in this branch launches a
    throwaway container on the same bridge, from the same image, and fetches the
    exact URL apk fetches. On the run that validated this branch:
    == the probe carries: 10.218.237.76/24, then
    == a container reaches the Alpine CDN. The network was always fine.
  • The Alpine CDN: APKINDEX.tar.gz answered 200 in 0.121 s from the
    maintainer's station the same morning.

And AlmaLinux built successfully in the same pass, seconds earlier, from the
same bridge.

The cause

waitForBuilder waited for incus exec -- true to answer, then for
cloud-init status --wait, whose failure is deliberately ignored because some
images carry no cloud-init.

Alpine's cloud image is one of them, and Alpine boots in about a second. Both
waits were satisfied while DHCP had not yet handed out a lease, so apk fetched
with no address. AlmaLinux takes long enough with systemd and cloud-init that the
lease is there by the time dnf runs.

What made it visible is the runner image jumping three weeks — 20260729.566 on
the last night that cleared this step, 20260819.586 on the night that did not.
The race was always there.

The fix

The wait is on the observable condition rather than a proxy for it: the
machine carries a global address, asked of the machine itself. That is #459's
rule one layer down, and it is exactly what the CI guard had to do by hand to
prove the network was fine.

A build instance that never gets one is refused by name, because a build that
proceeds without an address fails later and blames the package repository — which
is what happened for a week.

Proof, on the runner and in a test

The validating run of this branch: the incus leg is green, the first since
2026-08-21
, and Alpine builds:

== feint/alpine/3.21
  launching images:alpine/3.21/cloud
  installing openssh and enabling sshd
  removing host keys, machine id and cloud-init state
  publishing feint/alpine/3.21

TestTheBuilderIsNotDeclaredReadyWithoutAnAddress carries its own control: a
builder that does hold an address must not be waited on, or the test would
pass over a wait that always blocks. Two mutations — the wait bypassed, and an
empty address list counted as an address — both measured red, green after
restoration
.

The mutations were replayed by hand against a warm build cache rather than
through the falsification harness, which copies the repository and recompiles the
whole module per mutation: minutes for internal/core/machine, seconds this way.
The spec is declared in tools/falsify/specs/ so falsify:all replays it at
night, where that cost does not matter.

The CI guard, and why it stays

It proved the network before the build depended on it, and that is what let the
diagnosis land on our code instead of GitHub's. It stays for the next time: it
measures the filter, restores accept rules for the Incus bridge, and fails by
name
if a container still cannot reach a package repository.

The night of the 28th reported FAIL: the emulator did not answer /_feint/conformance — true, and thirty milliseconds too late to be useful,
because the report step runs under if: always() and met an emulator that never
started. That legibility half is recorded on #583 for its own change.

One consequence, found and fixed here

buildRecorder answered nothing to the address query, so every build in
incus_imagebuild_test.go waited out its three-minute deadline —
TestOneBuilderPerImageAndPerProcess took 360 s and failed. The recorder stands
in for a real host and a real builder has an address, so it now says so. 0.58 s
after.

Gates

mise run prepush 0 · actionlint and zizmor --persona regular --min-severity low clean on the workflow · the two mutations bite · and the workflow itself was
dispatched twice on this branch to measure, which is how the guard's own output
became evidence.

stephrobert and others added 2 commits August 28, 2026 16:21
…epository before an image build depends on it (#583)

Both legs died on the night of 2026-08-28, on every image, before a suite ran:

    apk add --no-cache openssh in the build instance:
      fetching https://dl-cdn.alpinelinux.org/alpine/v3.21/main: Permission denied

`Permission denied` on a connect() is EACCES — what an nftables or iptables
REJECT answers, not a missing route and not a CDN refusal. Nothing here moved:
the action is pinned by SHA, egress-policy is audit and silent, no native egress
firewall denial appears in the run, and the Alpine CDN answered 200 in 0.121 s
from the maintainer's station the same morning.

What moved is the runner image — 20260729.566 on the last night that cleared
this step, 20260819.586 on the night that did not. Docker's FORWARD DROP policy
and Incus's own accept rules share that filter, and an image that changes either
takes egress away from containers in exactly this shape.

So the step measures the filter, inserts accept rules for the Incus bridge, and
then **proves the repair before the build depends on it**: a throwaway container
fetches one real URL and the step refuses by name if it cannot. A prerequisite
asserted rather than demonstrated is what this repository spends its time
removing — and the night of the 28th reported `the emulator did not answer
/_feint/conformance`, which was true and thirty milliseconds too late to help.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t command needs (#583)

The nightly runtime proof died on both legs on 2026-08-28, on Alpine, before a
suite ran:

    apk add --no-cache openssh in the build instance:
      fetching https://dl-cdn.alpinelinux.org/alpine/v3.21/main: Permission denied

It is not the runner's filter and not the CDN. Measured the same day: a container
on the same bridge, launched from the same image, fetched that exact URL — the
new CI guard proves it before any build depends on it, and it passed. AlmaLinux
built successfully in the same pass, from the same bridge, seconds earlier.

The difference is when the fetch happens. `waitForBuilder` waited for
`incus exec -- true` to answer and then for `cloud-init status --wait`, whose
failure is deliberately ignored because some images carry no cloud-init. Alpine's
cloud image is one of them, and Alpine boots in about a second — so both waits
were satisfied while DHCP had not yet handed out a lease, and apk fetched with no
address. AlmaLinux takes long enough with systemd and cloud-init that the lease
is there by the time dnf runs.

The race was always there. What made it visible is a runner image that jumped
three weeks, from 20260729.566 on the last night that cleared this step to
20260819.586 on the night that did not.

So the wait is on the observable condition rather than a proxy for it — the
machine carries a global address, asked of the machine itself. That is #459's
rule one layer down, and it is what the CI guard had to do by hand to prove the
network was fine.

TestTheBuilderIsNotDeclaredReadyWithoutAnAddress carries its own control: a
builder that DOES hold an address must not be waited on, or the test would pass
over a wait that always blocks. Two mutations, both measured red, green after
restoration.

The build recorder gains the address case for the same reason: it stands in for a
real host, and a real builder has one. Without it every build in that file waited
out its three-minute deadline, which is how this consequence was found.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@stephrobert
stephrobert merged commit 2f90c42 into main Aug 28, 2026
42 of 44 checks passed
@stephrobert
stephrobert deleted the fix/583-the-build-container-reaches-the-network branch August 28, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant