test: aleph-vm dev-2.1 on the new TEE server (pyaleph 0.11.2 / aleph-cli 0.18.0 / scheduler 0.2.1) - #39
test: aleph-vm dev-2.1 on the new TEE server (pyaleph 0.11.2 / aleph-cli 0.18.0 / scheduler 0.2.1)#39odesenfans wants to merge 71 commits into
Conversation
Point the CRN deployment at the aleph-vm dev-accelerate branch (PR #977: gRPC supervisor rework, integration suite, directory-based VM migration) so the testnet migration test exercises it end to end before merge. crn-up.sh fetches aleph-vm.debian-12.deb from the branch's CI build (build-deb-package-and-integration-tests.yml); the latest dev-accelerate run is green and produces that artifact. Revert to version: 1.13.0 once aleph-vm#977 lands.
scheduler 0.1.2 (#188) makes node_watcher poll the v2 executions endpoint, so a gracefully-stopping VM stays observed (present as STOPPED) instead of dropping out of the v1 running-only list and being reported 'missing' for the ~34s graceful-shutdown window. That is what broke test_instance_stop_start on the dev-accelerate aleph-vm (whose stop is a graceful guest shutdown vs the old near-instant teardown). Pairs with aleph-vm's stop=stop_vm change, which keeps the stopped VM in the pool so v2 still lists it.
aleph-vm#977 (supervisor rework) merged into dev, so the temporary dev-accelerate branch pin is obsolete (that branch will be deleted). Point at dev to keep the migration and graceful-stop tests running against the merged code until it reaches main and a release is cut. Scheduler stays at 0.1.2 (graceful-stop tolerance).
Validate Phase 2 (gRPC wire + two-service packaging) on testnet. Phase 1 (#980) has merged to dev; phase2 carries it plus the Phase 2 changes.
The static AMD SEV-SNP server runs Ubuntu 24.04 (Python 3.12), but the branch path deployed the debian-12 .deb (cp311), so its vendored pydantic_core native extension failed to import there and the supervisor crash-looped (confidential test: :4020 connection refused). Provision CRN droplets as ubuntu-24.04 (CCN already is) and fetch/deploy the ubuntu-24.04 branch .deb so the Python ABI matches on every host including the TEE server.
Diagnosing the confidential create/registration on the SEV box: in the two-service split the agent serves /control/* and /v2/about/executions/list, so the confidential allocation result and the awaiting_confidential_init state live in its journal, not the supervisor daemon's. Also switch the TEE journals from -n 5000 to --since: the supervisor crash-loops during the static-server reinstall and blew the line budget before the test window.
Validate PR-1 of the agent/supervisor boundary split (contract layer + import-linter, behavior-neutral) on testnet.
…re origin The test_confidential init-session 'VM not found' failure reproduced 2x on #986 (od/agent-supervisor-contract), whose confidential-path code is import-only vs dev. Run against dev to determine if the breakage is pre-existing (#984/#985) rather than caused by #986. Will revert the pin afterwards.
…l failure is pre-existing on dev Isolation run (manifesto pinned to dev) failed test_confidential identically (VM not found at init-session), proving the breakage is pre-existing on dev (#984/#985), not caused by #986. Restoring the pin to the PR-1 branch.
…#990) Validate the complete combined stack (tip branch builds a .deb with all four PRs) on testnet integration. Combined-stack GitHub CI is green (PR #991). Known: the confidential SSH test is a pre-existing dev failure (needs the .deb-selection fix + SEV hw); the other 23 are expected to pass.
The generated supervisor.env omitted ALEPH_VM_SUPERVISOR_GRPC_SOCKET, so the agent ran its own in-process VmPool and never dialed the supervisor daemon. The daemon ran idle alongside it. Every prior green #27 validated the in-process path, not the two-process gRPC connector the boundary work is for. Set the socket so the agent talks to the daemon over gRPC. Expect split-mode gaps to surface (endpoints that still reach the in-process pool: backups, restore, confidential, migration, network recreation, GPU reservation, persistent programs).
The manifesto pinned od/fix-confidential-reconcile-teardown, deleted after #987 merged, so crn-up.sh could not fetch a build. dev now carries the full boundary stack + the firecracker Configuration vm_id fix, and crn-up.sh sets the gRPC socket. Deploy dev to exercise the two-process connector end-to-end for the first time.
…(aleph-vm#1001-#1010)
…24/24 on the testnet)
…-accelerate # Conflicts: # .github/workflows/pr-tests.yml # scripts/crn-up.sh
Point every component at its V-PROGRAM-capable pre-release so the full flow (CLI create/show/call with RA-TLS attestation) can be validated on the testnet: - pyaleph 0.10.3-rc7: first pinned release with the V-PROGRAM message handler, vms DB migration and cost model. - aleph-cli 0.17.0-rc1: vprogram command set (create/show/call with RA-TLS attestation verification). Subsumes the pin from PR #32. - scheduler-rs / scheduler-api vprogram-rc1: one-off images built from od/vprogram-scheduling (scheduler#193), adding V-PROGRAM ingestion, SEV-SNP capability matching and the v_programs allocation bucket. - aleph-vm od/vprogram-integration: dev (gRPC split) plus the full vprogram stack (#1050..#1079) including the SNP launch path, measured workload volumes and the attestation port host mapping.
Adds the first genuine end-to-end attestation test: deploy the fib-service measured workload with the Rust CLI, wait for scheduler placement on the SEV-SNP TEE server, then exercise vprogram call, which only returns a body after the AMD report chain verifies, the TLS key is bound to the report, and the launch measurement matches the ones the CLI pinned at create time. A wrong --expected-measurement must fail without printing any body. Pieces: - scripts/vprogram-artifacts.sh: fetch the prebuilt fixtures (runtime bundle, manifest template, fib-workload.ext4) from the vprogram-fixtures-1 release, sha256-pinned. The fixtures are nix-reproducible builds of aleph-vm od/vprogram-integration rev de9ed350; rebuild instructions in the release notes. - conftest: vprogram_dir + vprogram_runtime_hash fixtures. The bundle is uploaded per run (fresh CCN each time); the manifest template's zeroed bundle.ref is patched with the run's STORE hash before upload. - local-up.sh: export ALEPH_TESTNET_VPROGRAM_DIR when the fixtures dir exists; the test skips otherwise. - pr-tests.yml: fetch fixtures on the CCN + install cryptsetup-bin there (vprogram create runs veritysetup locally to hash the workload).
… sample First live run (31373843773) taught two things: - vprogram create --json emits a submission receipt (type: V-PROGRAM, item_hash, message_status) rather than a full message envelope; match on the type field instead of content.verification. - The rc1 CLI resolves the attested endpoint exactly once at readiness, before the CRN maps the attestation port (the mapping happens only after the SNP guest's measured boot: aleph-vm agent/run.py reconcile_vprogram_port_forwards runs post-_wait_until_running). Poll vprogram show as a fallback so the attestation path still gets exercised; drop the fallback once aleph-rs#318 ships in the pinned CLI.
Round-2 run (31378982391) launched the V-PROGRAM but the guest never started: the agent created the VM then hit 'did not reach RUNNING within 120.0s' twice, and no aleph-vm-controller@ unit ever appeared for it. Root cause: the aleph-vm deb ships both supervisor daemon implementations behind ALEPH_VM_SUPERVISOR_IMPL (default python), and the V-PROGRAM SNP auto-launch only exists in the Rust daemon (lifecycle.rs: session-dir derivation, verity sidecar force-insert, measured cmdline with workload_roothash; none of it exists in the python daemon). The python daemon parks the AlephQemuConfidentialInstance execution waiting for the operator session dance that classic confidential instances perform and V-PROGRAMs never do. Set ALEPH_VM_SUPERVISOR_IMPL=rust on the confidential CRN only (the TEE server, the sole node that can run V-PROGRAMs). The Rust daemon also implements the classic CVM session flow (confidential.rs), so test_confidential stays covered; DO CRNs keep the default python daemon that PR #27 has been validating. Also stop + reset-failed all aleph-vm-controller@ units in tee-reset.sh: the state wipe deletes controller.json files but left the units restart-looping (observed at restart counter 844k), flooding the journal and drowning the CI log capture.
Round 3 (31382627461) got the full chain working: measured SNP boot
under the Rust daemon, attestation port mapped, endpoint resolved on
the TEE server, and the guest's aleph-attest-agent bound :8443 with an
attested TLS identity - 3 seconds AFTER the test's first call. The CRN
maps the port at RUNNING (qemu up), a few seconds before guest
userspace finishes starting, so the first connection can land in that
gap.
Retry transport-level failures ('error sending request') for up to two
minutes; attestation verification failures still fail immediately.
Round 4 (31385362211): the SNP guest boots, aleph-attest-agent binds 0.0.0.0:8443, the DNAT host port is allocated and surfaced via the executions list, yet external connections to it fail for minutes (the CLI only sees 'error sending request'). The confidential instance's SSH forward works on the same host, so the failure is specific to the V-PROGRAM mapping. Capture the host-level view after tests: nft ruleset, interfaces, routes, listeners, plus two live probes: TEE host -> guest :8443 directly (guest reachability) and runner -> public host port (full DNAT path). Their combination localizes the break.
Round 5's diagnostics only probed the first dnat-to-:8443 rule, which turned out to be a STALE mapping from an earlier run (nft rules survive tee-reset): port 24006 answered 200 while the run's own port 24008 went unprobed. Loop over all rules so each mapping gets a host->guest and a runner->DNAT probe, with compact status-code output.
…ails Rounds 4-6 all show the identical signature: the guest's attest agent is up and listening, the run's own DNAT rule exists through the failure window (round 5's diagnostics listed it), a stale sibling rule to the same guest IP serves curl 200, yet vprogram call fails with 'error sending request' for the full retry window. Everything points at the CLI's RA-TLS verifier rejecting the handshake (most plausibly a launch measurement mismatch), which reqwest hides behind a transport-looking error until aleph-rs#319 ships. Capture ground truth at failure time: on the first failed call, curl the same endpoint with no verification and carry the HTTP status into the final assertion message. 200 there + CLI failure = verifier rejection, beyond doubt.
rc2 carries aleph-rs#318 (create --wait polls for the attested endpoint within the wait budget) and #319 (attestation rejections are named instead of hiding behind 'error sending request'). - manifesto: aleph-cli 0.17.0-rc1 -> 0.17.0-rc2. - test: remove the show-polling fallback; the ready payload's attested_endpoint is the CLI contract again. The warm-up retry stays (the port maps a few seconds before the guest binds), but with #319 a verification failure no longer matches the transient predicate, so it fails fast with the actual reason.
Run 32372006187: the compose VM booted, attested (nonce freshness verified) and the call reached the guest, but the whoami container was not up yet; the attest agent answered with an upstream-unreachable 502 body while the CLI exited 0, so the retry loop took the first sample as final. Retry on that body too, with a 240s budget for the in-guest podman load + compose up, and keep transport failures retried as before.
Root cause of runs 32372006187/32375910277, reproduced in a local qemu boot of the compose runtime: rc10's docker fallback saves the archive from the digest-pinned ref, and docker save name@sha256:... writes RepoTags: null, so the guest's podman load imports a bare image ID that podman-compose cannot match against the pinned image: string (short-name resolution fails without a registries.conf, fail-closed poweroff). Supply the archive ourselves with a TAGGED docker save and --image-archive, which the CLI maps verbatim: podman load restores the tag and compose resolves it from local storage (whoami verified up on :8080 in the local boot). Archive bytes stay verity-measured either way. Drop once the CLI saves archives under a podman-matchable reference. Also raise the job timeout to 75 min: two SNP boots per run pushed the suite past 60 (run 32375910277 was killed at the limit).
…shed aleph-vm aggregation All three registers-schema adopters (pyaleph 0.10.3-rc9, aleph-cli 0.17.0-rc11, aleph-vm od/vprogram-integration-3 rebuilt on dev 2df8d87e) move together; scheduler images bumped to the vprogram-rc2 build.
Run 32416035775 attempt 2 lost the documented nodestatus-balances race: the confidential rootfs upload still saw 'Insufficient balance' after 180s while the same upload passed on attempt 1 and every earlier run of the day. Give the asynchronous funding more room; real cost shortfalls still fail, just later.
The validation aggregation branch is fully merged (aleph-vm #1131, #1147, #1148); dev's tree matches the branch that passed run 32426136992.
scheduler-rs/scheduler-api move to the 2026-08-24 vprogram-rc3 build. aleph-vm stays pinned to dev, which gained the measured-CPU SNP launch (#1145), agent-owned VM disks (#1149) and the SNP vCPU probe retry (#1150) since the last green run.
dev gained agent-side backups with supervisor guest quiescence (#1151), disk admission before resource download (#1153), the dead proto reservation cleanup (#1154) and FAILED reporting for dead controller units (#1152) since the last green run.
The Siena/Zen4c TEE host reports x86 microcode 0x0AA0021C (SPL 28) since its 2026-08-24 reboot, above rc11's Zen4c EntrySign floor of 25, so the --min-tcb microcode=21 --accept-outdated-tcb override the V-PROGRAM tests carried since 2026-08-17 is obsolete. Dropping it also makes the runs exercise the real TCB gate again instead of lowering every silicon line's floor (which is what the CLI's 'accepting a TCB below the network floor' warning was about).
Throwaway harness for the 2.0 release readiness call: install the 1.15.0 release deb, boot a QEMU instance with a disk marker and an extra port forward, upgrade the CRN in place to the dev branch's CI deb via the new crn-up.sh --upgrade, and assert the instance survived untouched: same controller unit main PID and start time, exactly one QEMU process (no double boot), same vm_index in the controller config, same rootfs inode, listed by the new supervisor/agent, no adoption errors in the journals, port forward kept, marker readable, stop/start still working and coming back on the same rootfs file. A second scenario flips ALEPH_VM_SUPERVISOR_IMPL python -> rust -> python on the upgraded node with the same checks. Ported from od/aleph-vm-upgrade-checks onto the validated dev-era stack (pyaleph rc9, scheduler vprogram-rc3, cli rc11), without the TEE host steps: the static SNP server is on mainnet and must not be touched.
…st poll lands on a serving node Run 33010889646: --register restarted the supervisor right after staking, the scheduler's node watcher polled during the restart window, marked the node Unreachable, and the first instance message stayed unscheduled for 5 min (no reschedule trigger on recovery), past the dispatch timeout. Set ALEPH_VM_NODE_HASH, restart, wait for :4020, then link. Scenario B now fails fast with a clear message when the node was never upgraded (no supervisor-launcher), instead of blaming the rust swap.
… live on the TEE server The TEE server joins as static confidential CRN index 1 (baseline 1.15.0 installed on it too). Scenario A now creates a plain instance and an SEV instance (init-session + measured start + secret injection), upgrades every CRN in one pass (UPGRADE_STATIC=1 opts the static server in), and asserts both survived untouched; the SEV VM additionally has to report SEV active and a dm-crypt root after the upgrade, and to come back on the same rootfs after stop, start, re-init-session and secret injection. Scenario B gains an SEV variant: python -> rust -> python swap with the confidential VM live, stop/start under the Rust daemon. Host commands honour the per-CRN ssh-user (the TEE server is non-root with passwordless sudo), like crn-up.sh.
…17.0 / 0.2.0 releases Until now only the confidential TEE server ran ALEPH_VM_SUPERVISOR_IMPL=rust; the DigitalOcean CRNs took the launcher's python default, so every non-confidential test (instances, programs, migration, backups, port forwards, IPv6) on this testnet still validated the Python supervisor. crn-up.sh now writes ALEPH_VM_SUPERVISOR_IMPL (default rust, overridable from the environment) into supervisor.env on every CRN. Version pins: pyaleph 0.11.0 (with its Kubo v0.43.0), aleph-cli 0.17.0, scheduler-rs/api 0.2.0 (V-PROGRAM scheduling, aleph-vm-scheduler#193), aleph-vm from od/drop-attest-cli (PR #1168; #1169 is stacked on it and has no deb build, its kernel change is not part of the deb anyway).
Kubo v0.43.0 refuses to start when the config still carries the pre-0.42
Reprovider.* keys ("deprecated configuration detected. Manually migrate
'Reprovider' fields to 'Provide'"), so pyaleph never got its IPFS daemon
and the CCN failed its readiness wait on the first run of #37.
Run 2 of #37 (33112723550) ran the whole suite on the Rust supervisor: 28/31 green including the SEV instance; the three failures were harness drift, not supervisor behaviour: - aleph-cli 0.17.0 made the V-PROGRAM name a mandatory positional (aleph-rs#361); both vprogram create calls pass one, ahead of the optional-valued --wait. - tests/test_vm_upgrade.py ran under pr-tests without GH_TOKEN and with the deleted `dev` branch as candidate. It is opt-in now: the module skips unless ALEPH_VM_UPGRADE_BRANCH is set, which only upgrade-check.yml does, and that workflow's candidate is `main`. - aleph-vm deb from `main`: #1168 merged, everything dev carried is on main now.
…0.2.1, aleph-vm 2.0.1 rc (#1183) Pins the four components of the 2.0.1 / 0.18.0 release stack: - pyaleph 0.11.2 (strict V-PROGRAM scalars, aleph-message 1.3.1) - aleph-cli 0.18.0 (verified volumes, register-carrying verification) - scheduler-rs/api 0.2.1 (lossy history fetches, SDK 0.18.0) - aleph-vm branch od/bump-aleph-message-131 = main + released aleph-message 1.3.1 (PR #1183), the 2.0.1 release candidate. Claude-Session: https://claude.ai/code/session_01QkaSkidXtLL63prnqo7NS5
…ty volume)
Fixture bump: the 2026.08.31 "1.1" runtime build (aleph-vm c5391963,
PR #1176), the first with verified-volume support: the exec and compose
bundles gain the {verified_volumes} cmdline slot and the guest
/volumes/<i> verity mounts. Both bundles and manifests are published on
mainnet (exec native-storage, compose bundle on IPFS); sha256 pins
updated accordingly.
New test test_vprogram_compose_verified_volume: builds an ext4 volume
with mkfs.ext4 -d (nginx conf + marker.json), deploys a compose
V-PROGRAM with --volume, binds /volumes/0 subpaths into nginx and
asserts the marker comes back through the attested channel. The marker
only exists inside the verity volume, so a hit proves CLI verity
format/publish, guest verity-open + mount, and the podman bind half
(aleph-vm#1176 + aleph-rs#383) end to end.
Claude-Session: https://claude.ai/code/session_01QkaSkidXtLL63prnqo7NS5
…#1184) The 2026.08.31 "1.1" runtime bundles never boot on real SEV-SNP: the minimal guest kernel (aleph-vm#1169) lacks CONFIG_X86_PAT (+ its MTRR dependency) and hangs before the first console line, which run 33502834019 exposed as all three V-PROGRAM tests timing out against VMs the 90s RA-TLS gate kept tearing down. New fixtures are built from aleph-vm od/snp-guest-kernel-pat (main + the fragment fix, aleph-vm#1184), SNP-boot-validated on the TEE server before publishing, and hosted on this repo's vprogram-fixtures-2 prerelease until the fixed runtimes are republished on mainnet. Claude-Session: https://claude.ai/code/session_01QkaSkidXtLL63prnqo7NS5
tee-reset.sh kills QEMU and wipes aleph-vm state but left the tap devices behind (NO-CARRIER, pool addresses still configured). The next run's supervisor restarts its slice allocator and re-assigns the same IPv6 /124 to a fresh tap, so two interfaces own the prefix and the kernel keeps routing the guest's IPv6 into the dead tap (linkdown routes are used by default): run 33512021462 failed only the test_vprograms IPv6 assertion, with `ip -6 route get <guest-v6>` resolving to the stale vmtap8. IPv4 never collides because its pool index keeps advancing. Also stop the lingering aleph-vm-dhcp-* dnsmasq units that hold the taps. Claude-Session: https://claude.ai/code/session_01QkaSkidXtLL63prnqo7NS5
The workflow hardcoded 2001:bc8:702:32d::/64, the flexible-IP block of the previous TEE server. That server is now a mainnet CRN and the block stays attached to it, so a run against the new TEE server would hand VMs addresses that route to the old machine. Read the pool from the AMD_SEV_SNP_IPV6_POOL repository variable instead, next to the AMD_SEV_SNP_HOST secret it belongs with; when it is unset crn-up.sh falls back to the /64 derived from the host's own address.
Pin aleph-vm to the dev-2.1 branch (2.0.1 plus the 2.1 storage work, the scheduler-allocation reconciler and the NVIDIA CC probes), and bring back the --min-tcb microcode=21 --accept-outdated-tcb override on the attested calls: the TEE server that replaced the previous one on 2026-09-09 was delivered with BIOS 1.3.3 and microcode SPL 21, below the Zen4c EntrySign floor of 25, pending a Scaleway BIOS update.
GitHub Actions
Failure
foxpatch-aleph
left a comment
foxpatch-aleph
left a comment
There was a problem hiding this comment.
Solid, deeply-documented CI work overall: the upgrade invariants in test_vm_upgrade.py are genuinely rigorous, the scheduler-poll gating around restarts fixes a real race, and tee-reset.sh's tap cleanup addresses a subtle blackhole. However, the new upgrade-check.yml hardcodes the PREVIOUS TEE server's flexible IPv6 /64 (2001:bc8:702:32d::/64), the exact pool this PR migrates into vars.AMD_SEV_SNP_IPV6_POOL because the old server (now a mainnet CRN) kept it. If this workflow runs against the new server, the TEE CRN advertises a prefix routed to another machine, silently testing a misconfigured node. It should read the same repository variable as pr-tests.yml does.
.github/workflows/upgrade-check.yml (line 275): This hardcodes the flexible /64 of the PREVIOUS TEE server — the pool this very PR moves into vars.AMD_SEV_SNP_IPV6_POOL because it now belongs to the old machine (a mainnet CRN). With workflow_dispatch enabled, anyone triggering this harness against the new TEE server configures it to advertise a prefix routed elsewhere. Use STATIC_CRN_IPV6_POOL='${{ vars.AMD_SEV_SNP_IPV6_POOL }}' like pr-tests.yml does.
.github/workflows/pr-tests.yml (line 428): The loop appends lines to tee-network.txt ('attestation DNAT rule: $rule') that themselves match the pattern 'dnat.*:8443' while grep may still be streaming the file — if grep hasn't hit EOF yet, it re-feeds the appended lines back into the loop (each iteration takes ~10s of probes, so the cascade is slow but real). Snapshot the matches first (e.g. into a variable) or write the probe output to a separate file.
tests/test_vprograms.py (line 157): Per the PR description this assertion is expected to fail until the flexible /64 is attached to the new server and AMD_SEV_SNP_IPV6_POOL is set — i.e. the suite ships red. Consider exporting the pool through local-up.sh (like ALEPH_TESTNET_CONFIDENTIAL_*) and skipping this probe when it's empty, so the IPv6-gap failure doesn't mask other regressions until the Scaleway work lands.
tests/test_vprograms.py (line 12): Docstring says 'the 2026.08.31 "1.1" runtime', but scripts/vprogram-artifacts.sh documents the 2026.09.01 build (c5391963 + the aleph-vm#1184 SNP kernel fix; the 08.31 bundles hung before console). Same stale date in test_vprogram_compose.py's docstring.
scripts/vprogram-artifacts.sh (line 41): ALEPH_IPFS_URL is never used, and the CHECKSUMS comment says the compose bundle is 'fetched from Aleph storage ... the IPFS-hosted compose bundle by its CID' while SOURCES actually fetches it from the GitHub prerelease. Drop the variable and fix the comment.
tests/test_vm_upgrade.py (line 562): If _create_confidential_instance raises after the instance message was created (e.g. unlock fails), the finally only deletes vm — the SEV VM leaks. tee-reset wipes the TEE at job end so it's contained, but wrapping the creation in its own try/finally would keep the testnet's CRN capacity honest.
…guestunmount guestunmount always calls fusermount3, which Ubuntu 26.04 confines with an AppArmor profile that only allows unmounting under $HOME, /mnt, /media, /tmp and /run/user. The build dir is under /opt/aleph-ci-cache, so the unmount was denied even for root (audit: apparmor=DENIED operation=umount profile=fusermount3) and guestunmount gave up after a minute of retries, failing the artifact step on the new TEE server. The commands run as root and a plain umount is not confined.
GitHub Actions
Failure
foxpatch-aleph
left a comment
foxpatch-aleph
left a comment
There was a problem hiding this comment.
High-quality test-infrastructure work overall — careful fixture provenance, sha256-gated downloads, good teardown hygiene, and comments tied to real CI evidence. The blocking issue is that the PR itself states the new V-PROGRAM IPv6 probe is expected to fail (no flexible /64 on the new TEE server yet), yet tests/test_vprograms.py hard-asserts HTTP 200, so merging turns main's suite red until infrastructure lands; gate that assertion on the pool override being configured, consistent with how the repo's other fixtures skip. Secondary: upgrade-check.yml hardcodes the old server's IPv6 pool instead of the new vars.AMD_SEV_SNP_IPV6_POOL, and the new tcpdump dual-capture diagnostics are unreachable because the tap-interface grep pattern can never match ip addr output.
tests/test_vprograms.py (line 157): This assertion is expected to fail at merge time per the PR description ("Until a flexible /64 is attached to the new server and the variable set, the V-PROGRAM IPv6 reachability probe is expected to fail"), which makes the whole suite red on main until Scaleway delivers. Consider gating it on the pool override being configured — e.g. have local-up.sh export a flag when STATIC_CRN_IPV6_POOL is empty and pytest.skip just this IPv6 section — consistent with how the repo's other fixtures (vprogram_dir, confidential_*, rootfs) skip when their prerequisite is missing.
.github/workflows/upgrade-check.yml (line 275): This hardcodes STATIC_CRN_IPV6_POOL=2001:bc8:702:32d::/64, while pr-tests.yml was moved to vars.AMD_SEV_SNP_IPV6_POOL "so it moves with the AMD_SEV_SNP_HOST secret". If this /64 is the previous server's flexible pool (now a mainnet CRN), the upgrade harness will configure the new TEE server with a pool that is not routed to it. Use the same repository variable here for consistency.
.github/workflows/pr-tests.yml (line 408): The dual tcpdump capture is unreachable: ip addr output looks like 18: vmtap3: <NO-CARRIER,...> — there is no iface prefix — so tap is always empty and the if [ -n "$uplink" ] && [ -n "$tap" ] block never executes. The intended pattern is probably something like grep -oE '[0-9]+: vmtap[0-9]+' | awk '{print $2}' (or grep -oE 'vmtap[0-9]+:' | tr -d ':').
.github/workflows/pr-tests.yml (line 428): This reads tee-network.txt via a pipe while the loop body appends lines to tee-network.txt that themselves match dnat.*:8443 ("attestation DNAT rule: ..."), so termination depends on grep racing to EOF before the appends. Safer to snapshot the matches first, e.g. mapfile -t rules < <(grep -E "dnat.*:8443" tee-network.txt) and iterate over the array.
scripts/vprogram-artifacts.sh (line 41): ALEPH_IPFS_URL is defined but never used, and the comment above CHECKSUMS ("the IPFS-hosted compose bundle by its CID") contradicts SOURCES, which fetches the compose bundle from the vprogram-fixtures-2 GitHub prerelease. Drop the variable and fix the comment.
tests/test_vprograms.py (line 35): TCB_FLOOR_ARGS silences the TCB gate (--accept-outdated-tcb, min microcode=21) for every attested call in both V-PROGRAM test modules, including the fail-closed measurement check. The removal condition is well documented, but given the BIOS is outside your control (Scaleway ticket), consider a TODO/tracking issue so this doesn't quietly outlive the host fix.
.github/workflows/upgrade-check.yml (line 74): github.event.number is empty on push/workflow_dispatch events, so the droplet name renders as "testnets-pr--<run_id>". Cosmetic, but the "pr-" prefix is misleading for this harness.
…'s GRUB Built on Ubuntu 26.04 (e2fsprogs 1.47.2), the encrypted root filesystem gets the orphan_file and metadata_csum_seed features by default. The GRUB embedded in the confidential OVMF unlocks the LUKS volume but cannot read that filesystem: the guest prints "Slot 0 opened" then "Failed to find any grub configuration on the encrypted volume" and halts, and the confidential instance test times out waiting for SSH. Disable both features explicitly in mkfs.ext4 so the image does not depend on the build host's mke2fs.conf. Same change to send upstream to aleph-vm's examples/example_confidential_image/build_debian_image.sh, which this file is vendored from.
GitHub Actions
Failure
foxpatch-aleph
left a comment
foxpatch-aleph
left a comment
There was a problem hiding this comment.
Overall a well-built and unusually well-commented CI/test expansion: the crn-up.sh --upgrade phase, the register-before-link reordering, the tee-reset.sh controller/tap cleanup and the new upgrade/V-PROGRAM tests are all solid and consistent with existing patterns. However, the new upgrade-check.yml hardcodes the previous TEE server's IPv6 /64 (2001:bc8:702:32d::/64) even though this same PR moves pr-tests.yml to vars.AMD_SEV_SNP_IPV6_POOL precisely because the pool follows the server — and that prefix now belongs to a machine running a mainnet CRN, so an upgrade-check run would make the new TEE server allocate guest IPv6 from a routed-elsewhere prefix. The pr-tests.yml diagnostics also have a classic ssh-consumes-stdin bug in both while-read loops, and the V-PROGRAM IPv6 assertion is being merged while known-red per the PR description. None of these are hard to fix, but the pool one in particular should not ship as-is.
.github/workflows/upgrade-check.yml (line 275): This hardcodes STATIC_CRN_IPV6_POOL=2001:bc8:702:32d::/64, but this same PR changes pr-tests.yml to read the pool from vars.AMD_SEV_SNP_IPV6_POOL precisely because the pool must follow whatever host AMD_SEV_SNP_HOST points at — and this /64 is the OLD server's flexible IP, which per the PR description now fronts a mainnet CRN. Running the upgrade check against the new TEE server would configure it to allocate guest addresses from a prefix routed to a different machine, breaking (or worse, colliding with) the SEV half of the test. Use the same STATIC_CRN_IPV6_POOL='${{ vars.AMD_SEV_SNP_IPV6_POOL }}' here.
.github/workflows/pr-tests.yml (line 428): ssh reads stdin whenever stdin is not a TTY, so every $SSH/ssh invocation inside this while read -r rule loop consumes the grep output the loop is iterating — after the first iteration read gets EOF and the loop exits. The comment's stated goal (probe EVERY attestation DNAT rule) only ever probes one rule, and the same bug exists in the while read -r cidr loop at line 391. Add -n (or </dev/null) to the ssh calls inside both loops.
tests/test_vprograms.py (line 157): This assertion is being merged red: the PR description states the V-PROGRAM IPv6 probe is expected to fail until a flexible /64 is attached and AMD_SEV_SNP_IPV6_POOL is set, and pr-tests.yml prepares the fixtures unconditionally, so every run on main will fail here after all the attested-call assertions pass. Consider gating the probe on the pool actually being configured (skip when the variable is unset — the same temporary-accommodation style as TCB_FLOOR_ARGS) so the suite stays green and the probe is trivially re-enabled when the prefix lands, instead of training everyone to ignore a red build.
.github/workflows/upgrade-check.yml (line 11): The push trigger names od/upgrade-check-1.15-to-dev, but this file arrives via od/run-dev-2.1-new-tee and will never match main after merge, so post-merge the workflow only ever runs via workflow_dispatch while still occupying the shared amd-sev-snp-server concurrency group. If the harness is truly throwaway, consider not merging the file at all (or note in the header that only manual dispatch remains); otherwise point the trigger at the branch that should keep running it.
.github/workflows/pr-tests.yml (line 281): Minor hardening: vars.AMD_SEV_SNP_IPV6_POOL is interpolated directly into the shell command line, so a value containing a single quote would silently break out of the quoting into the ssh remote command. Repo variables are admin-controlled so the risk is low, but the standard pattern is to pass it via the step's env: block instead.
scripts/crn-up.sh (line 866): grep -q ' 200 ' requires a space after the status code. If the serving daemon's access log ends the line right at the status (or uses a different request-line format), the poll never matches and every --upgrade burns the full 180s wait only to log a WARNING. ' 200' or matching on the request path + status (e.g. 'GET /status/config HTTP/1.1" 200') would be more robust across both daemon implementations.
2 participants
Full-suite run of aleph-vm dev-2.1 on the new Aleph-Cloud-paid TEE server (Scaleway C6615, EPYC 8024P Siena, Ubuntu 26.04, set up 2026-09-09), with the rest of the stack as in #38:
dev-2.1(2.0.1 + bounded download caches #1161, storage status/reclaim/reconcile #1162, allocation reconciler #1173, NVIDIA CC probes #1196/#1197)Based on #38. Two changes on top:
--min-tcb microcode=21 --accept-outdated-tcboverride on attested calls is back: the new server was delivered with BIOS 1.3.3 / microcode SPL 21, below the Zen4c EntrySign floor of 25. Scaleway BIOS ticket pending; drop the override once the host reports >= 0x0aa00219.AMD_SEV_SNP_IPV6_POOLrepository variable instead of the hardcoded flexible /64 of the previous server (now a mainnet CRN). Until a flexible /64 is attached to the new server and the variable set, the V-PROGRAM IPv6 reachability probe is expected to fail.🤖 Generated with Claude Code
https://claude.ai/code/session_01RnureenxugkU48aFn5HL6h