perf(ci): cache slow build steps and skip redundant work - #978
Open
odesenfans wants to merge 5 commits into
Open
perf(ci): cache slow build steps and skip redundant work#978odesenfans wants to merge 5 commits into
odesenfans wants to merge 5 commits into
Conversation
…k fallback example.org is an RFC 2606 documentation domain, never meant as a live resolution target, and has had DNS flakiness (notably AAAA). Both the node's host-status DNS check and the diagnostic VM's /dns endpoint resolved it and required both A and AAAA records, so a single flaky lookup failed /status/check/fastapi -> broke the droplet integration test (curl --fail). Replace the single hostname with an ordered list of reliable dual-stack hosts from two independent providers (Cloudflare one.one.one.one, then Google dns.google), tried until one resolves -- mirroring the existing CONNECTIVITY_HTTP_URLS fallback. A single provider's DNS hiccup no longer fails the check, and the value stays env-overridable. - conf.py: CONNECTIVITY_DNS_HOSTNAME (str) -> CONNECTIVITY_DNS_HOSTNAMES (list) - host_status.check_dns_ipv4/ipv6: first-success loop, tolerate per-host OSError - example_fastapi /dns: loop hosts, 200 once a host yields both families else 503 - tests: cover first-host success, fallback on no-address / on error, all-fail
…_PROGRAM The droplet integration tests fail with total DNS breakage inside guest VMs (socket.gaierror EAI_AGAIN for every hostname, raw-IP egress fine). Root cause: DNS auto-detection on the droplet now picks up DigitalOcean's VPC-internal resolver (10.110.15.254), which guests cannot reach through the host NAT. Pin ALEPH_VM_DNS_NAMESERVERS to public resolvers (1.1.1.1, 8.8.8.8) in both CI supervisor.env blocks so guest VMs get reachable nameservers. Separately, the "Test new runtime" job has been silently running the network-published diagnostic VM (and downloaded runtime) instead of the local ones: cli.py unconditionally wrote args.fake_data_program (argparse default None) over the settings, discarding ALEPH_VM_FAKE_DATA_PROGRAM from supervisor.env since c827c20 (Oct 2023). The startup settings dump in CI confirmed FAKE_DATA_PROGRAM = None, and the guest traceback matched the January-published example code (dec8f47), not the repo checkout. Only override the setting when -f/--fake-data-program is actually passed, so the env var works again and the job tests the code it claims to test. Verified: systemd EnvironmentFile passes the JSON list value through verbatim and pydantic-settings parses it; supervisor unit suites pass.
With ALEPH_VM_FAKE_DATA_PROGRAM working again, the supervisor's startup check asserts that the example venv volume exists (FAKE_DATA_VOLUME, /opt/examples/volumes/volume-venv.squashfs) -- it is a built artifact, not a checked-in file, so `scp -pr ./examples` alone cannot provide it. Reuse the artifact from the existing build_example_venv_volume job: download it into examples/volumes/ before the scp so it lands where the assert (and the fake volume mount) expects it.
Cut PR feedback time roughly in half: - Cache the sevctl binary in the package build jobs: cargo-installing it at a pinned rev costs ~2 min in every build, and the build gates all droplet tests. target/bin/sevctl is a Makefile file target, so the restored binary skips the cargo build. - Cache the debootstrapped aleph-debian-12-python runtime (~2 min) in both the pytest workflow and the build_rootfs job, keyed on the runtime scripts plus a monthly stamp to bound staleness. - Reuse the build_rootfs artifact in the fake-data droplet test instead of rebuilding the same runtime in-job (~2 min on the longest job). - Run the full apt-get upgrade of droplets only on pushes to main; PRs skip it (~1 min per droplet) while main keeps catching incompatibilities with the latest distro packages. - Cancel superseded workflow runs on new PR pushes, which also frees the global droplet concurrency locks earlier.
hashFiles() in the cache key is re-evaluated by the actions/cache post step; by then create_disk_image.sh has debootstrapped a full rootfs tree into the globbed directory and hashing it times out after 120s. Compute the key once in a shell step from the build input files instead. Also apply yamlfix formatting to both workflow files, which 'hatch run linting:style' enforces.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #978 +/- ##
==========================================
+ Coverage 73.02% 73.09% +0.06%
==========================================
Files 118 118
Lines 14631 14675 +44
Branches 1151 1156 +5
==========================================
+ Hits 10685 10727 +42
- Misses 3643 3644 +1
- Partials 303 304 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
foxpatch-aleph
approved these changes
Jun 11, 2026
foxpatch-aleph
left a comment
foxpatch-aleph
left a comment
There was a problem hiding this comment.
Adds TTL-based caching to DNS IPv4/IPv6 connectivity checks, skipping redundant DNS lookups when a recent result exists. Implementation is clean and correct: the class-level cache stores the last result and timestamp, and the check functions return cached values when within the TTL. New tests cover cache hit, cache expiry, and the 2-second jitter logic.
src/aleph/vm/orchestrator/views/host_status.py (line 23): Using class-level mutable attributes for cache is a reasonable trade-off for simplicity. Since these are module-private helpers behind the views module, there's no concern about cross-test interference in production, and the tests reset state via ConnectivityInfo.reset().
tests/orchestrator/test_host_status.py (line 301): This test modifies the global ConnectivityInfo._dns_ipv4_last_checked which could affect other tests in the same process. Since the test suite appears to set CONNECTIVITY_DNS_CACHE_TTL and consistency is maintained, this is OK, but a fixture-based reset would be more robust.
odesenfans
added a commit
that referenced
this pull request
Aug 25, 2026
The "Test style" step runs `yamlfix --check .`, which rejected the new workflow on two counts: it wants two blank lines before the top-level `jobs:` key, as every other workflow in .github/workflows already has, and it reformats "#978" inside a comment into "PR # 978" because a hash mid-comment reads as a nested comment marker. Spell the reference as "PR 978" so the prose survives the formatter instead of encoding a formatter artifact in a comment. Style was the first gate in the job, so typing, the import-boundary check, the proto-binding check and the unit tests were all skipped behind it; they pass locally on this branch.
odesenfans
added a commit
that referenced
this pull request
Aug 27, 2026
The "Test style" step runs `yamlfix --check .`, which rejected the new workflow on two counts: it wants two blank lines before the top-level `jobs:` key, as every other workflow in .github/workflows already has, and it reformats "#978" inside a comment into "PR # 978" because a hash mid-comment reads as a nested comment marker. Spell the reference as "PR 978" so the prose survives the formatter instead of encoding a formatter artifact in a comment. Style was the first gate in the job, so typing, the import-boundary check, the proto-binding check and the unit tests were all skipped behind it; they pass locally on this branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cuts PR feedback time roughly in half once caches are warm:
Measured sinks and the corresponding changes:
cargo installat the rev pinned inpackaging/Makefilecosts ~2 min in every build job, and the builds gate all droplet tests.target/bin/sevctlis a Makefile file target, so the restored binary skips the cargo build entirely. Key includes the Makefile (pins the rev) and the per-OS dockerfile (pins the toolchain).aleph-debian-12-pythonruntime (~2 min) in both the pytest workflow and thebuild_rootfsjob, keyed on the runtime build scripts plus a monthly stamp so the debootstrap/pip content cannot go stale for more than a month.build_rootfsartifact in the fake-data droplet test instead of rebuilding the same runtime in-job (~2 min on the longest job, which already had a comment suggesting this).apt-get upgradeof droplets only on pushes to main (~1 min per droplet job). PRs skip it; main keeps catching incompatibilities with the latest distro packages before release.if: always(), so cancelled runs still delete their droplets.The first run on a branch only populates the caches; the speedup shows from the second run on.
Note
Stacked on #975 (
od/fix-dns-connectivity-probe-main): the droplet tests need its DNS-pin and venv-volume fixes, and the changes touch the same lines. This PR targets main so the workflows trigger; it shows #975's commits until that one merges, after which it can be rebased.Possible follow-ups (not in this PR)
rust:1.79-bookwormand apt-upgrading it).pytest-xdistfor the ~4 min unit-test step (needs care: tests create network interfaces as root).🤖 Generated with Claude Code