test(node-integration-tests): Free up CI disk space to prevent ENOSPC failures#22132
Open
mydea wants to merge 3 commits into
Open
test(node-integration-tests): Free up CI disk space to prevent ENOSPC failures#22132mydea wants to merge 3 commits into
mydea wants to merge 3 commits into
Conversation
… failures The full Node integration suite pulls several database docker images (mssql alone is ~1.5GB) and installs per-suite node_modules concurrently. On the default ubuntu-24.04 runner (~14GB free) this peak can exhaust the disk and fail unrelated suites with `ENOSPC: no space left on device`. Because the flaky-test detector files one issue per failing test, a single disk-full run spawns a batch of unrelated "flaky" issues that are really infra noise. Reclaim the large preinstalled toolchains this job never uses (Android SDK, .NET, GHC/ghcup, CodeQL) right after checkout to give the run ~20GB of headroom, and bracket it with `df` so a future squeeze is visible in the logs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Most hosted runners now report ~90GB free, where the integration suite's docker images and per-suite installs fit comfortably, so the unconditional ~30-60s toolchain cleanup was pure overhead. Gate it behind a `df` check so it only runs when free space drops below 40GB — protecting the constrained runners that actually hit ENOSPC while keeping it a no-op on healthy ones. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some Node integration runs fail with
ENOSPC: no space left on device. The suite pulls several database docker images (mssql alone ~1.5GB) and installs per-suitenode_modulesconcurrently, and on a small-disk runner this fills/. When it happens many unrelated suites fail at once — and because the flaky-test detector files one issue per failing test, a single disk-full run spawns a batch of unrelated "flaky" issues that are really infra noise.This adds a disk-reclaim step to
job_node_integration_tests, but gated behind adfcheck so it only runs when free space is actually low (< 40GB). On a healthy runner it's a sub-second no-op ("Sufficient disk space — skipping"); on a constrained one it reclaims the large preinstalled toolchains this job never uses (Android SDK, .NET, GHC/ghcup, CodeQL — a slow ~30-60srm), freeing ~20GB.dfis logged either way so a future squeeze stays visible.Root cause
Two runs hit ENOSPC (jobs
85837176962and85929899963); every issue linked below traces back to one of them, so none are genuine test flakes.It was a genuine byte shortage, not inodes — alongside the
mkdir/mkdtempfailures, one run failed withwrite /opt/mssql/lib/system.common.sfp: no space left on device, i.e. a file write running out of bytes.Why runs differ so much on the same label:
ubuntu-24.04is a label, not a fixed disk size. GitHub-hosted runners are ephemeral VMs from a fleet that isn't uniform, and the free space on/has been enlarged over time and rolled out gradually without changing the label. Nothing in the workflow pins a disk size, so one job lands on an old ~14-30GB-free VM (→ ENOSPC) while another lands on a ~145GB one (89GB free). The failing runs and a recently measured healthy run all useubuntu-24.04.The
dfgate keys on exactly the resource that ran out, so it fires precisely on the small-disk VMs and no-ops on the large ones. If GitHub has since fully migrated the fleet to larger disks it simply becomes a cheap permanent no-op; if the fleet is still mixed it protects the constrained runs. Either way it costs healthy runners nothing.Fixes #22076
Fixes #22071
Fixes #22070
Fixes #22106
Fixes #22073
Fixes #22072
Fixes #22077
Fixes #22074
Fixes #22075
Fixes #22108
Fixes #22107
🤖 Generated with Claude Code