Skip to content

Shard the CI test matrix into four time-balanced slices - #1028

Merged
TheGreatAxios merged 3 commits into
mainfrom
cl-7912-shard-the-ci-test-matrix-toward-30-90s-wall-clock
Sep 14, 2026
Merged

TheGreatAxios merged 3 commits into
mainfrom
cl-7912-shard-the-ci-test-matrix-toward-30-90s-wall-clock

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Collaborator

Summary

  • CI splits the same ./src ./tests ./evals ./scripts union into four time-balanced --shard=k/4 slices, balanced by the checked-in per-file durations in scripts/ci-timings.json
  • A non-blocking nightly one-process job reruns the whole union to catch cross-shard mock pollution the shards cannot see

Verification

  • bun run check passes (lint, typecheck, build, 7466-test guarded suite)
  • Gate test covers the shard matrix: all four slices present, shard command carries the suite union plus --shard/--timings
  • Local shard walls: 38s, 41s, 42s, 49s against a 144s one-process suite

Fixes CL-7912

@linear-code

linear-code Bot commented Sep 14, 2026

Copy link
Copy Markdown

CL-7912

@TheGreatAxios

Copy link
Copy Markdown
Collaborator Author

Review · Approve

Time-balanced 4-shard CI with checked-in timings, nightly one-process pollution detector, and a stronger gate test.

Findings (all verified against branch head, no blocking issues)

  • .github/workflows/ci.yml (test job) — union identical: every shard runs check:projects-dir-guard ./src ./tests ./evals ./scripts --shard=k/4 --timings=./scripts/ci-timings.json, the same path union as the test script. Seeds fixed at 424242 (scripts/test-paths.ts carries --seed 424242; guard forwards shard flags through to test:paths). fail-fast: false, guard delegation, and ripgrep install all intact in both the shard job and the detector job.
  • scripts/ci-timings.json — sane: 511 entries, all .test.ts, sum ~144s, skewed distribution (largest entries 14/9/7s, p50 6ms). Refresh policy is documented and actionable (exact regen command in the workflow comment and CONTRIBUTING, ~20% skew trigger). Verified bun test --help supports --shard/--timings/--update-timings, so the flags and regen path are real.
  • test-one-process — closes the CL-6967 masking gap structurally: unfiltered full-union run identical to the local bun run check gate, nightly cron plus workflow_dispatch, correctly skipped on PRs (confirmed skipping in the latest checks run), continue-on-error: true so a slow full run cannot hold the gate while still showing red for triage. Correct trade-off; latest run green across all four shards (45/45/56/58s) plus test (src) compat.
  • tests/unit/check-gate.test.ts — real coverage: asserts all four slices, the --shard template, --timings, and file-set equality between shard filters and suite union expanded to test files. Strictly stronger than a path-string compare.

Subsumption verdict: #1026 is fully subsumed — yes

Same 4 files in both PRs; CONTRIBUTING byte-identical at both heads; ci.yml identical except #1028 keeps the test-src protect-main compat job that #1026 drops (merging #1026 would delete the required test (src) context); #1028's gate test is strictly stronger (file-set expansion vs string compare); timings differ only by regen values. #1026 is branched from pre-2880f9a8c main (its head lacks the 4-path-shard ancestor), so its base is stale. #1028 also cleanly supersedes main's own 4-path-shard change (src-a/b/c matrix removed, test-src compat preserved). #1026 should be closed in favor of this PR.

Nits (non-blocking, file for later)

  • The test-src compat comment still says "The src-a/src-b/src-c matrix legs replaced..." — those legs no longer exist; should say the --shard legs.
  • 6 files under tests/fixtures/** are in the union but have no ci-timings.json entries (balancing-only impact; self-heals on next regen).

@TheGreatAxios
TheGreatAxios merged commit b509ed3 into main Sep 14, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant