ci(mutation): run incrementally per push, and measure the truth cold each week - #87
Merged
Merged
Conversation
…each week The per-push mutation run is the deepest gate here, and it was cold on every push that touched `src/` — which is the honest measurement, but it puts the whole cold duration in the path of every merge. `incremental: true` makes the routine run re-test only what changed. That is a trade rather than a free win: an incremental run reuses stored verdicts, and Stryker's own guidance is that reused verdicts for STATIC mutants can be stale. So the truth is measured cold once a week, and that run also refreshes the baseline the following week's pushes reuse. The weekly job is a thin caller — the logic lives once in `bymaxone/.github`, which skips the run entirely when nothing that can move the score has changed since the last successful one. What stays local is what is genuinely local: the schedule, staggered against the other libraries, and the ceilings, calibrated to this repository's measured cold duration.
There was a problem hiding this comment.
Pull request overview
This PR updates the repository’s mutation-testing setup to run incrementally on default-branch pushes (faster feedback) while adding a weekly scheduled cold run to periodically re-measure the true mutation score and refresh the incremental baseline, using the shared org reusable workflow.
Changes:
- Enabled Stryker incremental mode and configured a persisted incremental baseline file.
- Added a
mutation:fullscript that deletes the incremental baseline before running Stryker to force a cold measurement. - Introduced a scheduled “Mutation (full)” workflow and tuned CI’s mutation job timeout ceiling for cold-cache scenarios.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
stryker.config.json |
Enables incremental mode and writes the incremental baseline to reports/stryker-incremental.json. |
package.json |
Adds mutation:full to force a cold run by removing the incremental baseline before execution. |
.github/workflows/mutation-full.yml |
Adds a weekly scheduled workflow that calls the org reusable cold mutation workflow with a repo-specific schedule and timeout. |
.github/workflows/ci.yml |
Sets the mutation job timeout ceiling in the reusable CI caller to accommodate cold runs when needed. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…e docs Four findings, all fair, and one that the review only half-saw. `mutation:incremental` ran `stryker run --incremental` when the config now sets `incremental: true` — it was a synonym for `pnpm mutation`, and two names for one behaviour is how a contributor picks the wrong one. Removed. The operative docs described `pnpm mutation` as the full cold run. That became false with this change, and the failure is silent: someone measuring before a release would trust a reused baseline. `CLAUDE.md`, `CONTRIBUTING.md` and `docs/mutation_testing_plan.md` now name `pnpm mutation:full` for the cold measurement. The planning documents keep their historical wording — they record what was true when written. `mutation:full` deleted the incremental file with `force` alone, which does not remove a directory. That is the one thing in this change that must not fail quietly: if the file survives, the cold run silently is not cold. Now recursive. And the review was right that single-flight was missing, though not for the reason given — the default-branch guard IS present, in the reusable. What I had dropped when extracting the shared workflow was the `concurrency` group the local copy carried. Restored on the caller, without `cancel-in-progress`: a half-finished cold run leaves no baseline at all, which is worse than waiting.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
package.json:66
- The
mutation:incrementalscript was removed, but it’s still referenced indocs/mutation_testing_results.md(Reproduce section). This makes the documented command fail and can break any existing automation that invokes it. Consider keepingmutation:incrementalas an alias (incremental is now the default viastryker.config.json).
"mutation": "stryker run",
"mutation:full": "node -e \"require('node:fs').rmSync('reports/stryker-incremental.json',{force:true,recursive:true})\" && stryker run",
"mutation:dry-run": "stryker run --dryRunOnly",
The concurrency group was `mutation-full-${{ github.ref }}`, while the comment
above it claimed a scheduled run and a manual dispatch could not both write the
shared incremental baseline. A ref-scoped key does not deliver that: a dispatch
aimed at another branch lands in a different group, so the two would run at
once and the comment would be describing a guarantee that was not there.
The default-branch guard in the reusable means such a dispatch is refused
before it writes anything, so nothing was actually at risk — but a comment that
claims more than the mechanism provides is the kind of thing that gets trusted
later.
`peer-advisory-drift.yml` in these repositories already reached this
conclusion, in as many words: single-flight across the repository, deliberately
NOT keyed by `github.ref`, because what two runs race over is one piece of
shared state. This follows it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The change
The per-push mutation run becomes incremental, and a weekly cold run is added behind it.
Mutation is the deepest gate in this repository and it was cold on every push that touched
src/— the honest measurement, but it puts the whole cold duration in the path of every merge.incremental: truemakes the routine run re-test only what changed. That is a trade rather than a free win: an incremental run reuses stored verdicts, and Stryker's own guidance is that reused verdicts for static mutants can be stale. So the truth is measured cold once a week, and that run also refreshes the baseline the following week's pushes reuse.Where the logic lives
The weekly job is a thin caller. The 130 lines that decide what counts as "the score can have moved", why the job is pinned to the default branch, and why a partial baseline must never be saved live once in
bymaxone/.githuband are shared by every library.What stays here is what is genuinely local:
What the weekly run skips
It compares against the commit the last successful run measured and skips when nothing that can move the score has changed. That surface is deliberately wider than the per-push job's
^(src/): the score also moves when a test changes (a new test kills a mutant that survived), when the Stryker or Jest configuration changes, or when a dependency changes underneath the suite. Gating onsrc/alone would skip exactly the week someone spent writing tests — the week most likely to change the number.Why the ceiling is not the old default
The per-push run is fast now, but the ceiling exists for the cold one: the first run after this merges has no baseline, and neither does any run whose cache expired. The job is cancelled rather than failed when it lands, which reads as an infrastructure blip rather than a budget — that is exactly how nest-auth's mutation gate produced nothing for weeks.