ci: validate every entry when the validator itself changes - #1285
Open
MDA2AV wants to merge 1 commit into
Open
Conversation
validate.yml triggers on scripts/validate.sh, then builds its framework list from changed frameworks/ paths -- so a pull request that only touches the validator matches the trigger and then validates nothing. That is what happened when the static staleness probe landed in #1267 and again when it was hardened in #1284: both changed what every entry is checked against, and neither ran a single entry. A validator change now covers every enabled entry (185: 177 on the matrix, 8 compose entries sequential on the self-hosted box). The workflow file itself joins the trigger paths and the same rule, since editing the workflow can change what validation means just as much as editing the script. Also adds workflow_dispatch, taking an entry name, a comma-separated list, or "all", so a full sweep can be run on demand without inventing a commit. Timeouts raised to match what a full sweep reaches: 60 minutes on the matrix, because entries that build from source on a four-core hosted runner (h2o, the LTO Rust entries, Swift) do not finish in 30; and 300 on the self-hosted job, which runs all eight compose entries in one sequential loop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opens a full validation sweep over every enabled entry, and fixes the gap that made one impossible.
The gap
validate.ymltriggers onscripts/validate.sh, then builds its framework list from changedframeworks/**paths:A PR that only touches the validator matches the trigger and then validates nothing — the list is empty and both jobs are skipped by
if: ... != '[]'.That is not hypothetical. It happened when the static staleness probe landed in #1267, and again when it was hardened in #1284. Both changed what every entry is checked against; neither ran a single entry.
The change
scripts/validate.sh,scripts/validate-ws.py, or this workflow) now covers every enabled entry.workflow_dispatchtaking an entry name, a comma-separated list, orall, so a sweep can be run on demand without inventing a commit.Scale
185 enabled entries — 177 on the matrix (GitHub's cap is 256) and 8 compose entries running sequentially on the self-hosted box.
Timeouts were sized for the handful of entries a normal PR touches, so both are raised:
What to expect from this run
This is the first sweep against the hardened static probe, so it is also the blast-radius measurement for #1284. From a 12-entry local sample, no entry that passed the old probe fails the new one — every failure below already fails on
main:Two things that sample already surfaced, both independent of the hardening:
genhttp-11andgenhttp-11-kestrelserve from GenHTTP'sAssets.From, which never revalidates — 35s after the file changed on disk they still serve the original bytes. Their siblinggenhttp-11-ioxidepasses becauseIoxideFiles.Fromre-stats per request. Same framework, different handler.ioxideis stale on both the identity and variant paths out to 35s, despiteProgram.cs:100describing the plain path as read-per-request.Expect some failures in this sweep to be infrastructure rather than entries — a four-core hosted runner is not the benchmark box, and a few builds are slow or flaky there. Worth reading the first run as a survey, not a verdict.
🤖 Generated with Claude Code