Skip to content

ci: validate every entry when the validator itself changes - #1285

Open
MDA2AV wants to merge 1 commit into
mainfrom
ci/validate-all-frameworks
Open

ci: validate every entry when the validator itself changes#1285
MDA2AV wants to merge 1 commit into
mainfrom
ci/validate-all-frameworks

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Aug 23, 2026

Copy link
Copy Markdown
Owner

Opens a full validation sweep over every enabled entry, and fixes the gap that made one impossible.

The gap

validate.yml triggers on scripts/validate.sh, then builds its framework list from changed frameworks/** paths:

changed=$(git diff --name-only origin/$BASE...HEAD | grep '^frameworks/' | cut -d'/' -f2 | sort -u)

A PR that only touches the validator matches the trigger and then validates nothing — the list is empty and both jobs are skipped by if: ... != '[]'.

That is not hypothetical. It happened when the static staleness probe landed in #1267, and again when it was hardened in #1284. Both changed what every entry is checked against; neither ran a single entry.

The change

  • A validator change (scripts/validate.sh, scripts/validate-ws.py, or this workflow) now covers every enabled entry.
  • The workflow file joins the trigger paths, since editing it can change what validation means just as much as editing the script. That also makes this PR self-demonstrating: it validates everything.
  • Adds workflow_dispatch taking an entry name, a comma-separated list, or all, so a sweep can be run on demand without inventing a commit.

Scale

185 enabled entries — 177 on the matrix (GitHub's cap is 256) and 8 compose entries running sequentially on the self-hosted box.

Timeouts were sized for the handful of entries a normal PR touches, so both are raised:

job was now why
matrix 30 min 60 min a sweep reaches entries that build from source on a four-core hosted runner — h2o, the LTO Rust entries, Swift
self-hosted 90 min 300 min one sequential loop now covers all eight compose entries, not one or two

What to expect from this run

This is the first sweep against the hardened static probe, so it is also the blast-radius measurement for #1284. From a 12-entry local sample, no entry that passed the old probe fails the new one — every failure below already fails on main:

old probe new probes
bun, caddy, elysia, hono-bun, h2o-mruby, slimeweb, genhttp-11-ioxide PASS PASS
humming-bird, frankenphp-trueasync, genhttp-11, genhttp-11-kestrel, ioxide FAIL FAIL

Two things that sample already surfaced, both independent of the hardening:

  • genhttp-11 and genhttp-11-kestrel serve from GenHTTP's Assets.From, which never revalidates — 35s after the file changed on disk they still serve the original bytes. Their sibling genhttp-11-ioxide passes because IoxideFiles.From re-stats per request. Same framework, different handler.
  • ioxide is stale on both the identity and variant paths out to 35s, despite Program.cs:100 describing the plain path as read-per-request.

Expect some failures in this sweep to be infrastructure rather than entries — a four-core hosted runner is not the benchmark box, and a few builds are slow or flaky there. Worth reading the first run as a survey, not a verdict.

🤖 Generated with Claude Code

validate.yml triggers on scripts/validate.sh, then builds its framework list
from changed frameworks/ paths -- so a pull request that only touches the
validator matches the trigger and then validates nothing. That is what
happened when the static staleness probe landed in #1267 and again when it
was hardened in #1284: both changed what every entry is checked against, and
neither ran a single entry.

A validator change now covers every enabled entry (185: 177 on the matrix,
8 compose entries sequential on the self-hosted box). The workflow file
itself joins the trigger paths and the same rule, since editing the workflow
can change what validation means just as much as editing the script.

Also adds workflow_dispatch, taking an entry name, a comma-separated list, or
"all", so a full sweep can be run on demand without inventing a commit.

Timeouts raised to match what a full sweep reaches: 60 minutes on the matrix,
because entries that build from source on a four-core hosted runner (h2o, the
LTO Rust entries, Swift) do not finish in 30; and 300 on the self-hosted job,
which runs all eight compose entries in one sequential loop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant