Run the test suites CI was silently skipping - #3733
Conversation
`turbo test` runs a package's tests only if that package declares a `test` script, so a suite can sit in the repo for months without ever running. Four were in that state: @workflow/world (13 files, 160 tests), @workflow/cli (5 / 51), @workflow/nitro (1 / 30), and two files under packages/core/e2e that no workflow named. Wire each one up, and add scripts/check-test-suites-wired.mjs plus a lint job so the next unwired suite fails CI instead of going unnoticed. Fixes #3731 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (32 failed)astro-node (2 failed):
astro-quickjs (1 failed):
example-node (1 failed):
example-quickjs (1 failed):
express-node (1 failed):
express-quickjs (2 failed):
hono-quickjs (1 failed):
nest-node (1 failed):
nest-quickjs (1 failed):
nextjs-turbopack-quickjs (2 failed):
nextjs-webpack-node (1 failed):
nitro-quickjs (2 failed):
nuxt-quickjs (2 failed):
python-node (8 failed):
sveltekit-node (1 failed):
sveltekit-quickjs (1 failed):
tanstack-start-node (2 failed):
vite-node (1 failed):
vite-quickjs (1 failed):
🌐 Cross-language Conformance (9 failed)python (9 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ❌ ▲ Vercel Production | 3546 | 32 | 742 | 4320 |
| ✅ 💻 Local Development | 3922 | 0 | 558 | 4480 |
| ✅ 📦 Local Production | 3922 | 0 | 558 | 4480 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 558 | 4480 |
| ✅ 🪟 Windows | 320 | 0 | 0 | 320 |
| ❌ 🌐 Cross-language Conformance | 0 | 9 | 132 | 141 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| Total | 15659 | 41 | 2548 | 18248 |
Details by Category
❌ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ astro-node | 130 | 2 | 28 |
| ❌ astro-quickjs | 131 | 1 | 28 |
| ❌ example-node | 131 | 1 | 28 |
| ❌ example-quickjs | 131 | 1 | 28 |
| ❌ express-node | 131 | 1 | 28 |
| ❌ express-quickjs | 130 | 2 | 28 |
| ✅ fastify-node | 132 | 0 | 28 |
| ✅ fastify-quickjs | 132 | 0 | 28 |
| ✅ hono-node | 132 | 0 | 28 |
| ❌ hono-quickjs | 131 | 1 | 28 |
| ❌ nest-node | 131 | 1 | 28 |
| ❌ nest-quickjs | 131 | 1 | 28 |
| ✅ nextjs-turbopack-node | 157 | 0 | 3 |
| ❌ nextjs-turbopack-quickjs | 155 | 2 | 3 |
| ❌ nextjs-webpack-node | 156 | 1 | 3 |
| ✅ nextjs-webpack-quickjs | 157 | 0 | 3 |
| ✅ nitro-node | 132 | 0 | 28 |
| ❌ nitro-quickjs | 130 | 2 | 28 |
| ✅ nuxt-node | 132 | 0 | 28 |
| ❌ nuxt-quickjs | 130 | 2 | 28 |
| ❌ python-node | 0 | 8 | 152 |
| ❌ sveltekit-node | 150 | 1 | 9 |
| ❌ sveltekit-quickjs | 150 | 1 | 9 |
| ❌ tanstack-start-node | 130 | 2 | 28 |
| ✅ tanstack-start-quickjs | 132 | 0 | 28 |
| ❌ vite-node | 131 | 1 | 28 |
| ❌ vite-quickjs | 131 | 1 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 0 |
❌ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ python | 0 | 9 | 132 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 217526ms → this run 164269ms (Δ -53257ms, -24%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
| exit 1 | ||
| fi | ||
|
|
||
| wired-test-suites: |
There was a problem hiding this comment.
Can we get a better name?
There was a problem hiding this comment.
Renamed to no-unrun-tests / No Unrun Tests, matching the no-test-overrides / No Test Overrides sibling right above it — both are short negative invariants. The script moved to scripts/check-no-unrun-tests.mjs to match, and the step now reads "Check every test file is run by turbo test or a CI lane".
| '@workflow/core': patch | ||
| --- | ||
|
|
||
| Run `e2e/utils.test.ts` from the package's `test` script; it covers the e2e helpers themselves and needs no deployment, so nothing had been running it. |
There was a problem hiding this comment.
Maybe don't need two changeset entries just for internal test changes
There was a problem hiding this comment.
Dropped both. Nothing here changes shipped behaviour — the only edits to published packages are a test script and a vitest devDependency — so there is nothing to release. (No status check gates on the changeset, so this stays green.)
The PR only wires up existing suites and adds a CI check, so there is nothing to release. Rename the job to match its `no-test-overrides` sibling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
|
Backport to This is usually an infrastructure problem (e.g. the configured AI model could not be found, an AI Gateway error, or an opencode crash) rather than a merge conflict. Check the job logs linked above for details. Once the underlying issue is fixed, re-run the Backport to stable workflow manually via |
Fixes #3731.
turbo testruns a package's tests only if that package declares atestscript. A suite without one is skipped in silence — the files look like coverage, and the assertions inside them rot with nobody watching. #3731 found@workflow/worldin that state; checking the rest of the workspace found three more.What was unrun
@workflow/world@workflow/cli@workflow/nitropackages/core/e2e/utils.test.tspackages/core/e2e/manifest.test.tsAll 292 pass as-is, so this adds no red.
@workflow/world,@workflow/cliand@workflow/nitroget atestscript (vitest run src) and avitestdevDependency, matching every other package.The two
packages/core/e2efiles are a different shape of the same bug: that directory is run by dedicated workflow steps naming each file, and these two were named by none.e2e/utils.test.tstests the e2e helpers themselves and needs no deployment, so it joins the unit lane through@workflow/core'stestscript.e2e/manifest.test.tsasserts the shape of a built workbench manifest, so it gets a step in E2E Local Prod immediately afterlocal-build.test.ts— which is what runspnpm buildin the workbench app. It needs its own invocation rather than another file in that step, since vitest would otherwise run both concurrently, against a manifest that does not exist yet.The
spec-version.test.tsfailure in the issueAlready gone. #3634 rewrote that assertion when it moved the current version to 7 (sealed log), so the file now pins
SPEC_VERSION_CURRENT === SPEC_VERSION_SUPPORTS_SEALED_LOGrather than the stale compression version. Nothing to update here — but note that the fix landed blind, because the suite proving it still never ran.Keeping it wired
scripts/check-no-unrun-tests.mjswalks the workspace and requires every test file to be reachable one of two ways:testscript whose positional filters cover the file, soturbo testruns it — vitest treats positionals as filters, sovitest run srcbeside a test ine2e/drops that file quietly; orpackage.jsonscript, names the file (or a directory holding it) outright.Anything else fails. It runs as a new No Unrun Tests job in
lint.yml(node only, no install, alongside the other hygiene jobs). One waiver:@workflow/docs-typecheck, whose suite is far too slow for the unit lane and runs aspnpm test:docsfrom Docs Checks.Path matching is boundary-aware on purpose. A plain substring search reads
packages/worldas present in every workflow mentioningpackages/world-vercel, and a trailing/counts as continuation so that a lane namingpackages/core/e2e/local-build.test.tssays nothing about the other files in that directory. Without both rules the check passes on the very state it exists to catch.Verification
Each regression was re-introduced one at a time and the guard flagged it:
turbo testafter the fix:manifest.test.tswas checked against all eight workbench apps with manifests, built locally: 36 passed.CI confirms the suites actually execute rather than the job merely going green. Both unit lanes pass, and their logs show:
Windows matters here for two reasons, both now checked rather than assumed: nitro's assertions hard-code POSIX paths and hold because the source builds those specifiers with
pathe, and thee2e/utils.test.tspositional filter does match there.@workflow/world-postgresfails locally without Docker for testcontainers; it passes in CI (6 files / 193 tests).🤖 Generated with Claude Code