[core] Fail dev HMR cleanup on a stranded step registration - #3682
[core] Fail dev HMR cleanup on a stranded step registration#3682VaguelySerious wants to merge 1 commit into
Conversation
`dev.test.ts` fixtures that declare a `'use step'` function are imported by path from the generated `__step_registrations.js`, so deleting one only stops breaking the flow route once a rediscovery regenerates that file. When the dev server's watcher drops the unlink, the generated file keeps importing a path that no longer exists: the flow route stops compiling and every later workflow dispatch in the job gets a 500 that names a fixture from a test that already passed. Three of the last 120 `Tests` runs lost the whole ~30-minute Windows e2e window this way, each with ~10k flow-route 500s and no successful dispatch. Cleanup now waits for the generated step registrations to drop every file it deleted. If they still reference one when the budget expires, the fixture is written back so the shared dev server keeps serving the rest of the suite, and the hook fails naming the stranded path. The wait runs before the node_modules fixture-package removal so a restored fixture still resolves its import. The pre-e2e probe moves from `GET /api/chat` to `POST /.well-known/workflow/v1/flow?__health`. /api/chat compiles independently of the flow route, so it answered 405 in all three jobs while every dispatch was getting a 500. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 62fcd0f The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3474 | 0 | 738 | 4212 |
| ✅ 💻 Local Development | 3810 | 0 | 558 | 4368 |
| ✅ 📦 Local Production | 3810 | 0 | 558 | 4368 |
| ✅ 🐘 Local Postgres | 3810 | 0 | 558 | 4368 |
| ✅ 🪟 Windows | 312 | 0 | 0 | 312 |
| ✅ 🌐 Cross-language Conformance | 9 | 0 | 128 | 137 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| Total | 15252 | 0 | 2540 | 17792 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 128 | 0 | 28 |
| ✅ astro-quickjs | 128 | 0 | 28 |
| ✅ example-node | 128 | 0 | 28 |
| ✅ example-quickjs | 128 | 0 | 28 |
| ✅ express-node | 128 | 0 | 28 |
| ✅ express-quickjs | 128 | 0 | 28 |
| ✅ fastify-node | 128 | 0 | 28 |
| ✅ fastify-quickjs | 128 | 0 | 28 |
| ✅ hono-node | 128 | 0 | 28 |
| ✅ hono-quickjs | 128 | 0 | 28 |
| ✅ nest-node | 128 | 0 | 28 |
| ✅ nest-quickjs | 128 | 0 | 28 |
| ✅ nextjs-turbopack-node | 153 | 0 | 3 |
| ✅ nextjs-turbopack-quickjs | 153 | 0 | 3 |
| ✅ nextjs-webpack-node | 153 | 0 | 3 |
| ✅ nextjs-webpack-quickjs | 153 | 0 | 3 |
| ✅ nitro-node | 128 | 0 | 28 |
| ✅ nitro-quickjs | 128 | 0 | 28 |
| ✅ nuxt-node | 128 | 0 | 28 |
| ✅ nuxt-quickjs | 128 | 0 | 28 |
| ✅ python-node | 8 | 0 | 148 |
| ✅ sveltekit-node | 147 | 0 | 9 |
| ✅ sveltekit-quickjs | 147 | 0 | 9 |
| ✅ tanstack-start-node | 128 | 0 | 28 |
| ✅ tanstack-start-quickjs | 128 | 0 | 28 |
| ✅ vite-node | 128 | 0 | 28 |
| ✅ vite-quickjs | 128 | 0 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 130 | 0 | 26 |
| ✅ astro-stable-quickjs | 130 | 0 | 26 |
| ✅ express-stable-node | 130 | 0 | 26 |
| ✅ express-stable-quickjs | 130 | 0 | 26 |
| ✅ fastify-stable-node | 130 | 0 | 26 |
| ✅ fastify-stable-quickjs | 130 | 0 | 26 |
| ✅ hono-stable-node | 130 | 0 | 26 |
| ✅ hono-stable-quickjs | 130 | 0 | 26 |
| ✅ nest-stable-node | 130 | 0 | 26 |
| ✅ nest-stable-quickjs | 130 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 156 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 156 | 0 | 0 |
| ✅ nitro-stable-node | 130 | 0 | 26 |
| ✅ nitro-stable-quickjs | 130 | 0 | 26 |
| ✅ nuxt-stable-node | 130 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 130 | 0 | 26 |
| ✅ sveltekit-stable-node | 149 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 149 | 0 | 7 |
| ✅ tanstack-start-node | 130 | 0 | 26 |
| ✅ tanstack-start-quickjs | 130 | 0 | 26 |
| ✅ vite-stable-node | 130 | 0 | 26 |
| ✅ vite-stable-quickjs | 130 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 130 | 0 | 26 |
| ✅ astro-stable-quickjs | 130 | 0 | 26 |
| ✅ express-stable-node | 130 | 0 | 26 |
| ✅ express-stable-quickjs | 130 | 0 | 26 |
| ✅ fastify-stable-node | 130 | 0 | 26 |
| ✅ fastify-stable-quickjs | 130 | 0 | 26 |
| ✅ hono-stable-node | 130 | 0 | 26 |
| ✅ hono-stable-quickjs | 130 | 0 | 26 |
| ✅ nest-stable-node | 130 | 0 | 26 |
| ✅ nest-stable-quickjs | 130 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 156 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 156 | 0 | 0 |
| ✅ nitro-stable-node | 130 | 0 | 26 |
| ✅ nitro-stable-quickjs | 130 | 0 | 26 |
| ✅ nuxt-stable-node | 130 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 130 | 0 | 26 |
| ✅ sveltekit-stable-node | 149 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 149 | 0 | 7 |
| ✅ tanstack-start-node | 130 | 0 | 26 |
| ✅ tanstack-start-quickjs | 130 | 0 | 26 |
| ✅ vite-stable-node | 130 | 0 | 26 |
| ✅ vite-stable-quickjs | 130 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 130 | 0 | 26 |
| ✅ astro-stable-quickjs | 130 | 0 | 26 |
| ✅ express-stable-node | 130 | 0 | 26 |
| ✅ express-stable-quickjs | 130 | 0 | 26 |
| ✅ fastify-stable-node | 130 | 0 | 26 |
| ✅ fastify-stable-quickjs | 130 | 0 | 26 |
| ✅ hono-stable-node | 130 | 0 | 26 |
| ✅ hono-stable-quickjs | 130 | 0 | 26 |
| ✅ nest-stable-node | 130 | 0 | 26 |
| ✅ nest-stable-quickjs | 130 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 156 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 156 | 0 | 0 |
| ✅ nitro-stable-node | 130 | 0 | 26 |
| ✅ nitro-stable-quickjs | 130 | 0 | 26 |
| ✅ nuxt-stable-node | 130 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 130 | 0 | 26 |
| ✅ sveltekit-stable-node | 149 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 149 | 0 | 7 |
| ✅ tanstack-start-node | 130 | 0 | 26 |
| ✅ tanstack-start-quickjs | 130 | 0 | 26 |
| ✅ vite-stable-node | 130 | 0 | 26 |
| ✅ vite-stable-quickjs | 130 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 156 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 156 | 0 | 0 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 9 | 0 | 128 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 236286ms → this run 157785ms (Δ -78501ms, -33%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
Problem
Three of the last 120
Testsruns hadE2E Windows Tests (node)burn its full ~30-minutewindow and get cancelled with every e2e test failing on
[world-local] Queue message failed (HTTP 500), where thehandlerErroris a Next.js_errorpage rather than anything workflow-shaped:The flow route never answered a single request successfully in any of them. The dev server
log says why:
dev.test.ts> "should not log source map warnings for workflow node_modules imports"writes
workflows/source-map-warning-fixture.ts, which declares a'use step'function,and its
afterEachdeletes it. The generated__step_registrations.jsimports everydiscovered step file by path, so that deletion only stops breaking the flow route once a
rediscovery regenerates the file. When the watcher drops the unlink — which is what happens
on Windows — the generated file keeps importing a path that no longer exists, the flow route
stops compiling, and every workflow dispatch for the rest of the job gets a 500 pointing at
a fixture from a test that already passed.
The HMR log confirms nothing recovers it:
164 → 165 → 166 → 165 → 165workflows, thenthree
skips and no further rediscovery.afterEach'sprewarm()only waits forworkflow dev hmr: ready, printed once at startup, so nothing ever waited for thepost-deletion rediscovery in the first place.
The generated workflow bundle inlines workflow sources instead of importing them, so only
step-declaring fixtures can strand it. That is why
new-workflow.tsleaks harmlessly inthe same runs while the source-map fixture is fatal.
The existing fail-fast guard cannot see this
The Windows job already probes the dev server before the long e2e run, precisely to avoid
burning the job window on a wedged server. It probes
GET /api/chat, which compilesindependently of the flow route, so it reported
-> 405and let the suite run in all threejobs.
Changes
packages/core/e2e/dev.test.ts— after deleting fixtures, cleanup waits for thegenerated step registrations to stop importing them. If they still do when the budget
runs out, the fixture contents are written back so the shared dev server keeps serving
the rest of the suite, and the hook fails naming the stranded file. CI already skips the
e2e suite when
dev.test.tsfails, so this turns a 30-minute burn with ~10k misleading500s into an immediate, accurate failure. The convergence wait runs before the
node_modulesfixture-package removal so a restored fixture still resolves its import..github/workflows/tests.yml— the pre-e2e probe now POSTs/.well-known/workflow/v1/flow?__health, the SDK's own health endpoint on the routethat actually breaks. It returns 200 on a healthy dev server and would have returned 500
in all three jobs above.
Validation
Run locally against
workbench/nextjs-turbopackon macOS (all 9 dev tests, so includingthe flow-route HMR fuzz cases that also delete step-declaring files):
deletion converges in ~2s, so the watcher path itself is sound and the Windows miss is the
discriminator.
should follow Next flow-route HMR rebuild rules for body-only changesfails on macOS(
expected 2 to be 1on an exact HMR log count). It fails identically with unmodifiedorigin/main'sdev.test.tsagainst the same dev server, so it is pre-existing andunrelated. That case is skipped on Windows.
with the stranded path named, the fixture is restored, and
POST /.well-known/workflow/v1/flow?__healthstill answers 200.POST /.well-known/workflow/v1/flow?__healthreturns 200 on a healthy dev server and 500when a path reachable from
__step_registrations.jsdoes not resolve, whileGET /api/chatkeeps answering 405 — the asymmetry the second change fixes.The underlying Windows watcher miss is worth chasing separately; this PR stops it from
costing a whole job and makes it name itself.
Note on a related hole, not changed here
The Linux
e2e-local-devlane runspnpm vitest run packages/core/e2e/dev.test.ts; sleep 10and then continues into the full e2e suite, so a
dev.test.tsfailure there is discarded.That lane would swallow the new hook failure too. Tightening it would newly red the lane on
pre-existing
dev.test.tsflakes (the flow-route HMR fuzz case already fails on macOSagainst unmodified
main, see below), so it seemed worth raising separately rather thanbundling in.