fix(world-vercel): hold the ws channel registry on globalThis - #3699
fix(world-vercel): hold the ws channel registry on globalThis#3699shalabhc wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 3d940b8 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (235 failed)astro-node (6 failed):
astro-quickjs (9 failed):
example-node (9 failed):
example-quickjs (9 failed):
express-node (8 failed):
express-quickjs (9 failed):
fastify-node (11 failed):
fastify-quickjs (11 failed):
hono-node (8 failed):
hono-quickjs (7 failed):
nest-node (12 failed):
nest-quickjs (9 failed):
nextjs-turbopack-node (4 failed):
nextjs-turbopack-quickjs (9 failed):
nextjs-webpack-node (10 failed):
nextjs-webpack-quickjs (9 failed):
nitro-node (9 failed):
nitro-quickjs (9 failed):
nuxt-node (11 failed):
nuxt-quickjs (9 failed):
sveltekit-node (9 failed):
sveltekit-quickjs (9 failed):
tanstack-start-node (10 failed):
tanstack-start-quickjs (9 failed):
vite-node (10 failed):
vite-quickjs (10 failed):
💻 Local Development (4 failed)nextjs-webpack-canary-node (1 failed):
nextjs-webpack-stable-node (2 failed):
nextjs-webpack-stable-quickjs (1 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ❌ ▲ Vercel Production | 3343 | 235 | 742 | 4320 |
| ❌ 💻 Local Development | 3918 | 4 | 558 | 4480 |
| ✅ 📦 Local Production | 3922 | 0 | 558 | 4480 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 558 | 4480 |
| ✅ 🪟 Windows | 320 | 0 | 0 | 320 |
| ✅ 🌐 Cross-language Conformance | 9 | 0 | 132 | 141 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| Total | 15461 | 239 | 2548 | 18248 |
Details by Category
❌ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ astro-node | 126 | 6 | 28 |
| ❌ astro-quickjs | 123 | 9 | 28 |
| ❌ example-node | 123 | 9 | 28 |
| ❌ example-quickjs | 123 | 9 | 28 |
| ❌ express-node | 124 | 8 | 28 |
| ❌ express-quickjs | 123 | 9 | 28 |
| ❌ fastify-node | 121 | 11 | 28 |
| ❌ fastify-quickjs | 121 | 11 | 28 |
| ❌ hono-node | 124 | 8 | 28 |
| ❌ hono-quickjs | 125 | 7 | 28 |
| ❌ nest-node | 120 | 12 | 28 |
| ❌ nest-quickjs | 123 | 9 | 28 |
| ❌ nextjs-turbopack-node | 153 | 4 | 3 |
| ❌ nextjs-turbopack-quickjs | 148 | 9 | 3 |
| ❌ nextjs-webpack-node | 147 | 10 | 3 |
| ❌ nextjs-webpack-quickjs | 148 | 9 | 3 |
| ❌ nitro-node | 123 | 9 | 28 |
| ❌ nitro-quickjs | 123 | 9 | 28 |
| ❌ nuxt-node | 121 | 11 | 28 |
| ❌ nuxt-quickjs | 123 | 9 | 28 |
| ✅ python-node | 8 | 0 | 152 |
| ❌ sveltekit-node | 142 | 9 | 9 |
| ❌ sveltekit-quickjs | 142 | 9 | 9 |
| ❌ tanstack-start-node | 122 | 10 | 28 |
| ❌ tanstack-start-quickjs | 123 | 9 | 28 |
| ❌ vite-node | 122 | 10 | 28 |
| ❌ vite-quickjs | 122 | 10 | 28 |
❌ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ❌ nextjs-webpack-canary-node | 140 | 1 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ❌ nextjs-webpack-stable-node | 158 | 2 | 0 |
| ❌ nextjs-webpack-stable-quickjs | 159 | 1 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 0 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 9 | 0 | 132 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
📊 Workflow Benchmarks❌ The benchmark run for commit Backend:
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
The WS events transport opens a channel from the queue consumer, which the runtime builds from `getWorldHandlers()`, and looks it up on the write path through `getWorld()` — two `createWorld()` calls cached under two different `globalThis` symbols in @workflow/core. The registry that joins them was a module-scope Map, so the handoff only worked while both calls ran in one module instance of this package. Since #3493 bundled the world into the Next.js server output instead of leaving it external, that is no longer guaranteed: Next gives the instrumentation entry its own copy of the module. An app whose `register()` warms the world (`await getWorld()`) therefore creates the events-writing World in the instrumentation copy and the queue consumer's World in the route entry, and every lookup reads an empty Map — sockets open and carry nothing, silently, for the life of the process. Move the registry and the two once-per-process log latches onto a `Symbol.for()` key on globalThis, the same technique core already uses for the World cache, so the transport is correct however many copies a bundler makes. Signed-off-by: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Co-Authored-By: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com> Signed-off-by: Shalabh Chaturvedi <7066873+shalabhc@users.noreply.github.com>
5e30b16 to
3d940b8
Compare
| '@workflow/world-vercel': patch | ||
| --- | ||
|
|
||
| Hold the WebSocket events transport's channel registry on `globalThis` instead of at module scope, so an app that ends up with two copies of the bundled world in one process (for example a Next.js app whose `instrumentation.ts` warms the world) still resolves the channel its queue consumer opened instead of silently writing every event over HTTP. |
| * registry at module scope, opens went into one Map and every write read the | ||
| * other, empty one: socket open, every event silently on HTTP. | ||
| * | ||
| * `/v1` so a later change to `WsEventsTransport`'s shape gets a fresh registry |
pranaygp
left a comment
There was a problem hiding this comment.
I suspect this is papering over a more global bug/regression from #3493 that's allowing duplicate instances of modules in the first place. maybe we can fix that instead of papering over the fix for websockets/transports specifically with this solution
Due to bundling the world-vercel (#3493), we can have two copies of the module which cause websockets to not work from durabench.
Note in SDK use from e2e tests I have seen websockets work just fine. It breaks on durabench because of it's instrumentation initialization.
This PR fixes the issue by moving
const transportsout of module state intoglobalThis.Details
## WhatThe WS events transport's channel registry moves from module scope onto a
Symbol.for()key onglobalThis— the same technique@workflow/corealreadyuses for the World cache — along with the two once-per-process log latches.
Why
The transport's open and its lookup are reached through two different World
instances:
openWsChannelruns inside world-vercel'screateQueueHandler, and theruntime builds that consumer from
getWorldHandlers();resolveWsTransport(a lookup, never a create) throughgetWorld().Those are two
createWorld()calls, cached under two differentglobalThissymbols in core. The only thing that made them meet was sharing one module
instance of this package, because
const transportswas module-scope.#3493 bundled
@workflow/world-vercelinto the Next.js server output instead ofleaving it in
serverExternalPackages. Next then gives theinstrumentation.tsentry its own copy of the module, separate from the route entries. So in an
app whose
register()warms the world:…the flow route's later
getWorldHandlers()builds the consumer's World in theroute copy. Opens land in one Map; every lookup reads the other, empty one.
The socket is opened and carries nothing — every event silently falls back to
HTTP, deterministically, for the life of the process. That is what
vercel-labs/durabench'sapps/workflow-hosthit: sockets in the trace, zeroevent frames on them.
Nothing about the WS design was wrong, and this is not a reason to revert #3493
(its cold-start win is real). A module-scope singleton in a package that a
bundler may duplicate is the bug.
Verification
packages/world-vercel: 532 unit tests pass, typecheck and Biome clean.Two new tests in
ws-transport.test.ts: the registry is the object onglobalThis(and is evicted on release), andresolveWsTransportresolves achannel another copy of this module registered — the regression this fixes.
Out-of-tree repro (minimal Next app, a node_modules package shaped like this
one, two
globalThis-cached worlds, one variable at a time), on Next 16.2.11and 16.3.1, webpack and Turbopack:
register()warms the worldhttp(two Maps,hit=false)wswswswswsWith the fix the two module instances still exist — the open logs one instance
id and the lookup another — but they share the registry, so the lookup hits.
Notes for review
/v1suffix so a future change toWsEventsTransport'sshape gets a fresh registry rather than a structurally-incompatible hit from an
older copy sharing the process. Cross-copy use is structural (
request,release,close); there is noinstanceofon the transport.when
resolveWsTransportmisses (the silence here cost severalinvestigations); a transport assertion in the
e2e-vercel-ws-transportlane,which today passes whether events go over WS or fall back; and re-combining
getWorld/getWorldHandlers, which the code comment inpackages/core/src/runtime/world.tsalready contemplates and which is whatturned "duplicated module" into a systematic miss rather than a race.
registerOTelin itsinstrumentation, which is why CI never saw this; adding a world warm-up there
would give the lane the failing shape.