[world-vercel] Resume partial replay streams after validated events - #3546
[world-vercel] Resume partial replay streams after validated events#3546NathanColosimo wants to merge 10 commits into
Conversation
🦋 Changeset detectedLatest commit: c73a1d0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (234 failed)astro-node (7 failed):
astro-quickjs (11 failed):
example-node (11 failed):
example-quickjs (10 failed):
express-node (9 failed):
express-quickjs (11 failed):
fastify-node (6 failed):
fastify-quickjs (7 failed):
hono-node (6 failed):
hono-quickjs (7 failed):
nest-node (9 failed):
nest-quickjs (12 failed):
nextjs-turbopack-node (5 failed):
nextjs-turbopack-quickjs (8 failed):
nextjs-webpack-node (9 failed):
nextjs-webpack-quickjs (8 failed):
nitro-node (5 failed):
nitro-quickjs (12 failed):
nuxt-node (6 failed):
nuxt-quickjs (11 failed):
python-node (8 failed):
sveltekit-node (10 failed):
sveltekit-quickjs (13 failed):
tanstack-start-node (8 failed):
tanstack-start-quickjs (8 failed):
vite-node (7 failed):
vite-quickjs (10 failed):
🌐 Cross-language Conformance (9 failed)python (9 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ❌ ▲ Vercel Production | 3344 | 234 | 742 | 4320 |
| ✅ 💻 Local Development | 3922 | 0 | 558 | 4480 |
| ✅ 📦 Local Production | 3922 | 0 | 558 | 4480 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 558 | 4480 |
| ✅ 🪟 Windows | 320 | 0 | 0 | 320 |
| ❌ 🌐 Cross-language Conformance | 0 | 9 | 132 | 141 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| Total | 15457 | 243 | 2548 | 18248 |
Details by Category
❌ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ astro-node | 125 | 7 | 28 |
| ❌ astro-quickjs | 121 | 11 | 28 |
| ❌ example-node | 121 | 11 | 28 |
| ❌ example-quickjs | 122 | 10 | 28 |
| ❌ express-node | 123 | 9 | 28 |
| ❌ express-quickjs | 121 | 11 | 28 |
| ❌ fastify-node | 126 | 6 | 28 |
| ❌ fastify-quickjs | 125 | 7 | 28 |
| ❌ hono-node | 126 | 6 | 28 |
| ❌ hono-quickjs | 125 | 7 | 28 |
| ❌ nest-node | 123 | 9 | 28 |
| ❌ nest-quickjs | 120 | 12 | 28 |
| ❌ nextjs-turbopack-node | 152 | 5 | 3 |
| ❌ nextjs-turbopack-quickjs | 149 | 8 | 3 |
| ❌ nextjs-webpack-node | 148 | 9 | 3 |
| ❌ nextjs-webpack-quickjs | 149 | 8 | 3 |
| ❌ nitro-node | 127 | 5 | 28 |
| ❌ nitro-quickjs | 120 | 12 | 28 |
| ❌ nuxt-node | 126 | 6 | 28 |
| ❌ nuxt-quickjs | 121 | 11 | 28 |
| ❌ python-node | 0 | 8 | 152 |
| ❌ sveltekit-node | 141 | 10 | 9 |
| ❌ sveltekit-quickjs | 138 | 13 | 9 |
| ❌ tanstack-start-node | 124 | 8 | 28 |
| ❌ tanstack-start-quickjs | 124 | 8 | 28 |
| ❌ vite-node | 125 | 7 | 28 |
| ❌ vite-quickjs | 122 | 10 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 0 |
❌ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ python | 0 | 9 | 132 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 231540ms → this run 172562ms (Δ -58978ms, -25%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
e9e7488 to
dbdc9a2
Compare
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
VaguelySerious
left a comment
There was a problem hiding this comment.
LGTM, will let agent approve if no blockers found
VaguelySerious
left a comment
There was a problem hiding this comment.
AI review: blocking issues found
| params.limit !== undefined || | ||
| !lastEvent || | ||
| `eid:${lastEvent.eventId}` === cursor | ||
| retries === MAX_PARTIAL_EVENT_STREAM_RETRIES || |
There was a problem hiding this comment.
AI Review: Blocking
This loop now retries failures that made zero forward progress, with no backoff. Previously the guard was !lastEvent || \eid:${lastEvent.eventId}` === cursor, so a 5xx or transport failure arriving before the first frame threw immediately. Now the only gates are the retry counter, an explicit limit, and isRetryableEventRequestError— and that predicate returnstrueforcode === 'TRANSPORT'and for anystatus` in 500-599.
Measured on this branch vs origin/main, one getWorkflowRunEventsV4 call against an origin that only answers 503:
| wire requests | elapsed | |
|---|---|---|
main |
1 | - |
| this branch | 4 | 32 ms |
(undici MockAgent, .persist()ing a 503 on the LIST path, counting interceptor hits.)
So every event LIST — the hottest read path, one per replay — becomes a 4x immediate amplifier, and it amplifies precisely when the backend is already returning 5xx. There is no sleep between iterations, so the four requests leave in the same tick. Queue redelivery multiplies on top of that.
The truncated-stream resumption this PR is for only needs the retry when the cursor actually advanced. Two options: restore the forward-progress condition alongside the new counter, or keep the no-progress retry but require a backoff and restrict it to PartialEventStreamError/IncompleteFrameError rather than all of isRetryableEventRequestError.
Related: MAX_PARTIAL_EVENT_STREAM_RETRIES now also bounds whole-request retries that have nothing to do with a partial stream, so the name has drifted from what it gates.
| throw new Error(`v4 ${opName}: unexpected control frame`); | ||
| ): Promise<{ cursor: string | null; hasMore: boolean }> { | ||
| try { | ||
| for await (const frame of decodeFrames( |
There was a problem hiding this comment.
AI Review: Note
Dropping the materialized-response branch makes the client assume the server can never decline the replay stream. The backend still has a live decline path: when a hook_received carries resumeId + digest + the frame Accept but materialization returns no run state, it deliberately falls through to the CBOR mutation response and records workflow.hook_received_replay_stream_declined. The comment there is explicit that throwing would 500 a write that already succeeded and let each redelivery append another duplicate.
On that path this client now reaches eventFrameChunks, sees a non-frame content type, and throws a plain Error. That is not an IncompleteFrameError, so consumeEventFrameStream rethrows it unchanged, postReplayLogEvent does not resume it, and isRetryableEventPostError returns false — a hard failure after the write landed, which is the exact outcome the server-side fallback exists to avoid.
I checked how reachable it is: the decline requires a persisted pre-event-sourcing spec version, and there have been no hook_received events below specVersion 2 in the last 30 days, so it is dormant today. The concern is the coupling rather than current traffic — nothing in either repo records that the server may no longer decline, so the next decline reason added server-side breaks every deployed SDK carrying this change (and skew protection keeps those alive a long time). Worth either a comment here naming the contract or keeping a narrow fallback.
| get: (runId, eventId, params) => getEvent(runId, eventId, params, config), | ||
| list: (params) => getWorkflowRunEvents(params, config), | ||
| listByCorrelationId: (params) => getWorkflowRunEvents(params, config), | ||
| listByCorrelationId: (params) => |
There was a problem hiding this comment.
AI Review: Note
This is a real bug fix hiding in a transport PR: listByCorrelationId was routed to getWorkflowRunEvents, so it ignored correlationId entirely and returned the whole run's event log. It now reaches the correlation-id endpoint.
It is unmentioned in the PR body and in the changeset (which only describes stream resumption), and no test covers it — listByCorrelationId appears nowhere under packages/world-vercel/src/*.test.ts. Given the correlation-id-per-run scoping this path exists for, a test asserting the request carries correlationId and that the response is filtered to params.runId would be worth having, plus a changeset line so it shows up in the changelog.
Summary
hasMore: trueresultV4Response,reportTransportOutcome, and the complete/partial result state machineryProtocol contract
returnAll=trueasks workflow-server to drain storage pagination, but the server can still return a cleanhasMore: truesentinel when the configured event ceiling is reached. That is a valid bounded response, not an interrupted transport, so this PR returns it unchanged. Only a body that ends without a complete sentinel is resumed.The lazy-hook preload path requires a complete replay snapshot and intentionally rejects a bounded preload so the runtime can fall back to its normal replay-loading path.
Review scope
This PR owns transport recovery only. It does not introduce an event observer or prepare payloads. The documented
onEventAPI is introduced in #3548, where it is first consumed.Stack
Validation
pnpm --filter @workflow/world-vercel... buildpnpm --filter @workflow/world-vercel test— 542 tests passedgit diff --checkgpt-5.6-sol, xhigh): clean, no accepted/actionable P0–P2 findings; correctness confidence 0.91