[core] Prepare replay payloads as event frames arrive - #3548
[core] Prepare replay payloads as event frames arrive#3548NathanColosimo wants to merge 16 commits into
Conversation
🦋 Changeset detectedLatest commit: 13d4912 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (237 failed)astro-node (7 failed):
astro-quickjs (9 failed):
example-node (8 failed):
example-quickjs (9 failed):
express-node (9 failed):
express-quickjs (9 failed):
fastify-node (6 failed):
fastify-quickjs (11 failed):
hono-node (8 failed):
hono-quickjs (13 failed):
nest-node (6 failed):
nest-quickjs (10 failed):
nextjs-turbopack-node (5 failed):
nextjs-turbopack-quickjs (7 failed):
nextjs-webpack-node (6 failed):
nextjs-webpack-quickjs (5 failed):
nitro-node (9 failed):
nitro-quickjs (10 failed):
nuxt-node (9 failed):
nuxt-quickjs (13 failed):
python-node (8 failed):
sveltekit-node (8 failed):
sveltekit-quickjs (8 failed):
tanstack-start-node (11 failed):
tanstack-start-quickjs (13 failed):
vite-node (9 failed):
vite-quickjs (11 failed):
🌐 Cross-language Conformance (9 failed)python (9 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ❌ ▲ Vercel Production | 3341 | 237 | 742 | 4320 |
| ✅ 💻 Local Development | 3922 | 0 | 558 | 4480 |
| ✅ 📦 Local Production | 3922 | 0 | 558 | 4480 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 558 | 4480 |
| ✅ 🪟 Windows | 320 | 0 | 0 | 320 |
| ❌ 🌐 Cross-language Conformance | 0 | 9 | 132 | 141 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| Total | 15454 | 246 | 2548 | 18248 |
Details by Category
❌ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ astro-node | 125 | 7 | 28 |
| ❌ astro-quickjs | 123 | 9 | 28 |
| ❌ example-node | 124 | 8 | 28 |
| ❌ example-quickjs | 123 | 9 | 28 |
| ❌ express-node | 123 | 9 | 28 |
| ❌ express-quickjs | 123 | 9 | 28 |
| ❌ fastify-node | 126 | 6 | 28 |
| ❌ fastify-quickjs | 121 | 11 | 28 |
| ❌ hono-node | 124 | 8 | 28 |
| ❌ hono-quickjs | 119 | 13 | 28 |
| ❌ nest-node | 126 | 6 | 28 |
| ❌ nest-quickjs | 122 | 10 | 28 |
| ❌ nextjs-turbopack-node | 152 | 5 | 3 |
| ❌ nextjs-turbopack-quickjs | 150 | 7 | 3 |
| ❌ nextjs-webpack-node | 151 | 6 | 3 |
| ❌ nextjs-webpack-quickjs | 152 | 5 | 3 |
| ❌ nitro-node | 123 | 9 | 28 |
| ❌ nitro-quickjs | 122 | 10 | 28 |
| ❌ nuxt-node | 123 | 9 | 28 |
| ❌ nuxt-quickjs | 119 | 13 | 28 |
| ❌ python-node | 0 | 8 | 152 |
| ❌ sveltekit-node | 143 | 8 | 9 |
| ❌ sveltekit-quickjs | 143 | 8 | 9 |
| ❌ tanstack-start-node | 121 | 11 | 28 |
| ❌ tanstack-start-quickjs | 119 | 13 | 28 |
| ❌ vite-node | 123 | 9 | 28 |
| ❌ vite-quickjs | 121 | 11 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 0 |
❌ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ python | 0 | 9 | 132 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
388b6ea to
294d4b7
Compare
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
📊 Workflow Benchmarks❌ The benchmark run for commit Backend:
Streams
📜 Previous results (2)1460540Thu, 20 Aug 2026 18:03:52 GMT · run logs
Streams
c5d8448Wed, 19 Aug 2026 17:29:47 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
7471e77 to
184d506
Compare
VaguelySerious
left a comment
There was a problem hiding this comment.
AI review: blocking issues found
|
|
||
| function isMemoizablePrimitive(value: unknown): boolean { | ||
| if (value === null) return true; | ||
| function isCacheablePrimitive(value: unknown): boolean { |
There was a problem hiding this comment.
AI Review: Blocking
This drops the size bound that main deliberately had, and widens where memoization applies at the same time.
main's predicate was isMemoizablePrimitive with MAX_MEMOIZED_PRIMITIVE_LENGTH = 4096, capping both strings and bigints, and its comment stated the intent: "Objects and large strings/bigints always run hydrate again, producing a fresh VM-specific value from the separately cached prepared payload." isCacheablePrimitive has no length check, so every string is now retained regardless of size.
Verified with a 1 MiB string result, counting hydrate invocations across two replays of the same log:
| hydrate calls | meaning | |
|---|---|---|
main |
2 | re-hydrated per replay, not retained |
| this branch | 1 | retained for the invocation |
(main's small-string control memoizes at 1 call, so the cap was doing real work, not dead code.)
The retention is additive, not a substitution. getEventValue calls getPayload, which populates preparations[eventId] with the prepared plaintext bytes and never deletes them on success, and then cachePrimitive stores the hydrated value in primitiveValues[eventId]. Both live for the whole invocation, so a large-payload run holds the bytes and the UTF-16 string — roughly double the peak for that payload class.
The surface also grew from one call site to five. On main primitive memoization was reachable only through getStepResult at step.ts:304. getEventValue is now called from step.ts:193 (step errors), step.ts:304 (step results), hook.ts:355, hook.ts:427 (hook payloads) and abort-controller.ts:233. Hook payloads are webhook bodies, which is exactly the large-and-unbounded case the 4096 cap was protecting against.
If removing the cap is intentional, it needs to be stated with the memory reasoning, since the retention is now unbounded in payload size and the run's step count. Otherwise carry MAX_MEMOIZED_PRIMITIVE_LENGTH across.
| * replaying the accepted prefix. Throwing aborts the operation and is never | ||
| * treated as a transport failure or retried. | ||
| */ | ||
| onEvent?: (event: Event) => void; |
There was a problem hiding this comment.
AI Review: Note
The observer plumbing itself holds up well — I traced every path and the invariants are tight: EventObserverError is excluded from withV4ResponseBody's transport accounting, rethrown ahead of the LIST retry decision, rethrown rather than wrapped in EventPostResponseError by the continuation catch, and unwrapped to the caller's original throw at both public boundaries. events.push sits directly before onEvent, so events and observations stay in lockstep and the eid: continuation cursor observes each accepted event exactly once.
Two contract gaps:
onEventlands onListEventsParamsandCreateEventParamsbut notListEventsByCorrelationIdParams, which is the odd one out of the three list-shaped params in this file.- The doc does not say a world may ignore it. It reads as mandatory to anyone implementing the
Worldinterface, but the runtime'sprepareAllfull-scan fallback exists precisely so worlds that cannot observe streamed frames stay correct. Worth saying so here — that fallback is the reason this is optional, and it is not discoverable from the interface.
Summary
onEventobserver on event create/list operations, where streamed replay preparation first consumes itObserver contract
onEventruns synchronously after validation and before the next frame is consumed. Throwing aborts the operation; it is not a transport failure and is never retried. If an interrupted stream resumes, continuation starts after the last observed event, so every accepted event is observed exactly once.The callback itself is synchronous, but payload preparation may be asynchronous (the current codec stack uses Web Crypto). This PR overlaps that work with later frame delivery and VM setup; ordered consumers await only the payloads they actually reach.
Performance signal
The latest benchmark and previous behavior run were recorded on the original cumulative branch before this review split. Across those runs, inline STSO improved 4–20% at p75, 10–24% at p90, and 39–51% at p99; total 1020-step workflow overhead improved 7–23%.
Cold TTFS regressed in both runs, including the turbo control that does not preload or prepare replay events, so that signal is dominated by deployment cold-start variability. The benchmarked branch also contained the small tracing layer in #3523.
Stack
Validation
pnpm --filter @workflow/core... buildpnpm --filter @workflow/world-vercel... buildgit diff --checkgpt-5.6-sol, xhigh): clean, no actionable P0–P2 findings; correctness confidence 0.88