[core] Simplify replay payload preparation - #3547
Conversation
🦋 Changeset detectedLatest commit: 84b7a5a The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (230 failed)astro-node (6 failed):
astro-quickjs (7 failed):
example-node (10 failed):
example-quickjs (14 failed):
express-node (11 failed):
express-quickjs (10 failed):
fastify-node (8 failed):
fastify-quickjs (11 failed):
hono-node (10 failed):
hono-quickjs (8 failed):
nest-node (8 failed):
nest-quickjs (12 failed):
nextjs-turbopack-node (3 failed):
nextjs-turbopack-quickjs (7 failed):
nextjs-webpack-node (7 failed):
nextjs-webpack-quickjs (10 failed):
nitro-node (7 failed):
nitro-quickjs (11 failed):
nuxt-node (11 failed):
nuxt-quickjs (3 failed):
python-node (8 failed):
sveltekit-node (7 failed):
sveltekit-quickjs (9 failed):
tanstack-start-node (5 failed):
tanstack-start-quickjs (9 failed):
vite-node (7 failed):
vite-quickjs (11 failed):
🌐 Cross-language Conformance (9 failed)python (9 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ❌ ▲ Vercel Production | 3348 | 230 | 742 | 4320 |
| ✅ 💻 Local Development | 3922 | 0 | 558 | 4480 |
| ✅ 📦 Local Production | 3922 | 0 | 558 | 4480 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 558 | 4480 |
| ✅ 🪟 Windows | 320 | 0 | 0 | 320 |
| ❌ 🌐 Cross-language Conformance | 0 | 9 | 132 | 141 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| Total | 15461 | 239 | 2548 | 18248 |
Details by Category
❌ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ astro-node | 126 | 6 | 28 |
| ❌ astro-quickjs | 125 | 7 | 28 |
| ❌ example-node | 122 | 10 | 28 |
| ❌ example-quickjs | 118 | 14 | 28 |
| ❌ express-node | 121 | 11 | 28 |
| ❌ express-quickjs | 122 | 10 | 28 |
| ❌ fastify-node | 124 | 8 | 28 |
| ❌ fastify-quickjs | 121 | 11 | 28 |
| ❌ hono-node | 122 | 10 | 28 |
| ❌ hono-quickjs | 124 | 8 | 28 |
| ❌ nest-node | 124 | 8 | 28 |
| ❌ nest-quickjs | 120 | 12 | 28 |
| ❌ nextjs-turbopack-node | 154 | 3 | 3 |
| ❌ nextjs-turbopack-quickjs | 150 | 7 | 3 |
| ❌ nextjs-webpack-node | 150 | 7 | 3 |
| ❌ nextjs-webpack-quickjs | 147 | 10 | 3 |
| ❌ nitro-node | 125 | 7 | 28 |
| ❌ nitro-quickjs | 121 | 11 | 28 |
| ❌ nuxt-node | 121 | 11 | 28 |
| ❌ nuxt-quickjs | 129 | 3 | 28 |
| ❌ python-node | 0 | 8 | 152 |
| ❌ sveltekit-node | 144 | 7 | 9 |
| ❌ sveltekit-quickjs | 142 | 9 | 9 |
| ❌ tanstack-start-node | 127 | 5 | 28 |
| ❌ tanstack-start-quickjs | 123 | 9 | 28 |
| ❌ vite-node | 125 | 7 | 28 |
| ❌ vite-quickjs | 121 | 11 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 0 |
❌ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ python | 0 | 9 | 132 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
84e27c4 to
1e92acf
Compare
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
📊 Workflow Benchmarks❌ The benchmark run for commit Backend:
Streams
📜 Previous results (2)e0181d4Thu, 20 Aug 2026 18:00:09 GMT · run logs
Streams
2e2fa27Wed, 19 Aug 2026 17:27:53 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
VaguelySerious
left a comment
There was a problem hiding this comment.
AI review: no blocking issues
| } | ||
|
|
||
| async function decompressGzip(payload: Uint8Array): Promise<Uint8Array> { | ||
| if (nativeGzip) return nativeGzip.decompress(payload); |
There was a problem hiding this comment.
AI Review: Note
The gzip read path silently switches from the streaming codec to a blocking synchronous one. main's decompress always went through gunzipBytes → pipeThroughTransform(data, new DecompressionStream('gzip')); this prefers nativeGzip.decompress, i.e. zlib.gunzipSync, whenever process.getBuiltinModule resolves — Node 22.3+, so effectively every current runtime.
That matters because of what sits above it in this stack. #3548 starts preparation for every binary payload in the log as frames arrive, so a replay of a long run now runs N gunzipSync calls. Each is its own microtask, but the inflation inside each one blocks the event loop, where previously the work was spread across stream pumps. Any run still carrying gzip-prefixed payloads (written by a pre-zstd runtime, or with WORKFLOW_COMPRESSION_CODEC=gzip) gets that as a synchronous burst on the TTFS path.
The write path has the same shape at line 163 but a smaller blast radius: zstd wins selectWriteCodec on Node 22.15+, so sync gzip writes only affect Node 22.3–22.14 and the explicit gzip override. Note the zstd write path was already zstdCompressSync on main, so that half is not new.
Neither change is mentioned in the PR body or the changeset. If preferring native is deliberate for the latency win, it is worth saying so and confirming the sync burst is acceptable; if it is incidental to the extraction, keeping DecompressionStream for gzip would preserve the old behaviour.
I checked the things that could have gone wrong here and they are fine: ZSTD_LEVEL = 3 still reaches zstdCompressSync via the ZSTD_c_compressionLevel param mapping, canCompressGzip and the gzip/decompressGzip fallbacks agree so selectWriteCodec cannot pick a codec it then fails to run, and process.getBuiltinModule is the same resolution mechanism main used, so Node 18/20 keep the portable path rather than losing compression.
VaguelySerious
left a comment
There was a problem hiding this comment.
AI review: no blocking issues (re-review of the new fix(world): preserve omitted event payloads commit; the rest of the series is unchanged from my last review)
| * stored event so the strict serialized-data schema can still distinguish an | ||
| * omitted read payload from a malformed create request. | ||
| */ | ||
| function restoreOmittedEventPayload(value: unknown): unknown { |
There was a problem hiding this comment.
AI Review: Note
I verified the underlying break is real, and it is narrower than the fix. Parsing the four eventData shapes an empty-body v4 frame can present at decodeEventFrame's EventSchema.parse(meta):
| frame meta shape | before this commit | after |
|---|---|---|
step_completed, eventData absent |
REJECTS (eventData) |
PARSES |
step_completed, eventData: {}, no result |
PARSES | PARSES |
run_started, eventData: {}, no input |
PARSES | PARSES |
hook_received, eventData without payload |
PARSES | PARSES |
So only the eventData entirely absent shape was rejecting — a missing payload key inside a present eventData already parsed. That one shape is exactly what stripEventDataRefs emits, since it deletes the ref fields and then drops eventData when nothing is left. Worth fixing.
The concern is that the preprocessor is much wider than the shape it repairs. It injects the payload key into every payload-bearing event, so a run_created that previously parsed to eventData keys [deploymentId, workflowName] now parses to [deploymentId, workflowName, input], and 'input' in eventData flips false → true. Your own test codifies that with toHaveProperty('input', undefined). Since only the absent-eventData case needed help, making eventData optional (or .default({})) on the affected schemas would fix it without changing the parsed shape of every other event.
That shape change is observable past the parse boundary — cbor-x encodes an explicit undefined as CBOR f7 rather than omitting the key:
encode({x:1}) -> b90001617801
encode({x:1, input:undefined}) -> b9000261780165696e707574f7
decode(...) keys -> ['x', 'input']
So a parsed event that is ever re-encoded carries a key it did not have before. I looked for somewhere that bites and did not find one — the two Object.keys(eventData) sites are both safe (stripEventDataRefs uses delete, so its length check still sees 0; analyticsEventToEvent builds eventData from scratch and never parses). Flagging it because it is the kind of thing that surfaces later at a wire boundary rather than here.
Secondary: the preprocessor runs on every event parse and allocates two objects per event on the injection path, which is the resolveData: 'none' bulk-listing path — the one that exists specifically to avoid per-payload work, and the one #3548 now drives per frame.
One more thing worth separating out: origin/main rejects the same absent-eventData shape (verified against 5a59bb82e8). If a live read path re-parses a stripped event, that is a production bug being repaired as the 24th commit of a refactor PR three levels deep in a stack, where it will never be found again. I could not confirm such a path — world-postgres and world-local both parse then strip, so the stripped shape is not re-validated in-process — but if you did hit this for real, it deserves its own PR against main with the reproducing case.
| events: z.undefined(), | ||
| cursor: z.undefined(), | ||
| hasMore: z.undefined(), | ||
| // Materialized writes omit page metadata entirely. Zod 4.4 correctly |
There was a problem hiding this comment.
AI Review: Nit
This comment is the most useful sentence in the commit and it is on the wrong change — the z.undefined()-is-required behaviour is what motivates the EventSchema preprocessor too, but that hunk explains itself only as "worlds may omit a payload field" with no mention of the Zod version bump that changed the rule.
Worth repeating the Zod 4.4 rationale at restoreOmittedEventPayload, and worth a line in the changeset: the schema-loosening half is user-visible for anyone validating events with the exported EventSchema, and the current changeset text ("Simplify payload codecs and reuse prepared replay payloads across workflow VMs") does not hint at it.
Summary
serialization/replay.tsforwarding module; the runtime cache calls the real payload decoder directly@workflow/worldinstead of maintaining parallel implementations in core, CLI, and world-vercelThe production TypeScript diff is net 55 fewer lines. Most of the added lines are shared codec owners and regression tests replacing duplicated implementations in their consumers.
Why Web Crypto
The native Node AES path added substantial runtime detection, key-handle bookkeeping, and separate error behavior for a measured saving of only about 0.5 ms per 100 encrypted events. That tradeoff is not worthwhile in this PR.
importKey,encrypt, anddecryptremain the low-level implementation boundary. A later native or WASM implementation can replace that layer without changing serialization, replay caching, or callers.Replay boundary
decodePayloadperforms host-owned decrypt → decompress work.ReplayPayloadCachestarts that work once per binary event and retains the prepared bytes for the invocation.deserializePreparedReplayPayloadhydrates those bytes inside each workflow VM, producing a fresh realm-owned object graph on every replay.Compression telemetry is recorded when the cache performs preparation. Buffer-backed slices are copied into plain owned
Uint8Arrays before retention so pooled backing allocations are not retained accidentally.Compatibility
isEncrypted()remains the symmetricencrpredicate;isEncryptedData()matches bothencrand sealedencpenvelopesReadableStream.getReader()rather than async iteration, preserving Safari supportzstdCompressSyncStack
Validation
@workflow/core: 101 passed files, 1 skipped; 2,209 passed tests, 3 expected failures, 1 skipped@workflow/world-vercel: 26 passed files, 550 passed tests@workflow/web-shared: 24 passed files, 203 passed tests