Skip to content

[core] Simplify replay payload preparation - #3547

Open
NathanColosimo wants to merge 24 commits into
codex/replay-stream-recoveryfrom
codex/replay-sync-decode
Open

[core] Simplify replay payload preparation#3547
NathanColosimo wants to merge 24 commits into
codex/replay-stream-recoveryfrom
codex/replay-sync-decode

Conversation

@NathanColosimo

@NathanColosimo NathanColosimo commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • use one portable, asynchronous Web Crypto implementation for AES key import, encryption, and decryption in Node and browsers
  • reuse VM-independent decrypt/decompress results across fresh workflow VMs while still deserializing a fresh object graph inside each VM
  • remove the redundant serialization/replay.ts forwarding module; the runtime cache calls the real payload decoder directly
  • centralize persisted format-prefix and native gzip/zstd envelope handling in @workflow/world instead of maintaining parallel implementations in core, CLI, and world-vercel
  • pass the browser WASM zstd decoder explicitly through hydration options; the old registration API remains only as a deprecated compatibility fallback
  • keep existing public codec aliases and legacy non-binary pass-through signatures so this remains a non-breaking patch

The production TypeScript diff is net 55 fewer lines. Most of the added lines are shared codec owners and regression tests replacing duplicated implementations in their consumers.

Why Web Crypto

The native Node AES path added substantial runtime detection, key-handle bookkeeping, and separate error behavior for a measured saving of only about 0.5 ms per 100 encrypted events. That tradeoff is not worthwhile in this PR.

importKey, encrypt, and decrypt remain the low-level implementation boundary. A later native or WASM implementation can replace that layer without changing serialization, replay caching, or callers.

Replay boundary

  1. decodePayload performs host-owned decrypt → decompress work.
  2. ReplayPayloadCache starts that work once per binary event and retains the prepared bytes for the invocation.
  3. deserializePreparedReplayPayload hydrates those bytes inside each workflow VM, producing a fresh realm-owned object graph on every replay.

Compression telemetry is recorded when the cache performs preparation. Buffer-backed slices are copied into plain owned Uint8Arrays before retention so pooled backing allocations are not retained accidentally.

Compatibility

  • persisted envelope and encryption formats are unchanged
  • AES uses the same Web Crypto contract in every runtime
  • Node still uses native synchronous gzip/zstd when available; browser observability uses its lazy WASM zstd decoder
  • isEncrypted() remains the symmetric encr predicate; isEncryptedData() matches both encr and sealed encp envelopes
  • deprecated codec/replay compatibility exports remain available, but the runtime no longer depends on them
  • portable stream collection uses ReadableStream.getReader() rather than async iteration, preserving Safari support
  • the zstd fixture test skips cleanly on supported Node versions that do not expose zstdCompressSync

Stack

  1. [world-vercel] Resume partial replay streams after validated events #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — replay payload preparation simplification
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  6. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles

Validation

  • @workflow/core: 101 passed files, 1 skipped; 2,209 passed tests, 3 expected failures, 1 skipped
  • @workflow/world-vercel: 26 passed files, 550 passed tests
  • @workflow/web-shared: 24 passed files, 203 passed tests
  • typechecks: core, world-vercel, web-shared, CLI
  • focused encryption, replay-cache, compression, browser hydration, and serialized-data tests

@changeset-bot

changeset-bot Bot commented Aug 14, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 84b7a5a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages
Name Type
@workflow/core Patch
@workflow/cli Patch
@workflow/web-shared Patch
@workflow/world Patch
@workflow/world-vercel Patch
@workflow/builders Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/vitest Patch
@workflow/web Patch
workflow Patch
@workflow/world-testing Patch
@workflow/world-local Patch
@workflow/world-postgres Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/nuxt Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
example-nextjs-workflow-turbopack Ready Ready Preview, v0 Aug 21, 2026 8:55pm
example-nextjs-workflow-webpack Ready Ready Preview, v0 Aug 21, 2026 8:55pm
example-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-astro-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-express-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-fastify-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-hono-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-nestjs-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-nitro-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-nuxt-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-python-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-sveltekit-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-tanstack-start-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workbench-vite-workflow Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workflow-docs Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workflow-swc-playground Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workflow-tarballs Ready Ready Preview, v0 Aug 21, 2026 8:55pm
workflow-web Ready Ready Preview, v0 Aug 21, 2026 8:55pm

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (230 failed)

astro-node (6 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K1YKJ50GSGYQY3VYFFCY1E | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K1Z6HX0GSZA2FN6WY0XWQZ | 🔍 observability
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

astro-quickjs (7 failed):

  • webhookWorkflow | wrun_41M0K1ME5Q0GHHA44KMVNSF22E | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

example-node (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

example-quickjs (14 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

express-node (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

express-quickjs (10 failed):

  • webhookWorkflow | wrun_41M0K1ME5Q0GHHA44KMVNSF22E | 🔍 observability
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

fastify-node (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

fastify-quickjs (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

hono-node (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

hono-quickjs (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nest-node (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K1YKJ50GSGYQY3VYFFCY1E | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K1Z6HX0GSZA2FN6WY0XWQZ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes

nest-quickjs (12 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

nextjs-turbopack-node (3 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes

nextjs-turbopack-quickjs (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nextjs-webpack-node (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nextjs-webpack-quickjs (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nitro-node (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • distributedAbortController - reconnect to existing controller | wrun_41M0K27F0K0GW7P4VKWS20RDGQ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nitro-quickjs (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nuxt-node (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

nuxt-quickjs (3 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M0K1RB9Z0GZ7JBS5YVTDP7D6 | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1RBCR0GY6TXX9CE8BGDTK | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1RBCN0GSADT59ZP4VE250 | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1RBA60GNB57SAK5CXPMVE | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K1QFMC0GJN5KKNZGFNEXB5 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K1WNSF0GKDDPBJ5TH5P560 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K1WP4B0GN7PATD2QJK2FK3 | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K1WP200GHRDNQ0VD3KK2G7 | 🔍 observability

sveltekit-node (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K276HM0GH1RZ7KWDF9MXHT | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

sveltekit-quickjs (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

tanstack-start-node (5 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events

tanstack-start-quickjs (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly

vite-node (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XXDT0GXFZ0MEAQBXBJ6Q | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

vite-quickjs (11 failed):

  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1X0YQ0GNEFQMYVD1K3FQJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YWP60GJSQ84EK33ZJP98 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1XK810GV19J0QZSMJTG0K | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K20CEFPPN33NJ3Y61XJ7PT
  • promiseAllWorkflow | wrun_41M0K1RB9Z0GZ7JBS5YVTDP7D6
  • sleepingWorkflow | wrun_41M0K1RBCR0GY6TXX9CE8BGDTK
  • parallelSleepWorkflow | wrun_41M0K1RBCN0GSADT59ZP4VE250
  • nullByteWorkflow | wrun_41M0K1RBA60GNB57SAK5CXPMVE
  • cancelRun - cancelling a running workflow | wrun_41M0K1QFMC0GJN5KKNZGFNEXB5
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K1WNSF0GKDDPBJ5TH5P560
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K1WP4B0GN7PATD2QJK2FK3
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K1WP200GHRDNQ0VD3KK2G7

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

41 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nuxt)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (hono)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nuxt)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nextjs-webpack)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (nuxt)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (hono)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nextjs-webpack)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nuxt)
  • abortThrowIfAbortedWorkflow: throwIfAborted causes FatalError, no retries (astro)
  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (sveltekit)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (sveltekit)
  • distributedAbortController - manual abort triggers signal (tanstack-start)
  • distributedAbortController - reconnect to existing controller (fastify)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (vite)
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-turbopack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (tanstack-start)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (fastify)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (vite)
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook (vite)
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (tanstack-start)
  • Promise.all of disjoint-key writes: every key lands (express)
  • Promise.all of disjoint-key writes: every key lands (fastify)
  • Promise.all of disjoint-key writes: every key lands (nextjs-turbopack)
  • Promise.all of disjoint-key writes: every key lands (nextjs-webpack)
  • Promise.all of disjoint-key writes: every key lands (sveltekit)
  • promiseRaceWorkflow (sveltekit)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (express)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (fastify)
  • RetryableError respects custom retryAfter delay (nuxt)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nextjs-turbopack)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (tanstack-start)
  • sleepWinsRaceWorkflow (vite)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nextjs-webpack)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (vite)
  • webhookWorkflow (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nextjs-webpack)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (sveltekit)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

34 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:59Z · abandoned wrun_41M0K1JB2J0GGG3VXDM9DQCKWQ · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:15Z · abandoned wrun_41M0K1P0680GY23CJRKYM70RAW
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:15Z · abandoned wrun_41M0K1P06F0GJDBNZYT4PP7JDZ
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:15Z · abandoned wrun_41M0K1P06F0GJDBNZYT4PP7JDY
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:15Z · abandoned wrun_41M0K1P0M00GWCV3TSH8D2FYVX
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:15Z · abandoned wrun_41M0K1P06K0GZYK9FY984MD70H
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:48Z · abandoned wrun_41M0K1Q07W0GPFDEY6PXKTFT5D
  • run-pickup-stall · nullByteWorkflow (python) · at 20:56:16Z · abandoned wrun_41M0K1QVV00GVA8GB2NPNEDK3R
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:56:16Z · abandoned wrun_41M0K1QVXJ0GYAQN26711K4EST
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:56:16Z · abandoned wrun_41M0K1QVY30GJW6J5CGE397AG8
  • run-pickup-stall · sleepingWorkflow (python) · at 20:56:16Z · abandoned wrun_41M0K1QVV30GQSY0CEZ103ECC3
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:56Z · abandoned wrun_41M0K1R00J0GN0NF6PA9527TDD
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:48Z · abandoned wrun_41M0K1TKQH0GS0559GVP24YVM4
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:49Z · abandoned wrun_41M0K1TQX50GWAQWAJDKS9SK6T
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:59:09Z · abandoned wrun_01M0K1WSX8QRMNX6B2A6FN6AW0
  • cold-start-warmup · suite warmup (python) · at 20:59:24Z · abandoned wrun_01M0K1TE0PSDRGGMQVHA600PF5 · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:39Z · abandoned wrun_01M0K1Y35FVT4DGMRY0AJDBPDQ
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:39Z · abandoned wrun_01M0K1Y35NM9H5PSE84EFG30FJ
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:39Z · abandoned wrun_01M0K1Y35GZJNASTCZZV3CSNCX
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:39Z · abandoned wrun_01M0K1Y35RBPCYRKYQB5N9EDDW
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:39Z · abandoned wrun_01M0K1Y35TYATYXKVS861TFSRJ
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:00:39Z · abandoned wrun_01M0K1ZXS6AQGRFQD9NXWHK0EA
  • run-pickup-stall · promiseAllWorkflow (python) · at 21:00:39Z · abandoned wrun_01M0K1ZXSA5WYE5204H9P4HX0E
  • run-pickup-stall · sleepingWorkflow (python) · at 21:00:39Z · abandoned wrun_01M0K1ZXSE3Y8PD7GYV5ZP0PEP
  • run-pickup-stall · parallelSleepWorkflow (python) · at 21:00:39Z · abandoned wrun_01M0K1ZXSHD98ARYYENNBXK63G
  • run-pickup-stall · nullByteWorkflow (python) · at 21:00:39Z · abandoned wrun_01M0K1ZXSKWF5104EAA5SC4858
  • run-pickup-stall · AllInOneService.processNumber - static workflow method using sibling static step methods (nuxt) · at 21:01:36Z · abandoned wrun_01M0K21P0PN0VNXR8GEM4CSBMN
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:39Z · abandoned wrun_01M0K21RCSDBZEHNTJ32GBQD0F
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:39Z · abandoned wrun_01M0K21RD5C02AJEHP3HGSNC0W
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:39Z · abandoned wrun_01M0K21RD2P6A30VBRSR6VS8SZ
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:02:09Z · abandoned wrun_01M0K22NRWWZF0YV8WYS0BYVVE
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:02:09Z · abandoned wrun_01M0K22NRVP3G1N7SAM5GS8ZD9
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:02:39Z · abandoned wrun_01M0K23K08E2XA0RVQCMZDPE13
  • run-pickup-stall · pathsAliasWorkflow - TypeScript path aliases resolve correctly (nextjs-webpack) · at 21:05:55Z · abandoned wrun_01M0K29K1E9P9YWYSZ3W7VAQQB

E2E Test Summary

Summary
Passed Failed Skipped Total
❌ ▲ Vercel Production 3348 230 742 4320
✅ 💻 Local Development 3922 0 558 4480
✅ 📦 Local Production 3922 0 558 4480
✅ 🐘 Local Postgres 3922 0 558 4480
✅ 🪟 Windows 320 0 0 320
❌ 🌐 Cross-language Conformance 0 9 132 141
✅ vercel-multi-region 27 0 0 27
Total 15461 239 2548 18248
Details by Category

❌ ▲ Vercel Production

App Passed Failed Skipped
❌ astro-node 126 6 28
❌ astro-quickjs 125 7 28
❌ example-node 122 10 28
❌ example-quickjs 118 14 28
❌ express-node 121 11 28
❌ express-quickjs 122 10 28
❌ fastify-node 124 8 28
❌ fastify-quickjs 121 11 28
❌ hono-node 122 10 28
❌ hono-quickjs 124 8 28
❌ nest-node 124 8 28
❌ nest-quickjs 120 12 28
❌ nextjs-turbopack-node 154 3 3
❌ nextjs-turbopack-quickjs 150 7 3
❌ nextjs-webpack-node 150 7 3
❌ nextjs-webpack-quickjs 147 10 3
❌ nitro-node 125 7 28
❌ nitro-quickjs 121 11 28
❌ nuxt-node 121 11 28
❌ nuxt-quickjs 129 3 28
❌ python-node 0 8 152
❌ sveltekit-node 144 7 9
❌ sveltekit-quickjs 142 9 9
❌ tanstack-start-node 127 5 28
❌ tanstack-start-quickjs 123 9 28
❌ vite-node 125 7 28
❌ vite-quickjs 121 11 28

✅ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 160 0 0
✅ nextjs-turbopack-quickjs 160 0 0

❌ 🌐 Cross-language Conformance

App Passed Failed Skipped
❌ python 0 9 132

✅ vercel-multi-region

App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

📋 View full workflow run

@NathanColosimo NathanColosimo changed the title Decode replay payloads synchronously [core] Decode replay payloads synchronously Aug 14, 2026
@NathanColosimo
NathanColosimo force-pushed the codex/replay-sync-decode branch from 84e27c4 to 1e92acf Compare August 14, 2026 03:56
@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario outcome events virt replay violations
smoke-no-steps completed 3 0ms ok 0
smoke-one-step completed 6 0ms ok 0
hook-at-step-started completed 12 0ms ok 0
hook-at-step-completed completed 12 0ms ok 0
hook-at-hook-created completed 12 0ms ok 0
deadline-hook-wins completed 7 1.0h ok 0
deadline-expires completed 7 1.0h ok 0
long-sleep completed 11 30.0d ok 0
hook-never-arrives stalled 3 0ms skipped 0
step-retries-twice completed 10 2.0s ok 0
parallel-steps completed 9 0ms ok 0
hook-on-execution-state completed 12 0ms ok 0
peek-hook-before-branch completed 12 0ms ok 0
peek-hook-after-branch completed 12 0ms ok 0
peek-hook-at-registration completed 12 0ms ok 0
race-hook-before-probe completed 12 0ms ok 0
race-hook-after-probe completed 12 0ms ok 0
race-duplicate-delivery completed 13 0ms ok 0
attr-hook-before-step completed 11 0ms ok 0
attr-hook-after-step completed 11 0ms ok 0
attr-from-step-body completed 13 0ms ok 0
fork-hook-after-timeout completed 14 1.0m ok 0
fork-hook-before-timeout completed 14 1.0m ok 0
count-hook-after-timeout completed 17 1.0m ok 0
count-hook-before-timeout completed 20 1.0m ok 0
stale-read-step-count-fork completed 20 1.0m ok 0
stale-read-equal-step-counts completed 14 1.0m ok 0
step-vs-step-fork completed 12 0ms ok 0
step-vs-step-fork-fenced completed 12 0ms ok 0
fence-catches-benign-direction completed 12 5ms ok 0
in-flight-before-decision completed 17 1.0m ok 0
in-flight-before-decision-counted completed 17 1.0m ok 0
in-flight-after-decision completed 19 2.0m ok 0
stale-read-step-count-fork-fenced completed 20 1.0m ok 0
fork-hook-wins completed 13 1.0m ok 0
fork-timeout-wins completed 13 1.0m ok 0
unclaimed-payload-under-fork completed 17 1.0m ok 0
claimed-payload-under-fork completed 17 1.0m ok 0
writers-independent-step-bodies completed 12 0ms ok 0
writers-scripted-tempo completed 12 0ms ok 0
cancel-mid-step cancelled 7 0ms skipped 0

Full trace: world-sim.txt

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

The benchmark run for 84b7a5a failed. See the run logs for details.

commit 31488ca · Fri, 21 Aug 2026 10:38:22 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 478 (-29%) 💚 1395 🔴 (+35%) 🔻 1441 🔴 (+30%) 🔻 1505 🔴 (±0%) 30
TTFS stream 1056 (+284%) 🔻 1361 🔴 (+25%) 🔻 1376 🔴 (+25%) 🔻 1445 🔴 (+27%) 🔻 30
TTFS hook + stream 630 (-51%) 💚 1681 🔴 (+18%) 🔻 1721 🔴 (+3.5%) 2004 🔴 (-57%) 💚 30
Fan-out TTFS Promise.all(100 steps) 540 (+24%) 🔻 718 (+16%) 🔻 750 (+12%) 2407 (+23%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 4264 (-2.8%) 6796 (-9.3%) 7180 (-6.3%) 16329 (+85%) 🔻 10
STSO 1020 steps (inline) 112 (-12%) 178 (-27%) 💚 236 (-17%) 💚 333 (-27%) 💚 1019
WO 1020 steps 175515 (-24%) 💚 175515 (-24%) 💚 175515 (-24%) 💚 175515 (-24%) 💚 1
CRTT first chunk (pooled) 81 (-8.0%) 168 (+30%) 🔻 237 (+58%) 🔻 512 (+27%) 🔻 28

Streams

Scenario CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 126 (+18%) 117 (-5%) 274 (+29%) 584 (+15%) 104 (-17%) 10
size sweep (100/s, 160B-12KB) 113 (+9%) 117 (-11%) 157 (-29%) 372 (-31%) 108 (-34%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 111 (-18%) 164 (+21%) 309 (+51%) 566 (+2%) 238 (+65%) 3
replay eve-gpt-5.6-sol-2000t (1x) 130 (-49%) 120 (-51%) 183 (-87%) 641 (-86%) 560 (-76%) 2
replay eve-gpt-5.6-sol-2000t (2x) 157 (+5%) 135 (-24%) 185 (-41%) 348 (-64%) 189 (-67%) 3
📜 Previous results (2)

e0181d4

Thu, 20 Aug 2026 18:00:09 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1245 (+21%) 🔻 1310 🔴 (+16%) 🔻 1332 🔴 (+14%) 1362 🔴 (+11%) 30
TTFS stream 249 (+40%) 🔻 1304 🔴 (+16%) 🔻 1319 🔴 (+17%) 🔻 1341 🔴 (+14%) 30
TTFS hook + stream 506 (+1.8%) 1608 🔴 (+12%) 1636 🔴 (+11%) 1917 🔴 (+24%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 585 (+47%) 🔻 894 (±0%) 903 (-48%) 💚 2043 (+8.2%) 10
Fan-out TTLS Promise.all(100 steps) 4666 (-2.6%) 5220 (-16%) 💚 5258 (-16%) 💚 7418 (+2.9%) 10
STSO 1020 steps (inline) 124 (+3.3%) 243 (-1.6%) 288 (-4.0%) 501 (-0.6%) 1019
WO 1020 steps 230339 (-3.0%) 230339 (-3.0%) 230339 (-3.0%) 230339 (-3.0%) 1
CRTT first chunk (pooled) 87 (-5.4%) 162 (+23%) 🔻 267 (+77%) 🔻 360 (+118%) 🔻 28

Streams

Scenario CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 106 (-13%) 124 (-14%) 162 (-39%) 230 (-79%) 91.5 (-38%) 10
size sweep (100/s, 160B-12KB) 138 (+6%) 161 (-27%) 375 (-28%) 637 (-18%) 141 (-48%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 136 (+12%) 167 (+14%) 1978 (+673%) 3686 (+424%) 570 (+10%) 3
replay eve-gpt-5.6-sol-2000t (1x) 248 (+95%) 108 (-29%) 158 (-38%) 525 (-37%) 354 (-29%) 2
replay eve-gpt-5.6-sol-2000t (2x) 178 (+46%) 160 (-24%) 229 (-29%) 775 (-77%) 469 (-22%) 3

2e2fa27

Wed, 19 Aug 2026 17:27:53 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1335 (+365%) 🔻 1401 🔴 (+20%) 🔻 1405 🔴 (+4.9%) 1417 🔴 (-9.3%) 30
TTFS stream 1344 (+611%) 🔻 1429 🔴 (+28%) 🔻 1491 🔴 (+31%) 🔻 1529 🔴 (+31%) 🔻 30
TTFS hook + stream 1360 (+4.2%) 1741 🔴 (+26%) 🔻 1838 🔴 (+30%) 🔻 2018 🔴 (+18%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 1009 (-8.7%) 1379 (-40%) 💚 1465 (-47%) 💚 2996 (+8.1%) 10
Fan-out TTLS Promise.all(100 steps) 5087 (-3.7%) 6123 (-8.0%) 6777 (-1.5%) 10450 (+33%) 🔻 10
STSO 1020 steps (inline) 109 (-17%) 💚 221 (-16%) 💚 246 (-19%) 💚 379 (-25%) 💚 1019
WO 1020 steps 213351 (-14%) 213351 (-14%) 213351 (-14%) 213351 (-14%) 1
CRTT first chunk (pooled) 86 (+2.4%) 111 (-13%) 172 (-45%) 💚 239 (-62%) 💚 28

Streams

Scenario CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 104 (-8%) 142 (-14%) 645 (+178%) 4192 (+802%) 142 (-11%) 10
size sweep (100/s, 160B-12KB) 101 (-6%) 128 (-17%) 183 (-35%) 373 (-45%) 149 (-7%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 99 (-12%) 122 (-56%) 149 (-80%) 461 (-69%) 369 (-39%) 3
replay eve-gpt-5.6-sol-2000t (1x) 174 (+12%) 181 (+29%) 319 (+66%) 449 (-29%) 365 (-25%) 2
replay eve-gpt-5.6-sol-2000t (2x) 106 (-79%) 263 (-35%) 452 (-84%) 955 (-77%) 530 (-12%) 3
ℹ️ Metric definitions & methodology

Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@VaguelySerious VaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

}

async function decompressGzip(payload: Uint8Array): Promise<Uint8Array> {
if (nativeGzip) return nativeGzip.decompress(payload);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The gzip read path silently switches from the streaming codec to a blocking synchronous one. main's decompress always went through gunzipBytespipeThroughTransform(data, new DecompressionStream('gzip')); this prefers nativeGzip.decompress, i.e. zlib.gunzipSync, whenever process.getBuiltinModule resolves — Node 22.3+, so effectively every current runtime.

That matters because of what sits above it in this stack. #3548 starts preparation for every binary payload in the log as frames arrive, so a replay of a long run now runs N gunzipSync calls. Each is its own microtask, but the inflation inside each one blocks the event loop, where previously the work was spread across stream pumps. Any run still carrying gzip-prefixed payloads (written by a pre-zstd runtime, or with WORKFLOW_COMPRESSION_CODEC=gzip) gets that as a synchronous burst on the TTFS path.

The write path has the same shape at line 163 but a smaller blast radius: zstd wins selectWriteCodec on Node 22.15+, so sync gzip writes only affect Node 22.3–22.14 and the explicit gzip override. Note the zstd write path was already zstdCompressSync on main, so that half is not new.

Neither change is mentioned in the PR body or the changeset. If preferring native is deliberate for the latency win, it is worth saying so and confirming the sync burst is acceptable; if it is incidental to the extraction, keeping DecompressionStream for gzip would preserve the old behaviour.

I checked the things that could have gone wrong here and they are fine: ZSTD_LEVEL = 3 still reaches zstdCompressSync via the ZSTD_c_compressionLevel param mapping, canCompressGzip and the gzip/decompressGzip fallbacks agree so selectWriteCodec cannot pick a codec it then fails to run, and process.getBuiltinModule is the same resolution mechanism main used, so Node 18/20 keep the portable path rather than losing compression.

@VaguelySerious VaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues (re-review of the new fix(world): preserve omitted event payloads commit; the rest of the series is unchanged from my last review)

* stored event so the strict serialized-data schema can still distinguish an
* omitted read payload from a malformed create request.
*/
function restoreOmittedEventPayload(value: unknown): unknown {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

I verified the underlying break is real, and it is narrower than the fix. Parsing the four eventData shapes an empty-body v4 frame can present at decodeEventFrame's EventSchema.parse(meta):

frame meta shape before this commit after
step_completed, eventData absent REJECTS (eventData) PARSES
step_completed, eventData: {}, no result PARSES PARSES
run_started, eventData: {}, no input PARSES PARSES
hook_received, eventData without payload PARSES PARSES

So only the eventData entirely absent shape was rejecting — a missing payload key inside a present eventData already parsed. That one shape is exactly what stripEventDataRefs emits, since it deletes the ref fields and then drops eventData when nothing is left. Worth fixing.

The concern is that the preprocessor is much wider than the shape it repairs. It injects the payload key into every payload-bearing event, so a run_created that previously parsed to eventData keys [deploymentId, workflowName] now parses to [deploymentId, workflowName, input], and 'input' in eventData flips falsetrue. Your own test codifies that with toHaveProperty('input', undefined). Since only the absent-eventData case needed help, making eventData optional (or .default({})) on the affected schemas would fix it without changing the parsed shape of every other event.

That shape change is observable past the parse boundary — cbor-x encodes an explicit undefined as CBOR f7 rather than omitting the key:

encode({x:1})                  -> b90001617801
encode({x:1, input:undefined}) -> b9000261780165696e707574f7
decode(...) keys               -> ['x', 'input']

So a parsed event that is ever re-encoded carries a key it did not have before. I looked for somewhere that bites and did not find one — the two Object.keys(eventData) sites are both safe (stripEventDataRefs uses delete, so its length check still sees 0; analyticsEventToEvent builds eventData from scratch and never parses). Flagging it because it is the kind of thing that surfaces later at a wire boundary rather than here.

Secondary: the preprocessor runs on every event parse and allocates two objects per event on the injection path, which is the resolveData: 'none' bulk-listing path — the one that exists specifically to avoid per-payload work, and the one #3548 now drives per frame.

One more thing worth separating out: origin/main rejects the same absent-eventData shape (verified against 5a59bb82e8). If a live read path re-parses a stripped event, that is a production bug being repaired as the 24th commit of a refactor PR three levels deep in a stack, where it will never be found again. I could not confirm such a path — world-postgres and world-local both parse then strip, so the stripped shape is not re-validated in-process — but if you did hit this for real, it deserves its own PR against main with the reproducing case.

events: z.undefined(),
cursor: z.undefined(),
hasMore: z.undefined(),
// Materialized writes omit page metadata entirely. Zod 4.4 correctly

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Nit

This comment is the most useful sentence in the commit and it is on the wrong change — the z.undefined()-is-required behaviour is what motivates the EventSchema preprocessor too, but that hunk explains itself only as "worlds may omit a payload field" with no mention of the Zod version bump that changed the rule.

Worth repeating the Zod 4.4 rationale at restoreOmittedEventPayload, and worth a line in the changeset: the schema-loosening half is user-visible for anyone validating events with the exported EventSchema, and the current changeset text ("Simplify payload codecs and reuse prepared replay payloads across workflow VMs") does not hint at it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants