Skip to content

[core] Prepare replay payloads as event frames arrive - #3548

Open
NathanColosimo wants to merge 16 commits into
codex/replay-sync-decodefrom
codex/replay-stream-preparation
Open

[core] Prepare replay payloads as event frames arrive#3548
NathanColosimo wants to merge 16 commits into
codex/replay-sync-decodefrom
codex/replay-stream-preparation

Conversation

@NathanColosimo

@NathanColosimo NathanColosimo commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • introduce and document the synchronous onEvent observer on event create/list operations, where streamed replay preparation first consumes it
  • invoke observers in stream order immediately after each event validates, with natural response backpressure
  • make observer exceptions abort the operation without classifying them as transport failures or retrying event I/O, even when the thrown value resembles a transport error
  • resolve the run key as soon as a deployment-bearing replay event arrives
  • buffer decoded event references until that one key resolution completes, then start decrypt/decompress preparation for the buffered payloads together
  • prepare subsequent payloads as frames arrive and retain one shared preparation Promise per payload across fresh workflow VMs
  • keep a full scan of newly appended event-array positions as a correctness fallback for worlds that cannot observe streamed frames
  • reuse immutable primitive values for successful results, errors, hooks, and aborts while rehydrating object graphs per VM
  • compact prepared Buffer subviews before caching so small payloads do not retain large backing allocations

Observer contract

onEvent runs synchronously after validation and before the next frame is consumed. Throwing aborts the operation; it is not a transport failure and is never retried. If an interrupted stream resumes, continuation starts after the last observed event, so every accepted event is observed exactly once.

The callback itself is synchronous, but payload preparation may be asynchronous (the current codec stack uses Web Crypto). This PR overlaps that work with later frame delivery and VM setup; ordered consumers await only the payloads they actually reach.

Performance signal

The latest benchmark and previous behavior run were recorded on the original cumulative branch before this review split. Across those runs, inline STSO improved 4–20% at p75, 10–24% at p90, and 39–51% at p99; total 1020-step workflow overhead improved 7–23%.

Cold TTFS regressed in both runs, including the turbo control that does not preload or prepare replay events, so that signal is dominated by deployment cold-start variability. The benchmarked branch also contained the small tracing layer in #3523.

Stack

  1. [world-vercel] Resume partial replay streams after validated events #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — replay codec simplification
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation and event observer
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Retain workflow VMs across hooks #3604 — retain workflow VMs across hooks
  6. [core] Retain workflow VMs across attributes #3609 — retain workflow VMs across attributes
  7. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  8. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles
  9. [workflow] Share isolated serializer registries across replay bundles #3639 — shared serializer bootstrap

Validation

  • pnpm --filter @workflow/core... build
  • pnpm --filter @workflow/world-vercel... build
  • full world-vercel suite — 553 tests passed
  • full core suite — 2,214 tests passed, 3 expected failures, 1 skipped
  • focused replay cache and compression telemetry suites — 17 tests passed
  • git diff --check
  • final exact-branch autoreview (Codex, gpt-5.6-sol, xhigh): clean, no actionable P0–P2 findings; correctness confidence 0.88
  • GitHub checks for the current head are running

@changeset-bot

changeset-bot Bot commented Aug 14, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 13d4912

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages
Name Type
@workflow/core Patch
@workflow/world Patch
@workflow/world-vercel Patch
@workflow/builders Patch
@workflow/cli Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/web Patch
workflow Patch
@workflow/world-testing Patch
@workflow/world-local Patch
@workflow/world-postgres Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/nuxt Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
example-nextjs-workflow-turbopack Ready Ready Preview, v0 Aug 21, 2026 8:53pm
example-nextjs-workflow-webpack Building Building Preview, v0 Aug 21, 2026 8:53pm
example-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-astro-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-express-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-fastify-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-hono-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-nestjs-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-nitro-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-nuxt-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-python-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-sveltekit-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-tanstack-start-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workbench-vite-workflow Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workflow-docs Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workflow-swc-playground Building Building Preview, v0 Aug 21, 2026 8:53pm
workflow-tarballs Ready Ready Preview, v0 Aug 21, 2026 8:53pm
workflow-web Ready Ready Preview, v0 Aug 21, 2026 8:53pm

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (237 failed)

astro-node (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

astro-quickjs (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

example-node (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

example-quickjs (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

express-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

express-quickjs (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

fastify-node (6 failed):

  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

fastify-quickjs (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events

hono-node (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K1XXMD0GVG0PQ6FF387MMH | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

hono-quickjs (13 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

nest-node (6 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nest-quickjs (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

nextjs-turbopack-node (5 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • distributedAbortController - reconnect to existing controller | wrun_41M0K274T90GS1R3ET90Z1TM6H | 🔍 observability
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nextjs-turbopack-quickjs (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nextjs-webpack-node (6 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nextjs-webpack-quickjs (5 failed):

  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nitro-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K1XXMD0GVG0PQ6FF387MMH | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K1YY6P0GH833VVC5S1QSQP | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nitro-quickjs (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

nuxt-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nuxt-quickjs (13 failed):

  • webhookWorkflow | wrun_41M0K1KGDA0GS5E6J83DMSXDSN | 🔍 observability
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M0K1JB990GQFKGG7XGDB0SMR | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1MFDF0GJW6S0SVQPSXY9B | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1MFEH0GKY1KX3PVZGFDWM | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1NEGM0GPNEZY6MWK2GQNB | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K23B830GK0GF7DRMV0R946 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K23H8F0GN1SGVQWZ8HYE4M | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K23XRY0GZ1Y06ZJE2JVNGW | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K25XA20GY1KT8T39FHNWQ7 | 🔍 observability

sveltekit-node (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

sveltekit-quickjs (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

tanstack-start-node (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K1XXMD0GVG0PQ6FF387MMH | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K269B10GN4PZKVK4T1Z2NR | 🔍 observability
  • distributedAbortController - reconnect to existing controller | wrun_41M0K274T90GS1R3ET90Z1TM6H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

tanstack-start-quickjs (13 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

vite-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WKMC0GNVTFZF2RWN9JYA | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • distributedAbortController - reconnect to existing controller | wrun_41M0K274T90GS1R3ET90Z1TM6H | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

vite-quickjs (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1XBTF0GW2Q978R8NDZK20 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1YCHT0GHVKCWRP0PGGFPQ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1Z2TR0GMWG4BW191Y5GDJ | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1Y2VWPW6JRE34XDB1PZMV
  • promiseAllWorkflow | wrun_41M0K1JB990GQFKGG7XGDB0SMR
  • sleepingWorkflow | wrun_41M0K1MFDF0GJW6S0SVQPSXY9B
  • parallelSleepWorkflow | wrun_41M0K1MFEH0GKY1KX3PVZGFDWM
  • nullByteWorkflow | wrun_41M0K1NEGM0GPNEZY6MWK2GQNB
  • cancelRun - cancelling a running workflow | wrun_41M0K23B830GK0GF7DRMV0R946
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K23H8F0GN1SGVQWZ8HYE4M
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K23XRY0GZ1Y06ZJE2JVNGW
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K25XA20GY1KT8T39FHNWQ7

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

41 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (fastify)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nest)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (astro)
  • abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook (vite)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (hono)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (nitro)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (astro)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (example)
  • addTenWorkflow (nextjs-turbopack)
  • cancelRun via CLI - cancelling a running workflow (fastify)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (express)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (vite)
  • distributedAbortController - manual abort triggers signal (nextjs-webpack)
  • distributedAbortController - manual abort triggers signal (nuxt)
  • distributedAbortController - manual abort triggers signal (sveltekit)
  • distributedAbortController - reconnect to existing controller (express)
  • fire-and-forget: void setAttributes lands without awaiting (fastify)
  • health check (CLI) - workflow health command reports healthy endpoints (nextjs-webpack)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (express)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (hono)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nextjs-webpack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (astro)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (fastify)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-webpack)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (astro)
  • hookTokenReuseLoopWorkflow - same run recreates a hook with the same token after dispose() (express)
  • Promise.all of disjoint-key writes: every key lands (example)
  • Promise.all of disjoint-key writes: every key lands (nextjs-turbopack)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (nest)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nest)
  • setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (vite)
  • sleepWinsRaceWorkflow (vite)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (example)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nitro)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (astro)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (fastify)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nextjs-webpack)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (sveltekit)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

32 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:40Z · abandoned wrun_41M0K1HS430GNAB62HV09155M4 · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:54:56Z · abandoned wrun_41M0K1NE160GRAW5BP9P785MRV
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:56Z · abandoned wrun_41M0K1NE1B0GGVVGH8D3MZMHGV
  • run-pickup-stall · sleepingWorkflow (python) · at 20:54:56Z · abandoned wrun_41M0K1NE1B0GGVVGH8D3MZMHGT
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEDE0GYG5XQA9YFPDWQ9
  • run-pickup-stall · nullByteWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NE1E0GTYDD2NYR01HNC8
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:30Z · abandoned wrun_41M0K1PEH20GV9E14EEB6CV5DN
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:57Z · abandoned wrun_41M0K1Q9K30GXFVRVPSB8AF6B2
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9H10GGQQ9Z49CPA0MHS
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9KZ0GW9ADW04ZBJG1C9
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9KE0GXWNRM2F9CEC0A5
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:38Z · abandoned wrun_41M0K1QEKF0GW6G36H04DHQ2E7
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:57:52Z · abandoned wrun_01M0K1TEBKSG115Q4DWXHRPHJP
  • cold-start-warmup · suite warmup (python) · at 20:58:08Z · abandoned wrun_01M0K1R4E6ZHHK7GCTP8E8S8PH · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:58:23Z · abandoned wrun_01M0K1VSJQH3JTPMEFWKP2B91Z
  • run-pickup-stall · sleepingWorkflow (python) · at 20:58:23Z · abandoned wrun_01M0K1VSJYAANPS29D08A274VV
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:58:23Z · abandoned wrun_01M0K1VSJRMFKVGH4H7319JSVN
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:58:23Z · abandoned wrun_01M0K1VSJZP8QW3RD1FAH70ZVM
  • run-pickup-stall · nullByteWorkflow (python) · at 20:58:23Z · abandoned wrun_01M0K1VSK13CDFSNADS4NDPP4Z
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:29Z · abandoned wrun_41M0K1T6G10GKTA11RW57K9VG7
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:29Z · abandoned wrun_41M0K1SBPC0GGMTX40B8KQ873K
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:23Z · abandoned wrun_01M0K1XM6VBJ25WPG4PAFW3G5G
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:23Z · abandoned wrun_01M0K1XM6TKBJ5QM8A7V1N3T9Q
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:23Z · abandoned wrun_01M0K1XM6PS23EFVXX6ZN4ZD89
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:23Z · abandoned wrun_01M0K1XM6RENZ4YMVYVTDEJTCX
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:23Z · abandoned wrun_01M0K1XM6WP9HT0SZH16NWJ6VH
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:00:23Z · abandoned wrun_01M0K1ZET90R2G38B7TTAFX8FW
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:00:23Z · abandoned wrun_01M0K1ZETFTDER5TRKJ40YDRMN
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:00:23Z · abandoned wrun_01M0K1ZETRDFA4A2J72W93AWE0
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:00:54Z · abandoned wrun_01M0K20C6GFNC482N80GQCARXR
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:00:54Z · abandoned wrun_01M0K20C6F46XPV6X6EYR9M2XB
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:23Z · abandoned wrun_01M0K219DVPSCC2DB2FDQF0JD8

E2E Test Summary

Summary
Passed Failed Skipped Total
❌ ▲ Vercel Production 3341 237 742 4320
✅ 💻 Local Development 3922 0 558 4480
✅ 📦 Local Production 3922 0 558 4480
✅ 🐘 Local Postgres 3922 0 558 4480
✅ 🪟 Windows 320 0 0 320
❌ 🌐 Cross-language Conformance 0 9 132 141
✅ vercel-multi-region 27 0 0 27
Total 15454 246 2548 18248
Details by Category

❌ ▲ Vercel Production

App Passed Failed Skipped
❌ astro-node 125 7 28
❌ astro-quickjs 123 9 28
❌ example-node 124 8 28
❌ example-quickjs 123 9 28
❌ express-node 123 9 28
❌ express-quickjs 123 9 28
❌ fastify-node 126 6 28
❌ fastify-quickjs 121 11 28
❌ hono-node 124 8 28
❌ hono-quickjs 119 13 28
❌ nest-node 126 6 28
❌ nest-quickjs 122 10 28
❌ nextjs-turbopack-node 152 5 3
❌ nextjs-turbopack-quickjs 150 7 3
❌ nextjs-webpack-node 151 6 3
❌ nextjs-webpack-quickjs 152 5 3
❌ nitro-node 123 9 28
❌ nitro-quickjs 122 10 28
❌ nuxt-node 123 9 28
❌ nuxt-quickjs 119 13 28
❌ python-node 0 8 152
❌ sveltekit-node 143 8 9
❌ sveltekit-quickjs 143 8 9
❌ tanstack-start-node 121 11 28
❌ tanstack-start-quickjs 119 13 28
❌ vite-node 123 9 28
❌ vite-quickjs 121 11 28

✅ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 160 0 0
✅ nextjs-turbopack-quickjs 160 0 0

❌ 🌐 Cross-language Conformance

App Passed Failed Skipped
❌ python 0 9 132

✅ vercel-multi-region

App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

📋 View full workflow run

@NathanColosimo NathanColosimo changed the title Prepare replay payloads during event streaming [core] Prepare replay payloads as event frames arrive Aug 14, 2026
@NathanColosimo
NathanColosimo force-pushed the codex/replay-stream-preparation branch from 388b6ea to 294d4b7 Compare August 14, 2026 03:56
@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario outcome events virt replay violations
smoke-no-steps completed 3 0ms ok 0
smoke-one-step completed 6 0ms ok 0
hook-at-step-started completed 12 0ms ok 0
hook-at-step-completed completed 12 0ms ok 0
hook-at-hook-created completed 12 0ms ok 0
deadline-hook-wins completed 7 1.0h ok 0
deadline-expires completed 7 1.0h ok 0
long-sleep completed 11 30.0d ok 0
hook-never-arrives stalled 3 0ms skipped 0
step-retries-twice completed 10 2.0s ok 0
parallel-steps completed 9 0ms ok 0
hook-on-execution-state completed 12 0ms ok 0
peek-hook-before-branch completed 12 0ms ok 0
peek-hook-after-branch completed 12 0ms ok 0
peek-hook-at-registration completed 12 0ms ok 0
race-hook-before-probe completed 12 0ms ok 0
race-hook-after-probe completed 12 0ms ok 0
race-duplicate-delivery completed 13 0ms ok 0
attr-hook-before-step completed 11 0ms ok 0
attr-hook-after-step completed 11 0ms ok 0
attr-from-step-body completed 13 0ms ok 0
fork-hook-after-timeout completed 14 1.0m ok 0
fork-hook-before-timeout completed 14 1.0m ok 0
count-hook-after-timeout completed 17 1.0m ok 0
count-hook-before-timeout completed 20 1.0m ok 0
stale-read-step-count-fork completed 20 1.0m ok 0
stale-read-equal-step-counts completed 14 1.0m ok 0
step-vs-step-fork completed 12 0ms ok 0
step-vs-step-fork-fenced completed 12 0ms ok 0
fence-catches-benign-direction completed 12 5ms ok 0
in-flight-before-decision completed 17 1.0m ok 0
in-flight-before-decision-counted completed 17 1.0m ok 0
in-flight-after-decision completed 19 2.0m ok 0
stale-read-step-count-fork-fenced completed 20 1.0m ok 0
fork-hook-wins completed 13 1.0m ok 0
fork-timeout-wins completed 13 1.0m ok 0
unclaimed-payload-under-fork completed 17 1.0m ok 0
claimed-payload-under-fork completed 17 1.0m ok 0
writers-independent-step-bodies completed 12 0ms ok 0
writers-scripted-tempo completed 12 0ms ok 0
cancel-mid-step cancelled 7 0ms skipped 0

Full trace: world-sim.txt

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

The benchmark run for 13d4912 failed. See the run logs for details.

commit f0cf664 · Fri, 21 Aug 2026 11:12:49 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 374 (-45%) 💚 1378 🔴 (+34%) 🔻 1410 🔴 (+27%) 🔻 1431 🔴 (-4.9%) 30
TTFS stream 949 (+245%) 🔻 1404 🔴 (+29%) 🔻 1419 🔴 (+29%) 🔻 1428 🔴 (+26%) 🔻 30
TTFS hook + stream 546 (-57%) 💚 1663 🔴 (+16%) 🔻 1745 🔴 (+4.9%) 1968 🔴 (-58%) 💚 30
Fan-out TTFS Promise.all(100 steps) 478 (+9.9%) 715 (+15%) 🔻 2093 (+211%) 🔻 2242 (+15%) 10
Fan-out TTLS Promise.all(100 steps) 4533 (+3.4%) 5379 (-28%) 💚 6801 (-11%) 12761 (+45%) 🔻 10
STSO 1020 steps (inline) 119 (-6.3%) 219 (-10%) 250 (-12%) 356 (-22%) 💚 1019
WO 1020 steps 213147 (-8.0%) 213147 (-8.0%) 213147 (-8.0%) 213147 (-8.0%) 1
CRTT first chunk (pooled) 92 (+4.5%) 138 (+7.0%) 209 (+39%) 🔻 294 (-27%) 💚 28

Streams

Scenario CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 110 (+3%) 114 (-7%) 132 (-38%) 356 (-30%) 76.5 (-39%) 10
size sweep (100/s, 160B-12KB) 116 (+12%) 117 (-11%) 221 (±0%) 356 (-34%) 106 (-36%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 155 (+14%) 108 (-21%) 132 (-35%) 207 (-63%) 135 (-6%) 3
replay eve-gpt-5.6-sol-2000t (1x) 127 (-50%) 155 (-37%) 265 (-81%) 430 (-90%) 373 (-84%) 2
replay eve-gpt-5.6-sol-2000t (2x) 137 (-9%) 187 (+5%) 377 (+21%) 715 (-27%) 452 (-22%) 3
📜 Previous results (2)

1460540

Thu, 20 Aug 2026 18:03:52 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 250 (-76%) 💚 1356 🔴 (+20%) 🔻 1367 🔴 (+17%) 🔻 1504 🔴 (+23%) 🔻 30
TTFS stream 275 (+54%) 🔻 1372 🔴 (+23%) 🔻 1444 🔴 (+28%) 🔻 1730 🔴 (+47%) 🔻 30
TTFS hook + stream 406 (-18%) 💚 1683 🔴 (+17%) 🔻 1743 🔴 (+18%) 🔻 1752 🔴 (+13%) 30
Fan-out TTFS Promise.all(100 steps) 583 (+46%) 🔻 1000 (+12%) 1027 (-41%) 💚 2124 (+13%) 10
Fan-out TTLS Promise.all(100 steps) 5411 (+13%) 7824 (+27%) 🔻 8071 (+29%) 🔻 9210 (+28%) 🔻 10
STSO 1020 steps (inline) 134 (+12%) 269 (+8.9%) 308 (+2.7%) 490 (-2.8%) 1019
WO 1020 steps 257636 (+8.5%) 257636 (+8.5%) 257636 (+8.5%) 257636 (+8.5%) 1
CRTT first chunk (pooled) 98 (+6.5%) 171 (+30%) 🔻 201 (+33%) 🔻 505 (+206%) 🔻 28

Streams

Scenario CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 152 (+24%) 179 (+23%) 342 (+30%) 694 (-37%) 177 (+20%) 10
size sweep (100/s, 160B-12KB) 135 (+4%) 217 (-2%) 495 (-5%) 932 (+20%) 324 (+19%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 195 (+61%) 167 (+14%) 348 (+36%) 896 (+27%) 593 (+14%) 3
replay eve-gpt-5.6-sol-2000t (1x) 127 (±0%) 213 (+40%) 364 (+43%) 822 (-2%) 918 (+83%) 2
replay eve-gpt-5.6-sol-2000t (2x) 121 (-1%) 210 (±0%) 357 (+11%) 975 (-71%) 530 (-12%) 3

c5d8448

Wed, 19 Aug 2026 17:29:47 GMT · run logs

vercel / nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 247 (-14%) 1430 🔴 (+23%) 🔻 1456 🔴 (+8.7%) 3070 🔴 (+96%) 🔻 30
TTFS stream 292 (+54%) 🔻 1422 🔴 (+27%) 🔻 1468 🔴 (+29%) 🔻 1506 🔴 (+29%) 🔻 30
TTFS hook + stream 477 (-63%) 💚 1740 🔴 (+26%) 🔻 1840 🔴 (+31%) 🔻 2231 🔴 (+30%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 1482 (+34%) 🔻 3011 (+31%) 🔻 3206 (+16%) 🔻 3759 (+36%) 🔻 10
Fan-out TTLS Promise.all(100 steps) 5956 (+13%) 9255 (+39%) 🔻 9854 (+43%) 🔻 12880 (+63%) 🔻 10
STSO 1020 steps (inline) 133 (+0.8%) 279 (+6.1%) 326 (+7.2%) 476 (-5.7%) 1019
WO 1020 steps 262483 (+5.6%) 262483 (+5.6%) 262483 (+5.6%) 262483 (+5.6%) 1
CRTT first chunk (pooled) 103 (+23%) 🔻 169 (+33%) 🔻 415 (+32%) 🔻 463 (-26%) 💚 28

Streams

Scenario CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 147 (+30%) 258 (+56%) 366 (+58%) 1009 (+117%) 311 (+95%) 10
size sweep (100/s, 160B-12KB) 151 (+41%) 378 (+145%) 493 (+75%) 1145 (+69%) 798 (+396%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 133 (+19%) 162 (-42%) 222 (-70%) 470 (-68%) 400 (-34%) 3
replay eve-gpt-5.6-sol-2000t (1x) 137 (-12%) 159 (+14%) 207 (+8%) 817 (+30%) 618 (+27%) 2
replay eve-gpt-5.6-sol-2000t (2x) 124 (-75%) 224 (-45%) 365 (-87%) 861 (-80%) 751 (+24%) 3
ℹ️ Metric definitions & methodology

Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@VaguelySerious VaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found


function isMemoizablePrimitive(value: unknown): boolean {
if (value === null) return true;
function isCacheablePrimitive(value: unknown): boolean {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Blocking

This drops the size bound that main deliberately had, and widens where memoization applies at the same time.

main's predicate was isMemoizablePrimitive with MAX_MEMOIZED_PRIMITIVE_LENGTH = 4096, capping both strings and bigints, and its comment stated the intent: "Objects and large strings/bigints always run hydrate again, producing a fresh VM-specific value from the separately cached prepared payload." isCacheablePrimitive has no length check, so every string is now retained regardless of size.

Verified with a 1 MiB string result, counting hydrate invocations across two replays of the same log:

hydrate calls meaning
main 2 re-hydrated per replay, not retained
this branch 1 retained for the invocation

(main's small-string control memoizes at 1 call, so the cap was doing real work, not dead code.)

The retention is additive, not a substitution. getEventValue calls getPayload, which populates preparations[eventId] with the prepared plaintext bytes and never deletes them on success, and then cachePrimitive stores the hydrated value in primitiveValues[eventId]. Both live for the whole invocation, so a large-payload run holds the bytes and the UTF-16 string — roughly double the peak for that payload class.

The surface also grew from one call site to five. On main primitive memoization was reachable only through getStepResult at step.ts:304. getEventValue is now called from step.ts:193 (step errors), step.ts:304 (step results), hook.ts:355, hook.ts:427 (hook payloads) and abort-controller.ts:233. Hook payloads are webhook bodies, which is exactly the large-and-unbounded case the 4096 cap was protecting against.

If removing the cap is intentional, it needs to be stated with the memory reasoning, since the retention is now unbounded in payload size and the run's step count. Otherwise carry MAX_MEMOIZED_PRIMITIVE_LENGTH across.

* replaying the accepted prefix. Throwing aborts the operation and is never
* treated as a transport failure or retried.
*/
onEvent?: (event: Event) => void;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The observer plumbing itself holds up well — I traced every path and the invariants are tight: EventObserverError is excluded from withV4ResponseBody's transport accounting, rethrown ahead of the LIST retry decision, rethrown rather than wrapped in EventPostResponseError by the continuation catch, and unwrapped to the caller's original throw at both public boundaries. events.push sits directly before onEvent, so events and observations stay in lockstep and the eid: continuation cursor observes each accepted event exactly once.

Two contract gaps:

  1. onEvent lands on ListEventsParams and CreateEventParams but not ListEventsByCorrelationIdParams, which is the odd one out of the three list-shaped params in this file.
  2. The doc does not say a world may ignore it. It reads as mandatory to anyone implementing the World interface, but the runtime's prepareAll full-scan fallback exists precisely so worlds that cannot observe streamed frames stay correct. Worth saying so here — that fallback is the reason this is optional, and it is not discoverable from the interface.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants