Skip to content

[world-vercel] Resume partial replay streams after validated events - #3546

Open
NathanColosimo wants to merge 10 commits into
mainfrom
codex/replay-stream-recovery
Open

[world-vercel] Resume partial replay streams after validated events#3546
NathanColosimo wants to merge 10 commits into
mainfrom
codex/replay-stream-recovery

Conversation

@NathanColosimo

@NathanColosimo NathanColosimo commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • resume an interrupted framed replay response from the last fully validated event
  • preserve every clean response sentinel as authoritative, including a bounded hasMore: true result
  • bound continuation GET retries and retain the last accepted cursor across empty or interrupted suffixes
  • never repeat an accepted event POST when downloading its replay suffix eventually fails
  • classify truncated reads as transport failures while surfacing malformed CBOR and invalid frame metadata as protocol errors
  • make response-body consumption the single owner of dispatcher recycling outcomes for framed, single-event, and batch responses
  • preserve normal request deadlines for materialized writes while allowing framed event streams to drain without a whole-request deadline
  • remove the materialized-response compatibility path, V4Response, reportTransportOutcome, and the complete/partial result state machinery

Protocol contract

returnAll=true asks workflow-server to drain storage pagination, but the server can still return a clean hasMore: true sentinel when the configured event ceiling is reached. That is a valid bounded response, not an interrupted transport, so this PR returns it unchanged. Only a body that ends without a complete sentinel is resumed.

The lazy-hook preload path requires a complete replay snapshot and intentionally rejects a bounded preload so the runtime can fall back to its normal replay-loading path.

Review scope

This PR owns transport recovery only. It does not introduce an event observer or prepare payloads. The documented onEvent API is introduced in #3548, where it is first consumed.

Stack

  1. [world-vercel] Resume partial replay streams after validated events #3546 — replay stream recovery
  2. [core] Simplify replay payload preparation #3547 — replay codec simplification
  3. [core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
  4. [core] Trace fresh workflow replay phases #3523 — replay phase tracing
  5. [core] Retain workflow VMs across hooks #3604 — retain workflow VMs across hooks
  6. [core] Retain workflow VMs across attributes #3609 — retain workflow VMs across attributes
  7. [core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
  8. [workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles
  9. [workflow] Share isolated serializer registries across replay bundles #3639 — shared serializer bootstrap

Validation

  • pnpm --filter @workflow/world-vercel... build
  • pnpm --filter @workflow/world-vercel test — 542 tests passed
  • git diff --check
  • autoreview of the final materialized-body accounting commit (Codex, gpt-5.6-sol, xhigh): clean, no accepted/actionable P0–P2 findings; correctness confidence 0.91
  • GitHub checks for the current head are running

@changeset-bot

changeset-bot Bot commented Aug 14, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c73a1d0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages
Name Type
@workflow/world-vercel Patch
@workflow/cli Patch
@workflow/core Patch
@workflow/web Patch
workflow Patch
@workflow/world-testing Patch
@workflow/builders Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/nuxt Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
example-nextjs-workflow-turbopack Ready Ready Preview, v0 Aug 21, 2026 8:54pm
example-nextjs-workflow-webpack Ready Ready Preview, v0 Aug 21, 2026 8:54pm
example-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-astro-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-express-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-fastify-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-hono-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-nestjs-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-nitro-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-nuxt-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-python-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-sveltekit-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-tanstack-start-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workbench-vite-workflow Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workflow-docs Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workflow-swc-playground Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workflow-tarballs Ready Ready Preview, v0 Aug 21, 2026 8:54pm
workflow-web Ready Ready Preview, v0 Aug 21, 2026 8:54pm

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (234 failed)

astro-node (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

astro-quickjs (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

example-node (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

example-quickjs (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

express-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K1Y8VA0GT2SGZ5QCQZT6EG | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

express-quickjs (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

fastify-node (6 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events

fastify-quickjs (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

hono-node (6 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

hono-quickjs (7 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nest-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nest-quickjs (12 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands

nextjs-turbopack-node (5 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes Promise.all of disjoint-key writes: every key lands

nextjs-turbopack-quickjs (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nextjs-webpack-node (9 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

nextjs-webpack-quickjs (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nitro-node (5 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0K1X86F0GJSKTBEYMPYNRW1 | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K1Y8VA0GT2SGZ5QCQZT6EG | 🔍 observability
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nitro-quickjs (12 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

nuxt-node (6 failed):

  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

nuxt-quickjs (11 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M0K1JT7Z0GJ163X7NKPP83ET | 🔍 observability
  • sleepingWorkflow | wrun_41M0K1MWVF0GQYRFSHHRCB573R | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0K1MXY00GWVQYR6FMRKV198 | 🔍 observability
  • nullByteWorkflow | wrun_41M0K1NW7G0GG0G279Z2C2PNKC | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0K22N1C0GNAW9CWAFFNGFXT | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K22TA00GX2PYHJENQS4EPW | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K239E20GZ9GF4BNW2XA7XV | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K24F8H0GTAV7FPX0H6AXAV | 🔍 observability

sveltekit-node (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

sveltekit-quickjs (13 failed):

  • webhookWorkflow | wrun_41M0K1KXTD0GH9VKRT2F5X8RPE | 🔍 observability
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting
  • setAttributes Promise.all of disjoint-key writes: every key lands
  • setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
  • setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

tanstack-start-node (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1W89F0GGB5Z04JGRD0F7M | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

tanstack-start-quickjs (8 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

vite-node (7 failed):

  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0K1Y8VA0GT2SGZ5QCQZT6EG | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0K25EE60GVAAZB5CMDEPTA7 | 🔍 observability
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

vite-quickjs (10 failed):

  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1X5EP0GKA2BHKFBEBCKSJ | 🔍 observability
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1WGNV0GT65B67R6Z7AAT0 | 🔍 observability
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1XBC80GWAXXG56M9HBF5T | 🔍 observability
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • setAttributes start: initial attributes are seeded on run creation
  • setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
  • setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
  • setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
  • setAttributes fire-and-forget: void setAttributes lands without awaiting

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1Y0NG3N8EJWGF0ZE9QVS6
  • promiseAllWorkflow | wrun_41M0K1JT7Z0GJ163X7NKPP83ET
  • sleepingWorkflow | wrun_41M0K1MWVF0GQYRFSHHRCB573R
  • parallelSleepWorkflow | wrun_41M0K1MXY00GWVQYR6FMRKV198
  • nullByteWorkflow | wrun_41M0K1NW7G0GG0G279Z2C2PNKC
  • cancelRun - cancelling a running workflow | wrun_41M0K22N1C0GNAW9CWAFFNGFXT
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0K22TA00GX2PYHJENQS4EPW
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K239E20GZ9GF4BNW2XA7XV
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K24F8H0GTAV7FPX0H6AXAV

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

46 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nextjs-webpack)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (tanstack-start)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (example)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nitro)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nuxt)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (tanstack-start)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (express)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (tanstack-start)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (express)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (nextjs-webpack)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (tanstack-start)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (express)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (nest)
  • abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires (tanstack-start)
  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously (fastify)
  • distributedAbortController - reconnect to existing controller (fastify)
  • fire-and-forget: void setAttributes lands without awaiting (nextjs-turbopack)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (fastify)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (hono)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (hono)
  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nitro)
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook (nest)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (nest)
  • Promise.all of disjoint-key writes: every key lands (astro)
  • Promise.all of disjoint-key writes: every key lands (example)
  • Promise.all of disjoint-key writes: every key lands (express)
  • Promise.all of disjoint-key writes: every key lands (nuxt)
  • Promise.all of disjoint-key writes: every key lands (tanstack-start)
  • promiseRaceWorkflow (nextjs-webpack)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (fastify)
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run (sveltekit)
  • setAttributesInsideStepWorkflow: step-body calls append attributed native events (sveltekit)
  • sleepWinsRaceWorkflow (tanstack-start)
  • start: initial attributes are seeded on run creation (nextjs-turbopack)
  • start: reserved-prefix initial attributes are seeded with allowReservedAttributes (nextjs-turbopack)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (astro)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (example)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nest)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (sveltekit)
  • validation DX: invalid writes throw catchable FatalErrors naming rule and limit (tanstack-start)
  • workflow throw of a non-Error value round-trips verbatim as cause (nextjs-turbopack)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (example)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (hono)
  • workflow throws after awaited setAttributes: attribute still persists on the failed run (nest)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

32 infra events
  • cold-start-warmup · suite warmup (python) · at 20:54:26Z · abandoned wrun_41M0K1HBR80GNXQ5VAMKWZMY55 · (+7 more)
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:54:42Z · abandoned wrun_41M0K1N0DT0GQ06RCZDVED8H5P
  • run-pickup-stall · sleepingWorkflow (python) · at 20:54:42Z · abandoned wrun_41M0K1N0E00GYPSD244NK3WTXV
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:42Z · abandoned wrun_41M0K1N0E00GYPSD244NK3WTXW
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:42Z · abandoned wrun_41M0K1N0VR0GN7ZGCG3AFF9H6R
  • run-pickup-stall · nullByteWorkflow (python) · at 20:54:43Z · abandoned wrun_41M0K1N0E30GTDY23S221BV2ED
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:15Z · abandoned wrun_41M0K1P0AP0GGE8DJTM7J73FND
  • run-pickup-stall · nullByteWorkflow (python) · at 20:55:43Z · abandoned wrun_41M0K1PVQ60GG9BR4Y3PABZ104
  • run-pickup-stall · sleepingWorkflow (python) · at 20:55:43Z · abandoned wrun_41M0K1PVW10GJNNXSRX5PVKH5T
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:43Z · abandoned wrun_41M0K1PW150GQWW7FF0PDJG3W9
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:55:43Z · abandoned wrun_41M0K1PW1F0GJPZDTP3WWSMTT5
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:24Z · abandoned wrun_41M0K1Q08W0GQ5JQ20J0FCDTQJ
  • cold-start-warmup · suite warmup (tanstack-start) · at 20:57:44Z · abandoned wrun_01M0K1T6VQKYAWTVWNS5XVB2WM
  • cold-start-warmup · suite warmup (python) · at 20:58:06Z · abandoned wrun_01M0K1R27AEXN7KZEZRE8BYTZR · (+7 more)
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:14Z · abandoned wrun_41M0K1SR5E0GSG3PMJCMC3GPCN
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:14Z · abandoned wrun_41M0K1SKKS0GP4643C8CG9J98Q
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:58:21Z · abandoned wrun_01M0K1VQC1V702K45P84JSCK28
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:58:21Z · abandoned wrun_01M0K1VQC28KZF2SHAXAQTVDQ8
  • run-pickup-stall · sleepingWorkflow (python) · at 20:58:21Z · abandoned wrun_01M0K1VQC83KEX96MBGHGCJB1S
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:58:21Z · abandoned wrun_01M0K1VQC997D0C970CHDDRW6Q
  • run-pickup-stall · nullByteWorkflow (python) · at 20:58:21Z · abandoned wrun_01M0K1VQCBDPKHF2G6M2XV8XB7
  • run-pickup-stall · promiseAllWorkflow (python) · at 20:59:21Z · abandoned wrun_01M0K1XHZYZ1CPYA1CA8NJVNRE
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:21Z · abandoned wrun_01M0K1XHZPQAW247NE198070MY
  • run-pickup-stall · sleepingWorkflow (python) · at 20:59:21Z · abandoned wrun_01M0K1XJ035Y49TD38JY122HJJ
  • run-pickup-stall · nullByteWorkflow (python) · at 20:59:21Z · abandoned wrun_01M0K1XJ0A95X21WX97Q4051V9
  • run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:21Z · abandoned wrun_01M0K1XJ06NT926ZYJBKCKGDD1
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:00:21Z · abandoned wrun_01M0K1ZCKD9MPEKXCFSGF2DVSD
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:00:21Z · abandoned wrun_01M0K1ZCKSBPZ0E6AQS1XCPG2C
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:00:21Z · abandoned wrun_01M0K1ZCKVFK85DTZ4A27G0NN0
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:00:51Z · abandoned wrun_01M0K209YWTCG4HAVJZCXAM4PX
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:00:51Z · abandoned wrun_01M0K209Z7JTSD61FPBQNQWVZR
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:21Z · abandoned wrun_01M0K2176ZV5S0KR3ZVMAM5D0F

E2E Test Summary

Summary
Passed Failed Skipped Total
❌ ▲ Vercel Production 3344 234 742 4320
✅ 💻 Local Development 3922 0 558 4480
✅ 📦 Local Production 3922 0 558 4480
✅ 🐘 Local Postgres 3922 0 558 4480
✅ 🪟 Windows 320 0 0 320
❌ 🌐 Cross-language Conformance 0 9 132 141
✅ vercel-multi-region 27 0 0 27
Total 15457 243 2548 18248
Details by Category

❌ ▲ Vercel Production

App Passed Failed Skipped
❌ astro-node 125 7 28
❌ astro-quickjs 121 11 28
❌ example-node 121 11 28
❌ example-quickjs 122 10 28
❌ express-node 123 9 28
❌ express-quickjs 121 11 28
❌ fastify-node 126 6 28
❌ fastify-quickjs 125 7 28
❌ hono-node 126 6 28
❌ hono-quickjs 125 7 28
❌ nest-node 123 9 28
❌ nest-quickjs 120 12 28
❌ nextjs-turbopack-node 152 5 3
❌ nextjs-turbopack-quickjs 149 8 3
❌ nextjs-webpack-node 148 9 3
❌ nextjs-webpack-quickjs 149 8 3
❌ nitro-node 127 5 28
❌ nitro-quickjs 120 12 28
❌ nuxt-node 126 6 28
❌ nuxt-quickjs 121 11 28
❌ python-node 0 8 152
❌ sveltekit-node 141 10 9
❌ sveltekit-quickjs 138 13 9
❌ tanstack-start-node 124 8 28
❌ tanstack-start-quickjs 124 8 28
❌ vite-node 125 7 28
❌ vite-quickjs 122 10 28

✅ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 160 0 0
✅ nextjs-turbopack-quickjs 160 0 0

❌ 🌐 Cross-language Conformance

App Passed Failed Skipped
❌ python 0 9 132

✅ vercel-multi-region

App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

📋 View full workflow run

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit c73a1d0 · Fri, 21 Aug 2026 21:08:48 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 1123 (+66%) 🔻 1275 🔴 (+24%) 🔻 1377 🔴 (+24%) 🔻 1595 🔴 (+6.0%) 30
TTFS stream 317 (+15%) 🔻 1431 🔴 (+31%) 🔻 1504 🔴 (+37%) 🔻 1653 🔴 (+46%) 🔻 30
TTFS hook + stream 1530 (+20%) 🔻 1705 🔴 (+19%) 🔻 1724 🔴 (+3.7%) 1809 🔴 (-61%) 💚 30
Fan-out TTFS Promise.all(100 steps) 607 (+40%) 🔻 999 (+61%) 🔻 1061 (+58%) 🔻 1938 (-1.0%) 10
Fan-out TTLS Promise.all(100 steps) 1866 (-57%) 💚 3147 (-58%) 💚 4377 (-43%) 💚 5447 (-38%) 💚 10
STSO 1020 steps (inline) 107 (-16%) 💚 175 (-28%) 💚 193 (-32%) 💚 284 (-38%) 💚 1019
WO 1020 steps 172707 (-25%) 💚 172707 (-25%) 💚 172707 (-25%) 💚 172707 (-25%) 💚 1
CRTT first chunk (pooled) 83 (-5.7%) 161 (+25%) 🔻 191 (+27%) 🔻 331 (-18%) 💚 28

Streams

Scenario CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 109 (+2%) 119 (-3%) 156 (-26%) 820 (+62%) 122 (-3%) 10
size sweep (100/s, 160B-12KB) 140 (+35%) 155 (+18%) 331 (+50%) 587 (+9%) 243 (+47%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 145 (+7%) 117 (-14%) 146 (-28%) 327 (-41%) 272 (+89%) 3
replay eve-gpt-5.6-sol-2000t (1x) 149 (-42%) 127 (-49%) 165 (-88%) 665 (-85%) 497 (-79%) 2
replay eve-gpt-5.6-sol-2000t (2x) 124 (-17%) 483 (+171%) 831 (+166%) 1400 (+44%) 729 (+26%) 3
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 231540ms → this run 172562ms (Δ -58978ms, -25%)

100-150 ms  █░░┃                      main   6  this 126  +120
150-200 ms  ██████████░░░░░░░░░░░░░┃  main 344  this 814  +470
200-250 ms  █┃███████████             main 451  this  59  -392
250-300 ms  ┃███                      main 144  this  12  -132
300-350 ms  ┃                         main  45  this   4   -41
350-400 ms  ┃                         main   7  this   0    -7
400-450 ms  ┃                         main  10  this   1    -9
450-500 ms  ┃                         main   5  this   0    -5
500-550 ms  ┃                         main   1  this   1    +0
550-600 ms  ┃                         main   2  this   1    -1
600-650 ms  ┃                         main   1  this   0    -1
700-750 ms  ┃                         main   2  this   0    -2
800-850 ms  ┃                         main   1  this   1    +0
📈 CRTT drill-down vs main (RTT distributions & profiles)
variant  RTT 1ms→5s+             avg         p50          p90          p99     n
control  ······█▇▁▁···   116.1 (+5%)    97 (-1%)   156 (-26%)   820 (+62%)  3000
sweep    ······██▂▁···     126 (+7%)   107 (+3%)   331 (+50%)    587 (+9%)  3000
gw 1x    ·····▁█▆▁····  101.5 (-10%)    99 (±0%)   146 (-28%)   327 (-41%)  5295
eve 1x   ·····▁██▁▁▁··  116.2 (-63%)    98 (-6%)   165 (-88%)   665 (-85%)  5186
eve 2x   ·····▁▃█▅▂▁··  276.3 (+80%)  165 (+30%)  831 (+166%)  1400 (+44%)  7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control  ▅█▆▄▃▁▁▂▃▂  90–166ms
sweep    █▆▅▃▂▆▄▂▁▂  111–148ms
gw 1x    ▃▁▆▂▃▁▁█▂▄  94–120ms
eve 1x   ▂▁▃▅▁▂▃▅▂█  87–176ms
eve 2x   ▂▃▄▃▂▂▁█▇▂  160–500ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep  ▆▇█▆▄▁▄  122–129ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control  ██▄█▃▃▁▆▄▂  26–37ms
sweep    ▁▆█▃▄▆▃▃▄▃  40–63ms
gw 1x    ▃▂▅▁▃▃▃█▁▄  26–37ms
eve 1x   ▁▁▄▅▂▂▃▆▂█  20–33ms
eve 2x   ▅█▇▅▅▄▄▇▁▃  24–44ms
ℹ️ Metric definitions & methodology

Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). = main, = this run, = fill.

The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@NathanColosimo NathanColosimo changed the title Resume partial replay event streams [world-vercel] Resume partial replay streams after validated events Aug 14, 2026
@NathanColosimo
NathanColosimo force-pushed the codex/replay-stream-recovery branch from e9e7488 to dbdc9a2 Compare August 14, 2026 03:56
@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario outcome events virt replay violations
smoke-no-steps completed 3 0ms ok 0
smoke-one-step completed 6 0ms ok 0
hook-at-step-started completed 12 0ms ok 0
hook-at-step-completed completed 12 0ms ok 0
hook-at-hook-created completed 12 0ms ok 0
deadline-hook-wins completed 7 1.0h ok 0
deadline-expires completed 7 1.0h ok 0
long-sleep completed 11 30.0d ok 0
hook-never-arrives stalled 3 0ms skipped 0
step-retries-twice completed 10 2.0s ok 0
parallel-steps completed 9 0ms ok 0
hook-on-execution-state completed 12 0ms ok 0
peek-hook-before-branch completed 12 0ms ok 0
peek-hook-after-branch completed 12 0ms ok 0
peek-hook-at-registration completed 12 0ms ok 0
race-hook-before-probe completed 12 0ms ok 0
race-hook-after-probe completed 12 0ms ok 0
race-duplicate-delivery completed 13 0ms ok 0
attr-hook-before-step completed 11 0ms ok 0
attr-hook-after-step completed 11 0ms ok 0
attr-from-step-body completed 13 0ms ok 0
fork-hook-after-timeout completed 14 1.0m ok 0
fork-hook-before-timeout completed 14 1.0m ok 0
count-hook-after-timeout completed 17 1.0m ok 0
count-hook-before-timeout completed 20 1.0m ok 0
stale-read-step-count-fork completed 20 1.0m ok 0
stale-read-equal-step-counts completed 14 1.0m ok 0
step-vs-step-fork completed 12 0ms ok 0
step-vs-step-fork-fenced completed 12 0ms ok 0
fence-catches-benign-direction completed 12 5ms ok 0
in-flight-before-decision completed 17 1.0m ok 0
in-flight-before-decision-counted completed 17 1.0m ok 0
in-flight-after-decision completed 19 2.0m ok 0
stale-read-step-count-fork-fenced completed 20 1.0m ok 0
fork-hook-wins completed 13 1.0m ok 0
fork-timeout-wins completed 13 1.0m ok 0
unclaimed-payload-under-fork completed 17 1.0m ok 0
claimed-payload-under-fork completed 17 1.0m ok 0
writers-independent-step-bodies completed 12 0ms ok 0
writers-scripted-tempo completed 12 0ms ok 0
cancel-mid-step cancelled 7 0ms skipped 0

Full trace: world-sim.txt

@VaguelySerious VaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, will let agent approve if no blockers found

@VaguelySerious VaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: blocking issues found

params.limit !== undefined ||
!lastEvent ||
`eid:${lastEvent.eventId}` === cursor
retries === MAX_PARTIAL_EVENT_STREAM_RETRIES ||

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Blocking

This loop now retries failures that made zero forward progress, with no backoff. Previously the guard was !lastEvent || \eid:${lastEvent.eventId}` === cursor, so a 5xx or transport failure arriving before the first frame threw immediately. Now the only gates are the retry counter, an explicit limit, and isRetryableEventRequestError— and that predicate returnstrueforcode === 'TRANSPORT'and for anystatus` in 500-599.

Measured on this branch vs origin/main, one getWorkflowRunEventsV4 call against an origin that only answers 503:

wire requests elapsed
main 1 -
this branch 4 32 ms

(undici MockAgent, .persist()ing a 503 on the LIST path, counting interceptor hits.)

So every event LIST — the hottest read path, one per replay — becomes a 4x immediate amplifier, and it amplifies precisely when the backend is already returning 5xx. There is no sleep between iterations, so the four requests leave in the same tick. Queue redelivery multiplies on top of that.

The truncated-stream resumption this PR is for only needs the retry when the cursor actually advanced. Two options: restore the forward-progress condition alongside the new counter, or keep the no-progress retry but require a backoff and restrict it to PartialEventStreamError/IncompleteFrameError rather than all of isRetryableEventRequestError.

Related: MAX_PARTIAL_EVENT_STREAM_RETRIES now also bounds whole-request retries that have nothing to do with a partial stream, so the name has drifted from what it gates.

throw new Error(`v4 ${opName}: unexpected control frame`);
): Promise<{ cursor: string | null; hasMore: boolean }> {
try {
for await (const frame of decodeFrames(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

Dropping the materialized-response branch makes the client assume the server can never decline the replay stream. The backend still has a live decline path: when a hook_received carries resumeId + digest + the frame Accept but materialization returns no run state, it deliberately falls through to the CBOR mutation response and records workflow.hook_received_replay_stream_declined. The comment there is explicit that throwing would 500 a write that already succeeded and let each redelivery append another duplicate.

On that path this client now reaches eventFrameChunks, sees a non-frame content type, and throws a plain Error. That is not an IncompleteFrameError, so consumeEventFrameStream rethrows it unchanged, postReplayLogEvent does not resume it, and isRetryableEventPostError returns false — a hard failure after the write landed, which is the exact outcome the server-side fallback exists to avoid.

I checked how reachable it is: the decline requires a persisted pre-event-sourcing spec version, and there have been no hook_received events below specVersion 2 in the last 30 days, so it is dormant today. The concern is the coupling rather than current traffic — nothing in either repo records that the server may no longer decline, so the next decline reason added server-side breaks every deployed SDK carrying this change (and skew protection keeps those alive a long time). Worth either a comment here naming the contract or keeping a narrow fallback.

get: (runId, eventId, params) => getEvent(runId, eventId, params, config),
list: (params) => getWorkflowRunEvents(params, config),
listByCorrelationId: (params) => getWorkflowRunEvents(params, config),
listByCorrelationId: (params) =>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

This is a real bug fix hiding in a transport PR: listByCorrelationId was routed to getWorkflowRunEvents, so it ignored correlationId entirely and returned the whole run's event log. It now reaches the correlation-id endpoint.

It is unmentioned in the PR body and in the changeset (which only describes stream resumption), and no test covers it — listByCorrelationId appears nowhere under packages/world-vercel/src/*.test.ts. Given the correlation-id-per-run scoping this path exists for, a test asserting the request carries correlationId and that the response is filtered to params.runId would be worth having, plus a changeset line so it shows up in the changelog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants