[core] Trace fresh workflow replay phases by NathanColosimo · Pull Request #3523 · vercel/workflow

NathanColosimo · 2026-08-13T18:14:37Z

Summary

add focused spans for VM context creation, bundle compilation, bundle evaluation, input hydration, and replay execution
start the replay execution span before invoking user code so synchronous-before-first-await work is included
report whether the application bundle actually hit the compiled vm.Script cache
avoid a separate streamed-payload-preparation span: preparation intentionally overlaps event loading, while input hydration reports any residual blocking work
retain PR [core] Trace fresh workflow replay phases #3523 as the top layer so its deployed benchmark and trace history remain attached

Production trace shape

On 388 fresh-replay samples from the Next/Turbopack behavior run, median phase time was approximately:

3.0 ms — workflow.vm.create_context
33.0 ms — workflow.bundle.compile
91.4 ms — workflow.bundle.evaluate
1.2 ms — workflow.input.hydrate
18.4 ms — workflow.replay.execute

The earlier 6.3 ms workflow.replay.prepare_payloads measurement represented an overlapping wall-time window rather than a blocking phase, so this revision removes that span instead of preserving a misleading trace shape.

This PR is intentionally observability-only against its base. The performance implementation is reviewed in #3547 and #3548.

Stack

[world-vercel] Resume partial replay streams after validated events #3546 — replay stream recovery
[core] Simplify replay payload preparation #3547 — synchronous replay decode
[core] Prepare replay payloads as event frames arrive #3548 — streamed replay preparation
[core] Trace fresh workflow replay phases #3523 — replay phase tracing
[core] Prune schema modules from workflow bundles #3550 — prune sandbox dependency graph
[workflow] Load per-source replay bundles #3551 — per-source lazy VM bundles

Validation

core typecheck passed
full core suite: 2,116 tests passed, with 3 expected failures
focused tracing, script-cache, and replay-cache suites: 35 tests passed

changeset-bot · 2026-08-13T18:14:45Z

🦋 Changeset detected

Latest commit: 1a35510

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
@workflow/core	Patch
@workflow/builders	Patch
@workflow/cli	Patch
@workflow/next	Patch
@workflow/nitro	Patch
@workflow/vitest	Patch
@workflow/web-shared	Patch
@workflow/web	Patch
workflow	Patch
@workflow/world-testing	Patch
@workflow/astro	Patch
@workflow/nest	Patch
@workflow/rollup	Patch
@workflow/sveltekit	Patch
@workflow/vite	Patch
@workflow/nuxt	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2026-08-13T18:14:47Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
example-nextjs-workflow-turbopack	Ready	Preview, v0	Aug 21, 2026 8:54pm
example-nextjs-workflow-webpack	Ready	Preview, v0	Aug 21, 2026 8:54pm
example-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-astro-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-express-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-fastify-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-hono-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-nestjs-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-nitro-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-nuxt-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-python-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-sveltekit-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-tanstack-start-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workbench-vite-workflow	Ready	Preview, v0	Aug 21, 2026 8:54pm
workflow-docs	Ready	Preview, v0	Aug 21, 2026 8:54pm
workflow-swc-playground	Ready	Preview, v0	Aug 21, 2026 8:54pm
workflow-tarballs	Ready	Preview, v0	Aug 21, 2026 8:54pm
workflow-web	Ready	Preview, v0	Aug 21, 2026 8:54pm

github-actions · 2026-08-13T18:14:56Z

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (244 failed)

astro-node (8 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting

astro-quickjs (11 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes Promise.all of disjoint-key writes: every key lands
setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

example-node (9 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
distributedAbortController - manual abort triggers signal | wrun_41M0K23M5D0GK1235ZCKHS2R1Z | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes fire-and-forget: void setAttributes lands without awaiting

example-quickjs (12 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes Promise.all of disjoint-key writes: every key lands
setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

express-node (8 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes Promise.all of disjoint-key writes: every key lands

express-quickjs (10 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting

fastify-node (11 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

fastify-quickjs (8 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting

hono-node (9 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes Promise.all of disjoint-key writes: every key lands

hono-quickjs (10 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes Promise.all of disjoint-key writes: every key lands

nest-node (9 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes Promise.all of disjoint-key writes: every key lands

nest-quickjs (13 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes Promise.all of disjoint-key writes: every key lands

nextjs-turbopack-node (3 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events

nextjs-turbopack-quickjs (9 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting

nextjs-webpack-node (7 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes fire-and-forget: void setAttributes lands without awaiting

nextjs-webpack-quickjs (13 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes Promise.all of disjoint-key writes: every key lands
setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

nitro-node (11 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run
setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

nitro-quickjs (10 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

nuxt-node (7 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes fire-and-forget: void setAttributes lands without awaiting

nuxt-quickjs (10 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes validation DX: invalid writes throw catchable FatalErrors naming rule and limit

python-node (8 failed):

promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3 | 🔍 observability
sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3 | 🔍 observability
parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1 | 🔍 observability
nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD | 🔍 observability
cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56 | 🔍 observability
cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4 | 🔍 observability
sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK | 🔍 observability
resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z | 🔍 observability

sveltekit-node (7 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes fire-and-forget: void setAttributes lands without awaiting

sveltekit-quickjs (4 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes fire-and-forget: void setAttributes lands without awaiting

tanstack-start-node (9 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0K1TQQR0GVY3EHX0XZADW9H | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes workflow throws after awaited setAttributes: attribute still persists on the failed run

tanstack-start-quickjs (9 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
setAttributes start: initial attributes are seeded on run creation
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes Promise.all of disjoint-key writes: every key lands

vite-node (7 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_41M0K1VPH60GP1SW215ZKB165G | 🔍 observability
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes fire-and-forget: void setAttributes lands without awaiting

vite-quickjs (12 failed):

hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_41M0K1VAA50GSYHPCVWPV3EDR8 | 🔍 observability
concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_41M0K1TMCF0GYNYN1PVDK214NC | 🔍 observability
AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
setAttributes start: initial attributes are seeded on run creation
setAttributes start: reserved-prefix initial attributes are seeded with allowReservedAttributes
setAttributes setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly
setAttributes setAttributesInsideStepWorkflow: step-body calls append attributed native events
setAttributes fire-and-forget: void setAttributes lands without awaiting
setAttributes Promise.all of disjoint-key writes: every key lands

🌐 Cross-language Conformance (9 failed)

python (9 failed):

deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0K1ZE1VVFB967MCQ01CD795
promiseAllWorkflow | wrun_41M0K1JNVY0GGFK9N52C9G7CY3
sleepingWorkflow | wrun_41M0K1MNFA0GV59M7717VSGAA3
parallelSleepWorkflow | wrun_41M0K1MQB00GMS3TKYEC5N1DF1
nullByteWorkflow | wrun_41M0K1NME20GQ1YF6ZB1TBE6BD
cancelRun - cancelling a running workflow | wrun_41M0K201ED0GGS69K43HRNBQ56
cancelRun via CLI - cancelling a running workflow | wrun_41M0K207EW0GJ8H5APWJK11BW4
sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0K20X690GP6MMCQ5FG0BNVK
resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0K22KVF0GS8NG6HQMHFQM0Z

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

46 flaky tests

abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nest)
abortFromStepWorkflow: step abort cancels an in-flight sibling step (nest)
abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (sveltekit)
addTenWorkflow (astro)
addTenWorkflow (fastify)
cancelRun via CLI - cancelling a running workflow (sveltekit)
distributedAbortController - manual abort triggers signal (express)
distributedAbortController - reconnect to existing controller (example)
distributedAbortController - reconnect to existing controller (nest)
hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (fastify)
hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nuxt)
hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data (nextjs-webpack)
hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps (nextjs-webpack)
hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (nextjs-webpack)
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (express)
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (nextjs-turbopack)
hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable (tanstack-start)
no startIndex (reads all chunks) (express)
parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (fastify)
parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race (nextjs-webpack)
Promise.all of disjoint-key writes: every key lands (example)
Promise.all of disjoint-key writes: every key lands (fastify)
Promise.all of disjoint-key writes: every key lands (sveltekit)
resume-or-start route pattern - resumeHook retried after start() reaches the new run (express)
resume-or-start route pattern - resumeHook retried after start() reaches the new run (nextjs-webpack)
setAttributesInsideStepWorkflow: step-body calls append attributed native events (example)
setAttributesInsideStepWorkflow: step-body calls append attributed native events (express)
setAttributesInsideStepWorkflow: step-body calls append attributed native events (nextjs-webpack)
setAttributesInsideStepWorkflow: step-body calls append attributed native events (vite)
setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nextjs-turbopack)
setAttributesWorkflow: workflow-body calls append native attr_set events and merge correctly (nitro)
sleepWinsRaceWorkflow (tanstack-start)
sleepWinsRaceWorkflow (vite)
start: initial attributes are seeded on run creation (nextjs-turbopack)
start: reserved-prefix initial attributes are seeded with allowReservedAttributes (tanstack-start)
step throw of a non-Error value preserves it as cause on the wrapping FatalError (nextjs-turbopack)
stepWinsRaceWorkflow (vite)
validation DX: invalid writes throw catchable FatalErrors naming rule and limit (hono)
validation DX: invalid writes throw catchable FatalErrors naming rule and limit (nest)
validation DX: invalid writes throw catchable FatalErrors naming rule and limit (tanstack-start)
webhookWorkflow (hono)
workflow throws after awaited setAttributes: attribute still persists on the failed run (astro)
workflow throws after awaited setAttributes: attribute still persists on the failed run (hono)
workflow throws after awaited setAttributes: attribute still persists on the failed run (nest)
workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt)
workflow throws after awaited setAttributes: attribute still persists on the failed run (tanstack-start)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

32 infra events

cold-start-warmup · suite warmup (python) · at 20:54:40Z · abandoned wrun_41M0K1HSDJ0GME5BYMX59CNX6K · (+7 more)
run-pickup-stall · promiseAllWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFQ0GKFWCKBYBX5Q6TF
run-pickup-stall · nullByteWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEG00GYMTSZ31K3DAPBQ
run-pickup-stall · parallelSleepWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1M
run-pickup-stall · sleepingWorkflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NEFX0GR8J9V38R68QC1K
run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:54:57Z · abandoned wrun_41M0K1NF1Q0GMAYMPP86V96QQ6
run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 20:55:29Z · abandoned wrun_41M0K1PEDW0GZE4QZ5CFW30S42
run-pickup-stall · parallelSleepWorkflow (python) · at 20:55:57Z · abandoned wrun_41M0K1Q9SP0GTHYAKV39VBFX6P
run-pickup-stall · sleepingWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9SV0GK5F06DGM51605Q
run-pickup-stall · nullByteWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9W10GKRAC9NR0E87KFY
run-pickup-stall · promiseAllWorkflow (python) · at 20:55:58Z · abandoned wrun_41M0K1Q9V00GW4CNMJKV4AXAGS
run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:56:48Z · abandoned wrun_41M0K1QDWZ0GXPT291HQ94V9K2
run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 20:58:29Z · abandoned wrun_41M0K1T5SG0GRDC5WEMPDD22EV
run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 20:58:29Z · abandoned wrun_41M0K1T1P70GHK6GE0X7XYSHXJ
cold-start-warmup · suite warmup (python) · at 20:58:53Z · abandoned wrun_01M0K1SFM5SAP36XBNNF6BAB06 · (+7 more)
cold-start-warmup · suite warmup (tanstack-start) · at 20:59:04Z · abandoned wrun_01M0K1WMSYDP0A4AN0EZ2YNE7B
run-pickup-stall · promiseAllWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RSD3H8J0V61PC9W5MK
run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RRH9X8HXPZKXSEWHR3
run-pickup-stall · parallelSleepWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S01X7VQ0X3SWF6FY11
run-pickup-stall · sleepingWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4RYSY1H6DS6DJP1AJCJ
run-pickup-stall · nullByteWorkflow (python) · at 20:59:08Z · abandoned wrun_01M0K1X4S3C0M5KZEW6DRBHZ83
run-pickup-stall · parallelSleepWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCR8A6B5DQXHZ0XTPEG
run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCTEEFAHZFWC64SGN5T
run-pickup-stall · nullByteWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCWJRW9Q8395REQRV4X
run-pickup-stall · promiseAllWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCNNJ4CY13PGCBXDGSB
run-pickup-stall · sleepingWorkflow (python) · at 21:00:08Z · abandoned wrun_01M0K1YZCQJ9PV2MH9BM4DJF6A
run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0BSDW3JADX703S5W05
run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:08Z · abandoned wrun_01M0K20T0FWPYY292RKBY8CC9X
run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:01:08Z · abandoned wrun_01M0K20T0JG89Z3DB7M0QF7DGB
run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBY519CZXPW964N7N4B
run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 21:01:38Z · abandoned wrun_01M0K21QBZXRG7XF8BEYE0CAHE
run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 21:02:08Z · abandoned wrun_01M0K22MKP6JKDYT80XXGFSRPG

E2E Test Summary

Summary

	Passed	Failed	Skipped	Total
❌ ▲ Vercel Production	3334	244	742	4320
✅ 💻 Local Development	3922	0	558	4480
✅ 📦 Local Production	3922	0	558	4480
✅ 🐘 Local Postgres	3922	0	558	4480
✅ 🪟 Windows	320	0	0	320
❌ 🌐 Cross-language Conformance	0	9	132	141
✅ vercel-multi-region	27	0	0	27
Total	15447	253	2548	18248

Details by Category

❌ ▲ Vercel Production

App	Passed	Failed	Skipped
❌ astro-node	124	8	28
❌ astro-quickjs	121	11	28
❌ example-node	123	9	28
❌ example-quickjs	120	12	28
❌ express-node	124	8	28
❌ express-quickjs	122	10	28
❌ fastify-node	121	11	28
❌ fastify-quickjs	124	8	28
❌ hono-node	123	9	28
❌ hono-quickjs	122	10	28
❌ nest-node	123	9	28
❌ nest-quickjs	119	13	28
❌ nextjs-turbopack-node	154	3	3
❌ nextjs-turbopack-quickjs	148	9	3
❌ nextjs-webpack-node	150	7	3
❌ nextjs-webpack-quickjs	144	13	3
❌ nitro-node	121	11	28
❌ nitro-quickjs	122	10	28
❌ nuxt-node	125	7	28
❌ nuxt-quickjs	122	10	28
❌ python-node	0	8	152
❌ sveltekit-node	144	7	9
❌ sveltekit-quickjs	147	4	9
❌ tanstack-start-node	123	9	28
❌ tanstack-start-quickjs	123	9	28
❌ vite-node	125	7	28
❌ vite-quickjs	120	12	28

✅ 💻 Local Development

App	Passed	Skipped
✅ astro-stable-node	134	26
✅ astro-stable-quickjs	134	26
✅ express-stable-node	134	26
✅ express-stable-quickjs	134	26
✅ fastify-stable-node	134	26
✅ fastify-stable-quickjs	134	26
✅ hono-stable-node	134	26
✅ hono-stable-quickjs	134	26
✅ nest-stable-node	134	26
✅ nest-stable-quickjs	134	26
✅ nextjs-turbopack-canary-node	141	19
✅ nextjs-turbopack-canary-quickjs	141	19
✅ nextjs-turbopack-stable-node	160	0
✅ nextjs-turbopack-stable-quickjs	160	0
✅ nextjs-webpack-canary-node	141	19
✅ nextjs-webpack-canary-quickjs	141	19
✅ nextjs-webpack-stable-node	160	0
✅ nextjs-webpack-stable-quickjs	160	0
✅ nitro-stable-node	134	26
✅ nitro-stable-quickjs	134	26
✅ nuxt-stable-node	134	26
✅ nuxt-stable-quickjs	134	26
✅ sveltekit-stable-node	153	7
✅ sveltekit-stable-quickjs	153	7
✅ tanstack-start-node	134	26
✅ tanstack-start-quickjs	134	26
✅ vite-stable-node	134	26
✅ vite-stable-quickjs	134	26

✅ 📦 Local Production

App	Passed	Skipped
✅ astro-stable-node	134	26
✅ astro-stable-quickjs	134	26
✅ express-stable-node	134	26
✅ express-stable-quickjs	134	26
✅ fastify-stable-node	134	26
✅ fastify-stable-quickjs	134	26
✅ hono-stable-node	134	26
✅ hono-stable-quickjs	134	26
✅ nest-stable-node	134	26
✅ nest-stable-quickjs	134	26
✅ nextjs-turbopack-canary-node	141	19
✅ nextjs-turbopack-canary-quickjs	141	19
✅ nextjs-turbopack-stable-node	160	0
✅ nextjs-turbopack-stable-quickjs	160	0
✅ nextjs-webpack-canary-node	141	19
✅ nextjs-webpack-canary-quickjs	141	19
✅ nextjs-webpack-stable-node	160	0
✅ nextjs-webpack-stable-quickjs	160	0
✅ nitro-stable-node	134	26
✅ nitro-stable-quickjs	134	26
✅ nuxt-stable-node	134	26
✅ nuxt-stable-quickjs	134	26
✅ sveltekit-stable-node	153	7
✅ sveltekit-stable-quickjs	153	7
✅ tanstack-start-node	134	26
✅ tanstack-start-quickjs	134	26
✅ vite-stable-node	134	26
✅ vite-stable-quickjs	134	26

✅ 🐘 Local Postgres

App	Passed	Skipped
✅ astro-stable-node	134	26
✅ astro-stable-quickjs	134	26
✅ express-stable-node	134	26
✅ express-stable-quickjs	134	26
✅ fastify-stable-node	134	26
✅ fastify-stable-quickjs	134	26
✅ hono-stable-node	134	26
✅ hono-stable-quickjs	134	26
✅ nest-stable-node	134	26
✅ nest-stable-quickjs	134	26
✅ nextjs-turbopack-canary-node	141	19
✅ nextjs-turbopack-canary-quickjs	141	19
✅ nextjs-turbopack-stable-node	160	0
✅ nextjs-turbopack-stable-quickjs	160	0
✅ nextjs-webpack-canary-node	141	19
✅ nextjs-webpack-canary-quickjs	141	19
✅ nextjs-webpack-stable-node	160	0
✅ nextjs-webpack-stable-quickjs	160	0
✅ nitro-stable-node	134	26
✅ nitro-stable-quickjs	134	26
✅ nuxt-stable-node	134	26
✅ nuxt-stable-quickjs	134	26
✅ sveltekit-stable-node	153	7
✅ sveltekit-stable-quickjs	153	7
✅ tanstack-start-node	134	26
✅ tanstack-start-quickjs	134	26
✅ vite-stable-node	134	26
✅ vite-stable-quickjs	134	26

✅ 🪟 Windows

App	Passed	Failed	Skipped
✅ nextjs-turbopack-node	160	0	0
✅ nextjs-turbopack-quickjs	160	0	0

❌ 🌐 Cross-language Conformance

App	Passed	Failed	Skipped
❌ python	0	9	132

✅ vercel-multi-region

App	Passed	Failed	Skipped
✅ nextjs-turbopack	27	0	0

📋 View full workflow run

github-actions · 2026-08-13T18:15:00Z

📊 Workflow Benchmarks

❌ The benchmark run for 1a35510 failed. See the run logs for details.

commit d74f748 · Thu, 20 Aug 2026 18:02:44 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	248 (-76%) 💚	1391 🔴 (+23%) 🔻	1424 🔴 (+22%) 🔻	1739 🔴 (+42%) 🔻	30
TTFS	stream	277 (+56%) 🔻	1424 🔴 (+27%) 🔻	1473 🔴 (+31%) 🔻	1861 🔴 (+59%) 🔻	30
TTFS	hook + stream	374 (-25%) 💚	1623 🔴 (+13%)	1687 🔴 (+14%)	1835 🔴 (+19%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	557 (+40%) 🔻	967 (+8.0%)	2100 (+20%) 🔻	2302 (+22%) 🔻	10
Fan-out TTLS	Promise.all(100 steps)	4980 (+4.0%)	7820 (+27%) 🔻	8029 (+28%) 🔻	11536 (+60%) 🔻	10
STSO	1020 steps (inline)	135 (+13%)	274 (+11%)	319 (+6.3%)	575 (+14%)	1019
WO	1020 steps	265663 (+12%)	265663 (+12%)	265663 (+12%)	265663 (+12%)	1
CRTT	first chunk (pooled)	101 (+9.8%)	166 (+26%) 🔻	411 (+172%) 🔻	1142 (+592%) 🔻	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	134 (+9%)	298 (+106%)	368 (+39%)	1053 (-5%)	263 (+79%)	10
size sweep (100/s, 160B-12KB)	133 (+2%)	203 (-8%)	482 (-8%)	884 (+14%)	211 (-23%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	138 (+14%)	162 (+10%)	300 (+17%)	645 (-8%)	461 (-11%)	3
replay eve-gpt-5.6-sol-2000t (1x)	314 (+146%)	184 (+21%)	351 (+38%)	797 (-5%)	696 (+39%)	2
replay eve-gpt-5.6-sol-2000t (2x)	147 (+20%)	244 (+16%)	397 (+23%)	675 (-80%)	544 (-10%)	3

📜 Previous results (2)

`50faa3b`

Wed, 19 Aug 2026 17:28:21 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1335 (+365%) 🔻	1423 🔴 (+22%) 🔻	1441 🔴 (+7.6%)	1534 🔴 (-1.9%)	30
TTFS	stream	1342 (+610%) 🔻	1410 🔴 (+26%) 🔻	1430 🔴 (+25%) 🔻	1576 🔴 (+35%) 🔻	30
TTFS	hook + stream	1514 (+16%) 🔻	1693 🔴 (+23%) 🔻	1705 🔴 (+21%) 🔻	2082 🔴 (+21%) 🔻	30
Fan-out TTFS	Promise.all(100 steps)	1088 (-1.5%)	2594 (+13%)	2614 (-5.5%)	2987 (+7.8%)	10
Fan-out TTLS	Promise.all(100 steps)	5278 (±0%)	6182 (-7.1%)	6444 (-6.4%)	8094 (+2.7%)	10
STSO	1020 steps (inline)	120 (-9.1%)	224 (-15%)	249 (-18%) 💚	336 (-33%) 💚	1019
WO	1020 steps	216112 (-13%)	216112 (-13%)	216112 (-13%)	216112 (-13%)	1
CRTT	first chunk (pooled)	80 (-4.8%)	126 (-0.8%)	269 (-15%)	567 (-9.6%)	28

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	99 (-12%)	139 (-16%)	221 (-5%)	430 (-8%)	109 (-32%)	10
size sweep (100/s, 160B-12KB)	108 (±0%)	128 (-17%)	220 (-22%)	604 (-11%)	104 (-36%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	93 (-17%)	159 (-43%)	268 (-63%)	424 (-72%)	368 (-39%)	3
replay eve-gpt-5.6-sol-2000t (1x)	333 (+114%)	129 (-8%)	239 (+24%)	447 (-29%)	302 (-38%)	2
replay eve-gpt-5.6-sol-2000t (2x)	108 (-79%)	344 (-15%)	553 (-80%)	751 (-82%)	509 (-16%)	3

`50faa3b`

Tue, 18 Aug 2026 23:23:51 GMT · run logs

vercel / nextjs-turbopack

Metric	Scenario	Best (ms)	P75 (ms)	P90 (ms)	P99 (ms)	Samples
TTFS	step	1377 (+29%) 🔻	1439 🔴 (+20%) 🔻	1460 🔴 (+15%) 🔻	1585 🔴 (-0.9%)	30
TTFS	stream	225 (-27%) 💚	1436 🔴 (+29%) 🔻	1458 🔴 (+24%) 🔻	1865 🔴 (+51%) 🔻	30
TTFS	hook + stream	346 (-74%) 💚	1711 🔴 (+14%)	1742 🔴 (+12%)	1816 🔴 (-7.3%)	30
Fan-out TTFS	Promise.all(100 steps)	1521 (+25%) 🔻	3074 (+29%) 🔻	3100 (+25%) 🔻	3101 (+15%)	10
Fan-out TTLS	Promise.all(100 steps)	7331 (+37%) 🔻	27947 (+251%) 🔻	310173 (+3739%) 🔻	313908 (+3301%) 🔻	10
CRTT	first chunk (pooled)	108 (-7.7%)	164 (-19%) 💚	213 (-34%) 💚	416 (-34%) 💚	25

Streams

Scenario	CRTT 1st	p75	p90	p99	CDV max	iters
paced control (100/s, 60B)	138 (-15%)	197 (-45%)	272 (-60%)	1221 (-23%)	181 (-52%)	10
size sweep (100/s, 160B-12KB)	147 (-2%)	379 (+94%)	642 (+146%)	933 (-16%)	247 (+54%)	10
replay gateway-gpt-5.4-nano-2000t (1x)	113 (-33%)	196 (-5%)	355 (+9%)	821 (+42%)	242 (-39%)	3
replay eve-gpt-5.6-sol-2000t (1x)	147 (-33%)	156 (-20%)	194 (-30%)	424 (-14%)	302 (-30%)	2

ℹ️ Metric definitions & methodology

_{Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.}

_{Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.}

_{Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)}

_{Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles}

_{Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041}

_{🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600}

_{All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).}

_{Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.}

github-actions · 2026-08-13T18:18:41Z

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario	outcome	events	virt	replay
✅ `smoke-no-steps`	completed	3	0ms	ok
✅ `smoke-one-step`	completed	6	0ms	ok
✅ `hook-at-step-started`	completed	12	0ms	ok
✅ `hook-at-step-completed`	completed	12	0ms	ok
✅ `hook-at-hook-created`	completed	12	0ms	ok
✅ `deadline-hook-wins`	completed	7	1.0h	ok
✅ `deadline-expires`	completed	7	1.0h	ok
✅ `long-sleep`	completed	11	30.0d	ok
✅ `hook-never-arrives`	stalled	3	0ms	skipped
✅ `step-retries-twice`	completed	10	2.0s	ok
✅ `parallel-steps`	completed	9	0ms	ok
✅ `hook-on-execution-state`	completed	12	0ms	ok
✅ `peek-hook-before-branch`	completed	12	0ms	ok
✅ `peek-hook-after-branch`	completed	12	0ms	ok
✅ `peek-hook-at-registration`	completed	12	0ms	ok
✅ `race-hook-before-probe`	completed	12	0ms	ok
✅ `race-hook-after-probe`	completed	12	0ms	ok
✅ `race-duplicate-delivery`	completed	13	0ms	ok
✅ `attr-hook-before-step`	completed	11	0ms	ok
✅ `attr-hook-after-step`	completed	11	0ms	ok
✅ `attr-from-step-body`	completed	13	0ms	ok
✅ `fork-hook-after-timeout`	completed	14	1.0m	ok
✅ `fork-hook-before-timeout`	completed	14	1.0m	ok
✅ `count-hook-after-timeout`	completed	17	1.0m	ok
✅ `count-hook-before-timeout`	completed	20	1.0m	ok
✅ `stale-read-step-count-fork`	completed	20	1.0m	ok
✅ `stale-read-equal-step-counts`	completed	14	1.0m	ok
✅ `step-vs-step-fork`	completed	12	0ms	ok
✅ `step-vs-step-fork-fenced`	completed	12	0ms	ok
✅ `fence-catches-benign-direction`	completed	12	5ms	ok
✅ `in-flight-before-decision`	completed	17	1.0m	ok
❌ `in-flight-before-decision-counted`	completed	17	1.0m	ok
✅ `in-flight-after-decision`	completed	19	2.0m	ok
✅ `stale-read-step-count-fork-fenced`	completed	20	1.0m	ok
✅ `fork-hook-wins`	completed	13	1.0m	ok
✅ `fork-timeout-wins`	completed	13	1.0m	ok
✅ `unclaimed-payload-under-fork`	completed	17	1.0m	ok
✅ `claimed-payload-under-fork`	completed	17	1.0m	ok
✅ `writers-independent-step-bodies`	completed	12	0ms	ok
✅ `writers-scripted-tempo`	completed	12	0ms	ok
✅ `cancel-mid-step`	cancelled	7	0ms	skipped

Full trace: world-sim.txt

VaguelySerious

AI review: no blocking issues

VaguelySerious · 2026-08-21T20:27:24Z

+    'workflow.bundle.compile',
+    async (span) => {
+      const bundle = getCachedWorkflowScript(workflowCode, filename);
+      const lookup = getCachedWorkflowScript(workflowLookupCode, filename);


AI Review: Note

Giving the lookup script its own getCachedWorkflowScript call invalidates the bounding invariant documented in script-cache.ts. That cache is keyed by code at the top level and capped at MAX_BUNDLES = 8, and its doc block asserts: "In production this bound is never reached: a deployment is its own process serving exactly one build-time bundle literal ... so there is a single code key for the process lifetime."

workflowLookupCode embeds the workflow name, so each distinct workflow becomes its own top-level code entry. An app with 8 or more workflows now exceeds the cap in production, and MAX_BUNDLES effectively bounds bundle versions + distinct workflow names rather than bundle versions.

No correctness or real cost impact, and I do not think it needs restructuring: eviction only runs when a new code key is inserted and takes from the front, and touchBundle(workflowCode) moves the bundle to MRU at the start of every session, so the bundle is never the eviction candidate — only the one-line lookup scripts churn, and recompiling those is free. But that doc block is now wrong for any app with 8+ workflows and will mislead the next person tuning this cache. Either update it or key the lookup scripts separately from the bundle LRU.

The WorkflowBundleCompileCacheHit attribute correctly reports only the bundle's hit and the comment already calls out why the lookup script's own hit rate is not the signal — that part is right.

VaguelySerious · 2026-08-21T20:27:24Z

+}
+
+/** Keeps a parked workflow's ambient trace context aligned with each resume. */
+export async function createRefreshableTraceContext() {


AI Review: Note

The returned context is a live proxy over a mutable current, which breaks OTel's immutable-Context contract. Re-parenting a resumed workflow's ambient context is the right goal — spans from a resume should not attach to the trace of the request that first parked the run — but the mechanism has an asymmetry worth documenting.

setValue forwards to current.setValue(...), and OTel contexts are immutable, so that returns a concrete derived context rather than another proxy. The consequence: work whose context is the proxy itself re-parents on refresh(), but anything that derives a context one level deeper (any nested trace() inside the workflow body, which captures the concrete context at creation) does not. So the re-parenting applies to direct children only, and refresh() retroactively changes the parent that already-pending continuations resolve to.

That is defensible for what this is for, but it is subtle enough that the next reader will assume uniform behaviour. A comment on the proxy naming the setValue boundary would save that.

The rest of the span lifecycle checks out: startTraceSpan's ended flag makes the vmTrace.fail in createWorkflowSession's catch a no-op once endVmTrace() has run, so a post-bootstrap throw such as WorkflowNotRegisteredError correctly leaves the VM span OK rather than marking it failed; and void execution.then(replayTrace.end, replayTrace.fail) handles the rejection without competing with the copy returned to the caller.

vercel Bot had a problem deploying to Preview – workbench-python-workflow August 13, 2026 18:14 Failure

vercel Bot deployed to Preview – workflow-web August 13, 2026 18:15 View deployment

vercel Bot deployed to Preview – workflow-tarballs August 13, 2026 18:15 View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow August 13, 2026 18:15 View deployment

vercel Bot deployed to Preview – workbench-nitro-workflow August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – workbench-hono-workflow August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – workbench-express-workflow August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – workbench-astro-workflow August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – workbench-fastify-workflow August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – workbench-sveltekit-workflow August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – workbench-vite-workflow August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – example-workflow August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – workbench-tanstack-start-workflow August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow August 13, 2026 18:16 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack August 13, 2026 18:17 View deployment

vercel Bot deployed to Preview – workflow-docs August 13, 2026 18:17 View deployment

vercel Bot deployed to Preview – workflow-swc-playground August 13, 2026 18:18 View deployment

vercel Bot reviewed Aug 13, 2026

View reviewed changes

Comment thread packages/core/src/replay-payload-cache.ts Outdated

vercel Bot had a problem deploying to Preview – workbench-python-workflow August 13, 2026 23:30 Failure

vercel Bot deployed to Preview – workflow-web August 13, 2026 23:31 View deployment

vercel Bot deployed to Preview – workflow-tarballs August 13, 2026 23:31 View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow August 13, 2026 23:31 View deployment

vercel Bot deployed to Preview – workbench-hono-workflow August 13, 2026 23:32 View deployment

vercel Bot deployed to Preview – workbench-nitro-workflow August 13, 2026 23:32 View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow August 14, 2026 00:28 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack August 14, 2026 00:29 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack August 14, 2026 00:29 View deployment

vercel Bot deployed to Preview – workbench-vite-workflow August 14, 2026 00:29 View deployment

vercel Bot deployed to Preview – workflow-docs August 14, 2026 00:30 View deployment

vercel Bot deployed to Preview – workflow-swc-playground August 14, 2026 00:30 View deployment

vercel Bot had a problem deploying to Preview – workbench-python-workflow August 14, 2026 00:51 Failure

vercel Bot deployed to Preview – workflow-web August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workbench-nestjs-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workbench-nitro-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workbench-hono-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workbench-express-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workbench-fastify-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workbench-vite-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workbench-sveltekit-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – example-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workbench-tanstack-start-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workflow-tarballs August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workbench-nuxt-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – workbench-astro-workflow August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-webpack August 14, 2026 00:52 View deployment

vercel Bot deployed to Preview – example-nextjs-workflow-turbopack August 14, 2026 00:52 View deployment

This was referenced Aug 14, 2026

[world-vercel] Resume partial replay streams after validated events #3546

Open

[core] Prepare replay payloads as event frames arrive #3548

Open

[core] Simplify replay payload preparation #3547

Open

VaguelySerious approved these changes Aug 21, 2026

View reviewed changes

NathanColosimo added 4 commits August 21, 2026 13:49

Trace fresh workflow replay phases

ec86ef9

refactor(core): simplify workflow script cache API

8a0f882

Fix retained workflow tracing

82c1660

fix(core): keep tracing failure-safe

1a35510

Navigation Menu

Uh oh!

[core] Trace fresh workflow replay phases - #3523

[core] Trace fresh workflow replay phases#3523
NathanColosimo wants to merge 4 commits into
codex/replay-stream-preparationfrom
codex/speed-fresh-replay

NathanColosimo commented Aug 13, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 13, 2026 •

edited

Loading

`50faa3b`

`50faa3b`

Uh oh!

github-actions Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

VaguelySerious left a comment

Uh oh!

VaguelySerious Aug 21, 2026

Uh oh!

VaguelySerious Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

NathanColosimo commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Production trace shape

Stack

Validation

Uh oh!

changeset-bot Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

vercel Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 E2E Test Results

❌ Failed E2E Tests

🌐 Cross-language Conformance (9 failed)

⚠️ Flaky E2E Tests (passed on retry)

🛠 Infra Events (absorbed by the harness)

E2E Test Summary

Uh oh!

github-actions Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Workflow Benchmarks

50faa3b

50faa3b

Uh oh!

github-actions Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sim World

Uh oh!

Uh oh!

VaguelySerious left a comment

Choose a reason for hiding this comment

Uh oh!

VaguelySerious Aug 21, 2026

Choose a reason for hiding this comment

AI Review: Note

Uh oh!

VaguelySerious Aug 21, 2026

Choose a reason for hiding this comment

AI Review: Note

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NathanColosimo commented Aug 13, 2026 •

edited

Loading

changeset-bot Bot commented Aug 13, 2026 •

edited

Loading

vercel Bot commented Aug 13, 2026 •

edited

Loading

github-actions Bot commented Aug 13, 2026 •

edited

Loading

github-actions Bot commented Aug 13, 2026 •

edited

Loading

`50faa3b`

`50faa3b`

github-actions Bot commented Aug 13, 2026 •

edited

Loading