Skip to content

[workflow] Load per-source replay bundles - #3551

Draft
NathanColosimo wants to merge 10 commits into
codex/replay-prune-runtime-graphfrom
codex/replay-lazy-source-bundles
Draft

[workflow] Load per-source replay bundles#3551
NathanColosimo wants to merge 10 commits into
codex/replay-prune-runtime-graphfrom
codex/replay-lazy-source-bundles

Conversation

@NathanColosimo

@NathanColosimo NathanColosimo commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • emit one VM bundle per workflow source; workflows in the same source share one bundle
  • keep one flow route with a static workflow-ID → dynamic-import loader map
  • overlap the selected bundle load with world initialization and event setup
  • skip bundle loading for queue-only step deliveries
  • retain compiled scripts for every immutable production source bundle
  • keep watch loaders cache-safe and derive loader IDs from the same transform that emitted each bundle
  • encode generated VM modules as opaque artifacts so framework plugins cannot rewrite inert source

Result

For the Next/Turbopack benchmark app's 97_bench source, measured against the monolithic bundle at the base of this PR:

  • generated flow route: 1,327,514 B → 18,264 B (-98.6%)
  • selected benchmark VM code: 1,307,595 B → 133,859 B (-89.8%)
  • cold compile p50: 10.405 ms → 1.089 ms (-89.5%)
  • fresh-context evaluation p50: 7.484 ms → 0.356 ms (-95.2%)
  • one-time base64 artifact decode p50/p90: 0.037/0.044 ms

The compile/evaluation microbenchmarks use 60 uncached vm.Script compilations and 80 evaluations in fresh VM contexts on Node. The deployment benchmark and APM traces remain the end-to-end signal.

The correctness-preserving split deliberately includes any source that defines both workflows and custom serializers in every other source bundle. The workbench's unusually large 99_e2e.ts is such a hybrid, so the selected benchmark bundle is ~134 KB rather than the ~28 KB it would be without that shared registration. The next structural improvement is to extract serializer registration into a dedicated small VM prelude; omitting it would make replay deserialization incorrect.

The 19 split bundles total 5.28 MB decoded versus the old 1.31 MB monolith because shared sandbox dependencies and the hybrid serializer source repeat. Opaque base64 transport is 7.04 MB on disk before compression (1.33× decoded; gzip is close to the original text). This is a build/storage tradeoff: a replay imports, decodes, compiles, and evaluates only its selected source bundle, and the route caches that decoded promise.

Deployment benchmark

The final benchmark run completed successfully on deployment dpl_CHJqdRSi52QvbKm1KzXhEs4Aa1B2; the sticky comparison reports:

  • inline STSO p75 191 → 169 ms (-12%), p90 229 → 191 ms (-17%), p99 580 → 268 ms (-54%)
  • whole-run overhead for 1,020 steps 195,405 → 167,186 ms (-14%)
  • an earlier run of the same runtime code measured STSO p75/p90/p99 at 155/172/264 ms and whole-run overhead at 153,275 ms, so the improvement survives expected cold-start and infrastructure variance
  • TTFS p75 remains 1.12–1.50 s across scenarios, showing that dispatch, route cold start, and workflow-server work still dominate end-to-end startup

A final cold compile-miss trace has a 22.52 ms workflow.run; the surrounding fresh-replay path shows 6.88 ms bundle load (overlapped with 2.37 ms world init), followed by 3.00 ms context creation, 4.00 ms compile, 2.63 ms evaluate, 1.53 ms input hydrate, and 8.82 ms replay execute. Its 146 ms payload-preparation wall span begins before the replay and overlaps event loading and execution rather than representing 146 ms of blocking CPU work.

After the bundle and script caches are warm, the final sequential benchmark's first fresh VM replay trace is 6.88 ms: context creation 2.00 ms, compile hit 0.14 ms, evaluation 1.94 ms, input hydration 0.12 ms, and replay execution 2.41 ms. The comparable main trace is 186.10 ms (-96.3% for this representative warm-cache replay). Subsequent retained invocations in the final trace are generally 1–2.4 ms.

Validation

  • 295 focused tests across builders, runtime, tracing, payload cache, and stream recovery
  • complete builders suite: 226/226
  • affected package typechecks and builds
  • Next/Turbopack, Vite, and Nuxt production builds
  • exact Next HMR regression plus same-size/same-mtime workflow-ID rename regression
  • all eight stable/canary × Webpack/Turbopack × Node/QuickJS Next dev E2E lanes green
  • Linux and Windows unit lanes plus Node/QuickJS Windows Next E2E lanes green
  • standalone/BOPA import smoke test
  • final benchmark workflow green
  • final Codex and Claude autoreviews clean

@changeset-bot

changeset-bot Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 85cdc90

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
Name Type
@workflow/builders Patch
@workflow/core Patch
@workflow/next Patch
@workflow/world-testing Patch
@workflow/astro Patch
@workflow/cli Patch
@workflow/nest Patch
@workflow/nitro Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/web Patch
workflow Patch
@workflow/nuxt Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
example-nextjs-workflow-turbopack Ready Ready Preview, v0 Aug 21, 2026 11:24pm
example-nextjs-workflow-webpack Ready Ready Preview, v0 Aug 21, 2026 11:24pm
example-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-astro-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-express-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-fastify-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-hono-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-nestjs-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-nitro-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-nuxt-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-python-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-sveltekit-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-tanstack-start-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workbench-vite-workflow Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workflow-docs Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workflow-swc-playground Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workflow-tarballs Ready Ready Preview, v0 Aug 21, 2026 11:24pm
workflow-web Ready Ready Preview, v0 Aug 21, 2026 11:24pm

@github-actions

github-actions Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (53 failed)

astro-quickjs (3 failed):

  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step

example-node (1 failed):

  • distributedAbortController - reconnect to existing controller | wrun_41M0KACQT50GHDQE4YR1AAVMDT | 🔍 observability

example-quickjs (2 failed):

  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step

express-quickjs (2 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable | wrun_41M0KA9BG30GPQ0NYSH12SVTT4 | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal

fastify-node (3 failed):

  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0KA9JMM0GYRYZ6TS5WG94YG | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0KA9RC70GVH0G3S3CH2X30F | 🔍 observability
  • distributedAbortController - reconnect to existing controller | wrun_41M0KACQT50GHDQE4YR1AAVMDT | 🔍 observability

hono-node (1 failed):

  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0KA9JMM0GYRYZ6TS5WG94YG | 🔍 observability

hono-quickjs (1 failed):

  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires

nest-node (1 failed):

  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0KA9RC70GVH0G3S3CH2X30F | 🔍 observability

nest-quickjs (6 failed):

  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step
  • AbortController abortHookOrderingWorkflow [listener-first-abort-first]: addEventListener → hook.then → abort() → resumeHook

nextjs-turbopack-quickjs (3 failed):

  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires

nextjs-webpack-node (2 failed):

  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0KA9RC70GVH0G3S3CH2X30F | 🔍 observability
  • distributedAbortController - reconnect to existing controller | wrun_41M0KACQT50GHDQE4YR1AAVMDT | 🔍 observability

nitro-node (1 failed):

  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0KA9JMM0GYRYZ6TS5WG94YG | 🔍 observability

nitro-quickjs (4 failed):

  • AbortController abortFromStepWorkflow: step abort cancels an in-flight sibling step
  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortHookOrderingWorkflow [listener-first-abort-first]: addEventListener → hook.then → abort() → resumeHook

nuxt-node (2 failed):

  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0KA9RC70GVH0G3S3CH2X30F | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0KACA0B0GSMQYV5DEW7HBVF | 🔍 observability

nuxt-quickjs (2 failed):

  • webhookWorkflow | wrun_41M0KA9B8R0GP3K0Q55A0Y65YT | 🔍 observability
  • AbortController abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M0KA52NX0GWXH7C2KT3GVQE7 | 🔍 observability
  • sleepingWorkflow | wrun_41M0KA5Q300GGMWKWWWCQJ3TCV | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0KA5R580GTW1DV2JXH8WH7E | 🔍 observability
  • nullByteWorkflow | wrun_41M0KA62WF0GWGE58NK9WVV7G9 | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0KAB14R0GTD15FDE27WJYE0 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0KAB2MT0GGCJYERR06YSM18 | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0KADMEB0GM7G7FFB1J5QN5Q | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0KABWG20GPZ5KZN92VV6ZKX | 🔍 observability

sveltekit-node (1 failed):

  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0KA9RC70GVH0G3S3CH2X30F | 🔍 observability

sveltekit-quickjs (2 failed):

  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step

tanstack-start-node (3 failed):

  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_41M0KA9JMM0GYRYZ6TS5WG94YG | 🔍 observability
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_41M0KA9RC70GVH0G3S3CH2X30F | 🔍 observability
  • distributedAbortController - manual abort triggers signal | wrun_41M0KACA0B0GSMQYV5DEW7HBVF | 🔍 observability

tanstack-start-quickjs (2 failed):

  • AbortController abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals
  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires

vite-node (1 failed):

  • distributedAbortController - manual abort triggers signal | wrun_41M0KACA0B0GSMQYV5DEW7HBVF | 🔍 observability

vite-quickjs (2 failed):

  • AbortController abortThrowIfAbortedMidFlightWorkflow: throwIfAborted in a polling loop bails when abort fires
  • AbortController abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0KAF9GJN8QAYGKBB7QTE648
  • promiseAllWorkflow | wrun_41M0KA52NX0GWXH7C2KT3GVQE7
  • sleepingWorkflow | wrun_41M0KA5Q300GGMWKWWWCQJ3TCV
  • parallelSleepWorkflow | wrun_41M0KA5R580GTW1DV2JXH8WH7E
  • nullByteWorkflow | wrun_41M0KA62WF0GWGE58NK9WVV7G9
  • cancelRun - cancelling a running workflow | wrun_41M0KAB14R0GTD15FDE27WJYE0
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0KAB2MT0GGCJYERR06YSM18
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0KADMEB0GM7G7FFB1J5QN5Q
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0KABWG20GPZ5KZN92VV6ZKX

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

51 flaky tests
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (express)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (fastify)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (nextjs-webpack)
  • abortAnyInStepWorkflow: AbortSignal.any inside a step composes deserialized signals (vite)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (express)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (hono)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (nextjs-webpack)
  • abortDeterministicBranchFromStepWorkflow: branches stay consistent when abort comes from a step (tanstack-start)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (express)
  • abortFromStepWorkflow: step abort cancels an in-flight sibling step (vite)
  • abortHookOrderingWorkflow [hook-first-abort-first]: hook.then → addEventListener → abort() → resumeHook (nest)
  • abortHookOrderingWorkflow [listener-first-hook-first]: addEventListener → hook.then → resumeHook → abort() (express)
  • abortHookOrderingWorkflow [listener-first-hook-first]: addEventListener → hook.then → resumeHook → abort() (nest)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (fastify)
  • abortListenerWorkflow: signal.addEventListener fires on the deserialized step signal (sveltekit)
  • abortReasonWorkflow: abort reason preserved across boundaries (nitro)
  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • cancelRun via CLI - cancelling a running workflow (vite)
  • deploymentId: 'latest' is a no-op in non-Vercel worlds (sveltekit)
  • distributedAbortController - reconnect to existing controller (astro)
  • distributedAbortController - TTL expiration triggers signal (express)
  • getChunks returns same content as reading the stream (tanstack-start)
  • getTailIndex returns correct index after stream completes (astro)
  • getTailIndex returns correct index after stream completes (hono)
  • hook created by a fra1 run resolves by token and resumes the run (nextjs-turbopack)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (example)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (hono)
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue (nextjs-webpack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (hono)
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook (fastify)
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook (hono)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (fastify)
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token (sveltekit)
  • negative startIndex (reads from end) (example)
  • negative startIndex (reads from end) (nitro)
  • no startIndex (reads all chunks) (nest)
  • no startIndex (reads all chunks) (nuxt)
  • no startIndex (reads all chunks) (tanstack-start)
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions (nitro)
  • positive startIndex (skips first chunk) (nuxt)
  • regular Error retries until success (astro)
  • regular Error retries until success (nitro)
  • regular Error retries until success (nuxt)
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (vite)
  • sleepWinsRaceWorkflow (tanstack-start)
  • webhookWorkflow (express)
  • webhookWorkflow (nest)
  • webhookWorkflow (nextjs-webpack)
  • webhookWorkflow (sveltekit)
  • webhookWorkflow (tanstack-start)
  • webhookWorkflow (vite)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

35 infra events
  • cold-start-warmup · suite warmup (python) · at 23:24:31Z · abandoned wrun_41M0KA465Z0GYH8XFEGCPEP736 · (+7 more)
  • run-pickup-stall · nullByteWorkflow (python) · at 23:24:48Z · abandoned wrun_41M0KA7TSM0GTYR5QKP2JRSHNA
  • run-pickup-stall · promiseAllWorkflow (python) · at 23:24:48Z · abandoned wrun_41M0KA7TSB0GVEHFENAGZ3GQGG
  • run-pickup-stall · sleepingWorkflow (python) · at 23:24:48Z · abandoned wrun_41M0KA7TSG0GGECB0QPBXVCYKX
  • run-pickup-stall · parallelSleepWorkflow (python) · at 23:24:48Z · abandoned wrun_41M0KA7TSG0GGECB0QPBXVCYKY
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 23:24:48Z · abandoned wrun_41M0KA7V3K0GK1XJMD5S5PHRHM
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 23:25:21Z · abandoned wrun_41M0KA8VDG0GKT5GPVM1Y5JTF1
  • run-pickup-stall · sleepingWorkflow (python) · at 23:25:48Z · abandoned wrun_41M0KA9P2X0GJW24R90EZJAYHM
  • run-pickup-stall · promiseAllWorkflow (python) · at 23:25:48Z · abandoned wrun_41M0KA9P450GNYNJHSX0PHKJ1X
  • run-pickup-stall · nullByteWorkflow (python) · at 23:25:48Z · abandoned wrun_41M0KA9P430GWBGNTR5WZCT3BA
  • run-pickup-stall · parallelSleepWorkflow (python) · at 23:25:48Z · abandoned wrun_41M0KA9P880GGTK95C10MBJVW4
  • cold-start-warmup · suite warmup (tanstack-start) · at 23:26:11Z · abandoned wrun_01M0KAA1K75X1BGB9TYNXQS35W
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 23:26:39Z · abandoned wrun_41M0KA9TNQ0GP5PVK3VHVY5MMH
  • cold-start-warmup · suite warmup (python) · at 23:27:21Z · abandoned wrun_01M0KA9B2KRK1HES0PYRK4EGD2 · (+7 more)
  • run-pickup-stall · parallelSleepWorkflow (python) · at 23:27:36Z · abandoned wrun_01M0KAD08008GNHTY1FFCRBDKC
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 23:27:36Z · abandoned wrun_01M0KAD07VXHE0FJTAHPDF1S0T
  • run-pickup-stall · nullByteWorkflow (python) · at 23:27:36Z · abandoned wrun_01M0KAD081VFNJGEP218TQ5DXK
  • run-pickup-stall · sleepingWorkflow (python) · at 23:27:36Z · abandoned wrun_01M0KAD07Z17W8QR50MC2KKW09
  • run-pickup-stall · promiseAllWorkflow (python) · at 23:27:36Z · abandoned wrun_01M0KAD07VXHE0FJTAHPDF1S0V
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 23:28:20Z · abandoned wrun_41M0KACDZE0GQMTDZ07KNYWHTX
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 23:28:20Z · abandoned wrun_41M0KACJJ90GPCG5XD1150E5YN
  • run-pickup-stall · promiseAllWorkflow (python) · at 23:28:36Z · abandoned wrun_01M0KAETVE9TEXDD6RJWWB1QEP
  • run-pickup-stall · sleepingWorkflow (python) · at 23:28:36Z · abandoned wrun_01M0KAETVGFVKV03WJ2AB7DB2Q
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 23:28:36Z · abandoned wrun_01M0KAETVHTZX1MCE0VQEKDKZ0
  • run-pickup-stall · parallelSleepWorkflow (python) · at 23:28:36Z · abandoned wrun_01M0KAETVJBPX0SF0VJKXT7P4N
  • run-pickup-stall · nullByteWorkflow (python) · at 23:28:36Z · abandoned wrun_01M0KAETVNB41E6X16G0FM3KWY
  • run-pickup-stall · step throw of a non-Error value preserves it as cause on the wrapping FatalError (vite) · at 23:29:02Z · abandoned wrun_01M0KAFKXBG30D64EJP4WB09KT
  • run-pickup-stall · step-argument serialization failure is catchable in workflow code (vite) · at 23:29:03Z · abandoned wrun_01M0KAFNHKFKND99B6RH56785Z
  • run-pickup-stall · uncaught step-argument serialization failure fails the run as USER_ERROR without redelivery retries (vite) · at 23:29:03Z · abandoned wrun_01M0KAFNJMDZ84H9Z1V6YJP43B
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 23:29:36Z · abandoned wrun_01M0KAGNEW8W4PYVKZTGG3JEVS
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 23:29:36Z · abandoned wrun_01M0KAGNF8SAG5BAHQBN0DAT4K
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 23:29:36Z · abandoned wrun_01M0KAGNF4H26JB65B9CAW8Y43
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 23:30:06Z · abandoned wrun_01M0KAHJTG5BG8BK7BKPZSWZ89
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 23:30:06Z · abandoned wrun_01M0KAHJTKTQ00NSTPJ9ETBSX0
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 23:30:36Z · abandoned wrun_01M0KAJG2JXWV9P72HAGFB5FMK

E2E Test Summary

Summary
Passed Failed Skipped Total
❌ ▲ Vercel Production 3525 53 742 4320
✅ 💻 Local Development 3762 0 558 4320
✅ 📦 Local Production 3922 0 558 4480
✅ 🐘 Local Postgres 3922 0 558 4480
✅ 🪟 Windows 320 0 0 320
❌ 🌐 Cross-language Conformance 0 9 132 141
✅ vercel-multi-region 27 0 0 27
Total 15478 62 2548 18088
Details by Category

❌ ▲ Vercel Production

App Passed Failed Skipped
✅ astro-node 132 0 28
❌ astro-quickjs 129 3 28
❌ example-node 131 1 28
❌ example-quickjs 130 2 28
✅ express-node 132 0 28
❌ express-quickjs 130 2 28
❌ fastify-node 129 3 28
✅ fastify-quickjs 132 0 28
❌ hono-node 131 1 28
❌ hono-quickjs 131 1 28
❌ nest-node 131 1 28
❌ nest-quickjs 126 6 28
✅ nextjs-turbopack-node 157 0 3
❌ nextjs-turbopack-quickjs 154 3 3
❌ nextjs-webpack-node 155 2 3
✅ nextjs-webpack-quickjs 157 0 3
❌ nitro-node 131 1 28
❌ nitro-quickjs 128 4 28
❌ nuxt-node 130 2 28
❌ nuxt-quickjs 130 2 28
❌ python-node 0 8 152
❌ sveltekit-node 150 1 9
❌ sveltekit-quickjs 149 2 9
❌ tanstack-start-node 129 3 28
❌ tanstack-start-quickjs 130 2 28
❌ vite-node 131 1 28
❌ vite-quickjs 130 2 28

✅ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 160 0 0
✅ nextjs-turbopack-quickjs 160 0 0

❌ 🌐 Cross-language Conformance

App Passed Failed Skipped
❌ python 0 9 132

✅ vercel-multi-region

App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

📋 View full workflow run

@github-actions

github-actions Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 85cdc90 · Fri, 21 Aug 2026 23:37:35 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 125 (-86%) 💚 1179 🔴 (+15%) 🔻 1203 🔴 (+16%) 🔻 1210 🔴 (+11%) 30
TTFS stream 1066 (+290%) 🔻 1168 🔴 (+12%) 1188 🔴 (+9.3%) 1234 🔴 (+7.2%) 30
TTFS hook + stream 1337 (+149%) 🔻 1532 🔴 (+14%) 1629 🔴 (+13%) 1817 🔴 (+18%) 🔻 30
Fan-out TTFS Promise.all(100 steps) 392 (-27%) 💚 1800 (+16%) 🔻 1804 (+14%) 1842 (+15%) 10
Fan-out TTLS Promise.all(100 steps) 1926 (+27%) 🔻 3740 (-47%) 💚 4309 (-46%) 💚 5567 (-43%) 💚 10
STSO 1020 steps (inline) 114 (-16%) 💚 162 (-16%) 💚 180 (-22%) 💚 267 (-27%) 💚 1019
WO 1020 steps 160610 (-15%) 160610 (-15%) 160610 (-15%) 160610 (-15%) 1
CRTT first chunk (pooled) 93 (+13%) 132 (+12%) 221 (+17%) 🔻 462 (+14%) 28

Streams

Scenario CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 102 (-6%) 134 (-6%) 214 (±0%) 391 (-57%) 137 (-16%) 10
size sweep (100/s, 160B-12KB) 115 (+12%) 117 (-15%) 143 (-37%) 268 (-41%) 87 (-45%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 115 (+2%) 116 (-31%) 193 (-46%) 405 (-44%) 297 (-50%) 3
replay eve-gpt-5.6-sol-2000t (1x) 132 (-9%) 117 (-16%) 176 (-4%) 758 (+4%) 542 (-4%) 2
replay eve-gpt-5.6-sol-2000t (2x) 113 (-10%) 157 (-28%) 234 (-30%) 870 (+17%) 493 (+62%) 3
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 188787ms → this run 160507ms (Δ -28280ms, -15%)

100-150 ms  ██░░░░░░░░░░░┃            main  69  this 445  +376
150-200 ms  ████████████████┃███████  main 748  this 522  -226
200-250 ms  ┃████                     main 141  this  41  -100
250-300 ms  ┃                         main  40  this   6   -34
300-350 ms  ┃                         main   7  this   2    -5
350-400 ms  ┃                         main   5  this   2    -3
400-450 ms  ┃                         main   3  this   0    -3
450-500 ms  ┃                         main   2  this   0    -2
500-550 ms  ┃                         main   0  this   1    +1
550-600 ms  ┃                         main   1  this   0    -1
650-700 ms  ┃                         main   1  this   0    -1
700-750 ms  ┃                         main   1  this   0    -1
850-900 ms  ┃                         main   1  this   0    -1
📈 CRTT drill-down vs main (RTT distributions & profiles) 
variant  RTT 1ms→5s+             avg         p50         p90         p99     n
control  ······█▆▁····  106.2 (-12%)    93 (-5%)   214 (±0%)  391 (-57%)  3000
sweep    ·····▁█▆▁····   98.5 (-19%)   94 (-10%)  143 (-37%)  268 (-41%)  3000
gw 1x    ·····▁█▄▁····  105.3 (-31%)   91 (-20%)  193 (-46%)  405 (-44%)  5295
eve 1x   ·····▁█▅▁▁▁··  111.1 (-11%)   91 (-15%)   176 (-4%)   758 (+4%)  5186
eve 2x   ·····▁▅█▂▁···  149.9 (-11%)  115 (-14%)  234 (-30%)  870 (+17%)  7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control  █▅▆█▁▄▁▃▂▆  93–121ms
sweep    █▃▅▃▃▄▂▂▁▁  91–115ms
gw 1x    ▇▇▂██▄▃▁▂▂  89–122ms
eve 1x   ▄▃▇▃▁▂▄▆█▇  81–140ms
eve 2x   ▂▁▂▂▁▁▂█▄▁  104–322ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep  █▇█▇▃▁▂  96–100ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control  ▇▅█▂▁▄▃▄▄▆  25–36ms
sweep    ▅▃█▆▇▆▇▃▂▁  33–43ms
gw 1x    ▅▄▂▄█▅▂▁▁▄  23–35ms
eve 1x   ▃▅▇▁▂▃▆▄█▄  18–30ms
eve 2x   ▅▁▃▃█▄▆█▅▄  21–28ms
ℹ️ Metric definitions & methodology

Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.

The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@NathanColosimo NathanColosimo mentioned this pull request Aug 14, 2026
Open
This was referenced Aug 14, 2026
Open
Open
Open
@github-actions

github-actions Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟠 world-sim scenario book — 1 fail of 41 total

fence=per-spec

scenario outcome events virt replay violations
✅ smoke-no-steps completed 3 0ms ok 0
✅ smoke-one-step completed 6 0ms ok 0
✅ hook-at-step-started completed 12 0ms ok 0
✅ hook-at-step-completed completed 12 0ms ok 0
✅ hook-at-hook-created completed 12 0ms ok 0
✅ deadline-hook-wins completed 7 1.0h ok 0
✅ deadline-expires completed 7 1.0h ok 0
✅ long-sleep completed 11 30.0d ok 0
✅ hook-never-arrives stalled 3 0ms skipped 0
✅ step-retries-twice completed 10 2.0s ok 0
✅ parallel-steps completed 9 0ms ok 0
✅ hook-on-execution-state completed 12 0ms ok 0
✅ peek-hook-before-branch completed 12 0ms ok 0
✅ peek-hook-after-branch completed 12 0ms ok 0
✅ peek-hook-at-registration completed 12 0ms ok 0
✅ race-hook-before-probe completed 12 0ms ok 0
✅ race-hook-after-probe completed 12 0ms ok 0
✅ race-duplicate-delivery completed 13 0ms ok 0
✅ attr-hook-before-step completed 11 0ms ok 0
✅ attr-hook-after-step completed 11 0ms ok 0
✅ attr-from-step-body completed 13 0ms ok 0
✅ fork-hook-after-timeout completed 14 1.0m ok 0
✅ fork-hook-before-timeout completed 14 1.0m ok 0
✅ count-hook-after-timeout completed 17 1.0m ok 0
✅ count-hook-before-timeout completed 20 1.0m ok 0
✅ stale-read-step-count-fork completed 20 1.0m ok 0
✅ stale-read-equal-step-counts completed 14 1.0m ok 0
✅ step-vs-step-fork completed 12 0ms ok 0
✅ step-vs-step-fork-fenced completed 12 0ms ok 0
✅ fence-catches-benign-direction completed 12 5ms ok 0
✅ in-flight-before-decision completed 17 1.0m ok 0
❌ in-flight-before-decision-counted completed 17 1.0m ok 0
✅ in-flight-after-decision completed 19 2.0m ok 0
✅ stale-read-step-count-fork-fenced completed 20 1.0m ok 0
✅ fork-hook-wins completed 13 1.0m ok 0
✅ fork-timeout-wins completed 13 1.0m ok 0
✅ unclaimed-payload-under-fork completed 17 1.0m ok 0
✅ claimed-payload-under-fork completed 17 1.0m ok 0
✅ writers-independent-step-bodies completed 12 0ms ok 0
✅ writers-scripted-tempo completed 12 0ms ok 0
✅ cancel-mid-step cancelled 7 0ms skipped 0

Full trace: world-sim.txt

@NathanColosimo
NathanColosimo force-pushed the codex/replay-lazy-source-bundles branch from 8bdf832 to a2e263d Compare August 14, 2026 07:28
@NathanColosimo
NathanColosimo force-pushed the codex/replay-lazy-source-bundles branch from 45cfc76 to e202e05 Compare August 14, 2026 07:49
vercel[bot]
vercel Bot reviewed Aug 17, 2026
View reviewed changes

@vercel vercel Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:

Test asserts SPEC_VERSION_CURRENT === SPEC_VERSION_SUPPORTS_COMPRESSION (5) but the PR bumped SPEC_VERSION_CURRENT to SPEC_VERSION_SUPPORTS_SLOT_IDENTITY (6), so the assertion evaluates 6 === 5 and fails.

Fix on Vercel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vercel vercel[bot] vercel[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@NathanColosimo