[nest] Harden the NestJS integration for production - #3695
[nest] Harden the NestJS integration for production#3695VaguelySerious wants to merge 2 commits into
Conversation
Fixes correctness bugs on the workflow routes and closes the API gaps that made
a production deployment depend on undocumented setup.
Request and response fidelity:
- Recover the request body from `req.rawBody`, a Buffer/string body, or the
unread request stream, in that order. Re-serializing a parsed body with
JSON.stringify was silently rewriting webhook payloads: pretty-printed JSON
lost its whitespace (breaking HMAC verification), a Buffer became
{"type":"Buffer",...}, and a content type no parser claimed was dropped
entirely while the sender still got a 2xx.
- Write response bodies as bytes rather than through `.text()`, which replaced
every non-UTF-8 byte with U+FFFD, and keep every set-cookie value.
Routing:
- Serve GET/HEAD/OPTIONS on the flow route, which the generated bundle already
exports. HEAD is what getWorkflowPort() probes to identify a workflow server,
so a 404 there made local port detection fall back to an arbitrary port.
- Adopt `app.setGlobalPrefix()` for generated callback and webhook URLs, and add
a `basePath` option for a sub-path applied outside NestJS. A global prefix
previously moved the routes without moving the URLs, so runs were created and
then every delivery 404s forever with no diagnostic.
Startup and configuration:
- Fail startup when `skipBuild` is set and the bundles are absent, instead of
reporting a healthy boot and answering ERR_MODULE_NOT_FOUND per request.
- Default `skipBuild` to true on Vercel, where the bundles ship in the Build
Output and the filesystem is read-only.
- Add `forRootAsync`, export `WORKFLOW_MODULE_OPTIONS`, and provide the options
through DI instead of the process-global `configureWorkflowController`, which
two apps in one process overwrote for each other. It stays as a deprecated
fallback.
- Add opt-in `manageWorldLifecycle` so a self-hosted World's workers start with
the app and close on shutdown, and `preloadBundles` to keep the first queue
delivery off the module-load path.
- Report a load failure as 503 naming the build command that produces the
missing file, rather than letting a module-resolution stack escape as a 500.
- Pin `watch` off: the builder discards the esbuild contexts
createCombinedBundle returns in watch mode, so nothing rebuilt and the
contexts leaked.
Build:
- Externalize NestJS's absent optional peers from the workflow function too, not
only the app function. A step importing an @Injectable() service pulled
@nestjs/common into flow.func and failed the whole --vercel build on
`Could not resolve "class-validator"`.
- Honour config.externalPackages in the final combined and webhook esbuild
passes, which previously ignored it.
- `init` merges into an existing .swcrc instead of replacing it, so the --force
in a prebuild script no longer discards the app's own SWC configuration.
- Expose --base-path, --sourcemap, --max-duration, --runtime and --app-function,
which the builders accepted but the CLI could not reach.
Docs state plainly that the Nest injector, providers, request-scoped context,
guards, interceptors and the Nest Logger are unreachable from "use workflow" and
"use step" code, why (per-bundle module copies give a distinct class identity,
and on Vercel the workflow function is a separate function), and what to write
instead.
Unit tests go from 34 to 101, covering the conversion in both directions, option
resolution, base-path reconciliation, bundle validation, and the .swcrc merge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 52d679d The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3578 | 0 | 742 | 4320 |
| ✅ 💻 Local Development | 3922 | 0 | 558 | 4480 |
| ✅ 📦 Local Production | 3922 | 0 | 558 | 4480 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 558 | 4480 |
| ✅ 🪟 Windows | 320 | 0 | 0 | 320 |
| ✅ 🌐 Cross-language Conformance | 9 | 0 | 132 | 141 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| Total | 15700 | 0 | 2548 | 18248 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 132 | 0 | 28 |
| ✅ astro-quickjs | 132 | 0 | 28 |
| ✅ example-node | 132 | 0 | 28 |
| ✅ example-quickjs | 132 | 0 | 28 |
| ✅ express-node | 132 | 0 | 28 |
| ✅ express-quickjs | 132 | 0 | 28 |
| ✅ fastify-node | 132 | 0 | 28 |
| ✅ fastify-quickjs | 132 | 0 | 28 |
| ✅ hono-node | 132 | 0 | 28 |
| ✅ hono-quickjs | 132 | 0 | 28 |
| ✅ nest-node | 132 | 0 | 28 |
| ✅ nest-quickjs | 132 | 0 | 28 |
| ✅ nextjs-turbopack-node | 157 | 0 | 3 |
| ✅ nextjs-turbopack-quickjs | 157 | 0 | 3 |
| ✅ nextjs-webpack-node | 157 | 0 | 3 |
| ✅ nextjs-webpack-quickjs | 157 | 0 | 3 |
| ✅ nitro-node | 132 | 0 | 28 |
| ✅ nitro-quickjs | 132 | 0 | 28 |
| ✅ nuxt-node | 132 | 0 | 28 |
| ✅ nuxt-quickjs | 132 | 0 | 28 |
| ✅ python-node | 8 | 0 | 152 |
| ✅ sveltekit-node | 151 | 0 | 9 |
| ✅ sveltekit-quickjs | 151 | 0 | 9 |
| ✅ tanstack-start-node | 132 | 0 | 28 |
| ✅ tanstack-start-quickjs | 132 | 0 | 28 |
| ✅ vite-node | 132 | 0 | 28 |
| ✅ vite-quickjs | 132 | 0 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 0 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 9 | 0 | 132 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 237191ms → this run 221429ms (Δ -15762ms, -7%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Signed-off-by: Peter Wielander <mittgfu@gmail.com>
1 participant
Production-readiness pass over
@workflow/nest. Every item below was reproduced againstworkbench/nestbefore being fixed, and the fix re-verified the same way.Correctness bugs on the workflow routes
Webhook bodies were silently corrupted or dropped. The controller re-serialized the already-parsed Express body with
JSON.stringify. Measured by sending to real webhook tokens and reading back what the workflow received:{"message": "one", "n": 1}{"message":"one","n":1}<order id="7">…</order>""a=1&b=2""{"type":"Buffer","data":[…]}All four returned a success status, so a webhook signed over its raw body (Stripe, GitHub, Shopify, Slack) could not verify. The body is now recovered from
req.rawBody, aBuffer/string body, or the unread request stream, in that order; all four cases are byte-exact. The re-serializing fallback remains for an app withoutrawBody: trueand warns once.Response bodies were decoded as UTF-8 via
.text(), so four bytesff fe 00 01arrived as eight bytes of U+FFFD. They are now written as bytes.Headers.forEachalso kept only the lastset-cookie;getSetCookie()keeps them all.Only POST was wired on the flow route even though the generated bundle exports
GET/HEAD/OPTIONSas aliases.getWorkflowPort()probes withHEADand treats non-200 as "not a workflow server", so local port detection never identified a Nest app. All four methods now answer.app.setGlobalPrefix()broke everything silently. The routes moved; the generated callback URLs did not. Runs were created and then every queue delivery 404s forever.WorkflowModulenow reads the prefix duringonModuleInit(which NestJS runs after route registration) and adopts it, with abasePathoption for a sub-path applied outside NestJS. Verified end to end: a run undersetGlobalPrefix('api')completes with zero 404s, where before it looped indefinitely.skipBuildwith no bundles booted healthy and then answered every workflow request withERR_MODULE_NOT_FOUND. Startup now fails with the missing filenames and the command that produces them. A load failure at request time is a 503 with the same message instead of a raw stack.A step importing an
@Injectable()service failed the--vercelbuild withCould not resolve "class-validator". The optional-peer externals list applied only to the app function; the workflow function is built by the sharedVercelBuildOutputAPIBuilder, which had none. Confirmed causal with a negative control: removing the one line reproduces the failure, restoring it builds clean.API and lifecycle gaps
forRootAsyncfor options fromConfigService, andWORKFLOW_MODULE_OPTIONSexported. Options now flow through DI;configureWorkflowControllerstays as a deprecated fallback (its process-global state made two apps in one process overwrite each other).manageWorldLifecycle(opt-in) starts a self-hosted World's workers with the app and closes them on shutdown, replacing per-app bootstrap boilerplate.preloadBundles(default on) keeps ~1.6MB of module evaluation off the first queue delivery.skipBuilddefaults totrueon Vercel, so app modules no longer need aprocess.env.VERCELbranch.watchis pinned off and deprecated: the builder discarded the esbuild contextscreateCombinedBundlereturns in watch mode, so nothing rebuilt and the contexts leaked.initmerges into an existing.swcrcrather than replacing it. The docs put--forceinprebuild, which discarded any SWC config the app had.--base-path,--sourcemap,--max-duration,--runtimeand--app-functionare reachable from the CLI; the builders already accepted them.config.externalPackagesis honoured in the final combined and webhook esbuild passes (@workflow/builders), which previously ignored it.NestJS DI is documented as unavailable
Workflows and steps do not run inside the Nest application, and this was undocumented. The docs now state plainly that the injector, providers, request-scoped context and
AsyncLocalStorage(nestjs-cls), guards, interceptors, pipes, filters and the NestLoggerare all unreachable from"use workflow"and"use step"code, why (a step's bundle gets its own copy of any app file it imports, so a class token is a different object andapp.get()raisesUnknownElementException; on Vercel the workflow function is a separate function from the app), and what to write instead. No behaviour change: importing a Nest service into a step now builds, it simply gives an uninjected instance.Tests
34 → 101 unit tests: conversion in both directions (raw body, unparsed stream, Buffer, repeated headers, HTTP/2 pseudo-headers, binary response, multiple cookies, Fastify reply), option resolution, base-path reconciliation, bundle validation,
forRootAsync, and the.swcrcmerge.The full local e2e suite passes against the workbench app: 134 passed, 7 skipped, 0 failed. Repo-wide
typecheck,biome ci, and the docs snippet typecheck are clean.🤖 Generated with Claude Code