Skip to content

ci: restore TRPC and workers tests to Node 24 after native-addon teardown crash #34

Description

@absolutepraya

Problem

The TRPC test suite can abort at the native Node.js/addon layer during Vitest worker teardown when CI runs on Node 24. This is not a normal test assertion failure: the worker process crashes while better-sqlite3 is destroying native Statement objects, and Vitest/Tinypool only reports ERR_IPC_CHANNEL_CLOSED afterward because the child process is already dead.

PR #23 currently carries a temporary CI-only mitigation: local development and production remain on Node 24.18.1, while the combined CI tests job runs on Node 22.21.1. That mitigation has now restored TRPC and workers test execution, but CI and production should eventually be aligned on Node 24 again.

Related PR: #23

Observed failure

Representative native crash:

# node (vitest 1): void node::RemoveEnvironmentCleanupHook(...)
# Assertion failed: (env) != nullptr

Statement::~Statement()
.../node_modules/better-sqlite3/build/Release/better_sqlite3.node

After the native worker aborts, Vitest/Tinypool reports:

Error: Channel closed
Serialized Error: { code: 'ERR_IPC_CHANNEL_CLOSED' }

The Channel closed error is therefore secondary, not the root cause.

Root-cause analysis

The failure chain is:

Vitest worker teardown / GC
  -> better-sqlite3 native Statement wrapper destruction
  -> Node RemoveEnvironmentCleanupHook()
  -> no current Node Environment is available
  -> Node assertion aborts the worker
  -> Tinypool tries to message the dead worker
  -> ERR_IPC_CHANNEL_CLOSED

Node itself tracks this cleanup-hook design problem:

The Node 24 backport explicitly describes ObjectWrap destruction aborting in RemoveEnvironmentCleanupHook() when no Node.js Environment is current, which matches this repository's better-sqlite3 stack.

Why this appeared suddenly

CI previously configured the runtime as a floating Node 24 range. Older hosted-runner images resolved that to an older Node 24 patch (for example Node 24.6.0), and the suite was green.

After GitHub's hosted runner image changed, the same floating node-version: 24 began resolving to Node 24.19.0. At that point the TRPC suite started crashing during native teardown.

This is not specific to the public-list changes in PR #23. main itself reproduced the same native TRPC crash on Node 24.19.0.

Useful evidence:

What was already tried

1. Explicit SQLite database cleanup

TRPC tests now explicitly close per-test in-memory SQLite databases, and getInMemoryDB() closes the native handle if setup/migration throws.

This is correct lifecycle hygiene and should remain, but it is not sufficient to prevent the crash. Closing the database handle does not guarantee that every JavaScript/native Statement wrapper has already been garbage-collected before Vitest destroys the worker environment.

2. Serialize Vitest workers

Tried one-thread / serialized worker configurations, including:

The native teardown crash still reproduced. We should not continue adding Vitest pool/isolation workarounds unless new evidence points there.

3. Pin Node 24.18.1

Node 24.18.1 was tried because upstream Karakeep used it as a workaround for a related Node 24 / better-sqlite3 runtime crash.

For this repository's Vitest worker teardown case, Node 24.18.1 still reproduced:

Statement::~Statement()
RemoveEnvironmentCleanupHook()
Assertion failed: (env) != nullptr

Therefore the 24.18.1 pin alone is not sufficient for the test runner.

4. Node 22.21.1

An earlier experiment already showed that Node 22.21.1 successfully completes the TRPC and workers steps:

That workflow was later cancelled during E2E by a superseding run, but before cancellation:

  • Shared Package Tests: passed
  • TRPC Tests: passed
  • Workers Tests: passed

PR #23 now uses this as a targeted CI mitigation.

Current mitigation commits:

  • fix(ci): run test suite on Node 22: 811907d
  • workaround documentation: 51a19ae

Current proof that the mitigation works:

In that run, Shared Package Tests, TRPC Tests, and Workers Tests all complete successfully under the Node 22.21.1 CI override.

Current temporary state

The intended runtime split is:

Local development        -> Node 24.18.1 (.nvmrc)
Production / Docker      -> Node 24.18.1
Lint / format / typecheck -> Node 24.18.1
OpenAPI / quality jobs   -> Node 24.18.1
Combined CI tests        -> Node 22.21.1 (temporary override)
E2E application image    -> Node 24.18.1

This is an acceptable mitigation because it is narrowly scoped and production remains on Node 24, but it is not the desired permanent state.

Preferred permanent fix

Option A — adopt the fixed Node 24 release once nodejs/node#65042 ships

This is the preferred path because the failure originates in Node's native cleanup-hook lifecycle rather than application business logic.

Once the Node 24 backport is released in an installable Node 24 patch:

  • Update .nvmrc to a Node 24 release that contains the cleanup-hook fix.
  • Update Docker/runtime pins to the same supported Node 24 patch where appropriate.
  • Remove the node-version: "22.21.1" override from the CI tests job.
  • Let the shared GitHub setup action use .nvmrc for the test job again.
  • Run TRPC and workers tests repeatedly on Node 24 to make sure the native abort is actually gone rather than merely intermittent.
  • Run the full CI/E2E suite.
  • Remove the temporary Node-22 workaround notes from contributor/runtime documentation.

Option B — evaluate better-sqlite3 12.x if the Node 24 fix is delayed or insufficient

The repository currently uses better-sqlite3 11.x. A controlled dependency-upgrade experiment is worth doing separately because newer better-sqlite3 releases have newer Node 24 support/prebuild work.

Do not treat this as proven yet: there is currently no repository-specific evidence that upgrading to 12.x alone fixes this exact RemoveEnvironmentCleanupHook() teardown path.

Suggested experiment:

  • Create a focused branch that changes only better-sqlite3 (plus required type/lockfile changes).
  • Run TRPC tests on the current Node 24 runtime with no Node-22 override.
  • Run the suite multiple times to detect teardown flakiness.
  • Run workers and E2E to catch native/ABI/runtime regressions.
  • Review any breaking changes between 11.x and 12.x before adopting it.
  • If it reliably fixes Node 24 CI, upgrade the dependency and remove the Node-22 override.

Option C — application/test lifecycle cleanup only if new evidence requires it

Keep the existing explicit DB cleanup because it is correct, but do not attempt to manually null every test context, force GC, disable Vitest isolation, or otherwise program around the Node C++ lifecycle bug unless a minimal reproduction proves repository-owned references are still independently leaking after the upstream Node fix.

Permanent-fix acceptance criteria

This issue can be closed when:

  • The combined CI tests job runs on the same supported Node 24 line as the repository runtime; there is no Node 22 test-only override.
  • TRPC passes on Node 24 without RemoveEnvironmentCleanupHook() assertions.
  • Workers pass on Node 24.
  • There is no Statement::~Statement() native abort from better-sqlite3.
  • There is no downstream ERR_IPC_CHANNEL_CLOSED caused by a dead Vitest worker.
  • The full CI + E2E path passes.
  • The fix is shown stable across multiple consecutive CI runs, not only one run.
  • Temporary workaround comments/documentation are removed.
  • Local development, CI tests, and production no longer need different Node major versions for this reason.

Non-goals

  • Do not hide the failure by making TRPC non-blocking.
  • Do not remove TRPC coverage.
  • Do not permanently keep production on Node 22 solely for this test-runner problem.
  • Do not add more Vitest serialization/isolation hacks without evidence.

Summary

The current Node 22 CI override is a proper temporary mitigation backed by successful repository CI evidence. The permanent solution should restore Node 24 test execution by consuming the upstream Node cleanup-hook fix, or—if necessary—by proving that a newer better-sqlite3 release removes the problematic native lifecycle path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingseverity:mediumImportant gap or degraded experience with a reasonable workaroundstatus/untriaged

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions