Skip to content

CI: hub env-stashing wrote the string "undefined", un-skipping DB-gated suites - #176

Merged
TheGreatAxios merged 1 commit into
mainfrom
cl-ci-postgres-fix
Aug 21, 2026
Merged

TheGreatAxios merged 1 commit into
mainfrom
cl-ci-postgres-fix

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

Root cause

Every recent CI run — including main's own — failed the checks job's
bun run test step with connect ECONNREFUSED ::1:5432 and
127.0.0.1:5432. The dual-stack refusal was the tell: nothing was
listening on either loopback address because no Postgres was expected
there. checks deliberately has no service container; only
walking-skeleton does.

The connections came from three apps/hub suites that stash and clear
environment variables:

function clearEnvKey(key: EnvKey): void {
  // Prefer assignment over `delete process.env[key]` — eslint forbids dynamic delete.
  process.env[key] = undefined;
}

Node coerces process.env[k] = undefined to the string "undefined"
instead of removing the key. Bun did not, up to 1.3.x — but does from
1.4. The workflow pins bun-version: latest, which now resolves to
1.4.0 (bun test v1.4.0 (34cbb9a40) in the log), so the behaviour
flipped under CI with no repo change. That is why main went red wholesale
rather than at a specific commit.

bun test runs one process per package, so once
apps/hub/src/memory-mount.test.ts ran, DATABASE_URL and
EMBED_BASE_URL were the string "undefined" for the rest of the run:

  • mountMemory parses EMBED_BASE_URL as "string > 0". "undefined"
    passes, so the two tests asserting the unconfigured path proceeded to
    runMemoryMigrations and died on a socket. (The third test in that
    describe sets EMBED_BASE_URL="" explicitly and passed — exactly as the
    log shows.)
  • Every DB-gated suite guards on process.env["DATABASE_URL"] ?? "" or
    === undefined. A truthy "undefined" un-skipped all of them, and
    postgres.js, unable to parse that as a URL, fell back to its default
    host/port — localhost:5432, resolved on both stacks.

Log evidence, run
32425315884:
the hub package reports 131 pass / 2 skip / 18 fail / 1 error across
151 tests. The same suite locally reports 134 pass / 20 skip / 0 fail
across 154 tests — 18 tests that should skip instead ran and failed.

Reproduced locally by simulating Bun 1.4's coercion
(DATABASE_URL=undefined EMBED_BASE_URL=undefined bun test in
apps/hub): 151 tests, 17 fail — CI's numbers.

Fix

Clear the keys for real, in all three suites, with
Reflect.deleteProperty (dynamic delete is lint-banned here).

After the fix, apps/hub with no DATABASE_URL: 154 tests, 134 pass,
20 skip, 0 fail
.

No workflow change was needed: the service container is attached to the
job that needs it, and host resolution was never the problem.

Known remaining failure (pre-existing, unrelated)

walking-skeleton also fails on chat e2e > mention fan-out drives the mentioned run — list run events for run_…: expected HTTP 200, got 404 {"code":"not_found","message":"Run not found"} at
scripts/e2e/chat.test.ts:606. That is a product/e2e bug on a job that
does have Postgres, present in all three of main's last runs, and is out
of scope for this change.

Linear: https://linear.app/abklabs/issue/CL-6450

…ed suites

Three apps/hub suites cleared environment variables with
`process.env[key] = undefined`. Node — and Bun from 1.4 onward, the
version CI's `bun-version: latest` now resolves to — coerces that to the
*string* "undefined" rather than removing the key, so after the first
memory-mount test every later read of DATABASE_URL and EMBED_BASE_URL
saw a truthy value.

That poisoned the whole `bun test` process for @workbench/hub, which runs
in the `checks` job with no Postgres service:

- `mountMemory` accepted "undefined" as a configured EMBED_BASE_URL and
  went on to migrate, so the two tests asserting the unconfigured path
  died on a connection instead.
- Every DB-gated `describe.skipIf(...)`/`?? ""` gate un-skipped and
  handed postgres.js an unparseable URL, which falls back to its
  localhost:5432 default — hence `ECONNREFUSED ::1:5432` *and*
  `127.0.0.1:5432` on both stacks, with no service anywhere to refuse
  them.

Clear the keys for real with `Reflect.deleteProperty` (dynamic `delete`
is lint-banned). With Bun 1.4's coercion simulated locally the suite
reproduces CI exactly — 151 tests, 17 fail; after the fix it is 154
tests, 134 pass, 20 skip, 0 fail.
@TheGreatAxios
TheGreatAxios merged commit 9b45bbf into main Aug 21, 2026
0 of 2 checks passed
@TheGreatAxios
TheGreatAxios deleted the cl-ci-postgres-fix branch August 25, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant