perf(test): clone a database per test instead of a container

[LKSNDRTMLKV]

The fourth item from the CI/test-speed research, and the one the measurements pointed at: 171 tests consumed 86% of all test time, almost entirely in container startup.

What was happening

start_pg booted a fresh postgres:17 on every call — container start, a hardcoded 1500ms settle, CREATE ROLE, then the full migration set. Per test. 170 call sites did this, and the arithmetic lines up exactly with the 171 slow tests.

Why not just share it in a OnceLock

Because it does not work, which I confirmed rather than assumed: nextest runs each test in its own process. Two tests in one binary printed two distinct PIDs. Any process-local sharing is dead on arrival.

So the server has to outlive the test process and be discovered through the environment.

The change

start_pg gains a fast path. With ODAL_TEST_PG_ADMIN_URL set, it clones a per-test database from a migrated template on that server:

• ensure_template creates the odal_app role and a migrated template database, once per server. Guarded by a Postgres advisory lock, not a process-local Once — the racing parties are separate processes. The first to take the lock builds the template; the rest wait and find it there.
• clone_from_template issues CREATE DATABASE <unique> TEMPLATE <template>, a file copy inside Postgres. Measured at ~190ms, against 12–16s to boot and migrate a container.
• The migration pool is closed before the lock is released. CREATE DATABASE ... TEMPLATE refuses while anything is connected to the template, so leaving it open would break every clone that followed.

Unset, nothing changes. A bare cargo nextest run still starts a container per test — slowly, but working. TestPg now holds Option<ContainerAsync<..>>, None on the shared path.

start_pg_before deliberately keeps starting its own container: a test of a migration needs a server the migration has not been applied to.

The ninth copy

Chasing the remaining time turned up dpp-vault/tests/helpers/mod.rs starting its own postgres:17 — 109 call sites across 27 files, the actual bulk of the slow cohort.

just harness-check missed it because the function was called start_postgres, and I had written the rule as ^async fn start_pg. A gate written to catch exactly the copy already found. The rule now matches GenericImage::new("postgres" — starting a Postgres container at all, whatever the function is named — and was verified to fail on the old shape before being trusted.

start_postgres and PgContainer keep their names and delegate to the shared harness, so none of the 109 call sites changed.

Measured

	Before	After
pg_integration suite (20 tests)	57.7s	6.4s
One vault test, in isolation	6.47s	0.68s
dpp-vault integration (307 tests)	—	25.4s
Per-database setup	12–16s (container)	~190ms (clone)

Full integration tier — dal, vault, plugin-host, node — 1042 tests, all passing against the shared server, including the migration tests that still start their own.

Wiring

• just test-integration now runs the tiers behind scripts/shared-test-pg.sh, which starts one Postgres, polls pg_isready rather than sleeping a fixed guess, and removes it on any exit.
• just test-integration-isolated keeps the container-per-test arrangement. Worth having: it is the only thing that proves the fallback still works, and a suspected cross-test interaction deserves re-running under full isolation before it is believed.
• CI uses a job-level services: block with a health check, which is the idiomatic form and needs no script.

Worth a reviewer's attention

The isolation model changed. Tests previously had a private server; they now have a private database on a shared one. That is the same isolation for anything schema-scoped — which is everything these suites touch — but cluster-scoped state is now shared: roles, advisory locks, pg_stat views. Nothing in the suite asserts on those today, and all 1042 pass, but it is the assumption a future test could quietly break.

The odal_app role is created once per cluster rather than once per test, and CREATE ROLE failing as a duplicate is ignored for that reason.

Note on the required-status-check change

Merging #177 needed the main ruleset updated: dropping the check-features job orphaned a required check named "Feature-gated code compiles", which could then never report. That was my miss — I removed a job without reading the ruleset first. The check was removed from the required list with the operator's approval, and the ruleset was backed up before the edit. Clippy now runs with those exact features, so coverage went up, not down.

Verification

just check green: 842 tests, all gates including harness-check, cargo audit. Full integration tier run end to end against a shared server: 1042 passing. Fallback path re-verified by running pg_integration with the env var unset.

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 9 complexity · -4 duplication

Metric	Results
Complexity	9
Duplication	-4

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}