diff --git a/.redocly.lint-ignore.yaml b/.redocly.lint-ignore.yaml index 1f0b1d1..7546ab9 100644 --- a/.redocly.lint-ignore.yaml +++ b/.redocly.lint-ignore.yaml @@ -30,6 +30,9 @@ api/openapi.bundled.yaml: - '#/paths/~1identity~1ready/get' - '#/paths/~1integrator~1health/get' - '#/paths/~1integrator~1api~1v1~1templates~1{sector}/get' + - '#/paths/~1integrator~1api~1v1~1schemas/get' + - '#/paths/~1integrator~1api~1v1~1schemas~1{sector}/get' + - '#/paths/~1integrator~1api~1v1~1schemas~1{sector}~1{version}/get' - '#/paths/~1dpp~1{dppId}/get' - '#/paths/~1dpp~1{dppId}~1qr/get' - '#/paths/~101~1{gtin}/get' diff --git a/CHANGELOG.md b/CHANGELOG.md index aab64dd..70e5f85 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,8 @@ under the pre-1.0 conventions in [VERSIONING.md](docs/governance/VERSIONING.md): ## [Unreleased] +## [0.12.0] - 2026-08-23 + ### Breaking - **`publishValid` is now `sectorDataValid`.** *(Breaking: a response field is @@ -180,6 +182,47 @@ under the pre-1.0 conventions in [VERSIONING.md](docs/governance/VERSIONING.md): ### Added +- **Sector JSON Schemas are fetchable.** `GET /integrator/api/v1/schemas` lists + every sector with a schema, the version a new passport is validated against, + and every version a stored passport may legitimately record; + `/schemas/{sector}` serves the current one and `/schemas/{sector}/{version}` + a pinned one (a leading `v` is accepted). Unauthenticated. + + An SDK or dashboard previously had no way to see the contract before building + a body — the only feedback was a rejection from the create route, or the CSV + import template, which is an import artefact rather than a schema. These + resolve through the same registry the publish gate validates against, never a + copy: a second copy would drift, and the direction it drifts is the one where + a body passes here and fails at publish. + + **Every `description` is omitted from the served document, deliberately and + temporarily.** Those fields make regulatory assertions — act numbers, + adoption dates, effective dates, product-class scope, annex references — that + have not been verified against primary text; two electronics descriptions once + asserted an adoption date, an effective date, three named priority product + classes and a phase-two date for an act that does not exist. Inside a library + those are developer-facing comments; on a public endpoint they become a + product surface a consumer reads, caches and relies on. Everything that + decides accept or reject — types, `enum`, `required`, `pattern`, bounds, + `additionalProperties` — is served in full, so a client can pre-validate a + body and get the verdict the create route would give. The prose is restored + once the audit has verified it; the stripping is one function and one call + site, marked as such. + + `title` is kept: they are short labels, not assertions. + +- **A node says which catalogued sectors have no plugin loaded.** One line at + boot naming them, at `warn`. + + Passthrough is a legitimate configuration, so this does not refuse to boot. + But a sector with no plugin and a sector whose plugin found nothing wrong + produce the same thing — a determination with no findings — so from the + outside they were indistinguishable, and "no violations" read as "checked and + clean" when it could mean "never checked". `warn` rather than `info` because a + production node loads a full signed set from the release pipeline, so a gap + there is a misconfiguration, and `info` is where it would be missed. + + - **The CLI has an automated test tier.** `cli/tests/` runs the `odal` binary as a child process and asserts on exit codes, output, and what lands in `config.toml`. Nothing in the suite previously reached the CLI's behaviour — @@ -502,6 +545,33 @@ under the pre-1.0 conventions in [VERSIONING.md](docs/governance/VERSIONING.md): ### Changed +- **Shared test scaffolding has one home, and a gate that keeps it that way.** + The throwaway-Postgres harness and the in-memory `PassportRepository` double + now live behind `dpp-dal`'s dev-only `test-harness` feature, as + `dpp_dal::test_harness` and `dpp_dal::in_memory_repo`. + + Rust cannot share `#[cfg(test)]` code across crate boundaries, so copying is + the path of least resistance and nothing signalled when it happened. `start_pg` + had reached **eight copies that had drifted into six distinct + implementations**, each carrying its own hardcoded readiness sleep; the + repository double had reached three, with the `impl` blocks byte-identical and + the structs already diverging. `just harness-check` now fails the build when + either is defined outside its home, and was verified to fail before being + wired in. No new crate: every consumer already depends on `dpp-dal`, which is + `publish = false`, so none of this ships. + + Checked and deliberately left alone: the wire-shaped `serde_json::Value` + passport builders (a different thing from the typed ones) and the per-suite + auth doubles (different implementations, small, purpose-built). Merging those + would couple unrelated tests to one double's behaviour. + +- **Test keystores use `tempfile::tempdir()`.** Nine sites opened an Ed25519 + keystore at a hand-built path under `std::env::temp_dir()`, which gives no + restrictive permissions and no cleanup — every run left the file behind. + Severity was low and stays stated plainly: the passphrases are literals, the + keys are throwaway, and nothing production-adjacent reads those paths. + + - **The API description is authored multi-file and shipped as one file.** `api/openapi.yaml` is now a thin root — `info`, `servers`, security schemes, tags and a `$ref` per path — over `api/paths/` and `api/components/`. diff --git a/CLAUDE.md b/CLAUDE.md index 5bd08db..5cb81c8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -444,7 +444,10 @@ Test tiers: Two things that bite: - **A feature-gated suite that stops compiling fails only in CI.** `just test` skips them entirely. Run `just check` before pushing, not `just test`. -- **Adding a `#[cfg(test)]` helper does not make it reachable from another crate.** Rust cannot share test code across crate boundaries, which is why the Postgres harness is duplicated per suite. Follow the local copy rather than inventing a new one. +- **Adding a `#[cfg(test)]` helper does not make it reachable from another crate.** Rust cannot share test code across crate boundaries, so the reflex is to copy — and the Postgres harness reached eight copies that had drifted into six different implementations before anyone noticed. +- **Shared test scaffolding has one home, behind `dpp-dal`'s `test-harness` feature**, enabled from `[dev-dependencies]`. Two things live there today: `test_harness` (`start_pg`, `start_pg_raw`, `start_pg_before`) and `in_memory_repo` (`InMemoryPassportRepo`). **Do not write another one** — if the shared version cannot do what a suite needs, extend it there rather than forking a copy. `just harness-check` fails the build if you do. +- **Not everything that shares a name is duplication.** Checked and deliberately left alone: the three `serde_json::Value` passport builders (wire-shaped, for HTTP tests — a different thing from the typed `Passport` builders), and the `TestAuthProvider` / `AlwaysFail` doubles (different implementations per suite, small, purpose-built). Merging those would couple unrelated tests to one double's behaviour. The three *typed* `Passport` builders are a real candidate and are deferred, not rejected — extract them when a fourth appears, or when two of the three need the same new field. +- **A test keystore uses `tempfile::tempdir()`, never `std::env::temp_dir()`.** These files hold Ed25519 private keys; `tempfile` creates the directory with restrictive permissions and removes it on drop, and the hand-rolled path did neither. Return the `TempDir` alongside the store so the directory outlives it. ## Standing Conventions diff --git a/Cargo.lock b/Cargo.lock index 2b178f4..13ed7a1 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2168,7 +2168,7 @@ dependencies = [ [[package]] name = "dpp-cli" -version = "0.11.0" +version = "0.12.0" dependencies = [ "anyhow", "base64 0.23.1", @@ -2192,7 +2192,7 @@ dependencies = [ [[package]] name = "dpp-common" -version = "0.11.0" +version = "0.12.0" dependencies = [ "anyhow", "async-trait", @@ -2244,11 +2244,12 @@ dependencies = [ [[package]] name = "dpp-dal" -version = "0.11.0" +version = "0.12.0" dependencies = [ "anyhow", "async-trait", "chrono", + "dpp-dal", "dpp-domain", "dpp-types", "hex", @@ -2297,7 +2298,7 @@ dependencies = [ [[package]] name = "dpp-factor-data" -version = "0.11.0" +version = "0.12.0" dependencies = [ "chrono", "dpp-calc", @@ -2310,7 +2311,7 @@ dependencies = [ [[package]] name = "dpp-identity" -version = "0.11.0" +version = "0.12.0" dependencies = [ "anyhow", "axum", @@ -2324,6 +2325,7 @@ dependencies = [ "serde", "serde_json", "serial_test", + "tempfile", "thiserror 2.0.19", "tokio", "tower", @@ -2336,7 +2338,7 @@ dependencies = [ [[package]] name = "dpp-integrator" -version = "0.11.0" +version = "0.12.0" dependencies = [ "anyhow", "async-trait", @@ -2369,7 +2371,7 @@ dependencies = [ [[package]] name = "dpp-node" -version = "0.11.0" +version = "0.12.0" dependencies = [ "anyhow", "async-nats", @@ -2421,7 +2423,7 @@ dependencies = [ [[package]] name = "dpp-plugin-host" -version = "0.11.0" +version = "0.12.0" dependencies = [ "anyhow", "base64 0.23.1", @@ -2475,7 +2477,7 @@ dependencies = [ [[package]] name = "dpp-render" -version = "0.11.0" +version = "0.12.0" dependencies = [ "chrono", "dpp-digital-link", @@ -2487,7 +2489,7 @@ dependencies = [ [[package]] name = "dpp-resolver" -version = "0.11.0" +version = "0.12.0" dependencies = [ "anyhow", "axum", @@ -2539,7 +2541,7 @@ dependencies = [ [[package]] name = "dpp-seal" -version = "0.11.0" +version = "0.12.0" dependencies = [ "async-trait", "axum", @@ -2568,7 +2570,7 @@ dependencies = [ [[package]] name = "dpp-types" -version = "0.11.0" +version = "0.12.0" dependencies = [ "async-trait", "chrono", @@ -2585,7 +2587,7 @@ dependencies = [ [[package]] name = "dpp-vault" -version = "0.11.0" +version = "0.12.0" dependencies = [ "anyhow", "async-trait", @@ -2614,6 +2616,7 @@ dependencies = [ "serde_json", "sha2 0.11.0", "subtle", + "tempfile", "testcontainers", "testcontainers-modules", "thiserror 2.0.19", diff --git a/Cargo.toml b/Cargo.toml index 8099256..8f95728 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -17,7 +17,7 @@ members = [ ] [workspace.package] -version = "0.11.0" +version = "0.12.0" edition = "2024" authors = ["Odal Node "] license = "BSL-1.1" diff --git a/api/components/schemas/Problem.yaml b/api/components/schemas/Problem.yaml new file mode 100644 index 0000000..07a779d --- /dev/null +++ b/api/components/schemas/Problem.yaml @@ -0,0 +1,34 @@ +type: object +description: | + RFC 7807 / RFC 9457 problem details. The shape + `dpp-common::http_problem::Problem` produces, served as + `application/problem+json`. + + `type` is derived from `title`, so each distinct `title` used across the + codebase is a stable catalogue key that clients may depend on. +required: + - type + - title + - status +properties: + type: + type: string + format: uri + description: Absolute URI identifying the problem type. + example: https://problems.odal-node.io/not-found + title: + type: string + description: Short human-readable summary of the problem type. + example: Not Found + status: + type: integer + description: The HTTP status code, mirroring the status line. + example: 404 + detail: + type: string + description: Human-readable explanation for this specific occurrence. + example: "No schema for sector 'nosuchsector'. Known sectors: aluminium, battery." + instance: + type: string + format: uri-reference + description: URI reference identifying this specific occurrence. diff --git a/api/openapi.bundled.yaml b/api/openapi.bundled.yaml index deb679c..cb2f579 100644 --- a/api/openapi.bundled.yaml +++ b/api/openapi.bundled.yaml @@ -1,7 +1,7 @@ openapi: 3.1.0 info: title: Odal Node API - version: 0.11.0 + version: 0.12.0 description: | **Sovereign Digital Product Passport Infrastructure** @@ -2292,6 +2292,125 @@ paths: description: No template for this sector. '501': description: XLSX export not yet implemented. + /integrator/api/v1/schemas: + get: + operationId: listSectorSchemas + summary: List sector schemas and their versions + description: | + Every sector with a JSON Schema, the version a new passport is validated + against (`current`), and every version a stored passport may legitimately + record (`versions`). Unauthenticated. + tags: + - Integrator + responses: + '200': + description: The available sector schemas. + content: + application/json: + schema: + type: object + properties: + schemas: + type: array + items: + type: object + properties: + sector: + type: string + example: battery + current: + type: + - string + - 'null' + example: 2.6.0 + versions: + type: array + items: + type: string + example: + - 1.0.0 + - 2.6.0 + /integrator/api/v1/schemas/{sector}: + get: + operationId: getCurrentSectorSchema + summary: Fetch a sector's current JSON Schema + description: | + The schema a passport created today is validated against, resolved through + the same registry the publish gate uses — never a copy, which would drift in + the direction where a body passes here and fails at publish. Unauthenticated. + + Every `description` is omitted from the served document. Those fields make + regulatory assertions that have not been verified against primary text, and + serving them would turn developer-facing comments into a product surface. + Everything that decides accept or reject — types, `enum`, `required`, + `pattern`, bounds, `additionalProperties` — is served in full, so a client + can pre-validate a body and get the verdict the create route would give. + tags: + - Integrator + parameters: + - name: sector + in: path + required: true + schema: + type: string + example: battery + responses: + '200': + description: The sector's current JSON Schema. + content: + application/json: + schema: + type: object + '404': + description: No schema for this sector; the body names the known sectors. + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + /integrator/api/v1/schemas/{sector}/{version}: + get: + operationId: getPinnedSectorSchema + summary: Fetch a pinned version of a sector's JSON Schema + description: | + A stored passport records the `schemaVersion` it was written under, so a + client holding one needs that exact schema rather than whatever is current. + The version may be given with or without a leading `v`. Unauthenticated. + + Descriptions are omitted, as on the current-schema route. + tags: + - Integrator + parameters: + - name: sector + in: path + required: true + schema: + type: string + example: battery + - name: version + in: path + required: true + schema: + type: string + example: 2.6.0 + responses: + '200': + description: The sector's JSON Schema at that version. + content: + application/json: + schema: + type: object + '400': + description: The version is not a semver string. + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + '404': + description: No schema at that version; the body names what is available. + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' /integrator/api/v1/import/{sector}: post: operationId: importFile @@ -4100,6 +4219,41 @@ components: type: boolean did_document: $ref: '#/components/schemas/DidDocument' + Problem: + type: object + description: | + RFC 7807 / RFC 9457 problem details. The shape + `dpp-common::http_problem::Problem` produces, served as + `application/problem+json`. + + `type` is derived from `title`, so each distinct `title` used across the + codebase is a stable catalogue key that clients may depend on. + required: + - type + - title + - status + properties: + type: + type: string + format: uri + description: Absolute URI identifying the problem type. + example: https://problems.odal-node.io/not-found + title: + type: string + description: Short human-readable summary of the problem type. + example: Not Found + status: + type: integer + description: The HTTP status code, mirroring the status line. + example: 404 + detail: + type: string + description: Human-readable explanation for this specific occurrence. + example: 'No schema for sector ''nosuchsector''. Known sectors: aluminium, battery.' + instance: + type: string + format: uri-reference + description: URI reference identifying this specific occurrence. ImportCreatedEntry: type: object required: diff --git a/api/openapi.yaml b/api/openapi.yaml index a386149..d8d2307 100644 --- a/api/openapi.yaml +++ b/api/openapi.yaml @@ -1,7 +1,7 @@ openapi: 3.1.0 info: title: Odal Node API - version: 0.11.0 + version: 0.12.0 description: | **Sovereign Digital Product Passport Infrastructure** @@ -184,6 +184,12 @@ paths: $ref: paths/integrator_health.yaml /integrator/api/v1/templates/{sector}: $ref: paths/integrator_api_v1_templates_{sector}.yaml + /integrator/api/v1/schemas: + $ref: paths/integrator_api_v1_schemas.yaml + /integrator/api/v1/schemas/{sector}: + $ref: paths/integrator_api_v1_schemas_{sector}.yaml + /integrator/api/v1/schemas/{sector}/{version}: + $ref: paths/integrator_api_v1_schemas_{sector}_{version}.yaml /integrator/api/v1/import/{sector}: $ref: paths/integrator_api_v1_import_{sector}.yaml /integrator/api/v1/imports/{job_id}: diff --git a/api/paths/integrator_api_v1_schemas.yaml b/api/paths/integrator_api_v1_schemas.yaml new file mode 100644 index 0000000..a5d3bc1 --- /dev/null +++ b/api/paths/integrator_api_v1_schemas.yaml @@ -0,0 +1,37 @@ +get: + operationId: listSectorSchemas + summary: List sector schemas and their versions + description: | + Every sector with a JSON Schema, the version a new passport is validated + against (`current`), and every version a stored passport may legitimately + record (`versions`). Unauthenticated. + tags: + - Integrator + responses: + '200': + description: The available sector schemas. + content: + application/json: + schema: + type: object + properties: + schemas: + type: array + items: + type: object + properties: + sector: + type: string + example: battery + current: + type: + - string + - 'null' + example: 2.6.0 + versions: + type: array + items: + type: string + example: + - 1.0.0 + - 2.6.0 diff --git a/api/paths/integrator_api_v1_schemas_{sector}.yaml b/api/paths/integrator_api_v1_schemas_{sector}.yaml new file mode 100644 index 0000000..9a5e8a6 --- /dev/null +++ b/api/paths/integrator_api_v1_schemas_{sector}.yaml @@ -0,0 +1,36 @@ +get: + operationId: getCurrentSectorSchema + summary: Fetch a sector's current JSON Schema + description: | + The schema a passport created today is validated against, resolved through + the same registry the publish gate uses — never a copy, which would drift in + the direction where a body passes here and fails at publish. Unauthenticated. + + Every `description` is omitted from the served document. Those fields make + regulatory assertions that have not been verified against primary text, and + serving them would turn developer-facing comments into a product surface. + Everything that decides accept or reject — types, `enum`, `required`, + `pattern`, bounds, `additionalProperties` — is served in full, so a client + can pre-validate a body and get the verdict the create route would give. + tags: + - Integrator + parameters: + - name: sector + in: path + required: true + schema: + type: string + example: battery + responses: + '200': + description: The sector's current JSON Schema. + content: + application/json: + schema: + type: object + '404': + description: No schema for this sector; the body names the known sectors. + content: + application/problem+json: + schema: + $ref: ../components/schemas/Problem.yaml diff --git a/api/paths/integrator_api_v1_schemas_{sector}_{version}.yaml b/api/paths/integrator_api_v1_schemas_{sector}_{version}.yaml new file mode 100644 index 0000000..2386eba --- /dev/null +++ b/api/paths/integrator_api_v1_schemas_{sector}_{version}.yaml @@ -0,0 +1,43 @@ +get: + operationId: getPinnedSectorSchema + summary: Fetch a pinned version of a sector's JSON Schema + description: | + A stored passport records the `schemaVersion` it was written under, so a + client holding one needs that exact schema rather than whatever is current. + The version may be given with or without a leading `v`. Unauthenticated. + + Descriptions are omitted, as on the current-schema route. + tags: + - Integrator + parameters: + - name: sector + in: path + required: true + schema: + type: string + example: battery + - name: version + in: path + required: true + schema: + type: string + example: 2.6.0 + responses: + '200': + description: The sector's JSON Schema at that version. + content: + application/json: + schema: + type: object + '400': + description: The version is not a semver string. + content: + application/problem+json: + schema: + $ref: ../components/schemas/Problem.yaml + '404': + description: No schema at that version; the body names what is available. + content: + application/problem+json: + schema: + $ref: ../components/schemas/Problem.yaml diff --git a/crates/dpp-dal/Cargo.toml b/crates/dpp-dal/Cargo.toml index 2490634..c1ee417 100644 --- a/crates/dpp-dal/Cargo.toml +++ b/crates/dpp-dal/Cargo.toml @@ -28,14 +28,25 @@ chrono = { workspace = true } tracing = { workspace = true } metrics = { workspace = true } +testcontainers = { workspace = true, optional = true } + sqlx = { version = "0.9", default-features = false, features = [ "postgres", "runtime-tokio", "tls-rustls", "uuid", "chrono", "json", "migrate", "macros", ] } [features] integration-tests = [] +# Exposes `test_harness`: the shared Postgres container harness, used by this +# crate's own suites and by dpp-node's. Off by default so `testcontainers` and a +# container-spawning API stay out of every ordinary build of the DAL. Only ever +# enabled from a `[dev-dependencies]` entry. +test-harness = ["dep:testcontainers"] [dev-dependencies] +# This crate depends on itself with `test-harness` on, because its integration +# suites in `tests/` are separate crates and cannot see a feature-gated module +# otherwise. Legal and self-contained; it reads oddly, which is why it is noted. +dpp-dal = { path = ".", features = ["test-harness"] } testcontainers = { workspace = true } tokio = { workspace = true } uuid = { workspace = true } diff --git a/crates/dpp-dal/src/in_memory_repo.rs b/crates/dpp-dal/src/in_memory_repo.rs new file mode 100644 index 0000000..c5e2049 --- /dev/null +++ b/crates/dpp-dal/src/in_memory_repo.rs @@ -0,0 +1,129 @@ +//! An in-memory [`PassportRepository`], for suites that need the port without a +//! database. +//! +//! # Why it lives beside `PgPassportRepo` +//! +//! It is an alternative implementation of the same port, so it belongs with the +//! other one rather than in a test-support crate. Both consumers — +//! `dpp-node`'s suites and `dpp-vault`'s — already depend on `dpp-dal`. +//! +//! It was copied into three suites first. The `impl` blocks were byte-for-byte +//! identical; the structs had already diverged, one having grown +//! `Arc` + `Clone` that the other two lacked. That is the same drift the +//! Postgres harness went through on its way to eight copies and six +//! implementations, caught earlier. +//! +//! Gated behind `test-harness` with the container harness, and `dpp-dal` is +//! `publish = false`, so it ships nowhere. +//! +//! # What it is not +//! +//! Not a substitute for the Postgres suites. It stores passports in a map and +//! enforces none of the things the database does — no retention trigger, no +//! append-only audit, no app-role privilege boundary, no `LIKE` escaping. A test +//! asserting any of those must use [`start_pg`](crate::test_harness::start_pg). + +use std::collections::HashMap; +use std::sync::{Arc, Mutex}; + +use dpp_domain::domain::error::DppError; +use dpp_domain::domain::passport::{Passport, PassportId}; +use dpp_domain::domain::status::PassportStatus; +use dpp_domain::ports::passport_repo::PassportRepository; + +/// A [`PassportRepository`] backed by a `HashMap`. +/// +/// `Default` is the only constructor; it starts empty. +/// +/// `Clone` shares the same map rather than copying it — two clones see each +/// other's writes. That is what a suite handing the repo to a component while +/// keeping a handle to assert against needs, and it is why the store is behind +/// an `Arc`: of the three copies this replaces, one had already grown that +/// requirement and the other two had not. +#[derive(Default, Clone)] +pub struct InMemoryPassportRepo { + store: Arc>>, +} + +#[async_trait::async_trait] +impl PassportRepository for InMemoryPassportRepo { + async fn create(&self, passport: Passport) -> Result { + self.store + .lock() + .unwrap() + .insert(passport.id, passport.clone()); + Ok(passport) + } + + async fn find_by_id(&self, id: PassportId) -> Result, DppError> { + Ok(self.store.lock().unwrap().get(&id).cloned()) + } + + /// Returns any stored passport regardless of status. + /// + /// Deliberately not filtered: a suite using this double is exercising a + /// caller, not the publication policy, and a double that silently hid + /// non-published rows would make those callers look correct when they are + /// not. A test that needs the real filter needs the real repository. + async fn find_published_by_id(&self, id: PassportId) -> Result, DppError> { + self.find_by_id(id).await + } + + /// Always `None` — GTIN lookup is not modelled here. + /// + /// The real query matches a GS1 Digital Link path segment inside + /// `qrCodeUrl` and refuses non-numeric input so a `LIKE` metacharacter + /// cannot widen the match. Approximating that in a map would make a test + /// pass against behaviour the database does not have, so this answers + /// nothing rather than answering wrongly. + async fn find_published_by_gtin(&self, _gtin: &str) -> Result, DppError> { + Ok(None) + } + + async fn find_by_id_any_status(&self, id: PassportId) -> Result, DppError> { + self.find_by_id(id).await + } + + async fn update(&self, passport: Passport) -> Result { + self.store + .lock() + .unwrap() + .insert(passport.id, passport.clone()); + Ok(passport) + } + + async fn update_status( + &self, + id: PassportId, + status: PassportStatus, + ) -> Result { + let mut g = self.store.lock().unwrap(); + let mut p = g + .get(&id) + .cloned() + .ok_or_else(|| DppError::NotFound(id.to_string()))?; + p.status = status; + g.insert(id, p.clone()); + Ok(p) + } + + /// Every stored passport. Filters and paging are ignored. + async fn list( + &self, + _status: Option, + _q: Option<&str>, + _facility_id: Option<&str>, + _limit: u32, + _offset: u32, + ) -> Result, DppError> { + Ok(self.store.lock().unwrap().values().cloned().collect()) + } + + async fn count( + &self, + _status: Option, + _facility_id: Option<&str>, + ) -> Result { + Ok(self.store.lock().unwrap().len() as u64) + } +} diff --git a/crates/dpp-dal/src/lib.rs b/crates/dpp-dal/src/lib.rs index 42fc2ad..f075c02 100644 --- a/crates/dpp-dal/src/lib.rs +++ b/crates/dpp-dal/src/lib.rs @@ -3,3 +3,15 @@ //! Single backend: [`pg`] — PostgreSQL via sqlx. The [`pg`] module exposes //! one concrete struct per domain aggregate and re-exports them at crate root. pub mod pg; + +/// Shared throwaway-Postgres harness for integration suites across the +/// workspace. Dev-only: gated behind `test-harness`, which nothing outside a +/// `[dev-dependencies]` entry may enable. +#[cfg(feature = "test-harness")] +pub mod test_harness; + +/// An in-memory [`PassportRepository`](dpp_domain::ports::passport_repo::PassportRepository) +/// for suites that need the port without a database. Dev-only, same gate as +/// [`test_harness`]. +#[cfg(feature = "test-harness")] +pub mod in_memory_repo; diff --git a/crates/dpp-dal/src/test_harness.rs b/crates/dpp-dal/src/test_harness.rs new file mode 100644 index 0000000..fad4107 --- /dev/null +++ b/crates/dpp-dal/src/test_harness.rs @@ -0,0 +1,191 @@ +//! One throwaway-Postgres harness, shared by every integration suite that needs +//! a database. +//! +//! # Why it is shared at all +//! +//! Rust cannot share `#[cfg(test)]` code across crate boundaries, so a +//! `mod helpers` in one suite is invisible to the next. That is why `start_pg` +//! was copied into eight files instead of being written once — and by the time +//! it was extracted, those eight copies had already drifted into **six** +//! distinct implementations of the same fifteen lines. +//! +//! The cost was never aesthetic. Each copy carried its own hardcoded readiness +//! sleep, so getting the bootstrap sequence right was eight independent +//! problems; and a change to it — a new role grant, a different image pin — +//! meant eight edits or seven silent divergences. +//! +//! # Why it lives in `dpp-dal` and not its own crate +//! +//! Every consumer already depends on `dpp-dal`: this crate's own suites, and +//! `dpp-node`'s five. A crate holding one function for callers who could +//! already see it earns nothing, and the workspace takes no crate fission +//! before 1.0. +//! +//! Gated behind `test-harness`, off by default. `dpp-dal` is `publish = false`, +//! so this ships nowhere regardless; the feature keeps `testcontainers` and a +//! container-spawning API out of every ordinary build of the DAL. Nothing +//! outside a `[dev-dependencies]` entry may enable it. + +use crate::pg::{PgDal, sqlx}; +use testcontainers::{ + ContainerAsync, GenericImage, ImageExt, + core::{WaitFor, ports::ContainerPort}, + runners::AsyncRunner, +}; + +/// The image every suite tests against. One constant, so a pin bump is one edit. +const POSTGRES_IMAGE: (&str, &str) = ("postgres", "17"); + +/// How long to wait after the container reports ready before connecting. +/// +/// Postgres restarts once during first-time init, so the readiness line on +/// stderr appears *before* the server is actually reachable. Connecting into +/// that window fails, which is why every copy of this harness carried a sleep. +const POST_INIT_SETTLE: std::time::Duration = std::time::Duration::from_millis(1500); + +/// A running Postgres with the app role provisioned and no migrations applied. +/// +/// The building block. Suites that want the ordinary arrangement should call +/// [`start_pg`]; this exists for the ones that need to control which migrations +/// run, such as a test of a migration itself. +pub struct RawPg { + /// Superuser connection URL — DDL, raw trigger assertions, migrations. + pub admin_url: String, + /// Application-role URL. `odal_app` has no DDL and a narrow DELETE set, so + /// a test connecting as this role exercises the privileges production has. + pub app_url: String, + /// Held so the container outlives the test; dropping it stops the container. + pub container: ContainerAsync, +} + +/// A running Postgres with every migration applied and a connected [`PgDal`]. +pub struct TestPg { + /// Connected as `odal_app`, the role production uses. + pub dal: PgDal, + /// Superuser URL, kept for assertions that need to bypass the app role — + /// checking that a trigger fired, or that an append-only table refuses. + pub admin_url: String, + /// Application-role URL, for a suite that opens its own pool. + pub app_url: String, + _container: ContainerAsync, +} + +/// Start Postgres and provision the `odal_app` role. No migrations. +/// +/// The role is created exactly as `ops/bootstrap/pg-init.sh` does, so a suite +/// meets the same privilege boundary a deployed node does. +pub async fn start_pg_raw() -> RawPg { + let image = GenericImage::new(POSTGRES_IMAGE.0, POSTGRES_IMAGE.1) + .with_exposed_port(ContainerPort::Tcp(5432)) + .with_wait_for(WaitFor::message_on_stderr( + "database system is ready to accept connections", + )) + // The official Postgres image's own required env vars for a throwaway + // container — NOT the app's DATABASE_POSTGRES_PASS / DATABASE_APP_PASS + // scheme, which these deliberately do not mirror. + .with_env_var("POSTGRES_USER", "postgres") + .with_env_var("POSTGRES_PASSWORD", "test") + .with_env_var("POSTGRES_DB", "odal"); + + let container = image.start().await.expect("start postgres container"); + let port = container + .get_host_port_ipv4(5432) + .await + .expect("mapped port"); + let admin_url = format!("postgres://postgres:test@127.0.0.1:{port}/odal"); + let app_url = format!("postgres://odal_app:test@127.0.0.1:{port}/odal"); + + tokio::time::sleep(POST_INIT_SETTLE).await; + + let admin = sqlx::postgres::PgPoolOptions::new() + .max_connections(1) + .connect(&admin_url) + .await + .expect("admin connect"); + sqlx::query("CREATE ROLE odal_app LOGIN PASSWORD 'test'") + .execute(&admin) + .await + .expect("create app role"); + admin.close().await; + + RawPg { + admin_url, + app_url, + container, + } +} + +/// Start Postgres, apply every migration, and connect as `odal_app`. +/// +/// Migrations need DDL, so they run through the admin URL; the returned +/// [`PgDal`] then connects as the app role without re-running them, which +/// mirrors the ops workflow. +pub async fn start_pg() -> TestPg { + let raw = start_pg_raw().await; + + PgDal::migrate(&raw.admin_url) + .await + .expect("apply migrations via admin"); + + let dal = PgDal::connect(&raw.app_url).await.expect("app connect"); + + TestPg { + dal, + admin_url: raw.admin_url, + app_url: raw.app_url, + _container: raw.container, + } +} + +/// Start Postgres and apply migrations in order, stopping **before** the first +/// whose filename begins with `stop_before`. +/// +/// For testing a migration itself: bring the schema to the state that existed +/// just before it, then apply it and assert what it did. +/// +/// # Panics +/// +/// If `stop_before` matches no migration. A test pinned to a prefix that no +/// longer exists would otherwise apply every migration and silently assert +/// nothing — the failure mode this whole crate exists to reduce. +pub async fn start_pg_before(stop_before: &str) -> RawPg { + let raw = start_pg_raw().await; + + let admin = sqlx::postgres::PgPoolOptions::new() + .max_connections(1) + .connect(&raw.admin_url) + .await + .expect("admin connect"); + + let dir = concat!(env!("CARGO_MANIFEST_DIR"), "/../../ops/pg"); + let mut files: Vec<_> = std::fs::read_dir(dir) + .expect("read ops/pg") + .filter_map(|e| e.ok().map(|e| e.path())) + .filter(|p| p.extension().is_some_and(|x| x == "sql")) + .collect(); + files.sort(); + + let mut stopped = false; + for path in files { + let name = path.file_name().unwrap_or_default().to_string_lossy(); + if name.starts_with(stop_before) { + stopped = true; + break; + } + let sql = std::fs::read_to_string(&path).expect("read migration"); + // Repo-controlled migration text from ops/pg, never caller input. + sqlx::raw_sql(sqlx::AssertSqlSafe(sql)) + .execute(&admin) + .await + .unwrap_or_else(|e| panic!("apply {name}: {e}")); + } + admin.close().await; + + assert!( + stopped, + "no migration in ops/pg starts with '{stop_before}', so every migration was \ + applied and the test would assert against the wrong schema" + ); + + raw +} diff --git a/crates/dpp-dal/tests/pg_integration.rs b/crates/dpp-dal/tests/pg_integration.rs index c6ac2fc..3aab6dd 100644 --- a/crates/dpp-dal/tests/pg_integration.rs +++ b/crates/dpp-dal/tests/pg_integration.rs @@ -22,17 +22,13 @@ #![cfg(feature = "integration-tests")] -use testcontainers::{ - GenericImage, ImageExt, - core::{WaitFor, ports::ContainerPort}, - runners::AsyncRunner, -}; use uuid::Uuid; use dpp_dal::pg::{ PgApiKeyRepo, PgAuditRepo, PgDal, PgEvidenceDossierRepo, PgPassportRepo, PgScanTelemetryRepo, PgSnapshotOutboxRepo, sqlx, }; +use dpp_dal::test_harness::start_pg; use dpp_domain::{ domain::{ gtin::Gtin, @@ -54,62 +50,6 @@ use dpp_types::{ }; use sqlx::Row; -struct TestPg { - dal: PgDal, - /// Superuser URL kept for raw admin-side assertions (T4 trigger checks). - admin_url: String, - _container: testcontainers::ContainerAsync, -} - -async fn start_pg() -> TestPg { - let image = GenericImage::new("postgres", "17") - .with_exposed_port(ContainerPort::Tcp(5432)) - .with_wait_for(WaitFor::message_on_stderr( - "database system is ready to accept connections", - )) - // POSTGRES_USER/PASSWORD/DB are the official Postgres image's required - // env vars for this throwaway testcontainer — NOT the app's - // DATABASE_POSTGRES_PASS / DATABASE_APP_PASS scheme. - .with_env_var("POSTGRES_USER", "postgres") - .with_env_var("POSTGRES_PASSWORD", "test") - .with_env_var("POSTGRES_DB", "odal"); - - let container = image.start().await.expect("start postgres container"); - let port = container - .get_host_port_ipv4(5432) - .await - .expect("mapped port"); - let admin_url = format!("postgres://postgres:test@127.0.0.1:{port}/odal"); - - // Postgres restarts once during init — give it a moment, then provision - // the app role exactly like ops/bootstrap/pg-init.sh does. - tokio::time::sleep(std::time::Duration::from_millis(1500)).await; - let admin = sqlx::postgres::PgPoolOptions::new() - .max_connections(1) - .connect(&admin_url) - .await - .expect("admin connect"); - sqlx::query("CREATE ROLE odal_app LOGIN PASSWORD 'test'") - .execute(&admin) - .await - .expect("create app role"); - - // Migrations require DDL privileges; run them via the admin pool directly, - // then connect as odal_app (PgDal::migrate mirrors the ops/just workflow). - PgDal::migrate(&admin_url) - .await - .expect("apply 0001_init via admin"); - - let app_url = format!("postgres://odal_app:test@127.0.0.1:{port}/odal"); - let dal = PgDal::connect(&app_url).await.expect("app connect"); - - TestPg { - dal, - admin_url, - _container: container, - } -} - fn make_passport() -> Passport { Passport { id: PassportId::new(), diff --git a/crates/dpp-dal/tests/pg_seal_outbox.rs b/crates/dpp-dal/tests/pg_seal_outbox.rs index e94c170..c53456e 100644 --- a/crates/dpp-dal/tests/pg_seal_outbox.rs +++ b/crates/dpp-dal/tests/pg_seal_outbox.rs @@ -24,64 +24,14 @@ #![cfg(feature = "integration-tests")] use chrono::Utc; -use dpp_dal::pg::{PgDal, PgPassportRepo, PgSealOutboxRepo}; +use dpp_dal::pg::{PgPassportRepo, PgSealOutboxRepo}; +use dpp_dal::test_harness::start_pg; use dpp_domain::domain::passport::{ManufacturerInfo, Passport, PassportId}; use dpp_domain::domain::sector::Sector; use dpp_domain::domain::status::PassportStatus; use dpp_domain::ports::passport_repo::PassportRepository; use dpp_domain::ports::seal::{SealFormat, SealedEnvelope}; use dpp_types::SealOutbox; -use testcontainers::core::{ContainerPort, WaitFor}; -use testcontainers::runners::AsyncRunner; -use testcontainers::{GenericImage, ImageExt}; - -struct TestPg { - dal: PgDal, - _container: testcontainers::ContainerAsync, -} - -async fn start_pg() -> TestPg { - let image = GenericImage::new("postgres", "17") - .with_exposed_port(ContainerPort::Tcp(5432)) - .with_wait_for(WaitFor::message_on_stderr( - "database system is ready to accept connections", - )) - .with_env_var("POSTGRES_USER", "postgres") - .with_env_var("POSTGRES_PASSWORD", "test") - .with_env_var("POSTGRES_DB", "odal"); - - let container = image.start().await.expect("start postgres container"); - let port = container - .get_host_port_ipv4(5432) - .await - .expect("mapped port"); - let admin_url = format!("postgres://postgres:test@127.0.0.1:{port}/odal"); - - tokio::time::sleep(std::time::Duration::from_millis(1500)).await; - let admin = sqlx::postgres::PgPoolOptions::new() - .max_connections(1) - .connect(&admin_url) - .await - .expect("admin connect"); - sqlx::query("CREATE ROLE odal_app LOGIN PASSWORD 'test'") - .execute(&admin) - .await - .expect("create app role"); - - // Applying the whole migration set is itself part of what this file checks: - // 0028 has never run against a real Postgres before these tests. - PgDal::migrate(&admin_url) - .await - .expect("migrations apply, including 0028_seal_outbox"); - - let app_url = format!("postgres://odal_app:test@127.0.0.1:{port}/odal"); - let dal = PgDal::connect(&app_url).await.expect("app connect"); - - TestPg { - dal, - _container: container, - } -} /// A passport in the state the drain actually finds one in: published, signed, /// and retention-locked. The lock is the whole point — an unlocked row would diff --git a/crates/dpp-identity/Cargo.toml b/crates/dpp-identity/Cargo.toml index 8f01ee3..76b80fd 100644 --- a/crates/dpp-identity/Cargo.toml +++ b/crates/dpp-identity/Cargo.toml @@ -39,6 +39,10 @@ tracing = { workspace = true } tracing-subscriber = { workspace = true } [dev-dependencies] +# Test keystores hold Ed25519 private keys. `tempfile` creates the directory +# with restrictive permissions and removes it on drop; `env::temp_dir()` does +# neither, and left a file behind on every run. +tempfile = "3" uuid = { version = "1", features = ["v4"] } tracing-test = "0.2" tower = { version = "0.5", features = ["util"] } diff --git a/crates/dpp-identity/src/handlers/rotate_key.rs b/crates/dpp-identity/src/handlers/rotate_key.rs index d88d8d6..2dffb34 100644 --- a/crates/dpp-identity/src/handlers/rotate_key.rs +++ b/crates/dpp-identity/src/handlers/rotate_key.rs @@ -98,9 +98,19 @@ mod tests { use super::*; use crate::state::AppState; - fn temp_store() -> dpp_crypto::keystore::KeyStore { - let path = std::env::temp_dir().join(format!("rotate-test-{}.json", uuid::Uuid::now_v7())); - dpp_crypto::keystore::KeyStore::open(path, "test").expect("open store") + /// A throwaway keystore in a directory that is removed when the returned + /// `TempDir` drops. + /// + /// The `TempDir` comes back with it deliberately: it owns the directory the + /// store writes into, so a helper returning only the `KeyStore` would have + /// the file vanish underneath it. `tempfile` also creates the directory with + /// restrictive permissions, which `env::temp_dir()` does not — this holds + /// Ed25519 private keys, throwaway or not. + fn temp_store() -> (dpp_crypto::keystore::KeyStore, tempfile::TempDir) { + let dir = tempfile::tempdir().expect("temp dir"); + let store = dpp_crypto::keystore::KeyStore::open(dir.path().join("keystore.json"), "test") + .expect("open store"); + (store, dir) } /// Regression (custody runbook §"key rotation"): a JWS signed before a key @@ -111,7 +121,7 @@ mod tests { /// rotation — this is the fix the runbook flags as needing a green test. #[tokio::test] async fn signature_signed_before_rotation_still_verifies_after() { - let store = temp_store(); + let (store, _dir) = temp_store(); store.generate_key("op1").expect("provision initial key"); let payload = json!({"id": "dpp:test:1", "status": "published"}); diff --git a/crates/dpp-identity/src/handlers/verify.rs b/crates/dpp-identity/src/handlers/verify.rs index 08f6281..b35fbbe 100644 --- a/crates/dpp-identity/src/handlers/verify.rs +++ b/crates/dpp-identity/src/handlers/verify.rs @@ -90,9 +90,19 @@ mod tests { use super::*; - fn temp_store() -> dpp_crypto::keystore::KeyStore { - let path = std::env::temp_dir().join(format!("verify-test-{}.json", uuid::Uuid::now_v7())); - dpp_crypto::keystore::KeyStore::open(path, "test").expect("open store") + /// A throwaway keystore in a directory that is removed when the returned + /// `TempDir` drops. + /// + /// The `TempDir` comes back with it deliberately: it owns the directory the + /// store writes into, so a helper returning only the `KeyStore` would have + /// the file vanish underneath it. `tempfile` also creates the directory with + /// restrictive permissions, which `env::temp_dir()` does not — this holds + /// Ed25519 private keys, throwaway or not. + fn temp_store() -> (dpp_crypto::keystore::KeyStore, tempfile::TempDir) { + let dir = tempfile::tempdir().expect("temp dir"); + let store = dpp_crypto::keystore::KeyStore::open(dir.path().join("keystore.json"), "test") + .expect("open store"); + (store, dir) } fn app(store: dpp_crypto::keystore::KeyStore) -> Router { @@ -122,7 +132,7 @@ mod tests { #[tokio::test] async fn a_signature_this_service_issued_verifies_true() { - let store = temp_store(); + let (store, _dir) = temp_store(); store.generate_key("root").expect("provision key"); let payload = json!({"passportId": "abc", "productName": "Widget"}); let jws = signer::sign(&store, "root", &payload).expect("sign"); @@ -138,7 +148,7 @@ mod tests { #[tokio::test] async fn a_signature_over_different_content_is_rejected() { - let store = temp_store(); + let (store, _dir) = temp_store(); store.generate_key("root").expect("provision key"); let signed_payload = json!({"passportId": "abc"}); let jws = signer::sign(&store, "root", &signed_payload).expect("sign"); @@ -155,7 +165,7 @@ mod tests { #[tokio::test] async fn an_unknown_operator_is_rejected_without_provisioning_a_key() { - let store = temp_store(); + let (store, _dir) = temp_store(); let payload = json!({"passportId": "abc"}); let app = app(store); @@ -169,7 +179,7 @@ mod tests { #[tokio::test] async fn a_malformed_jws_is_rejected_not_a_500() { - let store = temp_store(); + let (store, _dir) = temp_store(); store.generate_key("root").expect("provision key"); let payload = json!({"passportId": "abc"}); @@ -189,7 +199,7 @@ mod tests { /// no-such-operator case above which never gets that far. #[tokio::test] async fn a_structurally_malformed_signature_segment_is_rejected_not_a_500() { - let store = temp_store(); + let (store, _dir) = temp_store(); store.generate_key("root").expect("provision key"); let payload = json!({"passportId": "abc"}); diff --git a/crates/dpp-identity/src/router.rs b/crates/dpp-identity/src/router.rs index 1d0fb93..1f9ee76 100644 --- a/crates/dpp-identity/src/router.rs +++ b/crates/dpp-identity/src/router.rs @@ -78,17 +78,28 @@ mod tests { use serial_test::serial; use tower::ServiceExt; - fn temp_store() -> dpp_crypto::keystore::KeyStore { - let path = std::env::temp_dir().join(format!("router-test-{}.json", uuid::Uuid::now_v7())); - dpp_crypto::keystore::KeyStore::open(path, "test").expect("open store") + /// A throwaway keystore in a directory that is removed when the returned + /// `TempDir` drops. + /// + /// The `TempDir` comes back with it deliberately: it owns the directory the + /// store writes into, so a helper returning only the `KeyStore` would have + /// the file vanish underneath it. `tempfile` also creates the directory with + /// restrictive permissions, which `env::temp_dir()` does not — this holds + /// Ed25519 private keys, throwaway or not. + fn temp_store() -> (dpp_crypto::keystore::KeyStore, tempfile::TempDir) { + let dir = tempfile::tempdir().expect("temp dir"); + let store = dpp_crypto::keystore::KeyStore::open(dir.path().join("keystore.json"), "test") + .expect("open store"); + (store, dir) } /// No X-Client-Cert-Subject header → 401 Unauthorized (enforcement on by default). #[tokio::test] #[serial] async fn mtls_rejects_internal_request_without_cert() { + let (store, _dir) = temp_store(); let state = crate::state::AppState { - store: Arc::new(temp_store()), + store: Arc::new(store), did_web_base_url: "http://localhost".into(), }; let app = super::build(state); @@ -111,7 +122,7 @@ mod tests { /// done in-process; these endpoints must simply not exist here. #[tokio::test] async fn public_router_has_no_internal_endpoints() { - let store = temp_store(); + let (store, _dir) = temp_store(); store.generate_key("root").expect("provision root key"); let state = crate::state::AppState { store: Arc::new(store), @@ -160,8 +171,9 @@ mod tests { #[serial] async fn mtls_rejects_wrong_cn() { unsafe { std::env::set_var("MTLS_PROXY_SHARED_SECRET", "s3cr3t") }; + let (store, _dir) = temp_store(); let state = crate::state::AppState { - store: Arc::new(temp_store()), + store: Arc::new(store), did_web_base_url: "http://localhost".into(), }; let app = super::build(state); diff --git a/crates/dpp-integrator/src/handlers/mod.rs b/crates/dpp-integrator/src/handlers/mod.rs index 264d422..1f90ad2 100644 --- a/crates/dpp-integrator/src/handlers/mod.rs +++ b/crates/dpp-integrator/src/handlers/mod.rs @@ -3,4 +3,5 @@ pub mod health; pub mod import; pub mod job_status; +pub mod schemas; pub mod templates; diff --git a/crates/dpp-integrator/src/handlers/schemas.rs b/crates/dpp-integrator/src/handlers/schemas.rs new file mode 100644 index 0000000..525a71f --- /dev/null +++ b/crates/dpp-integrator/src/handlers/schemas.rs @@ -0,0 +1,286 @@ +//! `GET /api/v1/schemas[/{sector}[/{version}]]` — serve the sector JSON Schemas +//! an SDK needs to build a passport body before it posts one. +//! +//! # Why this exists +//! +//! The only feedback available before this was a rejection from the create +//! route, or the CSV import template — which is an import artefact, not a +//! schema. The schemas already exist and the publish path already validates +//! against them; nothing served them. +//! +//! These are resolved through the same `VersionedSchemaRegistry` the publish +//! gate uses, never from a copy. A second copy would drift, and the direction it +//! drifts is the one where a body passes validation here and fails at publish. +//! +//! # Every `description` is stripped, deliberately and temporarily +//! +//! The schemas carry `description` fields that make regulatory assertions — act +//! numbers, adoption dates, effective dates, product-class scope, annex +//! references — and **none has been verified against primary text**. Two +//! electronics descriptions once asserted an adoption date, an effective date, +//! three named priority product classes and a phase-two date for an act that +//! does not exist. +//! +//! Inside a library those are developer-facing comments. On a public endpoint +//! they become a product surface that consumers read, cache and rely on, from a +//! compliance vendor — a much larger blast radius for a fabricated claim. So the +//! machine-readable contract is served and the prose is not. +//! +//! **This is a holding position, not the intended end state.** Restore the +//! descriptions once the prose audit has verified them against primary OJ text; +//! `strip_descriptions` and its call site are the only things to remove. Until +//! then an SDK gets what it actually needs to pre-validate — types, enums, +//! `required`, patterns, bounds — and no unaudited regulatory claim leaves the +//! node. +//! +//! `title` is kept: they are short labels ("Odal Node — Battery Sector Data +//! (v2.6.0)"), not assertions. + +use axum::{ + Json, + extract::Path, + http::StatusCode, + response::{IntoResponse, Response}, +}; +use dpp_common::http_problem; +use dpp_domain::catalog::SectorCatalog; +use dpp_domain::schemas::VersionedSchemaRegistry; +use serde_json::{Value, json}; + +/// `GET /api/v1/schemas` +/// +/// Every sector with a schema, and the versions it serves. `current` is the +/// version a new passport is written against; `versions` is everything a stored +/// passport may legitimately record. +pub async fn list_schemas() -> Response { + let registry = VersionedSchemaRegistry::new(); + let catalog = SectorCatalog::new(); + + let mut sectors: Vec<&str> = registry.sectors(); + sectors.sort_unstable(); + + let entries: Vec = sectors + .into_iter() + .map(|sector| { + let mut versions: Vec = registry + .versions_for(sector) + .into_iter() + .map(ToString::to_string) + .collect(); + versions.sort(); + json!({ + "sector": sector, + "current": catalog.current_schema_version(sector), + "versions": versions, + }) + }) + .collect(); + + (StatusCode::OK, Json(json!({ "schemas": entries }))).into_response() +} + +/// `GET /api/v1/schemas/{sector}` +/// +/// The sector's current schema — the one a passport created today is validated +/// against. +pub async fn get_current_schema(Path(sector): Path) -> Response { + let catalog = SectorCatalog::new(); + let Some(version) = catalog.current_schema_version(§or) else { + return unknown_sector(§or); + }; + serve(§or, version) +} + +/// `GET /api/v1/schemas/{sector}/{version}` +/// +/// A pinned version. A stored passport records the `schemaVersion` it was +/// written under, so an SDK holding one needs to keep fetching that exact +/// schema rather than whatever is current. +pub async fn get_pinned_schema(Path((sector, version)): Path<(String, String)>) -> Response { + serve(§or, version.trim_start_matches('v')) +} + +/// Resolve one `(sector, version)` from the registry and serve it, prose removed. +fn serve(sector: &str, version: &str) -> Response { + let registry = VersionedSchemaRegistry::new(); + let Ok(parsed) = version.parse() else { + return http_problem::bad_request(format!( + "'{version}' is not a semver version. Use the form '1.2.0'." + )) + .into_response(); + }; + let Some(raw) = registry.get(sector, &parsed) else { + return unknown_version(sector, version); + }; + + // An embedded schema parsed at boot in the registry, so this cannot fail in + // practice; a 500 is still the honest answer if it ever does. + let Ok(mut schema) = serde_json::from_str::(raw) else { + return http_problem::internal_error(format!( + "the schema for {sector} v{version} could not be read" + )) + .into_response(); + }; + strip_descriptions(&mut schema); + + (StatusCode::OK, Json(schema)).into_response() +} + +fn unknown_sector(sector: &str) -> Response { + let catalog = SectorCatalog::new(); + let mut known: Vec<&str> = catalog.keys(); + known.sort_unstable(); + http_problem::not_found(format!( + "No schema for sector '{sector}'. Known sectors: {}.", + known.join(", ") + )) + .into_response() +} + +fn unknown_version(sector: &str, version: &str) -> Response { + let registry = VersionedSchemaRegistry::new(); + let mut versions: Vec = registry + .versions_for(sector) + .into_iter() + .map(ToString::to_string) + .collect(); + versions.sort(); + if versions.is_empty() { + return unknown_sector(sector); + } + http_problem::not_found(format!( + "No schema for sector '{sector}' at version '{version}'. Available: {}.", + versions.join(", ") + )) + .into_response() +} + +/// Remove every `description` **keyword** from a JSON Schema, in place. +/// +/// Schema-aware rather than a blanket key removal: under `properties`, +/// `$defs`/`definitions` and `patternProperties` the keys are author-chosen +/// *names*, so a property legitimately called `description` would be deleted by +/// a naive walk — taking a real field out of the contract an SDK validates +/// against. No schema declares one today; this costs nothing and stops that +/// being a latent trap for whoever adds the first. +fn strip_descriptions(node: &mut Value) { + /// Keywords whose object values are keyed by author-chosen names, not by + /// schema keywords — descend into the values, never treat the keys as + /// keywords. + const NAME_KEYED: [&str; 4] = ["properties", "$defs", "definitions", "patternProperties"]; + + match node { + Value::Object(map) => { + map.remove("description"); + for (key, value) in map.iter_mut() { + if NAME_KEYED.contains(&key.as_str()) { + if let Value::Object(named) = value { + for schema in named.values_mut() { + strip_descriptions(schema); + } + } + } else { + strip_descriptions(value); + } + } + } + Value::Array(items) => { + for item in items { + strip_descriptions(item); + } + } + _ => {} + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn descriptions_are_removed_at_every_depth() { + let mut schema = json!({ + "description": "root prose", + "properties": { + "gtin": { "type": "string", "description": "nested prose" }, + "parts": { + "type": "array", + "items": { "type": "object", "description": "deep prose" } + } + }, + "$defs": { + "Thing": { "description": "def prose", "type": "object" } + }, + "allOf": [{ "description": "branch prose" }] + }); + strip_descriptions(&mut schema); + + let rendered = serde_json::to_string(&schema).unwrap(); + assert!( + !rendered.contains("prose"), + "no description may survive: {rendered}" + ); + } + + #[test] + fn a_property_named_description_survives() { + // The trap a blanket key removal would fall into: deleting a real field + // from the contract an SDK validates against. + let mut schema = json!({ + "description": "root prose", + "properties": { + "description": { "type": "string", "maxLength": 200 } + } + }); + strip_descriptions(&mut schema); + + assert_eq!( + schema["properties"]["description"]["type"], "string", + "a property *named* description is a field, not prose" + ); + assert!(schema.get("description").is_none(), "root prose must go"); + } + + #[test] + fn the_validation_contract_survives_stripping() { + // The point of serving these at all: an SDK must still be able to + // pre-validate. Everything that decides accept/reject has to remain. + let registry = VersionedSchemaRegistry::new(); + let raw = registry + .get("battery", &"2.6.0".parse().unwrap()) + .expect("battery 2.6.0 is embedded"); + let mut schema: Value = serde_json::from_str(raw).unwrap(); + strip_descriptions(&mut schema); + + assert!(schema.get("required").is_some(), "required must survive"); + assert!( + schema.get("properties").is_some(), + "properties must survive" + ); + assert_eq!(schema["additionalProperties"], json!(false)); + assert!( + schema["properties"]["gtin"]["pattern"].is_string(), + "a pattern is part of the contract" + ); + assert!( + serde_json::to_string(&schema).unwrap().contains("\"enum\""), + "enums are part of the contract" + ); + } + + #[test] + fn no_embedded_schema_keeps_a_description_after_stripping() { + let registry = VersionedSchemaRegistry::new(); + for (sector, version) in registry.list() { + let raw = registry.get(sector, version).expect("just listed"); + let mut schema: Value = serde_json::from_str(raw).unwrap(); + strip_descriptions(&mut schema); + assert!( + !serde_json::to_string(&schema) + .unwrap() + .contains("\"description\""), + "{sector} v{version} still carries a description keyword after stripping" + ); + } + } +} diff --git a/crates/dpp-integrator/src/router.rs b/crates/dpp-integrator/src/router.rs index 9dd5469..41e3fbf 100644 --- a/crates/dpp-integrator/src/router.rs +++ b/crates/dpp-integrator/src/router.rs @@ -17,7 +17,7 @@ use dpp_common::{ }; use crate::{ - handlers::{health, import, job_status, templates}, + handlers::{health, import, job_status, schemas, templates}, state::AppState, }; @@ -34,6 +34,12 @@ pub fn build(state: AppState) -> Router { Router::new() .route("/health", get(health::health_handler)) .route("/api/v1/templates/{sector}", get(templates::get_template)) + .route("/api/v1/schemas", get(schemas::list_schemas)) + .route("/api/v1/schemas/{sector}", get(schemas::get_current_schema)) + .route( + "/api/v1/schemas/{sector}/{version}", + get(schemas::get_pinned_schema), + ) .route( "/api/v1/import/{sector}", post(import::import_file).layer(DefaultBodyLimit::max(IMPORT_BODY_LIMIT)), @@ -89,6 +95,56 @@ mod tests { } } + /// The three schema routes serve, and what they serve carries no prose. + /// + /// End-to-end through the router rather than against `strip_descriptions` + /// directly: the unit test proves the function strips, this proves the route + /// actually calls it before the bytes leave. + #[tokio::test] + async fn schema_routes_serve_without_descriptions() { + for uri in [ + "/api/v1/schemas", + "/api/v1/schemas/battery", + "/api/v1/schemas/battery/2.6.0", + // A `v` prefix is accepted, since that is how the versions are + // spelled on disk and in the fixture directories. + "/api/v1/schemas/battery/v2.6.0", + ] { + let app = super::build(test_state()); + let req = Request::builder().uri(uri).body(Body::empty()).unwrap(); + let resp = app.oneshot(req).await.unwrap(); + assert_eq!(resp.status(), StatusCode::OK, "{uri} must serve"); + + let body = axum::body::to_bytes(resp.into_body(), 4 * 1024 * 1024) + .await + .unwrap(); + let text = String::from_utf8(body.to_vec()).unwrap(); + assert!( + !text.contains("\"description\""), + "{uri} leaked an unaudited regulatory description" + ); + } + } + + /// An unknown sector and an unknown version are told apart, and both name + /// what is available rather than only refusing. + #[tokio::test] + async fn schema_routes_refuse_helpfully() { + for (uri, status) in [ + ("/api/v1/schemas/nosuchsector", StatusCode::NOT_FOUND), + ("/api/v1/schemas/battery/9.9.9", StatusCode::NOT_FOUND), + ( + "/api/v1/schemas/battery/not-semver", + StatusCode::BAD_REQUEST, + ), + ] { + let app = super::build(test_state()); + let req = Request::builder().uri(uri).body(Body::empty()).unwrap(); + let resp = app.oneshot(req).await.unwrap(); + assert_eq!(resp.status(), status, "{uri}"); + } + } + /// Regression (red-team RT2-1): an import POST with no Bearer token must be /// rejected with 401 *before* the file is parsed, so anonymous callers can't /// drive the allocation-heavy parser. diff --git a/crates/dpp-node/Cargo.toml b/crates/dpp-node/Cargo.toml index 37e11de..eeafd78 100644 --- a/crates/dpp-node/Cargo.toml +++ b/crates/dpp-node/Cargo.toml @@ -79,6 +79,9 @@ s3 = ["dep:aws-sdk-s3"] integration-tests = ["s3"] [dev-dependencies] +# The shared Postgres harness the integration suites use. `test-harness` is a +# dev-only feature of dpp-dal; the normal dependency above stays featureless. +dpp-dal = { path = "../dpp-dal", features = ["test-harness"] } tower = { version = "0.5", features = ["util"] } serial_test = "4" wat = "1" diff --git a/crates/dpp-node/src/infra/ruleset.rs b/crates/dpp-node/src/infra/ruleset.rs index 3024292..292a794 100644 --- a/crates/dpp-node/src/infra/ruleset.rs +++ b/crates/dpp-node/src/infra/ruleset.rs @@ -139,15 +139,21 @@ mod tests { use super::*; use base64::Engine; - /// A throwaway publisher key store; returns the store, key id, and the - /// base64url public key a node would pin. - fn publisher() -> (KeyStore, String, String) { - let path = std::env::temp_dir().join(format!("ruleset-pub-{}.enc", uuid::Uuid::now_v7())); - let store = KeyStore::open_and_migrate(&path, "test-passphrase").expect("open keystore"); + /// A throwaway publisher key store; returns the store, key id, the + /// base64url public key a node would pin, and the directory holding it. + /// + /// The `TempDir` is returned rather than dropped because it owns the + /// directory the keystore writes into. `tempfile` creates that directory + /// with restrictive permissions and removes it on drop; `env::temp_dir()` + /// did neither, and left an Ed25519 private key behind on every run. + fn publisher() -> (KeyStore, String, String, tempfile::TempDir) { + let dir = tempfile::tempdir().expect("temp dir"); + let store = KeyStore::open_and_migrate(dir.path().join("publisher.enc"), "test-passphrase") + .expect("open keystore"); let entry = store.generate_key("publisher").expect("generate key"); let pubkey_b64 = base64::engine::general_purpose::URL_SAFE_NO_PAD.encode(entry.verifying_key.as_bytes()); - (store, "publisher".to_owned(), pubkey_b64) + (store, "publisher".to_owned(), pubkey_b64, dir) } fn bundle(store: &KeyStore, key_id: &str, version: &str, threshold: i64) -> SignedBundle { @@ -165,7 +171,7 @@ mod tests { #[test] fn signed_bundle_verifies_and_carries_version() { - let (store, kid, pubkey) = publisher(); + let (store, kid, pubkey, _dir) = publisher(); let b = bundle(&store, &kid, "2026-Q3.1", 5); let v = verify_bundle(&b, &pubkey, &DppCryptoVerifier).expect("must verify"); assert_eq!(v.version(), "2026-Q3.1"); @@ -174,7 +180,7 @@ mod tests { #[test] fn tampered_signature_is_refused() { - let (store, kid, pubkey) = publisher(); + let (store, kid, pubkey, _dir) = publisher(); let mut b = bundle(&store, &kid, "2026-Q3.1", 5); // Flip the second-to-last char of the JWS signature segment. The very // last base64url char of a 64-byte Ed25519 signature carries only 2 @@ -195,7 +201,7 @@ mod tests { #[test] fn tampered_content_is_refused() { - let (store, kid, pubkey) = publisher(); + let (store, kid, pubkey, _dir) = publisher(); let mut b = bundle(&store, &kid, "2026-Q3.1", 5); // Change the content without re-signing the manifest. b.content = serde_json::json!({ "textileFibreThreshold": 999 }); @@ -207,8 +213,8 @@ mod tests { #[test] fn wrong_publisher_key_is_refused() { - let (store, kid, _pubkey) = publisher(); - let (_other_store, _oid, other_pubkey) = publisher(); + let (store, kid, _pubkey, _dir) = publisher(); + let (_other_store, _oid, other_pubkey, _dir) = publisher(); let b = bundle(&store, &kid, "2026-Q3.1", 5); assert!(matches!( verify_bundle(&b, &other_pubkey, &DppCryptoVerifier), @@ -218,7 +224,7 @@ mod tests { #[test] fn active_ruleset_hot_swaps_a_verified_bundle() { - let (store, kid, pubkey) = publisher(); + let (store, kid, pubkey, _dir) = publisher(); let active = ActiveRuleset::baseline(); assert_eq!(active.version(), "baseline"); @@ -229,7 +235,7 @@ mod tests { assert_eq!(active.get().content["textileFibreThreshold"], 7); // A bad bundle leaves the active ruleset unchanged (fail-closed). - let (bad_store, bad_kid, _) = publisher(); + let (bad_store, bad_kid, _, _dir) = publisher(); let forged = bundle(&bad_store, &bad_kid, "evil", 0); assert!(active.load_and_swap(&forged, &pubkey).is_err()); assert_eq!(active.version(), "2026-Q3.2"); diff --git a/crates/dpp-node/src/plugins.rs b/crates/dpp-node/src/plugins.rs index a01d6d7..1b64cb0 100644 --- a/crates/dpp-node/src/plugins.rs +++ b/crates/dpp-node/src/plugins.rs @@ -58,10 +58,13 @@ pub fn boot(plugins_dir: &str) -> Result> { .unwrap_or(false); ensure_signing_policy(trusted_key.is_some(), discovered.len(), allow_unsigned)?; + let mut loaded: Vec = Vec::with_capacity(discovered.len()); + for (sector_key, path) in discovered { match LoadedPlugin::from_file(&engine, &path, §or_key, trusted_key.as_ref()) { Ok(plugin) => { tracing::info!(sector = %sector_key, path = %path.display(), "plugin loaded"); + loaded.push(sector_key.clone()); host.register(sector_key, plugin); } // Fail the boot rather than skip. This used to be a `warn!`, which @@ -90,9 +93,48 @@ pub fn boot(plugins_dir: &str) -> Result> { } } + report_sectors_without_plugins(&SectorCatalog::new(), &loaded); + Ok(host) } +/// Say which catalogued sectors are running with no plugin. +/// +/// Passthrough is a legitimate configuration, so this does not refuse to boot. +/// But a sector with no plugin and a sector whose plugin found nothing wrong +/// produce the same thing — a determination with no findings — so from the +/// outside they are indistinguishable. Left unsaid, "no violations" reads as +/// "checked and clean" when it may mean "never checked". +/// +/// `warn` rather than `info`: a production node loads a full signed set from the +/// release pipeline, so a gap there is a misconfiguration worth noticing, and +/// `info` is where it would be missed. A node deliberately running a subset in +/// development sees one line at boot and can ignore it. +fn report_sectors_without_plugins(catalog: &SectorCatalog, loaded: &[String]) { + let mut missing: Vec<&str> = catalog + .keys() + .into_iter() + .filter(|key| !loaded.iter().any(|l| l == key)) + .collect(); + missing.sort_unstable(); + + if missing.is_empty() { + tracing::info!( + sectors = catalog.len(), + "every catalogued sector has a plugin loaded" + ); + return; + } + + tracing::warn!( + sectors = %missing.join(", "), + count = missing.len(), + "no plugin loaded for these catalogued sectors; passports in them take the \ + passthrough path — declared values are carried verbatim, with no sector \ + validation and no findings" + ); +} + /// The registry that serves a sector with no Wasm plugin loaded for it. /// /// `PassthroughRegistry::new` ships the Apache-2.0 strategies; `register` @@ -179,6 +221,33 @@ fn ensure_signing_policy(has_key: bool, plugin_count: usize, allow_unsigned: boo mod tests { use super::*; + #[test] + fn a_sector_with_no_plugin_is_named() { + let catalog = SectorCatalog::new(); + // Every catalogued sector but the first is missing a plugin. + let keys: Vec = catalog.keys().into_iter().map(str::to_owned).collect(); + let loaded = vec![keys[0].clone()]; + + let missing: Vec<&str> = catalog + .keys() + .into_iter() + .filter(|k| !loaded.iter().any(|l| l == k)) + .collect(); + + assert_eq!( + missing.len(), + keys.len() - 1, + "every sector except the one loaded must be reported" + ); + assert!( + !missing.contains(&keys[0].as_str()), + "the loaded sector must not be reported as missing" + ); + // The reporting path itself must not panic on either branch. + report_sectors_without_plugins(&catalog, &loaded); + report_sectors_without_plugins(&catalog, &keys); + } + #[test] fn boot_with_empty_dir() { let tmp = std::env::temp_dir().join(format!("odal-test-{}", uuid::Uuid::now_v7())); diff --git a/crates/dpp-node/tests/import_job_store.rs b/crates/dpp-node/tests/import_job_store.rs index 29918c7..27dbcb6 100644 --- a/crates/dpp-node/tests/import_job_store.rs +++ b/crates/dpp-node/tests/import_job_store.rs @@ -9,14 +9,9 @@ #![cfg(feature = "integration-tests")] -use testcontainers::{ - GenericImage, ImageExt, - core::{WaitFor, ports::ContainerPort}, - runners::AsyncRunner, -}; use uuid::Uuid; -use dpp_dal::pg::{PgDal, sqlx}; +use dpp_dal::test_harness::start_pg; use dpp_integrator::{ domain::batch_runner::{BatchResult, CreatedItem, RowError}, domain::import_report::{FindingKind, ImportMode, ImportReport, ReportRow, RowFinding}, @@ -24,48 +19,10 @@ use dpp_integrator::{ }; use dpp_node::infra::pg_job_store::PgJobStore; -async fn start_pg() -> (PgDal, testcontainers::ContainerAsync) { - let image = GenericImage::new("postgres", "17") - .with_exposed_port(ContainerPort::Tcp(5432)) - .with_wait_for(WaitFor::message_on_stderr( - "database system is ready to accept connections", - )) - // POSTGRES_USER/PASSWORD/DB are the official Postgres image's required - // env vars for this throwaway testcontainer — NOT the app's - // DATABASE_POSTGRES_PASS / DATABASE_APP_PASS scheme. - .with_env_var("POSTGRES_USER", "postgres") - .with_env_var("POSTGRES_PASSWORD", "test") - .with_env_var("POSTGRES_DB", "odal"); - - let container = image.start().await.expect("start postgres container"); - let port = container - .get_host_port_ipv4(5432) - .await - .expect("mapped port"); - let admin_url = format!("postgres://postgres:test@127.0.0.1:{port}/odal"); - - tokio::time::sleep(std::time::Duration::from_millis(1500)).await; - - let admin = sqlx::postgres::PgPoolOptions::new() - .max_connections(1) - .connect(&admin_url) - .await - .expect("admin connect"); - sqlx::query("CREATE ROLE odal_app LOGIN PASSWORD 'test'") - .execute(&admin) - .await - .expect("create app role"); - - PgDal::migrate(&admin_url).await.expect("apply migrations"); - - let app_url = format!("postgres://odal_app:test@127.0.0.1:{port}/odal"); - let dal = PgDal::connect(&app_url).await.expect("app connect"); - (dal, container) -} - #[tokio::test(flavor = "multi_thread")] async fn job_lifecycle_persists_and_is_retrievable() { - let (dal, _container) = start_pg().await; + let _container = start_pg().await; + let dal = _container.dal.clone(); let store = PgJobStore::new(dal); let id = Uuid::now_v7(); @@ -124,7 +81,8 @@ async fn job_lifecycle_persists_and_is_retrievable() { // round-trip through real Postgres independent of `result`. #[tokio::test(flavor = "multi_thread")] async fn record_report_persists_and_survives_sql_round_trip() { - let (dal, _container) = start_pg().await; + let _container = start_pg().await; + let dal = _container.dal.clone(); let store = PgJobStore::new(dal); let id = Uuid::now_v7(); diff --git a/crates/dpp-node/tests/registry_outbox.rs b/crates/dpp-node/tests/registry_outbox.rs index 99553d6..ff35a63 100644 --- a/crates/dpp-node/tests/registry_outbox.rs +++ b/crates/dpp-node/tests/registry_outbox.rs @@ -30,13 +30,9 @@ use std::sync::atomic::{AtomicUsize, Ordering}; use async_trait::async_trait; use chrono::Utc; -use testcontainers::{ - GenericImage, ImageExt, - core::{WaitFor, ports::ContainerPort}, - runners::AsyncRunner, -}; use dpp_dal::pg::{PgDal, PgPassportRepo, PgRegistrySyncRepo, sqlx}; +use dpp_dal::test_harness::{start_pg, start_pg_before}; use dpp_domain::{ DppError, domain::{ @@ -67,41 +63,6 @@ use dpp_types::{RegistryStatusIntent, RegistrySyncOutbox, RegistrySyncStatus}; // ─── Harness ──────────────────────────────────────────────────────────────── -async fn start_pg() -> (PgDal, testcontainers::ContainerAsync) { - let image = GenericImage::new("postgres", "17") - .with_exposed_port(ContainerPort::Tcp(5432)) - .with_wait_for(WaitFor::message_on_stderr( - "database system is ready to accept connections", - )) - .with_env_var("POSTGRES_USER", "postgres") - .with_env_var("POSTGRES_PASSWORD", "test") - .with_env_var("POSTGRES_DB", "odal"); - - let container = image.start().await.expect("start postgres container"); - let port = container - .get_host_port_ipv4(5432) - .await - .expect("mapped port"); - let admin_url = format!("postgres://postgres:test@127.0.0.1:{port}/odal"); - - tokio::time::sleep(std::time::Duration::from_millis(1500)).await; - - let admin = sqlx::postgres::PgPoolOptions::new() - .max_connections(1) - .connect(&admin_url) - .await - .expect("admin connect"); - sqlx::query("CREATE ROLE odal_app LOGIN PASSWORD 'test'") - .execute(&admin) - .await - .expect("create app role"); - PgDal::migrate(&admin_url).await.expect("apply migrations"); - - let app_url = format!("postgres://odal_app:test@127.0.0.1:{port}/odal"); - let dal = PgDal::connect(&app_url).await.expect("app connect"); - (dal, container) -} - fn draft_passport() -> Passport { Passport { id: PassportId::new(), @@ -275,7 +236,8 @@ fn mock_with_poll( #[tokio::test(flavor = "multi_thread")] async fn publish_is_atomic_idempotent_and_drains_exactly_once() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let repo = PgPassportRepo::new(dal.clone()); @@ -328,7 +290,8 @@ async fn publish_is_atomic_idempotent_and_drains_exactly_once() { #[tokio::test(flavor = "multi_thread")] async fn drain_backs_off_on_transient_and_marks_terminal_rejection() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); // (c) transient failure → attempts++, still pending, pushed into the future. @@ -367,59 +330,6 @@ async fn drain_backs_off_on_transient_and_marks_terminal_rejection() { /// the clobbered rows), and the app URL is what `PgDal::connect` takes — it /// refuses a superuser role, since a superuser owns the audit table and the /// append-only trigger cannot bind it. -async fn start_pg_before_0024() -> (String, String, testcontainers::ContainerAsync) { - let image = GenericImage::new("postgres", "17") - .with_exposed_port(ContainerPort::Tcp(5432)) - .with_wait_for(WaitFor::message_on_stderr( - "database system is ready to accept connections", - )) - .with_env_var("POSTGRES_USER", "postgres") - .with_env_var("POSTGRES_PASSWORD", "test") - .with_env_var("POSTGRES_DB", "odal"); - - let container = image.start().await.expect("start postgres container"); - let port = container - .get_host_port_ipv4(5432) - .await - .expect("mapped port"); - let admin_url = format!("postgres://postgres:test@127.0.0.1:{port}/odal"); - - tokio::time::sleep(std::time::Duration::from_millis(1500)).await; - - let admin = sqlx::postgres::PgPoolOptions::new() - .max_connections(1) - .connect(&admin_url) - .await - .expect("admin connect"); - sqlx::query("CREATE ROLE odal_app LOGIN PASSWORD 'test'") - .execute(&admin) - .await - .expect("create app role"); - let app_url = format!("postgres://odal_app:test@127.0.0.1:{port}/odal"); - - let dir = concat!(env!("CARGO_MANIFEST_DIR"), "/../../ops/pg"); - let mut files: Vec<_> = std::fs::read_dir(dir) - .expect("read ops/pg") - .filter_map(|e| e.ok().map(|e| e.path())) - .filter(|p| p.extension().is_some_and(|x| x == "sql")) - .collect(); - files.sort(); - for path in files { - let name = path.file_name().unwrap().to_string_lossy().to_string(); - if name.starts_with("0024_") { - break; // stop at the migration under test - } - let sql = std::fs::read_to_string(&path).expect("read migration"); - // Repo-controlled migration text from ops/pg, not caller input. - sqlx::raw_sql(sqlx::AssertSqlSafe(sql)) - .execute(&admin) - .await - .unwrap_or_else(|e| panic!("apply {name}: {e}")); - } - admin.close().await; - (admin_url, app_url, container) -} - /// Insert a passport plus a `registry_sync` row in the shape the old /// `enqueue_status` left behind — an intent sitting in the `status` column. async fn insert_clobbered_row( @@ -456,7 +366,8 @@ async fn insert_clobbered_row( /// registrations in an existing deployment. #[tokio::test(flavor = "multi_thread")] async fn migration_0024_restores_registrations_lost_before_the_fix() { - let (admin_url, app_url, _c) = start_pg_before_0024().await; + let _c = start_pg_before("0024_").await; + let (admin_url, app_url) = (_c.admin_url.clone(), _c.app_url.clone()); let admin = sqlx::postgres::PgPoolOptions::new() .max_connections(1) .connect(&admin_url) @@ -541,7 +452,8 @@ async fn migration_0024_restores_registrations_lost_before_the_fix() { /// a row can stay pending indefinitely. #[tokio::test(flavor = "multi_thread")] async fn suspend_before_drain_must_not_drop_the_pending_registration() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let id = create_and_publish(&dal, &outbox).await; @@ -580,7 +492,8 @@ async fn suspend_before_drain_must_not_drop_the_pending_registration() { #[tokio::test(flavor = "multi_thread")] async fn suspend_enqueues_status_intent_and_counts_reflect_state() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let id = create_and_publish(&dal, &outbox).await; @@ -612,7 +525,8 @@ async fn suspend_enqueues_status_intent_and_counts_reflect_state() { /// publish transaction and by nothing else. #[tokio::test(flavor = "multi_thread")] async fn archiving_an_unpublished_draft_creates_no_outbox_row() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let repo = PgPassportRepo::new(dal.clone()); @@ -651,7 +565,8 @@ async fn archiving_an_unpublished_draft_creates_no_outbox_row() { /// passport that is live again. #[tokio::test(flavor = "multi_thread")] async fn republish_clears_a_stale_suspend_intent() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let id = create_and_publish(&dal, &outbox).await; @@ -694,7 +609,8 @@ async fn republish_clears_a_stale_suspend_intent() { /// submission as complete and never learned if it was later refused. #[tokio::test(flavor = "multi_thread")] async fn an_accepted_submission_is_not_yet_a_registration() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let id = create_and_publish(&dal, &outbox).await; @@ -722,7 +638,8 @@ async fn an_accepted_submission_is_not_yet_a_registration() { /// register the same product twice. #[tokio::test(flavor = "multi_thread")] async fn a_submitted_row_is_polled_not_resubmitted() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let id = create_and_publish(&dal, &outbox).await; @@ -758,7 +675,8 @@ async fn a_submitted_row_is_polled_not_resubmitted() { /// the submission was defective, so there is nothing to correct and resubmit. #[tokio::test(flavor = "multi_thread")] async fn a_deactivated_record_is_terminal_but_not_rejected() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let id = create_and_publish(&dal, &outbox).await; @@ -788,7 +706,8 @@ async fn a_deactivated_record_is_terminal_but_not_rejected() { /// surfaces those without fabricating a row that could never drain. #[tokio::test(flavor = "multi_thread")] async fn a_published_passport_with_no_outbox_row_is_counted() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let repo = PgPassportRepo::new(dal.clone()); @@ -827,7 +746,8 @@ async fn a_published_passport_with_no_outbox_row_is_counted() { /// A draft is not owed a registration, so it is not an orphan. #[tokio::test(flavor = "multi_thread")] async fn an_unpublished_draft_is_not_counted_as_unregistered() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let repo = PgPassportRepo::new(dal.clone()); @@ -867,7 +787,8 @@ async fn operator_verified_at( /// touching the rows, so re-verification resumes everything untouched. #[tokio::test(flavor = "multi_thread")] async fn an_expired_operator_holds_the_drain_without_losing_rows() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); let id = create_and_publish(&dal, &outbox).await; @@ -900,7 +821,8 @@ async fn an_expired_operator_holds_the_drain_without_losing_rows() { /// Never verified is the same refusal: there is no verified status to rely on. #[tokio::test(flavor = "multi_thread")] async fn a_never_verified_operator_also_holds_the_drain() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let outbox: Arc = Arc::new(PgRegistrySyncRepo::new(dal.clone())); create_and_publish(&dal, &outbox).await; diff --git a/crates/dpp-node/tests/seal_outbox.rs b/crates/dpp-node/tests/seal_outbox.rs index 829fdfd..896025b 100644 --- a/crates/dpp-node/tests/seal_outbox.rs +++ b/crates/dpp-node/tests/seal_outbox.rs @@ -40,13 +40,9 @@ use base64::engine::general_purpose::STANDARD as BASE64; use chrono::Utc; use hmac::{Hmac, KeyInit, Mac}; use sha2::{Digest, Sha256}; -use testcontainers::{ - GenericImage, ImageExt, - core::{WaitFor, ports::ContainerPort}, - runners::AsyncRunner, -}; -use dpp_dal::pg::{PgAuditRepo, PgDal, PgPassportRepo, PgSealOutboxRepo, sqlx}; +use dpp_dal::pg::{PgAuditRepo, PgPassportRepo, PgSealOutboxRepo}; +use dpp_dal::test_harness::start_pg; use dpp_domain::domain::passport::{ManufacturerInfo, Passport, PassportId}; use dpp_domain::domain::sector::Sector; use dpp_domain::domain::status::PassportStatus; @@ -146,40 +142,6 @@ async fn spawn_mock(state: Arc) -> String { // ─── Postgres harness ───────────────────────────────────────────────────────── -async fn start_pg() -> (PgDal, testcontainers::ContainerAsync) { - let image = GenericImage::new("postgres", "17") - .with_exposed_port(ContainerPort::Tcp(5432)) - .with_wait_for(WaitFor::message_on_stderr( - "database system is ready to accept connections", - )) - .with_env_var("POSTGRES_USER", "postgres") - .with_env_var("POSTGRES_PASSWORD", "test") - .with_env_var("POSTGRES_DB", "odal"); - - let container = image.start().await.expect("start postgres container"); - let port = container - .get_host_port_ipv4(5432) - .await - .expect("mapped port"); - let admin_url = format!("postgres://postgres:test@127.0.0.1:{port}/odal"); - - tokio::time::sleep(std::time::Duration::from_millis(1500)).await; - let admin = sqlx::postgres::PgPoolOptions::new() - .max_connections(1) - .connect(&admin_url) - .await - .expect("admin connect"); - sqlx::query("CREATE ROLE odal_app LOGIN PASSWORD 'test'") - .execute(&admin) - .await - .expect("create app role"); - PgDal::migrate(&admin_url).await.expect("apply migrations"); - - let app_url = format!("postgres://odal_app:test@127.0.0.1:{port}/odal"); - let dal = PgDal::connect(&app_url).await.expect("app connect"); - (dal, container) -} - // ─── Service harness ────────────────────────────────────────────────────────── fn auth() -> AuthContext { @@ -306,7 +268,8 @@ fn eideasy_adapter(cfg: dpp_seal::eideasy::EideasyConfig) -> Arc { /// print every record produced along the way. #[tokio::test] async fn publish_then_drain_seals_the_passport_end_to_end() { - let (dal, _pg) = start_pg().await; + let _pg = start_pg().await; + let dal = _pg.dal.clone(); let mock = Arc::new(MockState::default()); let base_url = spawn_mock(mock.clone()).await; @@ -460,7 +423,8 @@ async fn publish_then_drain_seals_the_passport_end_to_end() { /// silently be taken as covering the new signature. #[tokio::test] async fn a_republish_needs_and_gets_its_own_seal() { - let (dal, _pg) = start_pg().await; + let _pg = start_pg().await; + let dal = _pg.dal.clone(); let mock = Arc::new(MockState::default()); let base_url = spawn_mock(mock.clone()).await; @@ -555,7 +519,8 @@ async fn a_republish_needs_and_gets_its_own_seal() { /// simulation would actually catch that. #[tokio::test] async fn a_wrong_key_is_rejected_and_the_row_stays_pending() { - let (dal, _pg) = start_pg().await; + let _pg = start_pg().await; + let dal = _pg.dal.clone(); let mock = Arc::new(MockState::default()); let base_url = spawn_mock(mock.clone()).await; diff --git a/crates/dpp-node/tests/smoke.rs b/crates/dpp-node/tests/smoke.rs index 1784bb0..e5d2783 100644 --- a/crates/dpp-node/tests/smoke.rs +++ b/crates/dpp-node/tests/smoke.rs @@ -14,18 +14,14 @@ use std::sync::{Arc, OnceLock}; use async_trait::async_trait; use metrics_exporter_prometheus::{PrometheusBuilder, PrometheusHandle}; -use testcontainers::{ - GenericImage, ImageExt, - core::{WaitFor, ports::ContainerPort}, - runners::AsyncRunner, -}; use base64::Engine as _; use dpp_crypto::keystore::KeyStore; use dpp_dal::pg::{ PgApiKeyRepo, PgAuditRepo, PgDal, PgEvidenceDossierRepo, PgOperatorConfigRepo, PgPassportRepo, - PgRegistryIdentityRepo, PgScanTelemetryRepo, PgTransferRepo, PgWebhookRepo, sqlx, + PgRegistryIdentityRepo, PgScanTelemetryRepo, PgTransferRepo, PgWebhookRepo, }; +use dpp_dal::test_harness::{TestPg, start_pg}; use dpp_domain::domain::passport::PassportRef; use dpp_domain::{ DppError, GhostArchive, GhostRegistrySync, @@ -88,45 +84,6 @@ impl AuthProvider for TestAuthProvider { // DB setup helpers // --------------------------------------------------------------------------- -async fn start_pg() -> (PgDal, testcontainers::ContainerAsync) { - let image = GenericImage::new("postgres", "17") - .with_exposed_port(ContainerPort::Tcp(5432)) - .with_wait_for(WaitFor::message_on_stderr( - "database system is ready to accept connections", - )) - // POSTGRES_USER/PASSWORD/DB are the official Postgres image's required - // env vars for this throwaway testcontainer — NOT the app's - // DATABASE_POSTGRES_PASS / DATABASE_APP_PASS scheme. - .with_env_var("POSTGRES_USER", "postgres") - .with_env_var("POSTGRES_PASSWORD", "test") - .with_env_var("POSTGRES_DB", "odal"); - - let container = image.start().await.expect("start postgres container"); - let port = container - .get_host_port_ipv4(5432) - .await - .expect("mapped port"); - let admin_url = format!("postgres://postgres:test@127.0.0.1:{port}/odal"); - - tokio::time::sleep(std::time::Duration::from_millis(1500)).await; - - let admin = sqlx::postgres::PgPoolOptions::new() - .max_connections(1) - .connect(&admin_url) - .await - .expect("admin connect"); - sqlx::query("CREATE ROLE odal_app LOGIN PASSWORD 'test'") - .execute(&admin) - .await - .expect("create app role"); - - PgDal::migrate(&admin_url).await.expect("apply migrations"); - - let app_url = format!("postgres://odal_app:test@127.0.0.1:{port}/odal"); - let dal = PgDal::connect(&app_url).await.expect("app connect"); - (dal, container) -} - // --------------------------------------------------------------------------- // Node factory helpers // --------------------------------------------------------------------------- @@ -162,8 +119,14 @@ async fn start_node_with_dal(dal: PgDal) -> String { // In-process signing — the fused node signs via LocalIdentityService exactly // as main.rs does; the internal HTTP sign route is intentionally unmounted // (ATK-1). The same key store backs both signing and the did:web document. - let ks_path = std::env::temp_dir().join(format!("node-smoke-ks-{}.json", uuid::Uuid::now_v7())); - let key_store = Arc::new(KeyStore::open(&ks_path, "test-passphrase").expect("open key store")); + // `tempfile` creates the directory with restrictive permissions and removes + // it on drop; `env::temp_dir()` did neither, leaving an Ed25519 private key + // behind on every run. + let ks_dir = tempfile::tempdir().expect("temp dir"); + let key_store = Arc::new( + KeyStore::open(ks_dir.path().join("keystore.json"), "test-passphrase") + .expect("open key store"), + ); key_store .generate_key("root") .expect("provision root issuer key"); @@ -298,10 +261,12 @@ async fn seed_complete_operator(repo: &PgOperatorConfigRepo) { // Tier 1 — health + auth (uses a shared DB container; auth fires before DB) // --------------------------------------------------------------------------- -async fn start_db_and_node() -> (String, testcontainers::ContainerAsync) { - let (dal, container) = start_pg().await; - let node_url = start_node_with_dal(dal).await; - (node_url, container) +// Returns the whole `TestPg` rather than just the container: it owns the +// container privately, and the caller only needs to keep it alive. +async fn start_db_and_node() -> (String, TestPg) { + let pg = start_pg().await; + let node_url = start_node_with_dal(pg.dal.clone()).await; + (node_url, pg) } /// The ghost-honesty invariant, on the endpoint that now carries it. diff --git a/crates/dpp-node/tests/snapshot_outbox.rs b/crates/dpp-node/tests/snapshot_outbox.rs index 03a5ccc..5e35855 100644 --- a/crates/dpp-node/tests/snapshot_outbox.rs +++ b/crates/dpp-node/tests/snapshot_outbox.rs @@ -16,6 +16,7 @@ use async_trait::async_trait; use base64::Engine; use chrono::Utc; +use dpp_dal::in_memory_repo::InMemoryPassportRepo; use dpp_domain::{ DppError, domain::{ @@ -35,72 +36,6 @@ use dpp_node::infra::snapshot_drain::{MAX_ATTEMPTS, drain_once}; // In-memory ports // --------------------------------------------------------------------------- -#[derive(Default, Clone)] -struct InMemoryPassportRepo { - store: Arc>>, -} - -#[async_trait] -impl PassportRepository for InMemoryPassportRepo { - async fn create(&self, passport: Passport) -> Result { - self.store - .lock() - .unwrap() - .insert(passport.id, passport.clone()); - Ok(passport) - } - async fn find_by_id(&self, id: PassportId) -> Result, DppError> { - Ok(self.store.lock().unwrap().get(&id).cloned()) - } - async fn find_published_by_id(&self, id: PassportId) -> Result, DppError> { - self.find_by_id(id).await - } - async fn find_published_by_gtin(&self, _gtin: &str) -> Result, DppError> { - Ok(None) - } - async fn find_by_id_any_status(&self, id: PassportId) -> Result, DppError> { - self.find_by_id(id).await - } - async fn update(&self, passport: Passport) -> Result { - self.store - .lock() - .unwrap() - .insert(passport.id, passport.clone()); - Ok(passport) - } - async fn update_status( - &self, - id: PassportId, - status: PassportStatus, - ) -> Result { - let mut g = self.store.lock().unwrap(); - let mut p = g - .get(&id) - .cloned() - .ok_or_else(|| DppError::NotFound(id.to_string()))?; - p.status = status; - g.insert(id, p.clone()); - Ok(p) - } - async fn list( - &self, - _status: Option, - _q: Option<&str>, - _facility_id: Option<&str>, - _limit: u32, - _offset: u32, - ) -> Result, DppError> { - Ok(self.store.lock().unwrap().values().cloned().collect()) - } - async fn count( - &self, - _status: Option, - _facility_id: Option<&str>, - ) -> Result { - Ok(self.store.lock().unwrap().len() as u64) - } -} - /// Object store double. Optionally fails every write, to drive the retry path. #[derive(Default, Clone)] struct InMemorySnapshotStore { diff --git a/crates/dpp-node/tests/transfer_outbox.rs b/crates/dpp-node/tests/transfer_outbox.rs index cf5d26a..fc05751 100644 --- a/crates/dpp-node/tests/transfer_outbox.rs +++ b/crates/dpp-node/tests/transfer_outbox.rs @@ -20,14 +20,10 @@ use std::sync::Arc; use chrono::Utc; -use testcontainers::{ - GenericImage, ImageExt, - core::{WaitFor, ports::ContainerPort}, - runners::AsyncRunner, -}; use uuid::Uuid; use dpp_dal::pg::{PgDal, PgPassportRepo, PgRegistryTransferRepo, sqlx}; +use dpp_dal::test_harness::start_pg; use dpp_domain::{ domain::{ passport::{ManufacturerInfo, Passport, PassportId}, @@ -43,41 +39,6 @@ use dpp_types::{RegistryTransferOutbox, RegistryTransferStatus}; // ─── Harness ──────────────────────────────────────────────────────────────── -async fn start_pg() -> (PgDal, testcontainers::ContainerAsync) { - let image = GenericImage::new("postgres", "17") - .with_exposed_port(ContainerPort::Tcp(5432)) - .with_wait_for(WaitFor::message_on_stderr( - "database system is ready to accept connections", - )) - .with_env_var("POSTGRES_USER", "postgres") - .with_env_var("POSTGRES_PASSWORD", "test") - .with_env_var("POSTGRES_DB", "odal"); - - let container = image.start().await.expect("start postgres container"); - let port = container - .get_host_port_ipv4(5432) - .await - .expect("mapped port"); - let admin_url = format!("postgres://postgres:test@127.0.0.1:{port}/odal"); - - tokio::time::sleep(std::time::Duration::from_millis(1500)).await; - - let admin = sqlx::postgres::PgPoolOptions::new() - .max_connections(1) - .connect(&admin_url) - .await - .expect("admin connect"); - sqlx::query("CREATE ROLE odal_app LOGIN PASSWORD 'test'") - .execute(&admin) - .await - .expect("create app role"); - PgDal::migrate(&admin_url).await.expect("apply migrations"); - - let app_url = format!("postgres://odal_app:test@127.0.0.1:{port}/odal"); - let dal = PgDal::connect(&app_url).await.expect("app connect"); - (dal, container) -} - fn published_passport() -> Passport { Passport { id: PassportId::new(), @@ -178,7 +139,8 @@ async fn accept( /// transaction: the row exists, carries the signed record, and is drainable. #[tokio::test] async fn accepting_a_transfer_enqueues_a_pending_notification() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let (passport_id, outbox) = setup(&dal).await; let mut chain = TransferChain::new(passport_id, operator("did:web:acme.example", "Acme")); @@ -216,7 +178,8 @@ async fn accepting_a_transfer_enqueues_a_pending_notification() { /// would overwrite the first and the registry would never hear about it. #[tokio::test] async fn a_passport_transferred_twice_owes_two_notifications() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let (passport_id, outbox) = setup(&dal).await; let mut chain = TransferChain::new(passport_id, operator("did:web:acme.example", "Acme")); @@ -254,7 +217,8 @@ async fn a_passport_transferred_twice_owes_two_notifications() { /// one handover. #[tokio::test] async fn re_accepting_the_same_transfer_never_re_notifies() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let (passport_id, outbox) = setup(&dal).await; let mut chain = TransferChain::new(passport_id, operator("did:web:acme.example", "Acme")); @@ -287,7 +251,8 @@ async fn re_accepting_the_same_transfer_never_re_notifies() { /// attempt into the future — the notification is never lost, just deferred. #[tokio::test] async fn a_transient_failure_backs_off_without_losing_the_row() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let (passport_id, outbox) = setup(&dal).await; let mut chain = TransferChain::new(passport_id, operator("did:web:acme.example", "Acme")); @@ -323,7 +288,8 @@ async fn a_transient_failure_backs_off_without_losing_the_row() { /// A terminal rejection stops the row draining but keeps it for audit. #[tokio::test] async fn a_rejected_notification_is_kept_for_audit() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let (passport_id, outbox) = setup(&dal).await; let mut chain = TransferChain::new(passport_id, operator("did:web:acme.example", "Acme")); diff --git a/crates/dpp-node/tests/webhook_outbox.rs b/crates/dpp-node/tests/webhook_outbox.rs index 81fbfc6..61aaeeb 100644 --- a/crates/dpp-node/tests/webhook_outbox.rs +++ b/crates/dpp-node/tests/webhook_outbox.rs @@ -23,13 +23,9 @@ use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use axum::{Router, extract::State, http::HeaderMap, http::StatusCode, routing::post}; use hmac::{Hmac, KeyInit, Mac}; use sha2::Sha256; -use testcontainers::{ - GenericImage, ImageExt, - core::{WaitFor, ports::ContainerPort}, - runners::AsyncRunner, -}; -use dpp_dal::pg::{PgDal, PgWebhookRepo, sqlx}; +use dpp_dal::pg::{PgWebhookRepo, sqlx}; +use dpp_dal::test_harness::start_pg; use dpp_node::infra::webhook_drain::{MAX_ATTEMPTS, drain_once}; use dpp_types::{ NewWebhookSubscription, WebhookDeliveryRow, WebhookOutbox, WebhookSubscriptionStore, @@ -39,41 +35,6 @@ type HmacSha256 = Hmac; // ─── Postgres harness ───────────────────────────────────────────────────────── -async fn start_pg() -> (PgDal, testcontainers::ContainerAsync) { - let image = GenericImage::new("postgres", "17") - .with_exposed_port(ContainerPort::Tcp(5432)) - .with_wait_for(WaitFor::message_on_stderr( - "database system is ready to accept connections", - )) - .with_env_var("POSTGRES_USER", "postgres") - .with_env_var("POSTGRES_PASSWORD", "test") - .with_env_var("POSTGRES_DB", "odal"); - - let container = image.start().await.expect("start postgres container"); - let port = container - .get_host_port_ipv4(5432) - .await - .expect("mapped port"); - let admin_url = format!("postgres://postgres:test@127.0.0.1:{port}/odal"); - - tokio::time::sleep(std::time::Duration::from_millis(1500)).await; - - let admin = sqlx::postgres::PgPoolOptions::new() - .max_connections(1) - .connect(&admin_url) - .await - .expect("admin connect"); - sqlx::query("CREATE ROLE odal_app LOGIN PASSWORD 'test'") - .execute(&admin) - .await - .expect("create app role"); - PgDal::migrate(&admin_url).await.expect("apply migrations"); - - let app_url = format!("postgres://odal_app:test@127.0.0.1:{port}/odal"); - let dal = PgDal::connect(&app_url).await.expect("app connect"); - (dal, container) -} - // ─── Mock receiver ──────────────────────────────────────────────────────────── #[derive(Clone)] @@ -184,7 +145,8 @@ async fn one_due(outbox: &Arc) -> WebhookDeliveryRow { #[tokio::test(flavor = "multi_thread")] async fn delivers_signed_and_honours_subject_filter() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let store = PgWebhookRepo::new(dal.clone()); let outbox: Arc = Arc::new(PgWebhookRepo::new(dal.clone())); let (url, receiver) = start_receiver().await; @@ -246,7 +208,8 @@ async fn delivers_signed_and_honours_subject_filter() { #[tokio::test(flavor = "multi_thread")] async fn retries_then_exhausts_then_reconstructed_outbox_redelivers() { - let (dal, _c) = start_pg().await; + let _c = start_pg().await; + let dal = _c.dal.clone(); let store = PgWebhookRepo::new(dal.clone()); let outbox: Arc = Arc::new(PgWebhookRepo::new(dal.clone())); let (url, receiver) = start_receiver().await; diff --git a/crates/dpp-vault/Cargo.toml b/crates/dpp-vault/Cargo.toml index 3f3dd8d..4a17233 100644 --- a/crates/dpp-vault/Cargo.toml +++ b/crates/dpp-vault/Cargo.toml @@ -55,6 +55,13 @@ subtle = { workspace = true } integration-tests = [] [dev-dependencies] +# The shared in-memory PassportRepository double lives behind dpp-dal's +# dev-only `test-harness` feature; the normal dependency above stays featureless. +dpp-dal = { path = "../dpp-dal", features = ["test-harness"] } +# Test keystores hold Ed25519 private keys. `tempfile` creates the directory +# with restrictive permissions and removes it on drop; `env::temp_dir()` does +# neither, and left a file behind on every run. +tempfile = "3" tower = { version = "0.5", features = ["util"] } axum = { workspace = true } testcontainers = { workspace = true } diff --git a/crates/dpp-vault/tests/continuity_snapshot.rs b/crates/dpp-vault/tests/continuity_snapshot.rs index 0f3f6c7..4b20111 100644 --- a/crates/dpp-vault/tests/continuity_snapshot.rs +++ b/crates/dpp-vault/tests/continuity_snapshot.rs @@ -10,12 +10,12 @@ //! right passport. The byte-identical render + JWS-travels contract is //! unit-tested in `service::mod`. -use std::collections::HashMap; use std::sync::{Arc, Mutex}; use async_trait::async_trait; use chrono::Utc; +use dpp_dal::in_memory_repo::InMemoryPassportRepo; use dpp_domain::{ DppError, GhostArchive, GhostRegistrySync, compliance::passthrough_registry::PassthroughRegistry, @@ -25,7 +25,6 @@ use dpp_domain::{ sector::Sector, status::PassportStatus, }, - ports::passport_repo::PassportRepository, }; use dpp_types::{ api_key::ApiKeyScope, @@ -39,72 +38,6 @@ use dpp_vault::domain::service::{OperatorIdentity, PassportService}; // In-memory ports (no Docker/Postgres) — the two the lifecycle actually needs. // --------------------------------------------------------------------------- -#[derive(Default)] -struct InMemoryPassportRepo { - store: Mutex>, -} - -#[async_trait] -impl PassportRepository for InMemoryPassportRepo { - async fn create(&self, passport: Passport) -> Result { - self.store - .lock() - .unwrap() - .insert(passport.id, passport.clone()); - Ok(passport) - } - async fn find_by_id(&self, id: PassportId) -> Result, DppError> { - Ok(self.store.lock().unwrap().get(&id).cloned()) - } - async fn find_published_by_id(&self, id: PassportId) -> Result, DppError> { - self.find_by_id(id).await - } - async fn find_published_by_gtin(&self, _gtin: &str) -> Result, DppError> { - Ok(None) - } - async fn find_by_id_any_status(&self, id: PassportId) -> Result, DppError> { - self.find_by_id(id).await - } - async fn update(&self, passport: Passport) -> Result { - self.store - .lock() - .unwrap() - .insert(passport.id, passport.clone()); - Ok(passport) - } - async fn update_status( - &self, - id: PassportId, - status: PassportStatus, - ) -> Result { - let mut g = self.store.lock().unwrap(); - let mut p = g - .get(&id) - .cloned() - .ok_or_else(|| DppError::NotFound(id.to_string()))?; - p.status = status; - g.insert(id, p.clone()); - Ok(p) - } - async fn list( - &self, - _status: Option, - _q: Option<&str>, - _facility_id: Option<&str>, - _limit: u32, - _offset: u32, - ) -> Result, DppError> { - Ok(self.store.lock().unwrap().values().cloned().collect()) - } - async fn count( - &self, - _status: Option, - _facility_id: Option<&str>, - ) -> Result { - Ok(self.store.lock().unwrap().len() as u64) - } -} - /// Chains entries exactly as `PgAuditRepo` does, so the lifecycle's audit /// appends succeed. #[derive(Default)] @@ -220,10 +153,13 @@ fn auth() -> AuthContext { /// A `PassportService` with real signing + in-memory ports + the reconcile outbox. async fn build_service() -> (PassportService, InMemorySnapshotOutbox) { - let key_path = - std::env::temp_dir().join(format!("snapshot-test-{}.json", uuid::Uuid::new_v4())); + // `tempfile` creates the directory with restrictive permissions and removes + // it on drop; `env::temp_dir()` did neither, leaving an Ed25519 private key + // behind on every run. + let key_dir = tempfile::tempdir().expect("temp dir"); let store = - dpp_crypto::keystore::KeyStore::open(&key_path, "test-pass").expect("open keystore"); + dpp_crypto::keystore::KeyStore::open(key_dir.path().join("keystore.json"), "test-pass") + .expect("open keystore"); store.generate_key("root").expect("generate key"); let identity = Arc::new(dpp_vc::LocalIdentityService::new( Arc::new(store), diff --git a/crates/dpp-vault/tests/evidence_dossier.rs b/crates/dpp-vault/tests/evidence_dossier.rs index d051397..6beb31b 100644 --- a/crates/dpp-vault/tests/evidence_dossier.rs +++ b/crates/dpp-vault/tests/evidence_dossier.rs @@ -16,6 +16,7 @@ use async_trait::async_trait; use chrono::Utc; use uuid::Uuid; +use dpp_dal::in_memory_repo::InMemoryPassportRepo; use dpp_domain::{ DppError, GhostArchive, GhostRegistrySync, compliance::passthrough_registry::PassthroughRegistry, @@ -26,7 +27,6 @@ use dpp_domain::{ status::PassportStatus, transfer::{OperatorRole, ResponsibleOperator, TransferChain, TransferReason}, }, - ports::passport_repo::PassportRepository, }; use dpp_types::{ api_key::ApiKeyScope, @@ -43,72 +43,6 @@ use dpp_vault::domain::service::{OperatorIdentity, PassportService}; // In-memory ports (no Docker/Postgres — see module doc comment) // --------------------------------------------------------------------------- -#[derive(Default)] -struct InMemoryPassportRepo { - store: Mutex>, -} - -#[async_trait] -impl PassportRepository for InMemoryPassportRepo { - async fn create(&self, passport: Passport) -> Result { - self.store - .lock() - .unwrap() - .insert(passport.id, passport.clone()); - Ok(passport) - } - async fn find_by_id(&self, id: PassportId) -> Result, DppError> { - Ok(self.store.lock().unwrap().get(&id).cloned()) - } - async fn find_published_by_id(&self, id: PassportId) -> Result, DppError> { - self.find_by_id(id).await - } - async fn find_published_by_gtin(&self, _gtin: &str) -> Result, DppError> { - Ok(None) - } - async fn find_by_id_any_status(&self, id: PassportId) -> Result, DppError> { - self.find_by_id(id).await - } - async fn update(&self, passport: Passport) -> Result { - self.store - .lock() - .unwrap() - .insert(passport.id, passport.clone()); - Ok(passport) - } - async fn update_status( - &self, - id: PassportId, - status: PassportStatus, - ) -> Result { - let mut g = self.store.lock().unwrap(); - let mut p = g - .get(&id) - .cloned() - .ok_or_else(|| DppError::NotFound(id.to_string()))?; - p.status = status; - g.insert(id, p.clone()); - Ok(p) - } - async fn list( - &self, - _status: Option, - _q: Option<&str>, - _facility_id: Option<&str>, - _limit: u32, - _offset: u32, - ) -> Result, DppError> { - Ok(self.store.lock().unwrap().values().cloned().collect()) - } - async fn count( - &self, - _status: Option, - _facility_id: Option<&str>, - ) -> Result { - Ok(self.store.lock().unwrap().len() as u64) - } -} - /// Chains entries exactly as `dpp-dal::pg::repo_audit::PgAuditRepo` does — /// read the current head's `entry_hash` (or genesis), fold it into the new /// entry's hash, store both. Without this, `verify_audit_chain` would fail @@ -238,10 +172,13 @@ fn auth() -> AuthContext { /// ports, plus the DID the identity's did:web document actually publishes as /// (pathless form — see `dpp_vc::did_builder`). async fn build_service() -> (PassportService, Arc, String) { - let key_path = - std::env::temp_dir().join(format!("evidence-test-{}.json", uuid::Uuid::new_v4())); + // `tempfile` creates the directory with restrictive permissions and removes + // it on drop; `env::temp_dir()` did neither, leaving an Ed25519 private key + // behind on every run. + let key_dir = tempfile::tempdir().expect("temp dir"); let store = - dpp_crypto::keystore::KeyStore::open(&key_path, "test-pass").expect("open keystore"); + dpp_crypto::keystore::KeyStore::open(key_dir.path().join("keystore.json"), "test-pass") + .expect("open keystore"); store.generate_key("root").expect("generate key"); let base_url = "evidence-test.example.com".to_owned(); let issuer_did = format!("did:web:{}", base_url.replace(':', "%3A")); diff --git a/justfile b/justfile index 826de07..c276b3e 100644 --- a/justfile +++ b/justfile @@ -226,7 +226,7 @@ doc: cargo doc --workspace --no-deps # Fast gate (no Docker) — mirrors CI jobs: fmt, clippy, debug-prints, test-unit, audit -check: fmt-check lint debug-check subjects-check mod-rs-check spec-version-check outbound-check grants-check migrations-check check-plugins test check-integration audit +check: fmt-check lint debug-check subjects-check mod-rs-check harness-check spec-version-check outbound-check grants-check migrations-check check-plugins test check-integration audit # Full local CI mirror — adds integration-feature clippy + the Docker tiers (needs Docker running) ci: check lint-integration test-integration test-pg @@ -428,3 +428,13 @@ check-plugins: # Clean build artefacts clean: cargo clean + +# Refuse a forked copy of shared test scaffolding. +# +# Rust cannot share `#[cfg(test)]` code across crates, so copying is the path of +# least resistance and nothing signals when it happens: the Postgres harness +# reached eight copies and six divergent implementations before anyone counted. +# Both it and the in-memory PassportRepository double now live behind dpp-dal's +# `test-harness` feature; this is the signal that was missing. +harness-check: + bash scripts/harness-check.sh diff --git a/scripts/harness-check.sh b/scripts/harness-check.sh new file mode 100644 index 0000000..84ef0c8 --- /dev/null +++ b/scripts/harness-check.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Forbid re-copying shared test scaffolding that already has one home. +# +# Rust cannot share `#[cfg(test)]` code across crate boundaries, so copying is +# the path of least resistance and nothing signals when it happens. The Postgres +# harness reached eight copies that had drifted into six different +# implementations before anyone counted, and the in-memory PassportRepository +# double reached three. Both now live behind dpp-dal's `test-harness` feature. +# +# This is the signal that was missing. Each entry names a definition and the one +# file allowed to contain it; anywhere else is a fork. +# +# If the shared version cannot do what a suite needs, extend it in its home +# rather than adding an exception here. An exception list that grows is this +# check failing at its job. +set -euo pipefail + +# "||" +rules=( + "crates/dpp-dal/src/test_harness.rs|^async fn start_pg|use dpp_dal::test_harness::{start_pg, start_pg_raw, start_pg_before}" + "crates/dpp-dal/src/in_memory_repo.rs|^impl PassportRepository for InMemoryPassportRepo|use dpp_dal::in_memory_repo::InMemoryPassportRepo" +) + +status=0 +for rule in "${rules[@]}"; do + home="${rule%%|*}" + rest="${rule#*|}" + pattern="${rest%%|*}" + remedy="${rest#*|}" + + # One recursive grep, not a per-file loop: a loop over several hundred files + # stalls for a minute on Windows, which is how a gate stops being run. + hits=$(grep -rlE "$pattern" --include="*.rs" crates cli 2>/dev/null | grep -v "^${home}$" || true) + + if [ -n "$hits" ]; then + echo "ERROR: '$pattern' is defined outside $home:" + while IFS= read -r hit; do + echo " $hit" + done <<< "$hits" + echo " Use instead: $remedy" + status=1 + fi +done + +if [ "$status" -ne 0 ]; then + echo + echo "Shared test scaffolding has one home. Extend it there rather than forking a copy." + exit 1 +fi + +echo "harness-check: no forked copies of shared test scaffolding."