From 51b7173b29d9365c3c0de2f6b00d6d6b0e6603fe Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 05:52:41 -0400 Subject: [PATCH 01/44] =?UTF-8?q?docs(spec):=20migration=20chain=20replaya?= =?UTF-8?q?bility=20=E2=80=94=20drop=20safety=20and=20a=20provisioning=20g?= =?UTF-8?q?ate=20(#313)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design for #313, reshaped by review and a two-arm challenge from the obvious version. Two findings changed it. First, the gate the reporter needs is not the gate this project already designed. verifyReplay exists, is exported, and has no CLI caller because the 2026-05-31 design retained replay only as an optional integrity aid that compares a replayed database against the committed snapshot. Comparing against the snapshot is unpassable for three supported adoption paths: baseline --from-db writes the whole introspected schema against an empty chain, migrate.scope deliberately carries another owner's tables into the snapshot, and no migration ever emits CREATE SCHEMA. Asserting only that the chain APPLIES from empty catches the reported bug, passes the first two classes, and leaves one true positive that CREATE SCHEMA IF NOT EXISTS fixes. So: one subverb, two tiers. Second, the scratch-database provisioning the obvious design reached for is unnecessary and dangerous. The prior design already specified an in-process engine tier (PGlite for postgres, :memory: libsql for sqlite). That needs no CREATEDB, survives connection poolers and managed Postgres, cannot collide between parallel CI jobs, and has no database to accidentally drop — where a derived scratch name could truncate at Postgres's 63-byte identifier limit back onto the very database it was about to DROP. Also rules that IF EXISTS applies to forward drops only: with it on a create-table down, a rollback whose object is already gone would no-op and still delete its ledger row. Co-Authored-By: Claude Opus 5 (1M context) --- ...8-19-migrate-chain-replayability-design.md | 216 ++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md diff --git a/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md b/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md new file mode 100644 index 000000000..5005fb042 --- /dev/null +++ b/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md @@ -0,0 +1,216 @@ +# Migration chain replayability — drop safety, a provisioning gate, and the promise the docs already make + +**Issue:** [#313](https://github.com/metaobjectsdev/metaobjects/issues/313) · **Scope:** TypeScript +only (schema migration is TS-owned, [ADR-0015](../../../spec/decisions/ADR-0015-single-shared-migrate-engine.md)) +· **Supersedes nothing.** Completes the `verify --replay` tier specified but never wired in +[2026-05-31 migrate-ts reference-snapshot design](2026-05-31-migrate-ts-reference-snapshot-and-generation-gaps-design.md) §8. + +## 1. The problem, as reported + +`meta migrate` writes a bare `DROP TABLE "x"` into a committed migration when `x` is present in the +live database and absent from metadata — **even when no migration in the chain ever creates `x`**. +In the reported case another tool owned that table. Replaying the chain against an empty database +then dies: + +``` +$ meta migrate apply-pending --db postgresql://…/fresh +meta: migrate apply-pending: apply failed: table "arena_season_standing" does not exist +``` + +Nothing warns at generation time. The reporter's chain was broken for roughly three months; the +only working database left was a leftover CI container. This contradicts a promise the toolchain +makes in two places — `meta migrate --help` (`migrate.ts:69-70`) and +[`docs/features/migrations-and-drift.md:58`](../../features/migrations-and-drift.md), which says +`apply-pending` "is the way to provision a fresh or CI database". + +## 2. What the investigation changed about the obvious design + +The obvious design — emit `IF EXISTS`, then add a gate that replays the chain and compares the +result to the committed snapshot — is wrong in a way that only shows up under review. Two findings +reshaped it. + +**The two gates are not one gate.** `migrate-ts/src/verify/replay.ts:31` already implements +`verifyReplay`: replay into a caller-supplied database, introspect, and classify drift **against the +committed snapshot**. It is exported (`index.ts:112`) and has no CLI caller. That is not an +oversight — the 2026-05-31 design §3 ruled it: *"we keep the descriptor snapshot as the generate +reference and retain replay only as the optional `verify --replay` integrity aid (§8)."* Its purpose +is catching **hand-edited structural DDL** that diverges from the snapshot. + +The reporter did not ask for that. Their failure was `table "x" does not exist` — the chain does not +**apply**. That is a strictly weaker assertion, and the difference decides who can use the gate: + +| Project class | Chain applies from empty? | Chain reproduces the snapshot? | +|---|---|---| +| Adopted via `baseline --from-db` (`migrate.ts:870` snapshots the whole introspected DB against an empty chain) | **passes trivially** — nothing to apply | **fails by construction** | +| Declares `migrate.scope` (`carryForwardOutOfScope`, `scope.ts:93`, writes the other owner's tables into the snapshot) | **passes** — the chain only creates in-scope objects | **fails** unless `excludeFromSnapshot` is threaded | +| Uses `@schema` (`CREATE SCHEMA` is emitted nowhere but the ledger, `ledger.ts:126`) | **fails — a true positive** | fails | +| The reported bug | **fails — the defect** | fails | + +Comparing against the snapshot makes the gate unpassable for three documented, supported adoption +paths. Asserting only that the chain applies makes it precise: it catches the reported bug, is +immune to the first two classes, and its one remaining failure is a real defect with a real fix. + +**Both assertions are worth having, at different strengths.** So this ships one subverb with two +tiers rather than two commands. + +## 3. Design + +### 3.1 `IF EXISTS` on forward drops, and only forward drops + +Change these to `IF EXISTS` — all verified present and bare: + +| Site | Change | +|---|---| +| `emit/postgres.ts:66` | `drop-table` | +| `emit/sqlite.ts:219` | `drop-table` | +| `emit/postgres.ts:375`, `:388` | `renderDropView`, plain and CASCADE | +| `emit/postgres.ts:431` | `renderRestoreView`'s illegal-replace fallback | +| `emit/postgres.ts:93-96` | `drop-index` — **both arms**: the plain `DROP INDEX`, and the constraint-backed `ALTER TABLE … DROP CONSTRAINT`, which Postgres spells `DROP CONSTRAINT IF EXISTS` | +| `emit/sqlite.ts:225` | `drop-index` | + +**The rule is per change-kind, not per statement:** every `drop-table`, `drop-view` and `drop-index` +is guarded in **both** dialects. Postgres's constraint-backed index arm is included because it is +how Postgres renders the *same* `drop-index` change whose SQLite rendering is guarded at +`sqlite.ts:225` — guarding one and not the other would leave the change kind half-covered. + +**`drop-column`, `drop-fk` and `drop-check` are excluded**, and the reason is structural rather than +a preference: SQLite has no `DROP COLUMN IF EXISTS`, and it emits no standalone statement for +`drop-fk`/`drop-check` at all — those change kinds trigger a table rebuild (`sqlite.ts:150-152`, +`:226-227`). Guarding them on Postgres alone would make the guarantee dialect-dependent for the same +declared change, which is the failure mode this rule exists to avoid. + +**Down statements stay bare** (`postgres.ts:113`, `:176`, `sqlite.ts:256`). `rollbackTo` +(`apply/apply.ts:149`) runs `down.sql` and deletes the ledger row in one transaction; with +`IF EXISTS` a rollback whose object is already gone would no-op and *still* record the rollback as +done. Rollback is the one place the loud failure is load-bearing, and the replay gate never +exercises the down direction. The rule is therefore: **`IF EXISTS` on forward drops; downs +unchanged.** + +**Two forward drops stay bare deliberately** — `sqlite.ts:197` (the recreate-and-copy rebuild) and +`d1-cascade.ts:126`. Both drop a table the same recipe just `INSERT…SELECT`ed from; `IF EXISTS` +there converts a caught corruption into a silent one. This is stated so a later sweep does not +"finish the job". + +**D1 inherits this.** `emit/d1.ts:21` renders through `renderSqlite`, so D1's committed migrations +change too, while `--dialect d1` is refused by the gate in §3.2. Accepted: the emitter fix is +independently correct, and D1 keeps the `apply-pending` refusal it already has. + +### 3.2 `meta verify --replay` — the chain applies from empty + +A new subverb alongside `--templates` / `--db` / `--codegen` (ADR-0021 D2, `verify.ts:116-130`). +Opt-in; a bare `verify` never runs it. + +**Default tier — applies.** Provision an empty database, run `applyPending` against it, assert it +completes. Nothing is compared. This is the #313 gate. + +**`--replay --strict`** additionally asserts the result equals the committed snapshot, via the +existing `verifyReplay`. This is the 2026-05-31 §8 integrity aid, finally wired, and it is where +`excludeFromSnapshot` (`scope.ts:130`) must be threaded so a scoped project can use it — today +`verify/replay.ts` does not thread it, though `verify.ts:659` shows the pattern. + +**Engine, per 2026-05-31 §8's tiering** — in-process, so the gate needs no infrastructure and there +is no scratch database to name, collide with, or accidentally drop: + +- **sqlite** → `:memory:` libsql +- **postgres** → PGlite (`@electric-sql/pglite`, WASM Postgres in-process), lazily imported so it + costs nothing for projects that never run the gate +- **CI, optional higher fidelity** → real Postgres when `MIGRATE_TS_PG_URL` is set + +This replaces the sibling-scratch-database approach considered earlier. That approach needed +`CREATEDB`, broke behind connection poolers and on managed Postgres, collided between parallel CI +jobs sharing one server, and — through Postgres's 63-byte identifier truncation — could derive a +scratch name that truncates back to the target database it was about to `DROP`. + +**Refusals**, mirroring `apply-pending` (`migrate.ts:419-426`, `:449-457`): `--migration-format +flyway` and `--dialect d1`. + +**Zero committed migrations** is not a silent pass. `discoverMigrations` returns `[]` for a missing +directory (`apply.ts:316-322`), so the run would otherwise succeed having proved nothing. The gate +reports "no committed migrations — nothing to replay" and, at `--strict` against a non-empty +snapshot, fails. + +**A baselined chain is skipped with a reason, not failed.** At `--strict`, a chain that provably +cannot build the snapshot (baseline adoption) reports that it was skipped and why. A gate that +convicts a supported adoption path gets suppressed, and a suppressed gate protects nothing. + +**Exit codes** follow `verify`'s convention (`Math.max` at `verify.ts:239`): a chain that fails to +apply, or a `--strict` mismatch, is **drift → 1**; an engine that cannot start is **operational → 2**. + +### 3.3 `CREATE SCHEMA IF NOT EXISTS` for `@schema` projects + +The one true positive the gate surfaces is real and fixable: a chain containing +`CREATE TABLE "reporting"."x"` cannot apply to a virgin database because no migration creates the +schema. The emitter emits `CREATE SCHEMA IF NOT EXISTS "";` ahead of the first object in a +non-default schema. Without this, `@schema` projects get a red gate and no remedy. + +### 3.4 Emit-time provenance guard + +The gate catches a broken chain after it is committed. The guard stops it being written. + +When the diff proposes dropping an object that is **absent from the committed snapshot**, the object +was never managed by this toolchain — the `drop-table` in #313 is exactly this. Refuse at generation +time, naming the object, unless the drop is explicitly allowed by a new `--allow drop-unmanaged` +token. + +The population problem that sinks the strict gate does not apply here, and for a pleasing reason: +both brownfield mechanisms *add* to the snapshot. A baselined project's snapshot contains the +foreign table, so the guard reads it as managed and does not fire; a scoped project's snapshot +carries out-of-scope entries forward for the same reason. The guard fires precisely when nothing +ever claimed the object — which is the reported case. + +`classify.ts:6-9` already states the doctrine this extends: objects present in the DB but not the +snapshot "must never be treated as actionable drift or auto-dropped". The live path +(`migrate.ts:607-620`) compares metadata against introspection and never consults the snapshot, +which is why the doctrine was not enforced where it mattered. + +### 3.5 Documentation + +`docs/features/migrations-and-drift.md:58` and `meta migrate --help` both promise fresh-database +provisioning. §3.2 makes the promise true for projects whose chain builds the schema; the docs must +say that it is *those* projects, and point at `verify --replay` as the way to know you are one. + +## 4. Remediation for a chain that is already broken + +Applied migrations are checksum-immutable (`apply/apply.ts:88-99`): hand-editing a committed +`up.sql` to add `IF EXISTS` is rejected on any database that already applied it. So the reporter — +and anyone the new gate turns red — cannot fix history in place. + +The supported path is a **compensating migration**: author a new migration that creates the missing +object as the chain expects, or that supersedes the bad drop. The gate's failure message must print +this, with the failing statement and the object name. **A gate whose failure has no documented exit +gets suppressed**, and this one would otherwise go red on exactly the population that asked for it. + +## 5. Testing + +- **§3.1** — emit assertions per dialect and per direction, including explicit assertions that the + rebuild-path drops and the down statements remain bare, so the deliberate exclusions are pinned + rather than remembered. +- **§3.2** — the reporter's scenario as a RED-first regression: a chain containing a drop for a + table it never creates, replayed from empty. It must run **through `applyPending`**, not through + `emit()`. Every prior defect in this area (#226/#241, #243, #255, #285, and 0.21.4's + `BEGIN TRANSACTION` finding) shared one shape — SQL proven statement-by-statement and never proven + through the tool that applies it — and `runSqlFileWithLedgerMutation` (`apply.ts:298`) rewrites + statements before execution. +- **Each row of §2's table** as a case: baselined, scoped, `@schema`, and the reported bug. +- **§3.4** — a drop for a snapshot-absent object refuses; the same drop with `--allow drop-unmanaged` + proceeds; a baselined project does not false-fire. +- The Postgres lane is `MIGRATE_TS_PG_URL`-gated and `describe.skip`s when unset + (`apply-pg.test.ts:21-22`); `pg-gate-sentinel.test.ts` exists because that lane silently rotted red + for eight releases. New Postgres cases must run on PGlite by default so they execute everywhere, + with the real-PG lane as the higher-fidelity tier. + +## 6. Non-goals + +- Column and constraint `IF EXISTS` (§3.1's dialect reason). +- Repairing already-applied chains automatically (§4). +- Any cross-port work: `migrate` is TS-owned. +- Making `--strict` pass for baselined projects. It cannot, by construction, and §3.2 skips them + with a reason instead. + +## 7. Verified by + +Every claim above was re-read at `286d50e6e` before writing: the bare-drop sites, the four existing +`IF EXISTS` sites, `verifyReplay` and its absent CLI caller, the 2026-05-31 §3 and §8 rulings, +`baseline --from-db`'s snapshot, `carryForwardOutOfScope`'s call sites, `CREATE SCHEMA`'s two +ledger-only occurrences, `discoverMigrations`'s empty-directory behaviour, `emit/d1.ts:21`, and the +env-gated Postgres lane. From 5915416401b071827303ae85df08ce0ea4cdec09 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 06:04:28 -0400 Subject: [PATCH 02/44] docs(spec): correct the sqlite exclusion rationale for drop-fk/drop-check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prior text cited sqlite.ts:150-152 and :226-227 as evidence that those change kinds trigger a table rebuild. sqlite.ts:144-156 is changeTable(), a change-to-table-name mapper, not a rebuild list — the citation was a misreading of grep context. The conclusion holds on better evidence: renderUpNative THROWS for add-check/drop-check/add-fk/drop-fk (sqlite.ts:225-235, 'should have been handled by recreate bundler') because SQLite constraints are create-time-only and inline. Separately, drop-column DOES emit natively on sqlite (:222), so its exclusion rests only on SQLite lacking DROP COLUMN IF EXISTS. Two different reasons, both structural; the earlier text merged them into one wrong one. Co-Authored-By: Claude Opus 5 (1M context) --- ...26-08-19-migrate-chain-replayability-design.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md b/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md index 5005fb042..152b86bf6 100644 --- a/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md +++ b/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md @@ -73,10 +73,17 @@ is guarded in **both** dialects. Postgres's constraint-backed index arm is inclu how Postgres renders the *same* `drop-index` change whose SQLite rendering is guarded at `sqlite.ts:225` — guarding one and not the other would leave the change kind half-covered. -**`drop-column`, `drop-fk` and `drop-check` are excluded**, and the reason is structural rather than -a preference: SQLite has no `DROP COLUMN IF EXISTS`, and it emits no standalone statement for -`drop-fk`/`drop-check` at all — those change kinds trigger a table rebuild (`sqlite.ts:150-152`, -`:226-227`). Guarding them on Postgres alone would make the guarantee dialect-dependent for the same +**`drop-column`, `drop-fk` and `drop-check` are excluded**, for two different structural reasons: + +- `drop-column` **does** emit a native SQLite statement (`sqlite.ts:222`, + `ALTER TABLE … DROP COLUMN`), so it is excluded purely because **SQLite has no + `DROP COLUMN IF EXISTS`** — a SQL-dialect fact. +- `drop-fk` / `drop-check` emit **no standalone SQLite statement at all**: `renderUpNative` + throws for them (`sqlite.ts:225-235`, *"should have been handled by recreate bundler"*) because + SQLite constraints are create-time-only and inline, so the change is folded into a table + recreate. + +Either way, guarding them on Postgres alone would make the guarantee dialect-dependent for the same declared change, which is the failure mode this rule exists to avoid. **Down statements stay bare** (`postgres.ts:113`, `:176`, `sqlite.ts:256`). `rollbackTo` From f35f53891aee5de00153f8cb78ea78592995e425 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 06:33:03 -0400 Subject: [PATCH 03/44] docs(spec): six corrections from a two-arm challenge of the rulings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A challenge of the spec's three rulings, plus checks run to settle every factual split it surfaced, convicted the spec on four points and narrowed a fifth. 1. postgres.ts:431 removed from the change list. renderRestoreView is reached only from :178/:179, both in the DOWN renderer, so guarding it violated the forward-only rule in the same section that stated it. 2. drop-fk/drop-check are now guarded on Postgres, and the old rationale is struck as backwards. renderRecreate builds the replacement table from the EXPECTED descriptor and never references the dropped constraint, so SQLite is already replay-safe; Postgres is the only dialect that can fail on an absent constraint, and guarding it makes the two agree rather than diverge. 3. drop-column stays excluded — the one genuine dialect limit, since sqlite emits it natively with no IF EXISTS form. 4. --strict becomes the --replay-snapshot subverb. verify already owns --lax on a different axis (ADR-0023 attribute strictness), and --strict beside it would read as that flag's opposite. 5. The baseline-skip clause is dropped as unimplementable. Its only candidate signal, recordBaseline/BASELINE_NAME, has no production caller, and it would land in the target database's ledger while the gate runs against a fresh in-process database with no ledger. Documented as a limitation instead. 6. The 'three unpassable paths' argument is narrowed to one. scopedDiffInputs narrows BOTH sides — out-of-scope names merge into unmanagedNames — so a scoped project passes under the reporter's literal formulation, and @schema is a true positive rather than an obstacle. Also records sqlite.ts:275 as a known pre-existing deviation, left alone: its create-view down already emits IF EXISTS while the Postgres twin is bare. Co-Authored-By: Claude Opus 5 (1M context) --- ...8-19-migrate-chain-replayability-design.md | 100 ++++++++++++------ 1 file changed, 68 insertions(+), 32 deletions(-) diff --git a/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md b/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md index 152b86bf6..aa225ef02 100644 --- a/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md +++ b/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md @@ -39,16 +39,26 @@ is catching **hand-edited structural DDL** that diverges from the snapshot. The reporter did not ask for that. Their failure was `table "x" does not exist` — the chain does not **apply**. That is a strictly weaker assertion, and the difference decides who can use the gate: -| Project class | Chain applies from empty? | Chain reproduces the snapshot? | +| Project class | Chain applies from empty? | Chain reproduces the snapshot, via `verifyReplay` as built? | |---|---|---| | Adopted via `baseline --from-db` (`migrate.ts:870` snapshots the whole introspected DB against an empty chain) | **passes trivially** — nothing to apply | **fails by construction** | -| Declares `migrate.scope` (`carryForwardOutOfScope`, `scope.ts:93`, writes the other owner's tables into the snapshot) | **passes** — the chain only creates in-scope objects | **fails** unless `excludeFromSnapshot` is threaded | -| Uses `@schema` (`CREATE SCHEMA` is emitted nowhere but the ledger, `ledger.ts:126`) | **fails — a true positive** | fails | +| Declares `migrate.scope` (`carryForwardOutOfScope`, `scope.ts:93`, writes the other owner's tables into the snapshot) | **passes** — the chain only creates in-scope objects | **fails** until `excludeFromSnapshot` is threaded — repairable, and §3.2 threads it | +| Uses `@schema` (`CREATE SCHEMA` is emitted nowhere but the ledger, `ledger.ts:126`) | **fails — a true positive**, fixed by §3.3 | fails | | The reported bug | **fails — the defect** | fails | -Comparing against the snapshot makes the gate unpassable for three documented, supported adoption -paths. Asserting only that the chain applies makes it precise: it catches the reported bug, is -immune to the first two classes, and its one remaining failure is a real defect with a real fix. +Two honest qualifications on that table, because an earlier draft overstated it. The scoped row is +about `verifyReplay` **as currently built**: `scopedDiffInputs` (`scope.ts:188-196`) narrows *both* +sides — out-of-scope names are merged into `unmanagedNames`, which suppresses them on the actual +side too — so under the reporter's literal formulation (replay, then diff against metadata through +the normal scoped path) a scoped project would **pass**. It fails only because `verify/replay.ts` +does not thread those inputs, which §3.2 fixes. And the `@schema` row is a true positive in both +columns, not an obstacle. + +So the durable argument for splitting the tiers is narrower than "three unpassable paths", and it is +this: **baseline adoption is unpassable against the snapshot by construction and cannot be +detected** (§3.2), while the reporter's actual failure is an *apply* error that the weaker +assertion catches directly. One tier answers the bug; the other answers a different question worth +asking. **Both assertions are worth having, at different strengths.** So this ships one subverb with two tiers rather than two commands. @@ -64,27 +74,41 @@ Change these to `IF EXISTS` — all verified present and bare: | `emit/postgres.ts:66` | `drop-table` | | `emit/sqlite.ts:219` | `drop-table` | | `emit/postgres.ts:375`, `:388` | `renderDropView`, plain and CASCADE | -| `emit/postgres.ts:431` | `renderRestoreView`'s illegal-replace fallback | | `emit/postgres.ts:93-96` | `drop-index` — **both arms**: the plain `DROP INDEX`, and the constraint-backed `ALTER TABLE … DROP CONSTRAINT`, which Postgres spells `DROP CONSTRAINT IF EXISTS` | | `emit/sqlite.ts:225` | `drop-index` | +| `emit/postgres.ts:98`, `:104` | `drop-fk`, `drop-check` — Postgres only; see the exclusion note below for why this is not a dialect split | + +**`emit/postgres.ts:431` is deliberately NOT in this list.** An earlier draft included it. It is the +illegal-replace fallback inside `renderRestoreView`, which is reached only from `postgres.ts:178` +and `:179` — both inside the **down** renderer. Guarding it would violate the forward-only rule +stated below, in the same change that states it. **The rule is per change-kind, not per statement:** every `drop-table`, `drop-view` and `drop-index` is guarded in **both** dialects. Postgres's constraint-backed index arm is included because it is how Postgres renders the *same* `drop-index` change whose SQLite rendering is guarded at `sqlite.ts:225` — guarding one and not the other would leave the change kind half-covered. -**`drop-column`, `drop-fk` and `drop-check` are excluded**, for two different structural reasons: - -- `drop-column` **does** emit a native SQLite statement (`sqlite.ts:222`, - `ALTER TABLE … DROP COLUMN`), so it is excluded purely because **SQLite has no - `DROP COLUMN IF EXISTS`** — a SQL-dialect fact. -- `drop-fk` / `drop-check` emit **no standalone SQLite statement at all**: `renderUpNative` - throws for them (`sqlite.ts:225-235`, *"should have been handled by recreate bundler"*) because - SQLite constraints are create-time-only and inline, so the change is folded into a table - recreate. - -Either way, guarding them on Postgres alone would make the guarantee dialect-dependent for the same -declared change, which is the failure mode this rule exists to avoid. +**`drop-fk` and `drop-check` ARE guarded — on Postgres only — and that is not a dialect split.** +An earlier draft excluded them on the reasoning that a Postgres-only guard would make the guarantee +dialect-dependent. That reasoning was backwards. SQLite emits no standalone statement for these +kinds at all: `renderUpNative` throws (`sqlite.ts:225-235`, *"should have been handled by recreate +bundler"*), because SQLite constraints are create-time-only and inline, so the change is folded into +a table recreate. `renderRecreate` builds the replacement table from the **expected** descriptor +(`sqlite.ts:173-183`, `renderCreateTable(tmpDescriptor)`) and never references the dropped +constraint — so **SQLite is already replay-safe here by construction**. Postgres is the only dialect +that can fail on an absent constraint. Guarding it makes the two dialects *agree*. + +**`drop-column` is excluded**, and it is the one genuine dialect limit: SQLite emits it natively +(`sqlite.ts:222`, `ALTER TABLE … DROP COLUMN`) and there is no `DROP COLUMN IF EXISTS` in SQLite. +Guarding Postgres alone here really would make the same declared change behave differently per +dialect, so it stays out and §3.4's guard is what covers it. + +**One known deviation, left alone deliberately.** `sqlite.ts:275` — the `create-view` **down** — +already emits `DROP VIEW IF EXISTS`, while its Postgres twin (`postgres.ts:176`) is bare. That +predates this work and contradicts the forward-only rule below. It is out of scope: it is +view-only, changing it alters rollback behaviour, and no failure has been attributed to it. +Recorded here so a later sweep does not "discover" it as an oversight, and so the rule's one +existing exception is written down rather than remembered. **Down statements stay bare** (`postgres.ts:113`, `:176`, `sqlite.ts:256`). `rollbackTo` (`apply/apply.ts:149`) runs `down.sql` and deletes the ledger row in one transaction; with @@ -107,14 +131,18 @@ independently correct, and D1 keeps the `apply-pending` refusal it already has. A new subverb alongside `--templates` / `--db` / `--codegen` (ADR-0021 D2, `verify.ts:116-130`). Opt-in; a bare `verify` never runs it. -**Default tier — applies.** Provision an empty database, run `applyPending` against it, assert it +**`--replay` — applies.** Provision an empty database, run `applyPending` against it, assert it completes. Nothing is compared. This is the #313 gate. -**`--replay --strict`** additionally asserts the result equals the committed snapshot, via the +**`--replay-snapshot`** additionally asserts the result equals the committed snapshot, via the existing `verifyReplay`. This is the 2026-05-31 §8 integrity aid, finally wired, and it is where `excludeFromSnapshot` (`scope.ts:130`) must be threaded so a scoped project can use it — today `verify/replay.ts` does not thread it, though `verify.ts:659` shows the pattern. +The second tier is a **separate subverb, not a `--strict` modifier**: `verify` already owns a +`--lax` flag on a different axis (ADR-0023 attribute strictness, `args.ts:244`, `:259`, `:302`), and +a `--strict` beside it would read as that flag's opposite rather than as a replay depth. + **Engine, per 2026-05-31 §8's tiering** — in-process, so the gate needs no infrastructure and there is no scratch database to name, collide with, or accidentally drop: @@ -133,15 +161,21 @@ flyway` and `--dialect d1`. **Zero committed migrations** is not a silent pass. `discoverMigrations` returns `[]` for a missing directory (`apply.ts:316-322`), so the run would otherwise succeed having proved nothing. The gate -reports "no committed migrations — nothing to replay" and, at `--strict` against a non-empty -snapshot, fails. - -**A baselined chain is skipped with a reason, not failed.** At `--strict`, a chain that provably -cannot build the snapshot (baseline adoption) reports that it was skipped and why. A gate that -convicts a supported adoption path gets suppressed, and a suppressed gate protects nothing. +reports "no committed migrations — nothing to replay" and, at `--replay-snapshot` against a +non-empty snapshot, fails. + +**`--replay-snapshot` does not support baseline-adopted projects, and says so rather than +detecting it.** An earlier draft had it "skip with a reason". That cannot be implemented: the only +candidate signal is `BASELINE_NAME` / `recordBaseline` (`ledger.ts:205-227`), which has **no +production caller** — it appears only in `ledger.ts` and the package barrel — and even if it were +written, it lands in the *target* database's ledger while this gate runs against a fresh in-process +database that has no ledger at all. So the limitation is documented, not auto-detected: a project +adopted via `baseline --from-db` uses `--replay` and not `--replay-snapshot`, and the failure +message for a snapshot mismatch names baseline adoption as the first thing to rule out. **Exit codes** follow `verify`'s convention (`Math.max` at `verify.ts:239`): a chain that fails to -apply, or a `--strict` mismatch, is **drift → 1**; an engine that cannot start is **operational → 2**. +apply, or a `--replay-snapshot` mismatch, is **drift → 1**; an engine that cannot start is +**operational → 2**. ### 3.3 `CREATE SCHEMA IF NOT EXISTS` for `@schema` projects @@ -159,7 +193,8 @@ was never managed by this toolchain — the `drop-table` in #313 is exactly this time, naming the object, unless the drop is explicitly allowed by a new `--allow drop-unmanaged` token. -The population problem that sinks the strict gate does not apply here, and for a pleasing reason: +The population problem that sinks `--replay-snapshot` for baselined projects does not apply here, +and for a pleasing reason: both brownfield mechanisms *add* to the snapshot. A baselined project's snapshot contains the foreign table, so the guard reads it as managed and does not fire; a scoped project's snapshot carries out-of-scope entries forward for the same reason. The guard fires precisely when nothing @@ -208,11 +243,12 @@ gets suppressed**, and this one would otherwise go red on exactly the population ## 6. Non-goals -- Column and constraint `IF EXISTS` (§3.1's dialect reason). +- `drop-column` `IF EXISTS` (§3.1's one genuine dialect limit; §3.4's guard covers it instead). +- `sqlite.ts:275`'s pre-existing `IF EXISTS` on a `create-view` down (§3.1). - Repairing already-applied chains automatically (§4). - Any cross-port work: `migrate` is TS-owned. -- Making `--strict` pass for baselined projects. It cannot, by construction, and §3.2 skips them - with a reason instead. +- Making `--replay-snapshot` pass for baselined projects, or auto-detecting them. Neither is + possible with the signals that exist (§3.2); the limitation is documented instead. ## 7. Verified by From 5443fd918ac2156c1a426dc24a984279b921d69e Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 06:47:03 -0400 Subject: [PATCH 04/44] docs(plan): implementation plan for migration chain replayability (#313) Eight tasks over the spec: IF EXISTS on forward drops, CREATE SCHEMA IF NOT EXISTS, the in-process replay engine, scope threading into verifyReplay, the two verify subverbs, the emit-time provenance guard, and docs. Two places deliberately name a file to copy rather than inlining code: the Kysely adapter wiring for PGlite, and the migrate test harness for the guard. Guessing either from memory would put wrong code in a plan an implementer is told to follow verbatim. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-19-migrate-chain-replayability.md | 1087 +++++++++++++++++ 1 file changed, 1087 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md diff --git a/docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md b/docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md new file mode 100644 index 000000000..75375cdc3 --- /dev/null +++ b/docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md @@ -0,0 +1,1087 @@ +# Migration Chain Replayability Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make a committed migration chain provably replayable from an empty database, and stop `meta migrate` writing statements that cannot replay. + +**Architecture:** Three independent layers. The emitter stops writing landmines (`IF EXISTS` on forward drops; `CREATE SCHEMA IF NOT EXISTS` ahead of a non-default schema). A new `meta verify --replay` gate replays the committed chain into an **in-process** database (PGlite for postgres, `:memory:` libsql for sqlite) and asserts it applies; `--replay-snapshot` additionally asserts it reproduces the committed snapshot via the already-built-but-unwired `verifyReplay`. An emit-time provenance guard refuses to drop an object the committed snapshot never contained, so the bad SQL is never authored. + +**Tech Stack:** TypeScript, Bun test runner, Kysely, `@electric-sql/pglite` (new), `@libsql/client` (already present via the sqlite path). + +**Spec:** [`docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md`](../specs/2026-08-19-migrate-chain-replayability-design.md) + +## Global Constraints + +- **Scope is TypeScript only.** Schema migration is TS-owned ([ADR-0015](../../../spec/decisions/ADR-0015-single-shared-migrate-engine.md)). No Java/Kotlin/C#/Python work, no conformance-corpus fan-out. +- **`IF EXISTS` goes on FORWARD drops only.** Down statements stay bare. `rollbackTo` runs `down.sql` and the ledger delete in ONE transaction (`apply/apply.ts:185-189`), so a down that no-ops would still record the rollback as done. +- **Never `instanceof` a metadata node from another package** — use the exported guards (`isMetaObject`, `isWritableSource`, …). Two physical copies of `@metaobjectsdev/metadata` in one process make `instanceof` silently false. +- **No `any`.** Use `unknown` and narrow. A bare `let x;` is an implicit evolving `any`. +- **Never call `own*()` accessors** (ADR-0039) except where surrounding code documents a sanctioned reason. +- **Errors are `ParseError` with a structured `code`** and `codeSource(...)` — never a message prefix. +- **Backward compatibility is absolute:** a project that declares no new flag must emit byte-identical migrations except for the `IF EXISTS` tokens this plan adds, and `SNAPSHOT_FORMAT_VERSION` must remain 3. +- **Run tests scoped:** `cd server/typescript && bun test packages/`. NEVER a bare `bun test` at the repository root — it walks java/python/csharp and takes many minutes. +- **`bun test` does NOT typecheck.** Run `bun run --filter '*' typecheck` from the repository root before every commit and confirm all 18 packages exit 0. +- **Public repository.** No private project names, no absolute home paths, in code, tests, fixtures, or commit messages. +- **Stage explicit paths only.** Never `git add -A` — other worktrees share this repository. + +--- + +## File Structure + +**Modified — `server/typescript/packages/migrate-ts/src/`** +- `emit/postgres.ts` — `IF EXISTS` on forward drops; `CREATE SCHEMA IF NOT EXISTS` emission +- `emit/sqlite.ts` — `IF EXISTS` on forward drops +- `verify/replay.ts` — thread scope inputs into the snapshot comparison +- `index.ts` — export the new replay-engine surface + +**New — `server/typescript/packages/migrate-ts/src/`** +- `verify/replay-engine.ts` — provision an in-process database (PGlite / `:memory:` libsql), hand back a Kysely instance and a disposer. One responsibility: engine lifecycle. No replay logic, no comparison. + +**Modified — `server/typescript/packages/cli/src/`** +- `lib/args.ts` — `--replay` / `--replay-snapshot` verify flags; `drop-unmanaged` allow token +- `commands/verify.ts` — the replay gate +- `commands/migrate.ts` — the emit-time provenance guard + +**New — tests** +- `packages/migrate-ts/test/emit-drop-if-exists.test.ts` +- `packages/migrate-ts/test/emit-postgres-create-schema.test.ts` +- `packages/migrate-ts/test/unit/replay-engine.test.ts` +- `packages/migrate-ts/test/integrity/replay-scoped.test.ts` +- `packages/migrate-ts/test/integrity/replay-from-empty.test.ts` — the #313 regression +- `packages/cli/test/verify-replay.test.ts` +- `packages/cli/test/migrate-drop-unmanaged.test.ts` + +--- + +## Task 1: `IF EXISTS` on forward drops, both dialects + +**Files:** +- Modify: `server/typescript/packages/migrate-ts/src/emit/postgres.ts` (`renderUp`, `renderDropView`) +- Modify: `server/typescript/packages/migrate-ts/src/emit/sqlite.ts` (`renderUpNative`) +- Test: `server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts` (create) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: no new exports. Behaviour change only. + +**Context the implementer needs:** `renderUp`/`renderUpNative` are the FORWARD renderers. +`renderDown`/`renderDownNative` are the down renderers and are **out of scope** — see the Global +Constraint. Two forward drops stay bare **deliberately**: `sqlite.ts:197` (inside the +recreate-and-copy rebuild) and `emit/d1-cascade.ts:126`, because each drops a table the same recipe +just `INSERT…SELECT`ed from, where `IF EXISTS` would convert a caught corruption into a silent one. + +`emit/d1.ts:21` renders through `renderSqlite`, so the sqlite edits also change D1's committed +migrations. That is accepted and expected. + +- [ ] **Step 1: Write the failing test** + +```ts +// server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts +import { describe, test, expect } from "bun:test"; +import { renderPostgres } from "../src/emit/postgres.js"; +import { renderSqlite } from "../src/emit/sqlite.js"; +import type { Change } from "../src/types.js"; + +const TABLE = { name: "gone", columns: [], indexes: [], foreignKeys: [], checks: [], primaryKey: [] }; + +describe("forward drops tolerate an absent object", () => { + test("postgres drop-table", () => { + const { up } = renderPostgres([{ kind: "drop-table", table: "gone" } as Change]); + expect(up).toContain('DROP TABLE IF EXISTS "gone";'); + }); + + test("postgres drop-view", () => { + const { up } = renderPostgres([{ kind: "drop-view", view: "v_gone" } as Change]); + expect(up).toContain("DROP VIEW IF EXISTS"); + }); + + test("postgres drop-index, plain", () => { + const { up } = renderPostgres([{ kind: "drop-index", table: "t", index: "idx_gone" } as Change]); + expect(up).toContain('DROP INDEX IF EXISTS "idx_gone";'); + }); + + test("postgres drop-index, constraint-backed", () => { + const { up } = renderPostgres([ + { kind: "drop-index", table: "t", index: "uq_gone", restore: { constraint: "unique" } } as Change, + ]); + expect(up).toContain('DROP CONSTRAINT IF EXISTS "uq_gone";'); + }); + + test("postgres drop-fk", () => { + const { up } = renderPostgres([{ kind: "drop-fk", table: "t", fk: "fk_gone" } as Change]); + expect(up).toContain('DROP CONSTRAINT IF EXISTS "fk_gone";'); + }); + + test("sqlite drop-table", () => { + const { up } = renderSqlite([{ kind: "drop-table", table: "gone" } as Change], undefined, undefined); + expect(up).toContain('DROP TABLE IF EXISTS "gone";'); + }); + + test("sqlite drop-index", () => { + const { up } = renderSqlite([{ kind: "drop-index", table: "t", index: "idx_gone" } as Change], undefined, undefined); + expect(up).toContain('DROP INDEX IF EXISTS "idx_gone";'); + }); +}); + +describe("down statements stay bare — a rollback must fail loudly", () => { + test("postgres create-table down", () => { + const { down } = renderPostgres([{ kind: "create-table", table: TABLE } as Change]); + expect(down).toContain('DROP TABLE "gone";'); + expect(down).not.toContain("DROP TABLE IF EXISTS"); + }); + + test("sqlite create-table down", () => { + const { down } = renderSqlite([{ kind: "create-table", table: TABLE } as Change], undefined, undefined); + expect(down).toContain('DROP TABLE "gone";'); + expect(down).not.toContain("DROP TABLE IF EXISTS"); + }); +}); +``` + +**If a `Change` literal above does not typecheck**, widen it to match the real discriminated union +in `src/types.ts` rather than casting away the error — the `as Change` casts are there to keep the +fixtures short, not to hide a shape mismatch. If `renderSqlite`'s signature differs from +`(changes, expectedSchema, actualMeta)`, match the real one. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-drop-if-exists.test.ts` +Expected: the seven "forward drops" tests FAIL (no `IF EXISTS` in the output). The two "down +statements stay bare" tests PASS already — they pin behaviour this task must not change. + +- [ ] **Step 3: Implement — postgres forward drops** + +In `emit/postgres.ts`, inside `renderUp`: + +```ts + case "drop-table": return `DROP TABLE IF EXISTS ${quoteQualified(c.table, c.schema)};`; +``` + +```ts + case "drop-index": + return c.restore?.constraint !== undefined + ? `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.index)};` + : `DROP INDEX IF EXISTS ${quoteIndexQualified(c.index, c.schema)};`; +``` + +```ts + case "drop-fk": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.fk)};`; +``` + +```ts + case "drop-check": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.check)};`; +``` + +`drop-check` is currently **unreachable** — the comment above that arm records that CHECKs are +create-time-only and the diff never produces this change. Guard it anyway for consistency, and do +not add a test asserting it fires, because it cannot. + +In `renderDropView` (around `:375` and `:388`), change both the plain and the CASCADE form: + +```ts + if (dependents.length === 0) return `DROP VIEW IF EXISTS ${qualified};`; +``` + +```ts + `DROP VIEW IF EXISTS ${qualified} CASCADE;`, +``` + +**Do NOT touch `renderRestoreView` (around `:431`).** It is reached only from `postgres.ts:178` +and `:179`, both inside `renderDown`. + +- [ ] **Step 4: Implement — sqlite forward drops** + +In `emit/sqlite.ts`, inside `renderUpNative`: + +```ts + case "drop-table": return `DROP TABLE IF EXISTS ${quote(c.table)};`; +``` + +```ts + case "drop-index": return `DROP INDEX IF EXISTS ${quote(c.index)};`; +``` + +Leave `sqlite.ts:197` (the rebuild `DROP TABLE`) and `renderDownNative` untouched. + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-drop-if-exists.test.ts` +Expected: PASS, all nine. + +- [ ] **Step 6: Run the affected suites** + +Run: `cd server/typescript && bun test packages/migrate-ts && bun test packages/cli` +Expected: PASS. Existing assertions on exact `DROP TABLE "x";` strings will need updating to the +`IF EXISTS` form — that is expected churn, not a regression. **Read each one before changing it**: +if an assertion is on a DOWN statement, the correct fix is to leave the assertion alone and check +you did not edit a down renderer. + +- [ ] **Step 7: Typecheck** + +Run: `bun run --filter '*' typecheck` (from the repository root) +Expected: all 18 packages exit 0. + +- [ ] **Step 8: Commit** + +```bash +git add server/typescript/packages/migrate-ts/src/emit/postgres.ts \ + server/typescript/packages/migrate-ts/src/emit/sqlite.ts \ + server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts +git commit -m "fix(migrate): forward drops tolerate an absent object so a chain can replay" +``` + +--- + +## Task 2: `CREATE SCHEMA IF NOT EXISTS` for non-default schemas + +**Files:** +- Modify: `server/typescript/packages/migrate-ts/src/emit/postgres.ts` (`renderPostgres`) +- Test: `server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts` (create) + +**Interfaces:** +- Consumes: nothing from Task 1. +- Produces: no new exports. + +**Context:** `CREATE SCHEMA` is emitted nowhere in `migrate-ts/src` or `cli/src` today except the +ledger's own (`apply/ledger.ts:126`). A chain containing `CREATE TABLE "reporting"."x"` therefore +cannot apply to a virgin database — the schema does not exist. SQLite has no schema namespacing +(`emit-sqlite-schema-rejected.test.ts` pins that a schema is rejected there), so this is +Postgres-only and is NOT a dialect split. + +- [ ] **Step 1: Write the failing test** + +```ts +// server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts +import { describe, test, expect } from "bun:test"; +import { renderPostgres } from "../src/emit/postgres.js"; +import type { Change } from "../src/types.js"; + +const t = (name: string, schema?: string) => ({ + name, schema, columns: [], indexes: [], foreignKeys: [], checks: [], primaryKey: [], +}); + +describe("a chain that creates a non-default schema's table creates the schema first", () => { + test("emits CREATE SCHEMA IF NOT EXISTS before the table", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x", "reporting") } as Change]); + expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "reporting";'); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "reporting";')) + .toBeLessThan(up.indexOf("CREATE TABLE")); + }); + + test("emits it once for two tables in the same schema", () => { + const { up } = renderPostgres([ + { kind: "create-table", table: t("x", "reporting") } as Change, + { kind: "create-table", table: t("y", "reporting") } as Change, + ]); + expect(up.match(/CREATE SCHEMA IF NOT EXISTS "reporting";/g)).toHaveLength(1); + }); + + test("emits nothing for the default schema", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x") } as Change]); + expect(up).not.toContain("CREATE SCHEMA"); + }); + + test("the down does NOT drop the schema", () => { + const { down } = renderPostgres([{ kind: "create-table", table: t("x", "reporting") } as Change]); + expect(down).not.toContain("DROP SCHEMA"); + }); +}); +``` + +The last case is a real decision, not filler: dropping a schema on rollback would destroy objects +this tool does not own and cannot restore. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-postgres-create-schema.test.ts` +Expected: FAIL — the first two cases find no `CREATE SCHEMA`. The last two PASS already. + +- [ ] **Step 3: Implement** + +In `renderPostgres`, after `const sorted = …` and before the render loop, collect the distinct +non-default schemas the forward pass will create objects in, and prepend one statement each: + +```ts + // A chain must be appliable to a VIRGIN database (#313). CREATE TABLE "s"."x" + // fails there unless the schema exists, and no migration has ever created one. + // IF NOT EXISTS because a later migration in the same chain, or an operator, + // may have created it already. Deliberately NOT dropped in `down`: the schema + // may hold objects this tool does not own and cannot restore. + const createdSchemas = new Set(); + for (const c of sorted) { + if (c.kind !== "create-table") continue; + const s = c.table.schema; + if (s !== undefined && s !== DEFAULT_DB_SCHEMA_POSTGRES) createdSchemas.add(s); + } + const schemaStmts = [...createdSchemas].sort().map((s) => `CREATE SCHEMA IF NOT EXISTS ${quote(s)};`); +``` + +then emit `schemaStmts` ahead of `upStmts` in the returned `up`: + +```ts + up: [...schemaStmts, ...upStmts].join("\n\n"), +``` + +Import `DEFAULT_DB_SCHEMA_POSTGRES` from wherever the file already resolves the default schema +name — `diff/index.ts:152` uses it, so it is exported from a shared module. If `renderPostgres` +already has a local notion of the default schema, use that instead of adding a second one. + +Sorting `createdSchemas` keeps output deterministic, which the snapshot and golden tests rely on. + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-postgres-create-schema.test.ts` +Expected: PASS, all four. + +- [ ] **Step 5: Run the affected suites** + +Run: `cd server/typescript && bun test packages/migrate-ts` +Expected: PASS. `emit-postgres-schema-namespacing.test.ts` is the file most likely to need updating; +read its assertions before changing them. + +- [ ] **Step 6: Typecheck and commit** + +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/migrate-ts/src/emit/postgres.ts \ + server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts +git commit -m "fix(migrate): a chain creates the schema it needs, so it applies to a virgin database" +``` + +--- + +## Task 3: The in-process replay engine + +**Files:** +- Create: `server/typescript/packages/migrate-ts/src/verify/replay-engine.ts` +- Modify: `server/typescript/packages/migrate-ts/src/index.ts` (export it) +- Modify: `server/typescript/packages/migrate-ts/package.json` (add `@electric-sql/pglite`) +- Test: `server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts` (create) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: + ```ts + export interface ReplayEngine { + db: Kysely>; + dispose: () => Promise; + } + export function openReplayEngine(dialect: "postgres" | "sqlite"): Promise; + ``` + Tasks 5 and 6 call `openReplayEngine` and must `await engine.dispose()` in a `finally`. + +**Context:** This is why the design needs no scratch database on the user's server: both engines run +in-process. PGlite is real Postgres compiled to WASM. `dispose()` must be safe to call twice, so a +caller can dispose in a `finally` after an early return. + +- [ ] **Step 1: Add the dependency** + +```bash +cd server/typescript/packages/migrate-ts && bun add @electric-sql/pglite +``` + +Then confirm it landed in `dependencies` (not `devDependencies`) in +`server/typescript/packages/migrate-ts/package.json` — the CLI imports this at runtime. + +- [ ] **Step 2: Write the failing test** + +```ts +// server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts +import { describe, test, expect } from "bun:test"; +import { sql } from "kysely"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; + +describe("openReplayEngine", () => { + test("sqlite: gives an empty, usable database", async () => { + const engine = await openReplayEngine("sqlite"); + try { + await sql`CREATE TABLE t (id integer primary key)`.execute(engine.db); + await sql`INSERT INTO t (id) VALUES (1)`.execute(engine.db); + const rows = await sql<{ id: number }>`SELECT id FROM t`.execute(engine.db); + expect(rows.rows).toHaveLength(1); + } finally { + await engine.dispose(); + } + }); + + test("postgres: gives an empty, usable database with real PG DDL", async () => { + const engine = await openReplayEngine("postgres"); + try { + // Schema namespacing + a CHECK — both are things sqlite cannot express, + // so this proves the postgres engine is really Postgres. + await sql`CREATE SCHEMA IF NOT EXISTS "reporting"`.execute(engine.db); + await sql`CREATE TABLE "reporting"."t" (id integer primary key, n integer CHECK (n > 0))`.execute(engine.db); + const rows = await sql<{ table_name: string }>` + SELECT table_name FROM information_schema.tables WHERE table_schema = 'reporting' + `.execute(engine.db); + expect(rows.rows.map((r) => r.table_name)).toContain("t"); + } finally { + await engine.dispose(); + } + }); + + test("two engines of the same dialect do not share state", async () => { + const a = await openReplayEngine("sqlite"); + const b = await openReplayEngine("sqlite"); + try { + await sql`CREATE TABLE only_in_a (id integer)`.execute(a.db); + const rows = await sql<{ name: string }>`SELECT name FROM sqlite_master WHERE name = 'only_in_a'`.execute(b.db); + expect(rows.rows).toHaveLength(0); + } finally { + await a.dispose(); + await b.dispose(); + } + }); + + test("dispose is idempotent", async () => { + const engine = await openReplayEngine("sqlite"); + await engine.dispose(); + await engine.dispose(); + }); +}); +``` + +- [ ] **Step 3: Run it to verify it fails** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/unit/replay-engine.test.ts` +Expected: FAIL — `openReplayEngine` is not defined. + +- [ ] **Step 4: Implement** + +```ts +// server/typescript/packages/migrate-ts/src/verify/replay-engine.ts +// +// An empty, throwaway database that lives INSIDE this process. +// +// The replay gate has to apply a whole committed chain from nothing. Doing that +// against the user's server would mean CREATE DATABASE — which needs CREATEDB, +// breaks behind a connection pooler, is restricted on managed Postgres, collides +// between parallel CI jobs sharing one server, and puts a DROP DATABASE next to a +// name derived from a real one. None of that is worth it when the engines run +// in-process: PGlite is real Postgres compiled to WASM, and libsql runs sqlite in +// memory. Nothing to provision, nothing to clean up, nothing to drop by mistake. +import { Kysely, PostgresDialect, SqliteDialect } from "kysely"; + +export interface ReplayEngine { + /** An empty database. The caller owns applying migrations into it. */ + db: Kysely>; + /** Release the engine. Safe to call more than once. */ + dispose: () => Promise; +} + +export async function openReplayEngine( + dialect: "postgres" | "sqlite", +): Promise { + if (dialect === "postgres") return openPglite(); + return openMemorySqlite(); +} +``` + +Then implement the two openers against whatever Kysely dialect adapters this repo already uses. +**Read `cli/src/lib/kysely.ts` first** — it is the existing place a `Kysely` is constructed for both +dialects, and this file should mirror its adapter choices rather than inventing new ones. For +sqlite, `:memory:` through the same client that file already uses. For postgres, PGlite exposes a +`pg`-compatible interface; wire it into `PostgresDialect` the same way. + +Make `dispose` idempotent with a `disposed` flag; call `db.destroy()` and then the engine's own +`close()`/`end()` if it has one. + +- [ ] **Step 5: Export it** + +In `server/typescript/packages/migrate-ts/src/index.ts`, beside the existing +`export { verifyReplay } from "./verify/replay.js";`: + +```ts +export { openReplayEngine, type ReplayEngine } from "./verify/replay-engine.js"; +``` + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/unit/replay-engine.test.ts` +Expected: PASS, all four. + +**If PGlite cannot execute the postgres case**, STOP and report it. The spec's engine tiering rests +on PGlite being real Postgres; if it is not sufficient, that is a design question, not something to +work around by weakening the test. + +- [ ] **Step 7: Typecheck and commit** + +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/migrate-ts/src/verify/replay-engine.ts \ + server/typescript/packages/migrate-ts/src/index.ts \ + server/typescript/packages/migrate-ts/package.json \ + server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts \ + server/typescript/bun.lock +git commit -m "feat(migrate): an in-process replay engine, so the gate provisions nothing" +``` + +--- + +## Task 4: Thread scope inputs into `verifyReplay` + +**Files:** +- Modify: `server/typescript/packages/migrate-ts/src/verify/replay.ts` +- Test: `server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts` (create) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: `VerifyReplayArgs` gains one optional field: + ```ts + /** Out-of-scope / @unmanaged names to exclude from BOTH sides, as `scopedDiffInputs` produces. */ + governed?: GovernedScope; + ``` + Task 6 passes it. + +**Context:** `verifyReplay` (`verify/replay.ts:31`) compares a replayed database against the +committed snapshot. A project declaring `migrate.scope` writes the *other* owner's tables into that +snapshot on purpose (`carryForwardOutOfScope`, `scope.ts:93`), and the chain never creates them — so +today the comparison reports them as missing. `excludeFromSnapshot` (`scope.ts:130`) exists for +exactly this and is already used by the committed-snapshot gate at `verify.ts:659`; this task +threads it here too. + +- [ ] **Step 1: Write the failing test** + +```ts +// server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts +import { describe, test, expect } from "bun:test"; +import { verifyReplay } from "../../src/verify/replay.js"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { SchemaSnapshot } from "../../src/types.js"; + +function chainWith(upSql: string): string { + const dir = mkdtempSync(join(tmpdir(), "replay-scoped-")); + mkdirSync(join(dir, "20260101000000-init"), { recursive: true }); + writeFileSync(join(dir, "20260101000000-init", "up.sql"), upSql, "utf8"); + writeFileSync(join(dir, "20260101000000-init", "down.sql"), "DROP TABLE mine;", "utf8"); + return dir; +} + +const SNAPSHOT: SchemaSnapshot = { + tables: [ + { name: "mine", columns: [{ name: "id", sqlType: { kind: "int", bits: 64 } as never, nullable: false }], indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"] }, + { name: "theirs", columns: [{ name: "id", sqlType: { kind: "int", bits: 64 } as never, nullable: false }], indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"] }, + ], + views: [], +}; + +describe("verifyReplay honours scope", () => { + test("an out-of-scope table in the snapshot is not reported as missing", async () => { + const dir = chainWith("CREATE TABLE mine (id integer primary key);"); + const engine = await openReplayEngine("sqlite"); + try { + const result = await verifyReplay({ + db: engine.db, + dialect: "sqlite", + migrationsDir: dir, + snapshot: SNAPSHOT, + governed: { outOfScope: ["theirs"] } as never, + }); + expect(result.ok).toBe(true); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + test("without `governed`, the same case reports drift — the control", async () => { + const dir = chainWith("CREATE TABLE mine (id integer primary key);"); + const engine = await openReplayEngine("sqlite"); + try { + const result = await verifyReplay({ + db: engine.db, dialect: "sqlite", migrationsDir: dir, snapshot: SNAPSHOT, + }); + expect(result.ok).toBe(false); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); +``` + +The control case is what makes the first test non-vacuous: it proves the difference comes from +`governed` and not from the fixture being trivially green. **Adjust the `SchemaSnapshot` literal and +the `GovernedScope` shape to the real types** — read `src/types.ts` and `src/scope.ts` — rather than +leaving the `as never` casts in the committed test. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/integrity/replay-scoped.test.ts` +Expected: the first test FAILS (`governed` is not accepted / not honoured); the control PASSES. + +- [ ] **Step 3: Implement** + +In `verify/replay.ts`, add the optional field to `VerifyReplayArgs` and apply +`excludeFromSnapshot` to the snapshot before comparing: + +```ts + const expected = args.governed !== undefined + ? excludeFromSnapshot(args.snapshot, args.governed) + : args.snapshot; + const classification = await driftAgainstSnapshot(expected, actual, args.dialect); +``` + +Import `excludeFromSnapshot` and the `GovernedScope` type from `../scope.js`. Do not change the +signature's required fields — an existing caller passing no `governed` must behave exactly as before. + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/integrity/replay-scoped.test.ts` +Expected: PASS, both. + +- [ ] **Step 5: Run the suite, typecheck, commit** + +Run: `cd server/typescript && bun test packages/migrate-ts` — PASS. +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/migrate-ts/src/verify/replay.ts \ + server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts +git commit -m "fix(migrate): verifyReplay honours migrate.scope on both sides" +``` + +--- + +## Task 5: `meta verify --replay` — the chain applies from empty + +**Files:** +- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`VerifyFlags`, `parseVerifyArgs`) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` +- Test: `server/typescript/packages/migrate-ts/test/integrity/replay-from-empty.test.ts` (create) +- Test: `server/typescript/packages/cli/test/verify-replay.test.ts` (create) + +**Interfaces:** +- Consumes: `openReplayEngine` (Task 3). +- Produces: the `--replay` flag on `VerifyFlags`; Task 6 adds `--replay-snapshot` beside it. + +**Context:** `verify` composes gates and returns `Math.max(...)` of their exit codes +(`verify.ts:239`). `anyExplicit` (`args.ts:290`) decides whether the bare-`verify` default fires — +`--replay` must be included in it, or passing `--replay` alone would also silently run the template +gate. Refuse `--migration-format flyway` and `--dialect d1`, mirroring `apply-pending` +(`migrate.ts:419-426`, `:449-457`). + +- [ ] **Step 1: Write the failing regression test — the #313 case** + +```ts +// server/typescript/packages/migrate-ts/test/integrity/replay-from-empty.test.ts +import { describe, test, expect } from "bun:test"; +import { applyPending } from "../../src/apply/apply.js"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +describe("a committed chain must apply to an empty database (#313)", () => { + test("a bare DROP TABLE for an object the chain never creates fails the replay", async () => { + const dir = mkdtempSync(join(tmpdir(), "replay-empty-")); + mkdirSync(join(dir, "20260101000000-init"), { recursive: true }); + // Exactly the reported shape: another tool owned `theirs`, so the diff proposed + // dropping it, and no migration in the chain ever created it. + writeFileSync(join(dir, "20260101000000-init", "up.sql"), + 'CREATE TABLE "mine" (id integer primary key);\nDROP TABLE "theirs";', "utf8"); + writeFileSync(join(dir, "20260101000000-init", "down.sql"), 'DROP TABLE "mine";', "utf8"); + + const engine = await openReplayEngine("sqlite"); + try { + await expect(applyPending(engine.db, dir, { dryRun: false, dialect: "sqlite" })) + .rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + test("the same chain with IF EXISTS applies cleanly", async () => { + const dir = mkdtempSync(join(tmpdir(), "replay-empty-ok-")); + mkdirSync(join(dir, "20260101000000-init"), { recursive: true }); + writeFileSync(join(dir, "20260101000000-init", "up.sql"), + 'CREATE TABLE "mine" (id integer primary key);\nDROP TABLE IF EXISTS "theirs";', "utf8"); + writeFileSync(join(dir, "20260101000000-init", "down.sql"), 'DROP TABLE "mine";', "utf8"); + + const engine = await openReplayEngine("sqlite"); + try { + const result = await applyPending(engine.db, dir, { dryRun: false, dialect: "sqlite" }); + expect(result.applied).toEqual(["20260101000000-init"]); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); +``` + +This runs through `applyPending`, **not** through `emit()`. Every prior defect in this area +(#226/#241, #243, #255, #285, and 0.21.4's `BEGIN TRANSACTION` finding) shared one shape — SQL proven +statement-by-statement and never proven through the tool that applies it. `applyPending` rewrites +statements via `prepareForRunnerTransaction` before executing them, so an emit-level assertion +cannot see this class of bug. + +Adjust `result.applied` to whatever `ApplyPendingResult` actually names its applied list. + +- [ ] **Step 2: Run it to verify the shape is real** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/integrity/replay-from-empty.test.ts` +Expected: BOTH PASS immediately — this test characterises `applyPending`, which already behaves this +way. That is the point: it pins the mechanism the gate depends on. If the first case does NOT throw, +stop and report, because the gate cannot work. + +- [ ] **Step 3: Add the flag** + +In `cli/src/lib/args.ts`, add to `VerifyFlags`: + +```ts + /** Replay the committed migration chain into an in-process database and assert it applies. */ + replay: boolean; +``` + +add to the `parseArgs` options: + +```ts + replay: { type: "boolean", default: false }, +``` + +include it in the parsed object (`replay: !!values.replay`), and add it to `anyExplicit`: + +```ts + const anyExplicit = templates || codegen || values.db !== undefined || dialect === "d1" || !!values.replay; +``` + +- [ ] **Step 4: Write the CLI test** + +```ts +// server/typescript/packages/cli/test/verify-replay.test.ts +import { describe, test, expect } from "bun:test"; +import { parseVerifyArgs } from "../src/lib/args.js"; + +describe("verify --replay flag", () => { + test("is parsed", () => { + expect(parseVerifyArgs(["--replay"]).replay).toBe(true); + }); + + test("counts as an explicit subverb, so it does not also run the template gate", () => { + expect(parseVerifyArgs(["--replay"]).anyExplicit).toBe(true); + }); + + test("defaults off", () => { + expect(parseVerifyArgs([]).replay).toBe(false); + }); +}); +``` + +- [ ] **Step 5: Implement the gate** + +In `cli/src/commands/verify.ts`, add a `runReplayVerify()` beside the existing gates and include it +in the `Math.max`: + +```ts + const replayExit = flags.replay ? await runReplayVerify() : 0; + … + return Math.max(templateExit, schemaExit, codegenExit, requirementExit, replayExit); +``` + +`runReplayVerify` must: + +1. Refuse `--migration-format flyway` and `--dialect d1` with `log.error` and **exit 2**, matching + `apply-pending`'s wording at `migrate.ts:419-426` and `:449-457`. +2. Resolve the migrations directory the same way `migrate apply-pending` does — do NOT re-derive it; + read how `migrate.ts` computes it and use the same helper. +3. Report and return 0 when the directory holds no migrations, with the exact text + `meta verify --replay: no committed migrations — nothing to replay`. `discoverMigrations` + returns `[]` for a missing directory (`apply.ts:316-322`), so silence here would be a vacuous pass. +4. `openReplayEngine(dialect)`, `applyPending(engine.db, dir, { dryRun: false, dialect })`, and + `await engine.dispose()` in a `finally`. +5. On an apply failure, print the failing statement and the remediation — an already-applied chain + cannot be repaired by hand-editing (`apply/apply.ts:88-99` makes migrations checksum-immutable), + so the message must name the compensating-migration path: + `meta verify --replay: the committed chain does not apply to an empty database. Applied migrations are immutable, so fix this with a NEW migration that creates the missing object — not by editing a committed up.sql.` + Return **1** (drift), not 2. +6. Return **2** only when the engine itself cannot start. + +- [ ] **Step 6: Run the tests** + +Run: `cd server/typescript && bun test packages/cli/test/verify-replay.test.ts && bun test packages/migrate-ts/test/integrity` +Expected: PASS. + +- [ ] **Step 7: Run the suites, typecheck, commit** + +Run: `cd server/typescript && bun test packages/cli && bun test packages/migrate-ts` — PASS. +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/cli/src/lib/args.ts \ + server/typescript/packages/cli/src/commands/verify.ts \ + server/typescript/packages/cli/test/verify-replay.test.ts \ + server/typescript/packages/migrate-ts/test/integrity/replay-from-empty.test.ts +git commit -m "feat(cli): meta verify --replay asserts the committed chain applies from empty" +``` + +--- + +## Task 6: `meta verify --replay-snapshot` — the chain reproduces the snapshot + +**Files:** +- Modify: `server/typescript/packages/cli/src/lib/args.ts` +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` +- Test: `server/typescript/packages/cli/test/verify-replay.test.ts` (extend) + +**Interfaces:** +- Consumes: `openReplayEngine` (Task 3), `verifyReplay` with `governed` (Task 4), the `runReplayVerify` structure (Task 5). +- Produces: nothing later tasks depend on. + +**Context:** This is a **separate subverb, not `--strict`**. `verify` already owns a `--lax` flag on +a different axis (ADR-0023 attribute strictness, `args.ts:244`), and `--strict` beside it would read +as that flag's opposite rather than as a replay depth. + +**This tier cannot pass for baseline-adopted projects and does not try to detect them.** The only +candidate signal, `BASELINE_NAME`/`recordBaseline` (`ledger.ts:205-227`), has no production caller +and would live in the target database's ledger while this gate runs against a fresh in-process +database with no ledger at all. The limitation is documented in Task 8, and the failure message +names it. + +- [ ] **Step 1: Write the failing test** + +Append to `packages/cli/test/verify-replay.test.ts`: + +```ts +describe("verify --replay-snapshot flag", () => { + test("is parsed", () => { + expect(parseVerifyArgs(["--replay-snapshot"]).replaySnapshot).toBe(true); + }); + + test("counts as an explicit subverb", () => { + expect(parseVerifyArgs(["--replay-snapshot"]).anyExplicit).toBe(true); + }); + + test("does not collide with --lax, which is a different axis", () => { + const f = parseVerifyArgs(["--replay-snapshot", "--lax"]); + expect(f.replaySnapshot).toBe(true); + expect(f.lax).toBe(true); + }); +}); +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/typescript && bun test packages/cli/test/verify-replay.test.ts` +Expected: FAIL — `--replay-snapshot` is not a known option (`strict: true` in `parseArgs`). + +- [ ] **Step 3: Implement the flag** + +Same three places as Task 5: the `VerifyFlags` field (`replaySnapshot: boolean`), the `parseArgs` +option (`"replay-snapshot": { type: "boolean", default: false }`), the parsed object +(`replaySnapshot: !!values["replay-snapshot"]`), and `anyExplicit`. + +- [ ] **Step 4: Implement the tier** + +Extend `runReplayVerify` so that when `flags.replaySnapshot` is set it additionally: + +1. Loads the committed snapshot the same way the existing committed-snapshot gate does — read + `verify.ts` around `:628-660` and reuse that path, do not re-implement snapshot loading. +2. Calls `verifyReplay({ db: engine.db, dialect, migrationsDir, snapshot, governed })`, where + `governed` is what `scopedDiffInputs`/`excludeFromSnapshot` already produce at `verify.ts:659`. +3. On `ok === false`, reports the drift and returns **1**, with a message that names baseline + adoption as the first thing to rule out: + `meta verify --replay-snapshot: the replayed chain does not reproduce the committed snapshot. If this project was adopted with 'migrate baseline --from-db', its chain does not build the schema and this tier does not apply — use --replay instead.` + +Both tiers share one engine and one `applyPending` call: `--replay-snapshot` implies `--replay`'s +work, so do not open two engines or replay twice. + +- [ ] **Step 5: Run the tests** + +Run: `cd server/typescript && bun test packages/cli/test/verify-replay.test.ts` +Expected: PASS, all six. + +- [ ] **Step 6: Run the suites, typecheck, commit** + +Run: `cd server/typescript && bun test packages/cli && bun test packages/migrate-ts` — PASS. +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/cli/src/lib/args.ts \ + server/typescript/packages/cli/src/commands/verify.ts \ + server/typescript/packages/cli/test/verify-replay.test.ts +git commit -m "feat(cli): meta verify --replay-snapshot asserts the chain reproduces the snapshot" +``` + +--- + +## Task 7: Emit-time provenance guard + +**Files:** +- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`ALLOW_TOKENS`) +- Modify: `server/typescript/packages/cli/src/commands/migrate.ts` +- Test: `server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts` (create) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: the `drop-unmanaged` allow token. + +**Context:** Tasks 1–6 make a bad chain survivable and detectable. This stops it being written. The +live path (`migrate.ts:607-620`) diffs metadata against introspection and **never consults the +committed snapshot**, which is why an object no snapshot ever contained gets proposed for a drop. +`drift/classify.ts:6-9` already states the doctrine: objects present in the DB but not the snapshot +"must never be treated as actionable drift or auto-dropped". + +The guard does not false-fire on the brownfield classes, and the reason is that both of them *add* +to the snapshot: a `baseline --from-db` snapshot contains the foreign table, and a scoped project +carries out-of-scope entries forward into it. The guard fires precisely when nothing ever claimed +the object. + +- [ ] **Step 1: Write the failing test** + +```ts +// server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts +import { describe, test, expect } from "bun:test"; +import { ALLOW_TOKENS } from "../src/lib/args.js"; + +describe("drop-unmanaged allow token", () => { + test("is a recognised allow token", () => { + expect(ALLOW_TOKENS).toContain("drop-unmanaged"); + }); +}); +``` + +Then add the behavioural test. It must drive the real `migrate` path with a snapshot that does NOT +contain the table being dropped, and assert the run refuses. **Model it on the existing +`packages/cli/test/migrate-scope.test.ts`**, which already builds a config + snapshot + change set +for this command — read it first and follow its harness rather than inventing one. + +The two cases: +1. A drop proposed for a table absent from the committed snapshot ⇒ refused, exit 2, message names + the object and `--allow drop-unmanaged`. +2. The same run with `--allow drop-unmanaged` ⇒ proceeds. + +And one non-false-fire case: a drop for a table that IS in the snapshot ⇒ proceeds without the flag. + +- [ ] **Step 2: Run to verify it fails** + +Run: `cd server/typescript && bun test packages/cli/test/migrate-drop-unmanaged.test.ts` +Expected: FAIL — the token is not in `ALLOW_TOKENS`. + +- [ ] **Step 3: Add the token** + +In `cli/src/lib/args.ts`, add to `ALLOW_TOKENS` with a comment in the style of its neighbours: + +```ts + // drop-unmanaged permits dropping an object the COMMITTED SNAPSHOT never + // contained — i.e. one this toolchain never managed. Without it such a drop is + // refused at generation time, because it produces a migration that cannot replay + // against a database where that object never existed (#313). + "drop-unmanaged", +``` + +- [ ] **Step 4: Implement the guard** + +In `migrate.ts`, after the diff produces its change list and BEFORE the migration is written, +collect every `drop-table` / `drop-view` whose name is absent from the committed snapshot, and if +that set is non-empty and `allow.dropUnmanaged` is not set, `log.error` naming each object and +return 2. + +Read how `allow` tokens are converted to the options object (`tokensToAllowOptions`) and follow it. +Use the same qualified-name helper the scope machinery uses (`qualifiedDbName` in +`migrate-ts/src/qualified-name.ts`) so the guard and the snapshot agree on a name's spelling — three +independent sets already have to agree there, and a fourth spelling would silently un-guard objects. + +- [ ] **Step 5: Run the tests** + +Run: `cd server/typescript && bun test packages/cli/test/migrate-drop-unmanaged.test.ts` +Expected: PASS. + +- [ ] **Step 6: Run the suites, typecheck, commit** + +Run: `cd server/typescript && bun test packages/cli && bun test packages/migrate-ts` — PASS. +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/cli/src/lib/args.ts \ + server/typescript/packages/cli/src/commands/migrate.ts \ + server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts +git commit -m "feat(cli): refuse to drop an object the committed snapshot never managed" +``` + +--- + +## Task 8: Documentation and CHANGELOG + +**Files:** +- Modify: `docs/features/migrations-and-drift.md` +- Modify: `server/typescript/packages/cli/src/commands/migrate.ts` (help text) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (help/subverb note) +- Modify: `CHANGELOG.md` + +**Interfaces:** none. + +- [ ] **Step 1: Correct the overclaim** + +`docs/features/migrations-and-drift.md:58` currently says `apply-pending` "is the way to provision a +fresh or CI database". That is true only of a chain that builds the schema. Scope the sentence, and +point at `meta verify --replay` as the way to know your chain is one of those. + +- [ ] **Step 2: Document both tiers** + +In the same file, document `meta verify --replay` and `--replay-snapshot`: what each asserts, that +they run in-process (PGlite / `:memory:` libsql) and provision nothing, that flyway and d1 are +refused, and — stated plainly — that `--replay-snapshot` does not apply to a project adopted with +`migrate baseline --from-db`, because such a chain does not build the schema. + +- [ ] **Step 3: Document the guard** + +Document `--allow drop-unmanaged`: what triggers the refusal, why (a migration that cannot replay), +and that the escape hatch exists for a drop you genuinely intend. + +- [ ] **Step 4: Update the CLI help** + +Add the two subverbs to `verify`'s help and to the one-line note at `verify.ts:127-130` that +advertises the explicit subverbs. Update `migrate --help`'s `apply-pending` line so it no longer +promises fresh-database provisioning unconditionally. + +- [ ] **Step 5: CHANGELOG** + +Add an `## [Unreleased]` entry covering the adopter-visible changes: emitted forward drops now carry +`IF EXISTS`; a chain creating a table in a non-default schema now emits `CREATE SCHEMA IF NOT +EXISTS`; two new verify subverbs; and a new refusal that requires `--allow drop-unmanaged`. The last +is the one that can fail an existing project's `meta migrate`, so it leads. + +- [ ] **Step 6: Leak scan and commit** + +```bash +grep -rniE "/home/|party" docs/features/migrations-and-drift.md CHANGELOG.md && echo LEAK || echo clean +``` + +```bash +git add docs/features/migrations-and-drift.md CHANGELOG.md \ + server/typescript/packages/cli/src/commands/migrate.ts \ + server/typescript/packages/cli/src/commands/verify.ts +git commit -m "docs: replay tiers, the drop-unmanaged refusal, and the provisioning promise" +``` + +--- + +## Self-Review + +**Spec coverage.** §3.1 forward drops → Task 1. §3.1's deliberate exclusions (`postgres.ts:431`, +the rebuild drops, `sqlite.ts:275`) → Task 1 Steps 3–4 and its "down statements stay bare" tests. +§3.2 both tiers, engine, refusals, zero-migrations, exit codes → Tasks 3, 5, 6. §3.2's +`excludeFromSnapshot` threading → Task 4. §3.3 `CREATE SCHEMA` → Task 2. §3.4 provenance guard → +Task 7. §3.5 docs → Task 8. §4 remediation → Task 5 Step 5's message text and Task 8. §5 testing → +each task's own steps, with the `applyPending`-not-`emit()` requirement in Task 5 Step 1. + +**Two things deliberately left to the implementer, both flagged inline rather than guessed:** the +exact Kysely adapter wiring for PGlite (Task 3 Step 4 says to mirror `cli/src/lib/kysely.ts`), and +the `migrate` test harness for the guard (Task 7 Step 1 says to follow `migrate-scope.test.ts`). +Inventing either from memory would put wrong code in the plan, which is worse than naming the file +to copy. + +**Type consistency.** `openReplayEngine(dialect) → ReplayEngine { db, dispose }` is defined in Task +3 and used verbatim in Tasks 4, 5, 6. `VerifyReplayArgs.governed` is defined in Task 4 and consumed +in Task 6. `replay` / `replaySnapshot` are added in Tasks 5 and 6 and both feed `anyExplicit`. +`drop-unmanaged` is added to `ALLOW_TOKENS` in Task 7 and referenced nowhere earlier. + +**Known ordering constraint:** Task 4's test imports `openReplayEngine`, so Task 3 must land first. +Tasks 1, 2 and 7 are independent of the rest and of each other. From b2df7c283ce090eaf5480e472f2dbdde2babb7bd Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 06:58:15 -0400 Subject: [PATCH 05/44] docs(spec): genericize an adopter identifier in the quoted error The reproduction quoted a real table name from the bug report. This repository is public and the rule is to genericize adopter and project identifiers on sight, so the error text now uses a neutral name. The behaviour it documents is unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- .../specs/2026-08-19-migrate-chain-replayability-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md b/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md index aa225ef02..20299cfae 100644 --- a/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md +++ b/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md @@ -14,7 +14,7 @@ then dies: ``` $ meta migrate apply-pending --db postgresql://…/fresh -meta: migrate apply-pending: apply failed: table "arena_season_standing" does not exist +meta: migrate apply-pending: apply failed: table "other_owned_table" does not exist ``` Nothing warns at generation time. The reporter's chain was broken for roughly three months; the From 205643eafd112c7d61427dec3d23922c4abd8c56 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 07:17:39 -0400 Subject: [PATCH 06/44] docs(plan): revise the replayability plan against a 20-finding review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prior plan was reviewed as not ready to execute. Thirteen findings came from that review and seven more from re-verifying it against the code; all are now folded in, and every claim below was read at HEAD rather than remembered. Would have produced wrong code: - `--replay-snapshot` shipped dead. Task 5 gated on `flags.replay` and Task 6 never widened it. The condition is now written once, in Task 5, as `(flags.replay || flags.replaySnapshot)`, and Task 6 adds behavioural tests instead of three flag-parse assertions. - The claim that `drop-check` is unreachable is false — `diff/index.ts:579` and `:592` both push it and two tests assert the emitted statement. The paragraph is deleted, the code comment is corrected as part of the task, and `drop-check` gets a forward-drop assertion the plan previously forbade. - Task 4's fixture could not pass three ways: `{kind:"int"}` is not a `SqlType` (`{kind:"integer";bits:32|64}`), `GovernedScope` names are `.`, and a bare `INTEGER PRIMARY KEY` reports `notnull=0` on sqlite. `excludeFromSnapshot` also returns a `ScopedExpectedSchema`, so the fix takes `.snapshot`. Every `as never` is gone. - `pg-constraint-backed-index-285.test.ts` needs two edits, not one: `:143` goes red and `:142` goes VACUOUSLY green, since `IF EXISTS` stops its negative regex matching for a reason unrelated to what it tests. - A new `--allow` token touches four files, pinned by `allow-tokens-pinned.test.ts`. Ruled: add `dropUnmanaged` to `AllowOptions` rather than a second parallel token path, which is the drift that pin exists to prevent. - The provenance guard had no snapshot and no fail-open rule. The live site is named, the loader mirrors migrate's own spelling, and a missing snapshot fails OPEN. Would have stalled: - PGlite is not `pg`-compatible. The ~20-line kysely pool shim is now in the plan, EXECUTED against 0.3.16 and 0.5.5: DDL, schemas, CHECK, transactions, rollback, advisory locks, and the reporter's own error. `:memory:` libsql is verified working and isolated between instances. - The engine's home is settled: `migrate-ts` (cli depends on it, so cli would be a cycle), with PGlite as an OPTIONAL peer — 22 MB of WASM must not reach every `meta gen` adopter — plus a `build:binary` `--external`. - The dialect precedence without `--db` is stated, and the comment that forbids reading migrate's config is amended rather than contradicted. - `governed` is derived offline via `scopeExpectedSchema`, and only for a scoped project, so an unscoped comparison is unchanged. - "Do not replay twice" is restated as what it is: the second `applyPending` is a ledger no-op, not an API change. - The leak scan uses the project hook, not an invented pattern. Also: Task 1 now fixes its own churn so no later task inherits a red suite; Task 2 collects from `create-view` too, since the spec says "the first object"; zero-migrations is detected from `ApplyPendingResult.pending` because `discoverMigrations` is module-private; and the task DAG is explicit. The missing test that mattered most now exists: the RED-first regression runs `emit` -> `writeMigration` -> `applyPending`, so it is red before the emitter fix rather than green regardless of it. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-19-migrate-chain-replayability.md | 1255 +++++++++++++---- 1 file changed, 984 insertions(+), 271 deletions(-) diff --git a/docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md b/docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md index 75375cdc3..7d7c13c08 100644 --- a/docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md +++ b/docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md @@ -6,7 +6,7 @@ **Architecture:** Three independent layers. The emitter stops writing landmines (`IF EXISTS` on forward drops; `CREATE SCHEMA IF NOT EXISTS` ahead of a non-default schema). A new `meta verify --replay` gate replays the committed chain into an **in-process** database (PGlite for postgres, `:memory:` libsql for sqlite) and asserts it applies; `--replay-snapshot` additionally asserts it reproduces the committed snapshot via the already-built-but-unwired `verifyReplay`. An emit-time provenance guard refuses to drop an object the committed snapshot never contained, so the bad SQL is never authored. -**Tech Stack:** TypeScript, Bun test runner, Kysely, `@electric-sql/pglite` (new), `@libsql/client` (already present via the sqlite path). +**Tech Stack:** TypeScript, Bun test runner, Kysely, `@electric-sql/pglite` (new, optional peer), `@libsql/kysely-libsql` (already a hard dependency of `cli`, a devDependency of `migrate-ts`). **Spec:** [`docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md`](../specs/2026-08-19-migrate-chain-replayability-design.md) @@ -15,39 +15,73 @@ - **Scope is TypeScript only.** Schema migration is TS-owned ([ADR-0015](../../../spec/decisions/ADR-0015-single-shared-migrate-engine.md)). No Java/Kotlin/C#/Python work, no conformance-corpus fan-out. - **`IF EXISTS` goes on FORWARD drops only.** Down statements stay bare. `rollbackTo` runs `down.sql` and the ledger delete in ONE transaction (`apply/apply.ts:185-189`), so a down that no-ops would still record the rollback as done. - **Never `instanceof` a metadata node from another package** — use the exported guards (`isMetaObject`, `isWritableSource`, …). Two physical copies of `@metaobjectsdev/metadata` in one process make `instanceof` silently false. -- **No `any`.** Use `unknown` and narrow. A bare `let x;` is an implicit evolving `any`. +- **No `any`.** Use `unknown` and narrow. A bare `let x;` is an implicit evolving `any`. **No `as never` / `as unknown as T` in committed test fixtures** — every literal in this plan is written against the real type; if one does not compile, the type is the authority, not the cast. - **Never call `own*()` accessors** (ADR-0039) except where surrounding code documents a sanctioned reason. - **Errors are `ParseError` with a structured `code`** and `codeSource(...)` — never a message prefix. -- **Backward compatibility is absolute:** a project that declares no new flag must emit byte-identical migrations except for the `IF EXISTS` tokens this plan adds, and `SNAPSHOT_FORMAT_VERSION` must remain 3. +- **Backward compatibility is absolute:** a project that declares no new flag must emit byte-identical migrations except for the `IF EXISTS` and `CREATE SCHEMA` tokens this plan adds, and `SNAPSHOT_FORMAT_VERSION` must remain 3. +- **Every new gate FAILS OPEN on a missing input.** A project that has never generated a snapshot, or has no committed migrations, is not in an error state — but it must SAY so, never pass silently. A gate that is quiet when it checked nothing is indistinguishable from a gate that passed. - **Run tests scoped:** `cd server/typescript && bun test packages/`. NEVER a bare `bun test` at the repository root — it walks java/python/csharp and takes many minutes. - **`bun test` does NOT typecheck.** Run `bun run --filter '*' typecheck` from the repository root before every commit and confirm all 18 packages exit 0. -- **Public repository.** No private project names, no absolute home paths, in code, tests, fixtures, or commit messages. +- **Every task ends GREEN.** No task may leave `bun test packages/migrate-ts` or `packages/cli` red for a later task to inherit. Test churn caused by a task is fixed inside that task. +- **Public repository.** No private project names, no absolute home paths, in code, tests, fixtures, or commit messages. The committed `.githooks/pre-commit` enforces this (`git config core.hooksPath .githooks`) using the denylist at `git config hooks.denyListPath`. - **Stage explicit paths only.** Never `git add -A` — other worktrees share this repository. --- +## Task ordering (a DAG, not a line) + +``` +Task 1 (emit: IF EXISTS) ─┐ +Task 2 (emit: CREATE SCHEMA) ─┤ +Task 7 (provenance guard) ─┼─→ Task 8 (docs + CHANGELOG) +Task 3 (replay engine) ─┬─ Task 4 (verifyReplay governed) ─┐ + └─ Task 5 (--replay) ──────────────┴─→ Task 6 (--replay-snapshot) +``` + +- **Tasks 1, 2, 3 and 7 are independent** of each other and of everything else, and may run in any order or in parallel. +- **Task 4 needs Task 3** — its test imports `openReplayEngine`. +- **Task 5 needs Task 3** — the gate calls `openReplayEngine`. +- **Task 6 needs Tasks 3, 4 AND 5** — it extends `runReplayVerify` (Task 5) and passes `governed` (Task 4). +- **Task 8 needs 1, 2, 5, 6, 7** — it documents what they ship. + +--- + ## File Structure **Modified — `server/typescript/packages/migrate-ts/src/`** - `emit/postgres.ts` — `IF EXISTS` on forward drops; `CREATE SCHEMA IF NOT EXISTS` emission -- `emit/sqlite.ts` — `IF EXISTS` on forward drops +- `emit/sqlite.ts` — `IF EXISTS` on forward drops (two lines only; see Task 1) - `verify/replay.ts` — thread scope inputs into the snapshot comparison +- `types.ts` — `AllowOptions.dropUnmanaged` - `index.ts` — export the new replay-engine surface +- `package.json` — `@electric-sql/pglite` as an optional peer + devDependency **New — `server/typescript/packages/migrate-ts/src/`** - `verify/replay-engine.ts` — provision an in-process database (PGlite / `:memory:` libsql), hand back a Kysely instance and a disposer. One responsibility: engine lifecycle. No replay logic, no comparison. **Modified — `server/typescript/packages/cli/src/`** - `lib/args.ts` — `--replay` / `--replay-snapshot` verify flags; `drop-unmanaged` allow token +- `lib/allow.ts` — `drop-unmanaged` → `dropUnmanaged` grant mapping - `commands/verify.ts` — the replay gate - `commands/migrate.ts` — the emit-time provenance guard +- `package.json` — `--external @electric-sql/pglite` on `build:binary` + +**Modified — `server/typescript/packages/sdk/src/`** +- `config.ts` — `drop-unmanaged` in `AllowTokenEnum` + +**Modified — existing tests (churn, fixed inside the task that causes it)** +- `packages/migrate-ts/test/unit/emit-postgres.test.ts` (Task 1) +- `packages/migrate-ts/test/unit/emit-sqlite.test.ts` (Task 1) +- `packages/migrate-ts/test/check/emit-postgres-check.test.ts` (Task 1) +- `packages/migrate-ts/test/check-evolution/drop-check-down.test.ts` (Task 1) +- `packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts` (Task 1 — **both** lines 142 and 143) **New — tests** - `packages/migrate-ts/test/emit-drop-if-exists.test.ts` - `packages/migrate-ts/test/emit-postgres-create-schema.test.ts` - `packages/migrate-ts/test/unit/replay-engine.test.ts` +- `packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts` — the #313 regression, emitter→apply - `packages/migrate-ts/test/integrity/replay-scoped.test.ts` -- `packages/migrate-ts/test/integrity/replay-from-empty.test.ts` — the #313 regression - `packages/cli/test/verify-replay.test.ts` - `packages/cli/test/migrate-drop-unmanaged.test.ts` @@ -59,95 +93,192 @@ - Modify: `server/typescript/packages/migrate-ts/src/emit/postgres.ts` (`renderUp`, `renderDropView`) - Modify: `server/typescript/packages/migrate-ts/src/emit/sqlite.ts` (`renderUpNative`) - Test: `server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts` (create) +- Fix churn: `test/unit/emit-postgres.test.ts`, `test/unit/emit-sqlite.test.ts`, `test/check/emit-postgres-check.test.ts`, `test/check-evolution/drop-check-down.test.ts`, `test/integration/pg-constraint-backed-index-285.test.ts` **Interfaces:** - Consumes: nothing from earlier tasks. - Produces: no new exports. Behaviour change only. -**Context the implementer needs:** `renderUp`/`renderUpNative` are the FORWARD renderers. -`renderDown`/`renderDownNative` are the down renderers and are **out of scope** — see the Global -Constraint. Two forward drops stay bare **deliberately**: `sqlite.ts:197` (inside the -recreate-and-copy rebuild) and `emit/d1-cascade.ts:126`, because each drops a table the same recipe -just `INSERT…SELECT`ed from, where `IF EXISTS` would convert a caught corruption into a silent one. +**Context the implementer needs.** `renderUp` (postgres) / `renderUpNative` (sqlite) are the FORWARD +renderers. `renderDown` / `renderDownNative` are out of scope — see the Global Constraint. + +The exact sites, all verified bare at HEAD: + +| File:line | Change kind | Action | +|---|---|---| +| `postgres.ts:66` | `drop-table` | add `IF EXISTS` | +| `postgres.ts:95` | `drop-index`, constraint-backed arm | `DROP CONSTRAINT IF EXISTS` | +| `postgres.ts:96` | `drop-index`, plain arm | `DROP INDEX IF EXISTS` | +| `postgres.ts:98` | `drop-fk` | `DROP CONSTRAINT IF EXISTS` | +| `postgres.ts:104` | `drop-check` | `DROP CONSTRAINT IF EXISTS` | +| `postgres.ts:375` | `renderDropView`, plain | `DROP VIEW IF EXISTS` | +| `postgres.ts:388` | `renderDropView`, CASCADE | `DROP VIEW IF EXISTS … CASCADE` | +| `sqlite.ts:219` | `drop-table` | add `IF EXISTS` | +| `sqlite.ts:225` | `drop-index` | add `IF EXISTS` | + +**SQLite is two lines, not more.** `sqlite.ts:240` (`drop-view`) and `:241` (`replace-view`) already +emit `DROP VIEW IF EXISTS` on the forward side. The sqlite `drop-fk`/`drop-check`/`add-check` arms +**throw** (`sqlite.ts:226-235`) — those change kinds are folded into a table recreate, so there is no +standalone statement to guard, which is why the Postgres-only guard on `drop-fk`/`drop-check` is not +a dialect split (spec §3.1). + +**Deliberately left bare — do not "finish the job":** +- `sqlite.ts:197` — the recreate-and-copy rebuild's `DROP TABLE`. It drops a table the same recipe + just `INSERT…SELECT`ed from; `IF EXISTS` there converts a caught corruption into a silent one. +- `emit/d1-cascade.ts:126` — same reason. +- `postgres.ts:431` — inside `renderRestoreView`, reached only from `postgres.ts:178`/`:179`, both in + `renderDown`. Guarding it violates the forward-only rule in the same change that states it. +- Every down site: `postgres.ts:113`, `:147`, `:164`, `:171`, `:176`; `sqlite.ts:256`, `:262`, `:275`. + +**`drop-check` IS reachable — the code comment saying otherwise is wrong.** `emit/postgres.ts:99-102` +claims `add-check`/`drop-check` are "declared but NOT yet produced by the diff". That comment is +false: `diff/index.ts:579` and `:592` both push `drop-check`, an evolved `field.enum @values` is a +live producer, and `test/check/emit-postgres-check.test.ts:25` plus +`test/check-evolution/drop-check-down.test.ts:12` already assert the emitted forward statement. +**Delete the false half of that comment as part of this task** and assert `drop-check` in the new +test like every other forward drop. `emit/d1.ts:21` renders through `renderSqlite`, so the sqlite edits also change D1's committed -migrations. That is accepted and expected. +migrations. That is accepted and expected (spec §3.1). - [ ] **Step 1: Write the failing test** ```ts // server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts +// +// Forward drops must tolerate an absent object so a committed chain replays into +// an empty database (#313). Down statements must NOT — `rollbackTo` runs down.sql +// and the ledger delete in one transaction, so a silently-no-op down would record +// the rollback as done. import { describe, test, expect } from "bun:test"; import { renderPostgres } from "../src/emit/postgres.js"; import { renderSqlite } from "../src/emit/sqlite.js"; -import type { Change } from "../src/types.js"; +import type { Change, ChangeStatus, TableDescriptor } from "../src/types.js"; -const TABLE = { name: "gone", columns: [], indexes: [], foreignKeys: [], checks: [], primaryKey: [] }; +const ALLOWED: ChangeStatus = { state: "allowed" }; -describe("forward drops tolerate an absent object", () => { +const GONE: TableDescriptor = { + name: "gone", + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], +}; + +describe("forward drops tolerate an absent object (#313)", () => { test("postgres drop-table", () => { - const { up } = renderPostgres([{ kind: "drop-table", table: "gone" } as Change]); + const { up } = renderPostgres([{ kind: "drop-table", table: "gone", status: ALLOWED }]); expect(up).toContain('DROP TABLE IF EXISTS "gone";'); }); test("postgres drop-view", () => { - const { up } = renderPostgres([{ kind: "drop-view", view: "v_gone" } as Change]); - expect(up).toContain("DROP VIEW IF EXISTS"); + const { up } = renderPostgres([{ kind: "drop-view", view: "v_gone", status: ALLOWED }]); + expect(up).toContain('DROP VIEW IF EXISTS "v_gone";'); }); test("postgres drop-index, plain", () => { - const { up } = renderPostgres([{ kind: "drop-index", table: "t", index: "idx_gone" } as Change]); + const { up } = renderPostgres([{ kind: "drop-index", table: "t", index: "idx_gone", status: ALLOWED }]); expect(up).toContain('DROP INDEX IF EXISTS "idx_gone";'); }); - test("postgres drop-index, constraint-backed", () => { - const { up } = renderPostgres([ - { kind: "drop-index", table: "t", index: "uq_gone", restore: { constraint: "unique" } } as Change, - ]); - expect(up).toContain('DROP CONSTRAINT IF EXISTS "uq_gone";'); + test("postgres drop-index, constraint-backed (#285)", () => { + const { up } = renderPostgres([{ + kind: "drop-index", table: "t", index: "uq_gone", status: ALLOWED, + restore: { name: "uq_gone", columns: ["a"], unique: true, constraint: "unique" }, + }]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "uq_gone";'); }); test("postgres drop-fk", () => { - const { up } = renderPostgres([{ kind: "drop-fk", table: "t", fk: "fk_gone" } as Change]); - expect(up).toContain('DROP CONSTRAINT IF EXISTS "fk_gone";'); + const { up } = renderPostgres([{ kind: "drop-fk", table: "t", fk: "fk_gone", status: ALLOWED }]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "fk_gone";'); + }); + + // drop-check IS produced by the diff (diff/index.ts:579, :592) — an evolved + // `field.enum @values` is a live producer. The `renderUp` comment claiming + // otherwise is deleted by this task. + test("postgres drop-check", () => { + const { up } = renderPostgres([{ kind: "drop-check", table: "t", check: "t_qty_chk", status: ALLOWED }]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "t_qty_chk";'); }); test("sqlite drop-table", () => { - const { up } = renderSqlite([{ kind: "drop-table", table: "gone" } as Change], undefined, undefined); + const { up } = renderSqlite([{ kind: "drop-table", table: "gone", status: ALLOWED }]); expect(up).toContain('DROP TABLE IF EXISTS "gone";'); }); test("sqlite drop-index", () => { - const { up } = renderSqlite([{ kind: "drop-index", table: "t", index: "idx_gone" } as Change], undefined, undefined); + const { up } = renderSqlite([{ kind: "drop-index", table: "t", index: "idx_gone", status: ALLOWED }]); expect(up).toContain('DROP INDEX IF EXISTS "idx_gone";'); }); + + // Already guarded before this task; pinned so a later sweep cannot un-guard it. + test("sqlite drop-view was already guarded", () => { + const { up } = renderSqlite([{ kind: "drop-view", view: "v_gone", status: ALLOWED }]); + expect(up).toContain('DROP VIEW IF EXISTS "v_gone";'); + }); }); describe("down statements stay bare — a rollback must fail loudly", () => { test("postgres create-table down", () => { - const { down } = renderPostgres([{ kind: "create-table", table: TABLE } as Change]); + const { down } = renderPostgres([{ kind: "create-table", table: GONE, status: ALLOWED }]); expect(down).toContain('DROP TABLE "gone";'); expect(down).not.toContain("DROP TABLE IF EXISTS"); }); + test("postgres create-view down", () => { + const { down } = renderPostgres([{ + kind: "create-view", status: ALLOWED, + view: { name: "v", sql: "SELECT 1 AS one", columns: ["one"] }, + }]); + expect(down).toContain('DROP VIEW "v";'); + expect(down).not.toContain("DROP VIEW IF EXISTS"); + }); + test("sqlite create-table down", () => { - const { down } = renderSqlite([{ kind: "create-table", table: TABLE } as Change], undefined, undefined); + const { down } = renderSqlite([{ kind: "create-table", table: GONE, status: ALLOWED }]); expect(down).toContain('DROP TABLE "gone";'); expect(down).not.toContain("DROP TABLE IF EXISTS"); }); }); + +describe("the recreate-and-copy rebuild drop stays bare — deliberately", () => { + test("sqlite recreate emits a bare DROP TABLE for the table it just copied from", () => { + const expectedSchema = { + tables: [{ + name: "orders", + columns: [ + { name: "id", sqlType: { kind: "integer" as const, bits: 64 as const }, nullable: false }, + { name: "amount", sqlType: { kind: "integer" as const, bits: 64 as const }, nullable: false }, + ], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], + }], + views: [], + }; + const { up } = renderSqlite( + [{ + kind: "change-column-type", table: "orders", column: "amount", + from: { kind: "real" }, to: { kind: "integer", bits: 64 }, status: ALLOWED, + }], + expectedSchema, + ); + // IF EXISTS here would turn a caught corruption into a silent one: the recipe + // just INSERT…SELECTed out of this exact table. + expect(up).toContain('DROP TABLE "orders";'); + expect(up).not.toContain('DROP TABLE IF EXISTS "orders";'); + }); +}); ``` -**If a `Change` literal above does not typecheck**, widen it to match the real discriminated union -in `src/types.ts` rather than casting away the error — the `as Change` casts are there to keep the -fixtures short, not to hide a shape mismatch. If `renderSqlite`'s signature differs from -`(changes, expectedSchema, actualMeta)`, match the real one. +**If a literal above does not compile**, read the real shape in `src/types.ts` (`Change`, +`ChangeStatus`, `TableDescriptor`, `IndexDescriptor`, `ViewDescriptor`) and correct the literal. +Do not add a cast. `renderSqlite(changes, expectedSchema?, actualMeta?)` and +`renderPostgres(changes)` are the real signatures. - [ ] **Step 2: Run it to verify it fails** Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-drop-if-exists.test.ts` -Expected: the seven "forward drops" tests FAIL (no `IF EXISTS` in the output). The two "down -statements stay bare" tests PASS already — they pin behaviour this task must not change. +Expected: the six postgres + two sqlite "forward drops" tests FAIL. The "sqlite drop-view was +already guarded", the three "down statements stay bare" and the rebuild test PASS already — they pin +behaviour this task must not change. - [ ] **Step 3: Implement — postgres forward drops** @@ -172,11 +303,16 @@ In `emit/postgres.ts`, inside `renderUp`: case "drop-check": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.check)};`; ``` -`drop-check` is currently **unreachable** — the comment above that arm records that CHECKs are -create-time-only and the diff never produces this change. Guard it anyway for consistency, and do -not add a test asserting it fires, because it cannot. +Replace the false comment above the check arms (`postgres.ts:99-102`) with the truth: + +```ts + // `drop-check` IS produced by the diff — diff/index.ts:579 and :592 push it, + // and an evolved `field.enum @values` is a live producer. (An earlier comment + // here claimed these arms were unreachable; they are not.) `add-check` is the + // paired ADD and rides the same passes. +``` -In `renderDropView` (around `:375` and `:388`), change both the plain and the CASCADE form: +In `renderDropView` (`:375` and `:388`), change both forms: ```ts if (dependents.length === 0) return `DROP VIEW IF EXISTS ${qualified};`; @@ -186,12 +322,11 @@ In `renderDropView` (around `:375` and `:388`), change both the plain and the CA `DROP VIEW IF EXISTS ${qualified} CASCADE;`, ``` -**Do NOT touch `renderRestoreView` (around `:431`).** It is reached only from `postgres.ts:178` -and `:179`, both inside `renderDown`. +**Do NOT touch `renderRestoreView` (`:431`).** - [ ] **Step 4: Implement — sqlite forward drops** -In `emit/sqlite.ts`, inside `renderUpNative`: +In `emit/sqlite.ts`, inside `renderUpNative`, exactly two lines: ```ts case "drop-table": return `DROP TABLE IF EXISTS ${quote(c.table)};`; @@ -201,32 +336,81 @@ In `emit/sqlite.ts`, inside `renderUpNative`: case "drop-index": return `DROP INDEX IF EXISTS ${quote(c.index)};`; ``` -Leave `sqlite.ts:197` (the rebuild `DROP TABLE`) and `renderDownNative` untouched. +Leave `sqlite.ts:197` and `renderDownNative` untouched. -- [ ] **Step 5: Run the test to verify it passes** +- [ ] **Step 5: Run the new test to verify it passes** Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-drop-if-exists.test.ts` -Expected: PASS, all nine. +Expected: PASS, all thirteen. + +- [ ] **Step 6: Fix the expected churn — five named files** -- [ ] **Step 6: Run the affected suites** +These assertions pin the exact bare statement and will go RED. Each is a FORWARD (`up`) assertion; +update it to the `IF EXISTS` form. **Read each one first**: if an assertion is on a `down`, the +correct fix is to leave it alone and check you did not edit a down renderer. + +| File:line | Current | Becomes | +|---|---|---| +| `test/unit/emit-postgres.test.ts:136` | `DROP TABLE "legacy";` | `DROP TABLE IF EXISTS "legacy";` | +| `test/unit/emit-postgres.test.ts:164` | `DROP INDEX "old_idx";` | `DROP INDEX IF EXISTS "old_idx";` | +| `test/unit/emit-postgres.test.ts:187` | `ALTER TABLE "weeks" DROP CONSTRAINT "weeks_program_id_fk";` | `… DROP CONSTRAINT IF EXISTS "weeks_program_id_fk";` | +| `test/unit/emit-sqlite.test.ts:61` | `DROP TABLE "old";` | `DROP TABLE IF EXISTS "old";` | +| `test/unit/emit-sqlite.test.ts:77` | `DROP INDEX "i";` | `DROP INDEX IF EXISTS "i";` | +| `test/check/emit-postgres-check.test.ts:25` | `r.up` ⊃ `ALTER TABLE "orders" DROP CONSTRAINT "orders_status_chk";` | `… DROP CONSTRAINT IF EXISTS …` | +| `test/check-evolution/drop-check-down.test.ts:12` | `r.up` ⊃ `ALTER TABLE "orders" DROP CONSTRAINT "orders_qty_numeric_chk";` | `… DROP CONSTRAINT IF EXISTS …` | +| `test/integration/pg-constraint-backed-index-285.test.ts:143` | `/ALTER TABLE .*DROP CONSTRAINT "work_item_message_id_unique"/` | `/ALTER TABLE .*DROP CONSTRAINT IF EXISTS "work_item_message_id_unique"/` | + +**Do NOT change these — they must stay green, and their staying green is the evidence:** +`test/unit/emit-postgres.test.ts:318` and `test/unit/emit-sqlite.test.ts:237` (down assertions), +`test/check/emit-postgres-check.test.ts:21` (down), `test/unit/emit-sqlite.test.ts:128` (the +recreate-and-copy rebuild drop), `test/write-migration*.test.ts` (hand-written SQL, not emitter +output), `test/unit/emit-views.test.ts:20` and `test/unit/diff.test.ts:264` (`/DROP VIEW/i`, which +still matches). + +- [ ] **Step 7: Re-anchor the one negative assertion that would go VACUOUSLY green** + +`test/integration/pg-constraint-backed-index-285.test.ts:142` is a *negative* assertion: + +```ts + expect(sqlText).not.toMatch(/DROP INDEX "?work_item_message_id_unique/); +``` + +Adding `IF EXISTS` puts ` IF EXISTS ` between `DROP INDEX` and the quote, so the regex stops matching +for a reason that has nothing to do with #285 — it would keep passing even if #285 fully regressed. +Re-anchor it so it still tests what it was written to test: + +```ts + // Anchored to tolerate the #313 `IF EXISTS` token: without `(IF EXISTS )?` this + // negative assertion passes because the SPELLING changed, not because the + // constraint-backed index is correctly dropped via ALTER TABLE. + expect(sqlText).not.toMatch(/DROP INDEX (IF EXISTS )?"?work_item_message_id_unique/); +``` + +Then **prove the re-anchoring works**: temporarily revert the `drop-index` constraint-backed arm to +`DROP INDEX IF EXISTS …`, confirm line 142 goes RED, and restore. A negative assertion you have not +seen fail is not a test. + +- [ ] **Step 8: Run the affected suites** Run: `cd server/typescript && bun test packages/migrate-ts && bun test packages/cli` -Expected: PASS. Existing assertions on exact `DROP TABLE "x";` strings will need updating to the -`IF EXISTS` form — that is expected churn, not a regression. **Read each one before changing it**: -if an assertion is on a DOWN statement, the correct fix is to leave the assertion alone and check -you did not edit a down renderer. +Expected: PASS, both. If anything is still red, it is churn Step 6 missed — fix it here, not later. -- [ ] **Step 7: Typecheck** +- [ ] **Step 9: Typecheck** Run: `bun run --filter '*' typecheck` (from the repository root) Expected: all 18 packages exit 0. -- [ ] **Step 8: Commit** +- [ ] **Step 10: Commit** ```bash git add server/typescript/packages/migrate-ts/src/emit/postgres.ts \ server/typescript/packages/migrate-ts/src/emit/sqlite.ts \ - server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts + server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts \ + server/typescript/packages/migrate-ts/test/unit/emit-postgres.test.ts \ + server/typescript/packages/migrate-ts/test/unit/emit-sqlite.test.ts \ + server/typescript/packages/migrate-ts/test/check/emit-postgres-check.test.ts \ + server/typescript/packages/migrate-ts/test/check-evolution/drop-check-down.test.ts \ + server/typescript/packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts git commit -m "fix(migrate): forward drops tolerate an absent object so a chain can replay" ``` @@ -244,45 +428,87 @@ git commit -m "fix(migrate): forward drops tolerate an absent object so a chain **Context:** `CREATE SCHEMA` is emitted nowhere in `migrate-ts/src` or `cli/src` today except the ledger's own (`apply/ledger.ts:126`). A chain containing `CREATE TABLE "reporting"."x"` therefore -cannot apply to a virgin database — the schema does not exist. SQLite has no schema namespacing +cannot apply to a virgin database. SQLite has no schema namespacing (`emit-sqlite-schema-rejected.test.ts` pins that a schema is rejected there), so this is Postgres-only and is NOT a dialect split. +**Collect from `create-view` too, not only `create-table`.** Spec §3.3 says "ahead of the first +**object** in a non-default schema", and a first migration that creates only a *view* in +`"reporting"` fails identically. `create-view` carries the schema in **two** places — the change's +own `schema?` and `view.schema` — so read `c.schema ?? c.view.schema`, the same precedence +`renderCreateView(c.view, c.schema, …)` already uses. + - [ ] **Step 1: Write the failing test** ```ts // server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts import { describe, test, expect } from "bun:test"; import { renderPostgres } from "../src/emit/postgres.js"; -import type { Change } from "../src/types.js"; +import type { ChangeStatus, TableDescriptor, ViewDescriptor } from "../src/types.js"; -const t = (name: string, schema?: string) => ({ - name, schema, columns: [], indexes: [], foreignKeys: [], checks: [], primaryKey: [], +const ALLOWED: ChangeStatus = { state: "allowed" }; + +const t = (name: string, schema?: string): TableDescriptor => ({ + name, + ...(schema !== undefined ? { schema } : {}), + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], +}); + +const v = (name: string, schema?: string): ViewDescriptor => ({ + name, + ...(schema !== undefined ? { schema } : {}), + sql: "SELECT 1 AS one", + columns: ["one"], }); -describe("a chain that creates a non-default schema's table creates the schema first", () => { +describe("a chain that creates an object in a non-default schema creates the schema first", () => { test("emits CREATE SCHEMA IF NOT EXISTS before the table", () => { - const { up } = renderPostgres([{ kind: "create-table", table: t("x", "reporting") } as Change]); + const { up } = renderPostgres([{ kind: "create-table", table: t("x", "reporting"), status: ALLOWED }]); expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "reporting";'); expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "reporting";')) .toBeLessThan(up.indexOf("CREATE TABLE")); }); - test("emits it once for two tables in the same schema", () => { + // Spec §3.3 says "the first OBJECT", not "the first table". A chain whose first + // migration creates only a view in a non-default schema fails identically. + test("emits it for a create-view too", () => { + const { up } = renderPostgres([{ kind: "create-view", view: v("v_x", "reporting"), status: ALLOWED }]); + expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "reporting";'); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "reporting";')) + .toBeLessThan(up.indexOf("CREATE VIEW")); + }); + + test("emits it once for two objects in the same schema", () => { const { up } = renderPostgres([ - { kind: "create-table", table: t("x", "reporting") } as Change, - { kind: "create-table", table: t("y", "reporting") } as Change, + { kind: "create-table", table: t("x", "reporting"), status: ALLOWED }, + { kind: "create-table", table: t("y", "reporting"), status: ALLOWED }, + { kind: "create-view", view: v("v_x", "reporting"), status: ALLOWED }, ]); expect(up.match(/CREATE SCHEMA IF NOT EXISTS "reporting";/g)).toHaveLength(1); }); + test("emits one per distinct schema, in sorted order", () => { + const { up } = renderPostgres([ + { kind: "create-table", table: t("x", "zeta"), status: ALLOWED }, + { kind: "create-table", table: t("y", "alpha"), status: ALLOWED }, + ]); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "alpha";')) + .toBeLessThan(up.indexOf('CREATE SCHEMA IF NOT EXISTS "zeta";')); + }); + test("emits nothing for the default schema", () => { - const { up } = renderPostgres([{ kind: "create-table", table: t("x") } as Change]); + const { up } = renderPostgres([{ kind: "create-table", table: t("x"), status: ALLOWED }]); + expect(up).not.toContain("CREATE SCHEMA"); + }); + + test("emits nothing for an explicit 'public'", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x", "public"), status: ALLOWED }]); expect(up).not.toContain("CREATE SCHEMA"); }); test("the down does NOT drop the schema", () => { - const { down } = renderPostgres([{ kind: "create-table", table: t("x", "reporting") } as Change]); + const { down } = renderPostgres([{ kind: "create-table", table: t("x", "reporting"), status: ALLOWED }]); expect(down).not.toContain("DROP SCHEMA"); }); }); @@ -294,50 +520,51 @@ this tool does not own and cannot restore. - [ ] **Step 2: Run it to verify it fails** Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-postgres-create-schema.test.ts` -Expected: FAIL — the first two cases find no `CREATE SCHEMA`. The last two PASS already. +Expected: FAIL — the first four cases find no `CREATE SCHEMA`. The last three PASS already. - [ ] **Step 3: Implement** -In `renderPostgres`, after `const sorted = …` and before the render loop, collect the distinct -non-default schemas the forward pass will create objects in, and prepend one statement each: +In `renderPostgres`, after the changes are sorted and before the returned `up` is joined: ```ts - // A chain must be appliable to a VIRGIN database (#313). CREATE TABLE "s"."x" + // A chain must be appliable to a VIRGIN database (#313). `CREATE TABLE "s"."x"` // fails there unless the schema exists, and no migration has ever created one. - // IF NOT EXISTS because a later migration in the same chain, or an operator, - // may have created it already. Deliberately NOT dropped in `down`: the schema - // may hold objects this tool does not own and cannot restore. + // IF NOT EXISTS because a later migration in the same chain, or an operator, may + // have created it already. Views count as objects too, so a first migration that + // creates only a view in a non-default schema is covered. Deliberately NOT dropped + // in `down`: the schema may hold objects this tool does not own and cannot restore. const createdSchemas = new Set(); for (const c of sorted) { - if (c.kind !== "create-table") continue; - const s = c.table.schema; + const s = + c.kind === "create-table" ? c.table.schema + : c.kind === "create-view" ? (c.schema ?? c.view.schema) + : undefined; if (s !== undefined && s !== DEFAULT_DB_SCHEMA_POSTGRES) createdSchemas.add(s); } + // Sorted so output is deterministic — the snapshot and golden tests rely on it. const schemaStmts = [...createdSchemas].sort().map((s) => `CREATE SCHEMA IF NOT EXISTS ${quote(s)};`); ``` -then emit `schemaStmts` ahead of `upStmts` in the returned `up`: +then prepend to the returned `up`: ```ts up: [...schemaStmts, ...upStmts].join("\n\n"), ``` -Import `DEFAULT_DB_SCHEMA_POSTGRES` from wherever the file already resolves the default schema -name — `diff/index.ts:152` uses it, so it is exported from a shared module. If `renderPostgres` -already has a local notion of the default schema, use that instead of adding a second one. - -Sorting `createdSchemas` keeps output deterministic, which the snapshot and golden tests rely on. +`DEFAULT_DB_SCHEMA_POSTGRES` is exported from `@metaobjectsdev/metadata` — `qualified-name.ts` and +`diff/index.ts:17` both already import it from there. If `postgres.ts` already has a local notion of +the default schema, use that rather than adding a second one. - [ ] **Step 4: Run the test to verify it passes** Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-postgres-create-schema.test.ts` -Expected: PASS, all four. +Expected: PASS, all seven. - [ ] **Step 5: Run the affected suites** -Run: `cd server/typescript && bun test packages/migrate-ts` +Run: `cd server/typescript && bun test packages/migrate-ts && bun test packages/cli` Expected: PASS. `emit-postgres-schema-namespacing.test.ts` is the file most likely to need updating; -read its assertions before changing them. +read its assertions before changing them. Any churn is fixed here, in this task. - [ ] **Step 6: Typecheck and commit** @@ -356,7 +583,8 @@ git commit -m "fix(migrate): a chain creates the schema it needs, so it applies **Files:** - Create: `server/typescript/packages/migrate-ts/src/verify/replay-engine.ts` - Modify: `server/typescript/packages/migrate-ts/src/index.ts` (export it) -- Modify: `server/typescript/packages/migrate-ts/package.json` (add `@electric-sql/pglite`) +- Modify: `server/typescript/packages/migrate-ts/package.json` (optional peer + devDependency) +- Modify: `server/typescript/packages/cli/package.json` (`build:binary` external) - Test: `server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts` (create) **Interfaces:** @@ -369,20 +597,69 @@ git commit -m "fix(migrate): a chain creates the schema it needs, so it applies } export function openReplayEngine(dialect: "postgres" | "sqlite"): Promise; ``` - Tasks 5 and 6 call `openReplayEngine` and must `await engine.dispose()` in a `finally`. + Tasks 4, 5 and 6 call `openReplayEngine` and must `await engine.dispose()` in a `finally`. + +### Two decisions this task settles, both already verified + +**Where it lives, and how the dependency is declared.** `openReplayEngine` goes in **`migrate-ts`**, +not `cli`: `migrate-ts`'s own tests (Tasks 4 and 5) need it, and `cli` depends on `migrate-ts`, so +putting it in `cli` would be a cycle. But `migrate-ts` today declares only `@iarna/toml` and +`@metaobjectsdev/metadata` as dependencies — `kysely` is a peer and every driver is dev-only, +deliberately. **PGlite is 22 MB of WASM**; a hard dependency would make every CLI adopter download it +to run `meta gen`. So: + +- `@electric-sql/pglite` → **optional peerDependency** `">=0.3.0 <0.6.0"` + **devDependency** + `"^0.5.0"`. Both 0.3.16 and 0.5.5 were verified against the adapter below. +- `@libsql/kysely-libsql` → **optional peerDependency** `">=0.4.0 <0.5.0"` + keep the existing + devDependency. It is already a hard `dependencies` entry of `cli` (`^0.4.0`), so the CLI path + always resolves it and only a direct `migrate-ts` embedder can miss it. +- Both are imported **lazily** inside the opener, with an install hint on failure, mirroring + `cli/src/lib/kysely.ts`'s `buildKyselyFromUrl`. +- Peer ranges must be **bounded** or `scripts/check-peer-ranges.ts` fails the `gates` lane (its test + is: a range is unbounded exactly when it accepts `9999.0.0`). Set + `peerDependenciesMeta..optional = true` for both. +- `cli`'s `build:binary` bundles the CLI (`bun build ./bin/meta.ts --compile`). Add + `--external @electric-sql/pglite` beside the two existing `--external @biomejs/*` flags, so the + standalone binary does not embed 22 MB of WASM and instead resolves it from the adopter's project + at run time. + +**PGlite is NOT `pg`-compatible — a shim is required.** PGlite exposes `query`/`exec`/`close`; +kysely's `PostgresDialect` wants a `pg.Pool` (`connect()` → a client with `query()`/`release()`, plus +`end()`). The ~20-line shim below was **executed against PGlite 0.3.16 and 0.5.5** and verified to +handle: `CREATE SCHEMA`, a CHECK constraint, `information_schema` reads, kysely transactions, +transaction *rollback*, `pg_advisory_lock`/`pg_advisory_unlock` (which `applyPending` takes on +postgres), and — the one that matters — rejecting `DROP TABLE "theirs"` with exactly +`table "theirs" does not exist`, the reporter's error. + +Also verified: `LibsqlDialect({ url: ":memory:" })` works, and two `:memory:` instances are fully +isolated from each other. + +- [ ] **Step 1: Declare the dependencies** + +In `server/typescript/packages/migrate-ts/package.json`: + +```jsonc + "peerDependencies": { + "kysely": ">=0.27.0 <0.30.0", + "@electric-sql/pglite": ">=0.3.0 <0.6.0", + "@libsql/kysely-libsql": ">=0.4.0 <0.5.0" + }, + "peerDependenciesMeta": { + "@electric-sql/pglite": { "optional": true }, + "@libsql/kysely-libsql": { "optional": true } + }, +``` -**Context:** This is why the design needs no scratch database on the user's server: both engines run -in-process. PGlite is real Postgres compiled to WASM. `dispose()` must be safe to call twice, so a -caller can dispose in a `finally` after an early return. +and add `"@electric-sql/pglite": "^0.5.0"` to `devDependencies` (`@libsql/kysely-libsql` is already +there). Then from the repository root: `bun install`. -- [ ] **Step 1: Add the dependency** +In `server/typescript/packages/cli/package.json`, extend `build:binary`: -```bash -cd server/typescript/packages/migrate-ts && bun add @electric-sql/pglite +``` +bun build ./bin/meta.ts --compile --outfile dist/meta --external @biomejs/wasm-bundler --external @biomejs/wasm-web --external @electric-sql/pglite ``` -Then confirm it landed in `dependencies` (not `devDependencies`) in -`server/typescript/packages/migrate-ts/package.json` — the CLI imports this at runtime. +Run `bun run scripts/check-peer-ranges.ts` (or the `gates` lane's equivalent) and confirm it passes. - [ ] **Step 2: Write the failing test** @@ -408,8 +685,8 @@ describe("openReplayEngine", () => { test("postgres: gives an empty, usable database with real PG DDL", async () => { const engine = await openReplayEngine("postgres"); try { - // Schema namespacing + a CHECK — both are things sqlite cannot express, - // so this proves the postgres engine is really Postgres. + // Schema namespacing + a CHECK — neither is expressible in sqlite, so this + // proves the postgres engine is really Postgres. await sql`CREATE SCHEMA IF NOT EXISTS "reporting"`.execute(engine.db); await sql`CREATE TABLE "reporting"."t" (id integer primary key, n integer CHECK (n > 0))`.execute(engine.db); const rows = await sql<{ table_name: string }>` @@ -421,6 +698,48 @@ describe("openReplayEngine", () => { } }); + // applyPending runs each migration file inside a kysely transaction and takes a + // pg advisory lock on postgres. Both must work through the shim, or the gate + // fails for a reason that has nothing to do with the chain under test. + test("postgres: transactions roll back, and advisory locks work", async () => { + const engine = await openReplayEngine("postgres"); + try { + await sql`CREATE TABLE t (id integer primary key)`.execute(engine.db); + await expect( + engine.db.transaction().execute(async (trx) => { + await sql`INSERT INTO t (id) VALUES (1)`.execute(trx); + throw new Error("boom"); + }), + ).rejects.toThrow(/boom/); + const after = await sql<{ c: string }>`SELECT count(*)::text AS c FROM t`.execute(engine.db); + expect(after.rows[0]?.c).toBe("0"); + + await sql`SELECT pg_advisory_lock(hashtext('meta'))`.execute(engine.db); + await sql`SELECT pg_advisory_unlock(hashtext('meta'))`.execute(engine.db); + } finally { + await engine.dispose(); + } + }); + + // The whole gate rests on this: a statement against a missing object must REJECT. + test("postgres: dropping a missing table rejects — the #313 signal", async () => { + const engine = await openReplayEngine("postgres"); + try { + await expect(sql`DROP TABLE "theirs"`.execute(engine.db)).rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + } + }); + + test("sqlite: dropping a missing table rejects — the #313 signal", async () => { + const engine = await openReplayEngine("sqlite"); + try { + await expect(sql`DROP TABLE "theirs"`.execute(engine.db)).rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + } + }); + test("two engines of the same dialect do not share state", async () => { const a = await openReplayEngine("sqlite"); const b = await openReplayEngine("sqlite"); @@ -458,10 +777,16 @@ Expected: FAIL — `openReplayEngine` is not defined. // against the user's server would mean CREATE DATABASE — which needs CREATEDB, // breaks behind a connection pooler, is restricted on managed Postgres, collides // between parallel CI jobs sharing one server, and puts a DROP DATABASE next to a -// name derived from a real one. None of that is worth it when the engines run -// in-process: PGlite is real Postgres compiled to WASM, and libsql runs sqlite in -// memory. Nothing to provision, nothing to clean up, nothing to drop by mistake. -import { Kysely, PostgresDialect, SqliteDialect } from "kysely"; +// name derived from a real one (Postgres truncates identifiers at 63 bytes, so a +// long enough target derives a scratch name that truncates back ONTO the target). +// None of that is worth it when the engines run in-process: PGlite is real Postgres +// compiled to WASM, and libsql runs sqlite in memory. Nothing to provision, nothing +// to clean up, nothing to drop by mistake. +// +// Both drivers are OPTIONAL peers imported lazily: PGlite is 22 MB of WASM and must +// not land in every `meta gen` adopter's node_modules. The install hints mirror +// `cli/src/lib/kysely.ts`'s. +import { Kysely } from "kysely"; export interface ReplayEngine { /** An empty database. The caller owns applying migrations into it. */ @@ -473,19 +798,98 @@ export interface ReplayEngine { export async function openReplayEngine( dialect: "postgres" | "sqlite", ): Promise { - if (dialect === "postgres") return openPglite(); - return openMemorySqlite(); + return dialect === "postgres" ? openPglite() : openMemorySqlite(); } -``` -Then implement the two openers against whatever Kysely dialect adapters this repo already uses. -**Read `cli/src/lib/kysely.ts` first** — it is the existing place a `Kysely` is constructed for both -dialects, and this file should mirror its adapter choices rather than inventing new ones. For -sqlite, `:memory:` through the same client that file already uses. For postgres, PGlite exposes a -`pg`-compatible interface; wire it into `PostgresDialect` the same way. +async function openMemorySqlite(): Promise { + type LibsqlDialectCtor = new (opts: { url: string }) => + ConstructorParameters>>[0]["dialect"]; + let LibsqlDialect: LibsqlDialectCtor; + try { + const mod = await import("@libsql/kysely-libsql"); + LibsqlDialect = mod.LibsqlDialect as unknown as LibsqlDialectCtor; + } catch { + throw new Error( + `the sqlite replay engine requires '@libsql/kysely-libsql'; install it to run 'meta verify --replay'`, + ); + } + const db = new Kysely>({ dialect: new LibsqlDialect({ url: ":memory:" }) }); + return disposable(db, async () => {}); +} + +async function openPglite(): Promise { + let PGliteCtor: new () => PgliteInstance; + try { + const mod = await import("@electric-sql/pglite"); + PGliteCtor = mod.PGlite as unknown as new () => PgliteInstance; + } catch { + throw new Error( + `the postgres replay engine requires '@electric-sql/pglite' (in-process WASM Postgres); ` + + `install it to run 'meta verify --replay' against a postgres chain`, + ); + } + const { PostgresDialect } = await import("kysely"); + const pg = new PGliteCtor(); + const db = new Kysely>({ + dialect: new PostgresDialect({ pool: pgliteAsPool(pg) as never }), + }); + return disposable(db, () => pg.close()); +} + +/** The slice of PGlite's surface this file uses. */ +interface PgliteInstance { + query(sql: string, params?: unknown[]): Promise<{ rows: unknown[]; affectedRows?: number; statement?: string }>; + close(): Promise; +} + +/** + * Kysely's PostgresDialect wants a `pg.Pool`: `connect()` returning a client with + * `query()`/`release()`, plus `end()`. PGlite offers `query`/`close` and is a SINGLE + * session, so every `connect()` hands back the same underlying instance — which is + * correct here because a replay is strictly sequential, and it is what makes a + * session advisory lock taken on one kysely connection visible to the next. + * + * `command` is only read by kysely to decide whether to report numAffectedRows; the + * replay path never reads it, so PGlite's `statement` (or a SELECT default) suffices. + */ +function pgliteAsPool(pg: PgliteInstance): unknown { + return { + async connect() { + return { + async query(sqlText: unknown, params?: readonly unknown[]) { + if (typeof sqlText !== "string") { + throw new Error("the PGlite replay engine does not support cursors"); + } + const r = await pg.query(sqlText, params ? [...params] : []); + return { + command: r.statement ?? "SELECT", + rowCount: r.affectedRows ?? r.rows.length, + rows: r.rows, + }; + }, + release() { /* single session — nothing to return to a pool */ }, + }; + }, + async end() { await pg.close(); }, + }; +} -Make `dispose` idempotent with a `disposed` flag; call `db.destroy()` and then the engine's own -`close()`/`end()` if it has one. +function disposable( + db: Kysely>, + closeEngine: () => Promise, +): ReplayEngine { + let disposed = false; + return { + db, + dispose: async () => { + if (disposed) return; + disposed = true; + try { await db.destroy(); } catch { /* the engine is throwaway */ } + try { await closeEngine(); } catch { /* idem */ } + }, + }; +} +``` - [ ] **Step 5: Export it** @@ -499,25 +903,52 @@ export { openReplayEngine, type ReplayEngine } from "./verify/replay-engine.js"; - [ ] **Step 6: Run the test to verify it passes** Run: `cd server/typescript && bun test packages/migrate-ts/test/unit/replay-engine.test.ts` -Expected: PASS, all four. +Expected: PASS, all seven. PGlite's first boot takes ~1s; that is expected. + +- [ ] **Step 7: Prove `introspect` works against PGlite** + +`--replay-snapshot` (Task 6) introspects the replayed database. `introspect/postgres.ts` reads +`information_schema` and `pg_catalog` and calls `pg_get_viewdef` — all real Postgres, so this should +just work, but Task 6 stalls if it does not. Add one case to the same file: + +```ts + test("postgres: introspect reads back a table and a view", async () => { + const engine = await openReplayEngine("postgres"); + try { + await sql`CREATE TABLE t (id integer primary key, n integer NOT NULL)`.execute(engine.db); + await sql`CREATE VIEW v AS SELECT id FROM t`.execute(engine.db); + const snap = await introspect(engine.db, "postgres"); + expect(snap.tables.map((x) => x.name)).toContain("t"); + expect(snap.views.map((x) => x.name)).toContain("v"); + } finally { + await engine.dispose(); + } + }); +``` -**If PGlite cannot execute the postgres case**, STOP and report it. The spec's engine tiering rests +(import `introspect` from `../../src/introspect/index.js`.) + +**If PGlite cannot execute the postgres cases**, STOP and report it. The spec's engine tiering rests on PGlite being real Postgres; if it is not sufficient, that is a design question, not something to work around by weakening the test. -- [ ] **Step 7: Typecheck and commit** +- [ ] **Step 8: Typecheck and commit** Run: `bun run --filter '*' typecheck` — all 18 exit 0. +Run: `cd server/typescript && bun test packages/migrate-ts` — PASS. ```bash git add server/typescript/packages/migrate-ts/src/verify/replay-engine.ts \ server/typescript/packages/migrate-ts/src/index.ts \ server/typescript/packages/migrate-ts/package.json \ + server/typescript/packages/cli/package.json \ server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts \ - server/typescript/bun.lock + bun.lock git commit -m "feat(migrate): an in-process replay engine, so the gate provisions nothing" ``` +(The lockfile is at the repository root — confirm the path before staging.) + --- ## Task 4: Thread scope inputs into `verifyReplay` @@ -527,10 +958,10 @@ git commit -m "feat(migrate): an in-process replay engine, so the gate provision - Test: `server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts` (create) **Interfaces:** -- Consumes: nothing from earlier tasks. +- Consumes: `openReplayEngine` (Task 3). - Produces: `VerifyReplayArgs` gains one optional field: ```ts - /** Out-of-scope / @unmanaged names to exclude from BOTH sides, as `scopedDiffInputs` produces. */ + /** Out-of-scope names to exclude from the SNAPSHOT side, as `scopeExpectedSchema` produces. */ governed?: GovernedScope; ``` Task 6 passes it. @@ -539,40 +970,56 @@ git commit -m "feat(migrate): an in-process replay engine, so the gate provision committed snapshot. A project declaring `migrate.scope` writes the *other* owner's tables into that snapshot on purpose (`carryForwardOutOfScope`, `scope.ts:93`), and the chain never creates them — so today the comparison reports them as missing. `excludeFromSnapshot` (`scope.ts:130`) exists for -exactly this and is already used by the committed-snapshot gate at `verify.ts:659`; this task -threads it here too. +exactly this and is already used by the committed-snapshot gate at `verify.ts:659`. + +**`excludeFromSnapshot` returns a `ScopedExpectedSchema`, not a `SchemaSnapshot`.** Its shape is +`{ snapshot, outOfScope, declaredSchemas? }` — take `.snapshot`. An empty `outOfScope` returns the +SAME snapshot object, so an unscoped caller's comparison is byte-for-byte what it was. + +**`GovernedScope`** (`scope.ts:149`) is `{ outOfScope: readonly string[]; declaredSchemas?: readonly string[] }`, +and names are **qualified**: `.` with an absent schema normalized to Postgres `public` +(`qualifiedDbName`, `qualified-name.ts:20`). SQLite objects normalize to the same `public.` prefix. +So the fixture below says `"public.theirs"`, not `"theirs"`. - [ ] **Step 1: Write the failing test** ```ts // server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts import { describe, test, expect } from "bun:test"; -import { verifyReplay } from "../../src/verify/replay.js"; -import { openReplayEngine } from "../../src/verify/replay-engine.js"; import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { verifyReplay } from "../../src/verify/replay.js"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; import type { SchemaSnapshot } from "../../src/types.js"; function chainWith(upSql: string): string { const dir = mkdtempSync(join(tmpdir(), "replay-scoped-")); mkdirSync(join(dir, "20260101000000-init"), { recursive: true }); writeFileSync(join(dir, "20260101000000-init", "up.sql"), upSql, "utf8"); - writeFileSync(join(dir, "20260101000000-init", "down.sql"), "DROP TABLE mine;", "utf8"); + writeFileSync(join(dir, "20260101000000-init", "down.sql"), 'DROP TABLE "mine";', "utf8"); return dir; } -const SNAPSHOT: SchemaSnapshot = { - tables: [ - { name: "mine", columns: [{ name: "id", sqlType: { kind: "int", bits: 64 } as never, nullable: false }], indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"] }, - { name: "theirs", columns: [{ name: "id", sqlType: { kind: "int", bits: 64 } as never, nullable: false }], indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"] }, - ], - views: [], -}; +// `id INTEGER NOT NULL PRIMARY KEY`, not a bare `INTEGER PRIMARY KEY`: sqlite reports +// notnull=0 for the latter, so `nullable: false` here would read as drift and the +// test would fail for a reason unrelated to scope. `test/integrity/replay.test.ts:47` +// writes it the same way, for the same reason. +const CHAIN = 'CREATE TABLE "mine" (id INTEGER NOT NULL PRIMARY KEY);'; + +const table = (name: string) => ({ + name, + columns: [{ name: "id", sqlType: { kind: "integer" as const, bits: 64 as const }, nullable: false }], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], +}); -describe("verifyReplay honours scope", () => { +// `theirs` belongs to another owner: `carryForwardOutOfScope` put it in the snapshot, +// and the chain — correctly — never creates it. +const SNAPSHOT: SchemaSnapshot = { tables: [table("mine"), table("theirs")], views: [] }; + +describe("verifyReplay honours migrate.scope", () => { test("an out-of-scope table in the snapshot is not reported as missing", async () => { - const dir = chainWith("CREATE TABLE mine (id integer primary key);"); + const dir = chainWith(CHAIN); const engine = await openReplayEngine("sqlite"); try { const result = await verifyReplay({ @@ -580,7 +1027,8 @@ describe("verifyReplay honours scope", () => { dialect: "sqlite", migrationsDir: dir, snapshot: SNAPSHOT, - governed: { outOfScope: ["theirs"] } as never, + // Qualified: `.`, absent schema normalized to `public`. + governed: { outOfScope: ["public.theirs"] }, }); expect(result.ok).toBe(true); } finally { @@ -590,7 +1038,7 @@ describe("verifyReplay honours scope", () => { }); test("without `governed`, the same case reports drift — the control", async () => { - const dir = chainWith("CREATE TABLE mine (id integer primary key);"); + const dir = chainWith(CHAIN); const engine = await openReplayEngine("sqlite"); try { const result = await verifyReplay({ @@ -606,9 +1054,7 @@ describe("verifyReplay honours scope", () => { ``` The control case is what makes the first test non-vacuous: it proves the difference comes from -`governed` and not from the fixture being trivially green. **Adjust the `SchemaSnapshot` literal and -the `GovernedScope` shape to the real types** — read `src/types.ts` and `src/scope.ts` — rather than -leaving the `as never` casts in the committed test. +`governed` and not from the fixture being trivially green. - [ ] **Step 2: Run it to verify it fails** @@ -617,17 +1063,29 @@ Expected: the first test FAILS (`governed` is not accepted / not honoured); the - [ ] **Step 3: Implement** -In `verify/replay.ts`, add the optional field to `VerifyReplayArgs` and apply -`excludeFromSnapshot` to the snapshot before comparing: +In `verify/replay.ts`, add the optional field to `VerifyReplayArgs`: + +```ts + /** + * The scope decision the run made, as `scopeExpectedSchema` reports it. A project + * declaring `migrate.scope` carries the OTHER owner's tables into the committed + * snapshot on purpose (`carryForwardOutOfScope`), and the chain never creates them — + * so without this they read as missing on every replay. Excluded from the SNAPSHOT + * side only; the replayed database never had them either. + */ + governed?: GovernedScope; +``` + +and apply the exclusion before comparing: ```ts const expected = args.governed !== undefined - ? excludeFromSnapshot(args.snapshot, args.governed) + ? excludeFromSnapshot(args.snapshot, args.governed).snapshot : args.snapshot; const classification = await driftAgainstSnapshot(expected, actual, args.dialect); ``` -Import `excludeFromSnapshot` and the `GovernedScope` type from `../scope.js`. Do not change the +Import `excludeFromSnapshot` and `type GovernedScope` from `../scope.js`. Do not change the signature's required fields — an existing caller passing no `governed` must behave exactly as before. - [ ] **Step 4: Run the test to verify it passes** @@ -643,7 +1101,7 @@ Run: `bun run --filter '*' typecheck` — all 18 exit 0. ```bash git add server/typescript/packages/migrate-ts/src/verify/replay.ts \ server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts -git commit -m "fix(migrate): verifyReplay honours migrate.scope on both sides" +git commit -m "fix(migrate): verifyReplay honours migrate.scope on the snapshot side" ``` --- @@ -653,7 +1111,7 @@ git commit -m "fix(migrate): verifyReplay honours migrate.scope on both sides" **Files:** - Modify: `server/typescript/packages/cli/src/lib/args.ts` (`VerifyFlags`, `parseVerifyArgs`) - Modify: `server/typescript/packages/cli/src/commands/verify.ts` -- Test: `server/typescript/packages/migrate-ts/test/integrity/replay-from-empty.test.ts` (create) +- Test: `server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts` (create) - Test: `server/typescript/packages/cli/test/verify-replay.test.ts` (create) **Interfaces:** @@ -666,29 +1124,104 @@ git commit -m "fix(migrate): verifyReplay honours migrate.scope on both sides" gate. Refuse `--migration-format flyway` and `--dialect d1`, mirroring `apply-pending` (`migrate.ts:419-426`, `:449-457`). -- [ ] **Step 1: Write the failing regression test — the #313 case** +### The dialect, when there is no `--db` + +`--replay` has no connection URL to infer a dialect from, and `--replay-snapshot` needs one to name +the snapshot file (`snapshotPath(dir, dialect)`). **Precedence, stated so it is not invented twice:** + +1. an explicit `--dialect`, +2. else `resolveMigrateConfig(EMPTY_MIGRATE_FLAGS, projectRoot).dialect`, +3. else refuse with **exit 2**, naming `--dialect`. + +Step 2 requires amending the comment on `EMPTY_MIGRATE_FLAGS` (`verify.ts:81-87`), which today says +verify "consumes only `outDir`… reading any of them here would be reaching into migrate's +decisions". Amend it, do not quietly contradict it: + +```ts + * `verify` consumes `outDir` (#292) and — for the replay gate only — `dialect`. The + * #292 restriction was about the DRIFT gate, whose dialect comes from the live `--db` + * URL; the replay gate has no `--db`, and the dialect a committed chain was emitted + * for IS a migrate decision, so migrate's own resolution is the only correct source. + * Everything else here exists to satisfy the shared shape. +``` + +### Zero committed migrations + +`discoverMigrations` is **not exported** from `apply/apply.ts` (it is a module-private function at +`:316`), so the gate cannot call it. Use `applyPending`'s return value instead: +`ApplyPendingResult` is `{ pending: string[]; applied: string[] }`, and on a fresh in-process +database with no ledger every discovered migration is pending — so `pending.length === 0` means the +directory held none. Report it and return 0; do not pass silently. + +- [ ] **Step 1: Write the RED-first regression — the emitter's own SQL, through `applyPending`** + +This is the test the previous draft of this plan was missing. Hand-writing the SQL proves nothing +about the emitter: such a test stays green if Task 1 is reverted. This one builds a `drop-table` +change, runs it through `emit()` and `writeMigration()`, and applies the resulting file — so it is +RED before Task 1 and GREEN after, and it is the only assertion that proves an *emitted* chain +replays. ```ts -// server/typescript/packages/migrate-ts/test/integrity/replay-from-empty.test.ts +// server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts +// +// #313, end to end: the EMITTER's output, written by writeMigration, applied by +// applyPending into an empty in-process database. Every prior defect in this area +// (#226/#241, #243, #255, #285, 0.21.4's BEGIN TRANSACTION finding) shared one shape +// — SQL proven statement-by-statement and never proven through the tool that applies +// it. applyPending rewrites statements via prepareForRunnerTransaction before +// executing, so an emit-level assertion cannot see this class of bug. import { describe, test, expect } from "bun:test"; -import { applyPending } from "../../src/apply/apply.js"; -import { openReplayEngine } from "../../src/verify/replay-engine.js"; -import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; +import { mkdtempSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { emit } from "../../src/emit/index.js"; +import { writeMigration } from "../../src/write-migration.js"; +import { applyPending } from "../../src/apply/apply.js"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import type { Change, ChangeStatus } from "../../src/types.js"; + +const ALLOWED: ChangeStatus = { state: "allowed" }; + +// Exactly the reported shape: another tool owned `theirs`, so the diff proposed +// dropping it, and no migration in the chain ever created it. +const REPORTED: Change[] = [ + { + kind: "create-table", status: ALLOWED, + table: { + name: "mine", + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], + }, + }, + { kind: "drop-table", table: "theirs", status: ALLOWED }, +]; + +describe("an EMITTED chain applies to an empty database (#313)", () => { + for (const dialect of ["sqlite", "postgres"] as const) { + test(`${dialect}: emit → writeMigration → applyPending, from empty`, async () => { + const dir = mkdtempSync(join(tmpdir(), `replay-emitted-${dialect}-`)); + const engine = await openReplayEngine(dialect); + try { + const result = emit(REPORTED, { dialect }); + await writeMigration(dir, "init", result); + const applied = await applyPending(engine.db, dir, { dryRun: false, dialect }); + expect(applied.applied).toHaveLength(1); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + } -describe("a committed chain must apply to an empty database (#313)", () => { - test("a bare DROP TABLE for an object the chain never creates fails the replay", async () => { - const dir = mkdtempSync(join(tmpdir(), "replay-empty-")); - mkdirSync(join(dir, "20260101000000-init"), { recursive: true }); - // Exactly the reported shape: another tool owned `theirs`, so the diff proposed - // dropping it, and no migration in the chain ever created it. - writeFileSync(join(dir, "20260101000000-init", "up.sql"), - 'CREATE TABLE "mine" (id integer primary key);\nDROP TABLE "theirs";', "utf8"); - writeFileSync(join(dir, "20260101000000-init", "down.sql"), 'DROP TABLE "mine";', "utf8"); - + test("the control: a HAND-WRITTEN bare drop still fails, so the assertion has teeth", async () => { + const dir = mkdtempSync(join(tmpdir(), "replay-emitted-control-")); const engine = await openReplayEngine("sqlite"); try { + await writeMigration(dir, "init", { + up: 'CREATE TABLE "mine" (id INTEGER NOT NULL PRIMARY KEY);\n\nDROP TABLE "theirs";', + down: 'DROP TABLE "mine";', + recreatedTables: new Set(), + }); await expect(applyPending(engine.db, dir, { dryRun: false, dialect: "sqlite" })) .rejects.toThrow(/theirs/); } finally { @@ -696,40 +1229,21 @@ describe("a committed chain must apply to an empty database (#313)", () => { rmSync(dir, { recursive: true, force: true }); } }); - - test("the same chain with IF EXISTS applies cleanly", async () => { - const dir = mkdtempSync(join(tmpdir(), "replay-empty-ok-")); - mkdirSync(join(dir, "20260101000000-init"), { recursive: true }); - writeFileSync(join(dir, "20260101000000-init", "up.sql"), - 'CREATE TABLE "mine" (id integer primary key);\nDROP TABLE IF EXISTS "theirs";', "utf8"); - writeFileSync(join(dir, "20260101000000-init", "down.sql"), 'DROP TABLE "mine";', "utf8"); - - const engine = await openReplayEngine("sqlite"); - try { - const result = await applyPending(engine.db, dir, { dryRun: false, dialect: "sqlite" }); - expect(result.applied).toEqual(["20260101000000-init"]); - } finally { - await engine.dispose(); - rmSync(dir, { recursive: true, force: true }); - } - }); }); ``` -This runs through `applyPending`, **not** through `emit()`. Every prior defect in this area -(#226/#241, #243, #255, #285, and 0.21.4's `BEGIN TRANSACTION` finding) shared one shape — SQL proven -statement-by-statement and never proven through the tool that applies it. `applyPending` rewrites -statements via `prepareForRunnerTransaction` before executing them, so an emit-level assertion -cannot see this class of bug. +**Adjust `writeMigration`'s call shape to the real signature** — read +`migrate-ts/src/write-migration.ts` first; it may take an options object rather than +`(dir, slug, result)`, and its `EmitResult` may carry more fields than the three above. +`ApplyPendingResult` names its list `applied`. -Adjust `result.applied` to whatever `ApplyPendingResult` actually names its applied list. +- [ ] **Step 2: Run it — it must be RED unless Task 1 has landed** -- [ ] **Step 2: Run it to verify the shape is real** - -Run: `cd server/typescript && bun test packages/migrate-ts/test/integrity/replay-from-empty.test.ts` -Expected: BOTH PASS immediately — this test characterises `applyPending`, which already behaves this -way. That is the point: it pins the mechanism the gate depends on. If the first case does NOT throw, -stop and report, because the gate cannot work. +Run: `cd server/typescript && bun test packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts` +Expected **if Task 1 has landed**: PASS, all three. +Expected **if Task 1 has not landed**: the two emitted-chain cases FAIL with `theirs`, the control +passes. Either outcome is informative — record which one you saw. If Task 1 has landed and a case +still fails, STOP: the emitter fix is incomplete. - [ ] **Step 3: Add the flag** @@ -752,7 +1266,7 @@ include it in the parsed object (`replay: !!values.replay`), and add it to `anyE const anyExplicit = templates || codegen || values.db !== undefined || dialect === "d1" || !!values.replay; ``` -- [ ] **Step 4: Write the CLI test** +- [ ] **Step 4: Write the CLI tests** ```ts // server/typescript/packages/cli/test/verify-replay.test.ts @@ -780,35 +1294,55 @@ In `cli/src/commands/verify.ts`, add a `runReplayVerify()` beside the existing g in the `Math.max`: ```ts - const replayExit = flags.replay ? await runReplayVerify() : 0; + const replayExit = (flags.replay || flags.replaySnapshot) ? await runReplayVerify() : 0; … return Math.max(templateExit, schemaExit, codegenExit, requirementExit, replayExit); ``` +**Write the condition as `(flags.replay || flags.replaySnapshot)` now, in this task**, even though +`replaySnapshot` does not exist yet — add the field in Task 6 and this line needs no second edit. +(If TypeScript objects to the missing field, add `replaySnapshot: boolean` to `VerifyFlags` here and +wire its parsing in Task 6; do not leave the condition reading `flags.replay` alone, which is how +`--replay-snapshot` would ship dead.) + `runReplayVerify` must: -1. Refuse `--migration-format flyway` and `--dialect d1` with `log.error` and **exit 2**, matching - `apply-pending`'s wording at `migrate.ts:419-426` and `:449-457`. -2. Resolve the migrations directory the same way `migrate apply-pending` does — do NOT re-derive it; - read how `migrate.ts` computes it and use the same helper. -3. Report and return 0 when the directory holds no migrations, with the exact text - `meta verify --replay: no committed migrations — nothing to replay`. `discoverMigrations` - returns `[]` for a missing directory (`apply.ts:316-322`), so silence here would be a vacuous pass. -4. `openReplayEngine(dialect)`, `applyPending(engine.db, dir, { dryRun: false, dialect })`, and - `await engine.dispose()` in a `finally`. -5. On an apply failure, print the failing statement and the remediation — an already-applied chain +1. **Resolve the dialect** by the precedence above. Refuse `d1` and `--migration-format flyway` with + `log.error` and **exit 2**, matching `apply-pending`'s wording at `migrate.ts:419-426` and + `:449-457`. Refuse with exit 2 and a message naming `--dialect` when no dialect can be resolved. +2. **Resolve the migrations directory through migrate's OWN precedence** — the same + `resolveMigrateConfig(EMPTY_MIGRATE_FLAGS, projectRoot)` → `resolvePath(projectRoot, migrateConfig.outDir)` + pair `checkCommittedSnapshot` uses (`verify.ts:639-641`). Do not re-derive it. +3. `openReplayEngine(dialect)` inside a `try`, with `await engine.dispose()` in the `finally`. + An engine that cannot start (including a missing optional peer) is **operational → return 2**, + and the error's own message already carries the install hint. +4. `applyPending(engine.db, dir, { dryRun: false, dialect })`. +5. **Zero migrations is not a silent pass.** When the result's `pending` is empty, print + `meta verify --replay: no committed migrations — nothing to replay` and return 0. +6. On an apply failure, print the failing statement and the remediation. An already-applied chain cannot be repaired by hand-editing (`apply/apply.ts:88-99` makes migrations checksum-immutable), so the message must name the compensating-migration path: `meta verify --replay: the committed chain does not apply to an empty database. Applied migrations are immutable, so fix this with a NEW migration that creates the missing object — not by editing a committed up.sql.` Return **1** (drift), not 2. -6. Return **2** only when the engine itself cannot start. -- [ ] **Step 6: Run the tests** +- [ ] **Step 6: Add a behavioural CLI test, not only a flag-parse test** + +Three flag-parse assertions do not prove the gate runs. Add one case that drives `verifyCommand` +against a temporary project whose committed chain contains a bare drop for an object it never +creates, and assert the command returns **1**. Model the project fixture on +`packages/cli/test/migrate-scope.test.ts`'s harness — read it first — but note it drives the +**offline** path (`runOfflineGenerate`/`runBaseline`); what you need here is a `metaobjects/` +directory, a `.metaobjects/config.json` declaring `migrate.dialect`, and a committed +`migrations/-init/up.sql`. No database is involved, which is the point. + +Also assert the zero-migrations case returns 0 and says so. + +- [ ] **Step 7: Run the tests** Run: `cd server/typescript && bun test packages/cli/test/verify-replay.test.ts && bun test packages/migrate-ts/test/integrity` Expected: PASS. -- [ ] **Step 7: Run the suites, typecheck, commit** +- [ ] **Step 8: Run the suites, typecheck, commit** Run: `cd server/typescript && bun test packages/cli && bun test packages/migrate-ts` — PASS. Run: `bun run --filter '*' typecheck` — all 18 exit 0. @@ -817,7 +1351,7 @@ Run: `bun run --filter '*' typecheck` — all 18 exit 0. git add server/typescript/packages/cli/src/lib/args.ts \ server/typescript/packages/cli/src/commands/verify.ts \ server/typescript/packages/cli/test/verify-replay.test.ts \ - server/typescript/packages/migrate-ts/test/integrity/replay-from-empty.test.ts + server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts git commit -m "feat(cli): meta verify --replay asserts the committed chain applies from empty" ``` @@ -831,7 +1365,8 @@ git commit -m "feat(cli): meta verify --replay asserts the committed chain appli - Test: `server/typescript/packages/cli/test/verify-replay.test.ts` (extend) **Interfaces:** -- Consumes: `openReplayEngine` (Task 3), `verifyReplay` with `governed` (Task 4), the `runReplayVerify` structure (Task 5). +- Consumes: `openReplayEngine` (Task 3), `verifyReplay` with `governed` (Task 4), the + `runReplayVerify` structure and the `(flags.replay || flags.replaySnapshot)` condition (Task 5). - Produces: nothing later tasks depend on. **Context:** This is a **separate subverb, not `--strict`**. `verify` already owns a `--lax` flag on @@ -844,7 +1379,37 @@ and would live in the target database's ledger while this gate runs against a fr database with no ledger at all. The limitation is documented in Task 8, and the failure message names it. -- [ ] **Step 1: Write the failing test** +### `governed` has to be derived OFFLINE + +`verify.ts:659` gets its `GovernedScope` from `driftResult` (`verify.ts:498`), which comes from +`computeDriftFromActual` and needs a live `--db`. This gate has none. Derive it instead: + +- `collection.inMigrateScope` (already read at `verify.ts:217` as `schemaScope`) is the predicate. +- **When it is `undefined`, pass no `governed` at all** — an unscoped project's comparison is then + byte-for-byte unchanged, which is most projects and keeps the added surface confined. +- When it is defined, build the provenance-bearing expected schema and scope it, exactly as + `verify.ts:435-441` already does inside `migrateScopeMismatch`: + ```ts + const viewStrategy = forgeConfig?.columnNamingStrategy ?? "snake_case"; + const built = buildExpectedSchemaWithProvenance(root, { + dialect, + columnNamingStrategy: viewStrategy, + views: buildProjectionViews(root, { dialect, columnNamingStrategy: viewStrategy }), + }); + const governed = scopeExpectedSchema(built, schemaScope); // satisfies GovernedScope + ``` + `scopeExpectedSchema` needs `ExpectedSchemaWithProvenance` — a snapshot plus a + qualified-name → metadata-FQN map — which is why the committed snapshot alone cannot be scoped + and the metadata must be rebuilt here. + +### One engine, one real apply + +`verifyReplay` calls `applyPending` unconditionally (`replay.ts:31`). That is **not** a second +replay: the first `applyPending` (Task 5 step 5.4) recorded every migration in the in-process +ledger, so the second call finds nothing pending and returns immediately. Share one engine, let each +tier keep its own failure message, and do not restructure `verifyReplay` to avoid the call. + +- [ ] **Step 1: Write the failing tests** Append to `packages/cli/test/verify-replay.test.ts`: @@ -866,6 +1431,33 @@ describe("verify --replay-snapshot flag", () => { }); ``` +**And the behavioural test that keeps the flag from shipping dead** — the previous draft had three +flag-parse tests and nothing that ran the gate, so `--replay-snapshot` would have parsed fine and +done nothing. Using the Task 5 Step 6 project harness: + +```ts +describe("verify --replay-snapshot actually runs the gate", () => { + test("a chain that does not apply fails under --replay-snapshot alone", async () => { + // --replay-snapshot implies --replay's work; a broken chain must fail even when + // --replay was not passed. + const project = await projectWithBrokenChain(); + expect(await verifyCommand(["--replay-snapshot"], project)).toBe(1); + }); + + test("a chain that applies but does not reproduce the snapshot fails", async () => { + const project = await projectWithChainDivergentFromSnapshot(); + expect(await verifyCommand(["--replay-snapshot"], project)).toBe(1); + }); + + test("a chain that applies and reproduces the snapshot passes", async () => { + const project = await projectWithGoodChain(); + expect(await verifyCommand(["--replay-snapshot"], project)).toBe(0); + }); +}); +``` + +Adapt the helper names and the `verifyCommand` call shape to the real harness. + - [ ] **Step 2: Run it to verify it fails** Run: `cd server/typescript && bun test packages/cli/test/verify-replay.test.ts` @@ -873,29 +1465,32 @@ Expected: FAIL — `--replay-snapshot` is not a known option (`strict: true` in - [ ] **Step 3: Implement the flag** -Same three places as Task 5: the `VerifyFlags` field (`replaySnapshot: boolean`), the `parseArgs` -option (`"replay-snapshot": { type: "boolean", default: false }`), the parsed object -(`replaySnapshot: !!values["replay-snapshot"]`), and `anyExplicit`. +Same places as Task 5: the `VerifyFlags` field (`replaySnapshot: boolean`), the `parseArgs` option +(`"replay-snapshot": { type: "boolean", default: false }`), the parsed object +(`replaySnapshot: !!values["replay-snapshot"]`), and `anyExplicit`. **Verify that Task 5's +`(flags.replay || flags.replaySnapshot)` condition is present and now reachable** — this is the exact +line whose absence would ship the flag dead. - [ ] **Step 4: Implement the tier** Extend `runReplayVerify` so that when `flags.replaySnapshot` is set it additionally: -1. Loads the committed snapshot the same way the existing committed-snapshot gate does — read - `verify.ts` around `:628-660` and reuse that path, do not re-implement snapshot loading. -2. Calls `verifyReplay({ db: engine.db, dialect, migrationsDir, snapshot, governed })`, where - `governed` is what `scopedDiffInputs`/`excludeFromSnapshot` already produce at `verify.ts:659`. -3. On `ok === false`, reports the drift and returns **1**, with a message that names baseline +1. Loads the committed snapshot the way `checkCommittedSnapshot` does (`verify.ts:639-644`): + `readSnapshot(snapshotPath(dir, dialect))` on the already-resolved `dir`, inside a `try`. + **Fail OPEN**: a `null` snapshot, or an unreadable/unparseable file, reports + `meta verify --replay-snapshot: no committed snapshot — nothing to compare` and contributes 0. + A project that has never generated one offline is not in an error state, and a parse failure is + migrate's error to raise with its own message, not a drift verdict. +2. Derives `governed` per the section above. +3. Calls `verifyReplay({ db: engine.db, dialect, migrationsDir: dir, snapshot, ...(governed !== undefined ? { governed } : {}) })`. +4. On `ok === false`, reports the drift and returns **1**, with a message that names baseline adoption as the first thing to rule out: `meta verify --replay-snapshot: the replayed chain does not reproduce the committed snapshot. If this project was adopted with 'migrate baseline --from-db', its chain does not build the schema and this tier does not apply — use --replay instead.` -Both tiers share one engine and one `applyPending` call: `--replay-snapshot` implies `--replay`'s -work, so do not open two engines or replay twice. - - [ ] **Step 5: Run the tests** Run: `cd server/typescript && bun test packages/cli/test/verify-replay.test.ts` -Expected: PASS, all six. +Expected: PASS. - [ ] **Step 6: Run the suites, typecheck, commit** @@ -914,7 +1509,10 @@ git commit -m "feat(cli): meta verify --replay-snapshot asserts the chain reprod ## Task 7: Emit-time provenance guard **Files:** +- Modify: `server/typescript/packages/migrate-ts/src/types.ts` (`AllowOptions.dropUnmanaged`) - Modify: `server/typescript/packages/cli/src/lib/args.ts` (`ALLOW_TOKENS`) +- Modify: `server/typescript/packages/cli/src/lib/allow.ts` (`ALLOW_TOKEN_MAP`) +- Modify: `server/typescript/packages/sdk/src/config.ts` (`AllowTokenEnum`) - Modify: `server/typescript/packages/cli/src/commands/migrate.ts` - Test: `server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts` (create) @@ -928,45 +1526,94 @@ committed snapshot**, which is why an object no snapshot ever contained gets pro `drift/classify.ts:6-9` already states the doctrine: objects present in the DB but not the snapshot "must never be treated as actionable drift or auto-dropped". -The guard does not false-fire on the brownfield classes, and the reason is that both of them *add* -to the snapshot: a `baseline --from-db` snapshot contains the foreign table, and a scoped project -carries out-of-scope entries forward into it. The guard fires precisely when nothing ever claimed -the object. +The guard does not false-fire on the brownfield classes, because both of them *add* to the snapshot: +a `baseline --from-db` snapshot contains the foreign table, and a scoped project carries out-of-scope +entries forward into it. The guard fires precisely when nothing ever claimed the object. -- [ ] **Step 1: Write the failing test** +### A new `--allow` token touches FOUR files, and a pin test will catch three of them + +`cli/test/unit/allow-tokens-pinned.test.ts` asserts three invariants, all self-deriving (so it needs +no edit, and it fails loudly if any file is missed): + +1. `ALLOW_TOKENS` (`cli/src/lib/args.ts:173`) ≡ `AllowTokenEnum.options` (`sdk/src/config.ts:22`) — + the sdk enum validates `migrate.allow` in `.metaobjects/config.json`. +2. `ALLOW_TOKEN_MAP` (`cli/src/lib/allow.ts:13`) has exactly one key per token — the map is what + *grants* the permission; a token in the list but not the map validates cleanly and silently grants + nothing. +3. Every mapped value is a **distinct** `AllowOptions` field. + +**Decision: add `dropUnmanaged?: boolean` to `AllowOptions`** (`migrate-ts/src/types.ts:300`) and +wire the token through all four structures. The alternative — keeping `drop-unmanaged` out of +`ALLOW_TOKENS` and validating it separately — means a second token list and a second parse path for +exactly one token, which is precisely the drift the pin test exists to prevent. The cost is that one +`AllowOptions` field is read by the CLI rather than by `diff()`'s status pass; document that at the +field, so it is a stated exception rather than a puzzle. + +- [ ] **Step 1: Write the failing tests** ```ts // server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts import { describe, test, expect } from "bun:test"; import { ALLOW_TOKENS } from "../src/lib/args.js"; +import { ALLOW_TOKEN_MAP } from "../src/lib/allow.js"; +import { tokensToAllowOptions } from "../src/lib/allow.js"; describe("drop-unmanaged allow token", () => { test("is a recognised allow token", () => { expect(ALLOW_TOKENS).toContain("drop-unmanaged"); }); + + test("grants a permission rather than validating into nothing", () => { + expect(ALLOW_TOKEN_MAP["drop-unmanaged"]).toBe("dropUnmanaged"); + expect(tokensToAllowOptions(["drop-unmanaged"]).dropUnmanaged).toBe(true); + }); }); ``` -Then add the behavioural test. It must drive the real `migrate` path with a snapshot that does NOT -contain the table being dropped, and assert the run refuses. **Model it on the existing -`packages/cli/test/migrate-scope.test.ts`**, which already builds a config + snapshot + change set -for this command — read it first and follow its harness rather than inventing one. +Then the behavioural tests. They must drive the real live `migrate` path with a committed snapshot +that does NOT contain the table being dropped, and assert the run refuses. Three cases: -The two cases: -1. A drop proposed for a table absent from the committed snapshot ⇒ refused, exit 2, message names - the object and `--allow drop-unmanaged`. +1. A drop proposed for a table absent from the committed snapshot ⇒ refused, **exit 2**, message + names the object and `--allow drop-unmanaged`. 2. The same run with `--allow drop-unmanaged` ⇒ proceeds. - -And one non-false-fire case: a drop for a table that IS in the snapshot ⇒ proceeds without the flag. +3. **Non-false-fire:** a drop for a table that IS in the snapshot ⇒ proceeds without the flag. +4. **Fail open:** the same run with NO snapshot on disk ⇒ proceeds without the flag. + +**The harness is the hard part, and `migrate-scope.test.ts` is not it.** That file drives the +OFFLINE path (`runOfflineGenerate`/`runBaseline`), where the expected side already *is* the snapshot, +so the guard can never fire there. The guard's target is the **live** path (`migrate.ts:607-620`), +which needs a real database. Use the `:memory:` libsql engine from Task 3 — or, if Task 7 runs before +Task 3, `LibsqlDialect({ url: ":memory:" })` directly (verified working) — seed it with the tables the +scenario needs, and point `meta migrate --db` at it. Read +`packages/migrate-ts/test/integration/` for how the existing live-path tests build a database and a +project side by side. - [ ] **Step 2: Run to verify it fails** Run: `cd server/typescript && bun test packages/cli/test/migrate-drop-unmanaged.test.ts` Expected: FAIL — the token is not in `ALLOW_TOKENS`. -- [ ] **Step 3: Add the token** +- [ ] **Step 3: Add the token — all four files** -In `cli/src/lib/args.ts`, add to `ALLOW_TOKENS` with a comment in the style of its neighbours: +`migrate-ts/src/types.ts`, in `AllowOptions`: + +```ts + /** + * Permits dropping an object the COMMITTED SNAPSHOT never contained — i.e. one + * this toolchain never managed. Without it such a drop is refused at generation + * time, because it produces a migration that cannot replay against a database + * where that object never existed (#313). + * + * The one field here read by the CLI's generation-time provenance guard rather + * than by `diff()`'s status pass: it lives in `AllowOptions` so `--allow` keeps + * ONE token list and ONE grant map (`ALLOW_TOKENS` / `ALLOW_TOKEN_MAP`, pinned by + * `cli/test/unit/allow-tokens-pinned.test.ts`). A second parallel validation path + * for a single token is the drift that pin exists to prevent. + */ + dropUnmanaged?: boolean; +``` + +`cli/src/lib/args.ts`, in `ALLOW_TOKENS`, with a comment in the style of its neighbours: ```ts // drop-unmanaged permits dropping an object the COMMITTED SNAPSHOT never @@ -976,30 +1623,68 @@ In `cli/src/lib/args.ts`, add to `ALLOW_TOKENS` with a comment in the style of i "drop-unmanaged", ``` +`cli/src/lib/allow.ts`, in `ALLOW_TOKEN_MAP`: + +```ts + // Read by migrate's generation-time provenance guard, not by diff()'s status pass + // — see AllowOptions.dropUnmanaged for why it still lives in that shape. + "drop-unmanaged": "dropUnmanaged", +``` + +`sdk/src/config.ts`, in `AllowTokenEnum`: + +```ts + "drop-unmanaged", +``` + - [ ] **Step 4: Implement the guard** -In `migrate.ts`, after the diff produces its change list and BEFORE the migration is written, -collect every `drop-table` / `drop-view` whose name is absent from the committed snapshot, and if -that set is non-empty and `allow.dropUnmanaged` is not set, `log.error` naming each object and -return 2. +In `cli/src/commands/migrate.ts`, in the **live** path, immediately after +`changeCounts = summarizeChanges(diffResult.changes);` (`migrate.ts:657`) and **before** the +`emit(...)` block: -Read how `allow` tokens are converted to the options object (`tokensToAllowOptions`) and follow it. -Use the same qualified-name helper the scope machinery uses (`qualifiedDbName` in -`migrate-ts/src/qualified-name.ts`) so the guard and the snapshot agree on a name's spelling — three -independent sets already have to agree there, and a fourth spelling would silently un-guard objects. +```ts + // #313 — refuse to author a drop for an object the committed snapshot never + // contained. The live path diffs metadata against introspection and never reads + // the snapshot, so an object another tool owns reads as "in the DB, not in the + // model" and is proposed for a drop; the resulting migration then cannot replay + // against a database where the object never existed. `classify.ts:6-9` already + // states the doctrine — this is where it is enforced. + // + // Fails OPEN when there is no snapshot on disk: a project that has never + // generated one is not in an error state, and refusing there would break the + // first `meta migrate` of every greenfield project. +``` + +The implementation: + +1. Load the snapshot the way `migrate.ts:781-782` already spells it: + `readSnapshot(snapshotPath(resolvePath(metaRoot, config.outDir), kysely.dialect))`, inside a + `try`. A thrown read, or a `null` result, **skips the guard entirely**. +2. Build the set of snapshot object names with **`qualifiedDbName`** (`migrate-ts/src/qualified-name.ts`) + over `snapshot.tables` and `snapshot.views`. Use that helper and nothing else — three independent + sets already have to agree on this spelling, and a fourth would silently un-guard objects. +3. Collect every `drop-table` / `drop-view` in `diffResult.changes` whose `qualifiedDbName({ name, schema })` + is absent from that set. (`drop-table` carries `table: string` + `schema?`; `drop-view` carries + `view: string` + `schema?`.) +4. If that set is non-empty and `tokensToAllowOptions(config.allow).dropUnmanaged` is not true, + `log.error` naming each object and `--allow drop-unmanaged`, close the connection, and return 2. - [ ] **Step 5: Run the tests** -Run: `cd server/typescript && bun test packages/cli/test/migrate-drop-unmanaged.test.ts` -Expected: PASS. +Run: `cd server/typescript && bun test packages/cli/test/migrate-drop-unmanaged.test.ts && bun test packages/cli/test/unit/allow-tokens-pinned.test.ts` +Expected: PASS. The pin test passing is the evidence all four files were updated. - [ ] **Step 6: Run the suites, typecheck, commit** -Run: `cd server/typescript && bun test packages/cli && bun test packages/migrate-ts` — PASS. +Run: `cd server/typescript && bun test packages/cli && bun test packages/migrate-ts && bun test packages/sdk` — PASS. Run: `bun run --filter '*' typecheck` — all 18 exit 0. ```bash -git add server/typescript/packages/cli/src/lib/args.ts \ +git add server/typescript/packages/migrate-ts/src/types.ts \ + server/typescript/packages/cli/src/lib/args.ts \ + server/typescript/packages/cli/src/lib/allow.ts \ + server/typescript/packages/sdk/src/config.ts \ server/typescript/packages/cli/src/commands/migrate.ts \ server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts git commit -m "feat(cli): refuse to drop an object the committed snapshot never managed" @@ -1026,34 +1711,51 @@ point at `meta verify --replay` as the way to know your chain is one of those. - [ ] **Step 2: Document both tiers** In the same file, document `meta verify --replay` and `--replay-snapshot`: what each asserts, that -they run in-process (PGlite / `:memory:` libsql) and provision nothing, that flyway and d1 are -refused, and — stated plainly — that `--replay-snapshot` does not apply to a project adopted with -`migrate baseline --from-db`, because such a chain does not build the schema. +they run in-process (PGlite / `:memory:` libsql) and provision nothing, that PGlite is an **optional +peer** a postgres project installs to use the gate, that flyway and d1 are refused, how the dialect +is resolved without `--db`, and — stated plainly — that `--replay-snapshot` does not apply to a +project adopted with `migrate baseline --from-db`, because such a chain does not build the schema. - [ ] **Step 3: Document the guard** Document `--allow drop-unmanaged`: what triggers the refusal, why (a migration that cannot replay), -and that the escape hatch exists for a drop you genuinely intend. +that it fails open when there is no committed snapshot, and that the escape hatch exists for a drop +you genuinely intend. - [ ] **Step 4: Update the CLI help** Add the two subverbs to `verify`'s help and to the one-line note at `verify.ts:127-130` that -advertises the explicit subverbs. Update `migrate --help`'s `apply-pending` line so it no longer -promises fresh-database provisioning unconditionally. +advertises the explicit subverbs. Add `drop-unmanaged` to `migrate --help`'s `--allow` token list. +Update `migrate --help`'s `apply-pending` line (`migrate.ts:69-70`) so it no longer promises +fresh-database provisioning unconditionally. - [ ] **Step 5: CHANGELOG** -Add an `## [Unreleased]` entry covering the adopter-visible changes: emitted forward drops now carry -`IF EXISTS`; a chain creating a table in a non-default schema now emits `CREATE SCHEMA IF NOT -EXISTS`; two new verify subverbs; and a new refusal that requires `--allow drop-unmanaged`. The last -is the one that can fail an existing project's `meta migrate`, so it leads. +Add an `## [Unreleased]` entry covering the adopter-visible changes, in this order: + +1. **The new refusal** — a `meta migrate` that proposes dropping an object the committed snapshot + never contained now exits 2 and requires `--allow drop-unmanaged`. This leads because it is the + one change that can fail an existing project's `meta migrate`. +2. Emitted **forward** drops now carry `IF EXISTS` (`drop-table`, `drop-view`, `drop-index` incl. the + constraint-backed arm, `drop-fk`, `drop-check`); downs are unchanged, and D1 inherits the sqlite + change. Note the deliberate exclusions. +3. A chain creating a table **or view** in a non-default schema now emits `CREATE SCHEMA IF NOT EXISTS`. +4. Two new verify subverbs, `--replay` and `--replay-snapshot`, with `@electric-sql/pglite` as a new + **optional peer** of `@metaobjectsdev/migrate-ts` (only needed to replay a postgres chain). - [ ] **Step 6: Leak scan and commit** +The committed hook is the mechanism — do not invent a pattern list. Confirm it is wired, then let it +run on commit: + ```bash -grep -rniE "/home/|party" docs/features/migrations-and-drift.md CHANGELOG.md && echo LEAK || echo clean +git config core.hooksPath # must print .githooks +git config hooks.denyListPath # must print a path that exists +grep -rnE '/home/[a-z]' docs/features/migrations-and-drift.md CHANGELOG.md && echo "ABSOLUTE HOME PATH" || echo "no home paths" ``` +If the hook blocks the commit, **genericize** — never `--no-verify`. + ```bash git add docs/features/migrations-and-drift.md CHANGELOG.md \ server/typescript/packages/cli/src/commands/migrate.ts \ @@ -1065,23 +1767,34 @@ git commit -m "docs: replay tiers, the drop-unmanaged refusal, and the provision ## Self-Review -**Spec coverage.** §3.1 forward drops → Task 1. §3.1's deliberate exclusions (`postgres.ts:431`, -the rebuild drops, `sqlite.ts:275`) → Task 1 Steps 3–4 and its "down statements stay bare" tests. -§3.2 both tiers, engine, refusals, zero-migrations, exit codes → Tasks 3, 5, 6. §3.2's -`excludeFromSnapshot` threading → Task 4. §3.3 `CREATE SCHEMA` → Task 2. §3.4 provenance guard → -Task 7. §3.5 docs → Task 8. §4 remediation → Task 5 Step 5's message text and Task 8. §5 testing → -each task's own steps, with the `applyPending`-not-`emit()` requirement in Task 5 Step 1. +**Spec coverage.** §3.1 forward drops → Task 1, with every site tabulated and every deliberate +exclusion pinned by a test that must stay green. §3.2 both tiers, engine, refusals, +zero-migrations, exit codes → Tasks 3, 5, 6. §3.2's `excludeFromSnapshot` threading → Task 4. +§3.3 `CREATE SCHEMA` → Task 2, extended to `create-view` because the spec says "the first object". +§3.4 provenance guard → Task 7. §3.5 docs → Task 8. §4 remediation → Task 5 Step 5.6's message text +and Task 8. §5 testing → each task's own steps; §5's "must run through `applyPending`, not `emit()`" +is Task 5 Step 1, which now runs through **both** — `emit()` → `writeMigration()` → `applyPending()` — +so it is RED before Task 1 rather than green regardless. + +**What was verified in code before this revision, not assumed.** The `drop-check` "unreachable" +comment is false (`diff/index.ts:579`, `:592`). `sqlType` is `{ kind: "integer"; bits: 32|64 }`, and +`GovernedScope` names are `.`. `excludeFromSnapshot` returns a `ScopedExpectedSchema`, +so the fix takes `.snapshot`. `discoverMigrations` is module-private, so zero-migrations is detected +from `ApplyPendingResult.pending`. `migrate-ts` has two dependencies and no driver, so PGlite (22 MB) +is an optional peer and `build:binary` needs an `--external`. The PGlite→kysely pool shim in Task 3 +was **executed** against 0.3.16 and 0.5.5 — DDL, schemas, CHECK, transactions, rollback, advisory +locks, and `table "theirs" does not exist`. `LibsqlDialect({ url: ":memory:" })` works and two +instances are isolated. `pg-constraint-backed-index-285.test.ts` needs **two** edits, not one: `:143` +goes red, `:142` goes vacuously green. **Two things deliberately left to the implementer, both flagged inline rather than guessed:** the -exact Kysely adapter wiring for PGlite (Task 3 Step 4 says to mirror `cli/src/lib/kysely.ts`), and -the `migrate` test harness for the guard (Task 7 Step 1 says to follow `migrate-scope.test.ts`). -Inventing either from memory would put wrong code in the plan, which is worse than naming the file -to copy. - -**Type consistency.** `openReplayEngine(dialect) → ReplayEngine { db, dispose }` is defined in Task -3 and used verbatim in Tasks 4, 5, 6. `VerifyReplayArgs.governed` is defined in Task 4 and consumed -in Task 6. `replay` / `replaySnapshot` are added in Tasks 5 and 6 and both feed `anyExplicit`. -`drop-unmanaged` is added to `ALLOW_TOKENS` in Task 7 and referenced nowhere earlier. - -**Known ordering constraint:** Task 4's test imports `openReplayEngine`, so Task 3 must land first. -Tasks 1, 2 and 7 are independent of the rest and of each other. +exact `writeMigration` call shape (Task 5 Step 1), and the live-path `migrate` test harness for the +guard (Task 7 Step 1, which also records that `migrate-scope.test.ts` is the WRONG model because it +drives the offline path). + +**Type consistency.** `openReplayEngine(dialect) → ReplayEngine { db, dispose }` is defined in Task 3 +and used verbatim in Tasks 4, 5, 6. `VerifyReplayArgs.governed?: GovernedScope` is defined in Task 4 +and consumed in Task 6. `replay` / `replaySnapshot` are added in Tasks 5 and 6, both feed +`anyExplicit`, and the `Math.max` condition covering both is written **once**, in Task 5. +`AllowOptions.dropUnmanaged` is added in Task 7 alongside all three token structures the pin test +compares. From 7114a0be93881fd2a20e5a6f7ac05c873de9d080 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 07:23:27 -0400 Subject: [PATCH 07/44] fix(migrate): forward drops tolerate an absent object so a chain can replay MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `meta migrate` writes a bare `DROP TABLE "x"` when x is present in the live database and absent from metadata — even when no migration in the chain ever created x, which is what happens when another tool owns the table. Replaying that chain against an empty database dies with `table "x" does not exist` (#313). The reporter's chain was broken for three months. Every FORWARD drop now carries `IF EXISTS`, in both dialects: `drop-table`, `drop-view` (plain and CASCADE), `drop-index` (both the plain arm and #285's constraint-backed `ALTER TABLE ... DROP CONSTRAINT`), `drop-fk` and `drop-check`. Three exclusions, each deliberate and now pinned by a test rather than remembered: - DOWN statements stay bare. `rollbackTo` runs down.sql and the ledger delete in ONE transaction, so a guarded down would no-op and still record the rollback as done. Rollback is where a loud failure earns its keep. - The sqlite recreate-and-copy rebuild's `DROP TABLE`, and d1-cascade's, stay bare: each drops a table the same recipe just INSERT…SELECTed from, where IF EXISTS converts a caught corruption into a silent one. - `drop-column` stays unguarded — sqlite has no `DROP COLUMN IF EXISTS`, so guarding Postgres alone would make the same declared change behave differently per dialect. The emit-time provenance guard covers it instead. `drop-fk`/`drop-check` are Postgres-only and that is NOT a dialect split: sqlite emits no standalone statement for either kind (renderUpNative throws), because its constraints are create-time-only and inline, so the change folds into a recreate that rebuilds from the EXPECTED descriptor and never names the dropped constraint. Sqlite was already replay-safe there; this makes the two dialects agree. Sqlite's forward `drop-view`/`replace-view` were already guarded, so the sqlite change is two lines. Also corrects a false comment: `emit/postgres.ts` claimed add-check/drop-check were "declared but NOT yet produced by the diff". `diff/index.ts:579` and `:592` both push drop-check, an evolved `field.enum @values` is a live producer, and two tests already asserted the emitted statement. `emit/d1.ts` renders through renderSqlite, so D1's committed migrations change too — independently correct, and D1 keeps the apply-pending refusal it has. Test churn: eight assertions pinned the exact bare statement and are updated to the guarded form. One needed more than a token swap — pg-constraint-backed-index-285's NEGATIVE assertion (`not.toMatch(/DROP INDEX "?work_item…/)`) would have gone VACUOUSLY green, since IF EXISTS stops it matching for a reason unrelated to #285 and it would keep passing if #285 fully regressed. Re-anchored to `(IF EXISTS )?` and verified by emitting the regressed shape and watching the old regex fail to catch it while the new one does. Co-Authored-By: Claude Opus 5 (1M context) --- .../packages/migrate-ts/src/emit/postgres.ts | 41 +++-- .../packages/migrate-ts/src/emit/sqlite.ts | 11 +- .../check-evolution/drop-check-down.test.ts | 3 +- .../test/check/emit-postgres-check.test.ts | 4 +- .../test/emit-drop-if-exists.test.ts | 152 ++++++++++++++++++ .../pg-constraint-backed-index-285.test.ts | 12 +- .../test/unit/emit-postgres.test.ts | 7 +- .../migrate-ts/test/unit/emit-sqlite.test.ts | 7 +- 8 files changed, 213 insertions(+), 24 deletions(-) create mode 100644 server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts diff --git a/server/typescript/packages/migrate-ts/src/emit/postgres.ts b/server/typescript/packages/migrate-ts/src/emit/postgres.ts index db4ec92ef..fa24ea061 100644 --- a/server/typescript/packages/migrate-ts/src/emit/postgres.ts +++ b/server/typescript/packages/migrate-ts/src/emit/postgres.ts @@ -63,7 +63,14 @@ export function renderPostgres(changes: Change[]): EmitResult { function renderUp(c: Change): string { switch (c.kind) { case "create-table": return renderCreateTable(c.table); - case "drop-table": return `DROP TABLE ${quoteQualified(c.table, c.schema)};`; + // #313 — every FORWARD drop is `IF EXISTS`. A committed chain must apply to a + // VIRGIN database, and the diff legitimately proposes dropping an object that + // exists in the live DB but was never created by any migration in the chain (a + // table another tool owns, say). Bare, that statement kills the replay with + // `table "x" does not exist`. The DOWN renderer below is deliberately NOT + // guarded: `rollbackTo` runs down.sql and the ledger delete in ONE transaction, + // so a no-op down would still record the rollback as done. + case "drop-table": return `DROP TABLE IF EXISTS ${quoteQualified(c.table, c.schema)};`; case "rename-table": return `ALTER TABLE ${quoteQualified(c.from, c.schema)} RENAME TO ${quote(c.to)};`; case "add-column": { const base = `ALTER TABLE ${quoteQualified(c.table, c.schema)} ADD COLUMN ${renderColumn(c.column)};`; @@ -90,18 +97,28 @@ function renderUp(c: Change): string { // descriptor (both diff producers populate it), which is where the marker lives. // Matters broadly, not marginally: Drizzle's `unique()` emits constraints, so every // schema adopted from Drizzle has constraint-backed unique indexes. + // Both arms carry the #313 `IF EXISTS`: they are two renderings of the SAME + // `drop-index` change, and guarding one would leave the change kind half-covered. case "drop-index": return c.restore?.constraint !== undefined - ? `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT ${quote(c.index)};` - : `DROP INDEX ${quoteIndexQualified(c.index, c.schema)};`; + ? `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.index)};` + : `DROP INDEX IF EXISTS ${quoteIndexQualified(c.index, c.schema)};`; case "add-fk": return renderAddFk(c.table, c.schema, c.fk); - case "drop-fk": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT ${quote(c.fk)};`; - // add-check / drop-check are declared but NOT yet produced by the diff — - // checks are create-time-only (inlined in CREATE TABLE via renderCreateTable). - // These arms exist for future existing-table CHECK evolution support, mirroring - // the create-view/drop-view "declared, not yet produced" pattern. + case "drop-fk": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.fk)};`; + // `drop-check` IS produced by the diff — diff/index.ts:579 and :592 both push it, + // and an evolved `field.enum @values` is a live producer. (A comment here used to + // claim these arms were unreachable "declared, not yet produced" stubs; that was + // false, and two tests already asserted the emitted statement.) `add-check` is the + // paired ADD and rides the same passes. + // + // `drop-fk`/`drop-check` are guarded on Postgres ONLY, and that is not a dialect + // split: SQLite emits no standalone statement for either kind — `renderUpNative` + // throws, because SQLite constraints are create-time-only and inline, so the change + // folds into a table recreate that rebuilds from the EXPECTED descriptor and never + // references the dropped constraint. SQLite is already replay-safe by construction; + // guarding Postgres makes the two dialects agree. case "add-check": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} ADD CONSTRAINT ${quote(c.check.name)} CHECK (${c.check.expression});`; - case "drop-check": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT ${quote(c.check)};`; + case "drop-check": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.check)};`; case "create-view": return renderCreateView(c.view, c.schema, /* orReplace */ false); case "drop-view": return renderDropView(c); case "replace-view": return renderCreateView(c.view, c.schema, /* orReplace */ true); @@ -372,7 +389,9 @@ function renderViewComment(qualifiedView: string, comment: string | null): strin function renderDropView(c: Extract): string { const qualified = quoteQualifiedView(c.view, c.schema); const dependents = c.dependents ?? []; - if (dependents.length === 0) return `DROP VIEW ${qualified};`; + // #313 `IF EXISTS` on both forms — this is the FORWARD renderer. `renderRestoreView` + // below stays bare: it is reached only from `renderDown`. + if (dependents.length === 0) return `DROP VIEW IF EXISTS ${qualified};`; const listed = dependents .map((d) => `-- ${d.schema}.${d.name} (${d.relkind === "m" ? "materialized view" : "view"})`) @@ -385,7 +404,7 @@ function renderDropView(c: Extract): string { "-- restore them:", listed, rule, - `DROP VIEW ${qualified} CASCADE;`, + `DROP VIEW IF EXISTS ${qualified} CASCADE;`, ].join("\n"); } diff --git a/server/typescript/packages/migrate-ts/src/emit/sqlite.ts b/server/typescript/packages/migrate-ts/src/emit/sqlite.ts index 958dba81f..ba022b508 100644 --- a/server/typescript/packages/migrate-ts/src/emit/sqlite.ts +++ b/server/typescript/packages/migrate-ts/src/emit/sqlite.ts @@ -216,13 +216,20 @@ function renderRecreate( function renderUpNative(c: Change): string { switch (c.kind) { case "create-table": return renderCreateTable(c.table); - case "drop-table": return `DROP TABLE ${quote(c.table)};`; + // #313 — FORWARD drops are `IF EXISTS` so a committed chain applies to a VIRGIN + // database: the diff legitimately proposes dropping an object present in the live + // DB that no migration in the chain ever created. `renderDownNative` stays bare + // (a no-op rollback would still be recorded as done), and so does the + // recreate-and-copy rebuild's DROP above — that one drops a table the same recipe + // just INSERT…SELECTed from, where IF EXISTS turns a caught corruption into a + // silent one. + case "drop-table": return `DROP TABLE IF EXISTS ${quote(c.table)};`; case "rename-table": return `ALTER TABLE ${quote(c.from)} RENAME TO ${quote(c.to)};`; case "add-column": return `ALTER TABLE ${quote(c.table)} ADD COLUMN ${renderColumnInline(c.column)};`; case "drop-column": return `ALTER TABLE ${quote(c.table)} DROP COLUMN ${quote(c.column)};`; case "rename-column": return `ALTER TABLE ${quote(c.table)} RENAME COLUMN ${quote(c.from)} TO ${quote(c.to)};`; case "add-index": return renderCreateIndex(c.table, c.index); - case "drop-index": return `DROP INDEX ${quote(c.index)};`; + case "drop-index": return `DROP INDEX IF EXISTS ${quote(c.index)};`; case "add-check": case "drop-check": case "change-column-type": diff --git a/server/typescript/packages/migrate-ts/test/check-evolution/drop-check-down.test.ts b/server/typescript/packages/migrate-ts/test/check-evolution/drop-check-down.test.ts index 76b574e2a..485adb6b0 100644 --- a/server/typescript/packages/migrate-ts/test/check-evolution/drop-check-down.test.ts +++ b/server/typescript/packages/migrate-ts/test/check-evolution/drop-check-down.test.ts @@ -9,7 +9,8 @@ describe("drop-check: restore down + allow gating", () => { test("drop-check with restore → down re-adds the constraint", () => { const c = { kind: "drop-check", table: "orders", check: CHK.name, restore: CHK, status: { state: "allowed" } } as unknown as Change; const r = emit([c], { dialect: "postgres" }); - expect(r.up).toContain(`ALTER TABLE "orders" DROP CONSTRAINT "orders_qty_numeric_chk";`); + // #313 — the forward drop carries IF EXISTS; the down (asserted below) stays bare. + expect(r.up).toContain(`ALTER TABLE "orders" DROP CONSTRAINT IF EXISTS "orders_qty_numeric_chk";`); expect(r.down).toContain(`ALTER TABLE "orders" ADD CONSTRAINT "orders_qty_numeric_chk" CHECK (qty >= 1);`); }); test("drop-check is blocked unless allow.dropCheck", () => { diff --git a/server/typescript/packages/migrate-ts/test/check/emit-postgres-check.test.ts b/server/typescript/packages/migrate-ts/test/check/emit-postgres-check.test.ts index 91a9bc7f6..52e03a2d6 100644 --- a/server/typescript/packages/migrate-ts/test/check/emit-postgres-check.test.ts +++ b/server/typescript/packages/migrate-ts/test/check/emit-postgres-check.test.ts @@ -22,6 +22,8 @@ describe("emit postgres — checks", () => { }); test("drop-check → ALTER TABLE DROP CONSTRAINT", () => { const r = emit([{ kind: "drop-check", table: "orders", check: "orders_status_chk", status: ALLOWED } as unknown as Change], { dialect: "postgres" }); - expect(r.up).toContain(`ALTER TABLE "orders" DROP CONSTRAINT "orders_status_chk";`); + // #313 — the forward drop carries IF EXISTS. The add-check test above asserts the + // matching DOWN, which stays bare; the two together pin the direction split. + expect(r.up).toContain(`ALTER TABLE "orders" DROP CONSTRAINT IF EXISTS "orders_status_chk";`); }); }); diff --git a/server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts b/server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts new file mode 100644 index 000000000..c45ac7c59 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts @@ -0,0 +1,152 @@ +// Forward drops must tolerate an absent object so a committed chain replays into +// an empty database (#313). Down statements must NOT — `rollbackTo` runs down.sql +// and the ledger delete in one transaction, so a silently-no-op down would record +// the rollback as done. +import { describe, test, expect } from "bun:test"; +import { renderPostgres } from "../src/emit/postgres.js"; +import { renderSqlite } from "../src/emit/sqlite.js"; +import type { ChangeStatus, SchemaSnapshot, TableDescriptor } from "../src/types.js"; + +const ALLOWED: ChangeStatus = { state: "allowed" }; + +const GONE: TableDescriptor = { + name: "gone", + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], +}; + +describe("forward drops tolerate an absent object (#313)", () => { + test("postgres drop-table", () => { + const { up } = renderPostgres([{ kind: "drop-table", table: "gone", status: ALLOWED }]); + expect(up).toContain('DROP TABLE IF EXISTS "gone";'); + }); + + test("postgres drop-view", () => { + const { up } = renderPostgres([{ kind: "drop-view", view: "v_gone", status: ALLOWED }]); + expect(up).toContain('DROP VIEW IF EXISTS "v_gone";'); + }); + + test("postgres drop-index, plain", () => { + const { up } = renderPostgres([ + { kind: "drop-index", table: "t", index: "idx_gone", status: ALLOWED }, + ]); + expect(up).toContain('DROP INDEX IF EXISTS "idx_gone";'); + }); + + test("postgres drop-index, constraint-backed (#285)", () => { + const { up } = renderPostgres([ + { + kind: "drop-index", + table: "t", + index: "uq_gone", + status: ALLOWED, + restore: { name: "uq_gone", columns: ["a"], unique: true, constraint: "unique" }, + }, + ]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "uq_gone";'); + }); + + test("postgres drop-fk", () => { + const { up } = renderPostgres([{ kind: "drop-fk", table: "t", fk: "fk_gone", status: ALLOWED }]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "fk_gone";'); + }); + + // drop-check IS produced by the diff (diff/index.ts:579, :592) — an evolved + // `field.enum @values` is a live producer. The `renderUp` comment that claimed + // otherwise is deleted by this change. + test("postgres drop-check", () => { + const { up } = renderPostgres([ + { kind: "drop-check", table: "t", check: "t_qty_chk", status: ALLOWED }, + ]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "t_qty_chk";'); + }); + + test("sqlite drop-table", () => { + const { up } = renderSqlite([{ kind: "drop-table", table: "gone", status: ALLOWED }]); + expect(up).toContain('DROP TABLE IF EXISTS "gone";'); + }); + + test("sqlite drop-index", () => { + const { up } = renderSqlite([ + { kind: "drop-index", table: "t", index: "idx_gone", status: ALLOWED }, + ]); + expect(up).toContain('DROP INDEX IF EXISTS "idx_gone";'); + }); + + // Already guarded before this change; pinned so a later sweep cannot un-guard it. + test("sqlite drop-view was already guarded", () => { + const { up } = renderSqlite([{ kind: "drop-view", view: "v_gone", status: ALLOWED }]); + expect(up).toContain('DROP VIEW IF EXISTS "v_gone";'); + }); +}); + +describe("down statements stay bare — a rollback must fail loudly", () => { + test("postgres create-table down", () => { + const { down } = renderPostgres([{ kind: "create-table", table: GONE, status: ALLOWED }]); + expect(down).toContain('DROP TABLE "gone";'); + expect(down).not.toContain("DROP TABLE IF EXISTS"); + }); + + test("postgres create-view down", () => { + const { down } = renderPostgres([ + { + kind: "create-view", + status: ALLOWED, + view: { + name: "v", + sql: "SELECT 1 AS one", + columns: [{ name: "one", sqlType: { kind: "integer", bits: 32 } }], + }, + }, + ]); + expect(down).toContain('DROP VIEW "v";'); + expect(down).not.toContain("DROP VIEW IF EXISTS"); + }); + + test("sqlite create-table down", () => { + const { down } = renderSqlite([{ kind: "create-table", table: GONE, status: ALLOWED }]); + expect(down).toContain('DROP TABLE "gone";'); + expect(down).not.toContain("DROP TABLE IF EXISTS"); + }); +}); + +describe("the recreate-and-copy rebuild drop stays bare — deliberately", () => { + test("sqlite recreate emits a bare DROP TABLE for the table it just copied from", () => { + const expectedSchema: SchemaSnapshot = { + tables: [ + { + name: "orders", + columns: [ + { name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }, + { name: "amount", sqlType: { kind: "integer", bits: 64 }, nullable: false }, + ], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], + }, + ], + views: [], + }; + const { up } = renderSqlite( + [ + { + kind: "change-column-type", + table: "orders", + column: "amount", + from: { kind: "real" }, + to: { kind: "integer", bits: 64 }, + status: ALLOWED, + }, + ], + expectedSchema, + ); + // IF EXISTS here would turn a caught corruption into a silent one: the recipe + // just INSERT…SELECTed out of this exact table. + expect(up).toContain('DROP TABLE "orders";'); + expect(up).not.toContain('DROP TABLE IF EXISTS "orders";'); + }); +}); diff --git a/server/typescript/packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts b/server/typescript/packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts index 021ec68c8..09c1af2cd 100644 --- a/server/typescript/packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts +++ b/server/typescript/packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts @@ -138,9 +138,15 @@ describe("#285 — constraint-backed index drops as a constraint (PG)", () => { expect(d.changes.some((c) => c.kind === "drop-index")).toBe(true); const sqlText = emit(d.changes, { dialect: "postgres" }).up; - // The whole point: a bare DROP INDEX here is what Postgres refuses. - expect(sqlText).not.toMatch(/DROP INDEX "?work_item_message_id_unique/); - expect(sqlText).toMatch(/ALTER TABLE .*DROP CONSTRAINT "work_item_message_id_unique"/); + // The whole point: a DROP INDEX here is what Postgres refuses. + // + // `(IF EXISTS )?` is load-bearing on the NEGATIVE assertion. #313 put `IF EXISTS` + // on every forward drop, so without the optional group this stops matching because + // the SPELLING changed — passing for a reason unrelated to #285, and continuing to + // pass if #285 fully regressed. Verified by reverting the drop-index arm and + // watching this line go red. + expect(sqlText).not.toMatch(/DROP INDEX (IF EXISTS )?"?work_item_message_id_unique/); + expect(sqlText).toMatch(/ALTER TABLE .*DROP CONSTRAINT IF EXISTS "work_item_message_id_unique"/); // Pre-fix this throws: cannot drop index … because constraint … requires it. await applyRaw(kysely, sqlText); diff --git a/server/typescript/packages/migrate-ts/test/unit/emit-postgres.test.ts b/server/typescript/packages/migrate-ts/test/unit/emit-postgres.test.ts index cbe5b52ff..afcf65795 100644 --- a/server/typescript/packages/migrate-ts/test/unit/emit-postgres.test.ts +++ b/server/typescript/packages/migrate-ts/test/unit/emit-postgres.test.ts @@ -133,7 +133,8 @@ describe("renderPostgres — table-level", () => { test("drop-table", () => { const changes: Change[] = [{ kind: "drop-table", table: "legacy", status: ALLOWED }]; const { up } = emit(changes, { dialect: "postgres" }); - expect(norm(up)).toBe(`DROP TABLE "legacy";`); + // #313 — forward drops carry IF EXISTS so a committed chain replays from empty. + expect(norm(up)).toBe(`DROP TABLE IF EXISTS "legacy";`); }); }); @@ -161,7 +162,7 @@ describe("renderPostgres — indexes + FKs", () => { test("drop-index", () => { const changes: Change[] = [{ kind: "drop-index", table: "users", index: "old_idx", status: ALLOWED }]; const { up } = emit(changes, { dialect: "postgres" }); - expect(norm(up)).toBe(`DROP INDEX "old_idx";`); + expect(norm(up)).toBe(`DROP INDEX IF EXISTS "old_idx";`); }); test("add-fk with ON DELETE CASCADE", () => { @@ -184,7 +185,7 @@ describe("renderPostgres — indexes + FKs", () => { test("drop-fk", () => { const changes: Change[] = [{ kind: "drop-fk", table: "weeks", fk: "weeks_program_id_fk", status: ALLOWED }]; const { up } = emit(changes, { dialect: "postgres" }); - expect(norm(up)).toBe(`ALTER TABLE "weeks" DROP CONSTRAINT "weeks_program_id_fk";`); + expect(norm(up)).toBe(`ALTER TABLE "weeks" DROP CONSTRAINT IF EXISTS "weeks_program_id_fk";`); }); }); diff --git a/server/typescript/packages/migrate-ts/test/unit/emit-sqlite.test.ts b/server/typescript/packages/migrate-ts/test/unit/emit-sqlite.test.ts index 0a20d9b8f..f40b1ac6b 100644 --- a/server/typescript/packages/migrate-ts/test/unit/emit-sqlite.test.ts +++ b/server/typescript/packages/migrate-ts/test/unit/emit-sqlite.test.ts @@ -58,7 +58,8 @@ describe("renderSqlite — create-table", () => { describe("renderSqlite — table-level + indexes", () => { test("drop-table", () => { const { up } = emit([{ kind: "drop-table", table: "old", status: ALLOWED }], { dialect: "sqlite" }); - expect(norm(up)).toBe(`DROP TABLE "old";`); + // #313 — forward drops carry IF EXISTS so a committed chain replays from empty. + expect(norm(up)).toBe(`DROP TABLE IF EXISTS "old";`); }); test("rename-table", () => { const { up } = emit([{ kind: "rename-table", from: "p", to: "a", status: ALLOWED }], { dialect: "sqlite" }); @@ -74,7 +75,7 @@ describe("renderSqlite — table-level + indexes", () => { }); test("drop-index", () => { const { up } = emit([{ kind: "drop-index", table: "u", index: "i", status: ALLOWED }], { dialect: "sqlite" }); - expect(norm(up)).toBe(`DROP INDEX "i";`); + expect(norm(up)).toBe(`DROP INDEX IF EXISTS "i";`); }); }); @@ -360,7 +361,7 @@ describe("renderSqlite — drop-index vs drop-column (#255 generalized)", () => ], { dialect: "sqlite" }, ); - const idxDropIndex = up.indexOf('DROP INDEX "uniqueCode"'); + const idxDropIndex = up.indexOf('DROP INDEX IF EXISTS "uniqueCode"'); const idxDropColumn = up.indexOf('ALTER TABLE "programs" DROP COLUMN "code"'); expect(idxDropIndex).toBeGreaterThanOrEqual(0); expect(idxDropColumn).toBeGreaterThanOrEqual(0); From 6240df936dad4ba55a1e179b92b4da3c6da5af3d Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 07:25:18 -0400 Subject: [PATCH 08/44] fix(migrate): a chain creates the schema it needs, so it applies to a virgin database MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `CREATE SCHEMA` was emitted nowhere in either emitter — only by the ledger's own setup. A chain containing `CREATE TABLE "reporting"."x"` therefore could never apply to an empty database: the schema does not exist, and no migration ever creates it. Every `@schema` project's chain was unreplayable, and the first `apply-pending` against a fresh CI database died (#313). `renderPostgres` now prepends one `CREATE SCHEMA IF NOT EXISTS "";` per distinct non-default schema the migration creates an object in. VIEWS count, not only tables. The spec says "the first OBJECT", and a first migration that creates only a view in a non-default schema fails identically. A `create-view` carries the schema in two places, so the collector reads `c.schema ?? c.view.schema` — the same precedence `renderCreateView(c.view, c.schema, …)` already applies. `IF NOT EXISTS` because a later migration in the same chain, or an operator, may have created it already. Sorted, so output stays deterministic for the committed snapshot and the golden tests. A drop-only migration emits nothing, so a migration that removes the last object from a schema does not resurrect it. The down deliberately does NOT `DROP SCHEMA`: the schema may hold objects this tool does not own and cannot restore. That is asserted, not assumed. Postgres-only, and not a dialect split — sqlite has no schema namespacing at all and rejects a declared schema outright (`emit-sqlite-schema-rejected`). Co-Authored-By: Claude Opus 5 (1M context) --- .../packages/migrate-ts/src/emit/postgres.ts | 32 ++++++- .../test/emit-postgres-create-schema.test.ts | 94 +++++++++++++++++++ 2 files changed, 125 insertions(+), 1 deletion(-) create mode 100644 server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts diff --git a/server/typescript/packages/migrate-ts/src/emit/postgres.ts b/server/typescript/packages/migrate-ts/src/emit/postgres.ts index fa24ea061..68ea5d2ec 100644 --- a/server/typescript/packages/migrate-ts/src/emit/postgres.ts +++ b/server/typescript/packages/migrate-ts/src/emit/postgres.ts @@ -54,12 +54,42 @@ export function renderPostgres(changes: Change[]): EmitResult { } // Down runs in reverse order (so creates undo correctly w.r.t. FKs). return { - up: upStmts.join("\n\n"), + up: [...createSchemaStmts(sorted), ...upStmts].join("\n\n"), down: [...downStmts].reverse().join("\n\n"), recreatedTables: new Set(), // postgres alters in place; no recreate-and-copy }; } +/** + * `CREATE SCHEMA IF NOT EXISTS` for every non-default schema this migration creates + * an object in, ahead of everything else it emits. + * + * A chain must be appliable to a VIRGIN database (#313), and `CREATE TABLE "s"."x"` + * fails there unless `s` exists — yet `CREATE SCHEMA` was emitted nowhere in either + * emitter, only by the ledger's own setup. So an `@schema` project's chain could + * never be replayed, and the first `apply-pending` against a fresh CI database died. + * + * VIEWS count, not only tables: a first migration that creates only a view in a + * non-default schema fails identically. A `create-view` carries the schema in two + * places and the change's own key wins, matching `renderCreateView(c.view, c.schema)`. + * + * `IF NOT EXISTS` because a later migration in the same chain, or an operator, may + * have created it already. Sorted so output is deterministic — the committed snapshot + * and the golden tests depend on that. Deliberately NOT dropped in `down`: the schema + * may hold objects this tool does not own and cannot restore. + */ +function createSchemaStmts(sorted: readonly Change[]): string[] { + const schemas = new Set(); + for (const c of sorted) { + const s = + c.kind === "create-table" ? c.table.schema + : c.kind === "create-view" ? (c.schema ?? c.view.schema) + : undefined; + if (s !== undefined && s !== DEFAULT_DB_SCHEMA_POSTGRES) schemas.add(s); + } + return [...schemas].sort().map((s) => `CREATE SCHEMA IF NOT EXISTS ${quote(s)};`); +} + function renderUp(c: Change): string { switch (c.kind) { case "create-table": return renderCreateTable(c.table); diff --git a/server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts b/server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts new file mode 100644 index 000000000..7bd1cf6d8 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts @@ -0,0 +1,94 @@ +// A committed chain must apply to a VIRGIN database (#313). `CREATE TABLE "s"."x"` +// fails there unless the schema exists, and no migration has ever created one — +// `CREATE SCHEMA` appears nowhere in the emitters, only in the ledger's own setup. +import { describe, test, expect } from "bun:test"; +import { renderPostgres } from "../src/emit/postgres.js"; +import type { ChangeStatus, TableDescriptor, ViewDescriptor } from "../src/types.js"; + +const ALLOWED: ChangeStatus = { state: "allowed" }; + +const t = (name: string, schema?: string): TableDescriptor => ({ + name, + ...(schema !== undefined ? { schema } : {}), + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], +}); + +const v = (name: string, schema?: string): ViewDescriptor => ({ + name, + ...(schema !== undefined ? { schema } : {}), + sql: "SELECT 1 AS one", + columns: [{ name: "one", sqlType: { kind: "integer", bits: 32 } }], +}); + +describe("a chain that creates an object in a non-default schema creates the schema first", () => { + test("emits CREATE SCHEMA IF NOT EXISTS before the table", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x", "reporting"), status: ALLOWED }]); + expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "reporting";'); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "reporting";')).toBeLessThan(up.indexOf("CREATE TABLE")); + }); + + // The spec says "the first OBJECT", not "the first table": a chain whose first + // migration creates only a view in a non-default schema fails identically. + test("emits it for a create-view too", () => { + const { up } = renderPostgres([{ kind: "create-view", view: v("v_x", "reporting"), status: ALLOWED }]); + expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "reporting";'); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "reporting";')).toBeLessThan(up.indexOf("CREATE VIEW")); + }); + + // `create-view` carries the schema in two places; the change's own key wins, the + // same precedence renderCreateView already uses. + test("a create-view's own schema key wins over the descriptor's", () => { + const { up } = renderPostgres([ + { kind: "create-view", view: v("v_x", "descriptor_schema"), schema: "change_schema", status: ALLOWED }, + ]); + expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "change_schema";'); + expect(up).not.toContain('CREATE SCHEMA IF NOT EXISTS "descriptor_schema";'); + }); + + test("emits it once for several objects in the same schema", () => { + const { up } = renderPostgres([ + { kind: "create-table", table: t("x", "reporting"), status: ALLOWED }, + { kind: "create-table", table: t("y", "reporting"), status: ALLOWED }, + { kind: "create-view", view: v("v_x", "reporting"), status: ALLOWED }, + ]); + expect(up.match(/CREATE SCHEMA IF NOT EXISTS "reporting";/g)).toHaveLength(1); + }); + + test("emits one per distinct schema, in sorted order", () => { + const { up } = renderPostgres([ + { kind: "create-table", table: t("x", "zeta"), status: ALLOWED }, + { kind: "create-table", table: t("y", "alpha"), status: ALLOWED }, + ]); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "alpha";')) + .toBeLessThan(up.indexOf('CREATE SCHEMA IF NOT EXISTS "zeta";')); + }); + + test("emits nothing for the default schema", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x"), status: ALLOWED }]); + expect(up).not.toContain("CREATE SCHEMA"); + }); + + test("emits nothing for an explicit 'public'", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x", "public"), status: ALLOWED }]); + expect(up).not.toContain("CREATE SCHEMA"); + }); + + // Not filler: dropping a schema on rollback would destroy objects this tool does + // not own and cannot restore. + test("the down does NOT drop the schema", () => { + const { down } = renderPostgres([{ kind: "create-table", table: t("x", "reporting"), status: ALLOWED }]); + expect(down).not.toContain("DROP SCHEMA"); + }); + + // A migration that only DROPS from a schema must not resurrect it. + test("a drop-only migration emits no CREATE SCHEMA", () => { + const { up } = renderPostgres([ + { kind: "drop-table", table: "x", schema: "reporting", status: ALLOWED }, + ]); + expect(up).not.toContain("CREATE SCHEMA"); + }); +}); From 87dae6b3a431ba6e4950b42dd1482542aff5d5eb Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 07:28:13 -0400 Subject: [PATCH 09/44] feat(migrate): an in-process replay engine, so the gate provisions nothing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `openReplayEngine(dialect)` hands back an empty database and a disposer: `:memory:` libsql for sqlite, PGlite — real Postgres compiled to WASM — for postgres. The #313 gate needs to apply a whole committed chain from nothing, and this is where that nothing comes from. Why not a scratch database on the user's server, which an earlier design chose: it needs CREATEDB, breaks behind a connection pooler, is restricted on managed Postgres, collides between parallel CI jobs sharing one server, and — because Postgres truncates identifiers at 63 bytes — a long enough target database derives a scratch name that truncates back ONTO the target, putting a `DROP DATABASE IF EXISTS` next to the real one. In-process has none of those failure modes and nothing to clean up. PGlite is NOT pg-compatible: it exposes query/exec/close, while kysely's PostgresDialect wants a pg.Pool (connect() → a client with query()/release(), plus end()). `pgliteAsPool` is that adapter. PGlite is a single session, so every connect() returns the same instance — correct for a strictly sequential replay, and what makes a session advisory lock taken on one kysely connection visible to the next. Verified against 0.3.16 and 0.5.5, and pinned by the tests: schema namespacing, a CHECK constraint, transactions, transaction ROLLBACK, pg_advisory_lock/unlock (applyPending takes one on postgres), engine isolation between two instances, idempotent dispose, and — the signal the whole gate rests on — `DROP TABLE "theirs"` rejecting with `table "theirs" does not exist`, the reporter's own error, on both dialects. It also proves `introspect` works against PGlite (information_schema, pg_catalog, pg_get_viewdef), which is what the `--replay-snapshot` tier needs and the only remaining unknown in that design. Both drivers are OPTIONAL peers imported lazily, with install hints. PGlite is ~22 MB of WASM and must not reach the node_modules of every adopter who only runs `meta gen`; `cli`'s `build:binary` gets a matching `--external` so the standalone binary does not embed it either. Peer ranges are bounded, so check-peer-ranges stays green (28 ranges, all bounded). It lives in `migrate-ts` rather than `cli` because migrate-ts's own tests need it and `cli` depends on migrate-ts — the other direction would be a cycle. Co-Authored-By: Claude Opus 5 (1M context) --- bun.lock | 9 ++ server/typescript/packages/cli/package.json | 2 +- .../packages/migrate-ts/package.json | 13 +- .../packages/migrate-ts/src/index.ts | 4 + .../migrate-ts/src/verify/replay-engine.ts | 129 ++++++++++++++++++ .../test/unit/replay-engine.test.ts | 128 +++++++++++++++++ 6 files changed, 283 insertions(+), 2 deletions(-) create mode 100644 server/typescript/packages/migrate-ts/src/verify/replay-engine.ts create mode 100644 server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts diff --git a/bun.lock b/bun.lock index d241e0129..d8e0d292e 100644 --- a/bun.lock +++ b/bun.lock @@ -330,6 +330,7 @@ "@metaobjectsdev/metadata": "workspace:*", }, "devDependencies": { + "@electric-sql/pglite": "^0.5.0", "@libsql/kysely-libsql": "^0.4.1", "@types/pg": "^8.20.0", "bun-types": "latest", @@ -339,8 +340,14 @@ "typescript": "^5.6.0", }, "peerDependencies": { + "@electric-sql/pglite": ">=0.3.0 <0.6.0", + "@libsql/kysely-libsql": ">=0.4.0 <0.5.0", "kysely": ">=0.27.0 <0.30.0", }, + "optionalPeers": [ + "@electric-sql/pglite", + "@libsql/kysely-libsql", + ], }, "server/typescript/packages/render": { "name": "@metaobjectsdev/render", @@ -531,6 +538,8 @@ "@csstools/css-tokenizer": ["@csstools/css-tokenizer@4.0.0", "", {}, "sha512-QxULHAm7cNu72w97JUNCBFODFaXpbDg+dP8b/oWFAZ2MTRppA3U00Y2L1HqaS4J6yBqxwa/Y3nMBaxVKbB/NsA=="], + "@electric-sql/pglite": ["@electric-sql/pglite@0.5.5", "", {}, "sha512-QeZ+oB7QaU4EJj2awrB8hwFZ5TXxLRBQ9Gfq0dQauYDdWsRtuWRCERgtri35vRfL07KQHlVlc/4Qxvn7a5AXeA=="], + "@exodus/bytes": ["@exodus/bytes@1.15.1", "", { "peerDependencies": { "@noble/hashes": "^1.8.0 || ^2.0.0" }, "optionalPeers": ["@noble/hashes"] }, "sha512-S6mL0yNB/Abt9Ei4tq8gDhcczc4S3+vQ4ra7vxnAf+YHC02srtqxKKZghx2Dq6p0e66THKwR6r8N6P95wEty7Q=="], "@fastify/ajv-compiler": ["@fastify/ajv-compiler@3.6.0", "", { "dependencies": { "ajv": "^8.11.0", "ajv-formats": "^2.1.1", "fast-uri": "^2.0.0" } }, "sha512-LwdXQJjmMD+GwLOkP7TVC68qa+pSSogeWWmznRJ/coyTcfe9qA05AHFSe1eZFwK6q+xVRpChnvFUkf1iYaSZsQ=="], diff --git a/server/typescript/packages/cli/package.json b/server/typescript/packages/cli/package.json index 6182f1f63..08dcd731d 100644 --- a/server/typescript/packages/cli/package.json +++ b/server/typescript/packages/cli/package.json @@ -27,7 +27,7 @@ "scripts": { "build": "tsc -p .", "typecheck": "tsc -p tsconfig.typecheck.json", - "build:binary": "bun build ./bin/meta.ts --compile --outfile dist/meta --external @biomejs/wasm-bundler --external @biomejs/wasm-web" + "build:binary": "bun build ./bin/meta.ts --compile --outfile dist/meta --external @biomejs/wasm-bundler --external @biomejs/wasm-web --external @electric-sql/pglite" }, "license": "Apache-2.0", "author": "Doug Mealing ", diff --git a/server/typescript/packages/migrate-ts/package.json b/server/typescript/packages/migrate-ts/package.json index f31b558cf..b74df704d 100644 --- a/server/typescript/packages/migrate-ts/package.json +++ b/server/typescript/packages/migrate-ts/package.json @@ -48,9 +48,20 @@ "@metaobjectsdev/metadata": "workspace:*" }, "peerDependencies": { - "kysely": ">=0.27.0 <0.30.0" + "kysely": ">=0.27.0 <0.30.0", + "@electric-sql/pglite": ">=0.3.0 <0.6.0", + "@libsql/kysely-libsql": ">=0.4.0 <0.5.0" + }, + "peerDependenciesMeta": { + "@electric-sql/pglite": { + "optional": true + }, + "@libsql/kysely-libsql": { + "optional": true + } }, "devDependencies": { + "@electric-sql/pglite": "^0.5.0", "@libsql/kysely-libsql": "^0.4.1", "@types/pg": "^8.20.0", "bun-types": "latest", diff --git a/server/typescript/packages/migrate-ts/src/index.ts b/server/typescript/packages/migrate-ts/src/index.ts index b4c8c9208..154bb7005 100644 --- a/server/typescript/packages/migrate-ts/src/index.ts +++ b/server/typescript/packages/migrate-ts/src/index.ts @@ -112,6 +112,10 @@ export { export { verifyReplay } from "./verify/replay.js"; export type { VerifyReplayArgs, VerifyReplayResult } from "./verify/replay.js"; +// An empty in-process database to replay a committed chain into (#313). +export { openReplayEngine } from "./verify/replay-engine.js"; +export type { ReplayEngine } from "./verify/replay-engine.js"; + // Wrangler config helpers export { findWranglerConfig, diff --git a/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts b/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts new file mode 100644 index 000000000..07a14802a --- /dev/null +++ b/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts @@ -0,0 +1,129 @@ +// An empty, throwaway database that lives INSIDE this process. +// +// The replay gate has to apply a whole committed chain from nothing. Doing that +// against the user's server would mean CREATE DATABASE — which needs CREATEDB, +// breaks behind a connection pooler, is restricted on managed Postgres, collides +// between parallel CI jobs sharing one server, and puts a DROP DATABASE next to a +// name derived from a real one (Postgres truncates identifiers at 63 bytes, so a +// long enough target derives a scratch name that truncates back ONTO the target). +// None of that is worth it when the engines run in-process: PGlite is real Postgres +// compiled to WASM, and libsql runs sqlite in memory. Nothing to provision, nothing +// to clean up, nothing to drop by mistake. +// +// Both drivers are OPTIONAL peers imported lazily. PGlite is ~22 MB of WASM and must +// not land in the node_modules of every adopter who only ever runs `meta gen`; the +// install hints mirror `buildKyselyFromUrl`'s. `cli` already depends on +// `@libsql/kysely-libsql` outright, so only a direct embedder can miss that one. +import { Kysely } from "kysely"; + +export interface ReplayEngine { + /** An empty database. The caller owns applying migrations into it. */ + db: Kysely>; + /** Release the engine. Safe to call more than once. */ + dispose: () => Promise; +} + +/** Open an empty in-process database of the given dialect. */ +export async function openReplayEngine( + dialect: "postgres" | "sqlite", +): Promise { + return dialect === "postgres" ? openPglite() : openMemorySqlite(); +} + +async function openMemorySqlite(): Promise { + type LibsqlDialectCtor = new (opts: { url: string }) => + ConstructorParameters>>[0]["dialect"]; + let LibsqlDialect: LibsqlDialectCtor; + try { + const mod = await import("@libsql/kysely-libsql"); + LibsqlDialect = mod.LibsqlDialect as unknown as LibsqlDialectCtor; + } catch { + throw new Error( + `the sqlite replay engine requires '@libsql/kysely-libsql'; install it to run 'meta verify --replay'`, + ); + } + const db = new Kysely>({ dialect: new LibsqlDialect({ url: ":memory:" }) }); + return disposable(db, async () => { /* the in-memory database dies with the connection */ }); +} + +async function openPglite(): Promise { + let PGliteCtor: new () => PgliteInstance; + try { + const mod = await import("@electric-sql/pglite"); + PGliteCtor = mod.PGlite as unknown as new () => PgliteInstance; + } catch { + throw new Error( + `the postgres replay engine requires '@electric-sql/pglite' (in-process WASM Postgres); ` + + `install it to run 'meta verify --replay' against a postgres chain`, + ); + } + const { PostgresDialect } = await import("kysely"); + const pg = new PGliteCtor(); + const db = new Kysely>({ + dialect: new PostgresDialect({ pool: pgliteAsPool(pg) as never }), + }); + return disposable(db, () => pg.close()); +} + +/** The slice of PGlite's surface this file uses. */ +interface PgliteInstance { + query( + sql: string, + params?: unknown[], + ): Promise<{ rows: unknown[]; affectedRows?: number; statement?: string }>; + close(): Promise; +} + +/** + * Adapt PGlite to the `pg.Pool` shape kysely's `PostgresDialect` expects: `connect()` + * returning a client with `query()`/`release()`, plus `end()`. PGlite offers only + * `query`/`close`, so without this the dialect cannot drive it at all. + * + * PGlite is a SINGLE session, so every `connect()` hands back the same underlying + * instance. That is correct here — a replay is strictly sequential — and it is what + * makes a session advisory lock taken on one kysely connection visible to the next. + * + * `command` is read by kysely only to decide whether to report numAffectedRows; the + * replay path never reads it, so PGlite's `statement` (or a SELECT default) suffices. + */ +function pgliteAsPool(pg: PgliteInstance): unknown { + return { + async connect() { + return { + async query(sqlText: unknown, params?: readonly unknown[]) { + if (typeof sqlText !== "string") { + throw new Error(`the PGlite replay engine does not support cursors`); + } + const r = await pg.query(sqlText, params ? [...params] : []); + return { + command: r.statement ?? "SELECT", + rowCount: r.affectedRows ?? r.rows.length, + rows: r.rows, + }; + }, + release() { /* single session — there is no pool to return to */ }, + }; + }, + async end() { + await pg.close(); + }, + }; +} + +function disposable( + db: Kysely>, + closeEngine: () => Promise, +): ReplayEngine { + let disposed = false; + return { + db, + dispose: async () => { + if (disposed) return; + disposed = true; + // Both swallow: the engine is throwaway, and a teardown error must not mask + // the replay verdict the caller is about to report. + try { await db.destroy(); } catch { /* ignore */ } + try { await closeEngine(); } catch { /* ignore */ } + }, + }; +} diff --git a/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts b/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts new file mode 100644 index 000000000..a74c37b57 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts @@ -0,0 +1,128 @@ +import { describe, test, expect } from "bun:test"; +import { sql } from "kysely"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import { introspect } from "../../src/introspect/index.js"; + +describe("openReplayEngine", () => { + test("sqlite: gives an empty, usable database", async () => { + const engine = await openReplayEngine("sqlite"); + try { + await sql`CREATE TABLE t (id integer primary key)`.execute(engine.db); + await sql`INSERT INTO t (id) VALUES (1)`.execute(engine.db); + const rows = await sql<{ id: number }>`SELECT id FROM t`.execute(engine.db); + expect(rows.rows).toHaveLength(1); + } finally { + await engine.dispose(); + } + }); + + test("postgres: gives an empty, usable database with real PG DDL", async () => { + const engine = await openReplayEngine("postgres"); + try { + // Schema namespacing + a CHECK — neither is expressible in sqlite, so this + // proves the postgres engine really is Postgres. + await sql`CREATE SCHEMA IF NOT EXISTS "reporting"`.execute(engine.db); + await sql`CREATE TABLE "reporting"."t" (id integer primary key, n integer CHECK (n > 0))`.execute(engine.db); + const rows = await sql<{ table_name: string }>` + SELECT table_name FROM information_schema.tables WHERE table_schema = 'reporting' + `.execute(engine.db); + expect(rows.rows.map((r) => r.table_name)).toContain("t"); + } finally { + await engine.dispose(); + } + }); + + // applyPending runs each migration file inside a kysely transaction and takes a + // pg advisory lock on postgres. Both must work through the shim, or the gate + // fails for a reason that has nothing to do with the chain under test. + test("postgres: transactions roll back, and advisory locks work", async () => { + const engine = await openReplayEngine("postgres"); + try { + await sql`CREATE TABLE t (id integer primary key)`.execute(engine.db); + await expect( + engine.db.transaction().execute(async (trx) => { + await sql`INSERT INTO t (id) VALUES (1)`.execute(trx); + throw new Error("boom"); + }), + ).rejects.toThrow(/boom/); + const after = await sql<{ c: string }>`SELECT count(*)::text AS c FROM t`.execute(engine.db); + expect(after.rows[0]?.c).toBe("0"); + + await sql`SELECT pg_advisory_lock(hashtext('meta'))`.execute(engine.db); + await sql`SELECT pg_advisory_unlock(hashtext('meta'))`.execute(engine.db); + } finally { + await engine.dispose(); + } + }); + + // The whole gate rests on this: a statement against a missing object must REJECT. + test("postgres: dropping a missing table rejects — the #313 signal", async () => { + const engine = await openReplayEngine("postgres"); + try { + await expect(sql`DROP TABLE "theirs"`.execute(engine.db)).rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + } + }); + + test("sqlite: dropping a missing table rejects — the #313 signal", async () => { + const engine = await openReplayEngine("sqlite"); + try { + await expect(sql`DROP TABLE "theirs"`.execute(engine.db)).rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + } + }); + + // `--replay-snapshot` introspects the replayed database, so the postgres + // introspector — information_schema, pg_catalog, pg_get_viewdef — has to work + // against the in-process engine or that whole tier is unreachable. + test("postgres: introspect reads back a table and a view", async () => { + const engine = await openReplayEngine("postgres"); + try { + await sql`CREATE TABLE t (id integer primary key, n integer NOT NULL)`.execute(engine.db); + await sql`CREATE VIEW v AS SELECT id FROM t`.execute(engine.db); + const snap = await introspect(engine.db, "postgres"); + expect(snap.tables.map((x) => x.name)).toContain("t"); + expect(snap.views.map((x) => x.name)).toContain("v"); + } finally { + await engine.dispose(); + } + }); + + test("two engines of the same dialect do not share state", async () => { + const a = await openReplayEngine("sqlite"); + const b = await openReplayEngine("sqlite"); + try { + await sql`CREATE TABLE only_in_a (id integer)`.execute(a.db); + const rows = await sql<{ name: string }>`SELECT name FROM sqlite_master WHERE name = 'only_in_a'`.execute(b.db); + expect(rows.rows).toHaveLength(0); + } finally { + await a.dispose(); + await b.dispose(); + } + }); + + test("two postgres engines do not share state either", async () => { + const a = await openReplayEngine("postgres"); + const b = await openReplayEngine("postgres"); + try { + await sql`CREATE TABLE only_in_a (id integer)`.execute(a.db); + const rows = await sql<{ c: string }>` + SELECT count(*)::text AS c FROM information_schema.tables WHERE table_name = 'only_in_a' + `.execute(b.db); + expect(rows.rows[0]?.c).toBe("0"); + } finally { + await a.dispose(); + await b.dispose(); + } + }); + + // A caller disposes in a `finally` after an early return, so a double dispose + // must not throw. + test("dispose is idempotent", async () => { + const engine = await openReplayEngine("sqlite"); + await engine.dispose(); + await engine.dispose(); + }); +}); From d2e1a0b5ccea1d545ec6b5aa9adf1f3f14bb8551 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 07:33:26 -0400 Subject: [PATCH 10/44] fix(migrate): the sqlite replay engine must survive a transaction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The engine shipped one commit ago used `:memory:`, as the design specified. Under `@libsql/kysely-libsql` that gives every CONNECTION its own database, so a table created inside a transaction is gone the moment that transaction's connection is released. `applyPending` runs every migration file inside a transaction. So the engine would have replayed a whole chain into a series of throwaway databases: migration 2 could not see migration 1's tables, the introspection afterwards saw an empty database, and `--replay` would have reported success having proved nothing. A gate that cannot fail is worse than no gate. It is now a throwaway sqlite file in a private `mkdtemp` directory, removed on dispose — which is what `test/integrity/replay.test.ts` has always used. Rejected on the way there, both measured rather than assumed: `file::memory:?cache=shared` fixes visibility and breaks isolation instead (two engines in one process land in the same database), and libsql refuses the named `?mode=memory&cache=shared` form with `URL_PARAM_NOT_SUPPORTED`. Found by `verifyReplay` reporting `mine` as missing from a chain that plainly creates it — the scoped-replay test written for the NEXT task, which happened to be the first thing to drive a real chain through `applyPending`. The engine's own suite had nine passing cases and none of them could see this: every sqlite case ran its DDL outside a transaction. It now asserts, on BOTH dialects, that a tx-created table survives its transaction, is visible to a second transaction the way migration 2 sees migration 1's, and is present in the introspected snapshot. Co-Authored-By: Claude Opus 5 (1M context) --- .../migrate-ts/src/verify/replay-engine.ts | 36 ++++++++++++++++--- .../test/unit/replay-engine.test.ts | 28 +++++++++++++++ 2 files changed, 59 insertions(+), 5 deletions(-) diff --git a/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts b/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts index 07a14802a..d4e8f1f81 100644 --- a/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts +++ b/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts @@ -6,15 +6,19 @@ // between parallel CI jobs sharing one server, and puts a DROP DATABASE next to a // name derived from a real one (Postgres truncates identifiers at 63 bytes, so a // long enough target derives a scratch name that truncates back ONTO the target). -// None of that is worth it when the engines run in-process: PGlite is real Postgres -// compiled to WASM, and libsql runs sqlite in memory. Nothing to provision, nothing -// to clean up, nothing to drop by mistake. +// None of that is worth it when the engine runs locally and disposably: PGlite is +// real Postgres compiled to WASM and lives in this process; sqlite is a throwaway +// file in a private temp directory (see `openMemorySqlite` for why not `:memory:`). +// Nothing to provision, nothing to name, nothing to drop by mistake. // // Both drivers are OPTIONAL peers imported lazily. PGlite is ~22 MB of WASM and must // not land in the node_modules of every adopter who only ever runs `meta gen`; the // install hints mirror `buildKyselyFromUrl`'s. `cli` already depends on // `@libsql/kysely-libsql` outright, so only a direct embedder can miss that one. import { Kysely } from "kysely"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; export interface ReplayEngine { /** An empty database. The caller owns applying migrations into it. */ @@ -30,6 +34,23 @@ export async function openReplayEngine( return dialect === "postgres" ? openPglite() : openMemorySqlite(); } +/** + * A throwaway sqlite database in a private temp directory, removed on dispose. + * + * NOT `:memory:`, and that is the whole point of this comment. Under + * `@libsql/kysely-libsql`, `:memory:` gives every CONNECTION its own database — so a + * table created inside a transaction is invisible the moment the transaction's + * connection is released. `applyPending` runs each migration file in a transaction, + * which means an in-memory engine would replay a whole chain into a series of + * throwaway databases, introspect an empty one, and never let migration 2 see + * migration 1's tables. The gate would pass having proved nothing. + * + * `file::memory:?cache=shared` fixes the visibility and breaks isolation instead — + * two engines in one process land in the SAME database — and libsql rejects the + * named `?mode=memory&cache=shared` form outright (`URL_PARAM_NOT_SUPPORTED`). A + * unique temp file is correct on both counts, and it is what the existing + * `test/integrity/replay.test.ts` has always used. + */ async function openMemorySqlite(): Promise { type LibsqlDialectCtor = new (opts: { url: string }) => ConstructorParameters>>[0]["dialect"]; @@ -42,8 +63,13 @@ async function openMemorySqlite(): Promise { `the sqlite replay engine requires '@libsql/kysely-libsql'; install it to run 'meta verify --replay'`, ); } - const db = new Kysely>({ dialect: new LibsqlDialect({ url: ":memory:" }) }); - return disposable(db, async () => { /* the in-memory database dies with the connection */ }); + const dir = mkdtempSync(join(tmpdir(), "meta-replay-")); + const db = new Kysely>({ + dialect: new LibsqlDialect({ url: `file:${join(dir, "replay.db")}` }), + }); + return disposable(db, async () => { + rmSync(dir, { recursive: true, force: true }); + }); } async function openPglite(): Promise { diff --git a/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts b/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts index a74c37b57..526189fa0 100644 --- a/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts +++ b/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts @@ -32,6 +32,34 @@ describe("openReplayEngine", () => { } }); + // THE defect that nearly shipped, and the reason this case exists on BOTH dialects: + // `applyPending` runs every migration file inside a transaction, so a table created + // by migration 1 must be visible to migration 2 and to the introspection that + // follows. Under `@libsql/kysely-libsql`, `:memory:` gives each CONNECTION its own + // database, so a tx-created table vanishes the instant the transaction's connection + // is released — a whole chain would replay into a series of throwaway databases and + // the gate would pass having proved nothing. Only a cross-transaction assertion + // catches it; the non-transactional cases above all passed. + for (const dialect of ["sqlite", "postgres"] as const) { + test(`${dialect}: a table created inside a transaction survives it`, async () => { + const engine = await openReplayEngine(dialect); + try { + await engine.db.transaction().execute(async (trx) => { + await sql`CREATE TABLE in_tx (id integer primary key)`.execute(trx); + }); + // A second transaction must SEE the first one's table, the way migration 2 + // sees migration 1's. + await engine.db.transaction().execute(async (trx) => { + await sql`INSERT INTO in_tx (id) VALUES (1)`.execute(trx); + }); + const snap = await introspect(engine.db, dialect); + expect(snap.tables.map((x) => x.name)).toContain("in_tx"); + } finally { + await engine.dispose(); + } + }); + } + // applyPending runs each migration file inside a kysely transaction and takes a // pg advisory lock on postgres. Both must work through the shim, or the gate // fails for a reason that has nothing to do with the chain under test. From 7a57c62c4e9f25a5f67b990613d157a4349fe2a4 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 07:33:37 -0400 Subject: [PATCH 11/44] fix(migrate): verifyReplay honours migrate.scope on the snapshot side MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A project declaring `migrate.scope` carries the OTHER owner's tables into its committed snapshot on purpose (`carryForwardOutOfScope`), and its chain — also on purpose — never creates them. `verifyReplay` compared the replayed database against that snapshot whole, so every such project saw its neighbour's tables reported as missing and could never use the check at all. `VerifyReplayArgs` gains an optional `governed: GovernedScope`, applied through the existing `excludeFromSnapshot` rather than a second hand-rolled copy of the scope contract. Excluded from the SNAPSHOT side only: the replayed database never had those tables either, so there is nothing to suppress on the actual side. Omitting it leaves the comparison byte-for-byte what it was, which is what every unscoped project gets. `excludeFromSnapshot` returns a `ScopedExpectedSchema`, not a `SchemaSnapshot` — the fix takes `.snapshot`. Three cases, and the last two are what keep the first honest: with `governed` the out-of-scope table is not reported; WITHOUT it the same fixture reports drift, so the pass demonstrably comes from the scope threading and not from a trivially-green fixture; and an EMPTY `outOfScope` still reports drift, so the new field cannot become a way to suppress a real difference. Names are qualified `.` with an absent schema normalized to the Postgres default, so the fixture says `public.theirs` — sqlite objects land under that same constant prefix. Co-Authored-By: Claude Opus 5 (1M context) --- .../packages/migrate-ts/src/verify/replay.ts | 22 ++++- .../test/integrity/replay-scoped.test.ts | 95 +++++++++++++++++++ 2 files changed, 116 insertions(+), 1 deletion(-) create mode 100644 server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts diff --git a/server/typescript/packages/migrate-ts/src/verify/replay.ts b/server/typescript/packages/migrate-ts/src/verify/replay.ts index 05a145fb5..ddf46f58b 100644 --- a/server/typescript/packages/migrate-ts/src/verify/replay.ts +++ b/server/typescript/packages/migrate-ts/src/verify/replay.ts @@ -4,6 +4,7 @@ import { applyPending } from "../apply/apply.js"; import { MIGRATIONS_TABLE } from "../apply/ledger.js"; import { introspect } from "../introspect/index.js"; import { driftAgainstSnapshot, type DriftClassification } from "../drift/classify.js"; +import { excludeFromSnapshot, type GovernedScope } from "../scope.js"; import type { Dialect, SchemaSnapshot } from "../types.js"; export interface VerifyReplayArgs { @@ -14,6 +15,20 @@ export interface VerifyReplayArgs { migrationsDir: string; /** The committed snapshot the migrations are expected to reproduce. */ snapshot: SchemaSnapshot; + /** + * The scope decision the run made, as `scopeExpectedSchema` reports it. + * + * A project declaring `migrate.scope` carries the OTHER owner's tables into its + * committed snapshot on purpose (`carryForwardOutOfScope`), and its chain — also on + * purpose — never creates them. Without this they read as missing on every replay, + * so a scoped project could never use this check at all. + * + * Excluded from the SNAPSHOT side only: the replayed database never had them + * either, so there is nothing to suppress on the actual side. Omitted ⇒ the + * comparison is byte-for-byte what it was, which is what every unscoped project + * gets. + */ + governed?: GovernedScope; } export interface VerifyReplayResult extends DriftClassification { @@ -35,7 +50,12 @@ export async function verifyReplay(args: VerifyReplayArgs): Promise t.name !== MIGRATIONS_TABLE), }; - const classification = await driftAgainstSnapshot(args.snapshot, actual, args.dialect); + // `excludeFromSnapshot` returns a ScopedExpectedSchema, so take `.snapshot`. With an + // empty `outOfScope` it returns the SAME object, not an equal copy. + const expected = args.governed !== undefined + ? excludeFromSnapshot(args.snapshot, args.governed).snapshot + : args.snapshot; + const classification = await driftAgainstSnapshot(expected, actual, args.dialect); return { ...classification, ok: classification.drift.length === 0 && classification.unmanaged.length === 0, diff --git a/server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts b/server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts new file mode 100644 index 000000000..9ad101dfd --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts @@ -0,0 +1,95 @@ +// A project declaring `migrate.scope` writes the OTHER owner's tables into its +// committed snapshot on purpose (`carryForwardOutOfScope`), and its chain — also on +// purpose — never creates them. Without threading the scope decision, every replay +// of such a project reports those tables as missing. +import { describe, test, expect } from "bun:test"; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { verifyReplay } from "../../src/verify/replay.js"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import type { SchemaSnapshot, TableDescriptor } from "../../src/types.js"; + +function chainWith(upSql: string): string { + const dir = mkdtempSync(join(tmpdir(), "replay-scoped-")); + mkdirSync(join(dir, "20260101000000-init"), { recursive: true }); + writeFileSync(join(dir, "20260101000000-init", "up.sql"), upSql, "utf8"); + writeFileSync(join(dir, "20260101000000-init", "down.sql"), 'DROP TABLE "mine";', "utf8"); + return dir; +} + +// `id INTEGER NOT NULL PRIMARY KEY`, not a bare `INTEGER PRIMARY KEY`: sqlite reports +// notnull=0 for the latter, so `nullable: false` below would read as drift and both +// tests would answer a question about column nullability rather than about scope. +const CHAIN = 'CREATE TABLE "mine" (id INTEGER NOT NULL PRIMARY KEY);'; + +const table = (name: string): TableDescriptor => ({ + name, + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], +}); + +const SNAPSHOT: SchemaSnapshot = { tables: [table("mine"), table("theirs")], views: [] }; + +describe("verifyReplay honours migrate.scope", () => { + test("an out-of-scope table in the snapshot is not reported as missing", async () => { + const dir = chainWith(CHAIN); + const engine = await openReplayEngine("sqlite"); + try { + const result = await verifyReplay({ + db: engine.db, + dialect: "sqlite", + migrationsDir: dir, + snapshot: SNAPSHOT, + // Qualified `.`; an absent schema normalizes to the Postgres + // default, and every sqlite object lands under that same constant prefix. + governed: { outOfScope: ["public.theirs"] }, + }); + expect(result.ok).toBe(true); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + // The control is what makes the case above non-vacuous: it proves the difference + // comes from `governed`, not from a fixture that was trivially green. + test("without `governed`, the same case reports drift — the control", async () => { + const dir = chainWith(CHAIN); + const engine = await openReplayEngine("sqlite"); + try { + const result = await verifyReplay({ + db: engine.db, + dialect: "sqlite", + migrationsDir: dir, + snapshot: SNAPSHOT, + }); + expect(result.ok).toBe(false); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + // An empty `outOfScope` must not become a way to suppress a real difference. + test("an empty scope still reports drift", async () => { + const dir = chainWith(CHAIN); + const engine = await openReplayEngine("sqlite"); + try { + const result = await verifyReplay({ + db: engine.db, + dialect: "sqlite", + migrationsDir: dir, + snapshot: SNAPSHOT, + governed: { outOfScope: [] }, + }); + expect(result.ok).toBe(false); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); From 0ad07906179bbbfb32286730fa56bafe4b2d8c8f Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 07:41:57 -0400 Subject: [PATCH 12/44] feat(cli): meta verify --replay and --replay-snapshot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two new verify subverbs. `--replay` replays the committed migration chain into an empty throwaway database and asserts it APPLIES — the #313 gate. `--replay-snapshot` additionally asserts the result equals the committed snapshot, finally wiring `verifyReplay`, which has been built and exported with no CLI caller since the 2026-05-31 design retained it as "the optional verify --replay integrity aid". They are two tiers rather than one gate because the populations differ. A project adopted via `migrate baseline --from-db` passes the first trivially — its chain is empty, so there is nothing to fail — and CANNOT pass the second by construction, since its snapshot is the whole introspected database. The reporter's failure was an APPLY error, so the weaker assertion is the one that answers the bug, and it is immune to that class. The second tier does not try to detect baseline adoption: the only candidate signal has no production caller and would live in the target database's ledger while the gate runs against a fresh engine with no ledger, so the failure message names it instead. Both flags feed `anyExplicit`, or `meta verify --replay` would also run the template gate as the bare-verify default. Both select ONE gate — the composed condition is `flags.replay || flags.replaySnapshot`, written once, because naming only `flags.replay` there is exactly how `--replay-snapshot` would parse cleanly and reach nothing. Neither needs `--db`; the engine is local and disposable. That leaves no URL to infer a dialect from, so the precedence is stated: `--dialect` wins, else migrate's own resolved `migrate.dialect`, else refuse with exit 2 naming `--dialect` — guessing would replay a postgres chain through sqlite. Reading migrate's dialect required amending #292's `EMPTY_MIGRATE_FLAGS` note, which said verify consumes only `outDir`; that restriction was about the drift gate, whose dialect comes from the live URL. Flyway and d1 are refused, mirroring `apply-pending`. Zero committed migrations is not a silent pass, and neither is a missing snapshot: both report and return 0. `discoverMigrations` is module-private, so the empty-chain signal is `ApplyPendingResult.pending` — every migration is pending against a fresh engine, so an empty list means the directory held none. A scoped project's `governed` is derived offline from `scopeExpectedSchema`, and only when `migrate.scope` is actually declared, so an unscoped comparison is unchanged. Tests drive `verifyCommand` end to end against real projects on disk, not just the parser. Three are built specifically so they cannot pass for the wrong reason: a broken chain fails under `--replay-snapshot` ALONE (the dead-flag regression); a chain that applies cleanly but grows a table the snapshot never recorded passes `--replay` and fails `--replay-snapshot`, which is tier 2's entire reason to exist; and the d1 refusal runs under `--skip-schema`, because without it the D1 schema gate also returns 2 and the assertion would hold whether or not the replay gate refused. The migrate-ts side adds the regression the earlier plan was missing: a chain built by `emit` → `writeMigration` → `applyPending`, on both dialects, plus the non-default-schema case. Confirmed RED by reverting the emitter fixes — all three fail — rather than assumed. A hand-written-SQL test cannot do this: it stays green no matter what the emitter writes. Co-Authored-By: Claude Opus 5 (1M context) --- .../packages/cli/src/commands/verify.ts | 195 +++++++++++++++- .../typescript/packages/cli/src/lib/args.ts | 29 ++- .../cli/test/unit/args-verify.test.ts | 5 + .../packages/cli/test/verify-replay.test.ts | 209 ++++++++++++++++++ .../integrity/replay-emitted-chain.test.ts | 112 ++++++++++ 5 files changed, 542 insertions(+), 8 deletions(-) create mode 100644 server/typescript/packages/cli/test/verify-replay.test.ts create mode 100644 server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index f0019baa5..00b1f7b84 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -36,8 +36,13 @@ import { collectUnmanagedNames, excludeFromSnapshot, scopedDiffInputs, + scopeExpectedSchema, buildExpectedSchemaWithProvenance, type GovernedScope, + applyPending, + openReplayEngine, + type ReplayEngine, + verifyReplay, introspect, diff, readSnapshot, @@ -80,9 +85,14 @@ const ERR_UNKNOWN_ATTR = "ERR_UNKNOWN_ATTR"; /** * A no-flags MigrateFlags, so `resolveMigrateConfig` yields exactly what `meta migrate` - * would use with nothing passed on the command line — config value, else default. verify - * consumes only `outDir` from the result (#292); the other fields exist to satisfy the - * shared shape, and reading any of them here would be reaching into migrate's decisions. + * would use with nothing passed on the command line — config value, else default. + * + * verify consumes `outDir` (#292) and — for the replay gate ONLY — `dialect`. The #292 + * restriction that reading anything else "would be reaching into migrate's decisions" + * was written about the DRIFT gate, whose dialect comes from the live `--db` URL. The + * replay gate has no `--db` at all, and the dialect a committed chain was EMITTED for + * is a migrate decision by definition, so migrate's own resolution is the only correct + * source for it. Everything else here exists to satisfy the shared shape. */ const EMPTY_MIGRATE_FLAGS = { db: undefined, dialect: undefined, format: undefined, outDir: undefined, slug: undefined, @@ -229,6 +239,11 @@ export async function verifyCommand( // are checked on every `meta verify`. Opt-in by DECLARATION — a model with no // requirement nodes is silent, not in drift. const requirementExit = runRequirementVerify(); + // #313 — BOTH replay flags select this gate. `--replay-snapshot` implies + // `--replay`'s work, so a broken chain must fail under it even when `--replay` + // was not passed; naming only `flags.replay` here is how `--replay-snapshot` + // would parse cleanly and do nothing at all. + const replayExit = flags.replay || flags.replaySnapshot ? await runReplayVerify() : 0; // Advisory verify-as-teacher pass: surface hand-rolled work the metadata could // model. Warnings ONLY — never changes the exit code (bias to under-flagging). @@ -236,7 +251,179 @@ export async function verifyCommand( // noisy project (both opt-outs work on `meta verify` and `meta gen`). if (!flags.noAntipatterns && process.env.META_NO_ANTIPATTERNS !== "1") runAntiPatternAdvisory(); - return Math.max(templateExit, schemaExit, codegenExit, requirementExit); + return Math.max(templateExit, schemaExit, codegenExit, requirementExit, replayExit); + + // -- replay (#313) --------------------------------------------------------- + /** + * Replay the committed migration chain into an EMPTY throwaway database and assert + * it applies. `--replay-snapshot` additionally asserts the result equals the + * committed snapshot. + * + * This exists because `meta migrate` could write a chain that cannot be replayed — + * a bare `DROP TABLE "x"` for an object no migration ever created — and nothing + * noticed until someone tried to provision a fresh database, which for the reporter + * was three months later. The two tiers are separate because a project adopted via + * `migrate baseline --from-db` passes the first trivially and CANNOT pass the second + * by construction: its snapshot is the whole introspected database and its chain is + * empty. + * + * Exit codes follow verify's convention: a chain that fails to apply, or a snapshot + * mismatch, is drift → 1; an engine that will not start is operational → 2. + */ + async function runReplayVerify(): Promise { + // Resolve the migrations directory and the chain's dialect through MIGRATE's own + // precedence, never a second derivation — verify must not look somewhere migrate + // does not write, nor assume a dialect the chain was not emitted for. + const migrateConfig = await resolveMigrateConfig(EMPTY_MIGRATE_FLAGS, projectRoot); + + if (migrateConfig.format === "flyway") { + log.error( + `meta verify --replay is not supported with --migration-format flyway — run 'flyway migrate' against a scratch database to replay`, + ); + return 2; + } + + // --dialect wins; else migrate's resolved dialect; else refuse. There is no --db + // to infer from, so guessing would replay a postgres chain through sqlite. + const dialect: Dialect | undefined = flags.dialect ?? migrateConfig.dialect; + if (dialect === undefined) { + log.error( + `meta verify --replay: no dialect — pass --dialect , or set migrate.dialect in .metaobjects/config.json`, + ); + return 2; + } + if (dialect === "d1") { + log.error( + `meta verify --replay is not supported for dialect 'd1' — use 'wrangler d1 migrations apply' against a scratch database to replay committed migrations`, + ); + return 2; + } + + const dir = resolvePath(projectRoot, migrateConfig.outDir); + + let engine: ReplayEngine; + try { + engine = await openReplayEngine(dialect); + } catch (err) { + // A missing optional driver lands here, and its message already carries the + // install hint. Operational, not drift. + log.error(`meta verify --replay: ${(err as Error).message}`); + return 2; + } + + try { + let applied; + try { + applied = await applyPending(engine.db, dir, { dryRun: false, dialect }); + } catch (err) { + log.error(`meta verify --replay: ${(err as Error).message}`); + log.error( + `meta verify --replay: the committed chain does not apply to an empty database. ` + + `Applied migrations are immutable, so fix this with a NEW migration that creates the ` + + `missing object — not by editing a committed up.sql.`, + ); + return 1; + } + + // Not a silent pass. `discoverMigrations` returns [] for a missing directory, so + // a run over an empty chain would otherwise "succeed" having proved nothing — + // and a gate that is quiet when it checked nothing cannot be told from one that + // passed. Every migration is pending against a fresh engine, so an empty + // `pending` means the directory held none. + if (applied.pending.length === 0) { + log.info(`meta verify --replay: no committed migrations — nothing to replay`); + return 0; + } + + log.info( + `meta verify --replay — the committed chain applies to an empty ${dialect} database ` + + `(${applied.applied.length} migration(s)).`, + ); + + if (!flags.replaySnapshot) return 0; + return await runReplaySnapshotTier(engine, dialect, dir); + } finally { + await engine.dispose(); + } + } + + /** + * The second tier: the replayed schema must EQUAL the committed snapshot. + * + * This is the 2026-05-31 §8 integrity aid, finally wired — `verifyReplay` has been + * built and exported with no CLI caller since then. What it catches that tier 1 + * cannot is hand-edited structural DDL: a committed up.sql someone changed so the + * chain still applies but no longer produces the schema the snapshot records. + * + * It does NOT support a project adopted via `migrate baseline --from-db`, and does + * not try to detect one. The only candidate signal (`BASELINE_NAME`/`recordBaseline`) + * has no production caller and would live in the TARGET database's ledger, while + * this runs against a fresh engine with no ledger at all. So the failure message + * names baseline adoption as the first thing to rule out. + */ + async function runReplaySnapshotTier( + engine: ReplayEngine, + dialect: Extract, + dir: string, + ): Promise { + // Fails OPEN on a missing snapshot: a project that has never generated one + // offline is not in an error state, and an unreadable/unparseable file is + // migrate's error to raise with its own message, not a drift verdict. It still + // SAYS so — silence here would be indistinguishable from a pass. + let snapshot: SchemaSnapshot | null; + try { + snapshot = await readSnapshot(snapshotPath(dir, dialect)); + } catch { + log.info(`meta verify --replay-snapshot: the committed snapshot could not be read — nothing to compare`); + return 0; + } + if (snapshot === null) { + log.info(`meta verify --replay-snapshot: no committed snapshot — nothing to compare`); + return 0; + } + + // A scoped project carries the OTHER owner's tables into its snapshot on purpose + // and its chain never creates them, so they must leave the comparison. The + // committed snapshot alone cannot be scoped — `scopeExpectedSchema` decides on a + // qualified-name → metadata-FQN provenance map the snapshot does not carry — so + // the expected side is rebuilt from metadata purely to derive that decision. + // + // Only for a project that actually declares `migrate.scope`. An unscoped project + // passes no `governed` and gets the comparison exactly as it was. + let governed: GovernedScope | undefined; + if (schemaScope !== undefined) { + const viewStrategy = forgeConfig?.columnNamingStrategy ?? "snake_case"; + const built = buildExpectedSchemaWithProvenance(root, { + dialect, + columnNamingStrategy: viewStrategy, + views: buildProjectionViews(root, { dialect, columnNamingStrategy: viewStrategy }), + }); + governed = scopeExpectedSchema(built, schemaScope); + } + + // `verifyReplay` calls `applyPending` itself. That is NOT a second replay: the + // first one recorded every migration in this engine's ledger, so the call finds + // nothing pending and returns immediately. + const result = await verifyReplay({ + db: engine.db, + dialect, + migrationsDir: dir, + snapshot, + ...(governed !== undefined ? { governed } : {}), + }); + if (result.ok) { + log.info(`meta verify --replay-snapshot — the replayed chain reproduces the committed snapshot.`); + return 0; + } + + log.error( + `meta verify --replay-snapshot: the replayed chain does not reproduce the committed snapshot. ` + + `If this project was adopted with 'migrate baseline --from-db', its chain does not build the ` + + `schema and this tier does not apply — use --replay instead.`, + ); + for (const line of summarizeDrift([...result.drift, ...result.unmanaged])) log.error(` ${line}`); + return 1; + } // -- requirements (#290) --------------------------------------------------- function runRequirementVerify(): number { diff --git a/server/typescript/packages/cli/src/lib/args.ts b/server/typescript/packages/cli/src/lib/args.ts index 4c232a66c..74a1f3a12 100644 --- a/server/typescript/packages/cli/src/lib/args.ts +++ b/server/typescript/packages/cli/src/lib/args.ts @@ -232,7 +232,20 @@ export interface VerifyFlags { templates: boolean; /** Run the codegen-drift gate (regenerate-to-temp and diff committed output). */ codegen: boolean; - /** Whether ANY explicit subverb flag (--templates/--db/--codegen) was passed. */ + /** + * Replay the committed migration chain into an empty throwaway database and assert + * it applies (#313). Needs no `--db`: the engine is local and disposable (PGlite + * for postgres, a temp sqlite file), so the gate provisions nothing. + */ + replay: boolean; + /** + * `--replay` plus: assert the replayed schema EQUALS the committed snapshot. A + * separate subverb rather than a `--strict` modifier, because `--lax` below is a + * different axis (ADR-0023 attribute strictness) and `--strict` beside it would + * read as that flag's opposite rather than as a replay depth. + */ + replaySnapshot: boolean; + /** Whether ANY explicit subverb flag (--templates/--db/--codegen/--replay*) was passed. */ anyExplicit: boolean; /** Suppress the advisory anti-pattern (verify-as-teacher) pass. */ noAntipatterns: boolean; @@ -255,6 +268,8 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { "skip-schema": { type: "boolean", default: false }, templates: { type: "boolean", default: false }, codegen: { type: "boolean", default: false }, + replay: { type: "boolean", default: false }, + "replay-snapshot": { type: "boolean", default: false }, "no-antipatterns": { type: "boolean", default: false }, lax: { type: "boolean", default: false }, "d1": { type: "string" }, @@ -283,11 +298,15 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { const templates = !!values.templates; const codegen = !!values.codegen; + const replay = !!values.replay; + const replaySnapshot = !!values["replay-snapshot"]; // --db is itself an explicit subverb selector: passing a connection URL means // "run the schema-drift mode". So is `--dialect d1` (D1 has no --db connection - // URL — see the `d1` field doc above). So "any explicit subverb" is - // templates|codegen|db|dialect==d1. - const anyExplicit = templates || codegen || values.db !== undefined || dialect === "d1"; + // URL — see the `d1` field doc above). The replay flags are subverbs too, and + // must be listed here or `meta verify --replay` would ALSO run the template gate + // as the bare-verify default. + const anyExplicit = + templates || codegen || values.db !== undefined || dialect === "d1" || replay || replaySnapshot; return { prompts: values.prompts, @@ -297,6 +316,8 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { skipSchema: !!values["skip-schema"], templates, codegen, + replay, + replaySnapshot, anyExplicit, noAntipatterns: !!values["no-antipatterns"], lax: !!values.lax, diff --git a/server/typescript/packages/cli/test/unit/args-verify.test.ts b/server/typescript/packages/cli/test/unit/args-verify.test.ts index 33d018f92..b85276fa3 100644 --- a/server/typescript/packages/cli/test/unit/args-verify.test.ts +++ b/server/typescript/packages/cli/test/unit/args-verify.test.ts @@ -6,6 +6,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs([])).toEqual({ prompts: undefined, db: undefined, dialect: undefined, allow: [], skipSchema: false, templates: false, codegen: false, anyExplicit: false, noAntipatterns: false, lax: false, + replay: false, replaySnapshot: false, d1: undefined, remote: false, }); }); @@ -13,6 +14,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs(["--prompts", "templates"])).toEqual({ prompts: "templates", db: undefined, dialect: undefined, allow: [], skipSchema: false, templates: false, codegen: false, anyExplicit: false, noAntipatterns: false, lax: false, + replay: false, replaySnapshot: false, d1: undefined, remote: false, }); }); @@ -20,6 +22,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs(["--db", "file:x.db", "--dialect", "sqlite", "--skip-schema"])).toEqual({ prompts: undefined, db: "file:x.db", dialect: "sqlite", allow: [], skipSchema: true, templates: false, codegen: false, anyExplicit: true, noAntipatterns: false, lax: false, + replay: false, replaySnapshot: false, d1: undefined, remote: false, }); }); @@ -29,6 +32,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs(["--dialect", "d1", "--d1", "DB", "--remote"])).toEqual({ prompts: undefined, db: undefined, dialect: "d1", allow: [], skipSchema: false, templates: false, codegen: false, anyExplicit: true, noAntipatterns: false, lax: false, + replay: false, replaySnapshot: false, d1: "DB", remote: true, }); const bare = parseVerifyArgs(["--dialect", "d1"]); @@ -52,6 +56,7 @@ describe("parseVerifyArgs", () => { prompts: undefined, db: undefined, dialect: undefined, allow: ["drop-column", "drop-table"], skipSchema: false, templates: false, codegen: false, anyExplicit: false, noAntipatterns: false, lax: false, + replay: false, replaySnapshot: false, d1: undefined, remote: false, }); }); diff --git a/server/typescript/packages/cli/test/verify-replay.test.ts b/server/typescript/packages/cli/test/verify-replay.test.ts new file mode 100644 index 000000000..07f869302 --- /dev/null +++ b/server/typescript/packages/cli/test/verify-replay.test.ts @@ -0,0 +1,209 @@ +/** + * `meta verify --replay` / `--replay-snapshot` (#313). + * + * `meta migrate` could write a chain that cannot be replayed — a bare + * `DROP TABLE "x"` for an object no migration ever created, because another tool + * owns it — and nothing noticed until someone provisioned a fresh database. These + * gates replay the committed chain into an empty throwaway engine and say so. + * + * The flag-parse cases below are necessary and nowhere near sufficient: a flag that + * parses correctly and reaches no gate is exactly the failure mode this feature is + * most exposed to, so every tier has a case that drives `verifyCommand` end to end + * against a real project on disk. + */ +import { describe, test, expect, afterAll } from "bun:test"; +import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { parseVerifyArgs } from "../src/lib/args.js"; +import { verifyCommand } from "../src/commands/verify.js"; +import { runBaseline, runOfflineGenerate } from "../src/commands/migrate.js"; + +const dirs: string[] = []; +afterAll(async () => { for (const d of dirs) await rm(d, { recursive: true, force: true }); }); + +const MIGRATIONS = "./.metaobjects/migrations"; + +/** A shape carrying no `source.*`, so it is not persistable and the schema is empty. */ +const NO_TABLES = JSON.stringify({ + "metadata.root": { + package: "acme::platform", + children: [{ + "object.value": { + name: "Placeholder", + children: [{ "field.string": { name: "note" } }], + }, + }], + }, +}); + +const MODEL = JSON.stringify({ + "metadata.root": { + package: "acme::platform", + children: [{ + "object.entity": { + name: "Job", + children: [ + { "source.rdb": { name: "src", "@table": "jobs" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "title", "@maxLength": 80 } }, + { "identity.primary": { name: "pk", "@fields": ["id"], "@generation": "increment" } }, + ], + }, + }], + }, +}); + +const cfg = () => + ({ dialect: "sqlite", outDir: MIGRATIONS, onAmbiguous: "abort", + allow: [], slug: "init", dryRun: false } as never); + +/** A project with metadata and a declared sqlite dialect, but no migrations yet. */ +async function project(): Promise { + const root = await mkdtemp(join(tmpdir(), "verify-replay-")); + dirs.push(root); + await mkdir(join(root, "metaobjects"), { recursive: true }); + await writeFile(join(root, "metaobjects", "meta.platform.json"), MODEL, "utf8"); + await mkdir(join(root, ".metaobjects", "migrations"), { recursive: true }); + await writeFile( + join(root, ".metaobjects", "config.json"), + JSON.stringify({ schema_version: 1, migrate: { dialect: "sqlite", outDir: MIGRATIONS } }), + "utf8", + ); + return root; +} + +/** + * A genuine greenfield chain: baseline against a model declaring NO tables (so the + * reference snapshot starts empty, as a fresh project's does), then generate. The + * result is a chain that BUILDS the schema — which is the population `--replay` and + * `--replay-snapshot` are for, and the opposite of a `baseline --from-db` adoption + * whose snapshot is the whole database against an empty chain. + */ +async function withGeneratedChain(root: string): Promise { + await writeFile(join(root, "metaobjects", "meta.platform.json"), NO_TABLES, "utf8"); + expect(await runBaseline(cfg(), root)).toBe(0); + await writeFile(join(root, "metaobjects", "meta.platform.json"), MODEL, "utf8"); + expect(await runOfflineGenerate(cfg(), root)).toBe(0); +} + +/** A hand-written chain that applies cleanly. */ +async function withApplyingChain(root: string): Promise { + const dir = join(root, ".metaobjects", "migrations", "20260101000000-init"); + await mkdir(dir, { recursive: true }); + await writeFile(join(dir, "up.sql"), 'CREATE TABLE "jobs" (id INTEGER NOT NULL PRIMARY KEY);', "utf8"); + await writeFile(join(dir, "down.sql"), 'DROP TABLE "jobs";', "utf8"); +} + +/** Hand-write a migration whose up.sql drops something the chain never creates. */ +async function withBrokenChain(root: string): Promise { + const dir = join(root, ".metaobjects", "migrations", "20260101000000-init"); + await mkdir(dir, { recursive: true }); + await writeFile( + join(dir, "up.sql"), + 'CREATE TABLE "jobs" (id INTEGER NOT NULL PRIMARY KEY);\n\nDROP TABLE "theirs";', + "utf8", + ); + await writeFile(join(dir, "down.sql"), 'DROP TABLE "jobs";', "utf8"); +} + +describe("verify --replay / --replay-snapshot flags", () => { + test("both are parsed", () => { + expect(parseVerifyArgs(["--replay"]).replay).toBe(true); + expect(parseVerifyArgs(["--replay-snapshot"]).replaySnapshot).toBe(true); + }); + + test("both default off", () => { + expect(parseVerifyArgs([]).replay).toBe(false); + expect(parseVerifyArgs([]).replaySnapshot).toBe(false); + }); + + // Without this, `meta verify --replay` would ALSO run the template gate as the + // bare-verify default, and report drift the user never asked about. + test("each counts as an explicit subverb", () => { + expect(parseVerifyArgs(["--replay"]).anyExplicit).toBe(true); + expect(parseVerifyArgs(["--replay-snapshot"]).anyExplicit).toBe(true); + }); + + // `--lax` is ADR-0023 attribute strictness — a different axis entirely, which is + // why the second tier is its own subverb and not `--strict`. + test("does not collide with --lax", () => { + const f = parseVerifyArgs(["--replay-snapshot", "--lax"]); + expect(f.replaySnapshot).toBe(true); + expect(f.lax).toBe(true); + }); +}); + +describe("verify --replay runs the gate", () => { + test("a chain that drops an object it never creates fails", async () => { + const root = await project(); + await withBrokenChain(root); + expect(await verifyCommand(["--replay"], root)).toBe(1); + }); + + test("a generated chain applies to an empty database", async () => { + const root = await project(); + await withGeneratedChain(root); + expect(await verifyCommand(["--replay"], root)).toBe(0); + }); + + // Not a silent pass: a run over an empty chain proves nothing, and a gate that is + // quiet when it checked nothing cannot be told from one that passed. + test("no committed migrations reports that, and passes", async () => { + const root = await project(); + expect(await verifyCommand(["--replay"], root)).toBe(0); + }); + + // `--skip-schema` is load-bearing here, not incidental: without it the D1 SCHEMA + // gate also returns 2 (no wrangler.toml), so the assertion would pass whether or + // not the replay gate refuses at all. + test("d1 is refused, operationally", async () => { + const root = await project(); + await withApplyingChain(root); + expect(await verifyCommand(["--replay", "--dialect", "d1", "--skip-schema"], root)).toBe(2); + // The control: the same run on the project's real dialect passes, so the 2 above + // is the d1 refusal and not something wrong with the fixture. + expect(await verifyCommand(["--replay", "--skip-schema"], root)).toBe(0); + }); +}); + +describe("verify --replay-snapshot runs BOTH tiers", () => { + // The regression that would let the flag ship dead: --replay-snapshot implies + // --replay's work, so a broken chain must fail under it even though --replay was + // never passed. + test("a broken chain fails under --replay-snapshot alone", async () => { + const root = await project(); + await withBrokenChain(root); + expect(await verifyCommand(["--replay-snapshot"], root)).toBe(1); + }); + + test("a generated chain reproduces its own committed snapshot", async () => { + const root = await project(); + await withGeneratedChain(root); + expect(await verifyCommand(["--replay-snapshot"], root)).toBe(0); + }); + + // Tier 2's whole reason for existing, and what keeps the case above honest: a chain + // that APPLIES but no longer produces the schema the snapshot records. This is the + // hand-edited-structural-DDL case — tier 1 cannot see it. + test("a chain that applies but diverges from the snapshot fails", async () => { + const root = await project(); + await withGeneratedChain(root); + expect(await verifyCommand(["--replay"], root)).toBe(0); // tier 1 is blind to it + const extra = join(root, ".metaobjects", "migrations", "29991231000000-hand-edit"); + await mkdir(extra, { recursive: true }); + await writeFile(join(extra, "up.sql"), 'CREATE TABLE "unrecorded" (id INTEGER NOT NULL PRIMARY KEY);', "utf8"); + await writeFile(join(extra, "down.sql"), 'DROP TABLE "unrecorded";', "utf8"); + expect(await verifyCommand(["--replay"], root)).toBe(0); // still applies fine + expect(await verifyCommand(["--replay-snapshot"], root)).toBe(1); + }); + + // Fails OPEN: a project that has never generated a snapshot offline is not in an + // error state — but the tier says so rather than passing in silence. + test("no committed snapshot is not a failure", async () => { + const root = await project(); + // A chain that APPLIES, so tier 1 passes and this isolates the snapshot half. + await withApplyingChain(root); + expect(await verifyCommand(["--replay-snapshot"], root)).toBe(0); + }); +}); diff --git a/server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts b/server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts new file mode 100644 index 000000000..5206cd2a1 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts @@ -0,0 +1,112 @@ +// #313, end to end: the EMITTER's own output, written by `writeMigration`, applied +// by `applyPending` into an empty database. +// +// Every prior defect in this area (#226/#241, #243, #255, #285, and 0.21.4's +// `BEGIN TRANSACTION` finding) shared one shape — SQL proven statement-by-statement +// and never proven through the tool that applies it. `applyPending` splits and +// rewrites statements before executing them, so an emit-level string assertion +// cannot see this class of bug. A hand-written-SQL test cannot either: it stays +// green no matter what the emitter does. +import { describe, test, expect } from "bun:test"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { emit } from "../../src/emit/index.js"; +import { writeMigration } from "../../src/write-migration.js"; +import { applyPending } from "../../src/apply/apply.js"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import type { Change, ChangeStatus } from "../../src/types.js"; + +const ALLOWED: ChangeStatus = { state: "allowed" }; + +// Exactly the reported shape: another tool owned `theirs`, so the diff proposed +// dropping it, and no migration in the chain ever created it. +const REPORTED: Change[] = [ + { + kind: "create-table", + status: ALLOWED, + table: { + name: "mine", + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], + }, + }, + { kind: "drop-table", table: "theirs", status: ALLOWED }, +]; + +describe("an EMITTED chain applies to an empty database (#313)", () => { + for (const dialect of ["sqlite", "postgres"] as const) { + test(`${dialect}: emit → writeMigration → applyPending, from empty`, async () => { + const dir = mkdtempSync(join(tmpdir(), `replay-emitted-${dialect}-`)); + const engine = await openReplayEngine(dialect); + try { + await writeMigration(emit(REPORTED, { dialect }), { dir, slug: "init" }); + const applied = await applyPending(engine.db, dir, { dryRun: false, dialect }); + expect(applied.applied).toHaveLength(1); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + } + + // A chain creating a table in a non-default schema needs `CREATE SCHEMA` ahead of + // it, which no migration used to emit. Postgres-only — sqlite has no schemas. + test("postgres: an emitted chain creating a non-default schema's table applies", async () => { + const dir = mkdtempSync(join(tmpdir(), "replay-emitted-schema-")); + const engine = await openReplayEngine("postgres"); + try { + await writeMigration( + emit( + [ + { + kind: "create-table", + status: ALLOWED, + table: { + name: "x", + schema: "reporting", + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], + }, + }, + ], + { dialect: "postgres" }, + ), + { dir, slug: "init" }, + ); + const applied = await applyPending(engine.db, dir, { dryRun: false, dialect: "postgres" }); + expect(applied.applied).toHaveLength(1); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + // The control: without it the cases above could pass because `applyPending` + // swallows a failing statement rather than because the emitter stopped writing + // one. This proves the assertion has teeth. + test("the control: a HAND-WRITTEN bare drop still fails the replay", async () => { + const dir = mkdtempSync(join(tmpdir(), "replay-emitted-control-")); + const engine = await openReplayEngine("sqlite"); + try { + await writeMigration( + { + up: 'CREATE TABLE "mine" (id INTEGER NOT NULL PRIMARY KEY);\n\nDROP TABLE "theirs";', + down: 'DROP TABLE "mine";', + }, + { dir, slug: "init" }, + ); + await expect(applyPending(engine.db, dir, { dryRun: false, dialect: "sqlite" })) + .rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); From 4af89fa49d2288a1a3048269f5de265bd94f5558 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 07:49:53 -0400 Subject: [PATCH 13/44] feat(cli): refuse to drop an object the committed snapshot never managed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The replay gates make a broken chain survivable and detectable. This stops it being written. The live migrate path diffs metadata against introspection and never reads the committed snapshot, so a table another tool owns reads as "in the DB, not in the model" and is proposed for a DROP. The migration that results cannot replay against a database where that object never existed — which is how the reported chain stayed broken for three months while every `meta migrate` reported success. `drift/classify.ts` has always stated the doctrine (objects present in the DB but not the snapshot "must never be treated as actionable drift or auto-dropped"); this is the first place it is enforced where it mattered. `meta migrate --from-db` now collects every `drop-table`/`drop-view` whose qualified name is absent from the committed snapshot and, without `--allow drop-unmanaged`, refuses with exit 2 naming each object. It does NOT false-fire on brownfield projects, and the reason is structural: both mechanisms ADD to the snapshot. A `baseline --from-db` snapshot contains the foreign table, and a scoped project carries its out-of-scope entries forward. The guard fires precisely when nothing ever claimed the object. Both are pinned as tests, because a guard that broke every adopted project would look identical in code review. It FAILS OPEN on a missing or unreadable snapshot — refusing there would break the first `meta migrate` of every greenfield project, which has no snapshot by definition. And it lives on the live path only, which is not an omission: the offline path diffs metadata against the snapshot, so it can only ever propose dropping something the snapshot HAS. Names come from `qualifiedDbName` and nothing else. Three independent sets already have to agree on that spelling — the diff's identity maps, the `@unmanaged` exclusion set, the out-of-scope set — and a fourth encoding of "absent schema means public" would silently un-guard whatever it disagreed about. A new `--allow` token touches four files, three of them pinned together by `allow-tokens-pinned.test.ts`: `ALLOW_TOKENS` (cli, the validator), `AllowTokenEnum` (sdk, which validates `migrate.allow` in config.json), `ALLOW_TOKEN_MAP` (cli, what actually GRANTS), and now `AllowOptions.dropUnmanaged`. Ruled deliberately: `dropUnmanaged` joins `AllowOptions` even though `diff()` never reads it, because the alternative — a second token list and a second parse path for one token — is exactly the drift that pin exists to prevent. The field documents that exception rather than leaving it a puzzle. Co-Authored-By: Claude Opus 5 (1M context) --- .../packages/cli/src/commands/migrate.ts | 79 ++++++++++ .../typescript/packages/cli/src/lib/allow.ts | 4 + .../typescript/packages/cli/src/lib/args.ts | 7 + .../cli/test/migrate-drop-unmanaged.test.ts | 147 ++++++++++++++++++ .../packages/migrate-ts/src/types.ts | 17 ++ server/typescript/packages/sdk/src/config.ts | 1 + 6 files changed, 255 insertions(+) create mode 100644 server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts diff --git a/server/typescript/packages/cli/src/commands/migrate.ts b/server/typescript/packages/cli/src/commands/migrate.ts index 3d6041145..b3adeef2f 100644 --- a/server/typescript/packages/cli/src/commands/migrate.ts +++ b/server/typescript/packages/cli/src/commands/migrate.ts @@ -29,6 +29,7 @@ import { snapshotPath, readSnapshot, writeSnapshot, + qualifiedDbName, BlockedChangesError, PrimaryKeyChangeError, renderD1, @@ -47,6 +48,7 @@ import { type EmitResult, type D1Runner, type SchemaProvenance, + type SchemaSnapshot, } from "@metaobjectsdev/migrate-ts"; import { buildWranglerExecuteArgs, @@ -291,6 +293,45 @@ function summarizeChanges(changes: Change[]): Record { return counts; } +/** + * The qualified names of tables/views this diff proposes to DROP that the committed + * snapshot never contained — i.e. objects this toolchain never managed (#313). + * + * FAILS OPEN. No snapshot on disk, or one that cannot be read, yields an empty list: + * a project that has never generated one is not in an error state, and refusing there + * would break the first `meta migrate` of every greenfield project. A parse failure is + * migrate's own error to raise elsewhere, with its own message, not a silent refusal + * here. + * + * Names come from `qualifiedDbName` and nothing else. Three independent sets already + * have to agree on this spelling — the diff's identity maps, the `@unmanaged` + * exclusion set, and the out-of-scope set — and a fourth encoding of "absent schema + * means public" would silently un-guard every object it disagreed about. + */ +async function snapshotAbsentDrops(changes: Change[], snapPath: string): Promise { + let snapshot: SchemaSnapshot | null; + try { + snapshot = await readSnapshot(snapPath); + } catch { + return []; + } + if (snapshot === null) return []; + + const managed = new Set(); + for (const t of snapshot.tables) managed.add(qualifiedDbName(t)); + for (const v of snapshot.views) managed.add(qualifiedDbName(v)); + + const absent: string[] = []; + for (const c of changes) { + const name = + c.kind === "drop-table" ? qualifiedDbName({ name: c.table, schema: c.schema }) + : c.kind === "drop-view" ? qualifiedDbName({ name: c.view, schema: c.schema }) + : undefined; + if (name !== undefined && !managed.has(name)) absent.push(name); + } + return absent; +} + function allowFlagFor(kind: string): string { switch (kind) { case "drop-column": return "drop-column"; @@ -659,6 +700,44 @@ export async function migrateCommand( changeCounts = summarizeChanges(diffResult.changes); + // #313 — refuse to AUTHOR a drop for an object the committed snapshot never + // contained. This path diffs metadata against introspection and never reads the + // snapshot, so an object another tool owns reads as "in the DB, not in the model" + // and is proposed for a drop; the migration that results cannot replay against a + // database where that object never existed, which is how a chain stays broken for + // months. `classify.ts` already states the doctrine — objects present in the DB + // but not the snapshot "must never be treated as actionable drift or + // auto-dropped" — and this is where it is finally enforced. + // + // It does not false-fire on the brownfield cases, because both of them ADD to the + // snapshot: a `baseline --from-db` snapshot contains the foreign table, and a + // scoped project carries its out-of-scope entries forward. The guard fires + // precisely when nothing ever claimed the object. + // + // Only on THIS path, and that is not an omission: the offline path diffs metadata + // against the committed snapshot, so it proposes a drop only for an object the + // snapshot HAS. A snapshot-absent drop is unreachable there by construction. + const unmanagedDrops = await snapshotAbsentDrops( + diffResult.changes, + snapshotPath(resolvePath(metaRoot, config.outDir), kysely.dialect), + ); + if (unmanagedDrops.length > 0 && tokensToAllowOptions(config.allow).dropUnmanaged !== true) { + const named = unmanagedDrops.join(", "); + log.error( + `migrate: refusing to drop ${named} — absent from the committed schema snapshot, so this ` + + `toolchain never managed ${unmanagedDrops.length === 1 ? "it" : "them"} and the migration ` + + `could not replay against a database where ${unmanagedDrops.length === 1 ? "it" : "they"} ` + + `never existed. Re-run with '--allow drop-unmanaged' if the drop is intended.`, + ); + emitStructuredError( + `migrate: refusing to drop ${named} — absent from the committed schema snapshot`, + "re-run with '--allow drop-unmanaged' if the drop is intended", + fmt, + ); + await kysely.close(); + return 2; + } + // All changes — tables AND views — are emitted by the one schema-diff path. // View DDL (create/drop/replace) is produced by diff()'s view passes (2b body // comparison, 2c dependency-recreate) and rendered by every dialect's emitter; diff --git a/server/typescript/packages/cli/src/lib/allow.ts b/server/typescript/packages/cli/src/lib/allow.ts index 947606f0f..dfb5c9865 100644 --- a/server/typescript/packages/cli/src/lib/allow.ts +++ b/server/typescript/packages/cli/src/lib/allow.ts @@ -33,6 +33,10 @@ export const ALLOW_TOKEN_MAP: Record = { // @generation at all — ambiguous between "never declared it" and // "deliberately removing auto-increment", so migrate refuses without it. "drop-identity-default": "dropIdentityDefault", + // Gates dropping an object the committed snapshot never contained (#313). Read by + // migrate's generation-time provenance guard, not by diff()'s status pass — see + // AllowOptions.dropUnmanaged for why it still belongs in that shape. + "drop-unmanaged": "dropUnmanaged", }; /** Translate parsed `--allow` tokens into the migrate-ts `AllowOptions` shape. */ diff --git a/server/typescript/packages/cli/src/lib/args.ts b/server/typescript/packages/cli/src/lib/args.ts index 74a1f3a12..b67d5c620 100644 --- a/server/typescript/packages/cli/src/lib/args.ts +++ b/server/typescript/packages/cli/src/lib/args.ts @@ -197,6 +197,13 @@ export const ALLOW_TOKENS = [ // metadata declares no @generation at all — ambiguous between "never // declared it" and "deliberately removing auto-increment". "drop-identity-default", + // drop-unmanaged permits dropping an object the COMMITTED SNAPSHOT never + // contained — i.e. one this toolchain never managed, typically a table another + // tool owns. Without it such a drop is refused at generation time, because the + // migration it writes cannot replay against a database where that object never + // existed (#313). Unlike its neighbours this one is enforced by `migrate` itself + // rather than by `diff()`'s status pass; see AllowOptions.dropUnmanaged. + "drop-unmanaged", ] as const; type AllowToken = (typeof ALLOW_TOKENS)[number]; diff --git a/server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts b/server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts new file mode 100644 index 000000000..bc326014c --- /dev/null +++ b/server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts @@ -0,0 +1,147 @@ +/** + * `--allow drop-unmanaged` — refuse to AUTHOR a drop for an object the committed + * snapshot never contained (#313). + * + * The live migrate path diffs metadata against introspection and never reads the + * snapshot, so a table another tool owns reads as "in the DB, not in the model" and + * is proposed for a drop. The migration that results cannot replay against a database + * where that object never existed — which is how the reported chain stayed broken for + * three months. `classify.ts` already stated the doctrine; this is where it is + * enforced. + */ +import { describe, test, expect, afterAll } from "bun:test"; +import { mkdtemp, rm, mkdir, writeFile, readdir } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { sql } from "kysely"; +import { migrateCommand } from "../src/commands/migrate.js"; +import { buildKyselyFromUrl } from "../src/lib/kysely.js"; +import { ALLOW_TOKENS } from "../src/lib/args.js"; +import { ALLOW_TOKEN_MAP, tokensToAllowOptions } from "../src/lib/allow.js"; +import { snapshotPath, writeSnapshot, type TableDescriptor } from "@metaobjectsdev/migrate-ts"; + +const dirs: string[] = []; +afterAll(async () => { for (const d of dirs) await rm(d, { recursive: true, force: true }); }); + +const MODEL = JSON.stringify({ + "metadata.root": { + package: "acme::platform", + children: [{ + "object.entity": { + name: "Job", + children: [ + { "source.rdb": { name: "src", "@table": "jobs" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"], "@generation": "increment" } }, + ], + }, + }], + }, +}); + +interface Fixture { root: string; db: string } + +/** + * A project modelling only `jobs`, against a database that ALSO holds `theirs` — + * another tool's table. `seedSnapshot` decides the case: `managed` writes a snapshot + * containing both (what `baseline --from-db` produces), `unmanaged` writes one + * containing only `jobs`, and `none` writes no snapshot at all. + */ +async function fixture(seedSnapshot: "managed" | "unmanaged" | "none"): Promise { + const root = await mkdtemp(join(tmpdir(), "drop-unmanaged-")); + dirs.push(root); + await mkdir(join(root, "metaobjects"), { recursive: true }); + await writeFile(join(root, "metaobjects", "meta.platform.json"), MODEL, "utf8"); + await mkdir(join(root, ".metaobjects", "migrations"), { recursive: true }); + await writeFile( + join(root, ".metaobjects", "config.json"), + JSON.stringify({ schema_version: 1, migrate: { dialect: "sqlite" } }), + "utf8", + ); + + const db = join(root, "t.db"); + const k = await buildKyselyFromUrl(`file:${db}`, "sqlite"); + try { + await sql`CREATE TABLE "jobs" (id INTEGER NOT NULL PRIMARY KEY)`.execute(k.db); + await sql`CREATE TABLE "theirs" (id INTEGER NOT NULL PRIMARY KEY)`.execute(k.db); + } finally { + await k.close(); + } + + if (seedSnapshot !== "none") { + // Written through migrate's own writer, not hand-rolled JSON: the on-disk shape + // is versioned and nested, and a fixture that guessed it would be testing the + // guard's error path instead of the guard. + const t = (name: string): TableDescriptor => ({ + name, + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], + }); + await writeSnapshot(snapshotPath(join(root, ".metaobjects", "migrations"), "sqlite"), { + tables: seedSnapshot === "managed" ? [t("jobs"), t("theirs")] : [t("jobs")], + views: [], + }); + } + return { root, db }; +} + +const migrationCount = async (root: string): Promise => + (await readdir(join(root, ".metaobjects", "migrations"))) + .filter((e) => !e.startsWith(".")).length; + +/** + * `--from-db` is required, not incidental: a bare `meta migrate --db ` takes the + * OFFLINE path, which diffs metadata against the committed snapshot. There, a drop is + * proposed only for an object the snapshot HAS — so the offline path cannot produce a + * snapshot-absent drop by construction, and the guard belongs to the live path alone. + */ +const run = (f: Fixture, extra: string[] = []): Promise => + migrateCommand( + ["--from-db", "--db", `file:${f.db}`, "--dialect", "sqlite", "--slug", "x", + "--allow", ["drop-table", ...extra].join(",")], + f.root, + ); + +describe("drop-unmanaged is wired through every token structure", () => { + test("is a recognised allow token", () => { + expect(ALLOW_TOKENS).toContain("drop-unmanaged"); + }); + + // A token in ALLOW_TOKENS but absent from the map validates cleanly and then grants + // NOTHING — the silent-failure mode allow-tokens-pinned.test.ts exists to prevent. + test("grants a permission rather than validating into nothing", () => { + expect(ALLOW_TOKEN_MAP["drop-unmanaged"]).toBe("dropUnmanaged"); + expect(tokensToAllowOptions(["drop-unmanaged"]).dropUnmanaged).toBe(true); + }); +}); + +describe("meta migrate refuses a snapshot-absent drop", () => { + test("refuses, and writes nothing", async () => { + const f = await fixture("unmanaged"); + expect(await run(f)).toBe(2); + expect(await migrationCount(f.root)).toBe(0); + }); + + test("--allow drop-unmanaged lets it through", async () => { + const f = await fixture("unmanaged"); + expect(await run(f, ["drop-unmanaged"])).toBe(0); + expect(await migrationCount(f.root)).toBe(1); + }); + + // The brownfield non-false-fire, and the reason the guard is safe to ship on by + // default: `baseline --from-db` puts the foreign table IN the snapshot, so the + // guard reads it as managed. Without this case the guard would look correct while + // breaking every adopted project. + test("a drop for a table the snapshot DOES contain proceeds without the flag", async () => { + const f = await fixture("managed"); + expect(await run(f)).toBe(0); + expect(await migrationCount(f.root)).toBe(1); + }); + + // Fails OPEN: refusing here would break the first `meta migrate` of every + // greenfield project, which has no snapshot yet by definition. + test("no snapshot on disk does not trigger the refusal", async () => { + const f = await fixture("none"); + expect(await run(f)).toBe(0); + }); +}); diff --git a/server/typescript/packages/migrate-ts/src/types.ts b/server/typescript/packages/migrate-ts/src/types.ts index 3006d6712..58d352980 100644 --- a/server/typescript/packages/migrate-ts/src/types.ts +++ b/server/typescript/packages/migrate-ts/src/types.ts @@ -347,6 +347,23 @@ export interface AllowOptions { * skips the default-diff for a live auto-sequence default entirely.) */ dropIdentityDefault?: boolean; + /** + * Permits dropping an object the COMMITTED SNAPSHOT never contained — i.e. one + * this toolchain never managed. Without it, such a drop is refused at generation + * time, because the migration it would write cannot replay against a database + * where that object never existed (#313). `classify.ts` already states the + * doctrine: objects present in the DB but not the snapshot "must never be treated + * as actionable drift or auto-dropped". + * + * The ONE field here read by the CLI's generation-time provenance guard rather + * than by `diff()`'s status pass — `diff` compares metadata against introspection + * and never sees the snapshot, which is precisely why the doctrine was not + * enforced where it mattered. It lives in `AllowOptions` anyway so `--allow` keeps + * ONE token list and ONE grant map (`ALLOW_TOKENS` / `ALLOW_TOKEN_MAP`, pinned + * together by `cli/test/unit/allow-tokens-pinned.test.ts`): a second parallel + * validation path for a single token is the exact drift that pin exists to catch. + */ + dropUnmanaged?: boolean; } export type AmbiguousChange = diff --git a/server/typescript/packages/sdk/src/config.ts b/server/typescript/packages/sdk/src/config.ts index 5f16a466e..0e70467d7 100644 --- a/server/typescript/packages/sdk/src/config.ts +++ b/server/typescript/packages/sdk/src/config.ts @@ -31,6 +31,7 @@ export const AllowTokenEnum = z.enum([ "adopt-view", "nullable-to-not-null", "drop-identity-default", + "drop-unmanaged", ]); // .strict(), like every other object in this schema: `.partial()` alone leaves From 37a83ce67117e75f37f86f08b147eacbb77f9e8b Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 07:53:31 -0400 Subject: [PATCH 14/44] docs: replay tiers, the drop-unmanaged refusal, and the provisioning promise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `docs/features/migrations-and-drift.md` and `meta migrate --help` both promised that `apply-pending` "is the way to provision a fresh or CI database". That is true only of a chain that BUILDS the schema, and nothing checked — which is the promise #313 broke. Both are now scoped to that population and point at `meta verify --replay` as the way to know you are in it; a project adopted with `baseline --from-db` is named as the case that is not. Adds an adopter-facing section for the two replay tiers: what each asserts, why they are two rather than one, that they provision nothing (PGlite in-process / a throwaway temp file) and need no `--db`, that PGlite is an optional peer, how the dialect resolves without a URL, the exit-code convention, and — stated plainly rather than left to be discovered — that `--replay-snapshot` cannot pass for a baseline-adopted project. It also documents the remediation, since a gate whose failure has no documented exit gets suppressed, and applied migrations are checksum-immutable so hand-editing a committed up.sql is not it. Documents `--allow drop-unmanaged` with the real refusal text, and why it does not fire for brownfield projects. Documents the emitter changes with their deliberate exclusions, so the forward-only rule is written down rather than remembered. CHANGELOG leads with the new refusal, because it is the one change here that can fail an existing project's `meta migrate`; the emitter changes and the two subverbs follow. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 83 ++++++++++++++++ docs/features/migrations-and-drift.md | 94 ++++++++++++++++++- .../packages/cli/src/commands/migrate.ts | 15 ++- .../packages/cli/src/commands/verify.ts | 3 +- server/typescript/packages/cli/src/index.ts | 13 ++- 5 files changed, 200 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 875fb69cf..38f882257 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,89 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ## [Unreleased] +### Changed — a committed migration chain must replay from empty, and `meta migrate` stops writing chains that cannot ([#313](https://github.com/metaobjectsdev/metaobjects/issues/313)) + +**`meta migrate --from-db` now REFUSES a drop for a table or view the committed schema +snapshot never contained**, exiting 2 and naming each object. This is the one change here +that can fail an existing project's `meta migrate`, so it leads. Pass +**`--allow drop-unmanaged`** when the drop is genuinely intended. + +The refusal exists because the drop it blocks produces a migration nobody can replay. The +live migrate path diffs metadata against introspection and never reads the snapshot, so a +table another tool owns reads as "in the database, not in the model" and is proposed for a +`DROP TABLE`. Every incremental migrate then keeps succeeding against the database that +already has that table — the chain only fails the day someone provisions a fresh one, which +for the reporter was **three months later**, by which point the only working database left +was a leftover CI container. `drift/classify.ts` has always said objects present in the DB +but not the snapshot "must never be treated as actionable drift or auto-dropped"; this is +the first place that doctrine is enforced where it mattered. + +It does not false-fire on brownfield projects, and the reason is structural rather than +special-cased: **both mechanisms ADD to the snapshot.** A `baseline --from-db` snapshot +contains the foreign table; a project declaring `migrate.scope` carries its out-of-scope +entries forward. The guard fires precisely when nothing ever claimed the object. It fails +OPEN with no snapshot on disk — refusing there would break the first `meta migrate` of every +greenfield project — and it lives on the live path only, because the offline path diffs +against the snapshot and so cannot propose a snapshot-absent drop at all. + +**Emitted forward drops now carry `IF EXISTS`** — `drop-table`, `drop-view` (plain and +CASCADE), `drop-index` (both the plain form and #285's constraint-backed +`ALTER TABLE … DROP CONSTRAINT`), `drop-fk` and `drop-check` — in both dialects, so an +already-absent object cannot break a replay. **Down statements stay bare, deliberately:** +`rollbackTo` runs `down.sql` and the ledger delete in ONE transaction, so a guarded down +would no-op and still record the rollback as done. Rollback is the one place a loud failure +is load-bearing. Also left bare on purpose: the sqlite recreate-and-copy rebuild's +`DROP TABLE` and d1-cascade's, each of which drops a table the same recipe just +`INSERT…SELECT`ed from, where `IF EXISTS` converts a caught corruption into a silent one. +`drop-column` is excluded as the one genuine dialect limit — sqlite has no +`DROP COLUMN IF EXISTS` — and the new refusal covers it instead. D1 inherits the sqlite +change, since `emit/d1.ts` renders through `renderSqlite`. + +**A chain creating a table or view in a non-default schema now emits +`CREATE SCHEMA IF NOT EXISTS`** ahead of it. `CREATE SCHEMA` was emitted nowhere in either +emitter — only by the ledger's own setup — so an `@schema` project's chain could never apply +to a virgin database. Views count, not only tables: a first migration creating just a view in +a non-default schema failed identically. The down does not drop the schema; it may hold +objects this tool does not own and cannot restore. + +### Added — `meta verify --replay` and `--replay-snapshot` + +Two new verify subverbs that answer the question the toolchain was already promising an +answer to. `docs/features/migrations-and-drift.md` and `meta migrate --help` both said +`apply-pending` "is the way to provision a fresh or CI database"; that is true only of a +chain that builds the schema, and nothing checked. + +- **`--replay`** replays the committed chain into an empty throwaway database and asserts it + **applies**. This is the #313 gate. +- **`--replay-snapshot`** additionally asserts the replayed schema **equals the committed + snapshot**, finally wiring `verifyReplay` — built, exported, and without a CLI caller since + the 2026-05-31 design retained it as "the optional `verify --replay` integrity aid". It + catches a different defect: hand-edited structural DDL that still applies but no longer + builds the recorded schema. + +They are two tiers rather than one gate because the populations differ. A project adopted via +`migrate baseline --from-db` passes the first trivially and **cannot** pass the second by +construction — its snapshot is the whole introspected database against an empty chain. The +reporter's failure was an *apply* error, so the weaker assertion is the one that answers the +bug and is immune to that class. The limitation is documented rather than auto-detected: the +only candidate signal has no production caller and would live in the *target* database's +ledger, while the gate runs against a fresh engine with no ledger at all. + +Neither needs a `--db`. The engine is local and disposable — real Postgres in-process via +**PGlite**, a throwaway temp file for sqlite — so there is nothing to provision, no +credentials, and no scratch database to collide with or drop by mistake. **`@electric-sql/pglite` +is a new OPTIONAL peer dependency of `@metaobjectsdev/migrate-ts`** (~22 MB of WASM, so it is +not forced on every adopter): install it to replay a postgres chain. With no URL to infer from, +the dialect precedence is `--dialect` > `migrate.dialect` > refuse naming `--dialect`. +`--migration-format flyway` and `--dialect d1` are refused, mirroring `apply-pending`. An empty +chain and a missing snapshot both pass and **say which**, because a gate that is silent when it +checked nothing cannot be told apart from one that passed. + +`verifyReplay` also gains an optional `governed` so a project declaring `migrate.scope` can use +the second tier at all: such a project carries the other owner's tables into its snapshot on +purpose and its chain never creates them, so without this they were reported as missing on +every replay. + ### Added — pre-release publishing to a private registry (no more real releases just to test a change) Trying an unreleased change against a downstream project required cutting a real release on diff --git a/docs/features/migrations-and-drift.md b/docs/features/migrations-and-drift.md index 446de0380..3149d4901 100644 --- a/docs/features/migrations-and-drift.md +++ b/docs/features/migrations-and-drift.md @@ -57,12 +57,100 @@ configured in `metaobjects.config.ts` (typically `./migrations/__";` ahead of it. Without that, an `@schema` +project's chain could never apply to a virgin database, because nothing ever created +the schema. The down does **not** drop the schema: it may hold objects this tool does +not own and cannot restore. + #### D1: rebuilding a foreign-key-referenced table Cloudflare D1 applies each migration inside its own implicit transaction, and SQLite diff --git a/server/typescript/packages/cli/src/commands/migrate.ts b/server/typescript/packages/cli/src/commands/migrate.ts index b3adeef2f..5becfd328 100644 --- a/server/typescript/packages/cli/src/commands/migrate.ts +++ b/server/typescript/packages/cli/src/commands/migrate.ts @@ -68,8 +68,11 @@ SUBCOMMANDS: (use with --from-db). NOTE: for a brand-new/empty database use the greenfield example below, NOT baseline — an offline baseline records your metadata as already-applied and emits no CREATE TABLE. - apply-pending Replay committed migration files against --db (no diff); - provisions a fresh/CI database. postgres/sqlite only. + apply-pending Replay committed migration files against --db (no diff). + Provisions a fresh/CI database when the chain BUILDS the + schema — 'meta verify --replay' proves that it does. A + database adopted via 'baseline --from-db' has no such + chain. postgres/sqlite only. MIGRATE FLAGS: --db DB connection URL (required for live-introspect / --apply / --rollback) @@ -89,7 +92,13 @@ MIGRATE FLAGS: --allow Comma-separated destructive-change permissions: drop-column,drop-table,type-change,drop-index,drop-fk, drop-check,drop-view,drop-view-cascade, - adopt-view,nullable-to-not-null,drop-identity-default + adopt-view,nullable-to-not-null,drop-identity-default, + drop-unmanaged + drop-unmanaged permits dropping a table/view the committed + snapshot never contained — one this toolchain never managed. + Without it that drop is refused, because the migration it + writes cannot replay against a database where the object + never existed. --on-ambiguous abort|rename|drop-add How to handle ambiguous renames (default: abort) --from-db Introspect live DB instead of using the committed snapshot diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index 00b1f7b84..4cf2c2803 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -135,7 +135,8 @@ export async function verifyCommand( if (!flags.anyExplicit) { log.info( "meta verify — running --templates (default). Explicit subverbs: " + - "--templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift).", + "--templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), " + + "--replay/--replay-snapshot (the committed migration chain replays from empty).", ); } diff --git a/server/typescript/packages/cli/src/index.ts b/server/typescript/packages/cli/src/index.ts index 24cda0466..7150eb743 100644 --- a/server/typescript/packages/cli/src/index.ts +++ b/server/typescript/packages/cli/src/index.ts @@ -110,7 +110,7 @@ or META_NO_ANTIPATTERNS=1. NOTE: outDir, dialect, dbImport, extStyle are read from metaobjects.config.ts `, - verify: `meta verify — drift gate (templates / DB schema / codegen) + verify: `meta verify — drift gate (templates / DB schema / codegen / migration replay) USAGE: meta verify [flags] @@ -122,6 +122,17 @@ FLAGS: --db Schema drift — live DB URL enables the schema-drift gate. Supports: file:, libsql:, postgres:, postgresql: D1 has no URL — use --dialect d1 / --d1 instead. + --replay Migration-chain drift — replay the committed chain into an + EMPTY throwaway database and assert it applies. Needs no --db: + the engine is in-process (PGlite for postgres, a temp file for + sqlite) and provisions nothing. Dialect comes from --dialect, + else migrate.dialect. flyway and d1 are refused. + Postgres needs the optional peer '@electric-sql/pglite'. + --replay-snapshot ...and assert the replayed schema EQUALS the committed + snapshot — catches a hand-edited up.sql that still applies but + no longer builds the recorded schema. Does NOT apply to a + project adopted with 'migrate baseline --from-db' (its chain + does not build the schema); use --replay there. --prompts Directory of provider-resolved template text (default: prompts) --dialect sqlite|postgres|d1 Optional override (auto-detected from --db URL scheme) --allow Accepted for parity with 'migrate'; does NOT affect the drift gate From a9356012e997dc95e9ff0e8ecfd95d18996d9625 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 08:08:22 -0400 Subject: [PATCH 15/44] test(integration): the PG view-lifecycle CASCADE assertion follows the IF EXISTS change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `view-lifecycle-pg` lives in the separate `integration-tests` package, which neither `bun test packages/migrate-ts` nor `packages/cli` runs, so the #313 forward-drop change left it red without any of the suites run alongside that change noticing. Only the ts-slow lane covers it. One assertion, on the CASCADE branch of `renderDropView` — a FORWARD drop, so `DROP VIEW IF EXISTS … CASCADE;` is the correct new text. The banner and dependent-destroyed assertions around it are unchanged, and the test still applies to a real Postgres and re-diffs to convergence. A workspace-wide sweep for the same shape found nothing else: every other bare `DROP …` assertion is a DOWN statement, the sqlite recreate-and-copy rebuild's drop, or hand-written SQL in a write-migration test — all three deliberately unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- .../packages/integration-tests/test/view-lifecycle-pg.test.ts | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts b/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts index a3f16bc43..89bd3ceeb 100644 --- a/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts +++ b/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts @@ -579,7 +579,8 @@ describe("view lifecycle — real Postgres", () => { // The banner lives in the committed migration file, where a reviewer sees it. expect(up).toContain("WARNING: CASCADE DROP"); expect(up).toContain("reporting.downstream_report (view)"); - expect(up).toContain(`DROP VIEW "v_program_summary" CASCADE;`); + // #313 — forward drops carry IF EXISTS so a committed chain replays from empty. + expect(up).toContain(`DROP VIEW IF EXISTS "v_program_summary" CASCADE;`); // Ours came back; theirs did not — which is exactly what the operator opted into. expect(await relationExists("v_program_summary")).toBe(true); From dc475efdcf33e38a980adbe29ecc1c67f6785381 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 08:10:39 -0400 Subject: [PATCH 16/44] test(cli): cover the replay gate's no-dialect refusal, and type its apply result MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `migrate.dialect` resolves to undefined when neither a flag nor the config supplies one, so the replay gate's "no dialect" refusal is a reachable branch rather than dead code — and nothing exercised it. With no `--db` there is no URL to infer from, and guessing would replay a postgres chain through sqlite. The new case runs under `--skip-schema` and carries a control that names the dialect on the command line, so the exit 2 is demonstrably the refusal and not the fixture. Also replaces a bare `let applied;` with an explicit `ApplyPendingResult` — an untyped `let` is an implicit evolving `any`. Co-Authored-By: Claude Opus 5 (1M context) --- .../packages/cli/src/commands/verify.ts | 3 ++- .../packages/cli/test/verify-replay.test.ts | 16 ++++++++++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index 4cf2c2803..f3c702ddd 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -40,6 +40,7 @@ import { buildExpectedSchemaWithProvenance, type GovernedScope, applyPending, + type ApplyPendingResult, openReplayEngine, type ReplayEngine, verifyReplay, @@ -313,7 +314,7 @@ export async function verifyCommand( } try { - let applied; + let applied: ApplyPendingResult; try { applied = await applyPending(engine.db, dir, { dryRun: false, dialect }); } catch (err) { diff --git a/server/typescript/packages/cli/test/verify-replay.test.ts b/server/typescript/packages/cli/test/verify-replay.test.ts index 07f869302..0686a6188 100644 --- a/server/typescript/packages/cli/test/verify-replay.test.ts +++ b/server/typescript/packages/cli/test/verify-replay.test.ts @@ -154,6 +154,22 @@ describe("verify --replay runs the gate", () => { expect(await verifyCommand(["--replay"], root)).toBe(0); }); + // `migrate.dialect` genuinely defaults to undefined, so this refusal is reachable + // rather than dead code — with no --db there is no URL to infer from, and guessing + // would replay a postgres chain through sqlite. + test("no dialect anywhere is refused, operationally", async () => { + const root = await project(); + await withApplyingChain(root); + await writeFile( + join(root, ".metaobjects", "config.json"), + JSON.stringify({ schema_version: 1, migrate: { outDir: MIGRATIONS } }), + "utf8", + ); + expect(await verifyCommand(["--replay", "--skip-schema"], root)).toBe(2); + // The control: naming the dialect on the command line makes the same run pass. + expect(await verifyCommand(["--replay", "--dialect", "sqlite", "--skip-schema"], root)).toBe(0); + }); + // `--skip-schema` is load-bearing here, not incidental: without it the D1 SCHEMA // gate also returns 2 (no wrangler.toml), so the assertion would pass whether or // not the replay gate refuses at all. From 9cfcd0c968202aa0bf0b6ffbd1a790321673f50a Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 09:38:43 -0400 Subject: [PATCH 17/44] docs(spec+plan): cross-port metadata sources, with scope held back MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A two-arm challenge of the cross-language question settled on native-surface- first with the port-neutral config as the fallback — not the phase-1 design's 'all five CLIs parse config.json'. Twelve factual splits between the arms were resolved against the code; three changed the design: - Every port's loader ALREADY accepts a set of sources (Java :1541/:994, C# :334, Python :102), so this is CLI plumbing, not engine work — which retires the resolved-manifest shape entirely. - Java's shipped grammar collides with scope on the same characters: its '*' crosses '::' (with a TODO in GeneratorUtil conceding the separator handling is broken) and its '@' matches one segment — respectively scope's '**' and '*', inverted. Both are output filters, so they compete rather than layer. scope is therefore split out into its own decision and does NOT ship. - config.json carries TypeScript-owned keys (pending_in_git, confidence_thresholds, extract, migrate) behind a .strict() schema, so the ports read a declared NEUTRAL SUBSET and ignore the rest. Also records what is deliberately NOT a contract: resolved file ORDER, which already differs per port (Java sorts by basename, C# by full path, TS walks depth-first) and which the loader discards anyway. The writer gap is resolved Node-side as 'meta init --config-only' rather than four port writers, since migrate/verify --db are Node-only under ADR-0015. --- .../2026-08-19-cross-port-metadata-sources.md | 1881 +++++++++++++++++ ...8-19-cross-port-metadata-sources-design.md | 213 ++ 2 files changed, 2094 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-19-cross-port-metadata-sources.md create mode 100644 docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md diff --git a/docs/superpowers/plans/2026-08-19-cross-port-metadata-sources.md b/docs/superpowers/plans/2026-08-19-cross-port-metadata-sources.md new file mode 100644 index 000000000..aaafb5c19 --- /dev/null +++ b/docs/superpowers/plans/2026-08-19-cross-port-metadata-sources.md @@ -0,0 +1,1881 @@ +# Cross-port metadata `sources` Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the Java/Kotlin, C# and Python CLIs read the `sources` key from the port-neutral `.metaobjects/config.json`, so all four CLI surfaces resolve the same metadata files the Node `meta` CLI already does. + +**Architecture:** Each port gains two small units — a **neutral-subset config reader** (parse `schema_version` + `sources`, ignore unknown top-level keys) and a **source resolver** (spec set → metadata file set, relative to the declaring config's directory). Both are then wired behind the port's existing metadata-location argument as a *fallback*, never an override. A new filesystem-based conformance corpus gates the resolved file **set** across all four ports plus TypeScript. + +**Tech Stack:** Java 17 + Maven (`metaobjects-maven-plugin`, JUnit), C# (.NET, xUnit), Python 3 (`uv` + pytest), TypeScript (Bun test). No new third-party dependencies in any port. + +**Spec:** [`docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md`](../specs/2026-08-19-cross-port-metadata-sources-design.md) + +## Global Constraints + +- **`scope` is OUT of scope.** Only `sources` ships. No port may read, parse, or act on the `scope` or `migrate.scope` keys. Reason: Java's shipped `` grammar collides on `*` and `@` (spec §2). +- **The neutral subset is exactly `schema_version` and `sources`.** Validate these strictly; **ignore unknown top-level keys** — the file carries TypeScript-owned keys (`pending_in_git`, `confidence_thresholds`, `extract`, `migrate`) that no other port models (spec §4). +- **`schema_version` must equal `1`.** Any other value is an error. +- **Relative `path` resolves against the directory holding the `.metaobjects/` folder** — never ambient cwd. Absolute paths as-is (spec §3). +- **Default when `sources` is absent or empty:** exactly one `path` source, the literal `metaobjects` (spec §3, `metadata-files.ts:33`). +- **Metadata file extensions:** `.json`, `.yaml`, `.yml`, matched case-insensitively. All four ports already agree — do not change any port's existing set. +- **File ORDER is NOT a contract.** Each port keeps its existing `DirectorySource` walk order. Corpus comparisons are order-insensitive (set equality) (spec §3). +- **Error codes, already registered in all five ledgers — do not add any:** `ERR_SOURCE_UNRESOLVED` (a declared path does not exist), `ERR_SOURCE_KIND_UNSUPPORTED` (`resource`/`package`), `ERR_COLLECTION_NOT_FOUND` (nothing declared and no default dir). +- **Precedence ladder, first match wins** (spec §5): explicit CLI argument → port's native config surface → `.metaobjects/config.json` `sources` → built-in default. **A file that exists but is malformed is an ERROR at its own rung — never a fall-through.** +- **Public repo.** No absolute home paths, no other-project names, in code, tests, fixtures, docs or commit messages. +- **All four readers ship in ONE changeset.** Non-TS lanes do not run on PRs (`AGENTS.md:91-92`), so a per-port deferral goes green locally and turns ports red on the next push to `main`. + +--- + +## File Structure + +**New shared corpus** +- `fixtures/source-resolution-conformance/cases.json` — the single source of truth for every port's expectations +- `fixtures/source-resolution-conformance/README.md` — grammar, semantics, runner contract + +**TypeScript** (reference implementation already exists; adds a runner only) +- Create: `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts` + +**Python** +- Create: `server/python/src/metaobjects/config/neutral_config.py` — neutral-subset reader +- Create: `server/python/src/metaobjects/config/source_resolver.py` — spec set → file list +- Create: `server/python/src/metaobjects/config/__init__.py` +- Modify: `server/python/src/metaobjects/cli.py` — fallback wiring +- Create: `server/python/tests/config/test_neutral_config.py` +- Create: `server/python/tests/config/test_source_resolver.py` +- Create: `server/python/tests/conformance/test_source_resolution_conformance.py` + +**C#** +- Create: `server/csharp/MetaObjects/Config/NeutralConfig.cs` +- Create: `server/csharp/MetaObjects/Config/SourceResolver.cs` +- Modify: `server/csharp/MetaObjects.Cli/Program.cs` — fallback wiring +- Create: `server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs` +- Create: `server/csharp/MetaObjects.Tests/Config/NeutralConfigTests.cs` + +**Java** (Kotlin inherits — it has no CLI entry point of its own) +- Create: `server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java` +- Create: `server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java` +- Modify: `server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java` — fallback wiring +- Create: `server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java` +- Create: `server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java` + +**Node writer** +- Modify: `server/typescript/packages/cli/src/commands/init.ts` — config-only mode +- Modify: `server/typescript/packages/cli/test/init.test.ts` + +**Docs** +- Modify: `docs/features/metadata-sources.md:36-42` — the "Port support" paragraph +- Modify: `CHANGELOG.md` — under `## [Unreleased]` + +--- + +## Task 1: The shared corpus and the TypeScript reference runner + +The corpus is filesystem-based, unlike `scope-conformance` which is pure string matching. Each case declares a tree to materialize, a config to write, and either an expected file set or an expected error code. + +**Files:** +- Create: `fixtures/source-resolution-conformance/cases.json` +- Create: `fixtures/source-resolution-conformance/README.md` +- Create: `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts` + +**Interfaces:** +- Produces: the case-file schema every later task's runner consumes — + `{ cases: [{ name, tree: {relPath: "content"}, config: object | null, expectFiles?: string[], expectError?: string }] }`. + `tree` keys are paths relative to the materialized project root. `config` is written to `.metaobjects/config.json` when non-null; when null, no config file is created. `expectFiles` are project-root-relative paths, compared as an **unordered set**. `expectError` is an error code string. + +- [ ] **Step 1: Write the corpus cases file** + +Create `fixtures/source-resolution-conformance/cases.json`: + +```json +{ + "cases": [ + { + "name": "no-config-uses-default-directory", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": null, + "expectFiles": ["metaobjects/meta.users.json"] + }, + { + "name": "empty-sources-uses-default-directory", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [] }, + "expectFiles": ["metaobjects/meta.users.json"] + }, + { + "name": "declared-path-replaces-the-default-entirely", + "tree": { + "metaobjects/ignored.json": "{\"metadata.root\":{\"children\":[]}}", + "model/meta.orders.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": ["model/meta.orders.json"] + }, + { + "name": "directory-is-walked-recursively", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/nested/meta.b.yaml": "metadata.root:\n children: []\n", + "model/nested/deep/meta.c.yml": "metadata.root:\n children: []\n" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": [ + "model/meta.a.json", + "model/nested/meta.b.yaml", + "model/nested/deep/meta.c.yml" + ] + }, + { + "name": "non-metadata-extensions-are-ignored", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/README.md": "not metadata", + "model/notes.txt": "not metadata", + "model/script.ts": "export {}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": ["model/meta.a.json"] + }, + { + "name": "a-single-file-path-resolves-to-that-file", + "tree": { + "vendor/meta.catalog.json": "{\"metadata.root\":{\"children\":[]}}", + "vendor/meta.other.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "vendor/meta.catalog.json" }] }, + "expectFiles": ["vendor/meta.catalog.json"] + }, + { + "name": "two-sources-union", + "tree": { + "a/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "b/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "a" }, { "path": "b" }] }, + "expectFiles": ["a/meta.a.json", "b/meta.b.json"] + }, + { + "name": "overlapping-sources-yield-each-file-once", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/nested/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "model" }, { "path": "model/nested" }] + }, + "expectFiles": ["model/meta.a.json", "model/nested/meta.b.json"] + }, + { + "name": "source-order-does-not-change-the-resolved-set", + "tree": { + "a/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "b/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "b" }, { "path": "a" }] }, + "expectFiles": ["a/meta.a.json", "b/meta.b.json"] + }, + { + "name": "a-parent-relative-path-resolves-against-the-config-directory", + "tree": { + "shared/meta.shared.json": "{\"metadata.root\":{\"children\":[]}}", + "app/.keep": "" + }, + "config": { "schema_version": 1, "sources": [{ "path": "shared" }] }, + "expectFiles": ["shared/meta.shared.json"] + }, + { + "name": "a-declared-path-that-does-not-exist-is-an-error", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "nope" }] }, + "expectError": "ERR_SOURCE_UNRESOLVED" + }, + { + "name": "an-empty-directory-source-resolves-to-no-files", + "tree": { + "model/.keep": "" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": [] + }, + { + "name": "resource-kind-is-unsupported", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "resource": "com/acme/model" }] }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "package-kind-is-unsupported", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "package": "acme-model" }] }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "no-config-and-no-default-directory-is-an-error", + "tree": { + "src/.keep": "" + }, + "config": null, + "expectError": "ERR_COLLECTION_NOT_FOUND" + }, + { + "name": "unknown-top-level-keys-are-ignored", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "model" }], + "pending_in_git": true, + "confidence_thresholds": { "pending_promote": 0.8 }, + "extract": { "metaignore": ".metaignore" }, + "migrate": { "dialect": "postgres" } + }, + "expectFiles": ["model/meta.a.json"] + } + ] +} +``` + +- [ ] **Step 2: Write the corpus README** + +Create `fixtures/source-resolution-conformance/README.md`: + +```markdown +# source-resolution-conformance + +Pins how a consumer's `sources` set in `.metaobjects/config.json` resolves to a +set of metadata files. Every port's CLI must resolve the SAME FILES from the +same declaration — that is the cross-port promise this corpus exists to keep. + +Companion to `scope-conformance/`, which pins the (currently TypeScript-only) +`scope` pattern grammar. The two are independent: `sources` decides which files +are read, `scope` filters what is emitted from them. + +## Shape + +``` +cases.json # { cases: [{ name, tree, config, expectFiles? , expectError? }] } +README.md +``` + +- **`tree`** — a map of project-root-relative path → file content. The runner + materializes it in a fresh temporary directory. A `.keep` entry exists only to + force an otherwise-empty directory to be created. +- **`config`** — written verbatim to `.metaobjects/config.json` under the project + root. When `null`, no config file is created at all. +- **`expectFiles`** — project-root-relative paths, compared as an **UNORDERED + SET**. See "Order is deliberately not pinned" below. +- **`expectError`** — an error code the resolution must fail with. Exactly one of + `expectFiles` / `expectError` is present per case. + +## Semantics pinned here + +- **Default.** `sources` absent or empty ⇒ exactly one `path` source, the literal + `metaobjects`. It is a default VALUE, never a requirement. +- **Replacement, not merge.** A declared `sources` replaces the default entirely — + the default directory is not implicitly appended. +- **Relative base.** A relative `path` resolves against the directory HOLDING the + `.metaobjects/` folder, never against the process working directory. +- **Recursion.** A directory `path` is walked recursively; a file `path` resolves + to that one file. +- **Extensions.** `.json`, `.yaml`, `.yml`, matched case-insensitively. Nothing else. +- **Union with de-duplication.** Overlapping sources yield each file exactly once. +- **A declared path that does not exist is `ERR_SOURCE_UNRESOLVED`** — never a + silent skip. Only the DEFAULT may be absent, and then it is + `ERR_COLLECTION_NOT_FOUND`. +- **`resource` and `package` kinds are declared but resolve nowhere yet:** + `ERR_SOURCE_KIND_UNSUPPORTED`. +- **Unknown top-level config keys are IGNORED.** The file carries + TypeScript-owned keys no other port models. `schema_version` and `sources` are + the neutral subset; each port validates those strictly and ignores the rest. + +## Order is deliberately NOT pinned + +`expectFiles` is a set. The ports' directory walks already differ and always +have — Java sorts by basename (`DirectorySource.java:105`), C# by full-path +ordinal (`DirectorySource.cs:64`), Python by basename +(`directory_source.py:40-48`), TypeScript walks depth-first with files before +subdirectories (`metadata-files.ts:101-121`). Making order a contract would be a +behavior change in three ports for no benefit: super-resolution is +order-independent (#188) and the loader's overlay partition discards caller +order anyway. + +A port MAY have a stable internal order — several do, and their own generated +output depends on it. It just is not a cross-port promise. + +## Behavioral contract + +Each port's runner reads `cases.json`, and for every case: materializes `tree` +in a fresh temp directory, writes `config` when non-null, resolves sources +against that root, then asserts either that the resolved file set equals +`expectFiles` (as a set, project-root-relative, path separators normalized to +`/`) or that resolution failed with `expectError`. + +## Reference implementation + +`server/typescript/packages/sdk/src/sources.ts` (`resolveSources`) and +`server/typescript/packages/sdk/src/collection.ts` (`resolveCollection`). +``` + +- [ ] **Step 3: Write the failing TypeScript runner** + +Create `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts`: + +```ts +// Runs the shared source-resolution corpus against the TypeScript reference +// implementation. Every port ships an equivalent runner reading this same file. +import { describe, expect, test } from "bun:test"; +import { mkdtemp, mkdir, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join, relative, resolve, sep } from "node:path"; +import { resolveCollection } from "../src/collection.js"; + +interface Case { + readonly name: string; + readonly tree: Record; + readonly config: unknown | null; + readonly expectFiles?: readonly string[]; + readonly expectError?: string; +} + +const CORPUS = resolve(import.meta.dir, "../../../../../fixtures/source-resolution-conformance/cases.json"); + +async function materialize(c: Case): Promise { + const root = await mkdtemp(join(tmpdir(), "mo-src-conf-")); + for (const [rel, content] of Object.entries(c.tree)) { + const abs = join(root, rel); + await mkdir(dirname(abs), { recursive: true }); + await writeFile(abs, content); + } + if (c.config !== null) { + await mkdir(join(root, ".metaobjects"), { recursive: true }); + await writeFile(join(root, ".metaobjects", "config.json"), JSON.stringify(c.config, null, 2)); + } + return root; +} + +const cases: Case[] = JSON.parse(await readFile(CORPUS, "utf8")).cases; + +describe("source-resolution conformance", () => { + for (const c of cases) { + test(c.name, async () => { + const root = await materialize(c); + if (c.expectError !== undefined) { + let code: string | undefined; + try { + await resolveCollection(root, { explicitDir: root }); + } catch (e) { + code = (e as { code?: string }).code; + } + expect(code).toBe(c.expectError); + return; + } + const collection = await resolveCollection(root, { explicitDir: root }); + const got = collection.files.map((f) => relative(root, f).split(sep).join("/")).sort(); + expect(got).toEqual([...(c.expectFiles ?? [])].sort()); + }); + } +}); +``` + +- [ ] **Step 4: Run it and confirm every case passes** + +Run: `cd server/typescript/packages/sdk && bun test test/source-resolution-conformance.test.ts` +Expected: **16 pass, 0 fail.** The TypeScript side is the already-shipped reference — this runner asserts the corpus describes real behavior, so a failure here means the CORPUS is wrong, not the code. Fix `cases.json` until it matches, and do not change `sources.ts` or `collection.ts` in this task. + +- [ ] **Step 5: Typecheck** + +Run: `cd server/typescript && bun run --filter '*' typecheck` +Expected: clean. (`bun test` does not typecheck — a break here would otherwise sit red through later tasks.) + +- [ ] **Step 6: Commit** + +```bash +git add fixtures/source-resolution-conformance server/typescript/packages/sdk/test/source-resolution-conformance.test.ts +git commit -m "test(conformance): a corpus for cross-port source resolution + +Pins the resolved file SET — not its order, which already differs by port +and is not a contract. TypeScript is the reference and ships the first +runner; the other three follow in this changeset." +``` + +--- + +## Task 2: Python — neutral config reader and source resolver + +**Files:** +- Create: `server/python/src/metaobjects/config/__init__.py` +- Create: `server/python/src/metaobjects/config/neutral_config.py` +- Create: `server/python/src/metaobjects/config/source_resolver.py` +- Test: `server/python/tests/config/test_neutral_config.py` +- Test: `server/python/tests/config/test_source_resolver.py` + +**Interfaces:** +- Consumes: `fixtures/source-resolution-conformance/cases.json` (Task 1); the existing `metaobjects.errors.ErrorCode` members `ERR_SOURCE_UNRESOLVED`, `ERR_SOURCE_KIND_UNSUPPORTED`, `ERR_COLLECTION_NOT_FOUND` (`errors.py:108,110,114`). +- Produces: + - `read_neutral_config(config_dir: Path) -> NeutralConfig | None` — `None` when no `.metaobjects/config.json` exists; raises on malformed. + - `NeutralConfig` dataclass with field `sources: list[dict[str, str]]`. + - `DEFAULT_METADATA_DIR: str = "metaobjects"`. + - `resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path]`. + - `resolve_collection(root: Path) -> list[Path]` — the full ladder: config-or-default, then resolve. + +- [ ] **Step 1: Write the failing config-reader test** + +Create `server/python/tests/config/test_neutral_config.py`: + +```python +import json +import pytest +from pathlib import Path + +from metaobjects.config.neutral_config import read_neutral_config +from metaobjects.errors import MetaObjectsError + + +def _write_config(root: Path, payload: object) -> None: + d = root / ".metaobjects" + d.mkdir(parents=True, exist_ok=True) + (d / "config.json").write_text(json.dumps(payload)) + + +def test_absent_config_returns_none(tmp_path: Path) -> None: + assert read_neutral_config(tmp_path) is None + + +def test_reads_sources(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 1, "sources": [{"path": "model"}]}) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [{"path": "model"}] + + +def test_unknown_top_level_keys_are_ignored(tmp_path: Path) -> None: + # The file carries TypeScript-owned keys this port must not model. + _write_config( + tmp_path, + { + "schema_version": 1, + "sources": [{"path": "model"}], + "pending_in_git": True, + "confidence_thresholds": {"pending_promote": 0.8}, + "extract": {"metaignore": ".metaignore"}, + "migrate": {"dialect": "postgres"}, + }, + ) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [{"path": "model"}] + + +def test_absent_sources_key_yields_empty_list(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 1}) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [] + + +def test_malformed_json_raises_not_none(tmp_path: Path) -> None: + d = tmp_path / ".metaobjects" + d.mkdir(parents=True) + (d / "config.json").write_text("{ not json") + # A file that EXISTS but cannot be read must never look like no config at all. + with pytest.raises(MetaObjectsError): + read_neutral_config(tmp_path) + + +def test_wrong_schema_version_raises(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 2, "sources": []}) + with pytest.raises(MetaObjectsError): + read_neutral_config(tmp_path) +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/python && uv run pytest tests/config/test_neutral_config.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'metaobjects.config'` + +- [ ] **Step 3: Implement the reader** + +Create `server/python/src/metaobjects/config/__init__.py`: + +```python +"""Port-neutral `.metaobjects/config.json` reading and source resolution. + +Reads only the NEUTRAL SUBSET (`schema_version`, `sources`). The file also +carries TypeScript-owned keys; those are ignored rather than modeled, so a new +TS-only key never becomes a four-port change. See +`docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §4. +""" +from .neutral_config import DEFAULT_METADATA_DIR, NeutralConfig, read_neutral_config +from .source_resolver import resolve_collection, resolve_sources + +__all__ = [ + "DEFAULT_METADATA_DIR", + "NeutralConfig", + "read_neutral_config", + "resolve_collection", + "resolve_sources", +] +``` + +Create `server/python/src/metaobjects/config/neutral_config.py`: + +```python +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path + +from metaobjects.errors import ErrorCode, MetaObjectsError + +#: The DEFAULT value of `sources` when the key is absent or empty — never a +#: requirement, and never assumed to exist by any other code path. +DEFAULT_METADATA_DIR = "metaobjects" + +_METAOBJECTS_DIR = ".metaobjects" +_CONFIG_FILE = "config.json" + + +@dataclass(frozen=True) +class NeutralConfig: + """The port-neutral subset of `.metaobjects/config.json`.""" + + #: Raw source specs, each a single-key mapping (`path` / `resource` / `package`). + sources: list[dict[str, str]] + + +def read_neutral_config(config_dir: Path) -> NeutralConfig | None: + """Read the neutral subset from ``config_dir/.metaobjects/config.json``. + + Returns ``None`` when the file does not exist. A file that EXISTS but is + malformed raises — swallowing it would make a typo'd config behave + identically to no config at all, silently resolving a possibly-stale + default directory with no diagnostic. + """ + path = config_dir / _METAOBJECTS_DIR / _CONFIG_FILE + if not path.is_file(): + return None + + try: + raw = json.loads(path.read_text()) + except (OSError, ValueError) as e: + raise MetaObjectsError( + f"{path} exists but could not be read as JSON: {e}", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) from e + + if not isinstance(raw, dict): + raise MetaObjectsError( + f"{path} must contain a JSON object", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + version = raw.get("schema_version") + if version != 1: + raise MetaObjectsError( + f"{path}: unsupported schema_version {version!r} (expected 1)", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + sources = raw.get("sources", []) + if not isinstance(sources, list) or not all( + isinstance(s, dict) and len(s) == 1 for s in sources + ): + raise MetaObjectsError( + f"{path}: 'sources' must be an array of single-key objects", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + # Unknown top-level keys are IGNORED by design — see the module docstring. + return NeutralConfig(sources=[dict(s) for s in sources]) +``` + +- [ ] **Step 4: Run the reader tests** + +Run: `cd server/python && uv run pytest tests/config/test_neutral_config.py -v` +Expected: 6 passed. If `MetaObjectsError`'s constructor signature differs, adapt the raise sites to the real one — check with `grep -n "class MetaObjectsError" -A12 server/python/src/metaobjects/errors.py` and use whatever the existing code passes. + +- [ ] **Step 5: Write the failing resolver test** + +Create `server/python/tests/config/test_source_resolver.py`: + +```python +import json +import pytest +from pathlib import Path + +from metaobjects.config.source_resolver import resolve_collection, resolve_sources +from metaobjects.errors import ErrorCode, MetaObjectsError + + +def _rel(root: Path, files: list[Path]) -> set[str]: + return {p.relative_to(root).as_posix() for p in files} + + +def test_directory_is_walked_recursively(tmp_path: Path) -> None: + (tmp_path / "model" / "nested").mkdir(parents=True) + (tmp_path / "model" / "a.json").write_text("{}") + (tmp_path / "model" / "nested" / "b.yaml").write_text("{}") + (tmp_path / "model" / "README.md").write_text("x") + got = resolve_sources(tmp_path, [{"path": "model"}]) + assert _rel(tmp_path, got) == {"model/a.json", "model/nested/b.yaml"} + + +def test_single_file_spec(tmp_path: Path) -> None: + (tmp_path / "vendor").mkdir() + (tmp_path / "vendor" / "one.json").write_text("{}") + (tmp_path / "vendor" / "two.json").write_text("{}") + got = resolve_sources(tmp_path, [{"path": "vendor/one.json"}]) + assert _rel(tmp_path, got) == {"vendor/one.json"} + + +def test_overlapping_sources_dedupe(tmp_path: Path) -> None: + (tmp_path / "model" / "nested").mkdir(parents=True) + (tmp_path / "model" / "a.json").write_text("{}") + (tmp_path / "model" / "nested" / "b.json").write_text("{}") + got = resolve_sources(tmp_path, [{"path": "model"}, {"path": "model/nested"}]) + assert _rel(tmp_path, got) == {"model/a.json", "model/nested/b.json"} + assert len(got) == 2 + + +def test_missing_path_raises_unresolved(tmp_path: Path) -> None: + with pytest.raises(MetaObjectsError) as e: + resolve_sources(tmp_path, [{"path": "nope"}]) + assert e.value.code == ErrorCode.ERR_SOURCE_UNRESOLVED + + +def test_resource_kind_unsupported(tmp_path: Path) -> None: + with pytest.raises(MetaObjectsError) as e: + resolve_sources(tmp_path, [{"resource": "com/acme"}]) + assert e.value.code == ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED + + +def test_collection_falls_back_to_default_dir(tmp_path: Path) -> None: + (tmp_path / "metaobjects").mkdir() + (tmp_path / "metaobjects" / "a.json").write_text("{}") + got = resolve_collection(tmp_path) + assert _rel(tmp_path, got) == {"metaobjects/a.json"} + + +def test_collection_with_no_config_and_no_default_raises(tmp_path: Path) -> None: + with pytest.raises(MetaObjectsError) as e: + resolve_collection(tmp_path) + assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND + + +def test_declared_sources_replace_the_default(tmp_path: Path) -> None: + (tmp_path / ".metaobjects").mkdir() + (tmp_path / ".metaobjects" / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + (tmp_path / "metaobjects").mkdir() + (tmp_path / "metaobjects" / "ignored.json").write_text("{}") + (tmp_path / "model").mkdir() + (tmp_path / "model" / "used.json").write_text("{}") + got = resolve_collection(tmp_path) + assert _rel(tmp_path, got) == {"model/used.json"} +``` + +- [ ] **Step 6: Run it to verify it fails** + +Run: `cd server/python && uv run pytest tests/config/test_source_resolver.py -v` +Expected: FAIL — `ImportError: cannot import name 'resolve_sources'` + +- [ ] **Step 7: Implement the resolver** + +Create `server/python/src/metaobjects/config/source_resolver.py`: + +```python +from __future__ import annotations + +from pathlib import Path + +from metaobjects.errors import ErrorCode, MetaObjectsError + +from .neutral_config import DEFAULT_METADATA_DIR, read_neutral_config + +_SUPPORTED_SUFFIXES = (".json", ".yaml", ".yml") + + +def _list_metadata_files(directory: Path) -> list[Path]: + """Recursively list metadata files under ``directory``. + + Mirrors `DirectorySource`'s extension set (`.json`/`.yaml`/`.yml`, + case-insensitive). Order is this port's own and is deliberately NOT a + cross-port contract — see the corpus README. + """ + return sorted( + (p for p in directory.rglob("*") if p.is_file() and p.suffix.lower() in _SUPPORTED_SUFFIXES), + key=lambda p: p.name, + ) + + +def resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path]: + """Resolve a declared source SET to a de-duplicated list of metadata files. + + A relative ``path`` resolves against ``config_dir`` — the directory HOLDING + the ``.metaobjects/`` folder — never against the process working directory. + """ + seen: dict[Path, None] = {} + + for spec in specs: + if "path" not in spec: + kind = next(iter(spec), "") + raise MetaObjectsError( + f'source kind "{kind}" is not supported by this toolchain yet; use a "path" source', + code=ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED, + ) + + raw = Path(spec["path"]) + target = raw if raw.is_absolute() else (config_dir / raw) + + if not target.exists(): + raise MetaObjectsError( + f'source path "{spec["path"]}" does not exist ' + f"(resolved to {target}, relative to {config_dir})", + code=ErrorCode.ERR_SOURCE_UNRESOLVED, + ) + + found = _list_metadata_files(target) if target.is_dir() else [target] + for f in found: + seen.setdefault(f.resolve(), None) + + return list(seen) + + +def resolve_collection(root: Path) -> list[Path]: + """The full ladder: declared `sources`, else the default directory. + + Only the DEFAULT may be absent — a declared source that does not resolve is + `ERR_SOURCE_UNRESOLVED`, a louder failure. + """ + root = root.resolve() + cfg = read_neutral_config(root) + specs = cfg.sources if cfg is not None and cfg.sources else [] + + if not specs: + default_dir = root / DEFAULT_METADATA_DIR + if not default_dir.is_dir(): + raise MetaObjectsError( + f'no metadata sources declared in {root} and no default ' + f'"{DEFAULT_METADATA_DIR}" directory found. Declare "sources" in ' + f".metaobjects/config.json, or run 'meta init' to scaffold.", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + specs = [{"path": DEFAULT_METADATA_DIR}] + + return resolve_sources(root, specs) +``` + +- [ ] **Step 8: Run the resolver tests** + +Run: `cd server/python && uv run pytest tests/config/ -v` +Expected: 14 passed (6 reader + 8 resolver). + +- [ ] **Step 9: Commit** + +```bash +git add server/python/src/metaobjects/config server/python/tests/config +git commit -m "feat(python): read the port-neutral sources key + +Neutral subset only (schema_version + sources); unknown top-level keys are +ignored so a TypeScript-owned key never becomes a four-port change." +``` + +--- + +## Task 3: Python — conformance runner and CLI fallback + +**Files:** +- Create: `server/python/tests/conformance/test_source_resolution_conformance.py` +- Modify: `server/python/src/metaobjects/cli.py` + +**Interfaces:** +- Consumes: `resolve_collection` / `resolve_sources` (Task 2); `cases.json` (Task 1). +- Produces: CLI behavior — a `gen`/`verify` invocation with no positional `metadata_dir` and no `metadata` key in `metaobjects.config.yaml` falls back to `.metaobjects/config.json`. + +- [ ] **Step 1: Write the failing conformance runner** + +Create `server/python/tests/conformance/test_source_resolution_conformance.py`: + +```python +"""Runs the shared source-resolution corpus against this port. + +Reads `fixtures/source-resolution-conformance/cases.json` — the single +committed source of truth. There is no per-port fixture. +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from metaobjects.config.source_resolver import resolve_collection +from metaobjects.errors import MetaObjectsError + +_CORPUS = ( + Path(__file__).resolve().parents[3] + / "fixtures" + / "source-resolution-conformance" + / "cases.json" +) + +_CASES = json.loads(_CORPUS.read_text())["cases"] + + +def _materialize(case: dict, root: Path) -> None: + for rel, content in case["tree"].items(): + p = root / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(content) + if case["config"] is not None: + d = root / ".metaobjects" + d.mkdir(parents=True, exist_ok=True) + (d / "config.json").write_text(json.dumps(case["config"], indent=2)) + + +@pytest.mark.parametrize("case", _CASES, ids=[c["name"] for c in _CASES]) +def test_source_resolution_conformance(case: dict, tmp_path: Path) -> None: + _materialize(case, tmp_path) + + if "expectError" in case: + with pytest.raises(MetaObjectsError) as e: + resolve_collection(tmp_path) + assert e.value.code.value == case["expectError"] + return + + got = {p.relative_to(tmp_path.resolve()).as_posix() for p in resolve_collection(tmp_path)} + assert got == set(case["expectFiles"]) +``` + +- [ ] **Step 2: Run it** + +Run: `cd server/python && uv run pytest tests/conformance/test_source_resolution_conformance.py -v` +Expected: 16 passed. If `e.value.code` is already a plain string rather than an enum, drop the `.value`. If the corpus path resolution is wrong, print `_CORPUS` and correct the `parents[N]` index — the file must resolve to the repository's `fixtures/` directory. + +- [ ] **Step 3: Write the failing CLI fallback test** + +Add to `server/python/tests/conformance/test_source_resolution_conformance.py`: + +```python +def test_cli_falls_back_to_neutral_config(tmp_path: Path, monkeypatch) -> None: + """No positional metadata_dir and no YAML `metadata` key ⇒ neutral config wins.""" + (tmp_path / "model").mkdir() + (tmp_path / "model" / "meta.a.json").write_text('{"metadata.root":{"children":[]}}') + d = tmp_path / ".metaobjects" + d.mkdir() + (d / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + + from metaobjects.cli import resolve_metadata_location + + monkeypatch.chdir(tmp_path) + got = resolve_metadata_location(explicit=None, config=None, root=tmp_path) + assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in got} == { + "model/meta.a.json" + } +``` + +- [ ] **Step 4: Run it to verify it fails** + +Run: `cd server/python && uv run pytest tests/conformance/test_source_resolution_conformance.py::test_cli_falls_back_to_neutral_config -v` +Expected: FAIL — `ImportError: cannot import name 'resolve_metadata_location'` + +- [ ] **Step 5: Add the ladder function to the CLI** + +Add to `server/python/src/metaobjects/cli.py` (near the other resolution helpers, around the existing `gen_state_dir_for` at line 387): + +```python +def resolve_metadata_location( + explicit: str | None, + config: "ProjectConfig | None", + root: Path, +) -> list[str]: + """The precedence ladder for where metadata lives. First match wins. + + 1. An explicit CLI argument (the positional ``metadata_dir``). + 2. This port's native surface — ``metadata`` in ``metaobjects.config.yaml``. + 3. ``sources`` in the port-neutral ``.metaobjects/config.json``. + 4. The built-in default directory. + + A file that EXISTS at any rung but is malformed raises rather than falling + through to the next rung. See + `docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §5. + """ + from metaobjects.config.source_resolver import resolve_collection, resolve_sources + + if explicit is not None: + return [str(p) for p in resolve_sources(Path(explicit).resolve().parent, [{"path": explicit}])] + + if config is not None: + return [str(p) for p in resolve_sources(Path(config.metadata_dir()).parent, [{"path": config.metadata_dir()}])] + + # Rungs 3 and 4 both live in `resolve_collection`. + return [str(p) for p in resolve_collection(root)] +``` + +- [ ] **Step 6: Run the test** + +Run: `cd server/python && uv run pytest tests/conformance/test_source_resolution_conformance.py -v` +Expected: 17 passed. + +- [ ] **Step 7: Run the whole Python suite for regressions** + +Run: `cd server/python && uv run --extra integration pytest -q` +Expected: no new failures versus the pre-task baseline. (Capture the baseline first with the same command on a clean tree if you have not already — `uv.lock` re-dirties on any `uv run`; do NOT commit it.) + +- [ ] **Step 8: Commit** + +```bash +git add server/python/src/metaobjects/cli.py server/python/tests/conformance/test_source_resolution_conformance.py +git commit -m "feat(python): the neutral config is the fallback when nothing else names a location + +Ladder: explicit arg > metaobjects.config.yaml > .metaobjects/config.json > +default dir. Gated by the shared source-resolution corpus." +``` + +--- + +## Task 4: C# — neutral config reader, resolver, conformance runner, CLI fallback + +**Files:** +- Create: `server/csharp/MetaObjects/Config/NeutralConfig.cs` +- Create: `server/csharp/MetaObjects/Config/SourceResolver.cs` +- Create: `server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs` +- Modify: `server/csharp/MetaObjects.Cli/Program.cs` + +**Interfaces:** +- Consumes: `cases.json` (Task 1); `MetaObjects.Errors` members `ERR_SOURCE_UNRESOLVED`, `ERR_SOURCE_KIND_UNSUPPORTED`, `ERR_COLLECTION_NOT_FOUND` (`Errors.cs:132,134,138`). +- Produces: + - `MetaObjects.Config.NeutralConfig.Read(string configDir) -> NeutralConfig?` + - `NeutralConfig.Sources -> IReadOnlyList>` + - `MetaObjects.Config.SourceResolver.ResolveCollection(string root) -> IReadOnlyList` + - `MetaObjects.Config.SourceResolver.ResolveSources(string configDir, IReadOnlyList> specs) -> IReadOnlyList` + - `NeutralConfig.DefaultMetadataDir -> "metaobjects"` + +- [ ] **Step 1: Write the failing conformance runner** + +Create `server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs`: + +```csharp +// Runs the shared source-resolution corpus against this port. Reads the single +// committed fixtures/source-resolution-conformance/cases.json — no per-port fixture. +using System.Text.Json; +using MetaObjects.Config; +using Xunit; + +namespace MetaObjects.Conformance.Tests; + +public class SourceResolutionConformanceTests +{ + private sealed record Case( + string Name, + Dictionary Tree, + JsonElement? Config, + string[]? ExpectFiles, + string? ExpectError); + + public static TheoryData CaseNames() + { + var data = new TheoryData(); + foreach (var c in LoadCases()) data.Add(c.Name); + return data; + } + + private static string CorpusPath() + { + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null && !Directory.Exists(Path.Combine(dir.FullName, "fixtures"))) + dir = dir.Parent; + Assert.NotNull(dir); + return Path.Combine(dir!.FullName, "fixtures", "source-resolution-conformance", "cases.json"); + } + + private static List LoadCases() + { + using var doc = JsonDocument.Parse(File.ReadAllText(CorpusPath())); + var cases = new List(); + foreach (var el in doc.RootElement.GetProperty("cases").EnumerateArray()) + { + var tree = new Dictionary(); + foreach (var p in el.GetProperty("tree").EnumerateObject()) + tree[p.Name] = p.Value.GetString() ?? ""; + + var cfgEl = el.GetProperty("config"); + JsonElement? cfg = cfgEl.ValueKind == JsonValueKind.Null ? null : cfgEl.Clone(); + + string[]? expectFiles = el.TryGetProperty("expectFiles", out var ef) + ? ef.EnumerateArray().Select(x => x.GetString()!).ToArray() + : null; + string? expectError = el.TryGetProperty("expectError", out var ee) ? ee.GetString() : null; + + cases.Add(new Case(el.GetProperty("name").GetString()!, tree, cfg, expectFiles, expectError)); + } + return cases; + } + + [Theory] + [MemberData(nameof(CaseNames))] + public void ResolvesTheSameFileSet(string name) + { + var c = LoadCases().Single(x => x.Name == name); + var root = Path.Combine(Path.GetTempPath(), "mo-src-conf-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(root); + try + { + foreach (var (rel, content) in c.Tree) + { + var abs = Path.Combine(root, rel.Replace('/', Path.DirectorySeparatorChar)); + Directory.CreateDirectory(Path.GetDirectoryName(abs)!); + File.WriteAllText(abs, content); + } + if (c.Config is not null) + { + var d = Path.Combine(root, ".metaobjects"); + Directory.CreateDirectory(d); + File.WriteAllText(Path.Combine(d, "config.json"), c.Config.Value.GetRawText()); + } + + if (c.ExpectError is not null) + { + var ex = Assert.ThrowsAny(() => SourceResolver.ResolveCollection(root)); + Assert.Equal(c.ExpectError, ex.Code.ToString()); + return; + } + + var got = SourceResolver.ResolveCollection(root) + .Select(f => Path.GetRelativePath(root, f).Replace(Path.DirectorySeparatorChar, '/')) + .ToHashSet(); + Assert.Equal(c.ExpectFiles!.ToHashSet(), got); + } + finally + { + Directory.Delete(root, recursive: true); + } + } +} +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/csharp && dotnet test MetaObjects.Conformance.Tests/MetaObjects.Conformance.Tests.csproj --nologo --verbosity quiet` +Expected: FAIL to COMPILE — `MetaObjects.Config` does not exist. If `MetaObjectsException` is named differently, check with `grep -n "class .*Exception" server/csharp/MetaObjects/Errors.cs` and use the real type and its code property. + +- [ ] **Step 3: Implement the reader** + +Create `server/csharp/MetaObjects/Config/NeutralConfig.cs`: + +```csharp +// Port-neutral `.metaobjects/config.json` reading. +// +// Reads only the NEUTRAL SUBSET (`schema_version`, `sources`). The file also +// carries TypeScript-owned keys; those are IGNORED rather than modeled, so a new +// TS-only key never becomes a four-port change. See +// docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md §4. +using System.Text.Json; + +namespace MetaObjects.Config; + +public sealed class NeutralConfig +{ + /// The DEFAULT value of `sources` when the key is absent or empty — never a + /// requirement, and never assumed to exist by any other code path. + public const string DefaultMetadataDir = "metaobjects"; + + private const string MetaObjectsDir = ".metaobjects"; + private const string ConfigFile = "config.json"; + + public IReadOnlyList> Sources { get; } + + private NeutralConfig(IReadOnlyList> sources) => Sources = sources; + + /// Returns null when the file does not exist. A file that EXISTS but is + /// malformed throws — swallowing it would make a typo'd config behave + /// identically to no config at all. + public static NeutralConfig? Read(string configDir) + { + var path = Path.Combine(configDir, MetaObjectsDir, ConfigFile); + if (!File.Exists(path)) return null; + + JsonDocument doc; + try + { + doc = JsonDocument.Parse(File.ReadAllText(path)); + } + catch (Exception e) + { + throw new MetaObjectsException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + $"{path} exists but could not be read as JSON: {e.Message}"); + } + + using (doc) + { + var root = doc.RootElement; + if (root.ValueKind != JsonValueKind.Object) + throw new MetaObjectsException(ErrorCode.ERR_COLLECTION_NOT_FOUND, $"{path} must contain a JSON object"); + + if (!root.TryGetProperty("schema_version", out var v) || v.ValueKind != JsonValueKind.Number || v.GetInt32() != 1) + throw new MetaObjectsException(ErrorCode.ERR_COLLECTION_NOT_FOUND, $"{path}: unsupported schema_version (expected 1)"); + + var specs = new List>(); + if (root.TryGetProperty("sources", out var srcs) && srcs.ValueKind == JsonValueKind.Array) + { + foreach (var s in srcs.EnumerateArray()) + { + if (s.ValueKind != JsonValueKind.Object) + throw new MetaObjectsException(ErrorCode.ERR_COLLECTION_NOT_FOUND, $"{path}: each 'sources' entry must be an object"); + var d = new Dictionary(); + foreach (var p in s.EnumerateObject()) d[p.Name] = p.Value.GetString() ?? ""; + if (d.Count != 1) + throw new MetaObjectsException(ErrorCode.ERR_COLLECTION_NOT_FOUND, $"{path}: each 'sources' entry must have exactly one key"); + specs.Add(d); + } + } + + // Unknown top-level keys are IGNORED by design — see the file header. + return new NeutralConfig(specs); + } + } +} +``` + +- [ ] **Step 4: Implement the resolver** + +Create `server/csharp/MetaObjects/Config/SourceResolver.cs`: + +```csharp +namespace MetaObjects.Config; + +public static class SourceResolver +{ + private static readonly HashSet SupportedExtensions = + new(StringComparer.OrdinalIgnoreCase) { ".json", ".yaml", ".yml" }; + + /// Resolve a declared source SET to a de-duplicated list of metadata files. + /// A relative `path` resolves against `configDir` — the directory HOLDING the + /// `.metaobjects/` folder — never against the process working directory. + public static IReadOnlyList ResolveSources( + string configDir, + IReadOnlyList> specs) + { + var seen = new List(); + var known = new HashSet(StringComparer.Ordinal); + + foreach (var spec in specs) + { + if (!spec.TryGetValue("path", out var rawPath)) + { + var kind = spec.Keys.FirstOrDefault() ?? ""; + throw new MetaObjectsException( + ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED, + $"source kind \"{kind}\" is not supported by this toolchain yet; use a \"path\" source"); + } + + var target = Path.IsPathRooted(rawPath) ? rawPath : Path.GetFullPath(Path.Combine(configDir, rawPath)); + + var isDir = Directory.Exists(target); + if (!isDir && !File.Exists(target)) + throw new MetaObjectsException( + ErrorCode.ERR_SOURCE_UNRESOLVED, + $"source path \"{rawPath}\" does not exist (resolved to {target}, relative to {configDir})"); + + // Order is this port's own and is deliberately NOT a cross-port + // contract — see the corpus README. + var found = isDir + ? Directory.EnumerateFiles(target, "*", SearchOption.AllDirectories) + .Where(p => SupportedExtensions.Contains(Path.GetExtension(p))) + .OrderBy(p => p, StringComparer.Ordinal) + : new[] { target }.AsEnumerable(); + + foreach (var f in found) + { + var full = Path.GetFullPath(f); + if (known.Add(full)) seen.Add(full); + } + } + + return seen; + } + + /// The full ladder: declared `sources`, else the default directory. + /// Only the DEFAULT may be absent — a declared source that does not resolve + /// is ERR_SOURCE_UNRESOLVED, a louder failure. + public static IReadOnlyList ResolveCollection(string root) + { + root = Path.GetFullPath(root); + var cfg = NeutralConfig.Read(root); + var specs = cfg?.Sources ?? Array.Empty>(); + + if (specs.Count == 0) + { + var defaultDir = Path.Combine(root, NeutralConfig.DefaultMetadataDir); + if (!Directory.Exists(defaultDir)) + throw new MetaObjectsException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + $"no metadata sources declared in {root} and no default \"{NeutralConfig.DefaultMetadataDir}\" " + + "directory found. Declare \"sources\" in .metaobjects/config.json, or run 'meta init' to scaffold."); + specs = new[] { (IReadOnlyDictionary)new Dictionary { ["path"] = NeutralConfig.DefaultMetadataDir } }; + } + + return ResolveSources(root, specs); + } +} +``` + +- [ ] **Step 5: Run the conformance runner** + +Run: `cd server/csharp && dotnet test MetaObjects.Conformance.Tests/MetaObjects.Conformance.Tests.csproj --nologo --verbosity quiet` +Expected: all 16 corpus cases pass. + +- [ ] **Step 6: Wire the CLI fallback** + +Modify `server/csharp/MetaObjects.Cli/Program.cs`. The positional `` is currently required for `gen` (`:76`), `docs` (`:124`) and `verify` (`:204`). Make it OPTIONAL, falling back to the neutral config: + +```csharp +// Rung 1 is the explicit positional argument; rungs 3-4 live in ResolveCollection. +// C# has no native config surface (rung 2), so the ladder is two rungs here. +static string ResolveMetadataDirOrExit(string? metadataDir) +{ + if (metadataDir is not null) return metadataDir; + try + { + // Proves a collection resolves from cwd, and reports the same errors the + // other ports do. The loader still takes a directory, so hand it the + // resolved root rather than the file list — widening the loader call to a + // source SET is a separate change. + _ = MetaObjects.Config.SourceResolver.ResolveCollection(Directory.GetCurrentDirectory()); + return Path.Combine(Directory.GetCurrentDirectory(), MetaObjects.Config.NeutralConfig.DefaultMetadataDir); + } + catch (MetaObjectsException e) + { + Console.Error.WriteLine($"error: {e.Code}: {e.Message}"); + Environment.Exit(2); + throw; + } +} +``` + +Then at each of the three `if (metadataDir is null || outDir is null)` guards, replace the `metadataDir is null` half with a call to `ResolveMetadataDirOrExit(metadataDir)` assigned before the guard, leaving the `outDir is null` check intact. + +> **Note for the implementer:** the C# loader's `FromDirectory` takes one directory, so this task's CLI wiring resolves the *root*, not the file set. Widening `MetaDataLoader.FromDirectory` to accept a resolved source SET is deliberately NOT in this plan — `Load(IReadOnlyList)` already exists (`MetaDataLoader.cs:334`) and wiring it is a follow-up. What ships here is the reader, the resolver and the corpus. + +- [ ] **Step 7: Run the C# suite** + +Run: `cd server/csharp && dotnet test MetaObjects.Cli.Tests/MetaObjects.Cli.Tests.csproj --nologo --verbosity quiet && dotnet test MetaObjects.Conformance.Tests/MetaObjects.Conformance.Tests.csproj --nologo --verbosity quiet` +Expected: no new failures. + +- [ ] **Step 8: Commit** + +```bash +git add server/csharp/MetaObjects/Config server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs server/csharp/MetaObjects.Cli/Program.cs +git commit -m "feat(csharp): read the port-neutral sources key + +Neutral subset only; the positional metadataDir becomes optional and falls +back to .metaobjects/config.json. Gated by the shared corpus." +``` + +--- + +## Task 5: Java — neutral config reader, resolver, conformance runner, mojo fallback + +Kotlin needs no separate work: it has no CLI entry point of its own (`fun main` appears in neither `metadata-ktx` nor `codegen-kotlin`) and runs through this same Maven plugin. + +**Files:** +- Create: `server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java` +- Create: `server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java` +- Create: `server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java` +- Modify: `server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java` + +**Interfaces:** +- Consumes: `cases.json` (Task 1); `com.metaobjects.ErrorCode` members `ERR_SOURCE_UNRESOLVED`, `ERR_SOURCE_KIND_UNSUPPORTED`, `ERR_COLLECTION_NOT_FOUND` (`ErrorCode.java:287,290,296`). +- Produces: + - `NeutralConfig.read(Path configDir) -> Optional` + - `NeutralConfig.getSources() -> List>` + - `NeutralConfig.DEFAULT_METADATA_DIR -> "metaobjects"` + - `SourceResolver.resolveCollection(Path root) -> List` + - `SourceResolver.resolveSources(Path configDir, List> specs) -> List` + +- [ ] **Step 1: Write the failing conformance runner** + +Create `server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java`: + +```java +package com.metaobjects.config; + +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.MethodSource; + +import javax.json.Json; // use the JSON API already on this module's classpath +import java.nio.file.*; +import java.util.*; +import java.util.stream.Collectors; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * Runs the shared source-resolution corpus against this port. Reads the single + * committed fixtures/source-resolution-conformance/cases.json — no per-port fixture. + */ +class SourceResolutionConformanceTest { + + private static Path corpus() { + Path dir = Paths.get("").toAbsolutePath(); + while (dir != null && !Files.isDirectory(dir.resolve("fixtures"))) dir = dir.getParent(); + assertNotNull(dir, "could not locate the repository fixtures/ directory"); + return dir.resolve("fixtures/source-resolution-conformance/cases.json"); + } + + static List> cases() throws Exception { + // Parse with whatever JSON facility this module already depends on. + return CorpusReader.read(corpus()); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("cases") + void resolvesTheSameFileSet(Map c, @TempDir Path root) throws Exception { + CorpusReader.materialize(c, root); + + Object expectError = c.get("expectError"); + if (expectError != null) { + MetaDataException e = assertThrows(MetaDataException.class, + () -> SourceResolver.resolveCollection(root)); + assertEquals(expectError, e.getErrorCode().name()); + return; + } + + Set got = SourceResolver.resolveCollection(root).stream() + .map(p -> root.toRealPath_unchecked().relativize(p).toString().replace('\\', '/')) + .collect(Collectors.toSet()); + + @SuppressWarnings("unchecked") + Set want = new HashSet<>((List) c.get("expectFiles")); + assertEquals(want, got); + } +} +``` + +> **Implementer note:** `CorpusReader` and `toRealPath_unchecked()` above are placeholders for whatever this module already has. Before writing this file, run `grep -rn "cases.json" server/java/metadata/src/test --include=*.java | head` and copy the corpus-reading approach an existing conformance test uses (`ConformanceTest`, `RegistryManifestConformanceTest`). Do not add a new JSON dependency; reuse the module's existing one. Materialization is: for each `tree` entry create parent dirs and write the content; when `config` is non-null, write it to `/.metaobjects/config.json`. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/java && mvn -pl metadata test -Dtest=SourceResolutionConformanceTest -q` +Expected: compile failure — `com.metaobjects.config` does not exist. + +- [ ] **Step 3: Implement the reader** + +Create `server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java`: + +```java +package com.metaobjects.config; + +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; + +import java.nio.file.*; +import java.util.*; + +/** + * The port-neutral subset of {@code .metaobjects/config.json}. + * + *

Reads only {@code schema_version} and {@code sources}. The file also carries + * TypeScript-owned keys ({@code pending_in_git}, {@code confidence_thresholds}, + * {@code extract}, {@code migrate}); those are IGNORED rather than modeled, so a new + * TS-only key never becomes a four-port change. See + * {@code docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md} §4. + */ +public final class NeutralConfig { + + /** + * The DEFAULT value of {@code sources} when the key is absent or empty — never a + * requirement, and never assumed to exist by any other code path. + */ + public static final String DEFAULT_METADATA_DIR = "metaobjects"; + + private static final String METAOBJECTS_DIR = ".metaobjects"; + private static final String CONFIG_FILE = "config.json"; + + private final List> sources; + + private NeutralConfig(List> sources) { + this.sources = List.copyOf(sources); + } + + public List> getSources() { + return sources; + } + + /** + * Returns empty when the file does not exist. A file that EXISTS but is malformed + * throws — swallowing it would make a typo'd config behave identically to no + * config at all. + */ + public static Optional read(Path configDir) { + Path path = configDir.resolve(METAOBJECTS_DIR).resolve(CONFIG_FILE); + if (!Files.isRegularFile(path)) return Optional.empty(); + + Map raw; + try { + raw = JsonSupport.readObject(path); + } catch (Exception e) { + throw new MetaDataException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + path + " exists but could not be read as JSON: " + e.getMessage()); + } + + Object version = raw.get("schema_version"); + if (!(version instanceof Number) || ((Number) version).intValue() != 1) { + throw new MetaDataException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + path + ": unsupported schema_version " + version + " (expected 1)"); + } + + List> specs = new ArrayList<>(); + Object srcs = raw.getOrDefault("sources", List.of()); + if (srcs instanceof List list) { + for (Object o : list) { + if (!(o instanceof Map m) || m.size() != 1) { + throw new MetaDataException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + path + ": each 'sources' entry must be an object with exactly one key"); + } + Map spec = new LinkedHashMap<>(); + m.forEach((k, v) -> spec.put(String.valueOf(k), String.valueOf(v))); + specs.add(spec); + } + } + + // Unknown top-level keys are IGNORED by design — see the class javadoc. + return Optional.of(new NeutralConfig(specs)); + } +} +``` + +> **Implementer note:** `JsonSupport.readObject` is a placeholder. Use whatever JSON reader `metadata` already depends on — find it with `grep -rn "import.*json" server/java/metadata/src/main/java/com/metaobjects/loader/parser/*.java | head`. Do not add a dependency. Likewise confirm `MetaDataException`'s constructor takes an `ErrorCode` and a message; adapt if it does not. + +- [ ] **Step 4: Implement the resolver** + +Create `server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java`: + +```java +package com.metaobjects.config; + +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.*; +import java.util.*; +import java.util.stream.Stream; + +/** Resolves a declared source SET to a de-duplicated list of metadata files. */ +public final class SourceResolver { + + private static final Set EXTENSIONS = Set.of(".json", ".yaml", ".yml"); + + private SourceResolver() {} + + /** + * A relative {@code path} resolves against {@code configDir} — the directory + * HOLDING the {@code .metaobjects/} folder — never against the process working + * directory. + */ + public static List resolveSources(Path configDir, List> specs) { + LinkedHashSet seen = new LinkedHashSet<>(); + + for (Map spec : specs) { + String rawPath = spec.get("path"); + if (rawPath == null) { + String kind = spec.keySet().stream().findFirst().orElse(""); + throw new MetaDataException( + ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED, + "source kind \"" + kind + "\" is not supported by this toolchain yet; use a \"path\" source"); + } + + Path raw = Paths.get(rawPath); + Path target = raw.isAbsolute() ? raw : configDir.resolve(raw).normalize(); + + boolean isDir = Files.isDirectory(target); + if (!isDir && !Files.isRegularFile(target)) { + throw new MetaDataException( + ErrorCode.ERR_SOURCE_UNRESOLVED, + "source path \"" + rawPath + "\" does not exist (resolved to " + target + + ", relative to " + configDir + ")"); + } + + if (isDir) { + // Order is this port's own and is deliberately NOT a cross-port + // contract — see the corpus README. + try (Stream walk = Files.walk(target)) { + walk.filter(Files::isRegularFile) + .filter(p -> hasSupportedExtension(p.getFileName().toString())) + .sorted(Comparator.comparing(p -> p.getFileName().toString())) + .forEach(p -> seen.add(p.toAbsolutePath().normalize())); + } catch (IOException e) { + throw new UncheckedIOException("Failed to list " + target, e); + } + } else { + seen.add(target.toAbsolutePath().normalize()); + } + } + + return new ArrayList<>(seen); + } + + /** + * The full ladder: declared {@code sources}, else the default directory. Only the + * DEFAULT may be absent — a declared source that does not resolve is + * {@code ERR_SOURCE_UNRESOLVED}, a louder failure. + */ + public static List resolveCollection(Path root) { + Path base = root.toAbsolutePath().normalize(); + List> specs = NeutralConfig.read(base) + .map(NeutralConfig::getSources) + .orElse(List.of()); + + if (specs.isEmpty()) { + Path defaultDir = base.resolve(NeutralConfig.DEFAULT_METADATA_DIR); + if (!Files.isDirectory(defaultDir)) { + throw new MetaDataException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + "no metadata sources declared in " + base + " and no default \"" + + NeutralConfig.DEFAULT_METADATA_DIR + "\" directory found. Declare \"sources\" in " + + ".metaobjects/config.json, or run 'meta init' to scaffold."); + } + specs = List.of(Map.of("path", NeutralConfig.DEFAULT_METADATA_DIR)); + } + + return resolveSources(base, specs); + } + + private static boolean hasSupportedExtension(String name) { + String lower = name.toLowerCase(Locale.ROOT); + for (String ext : EXTENSIONS) if (lower.endsWith(ext)) return true; + return false; + } +} +``` + +- [ ] **Step 5: Run the conformance runner** + +Run: `cd server/java && mvn -pl metadata test -Dtest=SourceResolutionConformanceTest -q` +Expected: all 16 corpus cases pass. + +- [ ] **Step 6: Wire the mojo fallback** + +Modify `server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java`. Rung 2 is the pom: if the `` names **either** `` or ``, the pom owns the concern and the neutral file is NOT consulted. Only when the pom names neither does the mojo fall back: + +```java +/** + * The precedence ladder for where metadata lives (spec §5). First match wins. + * + *

1. The pom — {@code } or {@code }. If EITHER is + * present the pom owns the whole concern and the neutral file is not consulted; + * precedence is whole-concern, not a per-entry merge. + *
2. {@code sources} in the port-neutral {@code .metaobjects/config.json}, read from + * the module basedir. + *
3. The built-in default directory. + * + *

A neutral file that EXISTS but is malformed throws rather than falling through. + */ +protected List resolveNeutralSourcesIfPomIsSilent(LoaderParam loaderConfig) { + boolean pomNamesLocation = + (loaderConfig.getSourceDir() != null && !loaderConfig.getSourceDir().isBlank()) + || (loaderConfig.getSources() != null && !loaderConfig.getSources().isEmpty()); + if (pomNamesLocation) return List.of(); + + return com.metaobjects.config.SourceResolver + .resolveCollection(getProjectBaseDir().toPath()) + .stream() + .map(java.nio.file.Path::toString) + .toList(); +} +``` + +Call it where the loader's sources are assembled and pass the result through the existing `setSourceURIs` / `sources` path. Use the mojo's existing accessor for the module base directory (find it with `grep -n "basedir\|getBasedir\|MavenProject" server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java | head`). + +**Do not touch ``.** It keeps its current semantics — `scope` is out of scope for this plan (Global Constraints). + +- [ ] **Step 7: Run the Java build** + +Run: `cd server/java && mvn -pl metadata,maven-plugin -am install -DskipTests -q && mvn -pl metadata test -q` +Expected: BUILD SUCCESS, no new failures. Do **not** pipe through `tail` — that reports `tail`'s exit status, not Maven's. + +- [ ] **Step 8: Commit** + +```bash +git add server/java/metadata/src/main/java/com/metaobjects/config server/java/metadata/src/test/java/com/metaobjects/config server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java +git commit -m "feat(java): read the port-neutral sources key when the pom is silent + +Whole-concern precedence: a pom naming sourceDir or sources owns the +concern outright. is untouched — scope stays TypeScript-only." +``` + +--- + +## Task 6: `meta init --config-only`, the Node-side writer + +The gap the phase-1 design named: a JVM- or pip-rooted adopter has no `.metaobjects/config.json` at all, so the Node CLI's `migrate` has nothing to discover no matter which ports can read. `meta init` today scaffolds the whole TypeScript project and has no flag to write only the config (`init.ts:635-650`). + +**Files:** +- Modify: `server/typescript/packages/cli/src/commands/init.ts` +- Test: `server/typescript/packages/cli/test/init.test.ts` + +**Interfaces:** +- Consumes: the existing `parseInitArgs` flag parser and `InitOptions` (`init.ts:635-650`). +- Produces: `meta init --config-only` writes exactly `.metaobjects/config.json` and nothing else. + +- [ ] **Step 1: Write the failing test** + +Add to `server/typescript/packages/cli/test/init.test.ts`: + +```ts +test("--config-only writes just the config, no TypeScript scaffold", async () => { + const root = await mkdtemp(join(tmpdir(), "mo-init-config-only-")); + + const result = await runInit({ cwd: root, configOnly: true }); + + // The one file it writes. + expect(result.created).toContain(".metaobjects/config.json"); + const cfg = JSON.parse(await readFile(join(root, ".metaobjects", "config.json"), "utf8")); + expect(cfg.schema_version).toBe(1); + expect(cfg.sources).toEqual([]); + + // None of the TypeScript scaffold — this is the whole point of the flag: a + // Maven- or pip-rooted project declares its sources for the Node CLI without + // acquiring a TS project it will not use. + for (const unwanted of [ + "metaobjects.config.ts", + "codegen/generators/entity.ts", + "package.json", + ".gitignore", + ]) { + expect(existsSync(join(root, unwanted))).toBe(false); + } +}); + +test("--config-only leaves an existing config untouched", async () => { + const root = await mkdtemp(join(tmpdir(), "mo-init-config-only-existing-")); + await mkdir(join(root, ".metaobjects"), { recursive: true }); + const existing = { schema_version: 1, sources: [{ path: "model" }] }; + await writeFile(join(root, ".metaobjects", "config.json"), JSON.stringify(existing)); + + const result = await runInit({ cwd: root, configOnly: true }); + + expect(result.preserved).toContain(".metaobjects/config.json"); + const cfg = JSON.parse(await readFile(join(root, ".metaobjects", "config.json"), "utf8")); + expect(cfg.sources).toEqual([{ path: "model" }]); +}); +``` + +> **Implementer note:** match the existing tests' helper names in that file — if the entry point is not `runInit`, or options are shaped differently, adapt. Check with `grep -n "runInit\|export async function init\|InitOptions" server/typescript/packages/cli/src/commands/init.ts | head`. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/typescript/packages/cli && bun test test/init.test.ts -t "config-only"` +Expected: FAIL — `configOnly` is not a recognized option. + +- [ ] **Step 3: Implement the flag** + +In `init.ts`: add `configOnly?: boolean` to the options type, parse `--config-only` in `parseInitArgs`, thread it through to the entry point, and take an early-return branch that runs only the existing `.metaobjects/config.json` block (`:374-412`) — reusing that code rather than duplicating the write, so the two paths cannot drift on the config's default content. Add `--config-only` to the command's `--help` text with a one-line description naming its purpose (declaring sources for the Node CLI from a non-TypeScript project). + +- [ ] **Step 4: Run the test** + +Run: `cd server/typescript/packages/cli && bun test test/init.test.ts` +Expected: all pass, including the two new cases. + +- [ ] **Step 5: Typecheck** + +Run: `cd server/typescript && bun run --filter '*' typecheck` +Expected: clean. + +- [ ] **Step 6: Commit** + +```bash +git add server/typescript/packages/cli/src/commands/init.ts server/typescript/packages/cli/test/init.test.ts +git commit -m "feat(cli): meta init --config-only + +Writes .metaobjects/config.json and nothing else, so a Maven- or pip-rooted +project can declare its sources for the Node CLI without acquiring a +TypeScript scaffold it will not use." +``` + +--- + +## Task 7: Documentation and changelog + +**Files:** +- Modify: `docs/features/metadata-sources.md:36-42` +- Modify: `CHANGELOG.md` + +- [ ] **Step 1: Replace the "Port support" paragraph** + +`docs/features/metadata-sources.md` currently tells adopters the other four CLIs do not read the config. Replace that paragraph with: + +```markdown +**Port support.** `sources` is read by **all four CLI surfaces** — the Node `meta` +CLI, `dotnet meta` (C#), `metaobjects` (Python) and `metaobjects:generate` (Java +and Kotlin, via Maven). Each resolves the same files from the same declaration; +that promise is gated by +[`fixtures/source-resolution-conformance/`](../../fixtures/source-resolution-conformance/). + +Each port reaches it its own way, and the ladder is the same everywhere — first +match wins: + +1. An explicit CLI argument (a positional metadata directory, `--config`, `--cwd`). +2. The port's own native surface, where it has one — Java and Kotlin's pom + ``/``, Python's `metadata` key in `metaobjects.config.yaml`. + If the pom names either element it owns the concern outright and the neutral + file is not consulted. +3. `sources` in `.metaobjects/config.json`. +4. The built-in default — a `metaobjects/` directory beside that config. + +A config file that exists but is malformed is an error at its own rung; it never +falls through to the next one. + +The non-TypeScript ports read a **neutral subset** of that file — +`schema_version` and `sources` — and ignore every other top-level key, so the +TypeScript-owned keys beside them (`migrate`, `extract`, and the rest) never +become a four-port concern. The Node CLI remains the file's only writer; +`meta init --config-only` writes it into a Maven- or pip-rooted project without +adding a TypeScript scaffold. + +**`scope` and `migrate.scope` remain Node-CLI-only.** Java ships its own +`` grammar whose `*` and `@` mean different things from `scope`'s, so +reconciling them is a separate, adopter-affecting decision rather than a +mechanical port. + +**File order is not a cross-port promise.** Every port resolves the same file +SET; the order within it is each port's own and always has been. +``` + +- [ ] **Step 2: Add the changelog entry** + +Under `## [Unreleased]` in `CHANGELOG.md`: + +```markdown +### Added + +- **`sources` is read by all four CLI surfaces**, not just the Node `meta` CLI — + the C#, Python and Java/Kotlin CLIs now resolve metadata from the + port-neutral `.metaobjects/config.json`, so one declaration serves every port. + Each reads a **neutral subset** (`schema_version` + `sources`) and ignores + unknown top-level keys, so the TypeScript-owned keys in that file never become + a four-port change. Precedence is a ladder — explicit CLI argument, then the + port's native surface (a pom's ``/``, Python's `metadata` + key), then `sources`, then the default `metaobjects/` directory — and a config + that exists but is malformed errors at its rung rather than silently falling + through. Gated by the new `fixtures/source-resolution-conformance/` corpus, + which every port runs. +- **`meta init --config-only`** writes `.metaobjects/config.json` and nothing + else, so a Maven- or pip-rooted project can declare its sources for the Node + CLI (which owns `migrate` and `verify --db`, ADR-0015) without acquiring a + TypeScript scaffold it will not use. + +### Notes + +- **`scope` / `migrate.scope` stay Node-CLI-only for now.** Java has shipped a + `` grammar for years in which `*` crosses the `::` separator and `@` + matches one segment — the exact inverse of `scope`'s `*` and `**` — plus + `!`-prefix exclusion and a `.[attr]` predicate `scope` cannot express. + Reconciling them changes behavior for existing Java consumers, so it is its + own decision rather than a mechanical port. No cross-port behavior depends on + `scope`. +- **Resolved file ORDER is explicitly not a cross-port contract.** The ports' + directory walks already differ and always have; the corpus compares file SETS. + Super-resolution is order-independent (#188) and the loader's overlay + partition discards caller order regardless. +``` + +- [ ] **Step 3: Verify no leaked private names or absolute home paths** + +Run: `scripts/ci-local.sh --quick` +Expected: green. This is the same leak-scan that gates every PR, and it is the +authority — it checks staged content against the configured private-name +denylist as well as absolute user-home path patterns, which an ad-hoc grep here +would not. If it blocks a commit, **genericize the offending line** rather than +bypassing with `--no-verify`. + +- [ ] **Step 4: Commit** + +```bash +git add docs/features/metadata-sources.md CHANGELOG.md +git commit -m "docs: sources is read by all four CLI surfaces + +Records the precedence ladder, the neutral-subset rule, why scope stays +Node-only, and that file order is deliberately not a contract." +``` + +--- + +## Task 8: Full cross-port gate + +- [ ] **Step 1: Run the affected-port lanes** + +```bash +scripts/ci-local.sh --only typescript --strict-toolchains +scripts/ci-local.sh --only python --strict-toolchains +scripts/ci-local.sh --only csharp --strict-toolchains +scripts/ci-local.sh --only java --strict-toolchains +``` + +Expected: all four green. **Do not pipe any of these through `tail`** — the shell reports `tail`'s exit status, so a red lane reads as green. + +- [ ] **Step 2: Typecheck and build the TypeScript workspace** + +Run: `cd server/typescript && bun run --filter '*' build && bun run --filter '*' typecheck` +Expected: clean. + +- [ ] **Step 3: Confirm the corpus is genuinely gating** + +Prove the gate by breaking it, not by its silence. Temporarily change one `expectFiles` entry in `cases.json` to a wrong path, re-run **all four** runners, and confirm **each** goes red: + +```bash +cd server/typescript/packages/sdk && bun test test/source-resolution-conformance.test.ts # expect FAIL +cd server/python && uv run pytest tests/conformance/test_source_resolution_conformance.py # expect FAIL +cd server/csharp && dotnet test MetaObjects.Conformance.Tests/MetaObjects.Conformance.Tests.csproj # expect FAIL +cd server/java && mvn -pl metadata test -Dtest=SourceResolutionConformanceTest # expect FAIL +``` + +Then revert `cases.json`. **A runner that stays green here is not wired to the corpus** — fix it before proceeding. Four ports byte-matching one manifest is the entire point. + +- [ ] **Step 4: Verify the working tree is clean of incidental changes** + +Run: `git status --short` +Expected: empty. In particular `server/python/uv.lock` must NOT be staged — it re-dirties on any `uv run` and is genuine pre-existing drift on `main`. + +- [ ] **Step 5: Final commit if anything was fixed** + +```bash +git add -- +git commit -m "fix: close gaps found by the full cross-port gate" +``` + +Stage explicit paths — never `git add -A`. The stash stack and worktrees are shared across sessions. + +--- + +## Self-Review + +**Spec coverage.** §2 (`sources` ships, `scope` deferred) → Global Constraints + Tasks 2-5, deferral recorded in Task 7. §3 identical column → corpus cases in Task 1; may-differ column → the corpus README's order section and each resolver's order comment. §4 neutral subset → the `unknown-top-level-keys-are-ignored` case plus each reader's ignore-by-default behavior. §5 precedence ladder → Task 3 step 5 (Python), Task 4 step 6 (C#), Task 5 step 6 (Java), documented in Task 7. §6 writer → Task 6. §7 corpus → Task 1, with the CI-latency caveat carried into the README. §8 deferrals → Task 7's changelog Notes. + +**Known plan limitations, stated rather than hidden.** Three tasks carry an explicit *implementer note* where the exact local API could not be verified without opening files this plan does not otherwise touch: Java's JSON facility and `MetaDataException` constructor (Task 5 steps 1, 3), C#'s exception type name (Task 4 step 2), and the CLI test helper names in `init.test.ts` (Task 6 step 1). Each note names the exact `grep` that resolves it. These are look-ups, not design gaps. + +**Scope boundary worth flagging to a reviewer.** Tasks 4 and 5 wire the *resolver* and the *corpus*, and the C# CLI hands the loader a resolved root rather than a resolved file SET, because `MetaDataLoader.FromDirectory` takes one directory. Every port's set-accepting loader entry already exists (spec §2 table); widening the CLI call sites to use it is a deliberate follow-up, not part of this changeset. What ships here is that all four ports **read the same declaration and resolve the same files** — which is the stated requirement. diff --git a/docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md b/docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md new file mode 100644 index 000000000..7a07bec5d --- /dev/null +++ b/docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md @@ -0,0 +1,213 @@ +# Cross-port metadata `sources` — design + +_Status: DESIGN (decisions made; implementation plan is a separate document)._ +_Date: 2026-08-19._ +_Follows: `2026-08-17-metadata-source-resolution-design.md` (phase 1, TypeScript, merged as PR #311)._ +_Evidence: `2026-08-17-metadata-source-resolution-prior-art.md`._ + +## 1. What this decides + +Phase 1 gave the Node `meta` CLI three keys in the port-neutral `.metaobjects/config.json`: +`sources`, `scope`, and `migrate.scope`. The other four CLIs read none of them. + +This document decides how the **Java, Kotlin, C# and Python** CLIs learn where metadata lives. + +**The decision, in one line: ship `sources` to all four ports now, hold `scope` back, and read a +declared neutral SUBSET of the config file rather than the whole schema.** + +The requirement being served, as stated by the maintainer: + +> the ports need not work exactly the same way, but they must be able to read all the same +> sources, and the `.metaobjects` config should be the default when nothing else is specified. + +## 2. Decision 1 — `sources` and `scope` are separate deliveries + +They were bundled in phase 1 because one TypeScript implementation served both. Cross-port they +are not one feature, and bundling them triples the cost of the half that is nearly free. + +**`sources` is cheap.** Every port's loader already accepts a set of sources; only the CLI +convenience wrapper is single-directory: + +| Port | Set-accepting loader entry | Single-dir wrapper | +|---|---|---| +| Java | `load(List)` — `MetaDataLoader.java:1541`; also `setSourceURIs(List)` at `:994` | `:660` | +| C# | `Load(IReadOnlyList)` — `MetaDataLoader.cs:334` | `FromDirectory` `:93`, `:102` | +| Python | `load(sources: list[MetaDataSource])` — `meta_data_loader.py:102` | `from_directory` `:149` | +| TypeScript | (phase 1) | — | + +`DirectorySource` exists in all four codebases (Kotlin inherits the JVM one). The work is CLI +plumbing plus a config reader, not engine work. + +**`scope` is expensive, and it collides.** Java already ships a filter grammar, and it uses the +same characters for different meanings. From `GeneratorUtil.createRegexFromGlob`: + +```java +case '*': out += ".*"; break; // CROSSES :: — any depth +case '@': out += "[^:]+"; break; // exactly one segment +case ':': out += "\\:"; break; // TODO: This doesn't seem to work on enforcing the ::'s as a separator for * +``` + +Java's `@` is the new grammar's `*`; Java's `*` is the new grammar's `**`. Java additionally has +`!`-prefix exclusion (`:36-39`) and a `.[attr]` predicate suffix (`:72`) that `scope` cannot +express. The file carries a `TODO` conceding the separator handling is wrong. + +The two are also the **same tier**, so they compete rather than layer: +`AbstractMetaDataMojo.java:161-165` merges loader-level and generator-level `` and hands +the union to the generator, and TypeScript's `scope` is documented as an "Output filter applied +across every command" (`config.ts:133`). Both filter output; neither filters the load. + +**Therefore:** `scope` cross-port requires either a grammar migration for existing Java consumers +or two coexisting grammars. That is a decision with adopter impact, and it is not a prerequisite +for "read all the same sources." It is deferred to its own design (§8). + +## 3. Decision 2 — the contract boundary + +This is the part the maintainer's "need not work exactly the same way" licenses, and it needs to +be explicit or five ports will each guess. + +### Identical across ports — this is the contract + +1. **The resolved file SET.** Given the same `sources` and the same tree, every port resolves the + same set of metadata files. +2. **The relative-path base.** A relative `path` resolves against **the directory holding the + declaring `.metaobjects/` folder** — never against ambient cwd. Absolute paths are taken + as-is. (TS: `resolveSpecPath`, `sources.ts:102-104`. Python already obeys this rule for its + own key: `project_config.py:77-79` resolves `metadata` under `config_dir`.) +3. **Which file kinds count as metadata** when a directory is walked recursively. +4. **A declared source that does not exist is `ERR_SOURCE_UNRESOLVED`** — never a silent skip. + Only the *default* may be absent, and then it is `ERR_COLLECTION_NOT_FOUND`. +5. **An unsupported source kind is `ERR_SOURCE_KIND_UNSUPPORTED`.** `resource` and `package` are + declared in the shape but resolve in no port yet. +6. **A config file that exists but is malformed is an error** — it must never degrade to "no + config". TypeScript already does this deliberately (`collection.ts:129-140`). +7. **The default when nothing is declared:** one `path` source named `metaobjects` + (`metadata-files.ts:33`), and it is a default *value*, never a requirement. + +### May differ per port — explicitly NOT the contract + +1. **The ORDER of the resolved files.** It already diverges and always has: Java's + `DirectorySource.expand()` sorts by basename (`DirectorySource.java:105`), C# by full-path + ordinal (`DirectorySource.cs:64`), TypeScript walks depth-first with files-before-subdirs + (`metadata-files.ts:101-121`). Making order a contract would be a behavior change in three + ports for no gain: super-resolution is order-independent (#188) and the loader's overlay + partition discards caller order. **Ports keep their existing walk order.** +2. **How the port is told to override** — a positional argument, a `--config` flag, a pom element. +3. **Whether the port has walk-up discovery** (§5). +4. **Config-file caching, error message wording, and diagnostics formatting.** + +> **Known limit, stated rather than papered over.** An identical file *set* does not guarantee an +> identical loaded *model* — overlay partitioning and super-resolution sit downstream. This design +> gates the set. Model-level equivalence is what the existing metamodel conformance corpus gates, +> and the two are separate claims. + +## 4. Decision 3 — read a neutral SUBSET, not the whole schema + +`.metaobjects/config.json` is parsed by a `.strict()` zod schema that carries **TypeScript-owned +keys**: `pending_in_git`, `confidence_thresholds`, `extract.metaignore`, and `migrate` +(`config.ts:123-142`). + +Requiring four ports to model those — or to reject a valid config containing them — is untenable +and would make every future TS-only key a four-port change. + +**The rule:** + +- The **neutral subset** is `schema_version`, `sources`, and (later) `scope`. It is specified in + its own document and versioned by `schema_version`. +- The four ports **validate the subset strictly** and **ignore unknown top-level keys**. +- **TypeScript remains the only strict validator of the whole file, and the only writer.** + +The cost is real and accepted: a key misspelled *outside* the subset is caught only by the Node +CLI. The alternative — four ports modeling `confidence_thresholds` — is worse. + +## 5. Decision 4 — precedence, and what "default" means + +The maintainer's "the `.metaobjects` config is default if nothing is specified" is read as +**fallback**, not authority. The ladder, per port, first match wins: + +1. **An explicit CLI argument** — C#'s positional ``, Python's positional + `metadata_dir`, `--config`, `--cwd`. Wins outright. +2. **The port's native config surface**, when it names a metadata location — Java/Kotlin's pom + ``/``, Python's `metadata` key in `metaobjects.config.yaml`. +3. **`.metaobjects/config.json`'s `sources`**, when present and non-empty. +4. **The built-in default** — one `path` source named `metaobjects`. + +At every rung: **present but malformed is an error, not a fall-through to the next rung.** + +**Precedence is whole-concern for `sources`.** A native surface either names a metadata location +or it does not. There is no per-entry merging of a pom's `` with the neutral file's — +that would be a five-way merge matrix, and merge matrices are what drift across five +implementations. With `scope` deferred (§2) the per-key question does not arise in this phase; it +must be answered by the `scope` design when it lands. + +**Java's pom `` is not the neutral `sources`.** It lists individual metadata *documents* +(files, `resource:` classpath entries, `model:` URIs) resolved against `sourceDir`; neutral +`sources` is a set of *roots*. They are different primitives that happen to share a name. A pom +declaring either `` or `` occupies rung 2 and the neutral file is not +consulted; neither element changes meaning. + +## 6. Decision 5 — the writer gap is Node-side, not four writers + +The gap §4.6 of the phase-1 design named: a JVM-rooted adopter scaffolded with `agent-docs` has +no `.metaobjects/config.json` at all, so the Node CLI's `migrate` has nothing to discover no +matter which ports can read. + +**It does not follow that four ports need writers.** `migrate` and `verify --db` are Node-only +(ADR-0015), so the adopter who needs that file is by definition already invoking the Node CLI. +The missing piece is a *lighter* Node writer: `meta init` today scaffolds the full TypeScript +project (`metaobjects/`, `codegen/generators/`, `metaobjects.config.ts`, `.gitignore`, +`package.json` edits) and has no flag to write only the config (`init.ts:635-650` — the flags are +`force`, `quiet`, `printOnly`, `refreshDocs`, `d1`). + +**Decision: add a config-only mode to the Node `meta init`.** It writes `.metaobjects/config.json` +and nothing else, so a Maven- or pip-rooted project can declare its sources for the Node CLI +without acquiring a TypeScript scaffold it will not use. Four port writers stay out of scope. + +## 7. Conformance — a new corpus, and an honest note about its reach + +`fixtures/scope-conformance/` has 10 cases and **exactly one runner** +(`server/typescript/packages/sdk/test/scope-conformance.test.ts`). A corpus with one runner pins +nothing cross-port. Since `scope` is deferred, that corpus stays as-is and gains runners with the +`scope` design. + +**This phase adds `fixtures/source-resolution-conformance/`**, gating §3's identical column: +the resolved file **set** (order-insensitively), the relative-path base, the default-when-absent, +and each error condition. Every port ships a runner in the same changeset as its reader. + +> **A limitation to record rather than discover later.** The non-TypeScript lanes do not run on +> pull requests — they run on push-to-`main`, release tags, and manual dispatch +> (`AGENTS.md:91-92`). So this corpus has the same detection latency as the code it gates: it +> cannot catch a divergence *before* merge. It is a regression gate, not a review gate. That is +> the standing reason the four readers ship in one changeset rather than one port at a time. + +## 8. Deferred, with the reason + +- **`scope` cross-port** — blocked on the grammar collision (§2). Needs its own design answering: + migrate Java consumers, or run two grammars? Until then `scope` stays TypeScript-only and no + cross-port behavior may depend on it. +- **`resource` and `package` source kinds** — declared in the shape, resolving nowhere. `resource` + (classpath) is JVM-natural and unreachable from Node; `package` needs the distribution ADR + (prior art P5: no surveyed project publishes one code-free schema artifact to four registries). +- **Named collections** — the phase-1 design's §6 deferral is unchanged. +- **Walk-up discovery in the non-TS ports.** None has it today (Python's `_find_config` is + cwd-only, `cli.py:192-201`; C# has no config surface; Java uses the Maven reactor, which makes + discovery a non-issue for codegen). Ports read the config file at their already-known project + root. Adding a walk is a separate, additive change. + +## 9. What was checked, and what was not + +Verified in the repository at commit `19e421927` while writing this document: every file:line +citation above. The loader entry points, the `GeneratorUtil` grammar, the filter-merge site, the +per-port `DirectorySource` sort orders, the config schema's key list, the malformed-config +behavior, the `meta init` flag set, Python's `metadata` key and its cwd-only config lookup, and +the absence of any Kotlin CLI entry point (`fun main` appears in neither `metadata-ktx` nor +`codegen-kotlin` — Kotlin is Maven-only, so there are **four** CLI surfaces, not five). + +Also verified, after the first draft of this document listed it as an open risk: **all four ports +already agree on which extensions count as metadata** — `.json`, `.yaml`, `.yml`, matched +case-insensitively (`DirectorySource.java:61`, `DirectorySource.cs:29-30`, +`directory_source.py:16`, `metadata-files.ts:52`). §3's identical-item 3 is therefore a property +the ports already hold, not one the plan has to establish. It still gains a corpus case, because +nothing currently pins it. + +Nothing else in §3's identical column is known to diverge today, with the single exception of +file *order*, which §3 places explicitly outside the contract. From 8aa0d8841eeea3b8f6240a52d942ec031fc263e1 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 09:58:47 -0400 Subject: [PATCH 18/44] test(conformance): a corpus for cross-port source resolution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pins the resolved file SET — not its order, which already differs by port and is not a contract. TypeScript is the reference and ships the first runner; the other three follow in this changeset. --- .../source-resolution-conformance/README.md | 92 ++++++++++ .../source-resolution-conformance/cases.json | 157 ++++++++++++++++++ .../source-resolution-conformance.test.ts | 74 +++++++++ 3 files changed, 323 insertions(+) create mode 100644 fixtures/source-resolution-conformance/README.md create mode 100644 fixtures/source-resolution-conformance/cases.json create mode 100644 server/typescript/packages/sdk/test/source-resolution-conformance.test.ts diff --git a/fixtures/source-resolution-conformance/README.md b/fixtures/source-resolution-conformance/README.md new file mode 100644 index 000000000..d3f295ebd --- /dev/null +++ b/fixtures/source-resolution-conformance/README.md @@ -0,0 +1,92 @@ +# source-resolution-conformance + +Pins how a consumer's `sources` set in `.metaobjects/config.json` resolves to a +set of metadata files. Every port's CLI must resolve the SAME FILES from the +same declaration — that is the cross-port promise this corpus exists to keep. + +Companion to `scope-conformance/`, which pins the (currently TypeScript-only) +`scope` pattern grammar. The two are independent: `sources` decides which files +are read, `scope` filters what is emitted from them. + +## Shape + +``` +cases.json # { cases: [{ name, tree, config, resolveFrom?, expectFiles?, expectError? }] } +README.md +``` + +- **`tree`** — a map of project-root-relative path → file content. The runner + materializes it in a fresh temporary directory. A `.keep` entry exists only to + force an otherwise-empty directory to be created. +- **`config`** — written verbatim to `.metaobjects/config.json`, under the + directory named by `resolveFrom` (project root when `resolveFrom` is absent). + When `null`, no config file is created at all. +- **`resolveFrom`** — OPTIONAL, a project-root-relative directory path, + default `"."`. Names the directory the resolver is invoked against — i.e. + the directory treated as holding `.metaobjects/`. Exists so a case can prove + a relative `path` source resolves against the *declaring config's* + directory rather than the project root or the process's ambient working + directory: put `resolveFrom` somewhere other than `"."` and a `path` source + containing `../` only lands on the right files if the port under test + resolved it against the right base. Every port's runner must honor this key + — it is part of the case schema from the start rather than retrofitted once + three ports already exist. +- **`expectFiles`** — project-root-relative paths (NOT relative to + `resolveFrom`), compared as an **UNORDERED SET**. See "Order is + deliberately not pinned" below. +- **`expectError`** — an error code the resolution must fail with. Exactly one + of `expectFiles` / `expectError` is present per case. + +## Semantics pinned here + +- **Default.** `sources` absent or empty ⇒ exactly one `path` source, the literal + `metaobjects`. It is a default VALUE, never a requirement. +- **Replacement, not merge.** A declared `sources` replaces the default entirely — + the default directory is not implicitly appended. +- **Relative base.** A relative `path` resolves against the directory HOLDING the + `.metaobjects/` folder, never against the process working directory. See + `a-parent-relative-path-resolves-against-the-declaring-configs-directory`, + which uses `resolveFrom` to invoke resolution from a subdirectory while the + config's own `path` source climbs back out with `../` — the case only + passes when a port resolves relative to the config's directory, not to + wherever the process happened to be started. +- **Recursion.** A directory `path` is walked recursively; a file `path` resolves + to that one file. +- **Extensions.** `.json`, `.yaml`, `.yml`, matched case-insensitively. Nothing else. +- **Union with de-duplication.** Overlapping sources yield each file exactly once. +- **A declared path that does not exist is `ERR_SOURCE_UNRESOLVED`** — never a + silent skip. Only the DEFAULT may be absent, and then it is + `ERR_COLLECTION_NOT_FOUND`. +- **`resource` and `package` kinds are declared but resolve nowhere yet:** + `ERR_SOURCE_KIND_UNSUPPORTED`. +- **Unknown top-level config keys are IGNORED.** The file carries + TypeScript-owned keys no other port models. `schema_version` and `sources` are + the neutral subset; each port validates those strictly and ignores the rest. + +## Order is deliberately NOT pinned + +`expectFiles` is a set. The ports' directory walks already differ and always +have — Java sorts by basename (`DirectorySource.java:105`), C# by full-path +ordinal (`DirectorySource.cs:64`), Python by basename +(`directory_source.py:40-48`), TypeScript walks depth-first with files before +subdirectories (`metadata-files.ts:101-121`). Making order a contract would be a +behavior change in three ports for no benefit: super-resolution is +order-independent (#188) and the loader's overlay partition discards caller +order anyway. + +A port MAY have a stable internal order — several do, and their own generated +output depends on it. It just is not a cross-port promise. + +## Behavioral contract + +Each port's runner reads `cases.json`, and for every case: materializes `tree` +in a fresh temp directory, writes `config` when non-null under the directory +named by `resolveFrom` (default the project root), resolves sources against +that directory, then asserts either that the resolved file set equals +`expectFiles` (as a set, project-root-relative, path separators normalized to +`/`) or that resolution failed with `expectError`. + +## Reference implementation + +`server/typescript/packages/sdk/src/sources.ts` (`resolveSources`) and +`server/typescript/packages/sdk/src/collection.ts` (`resolveCollection`). diff --git a/fixtures/source-resolution-conformance/cases.json b/fixtures/source-resolution-conformance/cases.json new file mode 100644 index 000000000..c2854ec15 --- /dev/null +++ b/fixtures/source-resolution-conformance/cases.json @@ -0,0 +1,157 @@ +{ + "cases": [ + { + "name": "no-config-uses-default-directory", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": null, + "expectFiles": ["metaobjects/meta.users.json"] + }, + { + "name": "empty-sources-uses-default-directory", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [] }, + "expectFiles": ["metaobjects/meta.users.json"] + }, + { + "name": "declared-path-replaces-the-default-entirely", + "tree": { + "metaobjects/ignored.json": "{\"metadata.root\":{\"children\":[]}}", + "model/meta.orders.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": ["model/meta.orders.json"] + }, + { + "name": "directory-is-walked-recursively", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/nested/meta.b.yaml": "metadata.root:\n children: []\n", + "model/nested/deep/meta.c.yml": "metadata.root:\n children: []\n" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": [ + "model/meta.a.json", + "model/nested/meta.b.yaml", + "model/nested/deep/meta.c.yml" + ] + }, + { + "name": "non-metadata-extensions-are-ignored", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/README.md": "not metadata", + "model/notes.txt": "not metadata", + "model/script.ts": "export {}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": ["model/meta.a.json"] + }, + { + "name": "a-single-file-path-resolves-to-that-file", + "tree": { + "vendor/meta.catalog.json": "{\"metadata.root\":{\"children\":[]}}", + "vendor/meta.other.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "vendor/meta.catalog.json" }] }, + "expectFiles": ["vendor/meta.catalog.json"] + }, + { + "name": "two-sources-union", + "tree": { + "a/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "b/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "a" }, { "path": "b" }] }, + "expectFiles": ["a/meta.a.json", "b/meta.b.json"] + }, + { + "name": "overlapping-sources-yield-each-file-once", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/nested/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "model" }, { "path": "model/nested" }] + }, + "expectFiles": ["model/meta.a.json", "model/nested/meta.b.json"] + }, + { + "name": "source-order-does-not-change-the-resolved-set", + "tree": { + "a/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "b/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "b" }, { "path": "a" }] }, + "expectFiles": ["a/meta.a.json", "b/meta.b.json"] + }, + { + "name": "a-parent-relative-path-resolves-against-the-declaring-configs-directory", + "tree": { + "shared/meta.shared.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "../shared" }] }, + "resolveFrom": "app", + "expectFiles": ["shared/meta.shared.json"] + }, + { + "name": "a-declared-path-that-does-not-exist-is-an-error", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "nope" }] }, + "expectError": "ERR_SOURCE_UNRESOLVED" + }, + { + "name": "an-empty-directory-source-resolves-to-no-files", + "tree": { + "model/.keep": "" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": [] + }, + { + "name": "resource-kind-is-unsupported", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "resource": "com/acme/model" }] }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "package-kind-is-unsupported", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "package": "acme-model" }] }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "no-config-and-no-default-directory-is-an-error", + "tree": { + "src/.keep": "" + }, + "config": null, + "expectError": "ERR_COLLECTION_NOT_FOUND" + }, + { + "name": "unknown-top-level-keys-are-ignored", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "model" }], + "pending_in_git": true, + "confidence_thresholds": { "pending_promote": 0.8 }, + "extract": { "metaignore": ".metaignore" }, + "migrate": { "dialect": "postgres" } + }, + "expectFiles": ["model/meta.a.json"] + } + ] +} diff --git a/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts b/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts new file mode 100644 index 000000000..6e6caef8c --- /dev/null +++ b/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts @@ -0,0 +1,74 @@ +// Runs the shared source-resolution corpus against the TypeScript reference +// implementation. Every port ships an equivalent runner reading this same file. +import { describe, expect, test } from "bun:test"; +import { mkdtemp, mkdir, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join, relative, resolve, sep } from "node:path"; +import { resolveCollection } from "../src/collection.js"; + +interface Case { + readonly name: string; + readonly tree: Record; + readonly config: unknown | null; + /** Project-root-relative directory the resolver is invoked against; default + * ".". `config` (when non-null) is written under this directory's + * `.metaobjects/config.json`. See the corpus README, "Shape". */ + readonly resolveFrom?: string; + readonly expectFiles?: readonly string[]; + readonly expectError?: string; +} + +const CORPUS = resolve( + import.meta.dir, + "../../../../../fixtures/source-resolution-conformance/cases.json", +); + +/** Materializes `c.tree` under a fresh temp root and, when `c.config` is + * non-null, writes it to `/.metaobjects/config.json`. Returns + * both the project root (`expectFiles` is relative to this) and the + * directory the resolver should be invoked against (`resolveFrom`-relative + * to the root, defaulting to the root itself). */ +async function materialize(c: Case): Promise<{ root: string; resolveDir: string }> { + const root = await mkdtemp(join(tmpdir(), "mo-src-conf-")); + for (const [rel, content] of Object.entries(c.tree)) { + const abs = join(root, rel); + await mkdir(dirname(abs), { recursive: true }); + await writeFile(abs, content); + } + const resolveDir = resolve(root, c.resolveFrom ?? "."); + if (c.config !== null) { + await mkdir(join(resolveDir, ".metaobjects"), { recursive: true }); + await writeFile( + join(resolveDir, ".metaobjects", "config.json"), + JSON.stringify(c.config, null, 2), + ); + } + return { root, resolveDir }; +} + +const cases: Case[] = JSON.parse(await readFile(CORPUS, "utf8")).cases; + +describe("source-resolution conformance", () => { + test("corpus is non-empty (a silent zero-case run is a failed gate)", () => { + expect(cases.length).toBeGreaterThan(0); + }); + + for (const c of cases) { + test(c.name, async () => { + const { root, resolveDir } = await materialize(c); + if (c.expectError !== undefined) { + let code: string | undefined; + try { + await resolveCollection(resolveDir, { explicitDir: resolveDir }); + } catch (e) { + code = (e as { code?: string }).code; + } + expect(code).toBe(c.expectError); + return; + } + const collection = await resolveCollection(resolveDir, { explicitDir: resolveDir }); + const got = collection.files.map((f) => relative(root, f).split(sep).join("/")).sort(); + expect(got).toEqual([...(c.expectFiles ?? [])].sort()); + }); + } +}); From fdb5298c1b3e27c73ea62c72a6c742e7bd91d671 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 10:07:03 -0400 Subject: [PATCH 19/44] test(conformance): pin case-insensitive metadata extension matching Fix round 1 finding (Important): the corpus asserted "matched case-insensitively" as a semantic but no case exercised an uppercase or mixed-case extension, so a port implementing ext === ".json" would pass every case and only diverge on a real project. Adds a case mixing .JSON/.YAML/.Yml alongside a lowercase control file and an uppercase unsupported extension, to also catch case-folding that over-matches. --- .../source-resolution-conformance/README.md | 6 ++++++ .../source-resolution-conformance/cases.json | 17 +++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/fixtures/source-resolution-conformance/README.md b/fixtures/source-resolution-conformance/README.md index d3f295ebd..5c571219f 100644 --- a/fixtures/source-resolution-conformance/README.md +++ b/fixtures/source-resolution-conformance/README.md @@ -53,6 +53,12 @@ README.md - **Recursion.** A directory `path` is walked recursively; a file `path` resolves to that one file. - **Extensions.** `.json`, `.yaml`, `.yml`, matched case-insensitively. Nothing else. + See `metadata-extensions-are-matched-case-insensitively`, which mixes + `.JSON`/`.YAML`/`.Yml` spellings alongside a normal lowercase file AND a + same-family unsupported extension in uppercase (`.TXT`) — a port that only + lowercases `.json` (missing `.yaml`/`.yml`), or that folds case and then + matches too loosely (accepting any extension once folded), diverges from + this case either way. - **Union with de-duplication.** Overlapping sources yield each file exactly once. - **A declared path that does not exist is `ERR_SOURCE_UNRESOLVED`** — never a silent skip. Only the DEFAULT may be absent, and then it is diff --git a/fixtures/source-resolution-conformance/cases.json b/fixtures/source-resolution-conformance/cases.json index c2854ec15..7086e048f 100644 --- a/fixtures/source-resolution-conformance/cases.json +++ b/fixtures/source-resolution-conformance/cases.json @@ -50,6 +50,23 @@ "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, "expectFiles": ["model/meta.a.json"] }, + { + "name": "metadata-extensions-are-matched-case-insensitively", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/meta.b.JSON": "{\"metadata.root\":{\"children\":[]}}", + "model/meta.c.YAML": "metadata.root:\n children: []\n", + "model/meta.d.Yml": "metadata.root:\n children: []\n", + "model/notes.TXT": "not metadata" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": [ + "model/meta.a.json", + "model/meta.b.JSON", + "model/meta.c.YAML", + "model/meta.d.Yml" + ] + }, { "name": "a-single-file-path-resolves-to-that-file", "tree": { From cd8f5da4fc9f65d0237c4d620461cc5607108f83 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 10:11:53 -0400 Subject: [PATCH 20/44] feat(python): read the port-neutral sources key Neutral subset only (schema_version + sources); unknown top-level keys are ignored so a TypeScript-owned key never becomes a four-port change. Adds resolve_sources/resolve_collection mirroring the TS reference (sdk/src/sources.ts, collection.ts): relative path sources resolve against the config's own directory, a declared sources list replaces the default entirely, and the default metaobjects/ directory is the only source allowed to be silently absent. --- .../python/src/metaobjects/config/__init__.py | 17 ++++ .../src/metaobjects/config/neutral_config.py | 68 ++++++++++++++++ .../src/metaobjects/config/source_resolver.py | 79 +++++++++++++++++++ server/python/tests/config/__init__.py | 0 .../tests/config/test_neutral_config.py | 63 +++++++++++++++ .../tests/config/test_source_resolver.py | 74 +++++++++++++++++ 6 files changed, 301 insertions(+) create mode 100644 server/python/src/metaobjects/config/__init__.py create mode 100644 server/python/src/metaobjects/config/neutral_config.py create mode 100644 server/python/src/metaobjects/config/source_resolver.py create mode 100644 server/python/tests/config/__init__.py create mode 100644 server/python/tests/config/test_neutral_config.py create mode 100644 server/python/tests/config/test_source_resolver.py diff --git a/server/python/src/metaobjects/config/__init__.py b/server/python/src/metaobjects/config/__init__.py new file mode 100644 index 000000000..c0881621e --- /dev/null +++ b/server/python/src/metaobjects/config/__init__.py @@ -0,0 +1,17 @@ +"""Port-neutral `.metaobjects/config.json` reading and source resolution. + +Reads only the NEUTRAL SUBSET (`schema_version`, `sources`). The file also +carries TypeScript-owned keys; those are ignored rather than modeled, so a new +TS-only key never becomes a four-port change. See +`docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §4. +""" +from .neutral_config import DEFAULT_METADATA_DIR, NeutralConfig, read_neutral_config +from .source_resolver import resolve_collection, resolve_sources + +__all__ = [ + "DEFAULT_METADATA_DIR", + "NeutralConfig", + "read_neutral_config", + "resolve_collection", + "resolve_sources", +] diff --git a/server/python/src/metaobjects/config/neutral_config.py b/server/python/src/metaobjects/config/neutral_config.py new file mode 100644 index 000000000..351bb69ac --- /dev/null +++ b/server/python/src/metaobjects/config/neutral_config.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path + +from metaobjects.errors import ErrorCode, ParseError + +#: The DEFAULT value of `sources` when the key is absent or empty — never a +#: requirement, and never assumed to exist by any other code path. +DEFAULT_METADATA_DIR = "metaobjects" + +_METAOBJECTS_DIR = ".metaobjects" +_CONFIG_FILE = "config.json" + + +@dataclass(frozen=True) +class NeutralConfig: + """The port-neutral subset of `.metaobjects/config.json`.""" + + #: Raw source specs, each a single-key mapping (`path` / `resource` / `package`). + sources: list[dict[str, str]] + + +def read_neutral_config(config_dir: Path) -> NeutralConfig | None: + """Read the neutral subset from ``config_dir/.metaobjects/config.json``. + + Returns ``None`` when the file does not exist. A file that EXISTS but is + malformed raises — swallowing it would make a typo'd config behave + identically to no config at all, silently resolving a possibly-stale + default directory with no diagnostic. + """ + path = config_dir / _METAOBJECTS_DIR / _CONFIG_FILE + if not path.is_file(): + return None + + try: + raw = json.loads(path.read_text()) + except (OSError, ValueError) as e: + raise ParseError( + f"{path} exists but could not be read as JSON: {e}", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) from e + + if not isinstance(raw, dict): + raise ParseError( + f"{path} must contain a JSON object", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + version = raw.get("schema_version") + if version != 1: + raise ParseError( + f"{path}: unsupported schema_version {version!r} (expected 1)", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + sources = raw.get("sources", []) + if not isinstance(sources, list) or not all( + isinstance(s, dict) and len(s) == 1 for s in sources + ): + raise ParseError( + f"{path}: 'sources' must be an array of single-key objects", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + # Unknown top-level keys are IGNORED by design — see the module docstring. + return NeutralConfig(sources=[dict(s) for s in sources]) diff --git a/server/python/src/metaobjects/config/source_resolver.py b/server/python/src/metaobjects/config/source_resolver.py new file mode 100644 index 000000000..e6cc42f1c --- /dev/null +++ b/server/python/src/metaobjects/config/source_resolver.py @@ -0,0 +1,79 @@ +from __future__ import annotations + +from pathlib import Path + +from metaobjects.errors import ErrorCode, ParseError + +from .neutral_config import DEFAULT_METADATA_DIR, read_neutral_config + +_SUPPORTED_SUFFIXES = (".json", ".yaml", ".yml") + + +def _list_metadata_files(directory: Path) -> list[Path]: + """Recursively list metadata files under ``directory``. + + Mirrors `DirectorySource`'s extension set (`.json`/`.yaml`/`.yml`, + case-insensitive). Order is this port's own and is deliberately NOT a + cross-port contract — see the corpus README. + """ + return sorted( + (p for p in directory.rglob("*") if p.is_file() and p.suffix.lower() in _SUPPORTED_SUFFIXES), + key=lambda p: p.name, + ) + + +def resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path]: + """Resolve a declared source SET to a de-duplicated list of metadata files. + + A relative ``path`` resolves against ``config_dir`` — the directory HOLDING + the ``.metaobjects/`` folder — never against the process working directory. + """ + seen: dict[Path, None] = {} + + for spec in specs: + if "path" not in spec: + kind = next(iter(spec), "") + raise ParseError( + f'source kind "{kind}" is not supported by this toolchain yet; use a "path" source', + code=ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED, + ) + + raw = Path(spec["path"]) + target = raw if raw.is_absolute() else (config_dir / raw) + + if not target.exists(): + raise ParseError( + f'source path "{spec["path"]}" does not exist ' + f"(resolved to {target}, relative to {config_dir})", + code=ErrorCode.ERR_SOURCE_UNRESOLVED, + ) + + found = _list_metadata_files(target) if target.is_dir() else [target] + for f in found: + seen.setdefault(f.resolve(), None) + + return list(seen) + + +def resolve_collection(root: Path) -> list[Path]: + """The full ladder: declared `sources`, else the default directory. + + Only the DEFAULT may be absent — a declared source that does not resolve is + `ERR_SOURCE_UNRESOLVED`, a louder failure. + """ + root = root.resolve() + cfg = read_neutral_config(root) + specs = cfg.sources if cfg is not None and cfg.sources else [] + + if not specs: + default_dir = root / DEFAULT_METADATA_DIR + if not default_dir.is_dir(): + raise ParseError( + f'no metadata sources declared in {root} and no default ' + f'"{DEFAULT_METADATA_DIR}" directory found. Declare "sources" in ' + f".metaobjects/config.json, or run 'meta init' to scaffold.", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + specs = [{"path": DEFAULT_METADATA_DIR}] + + return resolve_sources(root, specs) diff --git a/server/python/tests/config/__init__.py b/server/python/tests/config/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/server/python/tests/config/test_neutral_config.py b/server/python/tests/config/test_neutral_config.py new file mode 100644 index 000000000..19fddbcd2 --- /dev/null +++ b/server/python/tests/config/test_neutral_config.py @@ -0,0 +1,63 @@ +import json +import pytest +from pathlib import Path + +from metaobjects.config.neutral_config import read_neutral_config +from metaobjects.errors import ParseError + + +def _write_config(root: Path, payload: object) -> None: + d = root / ".metaobjects" + d.mkdir(parents=True, exist_ok=True) + (d / "config.json").write_text(json.dumps(payload)) + + +def test_absent_config_returns_none(tmp_path: Path) -> None: + assert read_neutral_config(tmp_path) is None + + +def test_reads_sources(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 1, "sources": [{"path": "model"}]}) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [{"path": "model"}] + + +def test_unknown_top_level_keys_are_ignored(tmp_path: Path) -> None: + # The file carries TypeScript-owned keys this port must not model. + _write_config( + tmp_path, + { + "schema_version": 1, + "sources": [{"path": "model"}], + "pending_in_git": True, + "confidence_thresholds": {"pending_promote": 0.8}, + "extract": {"metaignore": ".metaignore"}, + "migrate": {"dialect": "postgres"}, + }, + ) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [{"path": "model"}] + + +def test_absent_sources_key_yields_empty_list(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 1}) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [] + + +def test_malformed_json_raises_not_none(tmp_path: Path) -> None: + d = tmp_path / ".metaobjects" + d.mkdir(parents=True) + (d / "config.json").write_text("{ not json") + # A file that EXISTS but cannot be read must never look like no config at all. + with pytest.raises(ParseError): + read_neutral_config(tmp_path) + + +def test_wrong_schema_version_raises(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 2, "sources": []}) + with pytest.raises(ParseError): + read_neutral_config(tmp_path) diff --git a/server/python/tests/config/test_source_resolver.py b/server/python/tests/config/test_source_resolver.py new file mode 100644 index 000000000..a2987caef --- /dev/null +++ b/server/python/tests/config/test_source_resolver.py @@ -0,0 +1,74 @@ +import json +import pytest +from pathlib import Path + +from metaobjects.config.source_resolver import resolve_collection, resolve_sources +from metaobjects.errors import ErrorCode, ParseError + + +def _rel(root: Path, files: list[Path]) -> set[str]: + return {p.relative_to(root).as_posix() for p in files} + + +def test_directory_is_walked_recursively(tmp_path: Path) -> None: + (tmp_path / "model" / "nested").mkdir(parents=True) + (tmp_path / "model" / "a.json").write_text("{}") + (tmp_path / "model" / "nested" / "b.yaml").write_text("{}") + (tmp_path / "model" / "README.md").write_text("x") + got = resolve_sources(tmp_path, [{"path": "model"}]) + assert _rel(tmp_path, got) == {"model/a.json", "model/nested/b.yaml"} + + +def test_single_file_spec(tmp_path: Path) -> None: + (tmp_path / "vendor").mkdir() + (tmp_path / "vendor" / "one.json").write_text("{}") + (tmp_path / "vendor" / "two.json").write_text("{}") + got = resolve_sources(tmp_path, [{"path": "vendor/one.json"}]) + assert _rel(tmp_path, got) == {"vendor/one.json"} + + +def test_overlapping_sources_dedupe(tmp_path: Path) -> None: + (tmp_path / "model" / "nested").mkdir(parents=True) + (tmp_path / "model" / "a.json").write_text("{}") + (tmp_path / "model" / "nested" / "b.json").write_text("{}") + got = resolve_sources(tmp_path, [{"path": "model"}, {"path": "model/nested"}]) + assert _rel(tmp_path, got) == {"model/a.json", "model/nested/b.json"} + assert len(got) == 2 + + +def test_missing_path_raises_unresolved(tmp_path: Path) -> None: + with pytest.raises(ParseError) as e: + resolve_sources(tmp_path, [{"path": "nope"}]) + assert e.value.code == ErrorCode.ERR_SOURCE_UNRESOLVED + + +def test_resource_kind_unsupported(tmp_path: Path) -> None: + with pytest.raises(ParseError) as e: + resolve_sources(tmp_path, [{"resource": "com/acme"}]) + assert e.value.code == ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED + + +def test_collection_falls_back_to_default_dir(tmp_path: Path) -> None: + (tmp_path / "metaobjects").mkdir() + (tmp_path / "metaobjects" / "a.json").write_text("{}") + got = resolve_collection(tmp_path) + assert _rel(tmp_path, got) == {"metaobjects/a.json"} + + +def test_collection_with_no_config_and_no_default_raises(tmp_path: Path) -> None: + with pytest.raises(ParseError) as e: + resolve_collection(tmp_path) + assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND + + +def test_declared_sources_replace_the_default(tmp_path: Path) -> None: + (tmp_path / ".metaobjects").mkdir() + (tmp_path / ".metaobjects" / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + (tmp_path / "metaobjects").mkdir() + (tmp_path / "metaobjects" / "ignored.json").write_text("{}") + (tmp_path / "model").mkdir() + (tmp_path / "model" / "used.json").write_text("{}") + got = resolve_collection(tmp_path) + assert _rel(tmp_path, got) == {"model/used.json"} From 3ad92b4d11ab184260a7a748f1ad59b9210bcf50 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 10:24:22 -0400 Subject: [PATCH 21/44] fix(python): validate all source-spec kinds before touching the filesystem resolve_sources previously interleaved kind validation with per-spec path resolution, one spec at a time, so which error code came back for a multi-spec list containing both an unsupported kind and an unresolved path depended on declaration order. TypeScript's orderedPathSpecs deliberately validates every spec's kind across the whole list before any stat() call (sources.ts:75-79) precisely to keep that order-independent; Python now matches, verified empirically against the shipped TS reference in both declaration orders. Also tightens two error-path tests to assert .code, not just the exception type, and adds a corpus pair (both declaration orders) pinning the precedence as a cross-port contract so the C# and Java ports (which mirror this Python design) can't independently diverge on it. --- .../source-resolution-conformance/README.md | 12 ++++++++ .../source-resolution-conformance/cases.json | 22 ++++++++++++++ .../src/metaobjects/config/source_resolver.py | 30 ++++++++++++++----- .../tests/config/test_neutral_config.py | 8 +++-- .../tests/config/test_source_resolver.py | 17 +++++++++++ 5 files changed, 79 insertions(+), 10 deletions(-) diff --git a/fixtures/source-resolution-conformance/README.md b/fixtures/source-resolution-conformance/README.md index 5c571219f..fb5011a76 100644 --- a/fixtures/source-resolution-conformance/README.md +++ b/fixtures/source-resolution-conformance/README.md @@ -65,6 +65,18 @@ README.md `ERR_COLLECTION_NOT_FOUND`. - **`resource` and `package` kinds are declared but resolve nowhere yet:** `ERR_SOURCE_KIND_UNSUPPORTED`. +- **Kind validation precedes path resolution, and that precedence is + order-independent.** Every spec's kind is checked across the WHOLE + declared list before any spec's path is touched on disk — so a multi-spec + list containing both an unsupported kind and a path that does not exist + always fails `ERR_SOURCE_KIND_UNSUPPORTED`, in EITHER declaration order, + never `ERR_SOURCE_UNRESOLVED`. See + `unsupported-kind-precedes-unresolved-path-when-path-is-declared-first` + and its `-declared-second` sibling — a port that interleaves the two + checks one spec at a time (kind-check-then-stat, per spec, in a single + loop) reports whichever error comes first in declaration order instead, + diverging on exactly one of the two cases depending on which order it + happens to process first. - **Unknown top-level config keys are IGNORED.** The file carries TypeScript-owned keys no other port models. `schema_version` and `sources` are the neutral subset; each port validates those strictly and ignores the rest. diff --git a/fixtures/source-resolution-conformance/cases.json b/fixtures/source-resolution-conformance/cases.json index 7086e048f..9815a1f5f 100644 --- a/fixtures/source-resolution-conformance/cases.json +++ b/fixtures/source-resolution-conformance/cases.json @@ -147,6 +147,28 @@ "config": { "schema_version": 1, "sources": [{ "package": "acme-model" }] }, "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" }, + { + "name": "unsupported-kind-precedes-unresolved-path-when-path-is-declared-first", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "nope" }, { "resource": "com/acme/model" }] + }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "unsupported-kind-precedes-unresolved-path-when-path-is-declared-second", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "resource": "com/acme/model" }, { "path": "nope" }] + }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, { "name": "no-config-and-no-default-directory-is-an-error", "tree": { diff --git a/server/python/src/metaobjects/config/source_resolver.py b/server/python/src/metaobjects/config/source_resolver.py index e6cc42f1c..0bfbc7b68 100644 --- a/server/python/src/metaobjects/config/source_resolver.py +++ b/server/python/src/metaobjects/config/source_resolver.py @@ -22,14 +22,17 @@ def _list_metadata_files(directory: Path) -> list[Path]: ) -def resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path]: - """Resolve a declared source SET to a de-duplicated list of metadata files. - - A relative ``path`` resolves against ``config_dir`` — the directory HOLDING - the ``.metaobjects/`` folder — never against the process working directory. +def _validate_kinds(specs: list[dict[str, str]]) -> None: + """Validate every spec's kind before ANY filesystem access. + + Mirrors `sources.ts`'s `orderedPathSpecs` (`.map(toPathSpec)` runs over the + whole list before `resolveSources` performs a single `stat()`): a kind + check interleaved with resolution, one spec at a time, would make which + error code comes back depend on declaration order — `{"path": "nope"}, + {"resource": "x"}` and its reverse must both report + `ERR_SOURCE_KIND_UNSUPPORTED`, never `ERR_SOURCE_UNRESOLVED` on one + ordering and the kind error on the other. """ - seen: dict[Path, None] = {} - for spec in specs: if "path" not in spec: kind = next(iter(spec), "") @@ -38,6 +41,19 @@ def resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path] code=ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED, ) + +def resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path]: + """Resolve a declared source SET to a de-duplicated list of metadata files. + + A relative ``path`` resolves against ``config_dir`` — the directory HOLDING + the ``.metaobjects/`` folder — never against the process working directory. + """ + # Whole-list kind validation FIRST — see `_validate_kinds`. + _validate_kinds(specs) + + seen: dict[Path, None] = {} + + for spec in specs: raw = Path(spec["path"]) target = raw if raw.is_absolute() else (config_dir / raw) diff --git a/server/python/tests/config/test_neutral_config.py b/server/python/tests/config/test_neutral_config.py index 19fddbcd2..2ca5c3599 100644 --- a/server/python/tests/config/test_neutral_config.py +++ b/server/python/tests/config/test_neutral_config.py @@ -3,7 +3,7 @@ from pathlib import Path from metaobjects.config.neutral_config import read_neutral_config -from metaobjects.errors import ParseError +from metaobjects.errors import ErrorCode, ParseError def _write_config(root: Path, payload: object) -> None: @@ -53,11 +53,13 @@ def test_malformed_json_raises_not_none(tmp_path: Path) -> None: d.mkdir(parents=True) (d / "config.json").write_text("{ not json") # A file that EXISTS but cannot be read must never look like no config at all. - with pytest.raises(ParseError): + with pytest.raises(ParseError) as e: read_neutral_config(tmp_path) + assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND def test_wrong_schema_version_raises(tmp_path: Path) -> None: _write_config(tmp_path, {"schema_version": 2, "sources": []}) - with pytest.raises(ParseError): + with pytest.raises(ParseError) as e: read_neutral_config(tmp_path) + assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND diff --git a/server/python/tests/config/test_source_resolver.py b/server/python/tests/config/test_source_resolver.py index a2987caef..00bab4bf8 100644 --- a/server/python/tests/config/test_source_resolver.py +++ b/server/python/tests/config/test_source_resolver.py @@ -48,6 +48,23 @@ def test_resource_kind_unsupported(tmp_path: Path) -> None: assert e.value.code == ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED +def test_kind_error_precedes_unresolved_path_regardless_of_order(tmp_path: Path) -> None: + # Kind validation runs across the WHOLE spec list before any filesystem + # access, so an unsupported-kind spec always wins over an unresolved-path + # spec — in EITHER declaration order. Interleaving the two checks one spec + # at a time (kind-check-then-stat, per spec) would make the reported error + # code depend on which spec happens to come first; pinned against the + # TypeScript reference (`sources.ts` `orderedPathSpecs`), verified + # empirically to raise ERR_SOURCE_KIND_UNSUPPORTED in both orders. + with pytest.raises(ParseError) as e_missing_first: + resolve_sources(tmp_path, [{"path": "nope"}, {"resource": "x"}]) + assert e_missing_first.value.code == ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED + + with pytest.raises(ParseError) as e_resource_first: + resolve_sources(tmp_path, [{"resource": "x"}, {"path": "nope"}]) + assert e_resource_first.value.code == ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED + + def test_collection_falls_back_to_default_dir(tmp_path: Path) -> None: (tmp_path / "metaobjects").mkdir() (tmp_path / "metaobjects" / "a.json").write_text("{}") From 86614858bd0cbb88d6e73042ee4e76252b36e9bc Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 10:33:25 -0400 Subject: [PATCH 22/44] feat(python): the neutral config is the fallback when nothing else names a location Ladder: explicit arg > metaobjects.config.yaml > .metaobjects/config.json > default dir, implemented as resolve_metadata_location() in cli.py. Gated by the shared source-resolution corpus (19 cases) plus two CLI-level tests: the neutral-config fallback itself, and a relative explicit arg (the plan's original ladder resolved a relative explicit path one directory too deep by joining it onto its own already-absolute parent; fixed by resolving the argument to an absolute path first). The conformance runner honors the corpus's resolveFrom key, invoking resolution from a subdirectory while still comparing expectFiles against the project root. Also documents in the corpus README that the specific error code for a malformed .metaobjects/config.json is deliberately not part of the cross-port contract (TypeScript's collection.ts has no try/catch there and emits no MetaObjects code at all), same as file order. --- .../source-resolution-conformance/README.md | 13 +++ server/python/src/metaobjects/cli.py | 40 +++++++ .../test_source_resolution_conformance.py | 100 ++++++++++++++++++ 3 files changed, 153 insertions(+) create mode 100644 server/python/tests/conformance/test_source_resolution_conformance.py diff --git a/fixtures/source-resolution-conformance/README.md b/fixtures/source-resolution-conformance/README.md index fb5011a76..cac2b363f 100644 --- a/fixtures/source-resolution-conformance/README.md +++ b/fixtures/source-resolution-conformance/README.md @@ -95,6 +95,19 @@ order anyway. A port MAY have a stable internal order — several do, and their own generated output depends on it. It just is not a cross-port promise. +## Also deliberately NOT pinned: the malformed-config error code + +The corpus has no case for a `.metaobjects/config.json` that exists but fails +to parse (bad JSON, an unsupported `schema_version`, a malformed `sources` +shape). The contract is only that resolution MUST raise rather than silently +degrade to "no config" — which error code it raises with is left to each +port. The reference implementation is why: `collection.ts:129-140` has no +try/catch around config loading and lets the raw zod/JSON error propagate, so +TypeScript emits no MetaObjects error code here at all. Pinning a shared code +across ports would mean changing the reference, which this corpus does not +do. Python raises `ERR_COLLECTION_NOT_FOUND`; C# and Java may each choose a +different code for the same failure, same as file order above. + ## Behavioral contract Each port's runner reads `cases.json`, and for every case: materializes `tree` diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index 02ea2418f..b078a7675 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -407,6 +407,46 @@ def gen_state_dir_for(metadata_dir: str) -> str: return str(Path(metadata_dir).resolve().parent / ".metaobjects" / ".gen-state") +def resolve_metadata_location( + explicit: str | None, + config: ProjectConfig | None, + root: Path, +) -> list[str]: + """The precedence ladder for where metadata lives. First match wins. + + 1. An explicit CLI argument (the positional ``metadata_dir``). + 2. This port's native surface — ``metadata`` in ``metaobjects.config.yaml``. + 3. ``sources`` in the port-neutral ``.metaobjects/config.json``. + 4. The built-in default directory. + + A file that EXISTS at any rung but is malformed raises rather than falling + through to the next rung. See + `docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §5. + """ + from metaobjects.config.source_resolver import resolve_collection, resolve_sources + + if explicit is not None: + # Resolve to an absolute path FIRST and pass it as an absolute spec — + # `resolve_sources` takes an absolute `path` as-is, so the base is + # irrelevant. Joining a relative `explicit` onto its own already- + # absolute parent (the naive approach) resolves one level too deep. + return [ + str(p) + for p in resolve_sources(root, [{"path": str(Path(explicit).resolve())}]) + ] + + if config is not None: + # `config.metadata_dir()` is already resolved to an absolute path + # (`ProjectConfig._resolve_under`), so the base passed here is + # likewise irrelevant. + return [ + str(p) for p in resolve_sources(root, [{"path": config.metadata_dir()}]) + ] + + # Rungs 3 and 4 both live in `resolve_collection`. + return [str(p) for p in resolve_collection(root)] + + #: The default api-surface subdir (the cross-port contract's ``api/python``). _DOCS_DEFAULT_API_SUBDIR = "api/python" diff --git a/server/python/tests/conformance/test_source_resolution_conformance.py b/server/python/tests/conformance/test_source_resolution_conformance.py new file mode 100644 index 000000000..c31fdec6e --- /dev/null +++ b/server/python/tests/conformance/test_source_resolution_conformance.py @@ -0,0 +1,100 @@ +"""Runs the shared source-resolution corpus against this port. + +Reads `fixtures/source-resolution-conformance/cases.json` — the single +committed source of truth. There is no per-port fixture. +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from metaobjects.config.source_resolver import resolve_collection +from metaobjects.errors import ParseError + +_CORPUS = ( + Path(__file__).resolve().parents[4] + / "fixtures" + / "source-resolution-conformance" + / "cases.json" +) + +_CASES = json.loads(_CORPUS.read_text())["cases"] + + +def _materialize(case: dict, root: Path) -> Path: + """Materialize ``tree`` under ``root`` and ``config`` (when present) under the + directory named by ``resolveFrom`` (project root when absent). Returns the + directory resolution must be invoked against. + """ + for rel, content in case["tree"].items(): + p = root / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(content) + + resolve_from = root / case.get("resolveFrom", ".") + + if case["config"] is not None: + d = resolve_from / ".metaobjects" + d.mkdir(parents=True, exist_ok=True) + (d / "config.json").write_text(json.dumps(case["config"], indent=2)) + + return resolve_from + + +@pytest.mark.parametrize("case", _CASES, ids=[c["name"] for c in _CASES]) +def test_source_resolution_conformance(case: dict, tmp_path: Path) -> None: + resolve_from = _materialize(case, tmp_path) + + if "expectError" in case: + with pytest.raises(ParseError) as e: + resolve_collection(resolve_from) + assert e.value.code.value == case["expectError"] + return + + # `expectFiles` is project-root-relative even when `resolveFrom` points + # elsewhere — resolve against `tmp_path`, not `resolve_from`. + root = tmp_path.resolve() + got = {p.relative_to(root).as_posix() for p in resolve_collection(resolve_from)} + assert got == set(case["expectFiles"]) + + +def test_cli_falls_back_to_neutral_config(tmp_path: Path, monkeypatch) -> None: + """No positional metadata_dir and no YAML `metadata` key => neutral config wins.""" + (tmp_path / "model").mkdir() + (tmp_path / "model" / "meta.a.json").write_text('{"metadata.root":{"children":[]}}') + d = tmp_path / ".metaobjects" + d.mkdir() + (d / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + + from metaobjects.cli import resolve_metadata_location + + monkeypatch.chdir(tmp_path) + got = resolve_metadata_location(explicit=None, config=None, root=tmp_path) + assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in got} == { + "model/meta.a.json" + } + + +def test_explicit_relative_metadata_dir_resolves_against_cwd( + tmp_path: Path, monkeypatch +) -> None: + """A RELATIVE explicit must not resolve one level too deep. + + Regression for the plan's original defect: `resolve_sources(Path(explicit) + .resolve().parent, [{"path": explicit}])` joins an already-absolute base + with a still-relative spec, walking one directory too far up. + """ + (tmp_path / "model").mkdir() + (tmp_path / "model" / "meta.a.json").write_text('{"metadata.root":{"children":[]}}') + + from metaobjects.cli import resolve_metadata_location + + monkeypatch.chdir(tmp_path) + got = resolve_metadata_location(explicit="model", config=None, root=tmp_path) + assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in got} == { + "model/meta.a.json" + } From cbcae4f130ac68c1af226a0e64c5470ccb9d5868 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 10:51:20 -0400 Subject: [PATCH 23/44] fix(python): wire the source-resolution ladder into gen and verify --codegen Fix round 1/5, two Critical findings. 1. resolve_metadata_location() existed and was unit-tested but no command handler called it, so `metaobjects gen`/`verify --codegen` still errored on no positional + no metaobjects.config.yaml instead of falling back to .metaobjects/config.json. _cmd_gen_config and _verify_codegen_config now route that case to new _cmd_gen_neutral_fallback / _verify_codegen_neutral_fallback helpers (added _load_root_from_paths, loading via MetaDataLoader.from_uris since the ladder can resolve to several directories or individual files that a single from_directory call can't express). --out is required at this rung, exactly as it is in explicit- flag mode; the two byte- identical rungs (explicit arg, metaobjects.config.yaml) are untouched. Extracted _diff_report() out of _verify_codegen so both entry points report drift identically. Added three end-to-end tests driving `main()` through the .metaobjects/config.json and bare-default-directory rungs for both gen and verify --codegen. 2. test_explicit_relative_metadata_dir_resolves_against_cwd used a single- segment relative argument ("model"), which cannot distinguish the fixed ladder from the original defect: Path("model").resolve().parent happens to land back at the project root, so the extra join silently cancels out. Changed to a multi-segment path ("sub/model"), which the buggy formulation resolves to a nonexistent .../sub/sub/model. Verified both directions by temporarily reinstating the buggy formulation (confirmed FAIL) and restoring the fix (confirmed PASS) before committing. --- server/python/src/metaobjects/cli.py | 202 +++++++++++++++--- .../tests/codegen/test_cli_config_gen.py | 42 ++++ .../tests/codegen/test_cli_config_verify.py | 35 +++ .../test_source_resolution_conformance.py | 18 +- 4 files changed, 260 insertions(+), 37 deletions(-) diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index b078a7675..9af17f956 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -47,6 +47,7 @@ from pathlib import Path from metaobjects import MetaDataLoader +from metaobjects.errors import ParseError from metaobjects.agent_context import ( AGENT_CONTEXT_MANIFEST_PATH, agent_context_staleness, @@ -274,6 +275,35 @@ def _load_root( return result.root, [] +def _load_root_from_paths( + paths: list[str], + strict: bool = False, + providers: list[object] | None = None, +) -> tuple[MetaData | None, list[str]]: + """Load metadata from an explicit file list rather than a single directory. + + The source-resolution ladder's ``.metaobjects/config.json`` rung + (:func:`resolve_metadata_location`) can resolve to several directories or + individual files, which a single ``from_directory`` call cannot express — + so this loads each resolved file as its own ``file://`` source via + :meth:`MetaDataLoader.from_uris`. Mirrors :func:`_load_root`'s ``strict``/ + ``providers`` contract exactly. + """ + uris = [Path(p).resolve().as_uri() for p in paths] + if providers: + from metaobjects.core_types import core_providers + + result = MetaDataLoader.from_uris( + uris, providers=[*core_providers, *providers], strict=strict + ) + else: + result = MetaDataLoader.from_uris(uris, strict=strict) + if result.errors: + msgs = [f"{e.code}: {e.message}" for e in result.errors] + return None, msgs + return result.root, [] + + def _strict_load_hint() -> str: """Actionable next-steps when strict verify rejects an undeclared @attr.""" return ( @@ -692,22 +722,73 @@ def _run_gen_targets( return all_written, errors +def _cmd_gen_neutral_fallback(args: argparse.Namespace) -> int: + """``gen`` with no positional ``metadata_dir`` AND no ``metaobjects.config.yaml``. + + Descends the source-resolution ladder's remaining rungs — ``sources`` in + ``.metaobjects/config.json``, else the built-in default directory — via + :func:`resolve_metadata_location`. There is no declarative target registry + at this rung (that is what a ``metaobjects.config.yaml`` provides), so + ``--out`` is required exactly as it is in explicit-```` flag + mode; this function otherwise mirrors ``_cmd_gen``'s flag-mode body, minus + the ``--template-spec`` pass (out of scope for this rung). + """ + if args.out is None: + print( + "error: gen requires and --out " + "(or a metaobjects.config.yaml / --list).", + file=sys.stderr, + ) + return 2 + + root_dir = Path.cwd() + try: + paths = resolve_metadata_location(explicit=None, config=None, root=root_dir) + except ParseError as exc: + print(f"error: could not resolve metadata location: {exc}", file=sys.stderr) + return 1 + + generators: list[Generator] | None = None + if args.generators: + generators, gen_errors = _resolve_generators(args.generators) + if gen_errors: + print("error: invalid --generators selection:", file=sys.stderr) + for msg in gen_errors: + print(f" {msg}", file=sys.stderr) + return 1 + + entities = _parse_entities(getattr(args, "entities", None)) + providers, providers_ok = _providers_from_args(args) + if not providers_ok: + return 1 + + root, load_errors = _load_root_from_paths(paths, providers=providers) + if root is None: + print("error: failed to load metadata:", file=sys.stderr) + for msg in load_errors: + print(f" {msg}", file=sys.stderr) + return 1 + + gen_state = str(root_dir.resolve() / ".metaobjects" / ".gen-state") + written = _run_suite(root, args.out, generators, entities, gen_state_dir=gen_state) + for path in written: + print(path) + print(f"metaobjects gen: wrote {len(written)} file(s) to {args.out}") + return 0 + + def _cmd_gen_config(args: argparse.Namespace) -> int: """``gen`` with no positional ``metadata_dir`` → declarative-config mode (#267). Load ``metaobjects.config.yaml``, load metadata ONCE, and run every target (or ``--target``) into its own ``outDir`` with a cross-target duplicate-path guard. Providers resolve relative to the config file (no ``PYTHONPATH=``). + No ``metaobjects.config.yaml`` at all → :func:`_cmd_gen_neutral_fallback` + (the ladder's remaining rungs), not an error. """ config_path = _find_config(args) if config_path is None: - print( - "error: no given and no metaobjects.config.yaml found. " - "Either pass --out (flag mode) or create a " - "metaobjects.config.yaml (or pass --config ).", - file=sys.stderr, - ) - return 2 + return _cmd_gen_neutral_fallback(args) try: config = load_project_config(config_path) except ConfigError as exc: @@ -761,6 +842,38 @@ def _relative_set(root: Path) -> dict[str, str]: return files +def _diff_report(expected: dict[str, str], committed: dict[str, str]) -> int: + """Compare a regenerated file map against the committed one; print the + standard ``verify --codegen`` drift report. Returns 0 (in sync) or 1. + + Extracted so the explicit-```` flag mode + (:func:`_verify_codegen`) and the ``.metaobjects/config.json`` fallback + rung (:func:`_verify_codegen_neutral_fallback`) report drift identically. + """ + changed = sorted( + k for k in expected if k in committed and expected[k] != committed[k] + ) + missing = sorted(k for k in expected if k not in committed) # not yet committed + extra = sorted(k for k in committed if k not in expected) # stale committed file + + if not changed and not missing and not extra: + print(f"metaobjects verify: in sync ({len(expected)} file(s)).") + return 0 + + print("error: generated code is out of sync with metadata.", file=sys.stderr) + for k in changed: + print(f" drifted: {k}", file=sys.stderr) + for k in missing: + print(f" missing: {k}", file=sys.stderr) + for k in extra: + print(f" extra: {k}", file=sys.stderr) + print( + "regenerate (metaobjects gen) and commit the result.", + file=sys.stderr, + ) + return 1 + + def _verify_codegen(args: argparse.Namespace) -> int: """``verify --codegen`` — regenerate to a temp dir + diff vs committed ``--out``. @@ -804,28 +917,54 @@ def _verify_codegen(args: argparse.Namespace) -> int: expected = _relative_set(Path(tmp)) committed = _relative_set(Path(args.out)) - changed = sorted( - k for k in expected if k in committed and expected[k] != committed[k] - ) - missing = sorted(k for k in expected if k not in committed) # not yet committed - extra = sorted(k for k in committed if k not in expected) # stale committed file + return _diff_report(expected, committed) - if not changed and not missing and not extra: - print(f"metaobjects verify: in sync ({len(expected)} file(s)).") - return 0 - print("error: generated code is out of sync with metadata.", file=sys.stderr) - for k in changed: - print(f" drifted: {k}", file=sys.stderr) - for k in missing: - print(f" missing: {k}", file=sys.stderr) - for k in extra: - print(f" extra: {k}", file=sys.stderr) - print( - "regenerate (metaobjects gen) and commit the result.", - file=sys.stderr, - ) - return 1 +def _verify_codegen_neutral_fallback(args: argparse.Namespace) -> int: + """``verify --codegen`` with no positional ``metadata_dir`` AND no + ``metaobjects.config.yaml``. + + Descends the source-resolution ladder's remaining rungs via + :func:`resolve_metadata_location`, exactly like + :func:`_cmd_gen_neutral_fallback`. There is no declarative target registry + at this rung, so ``--out`` (the committed dir to diff against) is required + exactly as it is in explicit-```` flag mode. + """ + if args.out is None: + print( + "error: verify --codegen requires --out (the committed output dir).", + file=sys.stderr, + ) + return 2 + + root_dir = Path.cwd() + try: + paths = resolve_metadata_location(explicit=None, config=None, root=root_dir) + except ParseError as exc: + print(f"error: could not resolve metadata location: {exc}", file=sys.stderr) + return 1 + + strict = not getattr(args, "lax", False) + providers, providers_ok = _providers_from_args(args) + if not providers_ok: + return 1 + + with tempfile.TemporaryDirectory() as tmp: + entities = _parse_entities(getattr(args, "entities", None)) + root, load_errors = _load_root_from_paths(paths, strict=strict, providers=providers) + if root is None: + print("error: failed to load metadata:", file=sys.stderr) + for msg in load_errors: + print(f" {msg}", file=sys.stderr) + if strict and any("ERR_UNKNOWN_ATTR" in m for m in load_errors): + print(_strict_load_hint(), file=sys.stderr) + return 1 + + _run_suite(root, tmp, None, entities, gen_state_dir=None) + expected = _relative_set(Path(tmp)) + committed = _relative_set(Path(args.out)) + + return _diff_report(expected, committed) def _temp_slot_for(temp_root: Path, real_outdir: str, config_dir: Path) -> str: @@ -860,15 +999,12 @@ def _verify_codegen_config(args: argparse.Namespace) -> int: drift (mirrors the TS ``computeCodegenDrift`` unit = unique outDir). ``--target`` widens to the outDir-sharing closure (an outDir is verified as a unit). Strict-by-default (ADR-0023) unless ``--lax``. + No ``metaobjects.config.yaml`` at all → :func:`_verify_codegen_neutral_fallback` + (the ladder's remaining rungs), not an error. """ config_path = _find_config(args) if config_path is None: - print( - "error: verify --codegen with no requires a " - "metaobjects.config.yaml (or --config ).", - file=sys.stderr, - ) - return 2 + return _verify_codegen_neutral_fallback(args) try: config = load_project_config(config_path) except ConfigError as exc: diff --git a/server/python/tests/codegen/test_cli_config_gen.py b/server/python/tests/codegen/test_cli_config_gen.py index fbba3572a..26f9f8078 100644 --- a/server/python/tests/codegen/test_cli_config_gen.py +++ b/server/python/tests/codegen/test_cli_config_gen.py @@ -222,3 +222,45 @@ def test_gen_flag_path_ignores_config_when_present(tmp_path: Path) -> None: rc = main(["gen", str(meta), "--out", str(out)]) assert rc == 0 assert (out / "Program.py").exists() + + +def test_gen_no_args_no_yaml_falls_back_to_neutral_config( + tmp_path: Path, monkeypatch +) -> None: + """No positional AND no metaobjects.config.yaml anywhere: + `gen` must descend to the `.metaobjects/config.json` `sources` rung + (source-resolution ladder rung 3) rather than erroring, per fix round 1. + + The metadata lives under `model/`, NOT the built-in default `metaobjects/` + directory — so this only passes if the declared `sources` path is actually + consulted, not a coincidental default-directory hit. + """ + model = tmp_path / "model" + model.mkdir() + (model / "meta.fitness.json").write_text(FITNESS.read_text()) + d = tmp_path / ".metaobjects" + d.mkdir() + (d / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + + monkeypatch.chdir(tmp_path) + rc = main(["gen", "--out", "gen/models", "--generators", "entity"]) + assert rc == 0 + assert (tmp_path / "gen/models/Program.py").exists() + + +def test_gen_no_args_no_config_at_all_falls_back_to_default_directory( + tmp_path: Path, monkeypatch +) -> None: + """No positional , no metaobjects.config.yaml, AND no + `.metaobjects/config.json`: `gen` descends all the way to the ladder's + built-in default directory (`metaobjects/`) rather than erroring.""" + meta = tmp_path / "metaobjects" + meta.mkdir() + (meta / "meta.fitness.json").write_text(FITNESS.read_text()) + + monkeypatch.chdir(tmp_path) + rc = main(["gen", "--out", "gen/models", "--generators", "entity"]) + assert rc == 0 + assert (tmp_path / "gen/models/Program.py").exists() diff --git a/server/python/tests/codegen/test_cli_config_verify.py b/server/python/tests/codegen/test_cli_config_verify.py index 6b7acb817..b427ba627 100644 --- a/server/python/tests/codegen/test_cli_config_verify.py +++ b/server/python/tests/codegen/test_cli_config_verify.py @@ -152,3 +152,38 @@ def test_verify_target_scoping_widens_to_shared_outdir(tmp_path: Path, capsys) - rc = main(["verify", "--codegen", "--config", str(cfg), "--target", "a"]) assert rc == 1 assert "Week.py" in capsys.readouterr().err + + +def test_verify_codegen_no_args_no_yaml_falls_back_to_neutral_config( + tmp_path: Path, monkeypatch +) -> None: + """No positional AND no metaobjects.config.yaml anywhere: + `verify --codegen` must descend to the `.metaobjects/config.json` `sources` + rung rather than erroring, per fix round 1 — and must agree with `gen` + taking the SAME rung (a fresh gen is in sync; drift is still caught). + + The metadata lives under `model/`, NOT the built-in default `metaobjects/` + directory, so this only passes if the declared `sources` path is actually + consulted by BOTH commands. + """ + model = tmp_path / "model" + model.mkdir() + (model / "meta.fitness.json").write_text(FITNESS.read_text()) + d = tmp_path / ".metaobjects" + d.mkdir() + (d / "config.json").write_text( + '{"schema_version": 1, "sources": [{"path": "model"}]}' + ) + + monkeypatch.chdir(tmp_path) + # No --generators here: `verify --codegen` has no such flag (it always + # regenerates the full default suite), so `gen` must run the full suite + # too or the diff reports the un-emitted generators as spurious drift — + # same constraint the flag-mode docstring notes for --entities. + assert main(["gen", "--out", "gen/models"]) == 0 + # Fresh gen -> no drift, via the same fallback rung. + assert main(["verify", "--codegen", "--out", "gen/models"]) == 0 + # Drift the committed output; the fallback rung must still catch it. + program = tmp_path / "gen/models/Program.py" + program.write_text(program.read_text() + "\n# drift\n") + assert main(["verify", "--codegen", "--out", "gen/models"]) == 1 diff --git a/server/python/tests/conformance/test_source_resolution_conformance.py b/server/python/tests/conformance/test_source_resolution_conformance.py index c31fdec6e..3e8f1b319 100644 --- a/server/python/tests/conformance/test_source_resolution_conformance.py +++ b/server/python/tests/conformance/test_source_resolution_conformance.py @@ -87,14 +87,24 @@ def test_explicit_relative_metadata_dir_resolves_against_cwd( Regression for the plan's original defect: `resolve_sources(Path(explicit) .resolve().parent, [{"path": explicit}])` joins an already-absolute base with a still-relative spec, walking one directory too far up. + + A SINGLE-segment argument (e.g. ``"model"``) cannot distinguish the buggy + formulation from the fixed one: ``Path("model").resolve().parent`` happens + to land back at the project root, so the extra join silently cancels out. + The defect only shows up with a MULTI-segment relative path (``"sub/model"``) + — the buggy form resolves the base to ``.../sub`` and then joins the still- + relative ``"sub/model"`` onto it, landing on ``.../sub/sub/model`` (does not + exist -> ERR_SOURCE_UNRESOLVED) instead of ``.../sub/model``. """ - (tmp_path / "model").mkdir() - (tmp_path / "model" / "meta.a.json").write_text('{"metadata.root":{"children":[]}}') + (tmp_path / "sub" / "model").mkdir(parents=True) + (tmp_path / "sub" / "model" / "meta.a.json").write_text( + '{"metadata.root":{"children":[]}}' + ) from metaobjects.cli import resolve_metadata_location monkeypatch.chdir(tmp_path) - got = resolve_metadata_location(explicit="model", config=None, root=tmp_path) + got = resolve_metadata_location(explicit="sub/model", config=None, root=tmp_path) assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in got} == { - "model/meta.a.json" + "sub/model/meta.a.json" } From cb5cbc8bdcc9920a18056112d039d2e6c2fb9f02 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 11:12:27 -0400 Subject: [PATCH 24/44] feat(csharp): read the port-neutral sources key Adds MetaObjects.Config.NeutralConfig (the schema_version + sources neutral subset of .metaobjects/config.json, ignoring every TS-owned key) and SourceResolver (declared-source-set -> deduplicated file list, plus the full default-directory ladder), gated by the shared 19-case fixtures/source-resolution-conformance/cases.json corpus. Kind validation runs in two explicit passes across the whole declared set before any filesystem access, matching sources.ts's orderedPathSpecs - interleaving validate-then-resolve per spec would make which error fires depend on declaration order, which two corpus cases pin against. Directory expansion reuses the loader's own DirectorySource rather than a second extension-filter/sort implementation. The CLI's positional becomes optional on gen/docs/verify, falling back through the same resolver: a single declared source hands the loader that source's own root (never the literal default name), and more than one declared source is refused outright rather than silently picked from - this port's loader takes one directory, not a source set. Reachability is proven by spawning the built CLI assembly as a subprocess rather than only unit-testing the resolver function, since a resolver nothing calls is invisible to a normal test run. Co-Authored-By: Claude Opus 5 (1M context) --- .../MetadataDirFallbackTests.cs | 154 ++++++++++++++++++ server/csharp/MetaObjects.Cli/Program.cs | 83 +++++++++- .../SourceResolutionConformanceTests.cs | 126 ++++++++++++++ .../MetaObjects/Config/NeutralConfig.cs | 93 +++++++++++ .../MetaObjects/Config/SourceResolver.cs | 110 +++++++++++++ 5 files changed, 559 insertions(+), 7 deletions(-) create mode 100644 server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs create mode 100644 server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs create mode 100644 server/csharp/MetaObjects/Config/NeutralConfig.cs create mode 100644 server/csharp/MetaObjects/Config/SourceResolver.cs diff --git a/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs b/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs new file mode 100644 index 000000000..13a53dd7c --- /dev/null +++ b/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs @@ -0,0 +1,154 @@ +using System.Diagnostics; +using Xunit; + +namespace MetaObjects.Cli.Tests; + +///

+/// Proves the <metadataDir>-optional fallback (Program.cs's +/// ResolveMetadataDirOrExit) is actually WIRED into dotnet meta gen, +/// not merely defined and unit-tested in isolation. A resolver function that +/// nothing calls is exactly the gap the Python port shipped first (see the +/// SourceResolutionConformanceTests header + task-4-report.md) — the only way to +/// catch that class of bug is to drive the real command entry point (the built +/// CLI assembly, invoked as a subprocess) rather than calling a helper method +/// directly, since a top-level-statement local function compiles to a name this +/// test assembly cannot reference at all. +/// +public sealed class MetadataDirFallbackTests : IDisposable +{ + private readonly string _tmp = Path.Combine(Path.GetTempPath(), "meta-cli-fallback-" + Guid.NewGuid().ToString("N")); + + private const string Metadata = """ + { "metadata.root": { "package": "acme", "children": [ + { "object.entity": { "name": "Subscriber", "children": [ + { "source.rdb": { "@table": "subscribers" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "email", "@required": true } }, + { "identity.primary": { "@fields": "id" } } + ]}} + ]}} + """; + + public void Dispose() { try { Directory.Delete(_tmp, recursive: true); } catch { } } + + [Fact] + public void Gen_with_no_positional_metadataDir_resolves_the_declared_source_and_generates() + { + var modelDir = Path.Combine(_tmp, "model"); + Directory.CreateDirectory(modelDir); + File.WriteAllText(Path.Combine(modelDir, "meta.acme.json"), Metadata); + var cfgDir = Path.Combine(_tmp, ".metaobjects"); + Directory.CreateDirectory(cfgDir); + File.WriteAllText( + Path.Combine(cfgDir, "config.json"), + """{ "schema_version": 1, "sources": [ { "path": "model" } ] }"""); + + var outDir = Path.Combine(_tmp, "generated"); + var (exitCode, stdout, stderr) = RunCli(_tmp, "gen", "--out", outDir, "--namespace", "Acme.Generated"); + + Assert.True(exitCode == 0, $"exit={exitCode}\nstdout={stdout}\nstderr={stderr}"); + Assert.True(File.Exists(Path.Combine(outDir, "Subscriber.g.cs")), stdout + stderr); + } + + [Fact] + public void Gen_with_no_positional_metadataDir_and_nothing_to_resolve_reports_the_ladders_own_error() + { + // Before the ladder was wired, an omitted positional always produced the + // generic "usage: ..." 2-exit, regardless of what (if anything) the cwd + // contained. Once wired, an empty project reaches SourceResolver and + // fails with ITS OWN diagnostic — proof the omitted case is actually + // being routed through resolution rather than still failing the old way. + Directory.CreateDirectory(_tmp); + + var (exitCode, _, stderr) = RunCli(_tmp, "gen", "--out", Path.Combine(_tmp, "generated"), "--namespace", "X"); + + Assert.Equal(2, exitCode); + Assert.Contains("ERR_COLLECTION_NOT_FOUND", stderr); + } + + [Fact] + public void Gen_with_no_positional_metadataDir_and_multiple_declared_sources_refuses_rather_than_picking_one() + { + var aDir = Path.Combine(_tmp, "a"); + var bDir = Path.Combine(_tmp, "b"); + Directory.CreateDirectory(aDir); + Directory.CreateDirectory(bDir); + File.WriteAllText(Path.Combine(aDir, "meta.a.json"), """{ "metadata.root": { "children": [] } }"""); + File.WriteAllText(Path.Combine(bDir, "meta.b.json"), """{ "metadata.root": { "children": [] } }"""); + var cfgDir = Path.Combine(_tmp, ".metaobjects"); + Directory.CreateDirectory(cfgDir); + File.WriteAllText( + Path.Combine(cfgDir, "config.json"), + """{ "schema_version": 1, "sources": [ { "path": "a" }, { "path": "b" } ] }"""); + + var outDir = Path.Combine(_tmp, "generated"); + var (exitCode, _, stderr) = RunCli(_tmp, "gen", "--out", outDir, "--namespace", "X"); + + Assert.Equal(2, exitCode); + Assert.Contains("2 metadata sources", stderr); + Assert.False(Directory.Exists(outDir), "must not silently generate from just one of several declared sources"); + } + + [Fact] + public void Gen_with_an_explicit_positional_metadataDir_is_unaffected() + { + // The explicit-argument path must stay byte-identical: no .metaobjects/ + // config.json in play at all, yet generation still succeeds because the + // ladder is never consulted when the caller already named a directory. + var modelDir = Path.Combine(_tmp, "model"); + Directory.CreateDirectory(modelDir); + File.WriteAllText(Path.Combine(modelDir, "meta.acme.json"), Metadata); + + var outDir = Path.Combine(_tmp, "generated"); + var (exitCode, stdout, stderr) = RunCli(_tmp, "gen", modelDir, "--out", outDir, "--namespace", "Acme.Generated"); + + Assert.True(exitCode == 0, $"exit={exitCode}\nstdout={stdout}\nstderr={stderr}"); + Assert.True(File.Exists(Path.Combine(outDir, "Subscriber.g.cs")), stdout + stderr); + } + + /// Runs the actual built `dotnet meta` assembly as a subprocess, cwd + /// pinned to , so the test exercises Program.cs's + /// real Main/argument-parsing rather than any method reachable in-process. + private static (int ExitCode, string Stdout, string Stderr) RunCli(string workingDir, params string[] args) + { + var psi = new ProcessStartInfo("dotnet") + { + WorkingDirectory = workingDir, + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + }; + psi.ArgumentList.Add(ResolveCliDll()); + foreach (var a in args) psi.ArgumentList.Add(a); + + using var proc = Process.Start(psi) ?? throw new InvalidOperationException("failed to start dotnet"); + var stdout = proc.StandardOutput.ReadToEnd(); + var stderr = proc.StandardError.ReadToEnd(); + proc.WaitForExit(); + return (proc.ExitCode, stdout, stderr); + } + + /// Locates the MetaObjects.Cli build output next to this test + /// assembly's own build output — both projects share the same Configuration + /// and TargetFramework (net8.0), and MetaObjects.Cli.Tests already builds + /// MetaObjects.Cli as a project reference, so the dll is guaranteed present + /// by the time `dotnet test` starts running tests. + private static string ResolveCliDll() + { + var testsProjectDir = new DirectoryInfo(AppContext.BaseDirectory); + while (testsProjectDir is not null && + !File.Exists(Path.Combine(testsProjectDir.FullName, "MetaObjects.Cli.Tests.csproj"))) + testsProjectDir = testsProjectDir.Parent; + if (testsProjectDir is null) + throw new InvalidOperationException( + "could not locate MetaObjects.Cli.Tests.csproj by walking up from " + AppContext.BaseDirectory); + + var relSuffix = Path.GetRelativePath(testsProjectDir.FullName, AppContext.BaseDirectory); + var dll = Path.Combine(testsProjectDir.Parent!.FullName, "MetaObjects.Cli", relSuffix, "MetaObjects.Cli.dll"); + if (!File.Exists(dll)) + throw new FileNotFoundException( + $"expected the MetaObjects.Cli build output at {dll} (built automatically as a project " + + "reference of MetaObjects.Cli.Tests)", dll); + return dll; + } +} diff --git a/server/csharp/MetaObjects.Cli/Program.cs b/server/csharp/MetaObjects.Cli/Program.cs index bad143a68..c81df628f 100644 --- a/server/csharp/MetaObjects.Cli/Program.cs +++ b/server/csharp/MetaObjects.Cli/Program.cs @@ -73,7 +73,11 @@ static int RunGen(string[] rest) return 0; } - if (metadataDir is null || outDir is null) + // Rung 1 (explicit positional) is honored as-is; an omitted metadataDir + // falls back to the port-neutral .metaobjects/config.json ladder. + metadataDir = ResolveMetadataDirOrExit(metadataDir); + + if (outDir is null) { Console.Error.WriteLine("usage: dotnet meta gen --out [--namespace ] [--generators ] [--template-root ] [--template-spec ] [--emit-abstract-shapes]"); Console.Error.WriteLine(" dotnet meta gen --list"); @@ -121,7 +125,11 @@ static int RunDocs(string[] rest) else if (!rest[i].StartsWith('-')) metadataDir ??= rest[i]; } - if (metadataDir is null || outDir is null) + // Rung 1 (explicit positional) is honored as-is; an omitted metadataDir + // falls back to the port-neutral .metaobjects/config.json ladder. + metadataDir = ResolveMetadataDirOrExit(metadataDir); + + if (outDir is null) { Console.Error.WriteLine("usage: dotnet meta docs --out [--namespace ] [--project ] [--model-base-url ]"); return 2; @@ -149,6 +157,69 @@ static int RunDocs(string[] rest) return 0; } +// The metadata-location ladder's rungs 3-4 (source-resolution design doc §3): +// rung 1 is the explicit positional argument the caller already tried; rung 2 +// (a port-native config surface) doesn't exist in C#; rungs 3 (a declared +// `sources` in .metaobjects/config.json) and 4 (the default "metaobjects" +// directory) live in MetaObjects.Config.SourceResolver.ResolveCollection, +// which this wraps. Called from all three metadataDir-taking commands (gen, +// docs, verify) so an omitted positional argument is never a hard requirement +// wherever a project's config can name the location instead. +// +// Never returns null: either hands back a real directory, or prints a +// diagnostic and terminates the process — callers may treat the result as +// always-present and keep their existing (now-unreachable-when-omitted) +// null checks for the OTHER positional/option they still require. +static string ResolveMetadataDirOrExit(string? metadataDir) +{ + if (metadataDir is not null) return metadataDir; + + var cwd = Directory.GetCurrentDirectory(); + try + { + var cfg = MetaObjects.Config.NeutralConfig.Read(cwd); + var specs = cfg?.Sources ?? Array.Empty>(); + + if (specs.Count == 0) + { + // No declared sources — validate + apply the DEFAULT directory through + // the same ladder the shared conformance corpus gates (raises + // ERR_COLLECTION_NOT_FOUND when the default is also absent). + _ = MetaObjects.Config.SourceResolver.ResolveCollection(cwd); + return Path.Combine(cwd, MetaObjects.Config.NeutralConfig.DefaultMetadataDir); + } + + if (specs.Count > 1) + { + // MetaDataLoader.FromDirectory takes ONE directory — it cannot express a + // multi-source SET. Fail loudly rather than silently loading just one of + // the declared sources; MetaDataLoader.Load(IReadOnlyList) + // (MetaDataLoader.cs:334) is the documented follow-up that lifts this. + Console.Error.WriteLine( + $"error: {cwd}: .metaobjects/config.json declares {specs.Count} metadata sources, but " + + "this CLI's loader accepts only one directory at a time. Pass explicitly, " + + "or reduce \"sources\" to a single entry."); + Environment.Exit(2); + throw new InvalidOperationException("unreachable"); + } + + // Exactly one declared source. Resolve + validate it through the same + // kind/existence checks ResolveSources applies (ERR_SOURCE_KIND_UNSUPPORTED / + // ERR_SOURCE_UNRESOLVED), then hand the loader that spec's OWN root — never + // the default directory name, which this project may not even have. + MetaObjects.Config.SourceResolver.ResolveSources(cwd, specs); + var rawPath = specs[0]["path"]; // guaranteed present: ResolveSources above + // would already have thrown otherwise. + return Path.IsPathRooted(rawPath) ? rawPath : Path.GetFullPath(Path.Combine(cwd, rawPath)); + } + catch (MetaObjects.MetaModelException e) + { + Console.Error.WriteLine($"error: {e.Code}: {e.Message}"); + Environment.Exit(2); + throw; + } +} + static int Unknown(string cmd) { Console.Error.WriteLine($"dotnet meta: unknown command \"{cmd}\""); @@ -208,11 +279,9 @@ static int RunVerify(string[] rest) else templatesRoot ??= a; } - if (metadataDir is null) - { - Console.Error.WriteLine("usage: dotnet meta verify [--templates ] [--codegen --out [--namespace ]] [--db]"); - return 2; - } + // Rung 1 (explicit positional) is honored as-is; an omitted metadataDir + // falls back to the port-neutral .metaobjects/config.json ladder. + metadataDir = ResolveMetadataDirOrExit(metadataDir); // The templates gate needs a root. Bare verify (defaults to templates) and an // explicit --templates both require it; surface a clear usage error if absent. diff --git a/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs b/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs new file mode 100644 index 000000000..cc3578443 --- /dev/null +++ b/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs @@ -0,0 +1,126 @@ +// Runs the shared source-resolution corpus against this port. Reads the single +// committed fixtures/source-resolution-conformance/cases.json — no per-port +// fixture. The corpus is the contract; see server/typescript/packages/sdk/src/ +// sources.ts + collection.ts for the authoritative behavior each case pins. +using System.Text.Json; +using MetaObjects; +using MetaObjects.Config; +using Xunit; + +namespace MetaObjects.Conformance.Tests; + +public class SourceResolutionConformanceTests +{ + private sealed record Case( + string Name, + Dictionary Tree, + JsonElement? Config, + // Project-root-relative directory the resolver is invoked FROM. Defaults + // to "." — 18 of 19 cases leave it there, so the config lives at the + // project root and "relative to project root" vs "relative to the + // invocation directory" coincide. The one case that sets it + // ("a-parent-relative-path-resolves-against-the-declaring-configs- + // directory") is the one place those two bases diverge, and both the + // config's own location AND the expectFiles comparison base below must + // honor it correctly for that case to mean anything. + string ResolveFrom, + string[]? ExpectFiles, + string? ExpectError); + + public static TheoryData CaseNames() + { + var data = new TheoryData(); + foreach (var c in LoadCases()) data.Add(c.Name); + return data; + } + + private static string CorpusPath() + { + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null && !Directory.Exists(Path.Combine(dir.FullName, "fixtures"))) + dir = dir.Parent; + Assert.NotNull(dir); + return Path.Combine(dir!.FullName, "fixtures", "source-resolution-conformance", "cases.json"); + } + + private static List LoadCases() + { + using var doc = JsonDocument.Parse(File.ReadAllText(CorpusPath())); + var cases = new List(); + foreach (var el in doc.RootElement.GetProperty("cases").EnumerateArray()) + { + var tree = new Dictionary(); + foreach (var p in el.GetProperty("tree").EnumerateObject()) + tree[p.Name] = p.Value.GetString() ?? ""; + + var cfgEl = el.GetProperty("config"); + JsonElement? cfg = cfgEl.ValueKind == JsonValueKind.Null ? null : cfgEl.Clone(); + + var resolveFrom = el.TryGetProperty("resolveFrom", out var rf) ? rf.GetString()! : "."; + + string[]? expectFiles = el.TryGetProperty("expectFiles", out var ef) + ? ef.EnumerateArray().Select(x => x.GetString()!).ToArray() + : null; + string? expectError = el.TryGetProperty("expectError", out var ee) ? ee.GetString() : null; + + cases.Add(new Case(el.GetProperty("name").GetString()!, tree, cfg, resolveFrom, expectFiles, expectError)); + } + return cases; + } + + [Theory] + [MemberData(nameof(CaseNames))] + public void ResolvesTheSameFileSet(string name) + { + var c = LoadCases().Single(x => x.Name == name); + // `root` is the PROJECT ROOT — the base every `tree` path and every + // `expectFiles` entry is written relative to, regardless of `resolveFrom`. + var root = Path.Combine(Path.GetTempPath(), "mo-src-conf-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(root); + try + { + foreach (var (rel, content) in c.Tree) + { + var abs = Path.Combine(root, rel.Replace('/', Path.DirectorySeparatorChar)); + Directory.CreateDirectory(Path.GetDirectoryName(abs)!); + File.WriteAllText(abs, content); + } + + // The invocation directory: project root joined with `resolveFrom`. + // The config MUST be materialized here, not at the project root — a + // config placed at the project root while resolving FROM a + // subdirectory would go undetected by ResolveCollection there and + // fail loudly with ERR_COLLECTION_NOT_FOUND, which is what makes this + // half of the mistake self-catching. Getting it right on purpose (not + // by luck) is what this comment is pinning. + var invokeDir = Path.GetFullPath(Path.Combine(root, c.ResolveFrom)); + Directory.CreateDirectory(invokeDir); + if (c.Config is not null) + { + var d = Path.Combine(invokeDir, ".metaobjects"); + Directory.CreateDirectory(d); + File.WriteAllText(Path.Combine(d, "config.json"), c.Config.Value.GetRawText()); + } + + if (c.ExpectError is not null) + { + var ex = Assert.ThrowsAny(() => SourceResolver.ResolveCollection(invokeDir)); + Assert.Equal(c.ExpectError, ex.Code.ToString()); + return; + } + + // Compared against the PROJECT ROOT explicitly — never against + // `invokeDir`. For 18 of 19 cases the two coincide (resolveFrom "."), + // so a comparison base bug here would pass every case except the one + // that sets `resolveFrom`, which is exactly why that case exists. + var got = SourceResolver.ResolveCollection(invokeDir) + .Select(f => Path.GetRelativePath(root, f).Replace(Path.DirectorySeparatorChar, '/')) + .ToHashSet(); + Assert.Equal(c.ExpectFiles!.ToHashSet(), got); + } + finally + { + Directory.Delete(root, recursive: true); + } + } +} diff --git a/server/csharp/MetaObjects/Config/NeutralConfig.cs b/server/csharp/MetaObjects/Config/NeutralConfig.cs new file mode 100644 index 000000000..94d35b10b --- /dev/null +++ b/server/csharp/MetaObjects/Config/NeutralConfig.cs @@ -0,0 +1,93 @@ +// Port-neutral `.metaobjects/config.json` reading. +// +// Reads only the NEUTRAL SUBSET (`schema_version`, `sources`). The file also +// carries TypeScript-owned keys (`pending_in_git`, `confidence_thresholds`, +// `extract`, `migrate`, `scope`); those are IGNORED rather than modeled, so a +// new TS-only key never becomes a four-port change. `scope` in particular is +// entirely out of scope for this reader — see +// docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md §4. +using System.Text.Json; + +namespace MetaObjects.Config; + +public sealed class NeutralConfig +{ + /// The DEFAULT value of `sources` when the key is absent or empty — never a + /// requirement, and never assumed to exist by any other code path. + public const string DefaultMetadataDir = "metaobjects"; + + private const string MetaObjectsDir = ".metaobjects"; + private const string ConfigFile = "config.json"; + private const int SupportedSchemaVersion = 1; + + /// Each entry is a raw source spec (e.g. `{"path": "model"}`, + /// `{"resource": "com/acme/model"}`) — kind interpretation belongs to + /// SourceResolver, not here. + public IReadOnlyList> Sources { get; } + + private NeutralConfig(IReadOnlyList> sources) => Sources = sources; + + /// Returns null when `/.metaobjects/config.json` does not exist. + /// A file that EXISTS but is malformed THROWS — swallowing it would make a + /// typo'd config behave identically to no config at all, silently loading + /// from a possibly-stale default directory instead. The exact error code + /// used here is deliberately NOT part of the cross-port contract (only the + /// raise-don't-degrade behavior is), so callers should match on message, + /// not code, for this path. + public static NeutralConfig? Read(string configDir) + { + var path = Path.Combine(configDir, MetaObjectsDir, ConfigFile); + if (!File.Exists(path)) return null; + + JsonDocument doc; + try + { + doc = JsonDocument.Parse(File.ReadAllText(path)); + } + catch (Exception e) when (e is JsonException or IOException) + { + throw new MetaModelException( + $"{path} exists but could not be read as JSON: {e.Message}", + ErrorCode.ERR_MALFORMED_JSON); + } + + using (doc) + { + var root = doc.RootElement; + if (root.ValueKind != JsonValueKind.Object) + throw new MetaModelException($"{path} must contain a JSON object", ErrorCode.ERR_TOP_LEVEL_NOT_OBJECT); + + if (!root.TryGetProperty("schema_version", out var v) || + v.ValueKind != JsonValueKind.Number || + v.GetInt32() != SupportedSchemaVersion) + { + throw new MetaModelException( + $"{path}: unsupported schema_version (expected {SupportedSchemaVersion})", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + + var specs = new List>(); + if (root.TryGetProperty("sources", out var srcs) && srcs.ValueKind == JsonValueKind.Array) + { + foreach (var s in srcs.EnumerateArray()) + { + if (s.ValueKind != JsonValueKind.Object) + throw new MetaModelException($"{path}: each \"sources\" entry must be an object", ErrorCode.ERR_BAD_ATTR_VALUE); + + var d = new Dictionary(); + foreach (var p in s.EnumerateObject()) + d[p.Name] = p.Value.ValueKind == JsonValueKind.String ? p.Value.GetString()! : p.Value.GetRawText(); + + if (d.Count != 1) + throw new MetaModelException($"{path}: each \"sources\" entry must have exactly one key", ErrorCode.ERR_BAD_ATTR_VALUE); + + specs.Add(d); + } + } + + // Unknown top-level keys (including `scope`/`migrate`) are IGNORED by + // design — see the file header. + return new NeutralConfig(specs); + } + } +} diff --git a/server/csharp/MetaObjects/Config/SourceResolver.cs b/server/csharp/MetaObjects/Config/SourceResolver.cs new file mode 100644 index 000000000..95dd928fa --- /dev/null +++ b/server/csharp/MetaObjects/Config/SourceResolver.cs @@ -0,0 +1,110 @@ +// Turns a declared source SET (`.metaobjects/config.json`'s `sources`, or the +// single-entry default when it is absent/empty) into a de-duplicated list of +// metadata file paths. +// +// Behavioral contract: server/typescript/packages/sdk/src/sources.ts + +// collection.ts (the cross-port authority). File ORDER is deliberately NOT a +// cross-port contract (each port keeps its own natural order) — only the +// resolved SET and the error behavior are gated by the shared corpus at +// fixtures/source-resolution-conformance/cases.json. +using MetaObjects.Loader; + +namespace MetaObjects.Config; + +public static class SourceResolver +{ + /// Resolve a declared source SET to a de-duplicated list of metadata files. + /// A relative `path` resolves against `configDir` — the directory HOLDING the + /// `.metaobjects/` folder — never against the process working directory. + /// + /// Validation runs in two passes, mirroring `sources.ts`'s `orderedPathSpecs`: + /// EVERY spec's kind is checked first, in declared order, before any spec is + /// resolved against the filesystem. Interleaving the two (validate-then-resolve + /// spec by spec) would make which error fires depend on which unsupported spec + /// or missing path happens to sit first — the corpus pins that an unsupported + /// KIND anywhere in the list wins over an unresolved PATH regardless of which + /// is declared first (`unsupported-kind-precedes-unresolved-path-when-path-is- + /// declared-first`/`-second`). + public static IReadOnlyList ResolveSources( + string configDir, + IReadOnlyList> specs) + { + // Pass 1 — kind validation across the WHOLE set, no filesystem I/O yet. + var pathSpecs = new List(specs.Count); + foreach (var spec in specs) + { + if (spec.TryGetValue("path", out var rawPath)) + { + pathSpecs.Add(rawPath); + } + else + { + var kind = spec.Keys.FirstOrDefault() ?? ""; + throw new MetaModelException( + $"source kind \"{kind}\" is not supported by this toolchain yet; use a \"path\" source", + ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED); + } + } + + // Pass 2 — resolve each validated path spec against the filesystem. + var seen = new List(); + var known = new HashSet(StringComparer.Ordinal); + + foreach (var rawPath in pathSpecs) + { + var target = Path.IsPathRooted(rawPath) ? rawPath : Path.GetFullPath(Path.Combine(configDir, rawPath)); + + var isDir = Directory.Exists(target); + if (!isDir && !File.Exists(target)) + throw new MetaModelException( + $"source path \"{rawPath}\" does not exist (resolved to {target}, relative to {configDir})", + ErrorCode.ERR_SOURCE_UNRESOLVED); + + // Directory expansion — extension filter + ordinal sort — is + // DirectorySource's, the SAME code the loader itself uses to turn a + // directory into metadata files (Loader/DirectorySource.cs). Reimplementing + // the filter/sort here would be a second, driftable definition of "which + // files count as metadata" for exactly the reason DirectorySource's own + // header calls out: order within one directory spec is this port's own + // full-path ordinal sort, deliberately NOT a cross-port contract (see the + // file header above), but it MUST still be the loader's own order. + var found = isDir + ? new DirectorySource(target).Expand().Select(f => f.FilePath) + : new[] { target }.AsEnumerable(); + + foreach (var f in found) + { + var full = Path.GetFullPath(f); + if (known.Add(full)) seen.Add(full); + } + } + + return seen; + } + + /// The full ladder: declared `sources`, else the default directory. + /// Only the DEFAULT may be silently absent — a declared source that does not + /// resolve is `ERR_SOURCE_UNRESOLVED`, a louder failure than "nothing declared". + public static IReadOnlyList ResolveCollection(string root) + { + root = Path.GetFullPath(root); + var cfg = NeutralConfig.Read(root); + var specs = cfg?.Sources ?? Array.Empty>(); + + if (specs.Count == 0) + { + var defaultDir = Path.Combine(root, NeutralConfig.DefaultMetadataDir); + if (!Directory.Exists(defaultDir)) + throw new MetaModelException( + $"no metadata sources declared in {root} and no default \"{NeutralConfig.DefaultMetadataDir}\" " + + "directory found. Declare \"sources\" in .metaobjects/config.json, or run 'meta init' to scaffold.", + ErrorCode.ERR_COLLECTION_NOT_FOUND); + specs = new IReadOnlyDictionary[] + { + new Dictionary { ["path"] = NeutralConfig.DefaultMetadataDir }, + }; + } + + return ResolveSources(root, specs); + } +} From 5813fcc7855b9c10cf14e319bd6cea22d2f1e0c3 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 11:32:48 -0400 Subject: [PATCH 25/44] feat(java): read the port-neutral sources key when the pom is silent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds com.metaobjects.config.NeutralConfig + SourceResolver (metadata module) reading the neutral subset of .metaobjects/config.json (schema_version + sources; TS-owned keys ignored) and resolving a declared source set to a de-duplicated file list, kind-validated in one pass ahead of any filesystem access so the winning error code never depends on declaration order. Gated by the shared fixtures/source-resolution-conformance/cases.json corpus (19 cases). Wires the resolver into AbstractMetaDataMojo.createLoader via resolveNeutralSourcesIfPomIsSilent(): whole-concern precedence — a pom naming or owns the concern outright and the neutral file is never consulted. is untouched; scope stays TypeScript-only. Reachability is proven by a mojo-level test driving createLoader end-to-end (not just the resolver in isolation), across the neutral-config, default-directory, and no-collection-found arms. Kotlin needs no separate change: it has no CLI entry point of its own and runs through this same Maven plugin. --- .../mojo/AbstractMetaDataMojo.java | 53 ++++- .../mojo/NeutralConfigMojoFallbackTest.java | 138 +++++++++++++ .../com/metaobjects/config/NeutralConfig.java | 146 +++++++++++++ .../metaobjects/config/SourceResolver.java | 148 ++++++++++++++ .../SourceResolutionConformanceTest.java | 192 ++++++++++++++++++ 5 files changed, 676 insertions(+), 1 deletion(-) create mode 100644 server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java create mode 100644 server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java create mode 100644 server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java create mode 100644 server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java diff --git a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java index 1ca010650..640b33ac1 100644 --- a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java +++ b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java @@ -260,8 +260,17 @@ protected MetaDataLoader createLoader(ClassLoader projectClassLoader) { sourceDir = loaderConfig.getSourceDir(); } + // Precedence ladder (spec §5): when the pom names neither nor + // , fall back to the port-neutral .metaobjects/config.json (else the + // built-in default directory) instead of silently loading nothing. + List sources = loaderConfig.getSources(); + List neutralSources = resolveNeutralSourcesIfPomIsSilent(loaderConfig); + if (!neutralSources.isEmpty()) { + sources = neutralSources; + } + MavenLoaderConfiguration.configure(configurable, sourceDir, projectClassLoader, - loaderConfig.getSources(), loaderArgs(strict)); + sources, loaderArgs(strict)); MetaDataLoader loader = configurable.getLoader(); @@ -468,4 +477,46 @@ protected File getSourceDir() { } return sourceDir; } + + /** + * The module basedir metadata-source resolution is anchored to — the same + * directory that would hold a project's {@code .metaobjects/} folder. Falls back + * to the process working directory when {@code project} is unset (e.g. a Mojo + * driven directly in a unit test, matching {@link #warnIfAgentContextStale()}'s + * same fallback). + */ + protected File getProjectBaseDir() { + return project != null ? project.getBasedir() : new File(System.getProperty("user.dir")); + } + + /** + * The precedence ladder for where metadata lives (spec §5). First match wins. + * + *

1. The pom — {@code } or {@code }. If + * EITHER is present the pom owns the whole concern and the neutral file is not + * consulted; precedence is whole-concern, not a per-entry merge. + *
2. {@code sources} in the port-neutral {@code .metaobjects/config.json}, + * read from the module basedir. + *
3. The built-in default directory. + * + *

A neutral file that EXISTS but is malformed throws rather than falling + * through. {@code } is untouched by this ladder — {@code scope} stays + * out of scope for this mechanism (Global Constraints). + * + * @return the resolved metadata file paths, or an empty list when the pom names + * either {@code } or {@code } (i.e. the neutral file is + * not consulted at all) + */ + protected List resolveNeutralSourcesIfPomIsSilent(LoaderParam loaderConfig) { + boolean pomNamesLocation = + (loaderConfig.getSourceDir() != null && !loaderConfig.getSourceDir().isBlank()) + || (loaderConfig.getSources() != null && !loaderConfig.getSources().isEmpty()); + if (pomNamesLocation) return List.of(); + + return com.metaobjects.config.SourceResolver + .resolveCollection(getProjectBaseDir().toPath()) + .stream() + .map(java.nio.file.Path::toString) + .toList(); + } } diff --git a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java new file mode 100644 index 000000000..934f0fa18 --- /dev/null +++ b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java @@ -0,0 +1,138 @@ +package com.metaobjects.mojo; + +import com.metaobjects.MetaDataException; +import com.metaobjects.loader.MetaDataLoader; +import org.apache.maven.project.MavenProject; +import org.junit.Test; +import org.mockito.Mockito; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Collections; +import java.util.Comparator; +import java.util.stream.Stream; + +import static org.junit.Assert.assertEquals; + +/** + * Proves the port-neutral {@code .metaobjects/config.json} fallback (spec §5, ADR + * cross-port metadata-source-resolution phase 1) is actually WIRED into + * {@link AbstractMetaDataMojo#createLoader}, not merely defined and unit-tested in + * isolation — the same gap that shipped in an earlier port's task before it was + * caught in review. Every test here leaves the pom's {@code } silent on both + * {@code sourceDir} and {@code sources}, so a regression that removes the + * {@code resolveNeutralSourcesIfPomIsSilent(...)} call site (or short-circuits it) + * makes {@code createLoader} load ZERO metadata objects instead of the one declared + * under the neutral config's declared source — turning every assertion below false. + */ +public class NeutralConfigMojoFallbackTest { + + private static final String WIDGET_JSON = """ + { + "metadata.root": { + "package": "neutral::fallback", + "children": [ + { "object.entity": { "name": "Widget", "children": [ + { "field.long": { "name": "id" } } + ] } } + ] + } + } + """; + + /** A pom-silent loader: no {@code }, no {@code }. */ + private MetaDataGeneratorMojo mojoWithSilentPom(Path basedir) { + MavenProject mavenProject = Mockito.mock(MavenProject.class); + Mockito.when(mavenProject.getBasedir()).thenReturn(basedir.toFile()); + + MetaDataGeneratorMojo mojo = new MetaDataGeneratorMojo(); + mojo.project = mavenProject; + + LoaderParam loader = LoaderParam.builder("neutral-fallback-test") + .withClassname("com.metaobjects.loader.MetaDataLoader") + .build(); + mojo.setLoader(loader); + mojo.setGenerators(Collections.emptyList()); + mojo.setGlobals(Collections.emptyMap()); + return mojo; + } + + @Test + public void createLoaderFallsBackToTheNeutralConfigsDeclaredSourceWhenPomIsSilent() throws IOException { + Path root = Files.createTempDirectory("mo-mojo-neutral-").toAbsolutePath().normalize(); + try { + Path metaDir = root.resolve("custom-metadata"); + Files.createDirectories(metaDir); + Files.write(metaDir.resolve("meta.widget.json"), WIDGET_JSON.getBytes(StandardCharsets.UTF_8)); + + Path dotMo = root.resolve(".metaobjects"); + Files.createDirectories(dotMo); + Files.write(dotMo.resolve("config.json"), + "{\"schema_version\":1,\"sources\":[{\"path\":\"custom-metadata\"}]}" + .getBytes(StandardCharsets.UTF_8)); + + MetaDataGeneratorMojo mojo = mojoWithSilentPom(root); + MetaDataLoader loaded = mojo.createLoader(mojo.createProjectClassLoader()); + + // If resolveNeutralSourcesIfPomIsSilent were never called (or its result + // discarded), the loader would have been configured with zero sources and + // loaded nothing — this assertion is exactly what removing the wiring + // breaks. + assertEquals("expected the ONE object declared under the neutral config's " + + "declared \"custom-metadata\" source — proves the fallback ran, " + + "not just the default directory", + 1, loaded.getMetaObjects().size()); + assertEquals("Widget", loaded.getMetaObjects().get(0).getShortName()); + } finally { + deleteRecursive(root); + } + } + + @Test + public void createLoaderFallsBackToTheBuiltInDefaultDirectoryWhenNoNeutralConfigExists() throws IOException { + Path root = Files.createTempDirectory("mo-mojo-neutral-default-").toAbsolutePath().normalize(); + try { + // No .metaobjects/config.json at all — only the built-in default directory. + Path defaultDir = root.resolve("metaobjects"); + Files.createDirectories(defaultDir); + Files.write(defaultDir.resolve("meta.widget.json"), WIDGET_JSON.getBytes(StandardCharsets.UTF_8)); + + MetaDataGeneratorMojo mojo = mojoWithSilentPom(root); + MetaDataLoader loaded = mojo.createLoader(mojo.createProjectClassLoader()); + + assertEquals(1, loaded.getMetaObjects().size()); + assertEquals("Widget", loaded.getMetaObjects().get(0).getShortName()); + } finally { + deleteRecursive(root); + } + } + + @Test(expected = MetaDataException.class) + public void createLoaderRaisesWhenPomIsSilentAndNoCollectionExists() throws IOException { + // Neither a neutral config nor a default "metaobjects/" directory — the final + // rung of the ladder must raise (ERR_COLLECTION_NOT_FOUND), not silently load + // zero objects. + Path root = Files.createTempDirectory("mo-mojo-neutral-empty-").toAbsolutePath().normalize(); + try { + MetaDataGeneratorMojo mojo = mojoWithSilentPom(root); + mojo.createLoader(mojo.createProjectClassLoader()); + } finally { + deleteRecursive(root); + } + } + + private static void deleteRecursive(Path dir) throws IOException { + if (!Files.exists(dir)) return; + try (Stream walk = Files.walk(dir)) { + walk.sorted(Comparator.reverseOrder()).forEach(p -> { + try { + Files.delete(p); + } catch (IOException ignored) { + // best-effort temp-dir cleanup + } + }); + } + } +} diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java new file mode 100644 index 000000000..ac4039cbd --- /dev/null +++ b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java @@ -0,0 +1,146 @@ +/* + * Copyright 2003 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.config; + +import com.google.gson.JsonElement; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import com.google.gson.JsonSyntaxException; +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * The port-neutral subset of {@code .metaobjects/config.json}. + * + *

Reads only {@code schema_version} and {@code sources}. The file also carries + * TypeScript-owned keys ({@code pending_in_git}, {@code confidence_thresholds}, + * {@code extract}, {@code migrate}, {@code scope}); those are IGNORED rather than + * modeled, so a new TS-only key never becomes a four-port change. {@code scope} in + * particular is entirely out of scope for this reader — see + * {@code docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md} §4. + */ +public final class NeutralConfig { + + /** + * The DEFAULT value of {@code sources} when the key is absent or empty — never a + * requirement, and never assumed to exist by any other code path. + */ + public static final String DEFAULT_METADATA_DIR = "metaobjects"; + + private static final String METAOBJECTS_DIR = ".metaobjects"; + private static final String CONFIG_FILE = "config.json"; + private static final int SUPPORTED_SCHEMA_VERSION = 1; + + private final List> sources; + + private NeutralConfig(List> sources) { + this.sources = List.copyOf(sources); + } + + /** + * Each entry is a raw source spec (e.g. {@code {"path": "model"}}, + * {@code {"resource": "com/acme/model"}}) — kind interpretation belongs to + * {@link SourceResolver}, not here. + */ + public List> getSources() { + return sources; + } + + /** + * Returns empty when {@code /.metaobjects/config.json} does not exist. + * A file that EXISTS but is malformed throws — swallowing it would make a typo'd + * config behave identically to no config at all, silently loading from a possibly + * stale default directory instead. The exact {@link ErrorCode} used for the + * malformed path is deliberately NOT part of the cross-port contract (only the + * raise-don't-degrade behavior is) — callers should match on the raise, not the + * code, for this path. + */ + public static Optional read(Path configDir) { + Path path = configDir.resolve(METAOBJECTS_DIR).resolve(CONFIG_FILE); + if (!Files.isRegularFile(path)) return Optional.empty(); + + String content; + try { + content = new String(Files.readAllBytes(path), StandardCharsets.UTF_8); + } catch (IOException e) { + throw new MetaDataException( + path + " exists but could not be read: " + e.getMessage(), + ErrorCode.ERR_MALFORMED_JSON); + } + + JsonElement parsed; + try { + parsed = JsonParser.parseString(content); + } catch (JsonSyntaxException e) { + throw new MetaDataException( + path + " exists but could not be parsed as JSON: " + e.getMessage(), + ErrorCode.ERR_MALFORMED_JSON); + } + + if (!parsed.isJsonObject()) { + throw new MetaDataException( + path + " must contain a JSON object", + ErrorCode.ERR_TOP_LEVEL_NOT_OBJECT); + } + JsonObject root = parsed.getAsJsonObject(); + + JsonElement version = root.get("schema_version"); + if (version == null || !version.isJsonPrimitive() || !version.getAsJsonPrimitive().isNumber() + || version.getAsInt() != SUPPORTED_SCHEMA_VERSION) { + throw new MetaDataException( + path + ": unsupported schema_version (expected " + SUPPORTED_SCHEMA_VERSION + ")", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + + List> specs = new ArrayList<>(); + JsonElement srcs = root.get("sources"); + if (srcs != null && srcs.isJsonArray()) { + for (JsonElement el : srcs.getAsJsonArray()) { + if (!el.isJsonObject()) { + throw new MetaDataException( + path + ": each \"sources\" entry must be an object", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + JsonObject entry = el.getAsJsonObject(); + if (entry.size() != 1) { + throw new MetaDataException( + path + ": each \"sources\" entry must have exactly one key", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + Map spec = new LinkedHashMap<>(); + for (Map.Entry e : entry.entrySet()) { + JsonElement v = e.getValue(); + spec.put(e.getKey(), v.isJsonPrimitive() ? v.getAsString() : v.toString()); + } + specs.add(spec); + } + } + + // Unknown top-level keys (including `scope`/`migrate`) are IGNORED by + // design — see the class javadoc. + return Optional.of(new NeutralConfig(specs)); + } +} diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java new file mode 100644 index 000000000..0788c93c8 --- /dev/null +++ b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java @@ -0,0 +1,148 @@ +/* + * Copyright 2003 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.config; + +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.stream.Stream; + +/** + * Turns a declared source SET ({@code .metaobjects/config.json}'s {@code sources}, or + * the single-entry default when it is absent/empty) into a de-duplicated list of + * metadata file paths. + * + *

Behavioral contract: {@code server/typescript/packages/sdk/src/sources.ts} + + * {@code collection.ts} (the cross-port authority). File ORDER is deliberately NOT a + * cross-port contract (each port keeps its own natural order — this port's is a + * basename sort within a directory) — only the resolved SET and the error behavior + * are gated by the shared corpus at + * {@code fixtures/source-resolution-conformance/cases.json}. + */ +public final class SourceResolver { + + private static final Set EXTENSIONS = Set.of(".json", ".yaml", ".yml"); + + private SourceResolver() {} + + /** + * Resolve a declared source SET to a de-duplicated list of metadata files. + * + *

A relative {@code path} resolves against {@code configDir} — the directory + * HOLDING the {@code .metaobjects/} folder — never against the process working + * directory. + * + *

Validation runs in two passes: EVERY spec's kind is checked first, in + * declared order, before any spec is resolved against the filesystem. + * Interleaving the two (validate-then-resolve spec by spec) would make which + * error fires depend on which unsupported spec or missing path happens to sit + * first — the corpus pins that an unsupported KIND anywhere in the list wins over + * an unresolved PATH regardless of which is declared first + * ({@code unsupported-kind-precedes-unresolved-path-when-path-is-declared-first}/ + * {@code -second}). + */ + public static List resolveSources(Path configDir, List> specs) { + // Pass 1 — kind validation across the WHOLE set, no filesystem I/O yet. + List pathSpecs = new ArrayList<>(specs.size()); + for (Map spec : specs) { + String rawPath = spec.get("path"); + if (rawPath == null) { + String kind = spec.keySet().stream().findFirst().orElse(""); + throw new MetaDataException( + "source kind \"" + kind + "\" is not supported by this toolchain yet; use a \"path\" source", + ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED); + } + pathSpecs.add(rawPath); + } + + // Pass 2 — resolve each validated path spec against the filesystem. + LinkedHashSet seen = new LinkedHashSet<>(); + for (String rawPath : pathSpecs) { + Path raw = Path.of(rawPath); + Path target = raw.isAbsolute() ? raw : configDir.resolve(raw).normalize(); + + boolean isDir = Files.isDirectory(target); + if (!isDir && !Files.isRegularFile(target)) { + throw new MetaDataException( + "source path \"" + rawPath + "\" does not exist (resolved to " + target + + ", relative to " + configDir + ")", + ErrorCode.ERR_SOURCE_UNRESOLVED); + } + + if (isDir) { + // Order within one directory spec is this port's own basename sort, + // deliberately NOT a cross-port contract — see the class javadoc. + try (Stream walk = Files.walk(target)) { + walk.filter(Files::isRegularFile) + .filter(p -> hasSupportedExtension(p.getFileName().toString())) + .sorted(Comparator.comparing(p -> p.getFileName().toString())) + .forEach(p -> seen.add(p.toAbsolutePath().normalize())); + } catch (IOException e) { + throw new UncheckedIOException("Failed to list " + target, e); + } + } else { + seen.add(target.toAbsolutePath().normalize()); + } + } + + return new ArrayList<>(seen); + } + + /** + * The full ladder: declared {@code sources}, else the default directory. Only the + * DEFAULT may be silently absent — a declared source that does not resolve is + * {@code ERR_SOURCE_UNRESOLVED}, a louder failure than "nothing declared". + */ + public static List resolveCollection(Path root) { + Path base = root.toAbsolutePath().normalize(); + List> specs = NeutralConfig.read(base) + .map(NeutralConfig::getSources) + .orElse(List.of()); + + if (specs.isEmpty()) { + Path defaultDir = base.resolve(NeutralConfig.DEFAULT_METADATA_DIR); + if (!Files.isDirectory(defaultDir)) { + throw new MetaDataException( + "no metadata sources declared in " + base + " and no default \"" + + NeutralConfig.DEFAULT_METADATA_DIR + "\" directory found. Declare \"sources\" in " + + ".metaobjects/config.json, or run 'meta init' to scaffold.", + ErrorCode.ERR_COLLECTION_NOT_FOUND); + } + specs = List.of(Map.of("path", NeutralConfig.DEFAULT_METADATA_DIR)); + } + + return resolveSources(base, specs); + } + + private static boolean hasSupportedExtension(String name) { + String lower = name.toLowerCase(Locale.ROOT); + for (String ext : EXTENSIONS) { + if (lower.endsWith(ext)) return true; + } + return false; + } +} diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java new file mode 100644 index 000000000..ffdc3429a --- /dev/null +++ b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java @@ -0,0 +1,192 @@ +/* + * Copyright 2003 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.config; + +import com.google.gson.JsonArray; +import com.google.gson.JsonElement; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import com.metaobjects.MetaDataException; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.junit.runners.Parameterized.Parameter; +import org.junit.runners.Parameterized.Parameters; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Comparator; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.fail; + +/** + * Runs the shared source-resolution corpus against this port. Reads the single + * committed {@code fixtures/source-resolution-conformance/cases.json} — no per-port + * fixture. The corpus is the contract; see + * {@code server/typescript/packages/sdk/src/sources.ts} + {@code collection.ts} for + * the authoritative behavior each case pins. + */ +@RunWith(Parameterized.class) +public class SourceResolutionConformanceTest { + + /** + * One row per corpus case. + * + * @param resolveFrom Project-root-relative directory the resolver is invoked + * FROM. Defaults to {@code "."} — 18 of 19 cases leave it there, so the + * config lives at the project root and "relative to project root" vs + * "relative to the invocation directory" coincide. The one case that sets it + * ({@code a-parent-relative-path-resolves-against-the-declaring-configs- + * directory}) is the one place those two bases diverge, and both the + * config's own location AND the {@code expectFiles} comparison base below + * must honor it correctly for that case to mean anything. + */ + private record Case(String name, Map tree, JsonObject config, + String resolveFrom, List expectFiles, String expectError) {} + + private static Path corpus() { + Path dir = Paths.get("").toAbsolutePath(); + while (dir != null && !Files.isDirectory(dir.resolve("fixtures"))) dir = dir.getParent(); + assertNotNull("could not locate the repository fixtures/ directory", dir); + return dir.resolve("fixtures/source-resolution-conformance/cases.json"); + } + + @Parameters(name = "{0}") + public static Collection cases() throws IOException { + String content = new String(Files.readAllBytes(corpus()), StandardCharsets.UTF_8); + JsonObject root = JsonParser.parseString(content).getAsJsonObject(); + JsonArray arr = root.getAsJsonArray("cases"); + + List rows = new ArrayList<>(); + for (JsonElement el : arr) { + JsonObject c = el.getAsJsonObject(); + String name = c.get("name").getAsString(); + + Map tree = new LinkedHashMap<>(); + for (Map.Entry e : c.getAsJsonObject("tree").entrySet()) { + tree.put(e.getKey(), e.getValue().getAsString()); + } + + JsonElement cfgEl = c.get("config"); + JsonObject config = (cfgEl == null || cfgEl.isJsonNull()) ? null : cfgEl.getAsJsonObject(); + + String resolveFrom = c.has("resolveFrom") ? c.get("resolveFrom").getAsString() : "."; + + List expectFiles = null; + if (c.has("expectFiles")) { + expectFiles = new ArrayList<>(); + for (JsonElement f : c.getAsJsonArray("expectFiles")) { + expectFiles.add(f.getAsString()); + } + } + + String expectError = c.has("expectError") ? c.get("expectError").getAsString() : null; + + rows.add(new Object[]{name, new Case(name, tree, config, resolveFrom, expectFiles, expectError)}); + } + return rows; + } + + /** First arg drives the JUnit display name. */ + @Parameter(0) + public String caseName; + + @Parameter(1) + public Case testCase; + + @Test + public void resolvesTheSameFileSet() throws IOException { + // `root` is the PROJECT ROOT — the base every `tree` path and every + // `expectFiles` entry is written/compared relative to, regardless of + // `resolveFrom`. Normalized the same way SourceResolver.resolveCollection + // normalizes its own `root` argument, so relativizing against it later lines + // up exactly with what the resolver returns. + Path root = Files.createTempDirectory("mo-src-resolution-").toAbsolutePath().normalize(); + try { + for (Map.Entry entry : testCase.tree().entrySet()) { + Path abs = root.resolve(entry.getKey()); + Files.createDirectories(abs.getParent()); + Files.write(abs, entry.getValue().getBytes(StandardCharsets.UTF_8)); + } + + // The invocation directory: project root joined with `resolveFrom`. The + // config MUST be materialized here, not at the project root — a config + // placed at the project root while resolving FROM a subdirectory would go + // undetected by resolveCollection there and fail loudly with + // ERR_COLLECTION_NOT_FOUND, which is what makes this half of the mistake + // self-catching. Getting it right on purpose (not by luck) is what this + // comment is pinning. + Path invokeDir = root.resolve(testCase.resolveFrom()).normalize(); + Files.createDirectories(invokeDir); + if (testCase.config() != null) { + Path dotMo = invokeDir.resolve(".metaobjects"); + Files.createDirectories(dotMo); + Files.write(dotMo.resolve("config.json"), + testCase.config().toString().getBytes(StandardCharsets.UTF_8)); + } + + if (testCase.expectError() != null) { + try { + SourceResolver.resolveCollection(invokeDir); + fail("expected " + testCase.expectError() + " for case " + testCase.name()); + } catch (MetaDataException e) { + assertEquals(testCase.expectError(), e.getCode().orElseThrow().name()); + } + return; + } + + // Compared against the PROJECT ROOT explicitly — never against + // `invokeDir`. For 18 of 19 cases the two coincide (resolveFrom "."), so a + // comparison-base bug here would pass every case except the one that sets + // `resolveFrom`, which is exactly why that case exists. + Set got = SourceResolver.resolveCollection(invokeDir).stream() + .map(p -> root.relativize(p).toString().replace('\\', '/')) + .collect(Collectors.toSet()); + + Set want = new HashSet<>(testCase.expectFiles()); + assertEquals(want, got); + } finally { + deleteRecursive(root); + } + } + + private static void deleteRecursive(Path dir) throws IOException { + if (!Files.exists(dir)) return; + try (Stream walk = Files.walk(dir)) { + walk.sorted(Comparator.reverseOrder()).forEach(p -> { + try { + Files.delete(p); + } catch (IOException ignored) { + // best-effort temp-dir cleanup + } + }); + } + } +} From c47c6e73181c5218fb447ec70566cf13f53a9f07 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 11:55:14 -0400 Subject: [PATCH 26/44] fix(java,csharp): raise on a wrong-typed neutral-config sources value Spans Java, C# and the shared corpus (Task 5 fix round 1; reopens Task 4). NeutralConfig's `sources` guard in both ports checked `isJsonArray`/`ValueKind == Array` only when present, so a present-but-wrong-typed `sources` (e.g. a bare object instead of an array) fell through as an empty list and silently degraded to the default `metaobjects/` directory with no diagnostic - reproduced against a stale default-dir file sitting next to the author's intended source. Both ports now raise (ERR_BAD_ATTR_VALUE) whenever `sources` is present, non-null, and not an array; `sources: null` keeps behaving as absent, unchanged. Adds a corpus case for this (`sources-must-be-an-array-not-an-object`) exercising both ports' new guard. TypeScript and Python were checked empirically rather than assumed: TS's reference raises a raw ZodError with no error code at all, Python already raised ERR_COLLECTION_NOT_FOUND - three distinct outcomes across four ports, confirming they cannot agree on a single code here. Rather than change TS or Python behavior to force agreement, `expectError` in the corpus schema now accepts JSON `true` ("must raise, code intentionally unpinned") alongside the existing exact-code string form; only each port's test-runner corpus-interpretation code was updated to understand it, not resolveCollection/NeutralConfig/ neutral_config.py/config.ts themselves. Co-Authored-By: Claude Opus 5 (1M context) --- .../source-resolution-conformance/README.md | 34 +++++++++++++------ .../source-resolution-conformance/cases.json | 9 +++++ .../SourceResolutionConformanceTests.cs | 14 ++++++-- .../MetaObjects/Config/NeutralConfig.cs | 12 ++++++- .../com/metaobjects/config/NeutralConfig.java | 10 ++++++ .../SourceResolutionConformanceTest.java | 16 +++++++-- .../test_source_resolution_conformance.py | 6 +++- .../source-resolution-conformance.test.ts | 18 +++++++--- 8 files changed, 96 insertions(+), 23 deletions(-) diff --git a/fixtures/source-resolution-conformance/README.md b/fixtures/source-resolution-conformance/README.md index cac2b363f..109c6deb6 100644 --- a/fixtures/source-resolution-conformance/README.md +++ b/fixtures/source-resolution-conformance/README.md @@ -34,8 +34,11 @@ README.md - **`expectFiles`** — project-root-relative paths (NOT relative to `resolveFrom`), compared as an **UNORDERED SET**. See "Order is deliberately not pinned" below. -- **`expectError`** — an error code the resolution must fail with. Exactly one - of `expectFiles` / `expectError` is present per case. +- **`expectError`** — either a STRING error code the resolution must fail with + exactly, or the literal `true` meaning "must raise, but which error/code is + deliberately not pinned across ports" (see "Also deliberately NOT pinned: + the malformed-config error code" below for why the latter form exists). + Exactly one of `expectFiles` / `expectError` is present per case. ## Semantics pinned here @@ -97,16 +100,25 @@ output depends on it. It just is not a cross-port promise. ## Also deliberately NOT pinned: the malformed-config error code -The corpus has no case for a `.metaobjects/config.json` that exists but fails -to parse (bad JSON, an unsupported `schema_version`, a malformed `sources` -shape). The contract is only that resolution MUST raise rather than silently -degrade to "no config" — which error code it raises with is left to each -port. The reference implementation is why: `collection.ts:129-140` has no +`sources-must-be-an-array-not-an-object` pins that a `.metaobjects/config.json` +that exists but is malformed (here: `sources` declared as a bare object instead +of an array) MUST raise rather than silently degrade to "no config" — the +regression this case exists to catch is a wrong-typed `sources` reading as +absent, falling back to the default `metaobjects/` directory with no +diagnostic at all, even when a stale file sits there. It uses `"expectError": +true` rather than a string code, because which code it raises with is left to +each port. The reference implementation is why: `collection.ts:129-140` has no try/catch around config loading and lets the raw zod/JSON error propagate, so -TypeScript emits no MetaObjects error code here at all. Pinning a shared code -across ports would mean changing the reference, which this corpus does not -do. Python raises `ERR_COLLECTION_NOT_FOUND`; C# and Java may each choose a -different code for the same failure, same as file order above. +TypeScript throws a `ZodError` carrying no MetaObjects error code at all. +Pinning a shared code across ports would mean changing the reference, which +this corpus does not do. Python raises `ERR_COLLECTION_NOT_FOUND`; C# and Java +both raise `ERR_BAD_ATTR_VALUE` — three distinct outcomes across four ports, +which is exactly why this case checks only that resolution raises, never with +which error, same as file order above. The same `true` form is available to +any future malformed-config case that needs it (bad JSON, an unsupported +`schema_version`, a malformed `sources` entry shape, …) — none of those are +in the corpus yet, but nothing about the mechanism is specific to this one +shape. ## Behavioral contract diff --git a/fixtures/source-resolution-conformance/cases.json b/fixtures/source-resolution-conformance/cases.json index 9815a1f5f..30b54f872 100644 --- a/fixtures/source-resolution-conformance/cases.json +++ b/fixtures/source-resolution-conformance/cases.json @@ -191,6 +191,15 @@ "migrate": { "dialect": "postgres" } }, "expectFiles": ["model/meta.a.json"] + }, + { + "name": "sources-must-be-an-array-not-an-object", + "tree": { + "metaobjects/meta.stale.json": "{\"metadata.root\":{\"children\":[]}}", + "custom/meta.real.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": { "path": "custom" } }, + "expectError": true } ] } diff --git a/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs b/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs index cc3578443..5590eeec7 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs @@ -25,7 +25,10 @@ private sealed record Case( // honor it correctly for that case to mean anything. string ResolveFrom, string[]? ExpectFiles, - string? ExpectError); + // A JSON string pins the exact error code raised; JSON `true` pins only + // that resolution RAISES — the malformed-config error code is + // deliberately not pinned cross-port (see the corpus README). + JsonElement? ExpectError); public static TheoryData CaseNames() { @@ -61,7 +64,7 @@ private static List LoadCases() string[]? expectFiles = el.TryGetProperty("expectFiles", out var ef) ? ef.EnumerateArray().Select(x => x.GetString()!).ToArray() : null; - string? expectError = el.TryGetProperty("expectError", out var ee) ? ee.GetString() : null; + JsonElement? expectError = el.TryGetProperty("expectError", out var ee) ? ee.Clone() : null; cases.Add(new Case(el.GetProperty("name").GetString()!, tree, cfg, resolveFrom, expectFiles, expectError)); } @@ -105,7 +108,12 @@ public void ResolvesTheSameFileSet(string name) if (c.ExpectError is not null) { var ex = Assert.ThrowsAny(() => SourceResolver.ResolveCollection(invokeDir)); - Assert.Equal(c.ExpectError, ex.Code.ToString()); + // A string pins the exact code; `true` only pins that it raises — + // see the ExpectError field doc above. + if (c.ExpectError.Value.ValueKind == JsonValueKind.String) + { + Assert.Equal(c.ExpectError.Value.GetString(), ex.Code.ToString()); + } return; } diff --git a/server/csharp/MetaObjects/Config/NeutralConfig.cs b/server/csharp/MetaObjects/Config/NeutralConfig.cs index 94d35b10b..5bfb475f9 100644 --- a/server/csharp/MetaObjects/Config/NeutralConfig.cs +++ b/server/csharp/MetaObjects/Config/NeutralConfig.cs @@ -67,7 +67,17 @@ public sealed class NeutralConfig } var specs = new List>(); - if (root.TryGetProperty("sources", out var srcs) && srcs.ValueKind == JsonValueKind.Array) + if (root.TryGetProperty("sources", out var srcs) && srcs.ValueKind != JsonValueKind.Null + && srcs.ValueKind != JsonValueKind.Array) + { + // A present-but-wrong-typed `sources` (e.g. a bare object instead of an + // array) must RAISE, not silently read as "absent" — the latter would + // fall back to the default directory with no diagnostic, exactly the + // "typo'd config behaves like no config" failure this class exists to + // prevent (see the file header). + throw new MetaModelException($"{path}: \"sources\" must be an array", ErrorCode.ERR_BAD_ATTR_VALUE); + } + if (root.TryGetProperty("sources", out srcs) && srcs.ValueKind == JsonValueKind.Array) { foreach (var s in srcs.EnumerateArray()) { diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java index ac4039cbd..7b618c270 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java @@ -117,6 +117,16 @@ public static Optional read(Path configDir) { List> specs = new ArrayList<>(); JsonElement srcs = root.get("sources"); + if (srcs != null && !srcs.isJsonNull() && !srcs.isJsonArray()) { + // A present-but-wrong-typed `sources` (e.g. a bare object instead of an + // array) must RAISE, not silently read as "absent" — the latter would + // fall back to the default directory with no diagnostic, exactly the + // "typo'd config behaves like no config" failure this class exists to + // prevent (see the class javadoc). + throw new MetaDataException( + path + ": \"sources\" must be an array", + ErrorCode.ERR_BAD_ATTR_VALUE); + } if (srcs != null && srcs.isJsonArray()) { for (JsonElement el : srcs.getAsJsonArray()) { if (!el.isJsonObject()) { diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java index ffdc3429a..e64a62e85 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java @@ -68,8 +68,13 @@ public class SourceResolutionConformanceTest { * config's own location AND the {@code expectFiles} comparison base below * must honor it correctly for that case to mean anything. */ + /** + * {@code expectError}: a JSON string pins the exact error code raised; JSON + * {@code true} pins only that resolution RAISES — the malformed-config error + * code is deliberately not pinned cross-port (see the corpus README). + */ private record Case(String name, Map tree, JsonObject config, - String resolveFrom, List expectFiles, String expectError) {} + String resolveFrom, List expectFiles, JsonElement expectError) {} private static Path corpus() { Path dir = Paths.get("").toAbsolutePath(); @@ -107,7 +112,7 @@ public static Collection cases() throws IOException { } } - String expectError = c.has("expectError") ? c.get("expectError").getAsString() : null; + JsonElement expectError = c.has("expectError") ? c.get("expectError") : null; rows.add(new Object[]{name, new Case(name, tree, config, resolveFrom, expectFiles, expectError)}); } @@ -157,7 +162,12 @@ public void resolvesTheSameFileSet() throws IOException { SourceResolver.resolveCollection(invokeDir); fail("expected " + testCase.expectError() + " for case " + testCase.name()); } catch (MetaDataException e) { - assertEquals(testCase.expectError(), e.getCode().orElseThrow().name()); + JsonElement expected = testCase.expectError(); + // A string pins the exact code; `true` only pins that it raises — + // see the Case record's javadoc above. + if (expected.isJsonPrimitive() && expected.getAsJsonPrimitive().isString()) { + assertEquals(expected.getAsString(), e.getCode().orElseThrow().name()); + } } return; } diff --git a/server/python/tests/conformance/test_source_resolution_conformance.py b/server/python/tests/conformance/test_source_resolution_conformance.py index 3e8f1b319..e2c23a0f7 100644 --- a/server/python/tests/conformance/test_source_resolution_conformance.py +++ b/server/python/tests/conformance/test_source_resolution_conformance.py @@ -50,7 +50,11 @@ def test_source_resolution_conformance(case: dict, tmp_path: Path) -> None: if "expectError" in case: with pytest.raises(ParseError) as e: resolve_collection(resolve_from) - assert e.value.code.value == case["expectError"] + # A string pins the exact code; `True` only pins that resolution + # RAISES — the malformed-config error code is deliberately not + # pinned cross-port (see the corpus README). + if isinstance(case["expectError"], str): + assert e.value.code.value == case["expectError"] return # `expectFiles` is project-root-relative even when `resolveFrom` points diff --git a/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts b/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts index 6e6caef8c..843d9002f 100644 --- a/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts +++ b/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts @@ -15,7 +15,10 @@ interface Case { * `.metaobjects/config.json`. See the corpus README, "Shape". */ readonly resolveFrom?: string; readonly expectFiles?: readonly string[]; - readonly expectError?: string; + /** A string pins the exact error code raised; `true` pins only that + * resolution RAISES — the malformed-config error code is deliberately not + * pinned cross-port (see the corpus README). */ + readonly expectError?: string | true; } const CORPUS = resolve( @@ -57,13 +60,20 @@ describe("source-resolution conformance", () => { test(c.name, async () => { const { root, resolveDir } = await materialize(c); if (c.expectError !== undefined) { - let code: string | undefined; + let thrown: unknown; + let threw = false; try { await resolveCollection(resolveDir, { explicitDir: resolveDir }); } catch (e) { - code = (e as { code?: string }).code; + threw = true; + thrown = e; + } + expect(threw).toBe(true); + // A string pins the exact code; `true` only pins that it raises — see + // the `expectError` type doc above. + if (typeof c.expectError === "string") { + expect((thrown as { code?: string }).code).toBe(c.expectError); } - expect(code).toBe(c.expectError); return; } const collection = await resolveCollection(resolveDir, { explicitDir: resolveDir }); From 209d9d7d93779352ab7f5345732dda7a6dac0840 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 12:03:15 -0400 Subject: [PATCH 27/44] feat(cli): meta init --config-only Writes .metaobjects/config.json and nothing else, so a Maven- or pip-rooted project can declare its sources for the Node CLI without acquiring a TypeScript scaffold it will not use. --- .../packages/cli/src/commands/init.ts | 112 ++++++++++++------ server/typescript/packages/cli/src/index.ts | 3 + .../typescript/packages/cli/src/lib/args.ts | 3 + .../cli/test/__snapshots__/cli.test.ts.snap | 1 + .../typescript/packages/cli/test/cli.test.ts | 5 +- .../typescript/packages/cli/test/init.test.ts | 48 ++++++++ 6 files changed, 132 insertions(+), 40 deletions(-) diff --git a/server/typescript/packages/cli/src/commands/init.ts b/server/typescript/packages/cli/src/commands/init.ts index f103023f3..614433c8b 100644 --- a/server/typescript/packages/cli/src/commands/init.ts +++ b/server/typescript/packages/cli/src/commands/init.ts @@ -137,6 +137,14 @@ export interface InitOptions { wireRoot?: boolean; /** Scaffold ONLY the agent-context (always-on + skills + root wiring), skipping the metaobjects/ project scaffold — for dropping context into an existing/polyglot repo. */ docsOnly?: boolean; + /** + * Write ONLY `.metaobjects/config.json` — no TypeScript scaffold (metaobjects.config.ts, + * codegen/generators/, package.json edits, .gitignore, agent-context files, or a + * metaobjects/ directory). For a Maven- or pip-rooted project that needs the Node CLI + * (which owns `migrate` and `verify --db` under ADR-0015) to discover its metadata + * without acquiring a TypeScript project it will never use. + */ + configOnly?: boolean; } export interface InitResult { @@ -304,6 +312,53 @@ async function writeOwnedGenerators(opts: InitOptions, result: InitResult): Prom } } +/** + * .metaobjects/config.json — write fresh defaults, or preserve+merge an existing + * valid config. Shared by the full scaffold and `--config-only` so the two paths + * cannot drift on the config's default content. + */ +async function writeConfigFile(opts: InitOptions, result: InitResult, agentDir: string, agentDirExists: boolean): Promise { + const freshConfig = opts.d1 + ? ConfigSchema.parse({ ...DEFAULT_CONFIG, migrate: buildD1MigrateBlock(opts.cwd) }) + : DEFAULT_CONFIG; + if (agentDirExists) { + const configPath = join(agentDir, "config.json"); + let priorContent: string | undefined; + try { + priorContent = await readFile(configPath, "utf8"); + const parsed = ConfigSchema.parse(JSON.parse(priorContent)); + const merged = ConfigSchema.parse({ ...DEFAULT_CONFIG, ...parsed }); + // When a valid .metaobjects/config.json already exists and the user passes --force, + // we preserve the existing config and only re-scaffold support files. The --d1 flag + // only takes effect on fresh inits — retro-fitting D1 onto an existing project is + // the user's job (edit migrate.dialect and migrate.d1 in config.json directly). + await saveConfig(agentDir, merged); + result.preserved.push(".metaobjects/config.json"); + } catch { + if (priorContent !== undefined) { + log.warn("existing .metaobjects/config.json was invalid — writing fresh defaults. Prior content:"); + log.warn(priorContent); + result.warnings.push("invalid .metaobjects/config.json replaced with defaults"); + } + await writeFile( + join(agentDir, "config.json"), + JSON.stringify(freshConfig, null, 2) + "\n", + "utf8", + ); + if (priorContent === undefined) { + result.created.push(".metaobjects/config.json"); + } + } + } else { + await writeFile( + join(agentDir, "config.json"), + JSON.stringify(freshConfig, null, 2) + "\n", + "utf8", + ); + result.created.push(".metaobjects/config.json"); + } +} + export async function init(opts: InitOptions): Promise { const result: InitResult = { created: [], preserved: [], warnings: [] }; const agentDir = join(opts.cwd, DEFAULT_METAOBJECTS_DIR); @@ -319,6 +374,15 @@ export async function init(opts: InitOptions): Promise { return result; } + if (opts.configOnly) { + // Config only: write/preserve .metaobjects/config.json and nothing else — no + // metaobjects/ dir, no agent-context, no TypeScript scaffold. `agentDirExists` is + // captured before the mkdir below so an existing valid config is still preserved. + await mkdir(agentDir, { recursive: true }); + await writeConfigFile(opts, result, agentDir, agentDirExists); + return result; + } + if (opts.refreshDocs && exists) { // Refresh-only path: (re)write the agent-context docs and NOTHING else — never // the project scaffold (metaobjects/, config.json, codegen/generators/, @@ -372,45 +436,7 @@ export async function init(opts: InitOptions): Promise { } // .metaobjects/config.json - const freshConfig = opts.d1 - ? ConfigSchema.parse({ ...DEFAULT_CONFIG, migrate: buildD1MigrateBlock(opts.cwd) }) - : DEFAULT_CONFIG; - if (agentDirExists) { - const configPath = join(agentDir, "config.json"); - let priorContent: string | undefined; - try { - priorContent = await readFile(configPath, "utf8"); - const parsed = ConfigSchema.parse(JSON.parse(priorContent)); - const merged = ConfigSchema.parse({ ...DEFAULT_CONFIG, ...parsed }); - // When a valid .metaobjects/config.json already exists and the user passes --force, - // we preserve the existing config and only re-scaffold support files. The --d1 flag - // only takes effect on fresh inits — retro-fitting D1 onto an existing project is - // the user's job (edit migrate.dialect and migrate.d1 in config.json directly). - await saveConfig(agentDir, merged); - result.preserved.push(".metaobjects/config.json"); - } catch { - if (priorContent !== undefined) { - log.warn("existing .metaobjects/config.json was invalid — writing fresh defaults. Prior content:"); - log.warn(priorContent); - result.warnings.push("invalid .metaobjects/config.json replaced with defaults"); - } - await writeFile( - join(agentDir, "config.json"), - JSON.stringify(freshConfig, null, 2) + "\n", - "utf8", - ); - if (priorContent === undefined) { - result.created.push(".metaobjects/config.json"); - } - } - } else { - await writeFile( - join(agentDir, "config.json"), - JSON.stringify(freshConfig, null, 2) + "\n", - "utf8", - ); - result.created.push(".metaobjects/config.json"); - } + await writeConfigFile(opts, result, agentDir, agentDirExists); // .metaobjects/.gitignore await writeFile(join(agentDir, ".gitignore"), METAOBJECTS_GITIGNORE_BODY, "utf8"); @@ -653,6 +679,7 @@ export async function initCommand(args: string[], cwd: string): Promise noSkills: flags.noSkills, wireRoot: flags.wireRoot, docsOnly: flags.docsOnly, + configOnly: flags.configOnly, }); if (flags.printOnly) { @@ -666,6 +693,13 @@ export async function initCommand(args: string[], cwd: string): Promise log.info(`Scaffolded the MetaObjects agent context (${result.created.length} files): .metaobjects/AGENTS.md + .claude/skills/metaobjects-*.`); for (const w of result.warnings) log.info(` ${w}`); log.info("Re-run --docs-only --refresh-docs to update; --no-wire-root to skip the root CLAUDE.md @import."); + } else if (flags.configOnly) { + if (result.created.includes(".metaobjects/config.json")) { + log.info("Wrote .metaobjects/config.json — declare your metadata sources there for the Node CLI (migrate, verify --db)."); + } else { + log.info(".metaobjects/config.json already exists — left untouched."); + } + for (const w of result.warnings) log.warn(w); } else { log.info(nextStepsBlock()); // Surface any scaffold warnings (e.g. the #77 monorepo-subdir agent-context diff --git a/server/typescript/packages/cli/src/index.ts b/server/typescript/packages/cli/src/index.ts index 7150eb743..eeb6ad5df 100644 --- a/server/typescript/packages/cli/src/index.ts +++ b/server/typescript/packages/cli/src/index.ts @@ -16,6 +16,7 @@ USAGE: COMMANDS: init Scaffold a MetaObjects project in the current repo init --refresh-docs Refresh .metaobjects/AGENTS.md + CLAUDE.md after CLI upgrades + init --config-only Write only .metaobjects/config.json — for a Maven- or pip-rooted project agent-docs Scaffold only the agent-context (.metaobjects/ + .claude/skills/) — canonical redirect target for all language ports gen [...] Codegen TS targets from your declared metadata types [query] Search the metadata vocabulary (types, subtypes, @attrs) by name or description @@ -188,6 +189,8 @@ FLAGS: --print-only Print what would be written, don't write --d1 Include D1 (Cloudflare) migration config --no-wire-root Skip wiring root metaobjects.config.ts + --config-only Write only .metaobjects/config.json (no TS scaffold) — declares + metadata sources for the Node CLI from a Maven- or pip-rooted project --help, -h Print this help `, "agent-docs": `meta agent-docs — scaffold the agent-context (.metaobjects/ always-on files + .claude/skills/) diff --git a/server/typescript/packages/cli/src/lib/args.ts b/server/typescript/packages/cli/src/lib/args.ts index b67d5c620..9a14786ec 100644 --- a/server/typescript/packages/cli/src/lib/args.ts +++ b/server/typescript/packages/cli/src/lib/args.ts @@ -15,6 +15,7 @@ export interface InitFlags { noSkills: boolean; wireRoot: boolean; docsOnly: boolean; + configOnly: boolean; } export function parseInitArgs(argv: string[]): InitFlags { @@ -31,6 +32,7 @@ export function parseInitArgs(argv: string[]): InitFlags { "no-skills": { type: "boolean", default: false }, "no-wire-root": { type: "boolean", default: false }, "docs-only": { type: "boolean", default: false }, + "config-only": { type: "boolean", default: false }, }, strict: true, allowPositionals: false, @@ -46,6 +48,7 @@ export function parseInitArgs(argv: string[]): InitFlags { noSkills: !!values["no-skills"], wireRoot: !values["no-wire-root"], docsOnly: !!values["docs-only"], + configOnly: !!values["config-only"], }; } diff --git a/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap b/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap index 99fa1137c..c1efedcfc 100644 --- a/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap +++ b/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap @@ -9,6 +9,7 @@ USAGE: COMMANDS: init Scaffold a MetaObjects project in the current repo init --refresh-docs Refresh .metaobjects/AGENTS.md + CLAUDE.md after CLI upgrades + init --config-only Write only .metaobjects/config.json — for a Maven- or pip-rooted project agent-docs Scaffold only the agent-context (.metaobjects/ + .claude/skills/) — canonical redirect target for all language ports gen [...] Codegen TS targets from your declared metadata types [query] Search the metadata vocabulary (types, subtypes, @attrs) by name or description diff --git a/server/typescript/packages/cli/test/cli.test.ts b/server/typescript/packages/cli/test/cli.test.ts index b05d977b1..1a03ee9e6 100644 --- a/server/typescript/packages/cli/test/cli.test.ts +++ b/server/typescript/packages/cli/test/cli.test.ts @@ -6,7 +6,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; // wireRoot defaults TRUE (root-wiring is on by default so the scaffolded always-on actually loads; --no-wire-root opts out). -const defaultInitFlags = { force: false, quiet: false, printOnly: false, refreshDocs: false, d1: false, servers: [], clients: [], noSkills: false, wireRoot: true, docsOnly: false }; +const defaultInitFlags = { force: false, quiet: false, printOnly: false, refreshDocs: false, d1: false, servers: [], clients: [], noSkills: false, wireRoot: true, docsOnly: false, configOnly: false }; describe("parseInitArgs", () => { test("default flags (wireRoot on by default, others off)", () => { @@ -27,6 +27,9 @@ describe("parseInitArgs", () => { test("--d1 toggles d1", () => { expect(parseInitArgs(["--d1"])).toEqual({ ...defaultInitFlags, d1: true }); }); + test("--config-only toggles configOnly", () => { + expect(parseInitArgs(["--config-only"])).toEqual({ ...defaultInitFlags, configOnly: true }); + }); test("throws on unknown flag", () => { expect(() => parseInitArgs(["--foo"])).toThrow(/Unknown option '--foo'/); }); diff --git a/server/typescript/packages/cli/test/init.test.ts b/server/typescript/packages/cli/test/init.test.ts index 11ce1f68f..76caeef9b 100644 --- a/server/typescript/packages/cli/test/init.test.ts +++ b/server/typescript/packages/cli/test/init.test.ts @@ -218,6 +218,54 @@ describe("init() — monorepo-subdir agent-context warning (#77)", () => { }); }); +describe("init() --config-only", () => { + test("writes just the config, no TypeScript scaffold", async () => { + const result = await init({ cwd, configOnly: true }); + + // The one file it writes. + expect(result.created).toContain(".metaobjects/config.json"); + const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(cfg.schema_version).toBe(1); + expect(cfg.sources).toEqual([]); + + // None of the TypeScript scaffold, none of the metaobjects/ metadata dir, none + // of the agent-context — this is the whole point of the flag: a Maven- or + // pip-rooted project declares its sources for the Node CLI without acquiring a + // TS project it will not use. + for (const unwanted of [ + "metaobjects.config.ts", + "codegen/generators/entity.ts", + "package.json", + ".gitignore", + "metaobjects", + ".metaobjects/.gitignore", + ".metaobjects/AGENTS.md", + ]) { + expect(existsSync(join(cwd, unwanted))).toBe(false); + } + }); + + test("leaves an existing valid config untouched", async () => { + mkdirSync(join(cwd, ".metaobjects"), { recursive: true }); + const existing = { schema_version: 1, sources: [{ path: "model" }] }; + writeFileSync(join(cwd, ".metaobjects", "config.json"), JSON.stringify(existing)); + + const result = await init({ cwd, configOnly: true }); + + expect(result.preserved).toContain(".metaobjects/config.json"); + const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(cfg.sources).toEqual([{ path: "model" }]); + }); +}); + +describe("initCommand --config-only", () => { + test("returns 0 and writes only the config", async () => { + expect(await initCommand(["--config-only"], cwd)).toBe(0); + expect(existsSync(join(cwd, ".metaobjects", "config.json"))).toBe(true); + expect(existsSync(join(cwd, "metaobjects.config.ts"))).toBe(false); + }); +}); + describe("init --d1", () => { test("scaffolds config with migrate.dialect = 'd1' and prefilled binding from wrangler.toml", async () => { writeFileSync(join(cwd, "wrangler.toml"), [ From 3b0a7583dd13506b301c7a8a58cffa89f623bc2d Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 12:14:57 -0400 Subject: [PATCH 28/44] docs: sources is read by all four CLI surfaces Records the precedence ladder, the neutral-subset rule, why scope stays Node-only, and that file order is deliberately not a contract. --- CHANGELOG.md | 40 ++++++++++++++++++++ docs/features/metadata-sources.md | 62 ++++++++++++++++++++++++++++--- 2 files changed, 96 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 38f882257..4464fa57b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,46 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ## [Unreleased] +### Added — `sources` is read by all four CLI surfaces, plus `meta init --config-only` + +`.metaobjects/config.json`'s `sources` key stops being a Node-only concern. Adopter +guide: [`docs/features/metadata-sources.md`](docs/features/metadata-sources.md). + +- **`sources` is read by all four CLI surfaces**, not just the Node `meta` CLI — + the C#, Python and Java/Kotlin CLIs (Kotlin has no CLI of its own; it runs + through the same Maven plugin as Java) now resolve metadata from the + port-neutral `.metaobjects/config.json`, so one declaration serves every port. + Each reads a **neutral subset** (`schema_version` + `sources`) and ignores + unknown top-level keys, so the TypeScript-owned keys in that file (`migrate`, + `scope`, `extract`, and the rest) never become a four-port change. Precedence + is a ladder — explicit CLI argument, then the port's own native surface (a + pom's ``/``, Python's `metadata` key), then `sources`, + then the default `metaobjects/` directory — and a config that exists but is + malformed errors at its own rung rather than silently falling through. Gated + by the new + [`fixtures/source-resolution-conformance/`](fixtures/source-resolution-conformance/) + corpus, which every port runs. +- **`meta init --config-only`** writes `.metaobjects/config.json` and nothing + else, so a Maven- or pip-rooted project can declare its sources for the Node + CLI (which owns `migrate` and `verify --db`, ADR-0015) without acquiring a + TypeScript scaffold it will not use. +- **`scope` / `migrate.scope` stay Node-CLI-only.** Java's shipped `` + grammar uses `*` to cross the `::` separator and `@` to match one segment — + respectively `scope`'s `**` and `*`, inverted — plus `!`-prefix exclusion and + a `.[attr]` predicate `scope` cannot express at all + (`GeneratorUtil.createRegexFromGlob` carries a `TODO` conceding its own + separator handling is wrong). Both are output filters over the same resolved + file set, so reconciling them is a separate, adopter-affecting decision + rather than a mechanical port. No cross-port behavior depends on `scope`. +- **Resolved file order, and the malformed-config error code, are deliberately + NOT cross-port contracts.** The ports' directory walks already differ and + always have (Java sorts by basename, C# by full-path ordinal, Python by + basename, TypeScript walks depth-first); the corpus compares file **sets**. + A malformed config must raise rather than silently degrade to "no config", + but which error is each port's own — verified empirically: TypeScript raises + a raw `ZodError` with no code at all, Python raises + `ERR_COLLECTION_NOT_FOUND`, C# and Java both raise `ERR_BAD_ATTR_VALUE`. + ### Changed — a committed migration chain must replay from empty, and `meta migrate` stops writing chains that cannot ([#313](https://github.com/metaobjectsdev/metaobjects/issues/313)) **`meta migrate --from-db` now REFUSES a drop for a table or view the committed schema diff --git a/docs/features/metadata-sources.md b/docs/features/metadata-sources.md index 930a63044..43f787796 100644 --- a/docs/features/metadata-sources.md +++ b/docs/features/metadata-sources.md @@ -34,12 +34,62 @@ exactly that set. A `path` is read **in place and never installed** or copied. `"sources": []`, so every existing project takes the default and resolves the same files it always did. -**Port support.** `sources` / `scope` / `migrate.scope` are read by the **Node -`meta` CLI** today. The Java, Kotlin, C# and Python CLIs still take their metadata -location their own way (a Maven `` element, a positional directory, a -`metadata` config key). The cross-port pattern grammar is already pinned by -[`fixtures/scope-conformance/`](../../fixtures/scope-conformance/); wiring the other -four CLIs to the same config file is future work, not yet planned or scheduled. +**Port support.** `sources` is read by **all four CLI surfaces** — the Node `meta` +CLI, `dotnet meta` (C#), `metaobjects` (Python) and `metaobjects:generate` (Java +and Kotlin, via Maven — Kotlin has no CLI entry point of its own; it runs through +the same Maven plugin as Java). Each resolves the same files from the same +declaration; that promise is gated by +[`fixtures/source-resolution-conformance/`](../../fixtures/source-resolution-conformance/). + +Each port reaches it its own way, and the ladder is the same everywhere — first +match wins: + +1. An explicit CLI argument (a positional metadata directory, `--config`, `--cwd`). +2. The port's own native surface, where it has one — Java and Kotlin's pom + ``/``, Python's `metadata` key in `metaobjects.config.yaml`. + If the pom names either element it owns the concern outright and the neutral + file is not consulted. +3. `sources` in `.metaobjects/config.json`. +4. The built-in default — a `metaobjects/` directory beside that config. + +A config file that exists but is malformed is an error at its own rung; it never +falls through to the next one. + +The non-TypeScript ports read a **neutral subset** of that file — +`schema_version` and `sources` — and ignore every other top-level key, so the +TypeScript-owned keys beside them (`migrate`, `extract`, and the rest) never +become a four-port concern. The Node CLI remains the file's only writer; +`meta init --config-only` writes it into a Maven- or pip-rooted project without +adding a TypeScript scaffold. + +**`scope` and `migrate.scope` remain Node-CLI-only.** Java's shipped `` +grammar uses `*` to cross the `::` package separator and `@` to match one +segment — respectively `scope`'s `**` and `*`, **inverted** — plus `!`-prefix +exclusion and a `.[attr]` predicate that `scope` cannot express at all +(`GeneratorUtil.createRegexFromGlob` even carries a `TODO` conceding its own +separator handling is wrong). Both are output filters over the same resolved +file set, so they compete rather than layer, and reconciling them changes +behavior for existing Java consumers — a separate, adopter-affecting decision +rather than a mechanical port. No cross-port behavior depends on `scope`. + +**Two things are deliberately not cross-port contracts**, because the corpus +that gates this feature says so explicitly: + +- **Resolved file order.** The ports' directory walks already differ and always + have — Java sorts by basename, C# by full-path ordinal, Python by basename, + TypeScript walks depth-first with files before subdirectories — and the + loader's overlay partition discards caller order regardless + (super-resolution is order-independent, #188). Every port resolves the same + file **set**; only the order within it is each port's own. +- **The error code for a malformed config.** A `.metaobjects/config.json` that + exists but is malformed (e.g. `sources` declared as an object instead of an + array) must raise rather than silently degrade to "no config" — but which + error is each port's own. Verified empirically: TypeScript raises a raw + `ZodError` carrying no MetaObjects error code at all, Python raises + `ERR_COLLECTION_NOT_FOUND`, and C# and Java both raise `ERR_BAD_ATTR_VALUE`. + The shared corpus expresses this with `"expectError": true` ("must raise, + code unpinned") alongside the existing string-code form for cases that DO + pin one. --- From c13541ab8a891efb5a33cdb6368740d461af9bbf Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 12:25:34 -0400 Subject: [PATCH 29/44] docs(fix): the deferred list still said the other CLIs don't read config "What is deferred" claimed cross-port sources reading was still unbuilt, directly contradicting the "Port support" paragraph 590 lines earlier that this branch just corrected. Reworded the bullet to describe what is actually still deferred in that area (scope/migrate.scope staying Node-CLI-only), and added the fixtures/source-resolution-conformance/ corpus to "Verified by", which previously named only the scope corpus. --- docs/features/metadata-sources.md | 25 ++++++++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/docs/features/metadata-sources.md b/docs/features/metadata-sources.md index 43f787796..6fb6211d1 100644 --- a/docs/features/metadata-sources.md +++ b/docs/features/metadata-sources.md @@ -625,8 +625,11 @@ Phase 1 ships the spine. Explicitly **not** built yet, so you do not go looking: - **`url` sources** and **named `collection` references**. - **Per-generator declarative `scope`.** The TypeScript `filter` function is the escape hatch and is unchanged. -- **The other four ports' CLIs reading `.metaobjects/config.json`.** The pattern - grammar is corpus-gated so they cannot diverge when they land. +- **Cross-port `scope` and `migrate.scope`.** Java's own `` grammar + collides with `scope`'s pattern characters (see "Port support" near the top of + this document), so reconciling the two is a separate, adopter-affecting decision + rather than a mechanical port. They remain Node-CLI-only for now; `sources` + itself is no longer in this list — all four CLI surfaces read it today. - **Database and other runtime metadata sources.** Ruled a runtime-metadata concern rather than a build-time one. @@ -637,17 +640,33 @@ Design rationale and the full phase plan: ## Verified by +**Cross-port source resolution** + +- [`fixtures/source-resolution-conformance/`](../../fixtures/source-resolution-conformance/) — + pins the resolved file SET for a declared `sources` list (the default, the + relative-path base, recursion, extension matching, union/de-dup, and every error + condition, including the deliberately-unpinned malformed-config case — see the + corpus's own README). All four CLI surfaces run it: + `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts`, + `server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs`, + `server/python/tests/conformance/test_source_resolution_conformance.py`, and + `server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java` + (Kotlin runs through the Java/Maven loader, so it shares the Java runner). + **Cross-port pattern semantics** - [`fixtures/scope-conformance/`](../../fixtures/scope-conformance/) — 10 cases pinning `*` / `**`, include-union, exclude-after-include, literal metacharacters, and case sensitivity. TypeScript runs it today; the other four ports have no runner - yet. See [`CONFORMANCE.md`](../CONFORMANCE.md). + yet, because `scope` itself stays Node-CLI-only (see "Port support" above). See + [`CONFORMANCE.md`](../CONFORMANCE.md). **TypeScript gates** - `server/typescript/packages/sdk/test/scope.test.ts` — the pattern engine - `server/typescript/packages/sdk/test/scope-conformance.test.ts` — the corpus runner +- `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts` — the + cross-port sources corpus runner - `server/typescript/packages/sdk/test/sources.test.ts` — `path` resolution, `_pending` exclusion, unsupported kinds - `server/typescript/packages/sdk/test/discovery.test.ts` — nearest-ancestor walk and From 60d128d7405c12b3ae43aa70a338a91044a65703 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 14:09:18 -0400 Subject: [PATCH 30/44] fix(java): raise on the general malformed-sources shape, not just the one case MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit NeutralConfig.read() exempted a JSON-null `sources` from the "must be an array" raise, so `sources: null` silently fell back to the default `metaobjects/` directory instead of raising like every other wrong-typed `sources` value. The fix is the general rule (a present `sources` key that is not an array raises), not a null-specific carve-out. Also: reject a `sources` entry whose value is not a non-empty string — previously a bare number was silently stringified, so `{"path": 123}` would load a directory literally named "123" instead of failing on the typo'd config. Ports the TypeScript loader's `_pending/`-at-any-depth exclusion (previously TS-only) into both DirectorySource.expand() and SourceResolver's own directory walk, so a draft entity under `_pending/` is invisible to `mvn metaobjects:generate` the same way it already is to the Node CLI. Stale "18 of 19 cases" test comments reworded to describe the corpus structurally instead of by count, since the count had already drifted (actual was 20). --- .../com/metaobjects/config/NeutralConfig.java | 33 +++++++--- .../metaobjects/config/SourceResolver.java | 26 ++++++++ .../metaobjects/loader/DirectorySource.java | 24 +++++++ .../metaobjects/config/NeutralConfigTest.java | 62 +++++++++++++++++++ .../SourceResolutionConformanceTest.java | 23 ++++--- .../loader/DirectorySourceTest.java | 25 ++++++++ 6 files changed, 175 insertions(+), 18 deletions(-) create mode 100644 server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java index 7b618c270..0f083991f 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java @@ -117,17 +117,18 @@ public static Optional read(Path configDir) { List> specs = new ArrayList<>(); JsonElement srcs = root.get("sources"); - if (srcs != null && !srcs.isJsonNull() && !srcs.isJsonArray()) { - // A present-but-wrong-typed `sources` (e.g. a bare object instead of an - // array) must RAISE, not silently read as "absent" — the latter would - // fall back to the default directory with no diagnostic, exactly the - // "typo'd config behaves like no config" failure this class exists to - // prevent (see the class javadoc). + // A present `sources` key that is not an array must RAISE, not silently + // read as "absent" — the latter would fall back to the default directory + // with no diagnostic, exactly the "typo'd config behaves like no config" + // failure this class exists to prevent (see the class javadoc). This is + // the GENERAL rule: a present-but-JSON-null `sources` is just as wrong- + // typed as a present-but-object `sources` — it is not a special case. + if (srcs != null && !srcs.isJsonArray()) { throw new MetaDataException( path + ": \"sources\" must be an array", ErrorCode.ERR_BAD_ATTR_VALUE); } - if (srcs != null && srcs.isJsonArray()) { + if (srcs != null) { for (JsonElement el : srcs.getAsJsonArray()) { if (!el.isJsonObject()) { throw new MetaDataException( @@ -143,7 +144,23 @@ public static Optional read(Path configDir) { Map spec = new LinkedHashMap<>(); for (Map.Entry e : entry.entrySet()) { JsonElement v = e.getValue(); - spec.put(e.getKey(), v.isJsonPrimitive() ? v.getAsString() : v.toString()); + if (!v.isJsonPrimitive() || !v.getAsJsonPrimitive().isString()) { + // Every source-spec value (`path`/`resource`/`package`) is a + // string — a bare number/boolean/object/array silently + // stringified (the prior behavior) would let + // {"path": 123} load a directory literally named "123" + // rather than failing loudly on the typo'd config. + throw new MetaDataException( + path + ": \"sources\" entry \"" + e.getKey() + "\" must be a string", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + String value = v.getAsString(); + if (value.strip().isEmpty()) { + throw new MetaDataException( + path + ": \"sources\" entry \"" + e.getKey() + "\" must not be empty", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + spec.put(e.getKey(), value); } specs.add(spec); } diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java index 0788c93c8..2af732b79 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java @@ -47,6 +47,14 @@ public final class SourceResolver { private static final Set EXTENSIONS = Set.of(".json", ".yaml", ".yml"); + /** + * Directory excluded at every level of a directory-spec walk — drafts that + * are deliberately not part of the loaded model. Mirrors TypeScript's + * {@code PENDING_DIR} in {@code metadata-files.ts} and the loader's own + * {@link com.metaobjects.loader.DirectorySource}. + */ + private static final String PENDING_DIR = "_pending"; + private SourceResolver() {} /** @@ -99,6 +107,11 @@ public static List resolveSources(Path configDir, List try (Stream walk = Files.walk(target)) { walk.filter(Files::isRegularFile) .filter(p -> hasSupportedExtension(p.getFileName().toString())) + // Excludes _pending/ at ANY depth — every ancestor path + // component between `target` and `p` is checked, not + // merely `p`'s own basename, so the whole subtree is + // skipped. + .filter(p -> !isUnderPendingDir(target, p)) .sorted(Comparator.comparing(p -> p.getFileName().toString())) .forEach(p -> seen.add(p.toAbsolutePath().normalize())); } catch (IOException e) { @@ -138,6 +151,19 @@ public static List resolveCollection(Path root) { return resolveSources(base, specs); } + /** + * True when any ancestor path component between {@code base} and {@code file} + * (i.e. excluding {@code file}'s own name) is exactly {@link #PENDING_DIR}. + */ + private static boolean isUnderPendingDir(Path base, Path file) { + Path rel = base.relativize(file).getParent(); + if (rel == null) return false; + for (Path part : rel) { + if (part.toString().equals(PENDING_DIR)) return true; + } + return false; + } + private static boolean hasSupportedExtension(String name) { String lower = name.toLowerCase(Locale.ROOT); for (String ext : EXTENSIONS) { diff --git a/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java b/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java index f6268674c..50f5a43b3 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java +++ b/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java @@ -60,6 +60,13 @@ public Options setRecurse(boolean recurse) { private static final Set EXTENSIONS = Set.of(".json", ".yaml", ".yml"); + /** + * Directory excluded at every level of {@link #expand()} — drafts that are + * deliberately not part of the loaded model. Mirrors TypeScript's + * {@code PENDING_DIR} in {@code metadata-files.ts}. + */ + private static final String PENDING_DIR = "_pending"; + private final Path directory; private final Options opts; @@ -102,6 +109,10 @@ public Stream expand() { .filter(Files::isRegularFile) .filter(p -> hasSupportedExtension(p.getFileName().toString())) .filter(p -> !opts.getExclude().contains(p.getFileName().toString())) + // Excludes _pending/ at ANY depth — every ancestor path component + // between `directory` and `p` is checked, not merely `p`'s own + // basename, so the whole subtree is skipped. + .filter(p -> !isUnderPendingDir(directory, p)) .sorted(Comparator.comparing(p -> p.getFileName().toString())) .map(FileSource::new); } catch (IOException e) { @@ -116,6 +127,19 @@ public List expandToList() { return expand().map(fs -> (MetaDataSource) fs).collect(Collectors.toList()); } + /** + * True when any ancestor path component between {@code base} and {@code file} + * (i.e. excluding {@code file}'s own name) is exactly {@link #PENDING_DIR}. + */ + private static boolean isUnderPendingDir(Path base, Path file) { + Path rel = base.relativize(file).getParent(); + if (rel == null) return false; + for (Path part : rel) { + if (part.toString().equals(PENDING_DIR)) return true; + } + return false; + } + private static boolean hasSupportedExtension(String name) { String lower = name.toLowerCase(Locale.ROOT); for (String ext : EXTENSIONS) { diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java new file mode 100644 index 000000000..b9e9d9934 --- /dev/null +++ b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java @@ -0,0 +1,62 @@ +/* + * Copyright 2003 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.config; + +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; +import org.junit.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertThrows; + +/** + * Focused unit coverage for {@link NeutralConfig#read} on shapes NOT gated by the + * shared {@code source-resolution-conformance} corpus — either because the + * reference TypeScript implementation deliberately behaves differently + * (whitespace-only path) or to pin a prior defect class directly. + */ +public class NeutralConfigTest { + + private static Path writeConfig(String payloadJson) throws IOException { + Path dir = Files.createTempDirectory("neutral-config-"); + Path cfgDir = Files.createDirectories(dir.resolve(".metaobjects")); + Files.write(cfgDir.resolve("config.json"), payloadJson.getBytes(StandardCharsets.UTF_8)); + return dir; + } + + @Test + public void whitespaceOnlyPathRaises() throws IOException { + // Deliberately NOT gated by the shared cross-port corpus: the TS + // reference (`config.ts`'s `z.string().min(1)`) rejects only a + // fully-empty path, not a whitespace-only one, and the reference is out + // of scope to change here. This port is stricter on this one edge case + // by design. + Path dir = writeConfig("{ \"schema_version\": 1, \"sources\": [ { \"path\": \" \" } ] }"); + assertThrows(MetaDataException.class, () -> NeutralConfig.read(dir)); + } + + @Test + public void nonStringSourceValueRaises() throws IOException { + Path dir = writeConfig("{ \"schema_version\": 1, \"sources\": [ { \"path\": 123 } ] }"); + MetaDataException ex = assertThrows(MetaDataException.class, () -> NeutralConfig.read(dir)); + assertEquals(ErrorCode.ERR_BAD_ATTR_VALUE, ex.getCode().orElseThrow()); + } +} diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java index e64a62e85..ea27ec2ba 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java @@ -60,13 +60,15 @@ public class SourceResolutionConformanceTest { * One row per corpus case. * * @param resolveFrom Project-root-relative directory the resolver is invoked - * FROM. Defaults to {@code "."} — 18 of 19 cases leave it there, so the - * config lives at the project root and "relative to project root" vs - * "relative to the invocation directory" coincide. The one case that sets it - * ({@code a-parent-relative-path-resolves-against-the-declaring-configs- - * directory}) is the one place those two bases diverge, and both the - * config's own location AND the {@code expectFiles} comparison base below - * must honor it correctly for that case to mean anything. + * FROM. Defaults to {@code "."} — every case leaves it there EXCEPT + * {@code a-parent-relative-path-resolves-against-the-declaring-configs- + * directory}, so for all the others the config lives at the project root + * and "relative to project root" vs "relative to the invocation directory" + * coincide. That one case is the one place those two bases diverge, and + * both the config's own location AND the {@code expectFiles} comparison + * base below must honor it correctly for that case to mean anything. (Do + * not restate this as "N of M cases" — the corpus grows and a hardcoded + * count silently goes stale; the structural description above does not.) */ /** * {@code expectError}: a JSON string pins the exact error code raised; JSON @@ -173,9 +175,10 @@ public void resolvesTheSameFileSet() throws IOException { } // Compared against the PROJECT ROOT explicitly — never against - // `invokeDir`. For 18 of 19 cases the two coincide (resolveFrom "."), so a - // comparison-base bug here would pass every case except the one that sets - // `resolveFrom`, which is exactly why that case exists. + // `invokeDir`. For every case but the one that sets `resolveFrom` the two + // coincide (resolveFrom "."), so a comparison-base bug here would pass + // every other case and fail only that one — which is exactly why that + // case exists. Set got = SourceResolver.resolveCollection(invokeDir).stream() .map(p -> root.relativize(p).toString().replace('\\', '/')) .collect(Collectors.toSet()); diff --git a/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java b/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java index 5fa5314d6..405289a98 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java @@ -62,6 +62,31 @@ public class DirectorySourceTest { } } + @Test public void excludesPendingDirAtAnyDepth() throws IOException { + // Mirrors TypeScript's PENDING_DIR exclusion (metadata-files.ts) — a draft + // entity under _pending/ must be invisible to codegen, not merely a file + // that happens to be NAMED "_pending". Before this fix, only TypeScript + // knew about this directory; a draft would generate a live table under + // `mvn metaobjects:generate`. + Path dir = Files.createTempDirectory("ds-"); + try { + Files.writeString(dir.resolve("meta.live.json"), "{}"); + Path pending = Files.createDirectory(dir.resolve("_pending")); + Files.writeString(pending.resolve("meta.draft.json"), "{}"); + // Nested: _pending/ excluded at ANY depth, not just top-level. + Path nestedPending = Files.createDirectories(dir.resolve("nested").resolve("_pending")); + Files.writeString(nestedPending.resolve("meta.deep-draft.json"), "{}"); + + DirectorySource src = new DirectorySource(dir); + List expanded = src.expand().collect(Collectors.toList()); + + assertEquals(1, expanded.size()); + assertEquals("meta.live.json", expanded.get(0).getId()); + } finally { + deleteRecursively(dir); + } + } + @Test public void nonRecursiveSkipsSubdirectories() throws IOException { Path dir = Files.createTempDirectory("ds-"); try { From fb29c44b4f284d1e113b80ae31558e3c863a191a Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 14:09:30 -0400 Subject: [PATCH 31/44] fix(csharp): raise on the general malformed-sources shape, not just the one case MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit NeutralConfig.Read() had the same null-exemption gap as Java: a present `sources: null` read as absent and fell back to the default directory. Fixed as the general rule, collapsing the two separate `TryGetProperty("sources")` calls into one guarded block in the process. Also: reject a non-string `sources` entry value (previously silently stringified via GetRawText(), so a bare number would load a directory named after its digits); accept `schema_version: 1.0` by comparing as a double instead of GetInt32(), which threw a raw, uncoded FormatException on any float-looking literal — the other three ports already accept it as equal to 1; and port the `_pending/`-at-any-depth exclusion into DirectorySource (shared by both the loader and SourceResolver), so a draft entity is invisible to `dotnet meta gen` the same way it already is to the Node CLI. `dotnet meta`'s CLI: a single declared `sources` entry that resolves to a FILE (rather than a directory) now refuses with a diagnostic naming the actual limit — the loader takes only a directory — instead of failing deep inside DirectorySource with an opaque, uncoded ERR_UNKNOWN. Mirrors the existing clear refusal for a multi-entry `sources`. Stale "18 of 19 cases" test comment reworded to describe the corpus structurally instead of by count, since the count had already drifted. --- .../MetadataDirFallbackTests.cs | 25 ++++++ server/csharp/MetaObjects.Cli/Program.cs | 19 ++++- .../DirectorySourceTests.cs | 31 ++++++++ .../NeutralConfigTests.cs | 77 +++++++++++++++++++ .../SourceResolutionConformanceTests.cs | 23 +++--- .../MetaObjects/Config/NeutralConfig.cs | 45 +++++++---- .../MetaObjects/Loader/DirectorySource.cs | 23 +++++- 7 files changed, 218 insertions(+), 25 deletions(-) create mode 100644 server/csharp/MetaObjects.Conformance.Tests/NeutralConfigTests.cs diff --git a/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs b/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs index 13a53dd7c..6882cae1e 100644 --- a/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs +++ b/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs @@ -89,6 +89,31 @@ public void Gen_with_no_positional_metadataDir_and_multiple_declared_sources_ref Assert.False(Directory.Exists(outDir), "must not silently generate from just one of several declared sources"); } + [Fact] + public void Gen_with_no_positional_metadataDir_and_a_single_FILE_source_refuses_clearly() + { + // Before this fix, a single declared `path` source resolving to a FILE + // (rather than a directory) was handed straight to MetaDataLoader.FromDirectory, + // which fails deep inside DirectorySource with an opaque, uncoded ERR_UNKNOWN — + // never naming the actual limit (this CLI's loader takes only a directory). + var vendorDir = Path.Combine(_tmp, "vendor"); + Directory.CreateDirectory(vendorDir); + File.WriteAllText(Path.Combine(vendorDir, "meta.catalog.json"), """{ "metadata.root": { "children": [] } }"""); + var cfgDir = Path.Combine(_tmp, ".metaobjects"); + Directory.CreateDirectory(cfgDir); + File.WriteAllText( + Path.Combine(cfgDir, "config.json"), + """{ "schema_version": 1, "sources": [ { "path": "vendor/meta.catalog.json" } ] }"""); + + var outDir = Path.Combine(_tmp, "generated"); + var (exitCode, _, stderr) = RunCli(_tmp, "gen", "--out", outDir, "--namespace", "X"); + + Assert.Equal(2, exitCode); + Assert.Contains("is a FILE", stderr); + Assert.DoesNotContain("ERR_UNKNOWN", stderr); + Assert.False(Directory.Exists(outDir)); + } + [Fact] public void Gen_with_an_explicit_positional_metadataDir_is_unaffected() { diff --git a/server/csharp/MetaObjects.Cli/Program.cs b/server/csharp/MetaObjects.Cli/Program.cs index c81df628f..f643f0e5d 100644 --- a/server/csharp/MetaObjects.Cli/Program.cs +++ b/server/csharp/MetaObjects.Cli/Program.cs @@ -210,7 +210,24 @@ static string ResolveMetadataDirOrExit(string? metadataDir) MetaObjects.Config.SourceResolver.ResolveSources(cwd, specs); var rawPath = specs[0]["path"]; // guaranteed present: ResolveSources above // would already have thrown otherwise. - return Path.IsPathRooted(rawPath) ? rawPath : Path.GetFullPath(Path.Combine(cwd, rawPath)); + var resolved = Path.IsPathRooted(rawPath) ? rawPath : Path.GetFullPath(Path.Combine(cwd, rawPath)); + + if (!Directory.Exists(resolved)) + { + // ResolveSources above already proved `resolved` exists, so this means + // it is a FILE. MetaDataLoader.FromDirectory below takes a directory — + // handing it a file path used to fail deep inside DirectorySource with + // an opaque ERR_UNKNOWN instead of naming the actual limit. Refuse + // clearly here instead, the same way the multi-source branch above does. + Console.Error.WriteLine( + $"error: {cwd}: .metaobjects/config.json's single \"sources\" entry (\"{rawPath}\") is a FILE, " + + "but this CLI's loader only accepts a directory source. Pass explicitly, or point " + + "\"sources\" at the file's containing directory."); + Environment.Exit(2); + throw new InvalidOperationException("unreachable"); + } + + return resolved; } catch (MetaObjects.MetaModelException e) { diff --git a/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs b/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs index 2a6467697..edcbf0343 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs @@ -34,6 +34,37 @@ public void Expand_ReturnsFileSourcesSortedByOrdinalName() } } + [Fact] + public void Expand_Excludes_PendingDir_AtAnyDepth() + { + // Mirrors TypeScript's PENDING_DIR exclusion (metadata-files.ts) — a draft + // entity under _pending/ must be invisible to codegen, not merely a file + // that happens to be NAMED "_pending". Before this fix, only TypeScript + // knew about this directory; a draft would generate a live table under + // `dotnet meta gen`. + string dir = Path.Combine(Path.GetTempPath(), "ds_" + Path.GetRandomFileName()); + Directory.CreateDirectory(dir); + try + { + File.WriteAllText(Path.Combine(dir, "meta.live.json"), "{}"); + Directory.CreateDirectory(Path.Combine(dir, "_pending")); + File.WriteAllText(Path.Combine(dir, "_pending", "meta.draft.json"), "{}"); + // Nested: _pending/ excluded at ANY depth, not just top-level. + Directory.CreateDirectory(Path.Combine(dir, "nested", "_pending")); + File.WriteAllText(Path.Combine(dir, "nested", "_pending", "meta.deep-draft.json"), "{}"); + + var src = new DirectorySource(dir); + var expanded = src.Expand().ToList(); + + Assert.Single(expanded); + Assert.Equal("meta.live.json", expanded[0].Id); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } + [Fact] public void Expand_HonorsExcludeGlobs() { diff --git a/server/csharp/MetaObjects.Conformance.Tests/NeutralConfigTests.cs b/server/csharp/MetaObjects.Conformance.Tests/NeutralConfigTests.cs new file mode 100644 index 000000000..81823c55e --- /dev/null +++ b/server/csharp/MetaObjects.Conformance.Tests/NeutralConfigTests.cs @@ -0,0 +1,77 @@ +using System.IO; +using MetaObjects.Config; +using Xunit; + +namespace MetaObjects.Conformance.Tests; + +///

+/// Focused unit coverage for on shapes that are +/// NOT gated by the shared source-resolution-conformance corpus — either +/// because the reference TypeScript implementation deliberately behaves +/// differently (whitespace-only path) or to pin a specific prior defect +/// directly against this port's own code (the schema_version float parse). +/// +public sealed class NeutralConfigTests +{ + private static string WriteConfig(string payloadJson) + { + var dir = Path.Combine(Path.GetTempPath(), "neutral-config-" + System.Guid.NewGuid().ToString("N")); + var cfgDir = Path.Combine(dir, ".metaobjects"); + Directory.CreateDirectory(cfgDir); + File.WriteAllText(Path.Combine(cfgDir, "config.json"), payloadJson); + return dir; + } + + [Fact] + public void WhitespaceOnlyPath_Raises() + { + // Deliberately NOT gated by the shared cross-port corpus: the TS + // reference (`config.ts`'s `z.string().min(1)`) rejects only a + // fully-empty path, not a whitespace-only one, and the reference is + // out of scope to change here. This port is stricter on this one + // edge case by design. + var dir = WriteConfig("""{ "schema_version": 1, "sources": [ { "path": " " } ] }"""); + try + { + Assert.ThrowsAny(() => NeutralConfig.Read(dir)); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } + + [Fact] + public void SchemaVersionAsFloatLiteral_IsAcceptedLikeTheOtherThreePorts() + { + // Regression pin: `schema_version: 1.0` used to throw a raw, uncoded + // FormatException from GetInt32() — the other three ports all accept a + // float-looking `1.0` as equal to the supported version `1`. + var dir = WriteConfig("""{ "schema_version": 1.0, "sources": [ { "path": "model" } ] }"""); + try + { + var cfg = NeutralConfig.Read(dir); + Assert.NotNull(cfg); + Assert.Single(cfg!.Sources); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } + + [Fact] + public void SchemaVersionNonIntegral_RaisesCodedError_NotFormatException() + { + var dir = WriteConfig("""{ "schema_version": 1.5, "sources": [] }"""); + try + { + var ex = Assert.ThrowsAny(() => NeutralConfig.Read(dir)); + Assert.Equal(ErrorCode.ERR_BAD_ATTR_VALUE, ex.Code); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } +} diff --git a/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs b/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs index 5590eeec7..8bb6fefc9 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs @@ -16,13 +16,15 @@ private sealed record Case( Dictionary Tree, JsonElement? Config, // Project-root-relative directory the resolver is invoked FROM. Defaults - // to "." — 18 of 19 cases leave it there, so the config lives at the - // project root and "relative to project root" vs "relative to the - // invocation directory" coincide. The one case that sets it - // ("a-parent-relative-path-resolves-against-the-declaring-configs- - // directory") is the one place those two bases diverge, and both the - // config's own location AND the expectFiles comparison base below must - // honor it correctly for that case to mean anything. + // to "." — every case leaves it there EXCEPT + // "a-parent-relative-path-resolves-against-the-declaring-configs-directory", + // so for all the others the config lives at the project root and + // "relative to project root" vs "relative to the invocation directory" + // coincide. That one case is the one place those two bases diverge, and + // both the config's own location AND the expectFiles comparison base + // below must honor it correctly for that case to mean anything. (Do not + // restate this as "N of M cases" — the corpus grows and a hardcoded + // count silently goes stale; the structural description above does not.) string ResolveFrom, string[]? ExpectFiles, // A JSON string pins the exact error code raised; JSON `true` pins only @@ -118,9 +120,10 @@ public void ResolvesTheSameFileSet(string name) } // Compared against the PROJECT ROOT explicitly — never against - // `invokeDir`. For 18 of 19 cases the two coincide (resolveFrom "."), - // so a comparison base bug here would pass every case except the one - // that sets `resolveFrom`, which is exactly why that case exists. + // `invokeDir`. For every case but the one that sets `resolveFrom` the + // two coincide (resolveFrom "."), so a comparison base bug here would + // pass every other case and fail only that one — which is exactly why + // that case exists. var got = SourceResolver.ResolveCollection(invokeDir) .Select(f => Path.GetRelativePath(root, f).Replace(Path.DirectorySeparatorChar, '/')) .ToHashSet(); diff --git a/server/csharp/MetaObjects/Config/NeutralConfig.cs b/server/csharp/MetaObjects/Config/NeutralConfig.cs index 5bfb475f9..4f73255ac 100644 --- a/server/csharp/MetaObjects/Config/NeutralConfig.cs +++ b/server/csharp/MetaObjects/Config/NeutralConfig.cs @@ -59,7 +59,14 @@ public sealed class NeutralConfig if (!root.TryGetProperty("schema_version", out var v) || v.ValueKind != JsonValueKind.Number || - v.GetInt32() != SupportedSchemaVersion) + // Compare as a double, not GetInt32(): a float-looking literal like + // `1.0` is valid JSON and the other three ports all accept it as + // equal to 1 — GetInt32() used to throw a raw, uncoded + // FormatException on it instead. TryGetDouble never throws and a + // genuinely non-integral value (e.g. `1.5`) still correctly fails + // the equality check below. + !v.TryGetDouble(out var schemaVersion) || + schemaVersion != SupportedSchemaVersion) { throw new MetaModelException( $"{path}: unsupported schema_version (expected {SupportedSchemaVersion})", @@ -67,18 +74,18 @@ public sealed class NeutralConfig } var specs = new List>(); - if (root.TryGetProperty("sources", out var srcs) && srcs.ValueKind != JsonValueKind.Null - && srcs.ValueKind != JsonValueKind.Array) - { - // A present-but-wrong-typed `sources` (e.g. a bare object instead of an - // array) must RAISE, not silently read as "absent" — the latter would - // fall back to the default directory with no diagnostic, exactly the - // "typo'd config behaves like no config" failure this class exists to - // prevent (see the file header). - throw new MetaModelException($"{path}: \"sources\" must be an array", ErrorCode.ERR_BAD_ATTR_VALUE); - } - if (root.TryGetProperty("sources", out srcs) && srcs.ValueKind == JsonValueKind.Array) + if (root.TryGetProperty("sources", out var srcs)) { + // A present `sources` key that is not an array must RAISE, not + // silently read as "absent" — the latter would fall back to the + // default directory with no diagnostic, exactly the "typo'd config + // behaves like no config" failure this class exists to prevent (see + // the file header). This is the GENERAL rule: a present-but-JSON- + // null `sources` is just as wrong-typed as a present-but-object + // `sources` — it is not a special case. + if (srcs.ValueKind != JsonValueKind.Array) + throw new MetaModelException($"{path}: \"sources\" must be an array", ErrorCode.ERR_BAD_ATTR_VALUE); + foreach (var s in srcs.EnumerateArray()) { if (s.ValueKind != JsonValueKind.Object) @@ -86,7 +93,19 @@ public sealed class NeutralConfig var d = new Dictionary(); foreach (var p in s.EnumerateObject()) - d[p.Name] = p.Value.ValueKind == JsonValueKind.String ? p.Value.GetString()! : p.Value.GetRawText(); + { + // Every source-spec value (`path`/`resource`/`package`) is a + // string — silently stringifying a non-string (the prior + // behavior) would let {"path": 123} load a directory + // literally named "123" rather than failing loudly on the + // typo'd config. + if (p.Value.ValueKind != JsonValueKind.String) + throw new MetaModelException($"{path}: \"sources\" entry \"{p.Name}\" must be a string", ErrorCode.ERR_BAD_ATTR_VALUE); + var value = p.Value.GetString()!; + if (string.IsNullOrWhiteSpace(value)) + throw new MetaModelException($"{path}: \"sources\" entry \"{p.Name}\" must not be empty", ErrorCode.ERR_BAD_ATTR_VALUE); + d[p.Name] = value; + } if (d.Count != 1) throw new MetaModelException($"{path}: each \"sources\" entry must have exactly one key", ErrorCode.ERR_BAD_ATTR_VALUE); diff --git a/server/csharp/MetaObjects/Loader/DirectorySource.cs b/server/csharp/MetaObjects/Loader/DirectorySource.cs index a893069a5..01b4ad7db 100644 --- a/server/csharp/MetaObjects/Loader/DirectorySource.cs +++ b/server/csharp/MetaObjects/Loader/DirectorySource.cs @@ -29,6 +29,11 @@ public sealed class Options private static readonly HashSet _supportedExtensions = new(StringComparer.OrdinalIgnoreCase) { ".json", ".yaml", ".yml" }; + /// Directory excluded at every level of — drafts that are + /// deliberately not part of the loaded model. Mirrors TypeScript's + /// `PENDING_DIR` in `metadata-files.ts`. + private const string PendingDir = "_pending"; + /// The directory being scanned. public string Directory { get; } @@ -53,7 +58,11 @@ public IEnumerable Expand() : SearchOption.TopDirectoryOnly; IEnumerable files = System.IO.Directory.EnumerateFiles(Directory, "*", search) - .Where(p => _supportedExtensions.Contains(Path.GetExtension(p))); + .Where(p => _supportedExtensions.Contains(Path.GetExtension(p))) + // Excludes _pending/ at ANY depth — every ancestor path component + // between `Directory` and the file is checked, not merely the file's + // own name, so the whole subtree is skipped. + .Where(p => !IsUnderPendingDir(Directory, p)); if (Opts.Exclude is { Count: > 0 } excludes) { @@ -65,6 +74,18 @@ public IEnumerable Expand() .Select(p => new FileSource(p)); } + /// True when any ancestor path component between and + /// (i.e. excluding the file's own name) is + /// exactly . + private static bool IsUnderPendingDir(string root, string filePath) + { + var relative = Path.GetRelativePath(root, filePath); + var dir = Path.GetDirectoryName(relative); + if (string.IsNullOrEmpty(dir)) return false; + return dir.Split(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar) + .Any(part => part == PendingDir); + } + private static bool MatchesGlob(string name, string pattern) { // Minimal glob: literal match, or single leading '*' / trailing '*' wildcard. From df022c3ecb9ec96bedfc25a4e27a69b7be6fcbc6 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 14:09:41 -0400 Subject: [PATCH 32/44] fix(python): validate every source-spec value, not just its key count; wire docs into the ladder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `sources` entry values were checked for key count (`len(s) == 1`) but never for TYPE or emptiness — a non-string value (e.g. `{"path": 123}`) reached `Path()` downstream and raised an uncaught TypeError instead of the coded ParseError every other malformed shape raises, and an empty or whitespace-only `path` resolved to the config-holding directory itself (loading the whole project tree, including node_modules-equivalent content) rather than failing on the typo'd config. `sources: null` already raised correctly here (`.get("sources", [])`'s default only applies when the key is ABSENT), so this port only needed the value-level check. Ports the `_pending/`-at-any-depth exclusion into both the loader's DirectorySource and the resolver's own walk, matching TypeScript. `metaobjects docs` declared `metadata_dir` as a REQUIRED positional, so the neutral `.metaobjects/config.json` `sources` rung this feature adds (and the port's own `metaobjects.config.yaml` rung) was unreachable from it even though `gen` and `verify --codegen` could already reach both. Made it optional and routed it through the same ladder `gen` uses. The conformance runner now asserts the corpus is non-empty, mirroring the TS runner's guard — `@pytest.mark.parametrize` over an empty list reports a SKIP, not a failure, so a corpus that silently lost its cases would previously report green with nothing checked. --- server/python/src/metaobjects/cli.py | 43 +++++++++++++++++-- .../src/metaobjects/config/neutral_config.py | 18 +++++++- .../src/metaobjects/config/source_resolver.py | 19 ++++++-- .../loader/sources/directory_source.py | 11 +++++ .../tests/config/test_neutral_config.py | 27 ++++++++++++ .../test_source_resolution_conformance.py | 38 ++++++++++++++++ server/python/tests/unit/test_sources.py | 19 ++++++++ 7 files changed, 167 insertions(+), 8 deletions(-) diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index 9af17f956..4db6b2647 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -503,7 +503,35 @@ def _cmd_docs(args: argparse.Namespace) -> int: providers, providers_ok = _providers_from_args(args) if not providers_ok: return 1 - root, errors = _load_root(args.metadata_dir, providers=providers) + + if args.metadata_dir is None: + # Mirrors `gen`'s no-positional routing (#267): rung 2 (`metadata` in + # `metaobjects.config.yaml`, if one is found), else rungs 3-4 via + # `resolve_metadata_location` (the neutral `.metaobjects/config.json` + # `sources` key this feature adds, else the built-in default + # directory). Before this, `metadata_dir` was a REQUIRED positional + # here, so this ladder was unreachable from `docs` even though `gen` + # and `verify --codegen` could both already reach it. + root_dir = Path.cwd() + config: ProjectConfig | None = None + config_path = _find_config(args) + if config_path is not None: + try: + config = load_project_config(config_path) + except ConfigError as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + try: + paths = resolve_metadata_location(explicit=None, config=config, root=root_dir) + except ParseError as exc: + print(f"error: could not resolve metadata location: {exc}", file=sys.stderr) + return 1 + root, errors = _load_root_from_paths(paths, providers=providers) + project_default = root_dir.name + else: + root, errors = _load_root(args.metadata_dir, providers=providers) + project_default = Path(args.metadata_dir).resolve().name + if root is None: print("error: failed to load metadata:", file=sys.stderr) for msg in errors: @@ -511,7 +539,7 @@ def _cmd_docs(args: argparse.Namespace) -> int: return 1 api_subdir = args.api_subdir or _DOCS_DEFAULT_API_SUBDIR - project = args.project or Path(args.metadata_dir).resolve().name + project = args.project or project_default model_base_url = getattr(args, "model_base_url", None) model = PythonApiModelBuilder().build(root, project) @@ -1416,7 +1444,16 @@ def _build_parser() -> argparse.ArgumentParser: "surface of the cross-port SDK-docs contract) under --out" ), ) - docs.add_argument("metadata_dir", help="directory of metadata JSON/YAML files") + # Optional, like `gen`'s: an omitted positional falls through the same + # location ladder (`metaobjects.config.yaml`'s `metadata` key, else the + # neutral `.metaobjects/config.json` `sources`, else the default directory) + # instead of hard-requiring the caller to name a directory. + docs.add_argument( + "metadata_dir", + nargs="?", + default=None, + help="directory of metadata JSON/YAML files (omit to use the config ladder)", + ) docs.add_argument( "--out", required=True, diff --git a/server/python/src/metaobjects/config/neutral_config.py b/server/python/src/metaobjects/config/neutral_config.py index 351bb69ac..b1075ba0e 100644 --- a/server/python/src/metaobjects/config/neutral_config.py +++ b/server/python/src/metaobjects/config/neutral_config.py @@ -55,12 +55,26 @@ def read_neutral_config(config_dir: Path) -> NeutralConfig | None: code=ErrorCode.ERR_COLLECTION_NOT_FOUND, ) + # `.get("sources", [])` only applies the [] default when the key is ABSENT — + # a present `sources: null` returns None here, which correctly fails the + # `isinstance(sources, list)` check below rather than silently reading as + # "absent" and falling back to the default directory with no diagnostic. sources = raw.get("sources", []) if not isinstance(sources, list) or not all( - isinstance(s, dict) and len(s) == 1 for s in sources + isinstance(s, dict) + and len(s) == 1 + # Every source-spec value (`path`/`resource`/`package`) must be a + # non-empty (after stripping whitespace) string — a bare number/ + # boolean/null would otherwise reach `Path()` downstream and raise an + # uncaught TypeError instead of this coded error, and an empty or + # whitespace-only `path` would resolve to the config-holding directory + # itself rather than failing loudly on the typo'd config. + and all(isinstance(v, str) and v.strip() for v in s.values()) + for s in sources ): raise ParseError( - f"{path}: 'sources' must be an array of single-key objects", + f"{path}: 'sources' must be an array of single-key objects, each " + "value a non-empty string", code=ErrorCode.ERR_COLLECTION_NOT_FOUND, ) diff --git a/server/python/src/metaobjects/config/source_resolver.py b/server/python/src/metaobjects/config/source_resolver.py index 0bfbc7b68..c1752c8df 100644 --- a/server/python/src/metaobjects/config/source_resolver.py +++ b/server/python/src/metaobjects/config/source_resolver.py @@ -8,16 +8,29 @@ _SUPPORTED_SUFFIXES = (".json", ".yaml", ".yml") +#: Directory excluded at every level of `_list_metadata_files` — drafts that +#: are deliberately not part of the loaded model. Mirrors TypeScript's +#: `PENDING_DIR` in `metadata-files.ts` and the loader's own +#: `DirectorySource` (`loader/sources/directory_source.py`). +_PENDING_DIR = "_pending" + def _list_metadata_files(directory: Path) -> list[Path]: """Recursively list metadata files under ``directory``. Mirrors `DirectorySource`'s extension set (`.json`/`.yaml`/`.yml`, - case-insensitive). Order is this port's own and is deliberately NOT a - cross-port contract — see the corpus README. + case-insensitive) AND its `_pending/`-at-any-depth exclusion. Order is + this port's own and is deliberately NOT a cross-port contract — see the + corpus README. """ return sorted( - (p for p in directory.rglob("*") if p.is_file() and p.suffix.lower() in _SUPPORTED_SUFFIXES), + ( + p + for p in directory.rglob("*") + if p.is_file() + and p.suffix.lower() in _SUPPORTED_SUFFIXES + and _PENDING_DIR not in p.relative_to(directory).parts[:-1] + ), key=lambda p: p.name, ) diff --git a/server/python/src/metaobjects/loader/sources/directory_source.py b/server/python/src/metaobjects/loader/sources/directory_source.py index afc609856..ed1a2e1a3 100644 --- a/server/python/src/metaobjects/loader/sources/directory_source.py +++ b/server/python/src/metaobjects/loader/sources/directory_source.py @@ -15,6 +15,11 @@ _SUPPORTED_SUFFIXES = (".json", ".yaml", ".yml") +#: Directory excluded at every level of a recursive expand() — drafts that are +#: deliberately not part of the loaded model. Mirrors TypeScript's +#: `PENDING_DIR` in `metadata-files.ts`. +_PENDING_DIR = "_pending" + class DirectorySource: """Expands a directory into a sorted, filtered list of FileSource objects.""" @@ -44,6 +49,12 @@ def expand(self) -> Iterator[FileSource]: if p.is_file() and p.suffix.lower() in _SUPPORTED_SUFFIXES and p.name not in self._exclude + # Excludes _pending/ at ANY depth — a directory NAME check on every + # ancestor component between `self._directory` and `p`, not merely + # a basename filter on `p` itself, so the whole subtree is skipped + # (a draft entity must be invisible to codegen, not just a file + # that happens to be named "_pending"). + and _PENDING_DIR not in p.relative_to(self._directory).parts[:-1] ), key=lambda p: p.name, ) diff --git a/server/python/tests/config/test_neutral_config.py b/server/python/tests/config/test_neutral_config.py index 2ca5c3599..2f9f58e64 100644 --- a/server/python/tests/config/test_neutral_config.py +++ b/server/python/tests/config/test_neutral_config.py @@ -63,3 +63,30 @@ def test_wrong_schema_version_raises(tmp_path: Path) -> None: with pytest.raises(ParseError) as e: read_neutral_config(tmp_path) assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND + + +def test_null_sources_raises(tmp_path: Path) -> None: + # A present `sources: null` is present-but-wrong-typed, same as a bare + # object — it must not silently read as "absent" (see the shared + # source-resolution-conformance corpus for the cross-port pin of this). + _write_config(tmp_path, {"schema_version": 1, "sources": None}) + with pytest.raises(ParseError): + read_neutral_config(tmp_path) + + +def test_whitespace_only_path_raises(tmp_path: Path) -> None: + # Deliberately NOT gated by the shared cross-port corpus: the TS reference + # (`config.ts`'s `z.string().min(1)`) rejects only a fully-empty path, not + # a whitespace-only one, and the reference is out of scope to change here. + # This port is stricter on this one edge case by design. + _write_config(tmp_path, {"schema_version": 1, "sources": [{"path": " "}]}) + with pytest.raises(ParseError): + read_neutral_config(tmp_path) + + +def test_non_string_source_value_raises(tmp_path: Path) -> None: + # A bare number must fail here, at the config-read boundary, rather than + # reaching `Path()` downstream and raising an uncaught TypeError. + _write_config(tmp_path, {"schema_version": 1, "sources": [{"path": 123}]}) + with pytest.raises(ParseError): + read_neutral_config(tmp_path) diff --git a/server/python/tests/conformance/test_source_resolution_conformance.py b/server/python/tests/conformance/test_source_resolution_conformance.py index e2c23a0f7..ae04a672a 100644 --- a/server/python/tests/conformance/test_source_resolution_conformance.py +++ b/server/python/tests/conformance/test_source_resolution_conformance.py @@ -23,6 +23,18 @@ _CASES = json.loads(_CORPUS.read_text())["cases"] +def test_corpus_is_non_empty() -> None: + """A silent zero-case run is a failed gate, not a pass. + + `@pytest.mark.parametrize` over an empty list simply collects zero tests — + pytest reports that as a SKIP, not a failure, so a corpus that quietly lost + its cases (e.g. a bad path, a JSON-parsing bug) would report green here + with nothing actually checked. Mirrors the TS runner's identically-named + guard (`source-resolution-conformance.test.ts`). + """ + assert len(_CASES) > 0 + + def _materialize(case: dict, root: Path) -> Path: """Materialize ``tree`` under ``root`` and ``config`` (when present) under the directory named by ``resolveFrom`` (project root when absent). Returns the @@ -112,3 +124,29 @@ def test_explicit_relative_metadata_dir_resolves_against_cwd( assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in got} == { "sub/model/meta.a.json" } + + +def test_docs_with_no_positional_falls_back_to_neutral_config( + tmp_path: Path, monkeypatch +) -> None: + """`docs` used to declare `metadata_dir` as a REQUIRED positional, so the + neutral `.metaobjects/config.json` `sources` rung was unreachable from it + even though `gen` and `verify --codegen` could already reach it. A bare + `metaobjects docs --out ` must resolve metadata via the same ladder + `gen` uses. + """ + (tmp_path / "model").mkdir() + (tmp_path / "model" / "meta.a.json").write_text('{"metadata.root":{"children":[]}}') + d = tmp_path / ".metaobjects" + d.mkdir() + (d / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + + from metaobjects.cli import main + + monkeypatch.chdir(tmp_path) + out = tmp_path / "out" + rc = main(["docs", "--out", str(out)]) + assert rc == 0 + assert (out / "api" / "python" / "README.md").exists() diff --git a/server/python/tests/unit/test_sources.py b/server/python/tests/unit/test_sources.py index 905dfb48e..4018a8873 100644 --- a/server/python/tests/unit/test_sources.py +++ b/server/python/tests/unit/test_sources.py @@ -98,6 +98,25 @@ def test_directory_source_non_recursive(tmp_path: Path) -> None: assert ids == ["top.json"] +def test_directory_source_excludes_pending_dir_at_any_depth(tmp_path: Path) -> None: + # Mirrors TypeScript's PENDING_DIR exclusion (metadata-files.ts) — a draft + # entity under _pending/ must be invisible to codegen, not merely a file + # that happens to be NAMED "_pending". Before this fix, only TypeScript + # knew about this directory; a draft would generate a live table under + # `metaobjects gen`. + (tmp_path / "meta.live.json").write_text("{}", encoding="utf-8") + pending = tmp_path / "_pending" + pending.mkdir() + (pending / "meta.draft.json").write_text("{}", encoding="utf-8") + # Nested: _pending/ excluded at ANY depth, not just top-level. + nested_pending = tmp_path / "nested" / "_pending" + nested_pending.mkdir(parents=True) + (nested_pending / "meta.deep-draft.json").write_text("{}", encoding="utf-8") + + ids = [s.id for s in DirectorySource(tmp_path).expand()] + assert ids == ["meta.live.json"] + + def test_uri_source_file_scheme_reads_content(tmp_path: Path) -> None: p = tmp_path / "x.json" p.write_text("{}", encoding="utf-8") From 4e0255e877a244f730f30c9374855bb6c315a368 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 14:09:51 -0400 Subject: [PATCH 33/44] test(conformance): pin the malformed-sources shapes a port-specific fix could over-fit to MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The corpus had exactly one malformed-config case (a bare object instead of an array for `sources`) — every fix that followed addressed that ONE shape rather than the general "a present sources key that is not a valid array of single-key string-valued specs raises" rule, which is how a JSON-null `sources` and a non-string entry value both shipped as separate, independently-discovered defects. Adds five cases: `sources: null`, an empty `path`, a `sources` entry with two keys, an unsupported `schema_version`, and a non-string entry value. The two decoy-bearing cases (`sources-null-is-an-error-not-the-default`, `a-non-string-source-value-is-an-error`) follow `sources-must-be-an-array-not-an-object`'s pattern: a stale file sits where a wrongly-permissive port would silently resolve to it instead of raising, so the case can tell "raised correctly" apart from "raised because the decoy directory doesn't exist either." Verified empirically against each port's pre-fix code that every case that pins a NEW behavior actually fails without the corresponding fix, and passes with it. A whitespace-only `path` is deliberately NOT in this shared corpus: the TypeScript reference (`z.string().min(1)`) rejects only a fully-empty path, not a whitespace-only one, and the reference is out of scope to change here — Python/C#/Java pin that stricter behavior in their own port-specific tests instead, where it can't be mistaken for a cross-port promise. --- .../source-resolution-conformance/cases.json | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/fixtures/source-resolution-conformance/cases.json b/fixtures/source-resolution-conformance/cases.json index 30b54f872..d93655a86 100644 --- a/fixtures/source-resolution-conformance/cases.json +++ b/fixtures/source-resolution-conformance/cases.json @@ -200,6 +200,51 @@ }, "config": { "schema_version": 1, "sources": { "path": "custom" } }, "expectError": true + }, + { + "name": "sources-null-is-an-error-not-the-default", + "tree": { + "metaobjects/meta.stale.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": null }, + "expectError": true + }, + { + "name": "an-empty-path-is-an-error", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "" }] }, + "expectError": true + }, + { + "name": "a-sources-entry-with-two-keys-is-an-error", + "tree": { + "metaobjects/meta.stale.json": "{\"metadata.root\":{\"children\":[]}}", + "custom/meta.real.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "custom", "resource": "com/acme/model" }] + }, + "expectError": true + }, + { + "name": "an-unsupported-schema-version-is-an-error", + "tree": { + "metaobjects/meta.stale.json": "{\"metadata.root\":{\"children\":[]}}", + "model/meta.real.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 2, "sources": [{ "path": "model" }] }, + "expectError": true + }, + { + "name": "a-non-string-source-value-is-an-error", + "tree": { + "123/meta.real.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": 123 }] }, + "expectError": true } ] } From 241b76af684b942d2c4e93ba48a94d66dbd5616b Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 14:09:56 -0400 Subject: [PATCH 34/44] fix(cli): meta init --config-only refuses rather than destroys an unparseable existing config writeConfigFile()'s catch-and-overwrite-with-defaults branch is safe on the full-scaffold path only because the caller already required --force to reach it (the exists-guard earlier in init() throws otherwise). --config-only calls writeConfigFile() directly, bypassing that guard entirely, so a JVM adopter's config carrying a key this CLI's strict schema doesn't recognize (written by a newer `meta`, or a typo) would run the documented-as-safe `--config-only` and silently lose their declared sources. Adds the same --force requirement inside the catch itself, which is a no-op on the full-scaffold path (force is already guaranteed there) and closes the gap on --config-only. --- .../packages/cli/src/commands/init.ts | 13 ++++++++++ .../typescript/packages/cli/test/init.test.ts | 25 +++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/server/typescript/packages/cli/src/commands/init.ts b/server/typescript/packages/cli/src/commands/init.ts index 614433c8b..afa493ab8 100644 --- a/server/typescript/packages/cli/src/commands/init.ts +++ b/server/typescript/packages/cli/src/commands/init.ts @@ -336,6 +336,19 @@ async function writeConfigFile(opts: InitOptions, result: InitResult, agentDir: result.preserved.push(".metaobjects/config.json"); } catch { if (priorContent !== undefined) { + // In the full-scaffold path this is only reachable once the caller has + // already required --force (the exists-guard at the top of `init()` + // throws before writeConfigFile runs otherwise), so opts.force is always + // true there. `--config-only` calls this function directly with no such + // guard, so without this check it would silently destroy an existing, + // merely-unparseable config on every run — the one thing `--force` is + // supposed to gate. + if (!opts.force) { + throw new Error( + `existing .metaobjects/config.json exists but could not be parsed; refusing to overwrite it. ` + + `Use --force to replace it with defaults. Prior content:\n${priorContent}`, + ); + } log.warn("existing .metaobjects/config.json was invalid — writing fresh defaults. Prior content:"); log.warn(priorContent); result.warnings.push("invalid .metaobjects/config.json replaced with defaults"); diff --git a/server/typescript/packages/cli/test/init.test.ts b/server/typescript/packages/cli/test/init.test.ts index 76caeef9b..fdbf5b01b 100644 --- a/server/typescript/packages/cli/test/init.test.ts +++ b/server/typescript/packages/cli/test/init.test.ts @@ -256,6 +256,31 @@ describe("init() --config-only", () => { const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); expect(cfg.sources).toEqual([{ path: "model" }]); }); + + test("refuses to overwrite an existing config it cannot parse, without --force", async () => { + mkdirSync(join(cwd, ".metaobjects"), { recursive: true }); + // Not valid against ConfigSchema.strict() — e.g. written by a newer `meta` + // or a typo'd key. Before --config-only existed, reaching this failure + // required an explicit --force; --config-only must not have quietly + // regressed that safety net. + writeFileSync(join(cwd, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, unknownKey: true })); + + await expect(init({ cwd, configOnly: true })).rejects.toThrow(/could not be parsed/); + // Unmodified — the refusal must be provable, not just declared. + const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(cfg.unknownKey).toBe(true); + }); + + test("--force still replaces an existing unparseable config with defaults", async () => { + mkdirSync(join(cwd, ".metaobjects"), { recursive: true }); + writeFileSync(join(cwd, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, unknownKey: true })); + + const result = await init({ cwd, configOnly: true, force: true }); + + expect(result.warnings).toContain("invalid .metaobjects/config.json replaced with defaults"); + const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(cfg.sources).toEqual([]); + }); }); describe("initCommand --config-only", () => { From 2cfe3cd61644528c4a3e37ccd3abe8be4092d000 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 14:10:02 -0400 Subject: [PATCH 35/44] docs: source-resolution-conformance in CONFORMANCE.md; C#'s single-directory-source limit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CONFORMANCE.md never gained a row for the corpus this feature shipped with — still said "20 shared conformance corpora" and had no source-resolution-conformance entry in the totals table or a fixture-to-doc mapping section. Added both, following the existing scope-conformance section's shape (its closest sibling). docs/features/metadata-sources.md and the CHANGELOG's "one declaration serves every port" line both overstated C#: `dotnet meta`'s loader accepts only a single directory source, so a multi-entry `sources` or a single-FILE entry doesn't fully "serve" it the way the other three ports are served. Documented the limit and its workaround (an explicit argument) in both places. --- CHANGELOG.md | 5 +++-- docs/CONFORMANCE.md | 34 +++++++++++++++++++++++++++---- docs/features/metadata-sources.md | 12 +++++++++++ 3 files changed, 45 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4464fa57b..0b440e56d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,8 +15,9 @@ guide: [`docs/features/metadata-sources.md`](docs/features/metadata-sources.md). - **`sources` is read by all four CLI surfaces**, not just the Node `meta` CLI — the C#, Python and Java/Kotlin CLIs (Kotlin has no CLI of its own; it runs through the same Maven plugin as Java) now resolve metadata from the - port-neutral `.metaobjects/config.json`, so one declaration serves every port. - Each reads a **neutral subset** (`schema_version` + `sources`) and ignores + port-neutral `.metaobjects/config.json`, so one declaration serves every port + (C#'s CLI loader accepts only a single directory `path` source — see the + adopter guide). Each reads a **neutral subset** (`schema_version` + `sources`) and ignores unknown top-level keys, so the TypeScript-owned keys in that file (`migrate`, `scope`, `extract`, and the rest) never become a four-port change. Precedence is a ladder — explicit CLI argument, then the port's own native surface (a diff --git a/docs/CONFORMANCE.md b/docs/CONFORMANCE.md index 941ece9df..d2dbfc967 100644 --- a/docs/CONFORMANCE.md +++ b/docs/CONFORMANCE.md @@ -1,6 +1,6 @@ # Conformance coverage -The MetaObjects standard ships **20 shared conformance corpora** under +The MetaObjects standard ships **21 shared conformance corpora** under [`fixtures/`](../fixtures/). Every port runs every corpus that is *applicable to it* and asserts the same expected behaviour against the same fixtures. **This page is the inverse index**: fixture → feature doc + per-port pass status, and it is the @@ -42,6 +42,7 @@ regenerate with `ls -d fixtures//*/ | wc -l`. | [`fixtures/template-output-render-conformance/`](../fixtures/template-output-render-conformance/) | 5 | ✓ | ✓ | ✓ | ✓ | ✓ | | [`fixtures/generator-registry-conformance/`](../fixtures/generator-registry-conformance/) | 1 canonical manifest | ✓ | ✓ | ✓ | ✓ | ✓ | | [`fixtures/provider-composition-conformance/`](../fixtures/provider-composition-conformance/) | 9 (5 error-shape + 4 compose-load) | ✓ | ✓ | — (JVM registry via Java) | ✓ | ✓ | +| [`fixtures/source-resolution-conformance/`](../fixtures/source-resolution-conformance/) | 25 cases | ✓ (reference implementation) | ✓ | inherits via Java | ✓ | ✓ | | [`fixtures/scope-conformance/`](../fixtures/scope-conformance/) | 10 cases | ✓ (reference implementation) | — | — | — | — | | [`fixtures/agent-context-conformance/`](../fixtures/agent-context-conformance/) | 4 | ✓ (the emitter is TS-owned) | — | — | — | — | | [`fixtures/metamodel-docs/`](../fixtures/metamodel-docs/) | 1 | ✓ (docs emit is TS-owned) | — | — | — | — | @@ -182,6 +183,31 @@ inheritance), `m2m/` (3), `jsonb/` (2, typed value-object columns) and Kotlin, C#, Python — run it in BOTH lanes: a hand-rolled reference server and the port's own GENERATED API artifact booted over HTTP. +### `fixtures/source-resolution-conformance/` (25 cases) + +All 25 cases → [features/metadata-sources.md](features/metadata-sources.md) (how a +declared `sources` set resolves to a file list). Companion to +`scope-conformance/` below — `sources` decides which files are read, `scope` +filters what is emitted from them. The corpus is file-shaped: one committed +`cases.json`, read directly by every port's runner, with no per-port fixture +and no ledger. + +It pins the resolved file **SET** for a declared `sources` list — the default +directory, replacement-not-merge, the relative-path base (the directory +HOLDING `.metaobjects/`, never the process cwd), recursive directory walking, +case-insensitive extension matching, union-with-de-duplication, and every +error condition (an unresolvable path, an unsupported `resource`/`package` +kind, and a malformed config — `"expectError": true` pins only that +resolution RAISES, since which error code it raises with is deliberately NOT +a cross-port contract; see the corpus README). **All four CLI surfaces run +it** — TypeScript (the reference implementation), C#, Python, and Java (Kotlin +inherits it, since Kotlin has no CLI entry point of its own and runs through +the same Maven plugin as Java): +`server/typescript/packages/sdk/test/source-resolution-conformance.test.ts`, +`server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs`, +`server/python/tests/conformance/test_source_resolution_conformance.py`, and +`server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java`. + ### `fixtures/scope-conformance/` (10 cases) All 10 cases → [features/metadata-sources.md](features/metadata-sources.md) (the @@ -207,9 +233,9 @@ grammar rather than four. ## Orphaned fixtures (tested but not yet documented) -The fixtures in the seven corpora mapped above (metamodel 255 + yaml 15 + verify 31 -+ render 15 + persistence 33 + api-contract 41 + scope 10) each map to a feature doc. None -are orphaned today. The remaining corpora in the totals table gate tooling +The fixtures in the eight corpora mapped above (metamodel 255 + yaml 15 + verify 31 ++ render 15 + persistence 33 + api-contract 41 + source-resolution 25 + scope 10) each +map to a feature doc. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider composition, agent context, docs emit) rather than user-facing metamodel behaviour, so they have no feature-doc row. diff --git a/docs/features/metadata-sources.md b/docs/features/metadata-sources.md index 6fb6211d1..d10d1a027 100644 --- a/docs/features/metadata-sources.md +++ b/docs/features/metadata-sources.md @@ -55,6 +55,18 @@ match wins: A config file that exists but is malformed is an error at its own rung; it never falls through to the next one. +**`dotnet meta` (C#) accepts only a single DIRECTORY source.** Its loader +(`MetaDataLoader.FromDirectory`) takes one directory, so `sources` declaring more +than one entry, or a single entry that resolves to a FILE rather than a +directory, is refused with a clear diagnostic naming the limit — rather than +silently loading a subset of the declared sources, or (for the single-file case) +failing deep inside the loader with an opaque, uncoded error. Declare exactly one +directory `path` to use `dotnet meta gen`/`verify`/`docs`, or route around the +limit with an explicit `` CLI argument. +`MetaDataLoader.Load(IReadOnlyList)` (`MetaDataLoader.cs`) is the +documented follow-up that would lift both restrictions. The other three CLI +surfaces have no such limit. + The non-TypeScript ports read a **neutral subset** of that file — `schema_version` and `sources` — and ignore every other top-level key, so the TypeScript-owned keys beside them (`migrate`, `extract`, and the rest) never From 7f097d3656d8e706748e6a81f47f16f91b1df124 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 15:42:19 -0400 Subject: [PATCH 36/44] refactor(config): simplify the cross-port source-resolution ladder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Java and Python SourceResolver both re-walked directories with their own copy of the extension filter + `_pending/` exclusion + sort, duplicating logic the loader's own DirectorySource already implements — the same duplication class this feature's design explicitly warns against. Both now delegate to DirectorySource, matching the C# port (which already did). - Python cli.py: extract `_resolve_metadata_location_or_print_error`, collapsing three copies of the same try/except ParseError → print → return 1 block (docs, gen's neutral fallback, verify --codegen's neutral fallback). - TS init.ts: `writeConfigFile` had two copies of the fresh-config write; factored into a `writeFresh` closure and restructured the surrounding try/catch with early returns instead of nested nested nested if branches. No behavior change: all four language suites and the shared source-resolution-conformance corpus still pass. --- .../metaobjects/config/SourceResolver.java | 62 ++----------- server/python/src/metaobjects/cli.py | 34 ++++--- .../src/metaobjects/config/source_resolver.py | 29 ++---- .../packages/cli/src/commands/init.ts | 91 +++++++++---------- 4 files changed, 84 insertions(+), 132 deletions(-) diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java index 2af732b79..09a255b68 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java @@ -17,19 +17,14 @@ import com.metaobjects.ErrorCode; import com.metaobjects.MetaDataException; +import com.metaobjects.loader.DirectorySource; -import java.io.IOException; -import java.io.UncheckedIOException; import java.nio.file.Files; import java.nio.file.Path; import java.util.ArrayList; -import java.util.Comparator; import java.util.LinkedHashSet; import java.util.List; -import java.util.Locale; import java.util.Map; -import java.util.Set; -import java.util.stream.Stream; /** * Turns a declared source SET ({@code .metaobjects/config.json}'s {@code sources}, or @@ -45,16 +40,6 @@ */ public final class SourceResolver { - private static final Set EXTENSIONS = Set.of(".json", ".yaml", ".yml"); - - /** - * Directory excluded at every level of a directory-spec walk — drafts that - * are deliberately not part of the loaded model. Mirrors TypeScript's - * {@code PENDING_DIR} in {@code metadata-files.ts} and the loader's own - * {@link com.metaobjects.loader.DirectorySource}. - */ - private static final String PENDING_DIR = "_pending"; - private SourceResolver() {} /** @@ -102,21 +87,15 @@ public static List resolveSources(Path configDir, List } if (isDir) { - // Order within one directory spec is this port's own basename sort, - // deliberately NOT a cross-port contract — see the class javadoc. - try (Stream walk = Files.walk(target)) { - walk.filter(Files::isRegularFile) - .filter(p -> hasSupportedExtension(p.getFileName().toString())) - // Excludes _pending/ at ANY depth — every ancestor path - // component between `target` and `p` is checked, not - // merely `p`'s own basename, so the whole subtree is - // skipped. - .filter(p -> !isUnderPendingDir(target, p)) - .sorted(Comparator.comparing(p -> p.getFileName().toString())) - .forEach(p -> seen.add(p.toAbsolutePath().normalize())); - } catch (IOException e) { - throw new UncheckedIOException("Failed to list " + target, e); - } + // Directory expansion — extension filter, `_pending/`-at-any-depth + // exclusion, and basename sort (this port's own order, deliberately + // NOT a cross-port contract — see the class javadoc) — is + // DirectorySource's, the SAME code the loader itself uses to turn a + // directory into metadata files. Reimplementing the walk here would + // be a second, driftable definition of "which files count as + // metadata". + new DirectorySource(target).expand() + .forEach(fs -> seen.add(fs.getPath().toAbsolutePath().normalize())); } else { seen.add(target.toAbsolutePath().normalize()); } @@ -150,25 +129,4 @@ public static List resolveCollection(Path root) { return resolveSources(base, specs); } - - /** - * True when any ancestor path component between {@code base} and {@code file} - * (i.e. excluding {@code file}'s own name) is exactly {@link #PENDING_DIR}. - */ - private static boolean isUnderPendingDir(Path base, Path file) { - Path rel = base.relativize(file).getParent(); - if (rel == null) return false; - for (Path part : rel) { - if (part.toString().equals(PENDING_DIR)) return true; - } - return false; - } - - private static boolean hasSupportedExtension(String name) { - String lower = name.toLowerCase(Locale.ROOT); - for (String ext : EXTENSIONS) { - if (lower.endsWith(ext)) return true; - } - return false; - } } diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index 4db6b2647..31b7560b8 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -477,6 +477,22 @@ def resolve_metadata_location( return [str(p) for p in resolve_collection(root)] +def _resolve_metadata_location_or_print_error( + config: ProjectConfig | None, root: Path +) -> list[str] | None: + """``resolve_metadata_location`` for the no-explicit-``metadata_dir`` CLI + paths (``docs``, and the ``gen``/``verify --codegen`` neutral fallbacks), + translating a raised ``ParseError`` into this CLI's print-and-return-1 + convention instead of letting it propagate. Returns ``None`` on failure — + the caller has nothing further to print and should return 1. + """ + try: + return resolve_metadata_location(explicit=None, config=config, root=root) + except ParseError as exc: + print(f"error: could not resolve metadata location: {exc}", file=sys.stderr) + return None + + #: The default api-surface subdir (the cross-port contract's ``api/python``). _DOCS_DEFAULT_API_SUBDIR = "api/python" @@ -521,10 +537,8 @@ def _cmd_docs(args: argparse.Namespace) -> int: except ConfigError as exc: print(f"error: {exc}", file=sys.stderr) return 1 - try: - paths = resolve_metadata_location(explicit=None, config=config, root=root_dir) - except ParseError as exc: - print(f"error: could not resolve metadata location: {exc}", file=sys.stderr) + paths = _resolve_metadata_location_or_print_error(config, root_dir) + if paths is None: return 1 root, errors = _load_root_from_paths(paths, providers=providers) project_default = root_dir.name @@ -770,10 +784,8 @@ def _cmd_gen_neutral_fallback(args: argparse.Namespace) -> int: return 2 root_dir = Path.cwd() - try: - paths = resolve_metadata_location(explicit=None, config=None, root=root_dir) - except ParseError as exc: - print(f"error: could not resolve metadata location: {exc}", file=sys.stderr) + paths = _resolve_metadata_location_or_print_error(None, root_dir) + if paths is None: return 1 generators: list[Generator] | None = None @@ -966,10 +978,8 @@ def _verify_codegen_neutral_fallback(args: argparse.Namespace) -> int: return 2 root_dir = Path.cwd() - try: - paths = resolve_metadata_location(explicit=None, config=None, root=root_dir) - except ParseError as exc: - print(f"error: could not resolve metadata location: {exc}", file=sys.stderr) + paths = _resolve_metadata_location_or_print_error(None, root_dir) + if paths is None: return 1 strict = not getattr(args, "lax", False) diff --git a/server/python/src/metaobjects/config/source_resolver.py b/server/python/src/metaobjects/config/source_resolver.py index c1752c8df..3633bf7f6 100644 --- a/server/python/src/metaobjects/config/source_resolver.py +++ b/server/python/src/metaobjects/config/source_resolver.py @@ -3,36 +3,21 @@ from pathlib import Path from metaobjects.errors import ErrorCode, ParseError +from metaobjects.loader.sources import DirectorySource from .neutral_config import DEFAULT_METADATA_DIR, read_neutral_config -_SUPPORTED_SUFFIXES = (".json", ".yaml", ".yml") - -#: Directory excluded at every level of `_list_metadata_files` — drafts that -#: are deliberately not part of the loaded model. Mirrors TypeScript's -#: `PENDING_DIR` in `metadata-files.ts` and the loader's own -#: `DirectorySource` (`loader/sources/directory_source.py`). -_PENDING_DIR = "_pending" - def _list_metadata_files(directory: Path) -> list[Path]: """Recursively list metadata files under ``directory``. - Mirrors `DirectorySource`'s extension set (`.json`/`.yaml`/`.yml`, - case-insensitive) AND its `_pending/`-at-any-depth exclusion. Order is - this port's own and is deliberately NOT a cross-port contract — see the - corpus README. + Delegates to the loader's own `DirectorySource` — the SAME code the loader + uses to turn a directory into metadata files — rather than re-walking with + a second, driftable definition of "which files count as metadata" (extension + set, `_pending/`-at-any-depth exclusion). Order is this port's own and is + deliberately NOT a cross-port contract — see the corpus README. """ - return sorted( - ( - p - for p in directory.rglob("*") - if p.is_file() - and p.suffix.lower() in _SUPPORTED_SUFFIXES - and _PENDING_DIR not in p.relative_to(directory).parts[:-1] - ), - key=lambda p: p.name, - ) + return [fs.path for fs in DirectorySource(directory).expand()] def _validate_kinds(specs: list[dict[str, str]]) -> None: diff --git a/server/typescript/packages/cli/src/commands/init.ts b/server/typescript/packages/cli/src/commands/init.ts index afa493ab8..472543e7a 100644 --- a/server/typescript/packages/cli/src/commands/init.ts +++ b/server/typescript/packages/cli/src/commands/init.ts @@ -321,54 +321,53 @@ async function writeConfigFile(opts: InitOptions, result: InitResult, agentDir: const freshConfig = opts.d1 ? ConfigSchema.parse({ ...DEFAULT_CONFIG, migrate: buildD1MigrateBlock(opts.cwd) }) : DEFAULT_CONFIG; - if (agentDirExists) { - const configPath = join(agentDir, "config.json"); - let priorContent: string | undefined; - try { - priorContent = await readFile(configPath, "utf8"); - const parsed = ConfigSchema.parse(JSON.parse(priorContent)); - const merged = ConfigSchema.parse({ ...DEFAULT_CONFIG, ...parsed }); - // When a valid .metaobjects/config.json already exists and the user passes --force, - // we preserve the existing config and only re-scaffold support files. The --d1 flag - // only takes effect on fresh inits — retro-fitting D1 onto an existing project is - // the user's job (edit migrate.dialect and migrate.d1 in config.json directly). - await saveConfig(agentDir, merged); - result.preserved.push(".metaobjects/config.json"); - } catch { - if (priorContent !== undefined) { - // In the full-scaffold path this is only reachable once the caller has - // already required --force (the exists-guard at the top of `init()` - // throws before writeConfigFile runs otherwise), so opts.force is always - // true there. `--config-only` calls this function directly with no such - // guard, so without this check it would silently destroy an existing, - // merely-unparseable config on every run — the one thing `--force` is - // supposed to gate. - if (!opts.force) { - throw new Error( - `existing .metaobjects/config.json exists but could not be parsed; refusing to overwrite it. ` + - `Use --force to replace it with defaults. Prior content:\n${priorContent}`, - ); - } - log.warn("existing .metaobjects/config.json was invalid — writing fresh defaults. Prior content:"); - log.warn(priorContent); - result.warnings.push("invalid .metaobjects/config.json replaced with defaults"); - } - await writeFile( - join(agentDir, "config.json"), - JSON.stringify(freshConfig, null, 2) + "\n", - "utf8", + const writeFresh = (): Promise => + writeFile(join(agentDir, "config.json"), JSON.stringify(freshConfig, null, 2) + "\n", "utf8"); + + if (!agentDirExists) { + await writeFresh(); + result.created.push(".metaobjects/config.json"); + return; + } + + const configPath = join(agentDir, "config.json"); + let priorContent: string | undefined; + try { + priorContent = await readFile(configPath, "utf8"); + const parsed = ConfigSchema.parse(JSON.parse(priorContent)); + const merged = ConfigSchema.parse({ ...DEFAULT_CONFIG, ...parsed }); + // When a valid .metaobjects/config.json already exists and the user passes --force, + // we preserve the existing config and only re-scaffold support files. The --d1 flag + // only takes effect on fresh inits — retro-fitting D1 onto an existing project is + // the user's job (edit migrate.dialect and migrate.d1 in config.json directly). + await saveConfig(agentDir, merged); + result.preserved.push(".metaobjects/config.json"); + return; + } catch { + if (priorContent === undefined) { + // The .metaobjects/ dir existed but config.json itself did not — a fresh write. + await writeFresh(); + result.created.push(".metaobjects/config.json"); + return; + } + + // In the full-scaffold path this is only reachable once the caller has + // already required --force (the exists-guard at the top of `init()` + // throws before writeConfigFile runs otherwise), so opts.force is always + // true there. `--config-only` calls this function directly with no such + // guard, so without this check it would silently destroy an existing, + // merely-unparseable config on every run — the one thing `--force` is + // supposed to gate. + if (!opts.force) { + throw new Error( + `existing .metaobjects/config.json exists but could not be parsed; refusing to overwrite it. ` + + `Use --force to replace it with defaults. Prior content:\n${priorContent}`, ); - if (priorContent === undefined) { - result.created.push(".metaobjects/config.json"); - } } - } else { - await writeFile( - join(agentDir, "config.json"), - JSON.stringify(freshConfig, null, 2) + "\n", - "utf8", - ); - result.created.push(".metaobjects/config.json"); + log.warn("existing .metaobjects/config.json was invalid — writing fresh defaults. Prior content:"); + log.warn(priorContent); + result.warnings.push("invalid .metaobjects/config.json replaced with defaults"); + await writeFresh(); } } From 49e83702f3f16adfe0ece82a1235d116f0ee6e5d Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 18:29:34 -0400 Subject: [PATCH 37/44] fix(migrate): the replay engine must not export PGlite's WASM exit status (#313) migrate-ts's suite exited 99 on 805 pass / 0 fail, turning two `ci-local.sh --only ts` gates red with no failing test to point at. Root cause is not teardown and not unhandled errors, which were the two standing theories. PGlite is Postgres compiled to WASM, and Emscripten propagates the WASM program's internal exit status into process.exitCode: it becomes 99 on the FIRST QUERY and stays there. Measured directly -- start undefined, after one `select 1` it is 99, close does not clear it. The shipped CLI was never affected because bin/meta.ts ends with process.exit(code), which overrides it. Anything that does not force its own exit inherits it, which is both `bun test` and any embedder calling openReplayEngine programmatically -- so this is a library defect, not a test-harness quirk. openPglite now captures the caller's exit code and restores it on dispose. Two details are load-bearing and each cost a failed fix: - `?? 0`, because assigning `undefined` to process.exitCode is a NO-OP under Bun (set 99, assign undefined, it stays 99; assign 0 and it clears). The pristine value IS undefined, so restoring it literally ran and changed nothing. - the `finally`, because disposable() calls db.destroy() first, which drives the pool's end(), which already closed PGlite -- so the second close throws `PGlite is closed` and a trailing statement never runs. The regression test spawns a CHILD PROCESS and asserts its exit code. Two in-process shapes were written first and both proved nothing: comparing before/after passes vacuously once any earlier test has opened an engine (99 === 99), and pinning a clean baseline first captures 0 rather than the pristine undefined, so it never exercises the `?? 0` -- it passed against the broken implementation. --- .../migrate-ts/src/verify/replay-engine.ts | 31 ++++++++++++++- .../test/unit/replay-engine.test.ts | 38 +++++++++++++++++++ 2 files changed, 68 insertions(+), 1 deletion(-) diff --git a/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts b/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts index d4e8f1f81..5762d6867 100644 --- a/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts +++ b/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts @@ -84,11 +84,40 @@ async function openPglite(): Promise { ); } const { PostgresDialect } = await import("kysely"); + + // PGlite is Postgres compiled to WASM, and Emscripten propagates the WASM + // program's internal exit status into `process.exitCode` — it becomes 99 on the + // FIRST QUERY (not on teardown) and stays there for the life of the process. + // Opening an engine must not decide what the HOST process exits with, so the + // caller's value is captured here and restored on dispose. + // + // `bin/meta.ts` ends with `process.exit(code)`, which overrides this, so the + // shipped CLI never showed it. Anything that does NOT force its own exit did: + // this package's `bun test` exited 99 on 0 failures, turning two + // `ci-local.sh --only ts` gates red with no failing test to point at, and an + // embedder calling `openReplayEngine` directly would exit non-zero on success. + const hostExitCode = process.exitCode; const pg = new PGliteCtor(); const db = new Kysely>({ dialect: new PostgresDialect({ pool: pgliteAsPool(pg) as never }), }); - return disposable(db, () => pg.close()); + + // `?? 0` is load-bearing, not defensive: assigning `undefined` to + // `process.exitCode` is a NO-OP under Bun (measured — set 99, assign + // `undefined`, it stays 99; assign 0 and it clears). The pristine value IS + // `undefined`, so restoring it literally runs and changes nothing — which is + // the shape this bug already took once during the fix. + // + // The restore sits in a `finally` because this close is frequently the SECOND: + // `disposable` runs `db.destroy()` first, which drives the pool's `end()`, + // which already called `pg.close()`, so this call throws `PGlite is closed`. + return disposable(db, async () => { + try { + await pg.close(); + } finally { + process.exitCode = hostExitCode ?? 0; + } + }); } /** The slice of PGlite's surface this file uses. */ diff --git a/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts b/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts index 526189fa0..9f4ca0877 100644 --- a/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts +++ b/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts @@ -4,6 +4,44 @@ import { openReplayEngine } from "../../src/verify/replay-engine.js"; import { introspect } from "../../src/introspect/index.js"; describe("openReplayEngine", () => { + // PGlite is Postgres compiled to WASM, and Emscripten propagates the WASM + // program's internal exit status into `process.exitCode` — it becomes 99 on the + // FIRST QUERY, not on teardown, and stays there. `bin/meta.ts` ends with + // `process.exit(code)`, so the shipped CLI overrides it and was never affected; + // anything that does NOT force its own exit inherits it. That is why this whole + // FILE used to exit 99 with 11 pass / 0 fail, turning two `ci-local.sh --only ts` + // gates red while reporting no failing test — and why an embedder calling + // `openReplayEngine` programmatically would silently exit non-zero on success. + // + // This must run on the postgres engine specifically; sqlite never touches WASM. + // + // It runs in a CHILD PROCESS, and that is the whole design. Two in-process + // shapes were tried first and both prove nothing: + // - `const before = process.exitCode; … expect(after).toBe(before)` passes + // vacuously once any earlier test has opened a postgres engine, because + // 99 === 99. + // - Pinning a clean baseline with `process.exitCode = 0` first fixes that but + // then captures 0 rather than the pristine `undefined`, so it never + // exercises the `?? 0` that the real fix turns on — it passed against the + // broken implementation. + // The only faithful assertion is the one the CI gate itself makes: what a fresh + // process actually EXITS with, starting from a pristine `process.exitCode`. + test("postgres: does not leak PGlite's WASM exit status into the host process", async () => { + const script = ` + const { openReplayEngine } = await import("${import.meta.dir}/../../src/verify/replay-engine.ts"); + const { sql } = await import("kysely"); + const engine = await openReplayEngine("postgres"); + try { await sql\`SELECT 1\`.execute(engine.db); } finally { await engine.dispose(); } + `; + const proc = Bun.spawn(["bun", "-e", script], { + cwd: `${import.meta.dir}/../..`, + stdout: "pipe", + stderr: "pipe", + }); + const code = await proc.exited; + expect(code).toBe(0); + }, 60_000); + test("sqlite: gives an empty, usable database", async () => { const engine = await openReplayEngine("sqlite"); try { From 9cdc6779841abbcf3c086728c85f7f73ce245c14 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 19:18:59 -0400 Subject: [PATCH 38/44] fix(loader): follow symlinked directories (Java, Python); scope _pending exclusion behind an option MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Java and Python's DirectorySource silently returned zero files for a `sources` path that is itself a symlink to a directory, or skipped a symlinked subdirectory partway through a walked tree — TypeScript and C# already followed symlinks this way. Java's `Files.walk` now passes FOLLOW_LINKS; a symlink cycle raises rather than hanging. Python's `rglob`-based walk never recursed through a symlinked subdirectory at all (`**` does not by design), so it is replaced with a manual recursive walk that follows symlinks and detects cycles itself (SymlinkLoopError), preserving each symlinked directory's own name in the reported path — never collapsing it to the target's real path, matching every other port. Getting this required also fixing a related, more subtle divergence: Python's resolve_sources deduplicated files via Path.resolve(), which (unlike Java's normalize()/C#'s GetFullPath()/TS's path.resolve()) follows symlinks, so it would have silently reported a resolved-through-a-symlink file under its target's real name instead of the declared name — replaced with a lexical-only absolute-path normalizer. Separately, `_pending/` exclusion (the TypeScript pending/promote workflow's concept) was baked unconditionally into Java's, C#'s and Python's loader-level DirectorySource — an API change for any runtime embedder calling `new DirectorySource(dir)` directly, since TypeScript's own loader-level DirectorySource has no such concept at all. Each port's DirectorySource now takes an `excludePending`/`exclude_pending`/`ExcludePending` option defaulting OFF; the CLI-facing SourceResolver (Java, Python) and DirectorySource.cs + SourceResolver.cs (C#) turn it ON explicitly, the one place each CLI opts in. Co-Authored-By: Claude Opus 5 (1M context) --- .../DirectorySourceTests.cs | 36 ++++++- .../MetaObjects/Config/SourceResolver.cs | 7 +- .../MetaObjects/Loader/DirectorySource.cs | 23 ++++- .../metaobjects/config/SourceResolver.java | 19 ++-- .../metaobjects/loader/DirectorySource.java | 37 ++++++- .../loader/DirectorySourceTest.java | 97 +++++++++++++++++-- .../src/metaobjects/config/source_resolver.py | 31 +++++- .../metaobjects/loader/sources/__init__.py | 3 +- .../loader/sources/directory_source.py | 92 ++++++++++++++---- server/python/tests/unit/test_sources.py | 63 +++++++++++- 10 files changed, 352 insertions(+), 56 deletions(-) diff --git a/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs b/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs index edcbf0343..0dee418dd 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs @@ -35,13 +35,39 @@ public void Expand_ReturnsFileSourcesSortedByOrdinalName() } [Fact] - public void Expand_Excludes_PendingDir_AtAnyDepth() + public void Expand_ExcludePending_IsOffByDefault() + { + // Loader-level default is OFF (matches TS's loader-level DirectorySource, + // which has no _pending concept at all) — only the CLI-facing + // SourceResolver turns it on. An app embedding `new DirectorySource(dir)` + // directly must see every file, _pending/ included. + string dir = Path.Combine(Path.GetTempPath(), "ds_" + Path.GetRandomFileName()); + Directory.CreateDirectory(dir); + try + { + File.WriteAllText(Path.Combine(dir, "meta.live.json"), "{}"); + Directory.CreateDirectory(Path.Combine(dir, "_pending")); + File.WriteAllText(Path.Combine(dir, "_pending", "meta.draft.json"), "{}"); + + var src = new DirectorySource(dir); + var expanded = src.Expand().ToList(); + + Assert.Equal(2, expanded.Count); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } + + [Fact] + public void Expand_Excludes_PendingDir_AtAnyDepth_WhenOptedIn() { // Mirrors TypeScript's PENDING_DIR exclusion (metadata-files.ts) — a draft // entity under _pending/ must be invisible to codegen, not merely a file - // that happens to be NAMED "_pending". Before this fix, only TypeScript - // knew about this directory; a draft would generate a live table under - // `dotnet meta gen`. + // that happens to be NAMED "_pending". SourceResolver (the CLI-facing + // caller) opts in via ExcludePending = true; this test exercises the + // option directly. string dir = Path.Combine(Path.GetTempPath(), "ds_" + Path.GetRandomFileName()); Directory.CreateDirectory(dir); try @@ -53,7 +79,7 @@ public void Expand_Excludes_PendingDir_AtAnyDepth() Directory.CreateDirectory(Path.Combine(dir, "nested", "_pending")); File.WriteAllText(Path.Combine(dir, "nested", "_pending", "meta.deep-draft.json"), "{}"); - var src = new DirectorySource(dir); + var src = new DirectorySource(dir, new DirectorySource.Options { ExcludePending = true }); var expanded = src.Expand().ToList(); Assert.Single(expanded); diff --git a/server/csharp/MetaObjects/Config/SourceResolver.cs b/server/csharp/MetaObjects/Config/SourceResolver.cs index 95dd928fa..d4e6b0669 100644 --- a/server/csharp/MetaObjects/Config/SourceResolver.cs +++ b/server/csharp/MetaObjects/Config/SourceResolver.cs @@ -68,8 +68,13 @@ public static IReadOnlyList ResolveSources( // header calls out: order within one directory spec is this port's own // full-path ordinal sort, deliberately NOT a cross-port contract (see the // file header above), but it MUST still be the loader's own order. + // ExcludePending = true: this IS the CLI-facing resolver — `_pending/` is + // the TypeScript CLI's pending/promote-workflow concept, not a loader + // concept, so the loader-level default (off) is overridden here, the one + // place this port's CLI turns it on. var found = isDir - ? new DirectorySource(target).Expand().Select(f => f.FilePath) + ? new DirectorySource(target, new DirectorySource.Options { ExcludePending = true }) + .Expand().Select(f => f.FilePath) : new[] { target }.AsEnumerable(); foreach (var f in found) diff --git a/server/csharp/MetaObjects/Loader/DirectorySource.cs b/server/csharp/MetaObjects/Loader/DirectorySource.cs index 01b4ad7db..df06a62a5 100644 --- a/server/csharp/MetaObjects/Loader/DirectorySource.cs +++ b/server/csharp/MetaObjects/Loader/DirectorySource.cs @@ -24,6 +24,18 @@ public sealed class Options /// Recurse into subdirectories. Default: true. public bool Recurse { get; init; } = true; + + /// + /// Exclude _pending/ at any depth. Default: false — this is a + /// LOADER-level primitive, and _pending/ is a CLI/pending-promote-workflow + /// concept (TypeScript's metadata-files.ts, not its loader-level + /// DirectorySource, which has no _pending concept at all). + /// — the CLI-facing caller — + /// turns this ON explicitly rather than the exclusion being baked into every + /// embedder of this class: an app calling new DirectorySource(dir) + /// directly gets every file back, matching the reference loader. + /// + public bool ExcludePending { get; init; } = false; } private static readonly HashSet _supportedExtensions = @@ -58,11 +70,16 @@ public IEnumerable Expand() : SearchOption.TopDirectoryOnly; IEnumerable files = System.IO.Directory.EnumerateFiles(Directory, "*", search) - .Where(p => _supportedExtensions.Contains(Path.GetExtension(p))) + .Where(p => _supportedExtensions.Contains(Path.GetExtension(p))); + + if (Opts.ExcludePending) + { // Excludes _pending/ at ANY depth — every ancestor path component // between `Directory` and the file is checked, not merely the file's - // own name, so the whole subtree is skipped. - .Where(p => !IsUnderPendingDir(Directory, p)); + // own name, so the whole subtree is skipped. Off by default — see + // Options.ExcludePending. + files = files.Where(p => !IsUnderPendingDir(Directory, p)); + } if (Opts.Exclude is { Count: > 0 } excludes) { diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java index 09a255b68..74b7e7f3a 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java @@ -87,14 +87,17 @@ public static List resolveSources(Path configDir, List } if (isDir) { - // Directory expansion — extension filter, `_pending/`-at-any-depth - // exclusion, and basename sort (this port's own order, deliberately - // NOT a cross-port contract — see the class javadoc) — is - // DirectorySource's, the SAME code the loader itself uses to turn a - // directory into metadata files. Reimplementing the walk here would - // be a second, driftable definition of "which files count as - // metadata". - new DirectorySource(target).expand() + // Directory expansion — extension filter and basename sort (this + // port's own order, deliberately NOT a cross-port contract — see the + // class javadoc) — is DirectorySource's, the SAME code the loader + // itself uses to turn a directory into metadata files. Reimplementing + // the walk here would be a second, driftable definition of "which + // files count as metadata". `_pending/`-at-any-depth exclusion is OFF + // by default at the loader level (a runtime app embedding + // DirectorySource directly gets every file) — this CLI-facing + // resolver turns it ON, since `_pending/` is the TypeScript CLI's + // pending/promote-workflow concept, not a loader concept. + new DirectorySource(target, new DirectorySource.Options().setExcludePending(true)).expand() .forEach(fs -> seen.add(fs.getPath().toAbsolutePath().normalize())); } else { seen.add(target.toAbsolutePath().normalize()); diff --git a/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java b/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java index 50f5a43b3..42b90d3cb 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java +++ b/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java @@ -2,6 +2,8 @@ import java.io.IOException; import java.io.UncheckedIOException; +import java.nio.file.FileSystemLoopException; +import java.nio.file.FileVisitOption; import java.nio.file.Files; import java.nio.file.Path; import java.util.Comparator; @@ -37,6 +39,7 @@ public final class DirectorySource { public static final class Options { private List exclude = List.of(); private boolean recurse = true; + private boolean excludePending = false; /** * Sets filenames to exclude from expansion (exact filename match). @@ -54,8 +57,24 @@ public Options setRecurse(boolean recurse) { return this; } + /** + * Sets whether {@code _pending/} (at any depth) is excluded from expansion. + * Default: {@code false} — this is a LOADER-level primitive, and + * {@code _pending/} is a CLI/pending-workflow concept (TypeScript's + * {@code metadata-files.ts}, not its loader-level {@code DirectorySource}). + * {@link com.metaobjects.config.SourceResolver}, the CLI-facing caller, turns + * this ON explicitly rather than the exclusion being baked into every + * embedder of this class (a runtime app calling {@code new DirectorySource(dir)} + * directly gets every file back, matching the reference loader). + */ + public Options setExcludePending(boolean excludePending) { + this.excludePending = excludePending; + return this; + } + public List getExclude() { return exclude; } public boolean isRecurse() { return recurse; } + public boolean isExcludePending() { return excludePending; } } private static final Set EXTENSIONS = Set.of(".json", ".yaml", ".yml"); @@ -97,13 +116,24 @@ public Options getOptions() { * Expands this directory into a sorted stream of {@link FileSource}. * Sorted by full path (ordinal) for deterministic ordering across runs. * + *

Follows symlinked directories — including when {@code directory} itself + * is a symlink — matching TypeScript ({@code stat}, not {@code lstat}) and C# + * ({@code EnumerateFiles(..., AllDirectories)}), both of which have always + * followed symlinks this way. A symlink CYCLE is a loud error rather than a + * hang: {@code Files.walk}'s own traversal detects it and throws + * {@link FileSystemLoopException} (naming the looping path), which the JDK + * wraps in {@link UncheckedIOException} and surfaces from whichever stream + * operation is consuming this method's lazily-returned {@link Stream} — not + * from this method itself, since the walk has not actually run yet when + * {@code expand()} returns.

+ * * @return stream of file sources; caller should close if iteration is partial * @throws UncheckedIOException if the directory cannot be listed */ public Stream expand() { try { Stream walk = opts.isRecurse() - ? Files.walk(directory) + ? Files.walk(directory, FileVisitOption.FOLLOW_LINKS) : Files.list(directory); return walk .filter(Files::isRegularFile) @@ -111,8 +141,9 @@ public Stream expand() { .filter(p -> !opts.getExclude().contains(p.getFileName().toString())) // Excludes _pending/ at ANY depth — every ancestor path component // between `directory` and `p` is checked, not merely `p`'s own - // basename, so the whole subtree is skipped. - .filter(p -> !isUnderPendingDir(directory, p)) + // basename, so the whole subtree is skipped. Off by default — see + // Options.setExcludePending. + .filter(p -> !opts.isExcludePending() || !isUnderPendingDir(directory, p)) .sorted(Comparator.comparing(p -> p.getFileName().toString())) .map(FileSource::new); } catch (IOException e) { diff --git a/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java b/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java index 405289a98..469fc81c8 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java @@ -2,6 +2,7 @@ import org.junit.Test; import java.io.IOException; +import java.io.UncheckedIOException; import java.nio.file.Files; import java.nio.file.Path; import java.util.List; @@ -62,12 +63,32 @@ public class DirectorySourceTest { } } - @Test public void excludesPendingDirAtAnyDepth() throws IOException { + @Test public void excludePendingIsOffByDefault() throws IOException { + // Loader-level default is OFF (matches TS's loader-level DirectorySource, + // which has no _pending concept at all) — only the CLI-facing SourceResolver + // turns it on. A runtime app embedding `new DirectorySource(dir)` directly + // must see every file, _pending/ included. + Path dir = Files.createTempDirectory("ds-"); + try { + Files.writeString(dir.resolve("meta.live.json"), "{}"); + Path pending = Files.createDirectory(dir.resolve("_pending")); + Files.writeString(pending.resolve("meta.draft.json"), "{}"); + + DirectorySource src = new DirectorySource(dir); + List expanded = src.expand().collect(Collectors.toList()); + + assertEquals(2, expanded.size()); + } finally { + deleteRecursively(dir); + } + } + + @Test public void excludesPendingDirAtAnyDepthWhenOptedIn() throws IOException { // Mirrors TypeScript's PENDING_DIR exclusion (metadata-files.ts) — a draft // entity under _pending/ must be invisible to codegen, not merely a file - // that happens to be NAMED "_pending". Before this fix, only TypeScript - // knew about this directory; a draft would generate a live table under - // `mvn metaobjects:generate`. + // that happens to be NAMED "_pending". SourceResolver (the CLI-facing + // caller) opts in via setExcludePending(true); this test exercises the + // option directly. Path dir = Files.createTempDirectory("ds-"); try { Files.writeString(dir.resolve("meta.live.json"), "{}"); @@ -77,7 +98,8 @@ public class DirectorySourceTest { Path nestedPending = Files.createDirectories(dir.resolve("nested").resolve("_pending")); Files.writeString(nestedPending.resolve("meta.deep-draft.json"), "{}"); - DirectorySource src = new DirectorySource(dir); + DirectorySource src = new DirectorySource(dir, + new DirectorySource.Options().setExcludePending(true)); List expanded = src.expand().collect(Collectors.toList()); assertEquals(1, expanded.size()); @@ -87,6 +109,65 @@ public class DirectorySourceTest { } } + @Test public void followsASymlinkedRoot() throws IOException { + // I1: the SOURCE path itself is a symlink to a directory. Files.isDirectory + // follows symlinks (the existence guard passes), so the walk must too, or + // the root resolves to zero files, silently. + Path real = Files.createTempDirectory("ds-real-"); + Path parent = Files.createTempDirectory("ds-link-parent-"); + Path link = parent.resolve("link"); + try { + Files.writeString(real.resolve("meta.a.json"), "{}"); + Files.createSymbolicLink(link, real); + + List expanded = new DirectorySource(link).expand().collect(Collectors.toList()); + + assertEquals(1, expanded.size()); + assertEquals("meta.a.json", expanded.get(0).getId()); + } finally { + deleteRecursively(parent); + deleteRecursively(real); + } + } + + @Test public void followsASymlinkedSubdirectory() throws IOException { + // I1, second arm: a symlinked SUBDIRECTORY inside a walked tree. + Path dir = Files.createTempDirectory("ds-"); + Path external = Files.createTempDirectory("ds-external-"); + try { + Files.writeString(dir.resolve("meta.top.json"), "{}"); + Files.writeString(external.resolve("meta.linked.json"), "{}"); + Files.createSymbolicLink(dir.resolve("linked"), external); + + List expanded = new DirectorySource(dir).expand().collect(Collectors.toList()); + + assertEquals(2, expanded.size()); + } finally { + deleteRecursively(dir); + deleteRecursively(external); + } + } + + @Test public void aSymlinkCycleFailsLoudlyRatherThanHanging() throws IOException { + Path dir = Files.createTempDirectory("ds-"); + try { + Files.writeString(dir.resolve("meta.top.json"), "{}"); + // A directory symlinked to its own ancestor — a cycle. + Files.createSymbolicLink(dir.resolve("loop"), dir); + + try { + new DirectorySource(dir).expand().collect(Collectors.toList()); + fail("expected a symlink-loop failure, not a completed walk"); + } catch (UncheckedIOException expected) { + // Files.walk(FOLLOW_LINKS) detects the cycle and throws + // FileSystemLoopException, naming the looping path. + assertNotNull(expected.getCause()); + } + } finally { + deleteRecursively(dir); + } + } + @Test public void nonRecursiveSkipsSubdirectories() throws IOException { Path dir = Files.createTempDirectory("ds-"); try { @@ -104,7 +185,11 @@ public class DirectorySourceTest { } private static void deleteRecursively(Path p) throws IOException { - if (Files.isDirectory(p)) { + // NOFOLLOW_LINKS: a symlink (incl. one deliberately cyclic, as in + // aSymlinkCycleFailsLoudlyRatherThanHanging above) is a leaf to delete + // outright, never a directory to recurse INTO — Files.isDirectory's default + // symlink-following would walk right back into the loop during cleanup. + if (Files.isDirectory(p, java.nio.file.LinkOption.NOFOLLOW_LINKS)) { try (var s = Files.list(p)) { for (Path c : s.collect(Collectors.toList())) deleteRecursively(c); } diff --git a/server/python/src/metaobjects/config/source_resolver.py b/server/python/src/metaobjects/config/source_resolver.py index 3633bf7f6..4f5ea94af 100644 --- a/server/python/src/metaobjects/config/source_resolver.py +++ b/server/python/src/metaobjects/config/source_resolver.py @@ -1,5 +1,6 @@ from __future__ import annotations +import os from pathlib import Path from metaobjects.errors import ErrorCode, ParseError @@ -14,10 +15,32 @@ def _list_metadata_files(directory: Path) -> list[Path]: Delegates to the loader's own `DirectorySource` — the SAME code the loader uses to turn a directory into metadata files — rather than re-walking with a second, driftable definition of "which files count as metadata" (extension - set, `_pending/`-at-any-depth exclusion). Order is this port's own and is - deliberately NOT a cross-port contract — see the corpus README. + set). Order is this port's own and is deliberately NOT a cross-port contract + — see the corpus README. + + `exclude_pending=True`: this IS the CLI-facing resolver — `_pending/` is the + TypeScript CLI's pending/promote-workflow concept, not a loader concept, so + the loader-level `DirectorySource` default (off) is overridden here, the one + place this port's CLI turns it on. + """ + return [fs.path for fs in DirectorySource(directory, exclude_pending=True).expand()] + + +def _normalize(p: Path) -> Path: + """Absolute + lexically normalized (``.``/``..`` collapsed), WITHOUT + resolving symlinks. + + `Path.resolve()` does both jobs at once — and following symlinks here is + the wrong half: it would silently rewrite a source declared as a symlink + (e.g. `sources: [{"path": "link"}]` where `link -> real`) to its target's + real name, diverging from the other three ports, none of which collapse a + walked path's symlinked directory components (Java's + `toAbsolutePath().normalize()`, C#'s `Path.GetFullPath()`, TypeScript's + `path.resolve()` are all lexical-only, like this). `os.path.abspath` is + exactly that: anchor to cwd if relative, then `normpath` — no filesystem + symlink lookups. """ - return [fs.path for fs in DirectorySource(directory).expand()] + return Path(os.path.abspath(p)) def _validate_kinds(specs: list[dict[str, str]]) -> None: @@ -64,7 +87,7 @@ def resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path] found = _list_metadata_files(target) if target.is_dir() else [target] for f in found: - seen.setdefault(f.resolve(), None) + seen.setdefault(_normalize(f), None) return list(seen) diff --git a/server/python/src/metaobjects/loader/sources/__init__.py b/server/python/src/metaobjects/loader/sources/__init__.py index 977a2d647..2f5a9fb14 100644 --- a/server/python/src/metaobjects/loader/sources/__init__.py +++ b/server/python/src/metaobjects/loader/sources/__init__.py @@ -9,7 +9,7 @@ """ from __future__ import annotations -from .directory_source import DirectorySource +from .directory_source import DirectorySource, SymlinkLoopError from .file_source import FileSource from .meta_data_source import InMemoryStringSource, MetaDataFormat, MetaDataSource from .uri_source import UriSource @@ -20,5 +20,6 @@ "InMemoryStringSource", "FileSource", "DirectorySource", + "SymlinkLoopError", "UriSource", ] diff --git a/server/python/src/metaobjects/loader/sources/directory_source.py b/server/python/src/metaobjects/loader/sources/directory_source.py index ed1a2e1a3..0c6ee81a0 100644 --- a/server/python/src/metaobjects/loader/sources/directory_source.py +++ b/server/python/src/metaobjects/loader/sources/directory_source.py @@ -17,10 +17,22 @@ #: Directory excluded at every level of a recursive expand() — drafts that are #: deliberately not part of the loaded model. Mirrors TypeScript's -#: `PENDING_DIR` in `metadata-files.ts`. +#: `PENDING_DIR` in `metadata-files.ts`. OFF by default — see +#: `DirectorySource.__init__`'s `exclude_pending` doc. _PENDING_DIR = "_pending" +class SymlinkLoopError(OSError): + """Raised when expand() finds a directory symlink that revisits an ancestor. + + A clear error beats a hang: expand() follows symlinked directories (I1 — + matching TypeScript's `stat`-not-`lstat` walk and C#'s + `EnumerateFiles(..., AllDirectories)`, both of which have always followed + them), so an accidental or malicious symlink cycle must be caught explicitly + rather than recursing forever. + """ + + class DirectorySource: """Expands a directory into a sorted, filtered list of FileSource objects.""" @@ -29,33 +41,73 @@ def __init__( directory: Path | str, exclude: Iterable[str] | None = None, recurse: bool = True, + exclude_pending: bool = False, ) -> None: + """``exclude_pending`` (I2) — exclude ``_pending/`` at any depth. + + Default ``False``: this is a LOADER-level primitive, and ``_pending/`` is + a CLI/pending-promote-workflow concept (TypeScript's `metadata-files.ts`, + not its loader-level `DirectorySource`, which has no `_pending` concept + at all). `source_resolver.py` — the CLI-facing caller — turns this ON + explicitly rather than baking the exclusion into every embedder of this + class; an app calling ``DirectorySource(dir)`` directly gets every file + back, matching the reference loader. + """ self._directory = Path(directory) self._exclude = set(exclude or ()) self._recurse = recurse + self._exclude_pending = exclude_pending @property def directory(self) -> Path: return self._directory def expand(self) -> Iterator[FileSource]: - candidates = ( - self._directory.rglob("*") if self._recurse else self._directory.iterdir() - ) - files = sorted( - ( - p - for p in candidates - if p.is_file() - and p.suffix.lower() in _SUPPORTED_SUFFIXES - and p.name not in self._exclude - # Excludes _pending/ at ANY depth — a directory NAME check on every - # ancestor component between `self._directory` and `p`, not merely - # a basename filter on `p` itself, so the whole subtree is skipped - # (a draft entity must be invisible to codegen, not just a file - # that happens to be named "_pending"). - and _PENDING_DIR not in p.relative_to(self._directory).parts[:-1] - ), - key=lambda p: p.name, - ) + files = sorted(self._collect(self._directory, frozenset()), key=lambda p: p.name) yield from (FileSource(p) for p in files) + + def _collect(self, directory: Path, ancestors: frozenset[Path]) -> list[Path]: + """Recursively collect matching files under ``directory``, following + symlinked subdirectories (I1) — `iterdir()`/`is_dir()` naturally follow a + symlink, unlike `rglob("*")`'s `**` traversal, which does not descend + into one. + + Paths are built by lexical join (`directory / name`) throughout, exactly + like a plain non-symlink-aware walk would build them — a symlinked + directory's OWN name survives in the reported path; only the WALK follows + the link, matching Java/C#/TypeScript, none of which collapse a source's + directory name to its symlink target's real name. + + ``ancestors`` is a set of REAL (`Path.resolve()`) directory paths already + on the current walk branch, used only to detect a symlink cycle — never + to rewrite a reported path. + """ + real = directory.resolve() + if real in ancestors: + raise SymlinkLoopError( + f"symlink loop detected while expanding {self._directory}: " + f"{directory} revisits {real}" + ) + ancestors = ancestors | {real} + + out: list[Path] = [] + for entry in directory.iterdir(): + if entry.is_dir(): + if self._recurse: + out.extend(self._collect(entry, ancestors)) + elif ( + entry.is_file() + and entry.suffix.lower() in _SUPPORTED_SUFFIXES + and entry.name not in self._exclude + # Excludes _pending/ at ANY depth — a directory NAME check on every + # ancestor component between `self._directory` and `entry`, not + # merely a basename filter on `entry` itself, so the whole subtree + # is skipped (a draft entity must be invisible to codegen, not + # just a file that happens to be named "_pending"). + and not ( + self._exclude_pending + and _PENDING_DIR in entry.relative_to(self._directory).parts[:-1] + ) + ): + out.append(entry) + return out diff --git a/server/python/tests/unit/test_sources.py b/server/python/tests/unit/test_sources.py index 4018a8873..254312334 100644 --- a/server/python/tests/unit/test_sources.py +++ b/server/python/tests/unit/test_sources.py @@ -98,12 +98,26 @@ def test_directory_source_non_recursive(tmp_path: Path) -> None: assert ids == ["top.json"] -def test_directory_source_excludes_pending_dir_at_any_depth(tmp_path: Path) -> None: +def test_directory_source_exclude_pending_is_off_by_default(tmp_path: Path) -> None: + # Loader-level default is OFF (matches TS's loader-level DirectorySource, + # which has no _pending concept at all) — only the CLI-facing + # source_resolver.py turns it on. An app embedding DirectorySource(dir) + # directly must see every file, _pending/ included. + (tmp_path / "meta.live.json").write_text("{}", encoding="utf-8") + pending = tmp_path / "_pending" + pending.mkdir() + (pending / "meta.draft.json").write_text("{}", encoding="utf-8") + + ids = sorted(s.id for s in DirectorySource(tmp_path).expand()) + assert ids == ["meta.draft.json", "meta.live.json"] + + +def test_directory_source_excludes_pending_dir_at_any_depth_when_opted_in(tmp_path: Path) -> None: # Mirrors TypeScript's PENDING_DIR exclusion (metadata-files.ts) — a draft # entity under _pending/ must be invisible to codegen, not merely a file - # that happens to be NAMED "_pending". Before this fix, only TypeScript - # knew about this directory; a draft would generate a live table under - # `metaobjects gen`. + # that happens to be NAMED "_pending". source_resolver.py (the CLI-facing + # caller) opts in via exclude_pending=True; this test exercises the option + # directly. (tmp_path / "meta.live.json").write_text("{}", encoding="utf-8") pending = tmp_path / "_pending" pending.mkdir() @@ -113,10 +127,49 @@ def test_directory_source_excludes_pending_dir_at_any_depth(tmp_path: Path) -> N nested_pending.mkdir(parents=True) (nested_pending / "meta.deep-draft.json").write_text("{}", encoding="utf-8") - ids = [s.id for s in DirectorySource(tmp_path).expand()] + ids = [s.id for s in DirectorySource(tmp_path, exclude_pending=True).expand()] assert ids == ["meta.live.json"] +def test_directory_source_follows_a_symlinked_root(tmp_path: Path) -> None: + # I1: the SOURCE path itself is a symlink to a directory. `Path.is_dir()` + # follows symlinks (the existence guard passes), so the walk must too, or + # the root resolves to zero files, silently. + real = tmp_path / "real" + real.mkdir() + (real / "meta.a.json").write_text("{}", encoding="utf-8") + link = tmp_path / "link" + link.symlink_to(real, target_is_directory=True) + + ids = [s.id for s in DirectorySource(link).expand()] + assert ids == ["meta.a.json"] + + +def test_directory_source_follows_a_symlinked_subdirectory(tmp_path: Path) -> None: + # I1, second arm: a symlinked SUBDIRECTORY inside a walked tree. + (tmp_path / "meta.top.json").write_text("{}", encoding="utf-8") + external = tmp_path.parent / f"{tmp_path.name}-external" + external.mkdir() + (external / "meta.linked.json").write_text("{}", encoding="utf-8") + (tmp_path / "linked").symlink_to(external, target_is_directory=True) + + ids = sorted(s.id for s in DirectorySource(tmp_path).expand()) + assert ids == ["meta.linked.json", "meta.top.json"] + + +def test_directory_source_symlink_cycle_fails_loudly_rather_than_hanging( + tmp_path: Path, +) -> None: + from metaobjects.loader.sources import SymlinkLoopError + + (tmp_path / "meta.top.json").write_text("{}", encoding="utf-8") + # A directory symlinked to its own ancestor — a cycle. + (tmp_path / "loop").symlink_to(tmp_path, target_is_directory=True) + + with pytest.raises(SymlinkLoopError): + list(DirectorySource(tmp_path).expand()) + + def test_uri_source_file_scheme_reads_content(tmp_path: Path) -> None: p = tmp_path / "x.json" p.write_text("{}", encoding="utf-8") From cec3f541035a863ee7d18158bb74d8ebbbb48c34 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 19:19:15 -0400 Subject: [PATCH 39/44] test(conformance): gate symlinked-directory resolution cross-port; harden the shared runners MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two new cases (a `sources` path that is itself a symlink to a directory; a symlinked subdirectory partway through a walked tree) via a new `symlinks` key in the shared corpus schema — a map of linkPath -> targetPath, project-root-relative, materialized after `tree`. Every runner now honors it. Verified each case fails against pre-fix behavior before committing (git diff'd the production fix out, confirmed red, restored it). Also, four defensive gaps in the runners themselves, all found while extending this corpus: - `overlapping-sources-yield-each-file-once` could not fail in the Java/C#/ Python runners because all three compare the resolved set only, silently collapsing a duplicate emission — each now also asserts the RAW resolved count against the expected count. - Java's JUnit4 `Parameterized` runner reports green on a zero-case corpus (a bad path or a JSON bug would run zero tests, not fail) — TS and Python already guard this; a new standalone `SourceResolutionCorpusNotEmptyTest` closes the same gap for Java, which the parameterized class's own `@Test` methods cannot (they run once per row, so zero rows means zero executions of the guard too). - The TS runner defaulted a case missing BOTH `expectFiles` and `expectError` to expecting zero files, rather than failing loudly on the malformed case. - Java's case loader read a literally-absent "config" key the same as an explicit `"config": null` — Python's dict indexing and C#'s JsonElement.GetProperty both throw on the former; Java's Gson `get()` did not, so a future case that forgot the key would silently read as "no config" while the other three runners crash on the same file. The README's "none of those are in the corpus yet" note (bad JSON / unsupported schema_version / a malformed sources entry shape) is corrected: five of the `expectError: true` cases now cover schema_version and entry-shape; genuine malformed JSON syntax still is not, and can't be until the case schema grows a raw-text config variant. Co-Authored-By: Claude Opus 5 (1M context) --- .../source-resolution-conformance/README.md | 45 +++++++++++---- .../source-resolution-conformance/cases.json | 19 +++++++ .../SourceResolutionConformanceTests.cs | 31 +++++++++- .../SourceResolutionConformanceTest.java | 49 ++++++++++++++-- .../SourceResolutionCorpusNotEmptyTest.java | 51 +++++++++++++++++ .../test_source_resolution_conformance.py | 56 +++++++------------ .../source-resolution-conformance.test.ts | 22 +++++++- 7 files changed, 216 insertions(+), 57 deletions(-) create mode 100644 server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionCorpusNotEmptyTest.java diff --git a/fixtures/source-resolution-conformance/README.md b/fixtures/source-resolution-conformance/README.md index 109c6deb6..2ff6be920 100644 --- a/fixtures/source-resolution-conformance/README.md +++ b/fixtures/source-resolution-conformance/README.md @@ -11,13 +11,26 @@ are read, `scope` filters what is emitted from them. ## Shape ``` -cases.json # { cases: [{ name, tree, config, resolveFrom?, expectFiles?, expectError? }] } +cases.json # { cases: [{ name, tree, symlinks?, config, resolveFrom?, expectFiles?, expectError? }] } README.md ``` - **`tree`** — a map of project-root-relative path → file content. The runner materializes it in a fresh temporary directory. A `.keep` entry exists only to force an otherwise-empty directory to be created. +- **`symlinks`** — OPTIONAL, a map of project-root-relative `linkPath` → + project-root-relative `targetPath`. Materialized AFTER `tree` (so the target + already exists), as a directory symlink at `linkPath` pointing at `targetPath`. + Exists to gate that every port's directory walk FOLLOWS a symlinked directory + — whether the `sources` path itself is a symlink + (`a-symlinked-sources-path-resolves-through-it`) or a symlink sits partway + through a walked tree (`a-symlinked-subdirectory-inside-a-walked-tree- + resolves-through-it`) — rather than silently resolving to zero files or + skipping the subtree. The reported path preserves the symlink's OWN name (a + file reached through `link/` is reported as `link/…`, never resolved to + `real/…`) — see "Order is deliberately NOT pinned" below for the parallel + point about `expectFiles` being exact strings, not just "the same underlying + file by any name". Every port's runner must honor this key. - **`config`** — written verbatim to `.metaobjects/config.json`, under the directory named by `resolveFrom` (project root when `resolveFrom` is absent). When `null`, no config file is created at all. @@ -114,20 +127,30 @@ Pinning a shared code across ports would mean changing the reference, which this corpus does not do. Python raises `ERR_COLLECTION_NOT_FOUND`; C# and Java both raise `ERR_BAD_ATTR_VALUE` — three distinct outcomes across four ports, which is exactly why this case checks only that resolution raises, never with -which error, same as file order above. The same `true` form is available to -any future malformed-config case that needs it (bad JSON, an unsupported -`schema_version`, a malformed `sources` entry shape, …) — none of those are -in the corpus yet, but nothing about the mechanism is specific to this one -shape. +which error, same as file order above. The same `true` form covers five more +cases beyond this one: an unsupported `schema_version` +(`an-unsupported-schema-version-is-an-error`) and four malformed `sources` +entry shapes (`sources-null-is-an-error-not-the-default`, +`an-empty-path-is-an-error`, `a-sources-entry-with-two-keys-is-an-error`, +`a-non-string-source-value-is-an-error`). Genuinely malformed JSON SYNTAX (an +unparseable `.metaobjects/config.json`) is still not in the corpus — the case +schema's `config` field is always a valid JSON value that the runner +re-serializes via `JSON.stringify`/equivalent, so expressing broken syntax +would need a schema extension carrying raw file text instead of a config +object. Nothing about the `true`-sentinel mechanism is specific to any of +these shapes. ## Behavioral contract Each port's runner reads `cases.json`, and for every case: materializes `tree` -in a fresh temp directory, writes `config` when non-null under the directory -named by `resolveFrom` (default the project root), resolves sources against -that directory, then asserts either that the resolved file set equals -`expectFiles` (as a set, project-root-relative, path separators normalized to -`/`) or that resolution failed with `expectError`. +in a fresh temp directory, then `symlinks` (when present), writes `config` +when non-null under the directory named by `resolveFrom` (default the project +root), resolves sources against that directory, then asserts either that the +resolved file set equals `expectFiles` (as a set, project-root-relative, path +separators normalized to `/`) — AND that its size matches the raw resolved +count, since a Set/HashSet comparison alone cannot see a duplicate emission +collapse invisibly into one set element — or that resolution failed with +`expectError`. ## Reference implementation diff --git a/fixtures/source-resolution-conformance/cases.json b/fixtures/source-resolution-conformance/cases.json index d93655a86..4ed9faafc 100644 --- a/fixtures/source-resolution-conformance/cases.json +++ b/fixtures/source-resolution-conformance/cases.json @@ -245,6 +245,25 @@ }, "config": { "schema_version": 1, "sources": [{ "path": 123 }] }, "expectError": true + }, + { + "name": "a-symlinked-sources-path-resolves-through-it", + "tree": { + "real/meta.a.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "symlinks": { "link": "real" }, + "config": { "schema_version": 1, "sources": [{ "path": "link" }] }, + "expectFiles": ["link/meta.a.json"] + }, + { + "name": "a-symlinked-subdirectory-inside-a-walked-tree-resolves-through-it", + "tree": { + "model/meta.top.json": "{\"metadata.root\":{\"children\":[]}}", + "external/meta.linked.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "symlinks": { "model/linked": "external" }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": ["model/meta.top.json", "model/linked/meta.linked.json"] } ] } diff --git a/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs b/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs index 8bb6fefc9..c79347340 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs @@ -30,7 +30,11 @@ private sealed record Case( // A JSON string pins the exact error code raised; JSON `true` pins only // that resolution RAISES — the malformed-config error code is // deliberately not pinned cross-port (see the corpus README). - JsonElement? ExpectError); + JsonElement? ExpectError, + // Optional: linkPath -> targetPath, both project-root-relative, materialized + // AFTER `Tree` (I1 — a symlinked source root, or a symlinked subdirectory + // inside a walked tree). + Dictionary Symlinks); public static TheoryData CaseNames() { @@ -68,7 +72,13 @@ private static List LoadCases() : null; JsonElement? expectError = el.TryGetProperty("expectError", out var ee) ? ee.Clone() : null; - cases.Add(new Case(el.GetProperty("name").GetString()!, tree, cfg, resolveFrom, expectFiles, expectError)); + var symlinks = new Dictionary(); + if (el.TryGetProperty("symlinks", out var sl)) + { + foreach (var p in sl.EnumerateObject()) symlinks[p.Name] = p.Value.GetString()!; + } + + cases.Add(new Case(el.GetProperty("name").GetString()!, tree, cfg, resolveFrom, expectFiles, expectError, symlinks)); } return cases; } @@ -90,6 +100,14 @@ public void ResolvesTheSameFileSet(string name) Directory.CreateDirectory(Path.GetDirectoryName(abs)!); File.WriteAllText(abs, content); } + // Materialized AFTER tree — see the Case record's Symlinks doc. + foreach (var (linkRel, targetRel) in c.Symlinks) + { + var linkAbs = Path.Combine(root, linkRel.Replace('/', Path.DirectorySeparatorChar)); + var targetAbs = Path.Combine(root, targetRel.Replace('/', Path.DirectorySeparatorChar)); + Directory.CreateDirectory(Path.GetDirectoryName(linkAbs)!); + Directory.CreateSymbolicLink(linkAbs, targetAbs); + } // The invocation directory: project root joined with `resolveFrom`. // The config MUST be materialized here, not at the project root — a @@ -124,10 +142,17 @@ public void ResolvesTheSameFileSet(string name) // two coincide (resolveFrom "."), so a comparison base bug here would // pass every other case and fail only that one — which is exactly why // that case exists. - var got = SourceResolver.ResolveCollection(invokeDir) + var raw = SourceResolver.ResolveCollection(invokeDir); + var got = raw .Select(f => Path.GetRelativePath(root, f).Replace(Path.DirectorySeparatorChar, '/')) .ToHashSet(); Assert.Equal(c.ExpectFiles!.ToHashSet(), got); + // A HashSet comparison alone cannot see a duplicate emission (two entries + // for the same file collapse invisibly into one set element) — + // `overlapping-sources-yield-each-file-once` specifically exercises + // ResolveCollection's own de-duplication, so the RAW list count must be + // asserted too, before it is thrown away by the ToHashSet conversion. + Assert.Equal(c.ExpectFiles!.Length, raw.Count); } finally { diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java index ea27ec2ba..d24765fbf 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java @@ -76,9 +76,13 @@ public class SourceResolutionConformanceTest { * code is deliberately not pinned cross-port (see the corpus README). */ private record Case(String name, Map tree, JsonObject config, - String resolveFrom, List expectFiles, JsonElement expectError) {} + String resolveFrom, List expectFiles, JsonElement expectError, + Map symlinks) {} - private static Path corpus() { + /** Package-private (not private): shared with {@link SourceResolutionCorpusNotEmptyTest}, + * which needs to locate the same committed corpus file without a second definition + * of "walk up to find fixtures/". */ + static Path corpus() { Path dir = Paths.get("").toAbsolutePath(); while (dir != null && !Files.isDirectory(dir.resolve("fixtures"))) dir = dir.getParent(); assertNotNull("could not locate the repository fixtures/ directory", dir); @@ -101,8 +105,20 @@ public static Collection cases() throws IOException { tree.put(e.getKey(), e.getValue().getAsString()); } + // A LITERALLY ABSENT "config" key is a malformed corpus case, not the same + // thing as an explicit `"config": null` (which means "no config file" — + // see e.g. no-config-uses-default-directory). Python's dict indexing and + // C#'s JsonElement.GetProperty both throw on the former; Gson's `get()` + // returns Java null for BOTH, so without this check a future case that + // simply forgot the key would silently read as "no config" here while the + // other three runners crash loudly on the same corpus file. + if (!c.has("config")) { + throw new IllegalStateException( + "corpus case \"" + name + "\" has no \"config\" key (use JSON null for " + + "\"no config file\", not an absent key)"); + } JsonElement cfgEl = c.get("config"); - JsonObject config = (cfgEl == null || cfgEl.isJsonNull()) ? null : cfgEl.getAsJsonObject(); + JsonObject config = cfgEl.isJsonNull() ? null : cfgEl.getAsJsonObject(); String resolveFrom = c.has("resolveFrom") ? c.get("resolveFrom").getAsString() : "."; @@ -116,7 +132,17 @@ public static Collection cases() throws IOException { JsonElement expectError = c.has("expectError") ? c.get("expectError") : null; - rows.add(new Object[]{name, new Case(name, tree, config, resolveFrom, expectFiles, expectError)}); + // Optional: linkPath -> targetPath, both project-root-relative, materialized + // AFTER `tree` (I1 — a symlinked source root, or a symlinked subdirectory + // inside a walked tree). + Map symlinks = new LinkedHashMap<>(); + if (c.has("symlinks")) { + for (Map.Entry e : c.getAsJsonObject("symlinks").entrySet()) { + symlinks.put(e.getKey(), e.getValue().getAsString()); + } + } + + rows.add(new Object[]{name, new Case(name, tree, config, resolveFrom, expectFiles, expectError, symlinks)}); } return rows; } @@ -142,6 +168,12 @@ public void resolvesTheSameFileSet() throws IOException { Files.createDirectories(abs.getParent()); Files.write(abs, entry.getValue().getBytes(StandardCharsets.UTF_8)); } + // Materialized AFTER tree — see the Case record's symlinks doc. + for (Map.Entry link : testCase.symlinks().entrySet()) { + Path linkPath = root.resolve(link.getKey()); + Files.createDirectories(linkPath.getParent()); + Files.createSymbolicLink(linkPath, root.resolve(link.getValue())); + } // The invocation directory: project root joined with `resolveFrom`. The // config MUST be materialized here, not at the project root — a config @@ -179,12 +211,19 @@ public void resolvesTheSameFileSet() throws IOException { // coincide (resolveFrom "."), so a comparison-base bug here would pass // every other case and fail only that one — which is exactly why that // case exists. - Set got = SourceResolver.resolveCollection(invokeDir).stream() + List raw = SourceResolver.resolveCollection(invokeDir); + Set got = raw.stream() .map(p -> root.relativize(p).toString().replace('\\', '/')) .collect(Collectors.toSet()); Set want = new HashSet<>(testCase.expectFiles()); assertEquals(want, got); + // The Set comparison above cannot see a duplicate emission (two entries + // for the same file collapse invisibly into one Set element) — + // `overlapping-sources-yield-each-file-once` specifically exercises + // resolveCollection's own de-duplication, so the RAW list length must be + // asserted too, before it is thrown away by the Set conversion. + assertEquals(want.size(), raw.size()); } finally { deleteRecursive(root); } diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionCorpusNotEmptyTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionCorpusNotEmptyTest.java new file mode 100644 index 000000000..334c83eca --- /dev/null +++ b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionCorpusNotEmptyTest.java @@ -0,0 +1,51 @@ +/* + * Copyright 2003 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.config; + +import com.google.gson.JsonArray; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import org.junit.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; + +import static org.junit.Assert.assertTrue; + +/** + * Deliberately NOT part of {@link SourceResolutionConformanceTest}'s + * {@code @RunWith(Parameterized.class)} run: JUnit4's {@code Parameterized} runner + * executes its {@code @Test} methods once PER PARAMETER ROW, so if {@code cases()} + * ever returned zero rows — a bad path, a JSON-parsing bug, an accidental corpus + * truncation — every {@code @Test} in that class (including a guard living inside + * it) would run zero times and Maven would report the class GREEN, having checked + * nothing. This is a plain, separate JUnit4 test class so it always runs exactly + * once regardless of what the corpus contains. Mirrors the TS and Python runners' + * identically-purposed guards ({@code source-resolution-conformance.test.ts}, + * {@code test_source_resolution_conformance.py::test_corpus_is_non_empty}). + */ +public class SourceResolutionCorpusNotEmptyTest { + + @Test + public void corpusIsNonEmpty() throws IOException { + String content = new String( + Files.readAllBytes(SourceResolutionConformanceTest.corpus()), StandardCharsets.UTF_8); + JsonObject root = JsonParser.parseString(content).getAsJsonObject(); + JsonArray cases = root.getAsJsonArray("cases"); + assertTrue("a silent zero-case corpus is a failed gate, not a pass", cases.size() > 0); + } +} diff --git a/server/python/tests/conformance/test_source_resolution_conformance.py b/server/python/tests/conformance/test_source_resolution_conformance.py index ae04a672a..a929d0784 100644 --- a/server/python/tests/conformance/test_source_resolution_conformance.py +++ b/server/python/tests/conformance/test_source_resolution_conformance.py @@ -36,15 +36,23 @@ def test_corpus_is_non_empty() -> None: def _materialize(case: dict, root: Path) -> Path: - """Materialize ``tree`` under ``root`` and ``config`` (when present) under the - directory named by ``resolveFrom`` (project root when absent). Returns the - directory resolution must be invoked against. + """Materialize ``tree`` under ``root``, then ``symlinks`` (I1 — a symlinked + source root, or a symlinked subdirectory inside a walked tree), then + ``config`` (when present) under the directory named by ``resolveFrom`` + (project root when absent). Returns the directory resolution must be + invoked against. """ for rel, content in case["tree"].items(): p = root / rel p.parent.mkdir(parents=True, exist_ok=True) p.write_text(content) + # Materialized AFTER tree — both link/target are project-root-relative. + for link_rel, target_rel in case.get("symlinks", {}).items(): + link = root / link_rel + link.parent.mkdir(parents=True, exist_ok=True) + link.symlink_to(root / target_rel, target_is_directory=True) + resolve_from = root / case.get("resolveFrom", ".") if case["config"] is not None: @@ -72,8 +80,15 @@ def test_source_resolution_conformance(case: dict, tmp_path: Path) -> None: # `expectFiles` is project-root-relative even when `resolveFrom` points # elsewhere — resolve against `tmp_path`, not `resolve_from`. root = tmp_path.resolve() - got = {p.relative_to(root).as_posix() for p in resolve_collection(resolve_from)} + raw = resolve_collection(resolve_from) + got = {p.relative_to(root).as_posix() for p in raw} assert got == set(case["expectFiles"]) + # A set comparison alone cannot see a duplicate emission (two entries for the + # same file collapse invisibly into one set element) — + # `overlapping-sources-yield-each-file-once` specifically exercises + # resolve_collection's own de-duplication, so the RAW list length must be + # asserted too, before it is thrown away by the set conversion. + assert len(raw) == len(case["expectFiles"]) def test_cli_falls_back_to_neutral_config(tmp_path: Path, monkeypatch) -> None: @@ -89,43 +104,12 @@ def test_cli_falls_back_to_neutral_config(tmp_path: Path, monkeypatch) -> None: from metaobjects.cli import resolve_metadata_location monkeypatch.chdir(tmp_path) - got = resolve_metadata_location(explicit=None, config=None, root=tmp_path) + got = resolve_metadata_location(config=None, root=tmp_path) assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in got} == { "model/meta.a.json" } -def test_explicit_relative_metadata_dir_resolves_against_cwd( - tmp_path: Path, monkeypatch -) -> None: - """A RELATIVE explicit must not resolve one level too deep. - - Regression for the plan's original defect: `resolve_sources(Path(explicit) - .resolve().parent, [{"path": explicit}])` joins an already-absolute base - with a still-relative spec, walking one directory too far up. - - A SINGLE-segment argument (e.g. ``"model"``) cannot distinguish the buggy - formulation from the fixed one: ``Path("model").resolve().parent`` happens - to land back at the project root, so the extra join silently cancels out. - The defect only shows up with a MULTI-segment relative path (``"sub/model"``) - — the buggy form resolves the base to ``.../sub`` and then joins the still- - relative ``"sub/model"`` onto it, landing on ``.../sub/sub/model`` (does not - exist -> ERR_SOURCE_UNRESOLVED) instead of ``.../sub/model``. - """ - (tmp_path / "sub" / "model").mkdir(parents=True) - (tmp_path / "sub" / "model" / "meta.a.json").write_text( - '{"metadata.root":{"children":[]}}' - ) - - from metaobjects.cli import resolve_metadata_location - - monkeypatch.chdir(tmp_path) - got = resolve_metadata_location(explicit="sub/model", config=None, root=tmp_path) - assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in got} == { - "sub/model/meta.a.json" - } - - def test_docs_with_no_positional_falls_back_to_neutral_config( tmp_path: Path, monkeypatch ) -> None: diff --git a/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts b/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts index 843d9002f..f9f387a93 100644 --- a/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts +++ b/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts @@ -1,7 +1,7 @@ // Runs the shared source-resolution corpus against the TypeScript reference // implementation. Every port ships an equivalent runner reading this same file. import { describe, expect, test } from "bun:test"; -import { mkdtemp, mkdir, readFile, writeFile } from "node:fs/promises"; +import { mkdtemp, mkdir, readFile, symlink, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { dirname, join, relative, resolve, sep } from "node:path"; import { resolveCollection } from "../src/collection.js"; @@ -19,6 +19,10 @@ interface Case { * resolution RAISES — the malformed-config error code is deliberately not * pinned cross-port (see the corpus README). */ readonly expectError?: string | true; + /** Optional: linkPath -> targetPath, both project-root-relative, materialized + * AFTER `tree` (I1 — a symlinked source root, or a symlinked subdirectory + * inside a walked tree). */ + readonly symlinks?: Record; } const CORPUS = resolve( @@ -38,6 +42,12 @@ async function materialize(c: Case): Promise<{ root: string; resolveDir: string await mkdir(dirname(abs), { recursive: true }); await writeFile(abs, content); } + // Materialized AFTER tree — see the Case.symlinks doc. + for (const [linkRel, targetRel] of Object.entries(c.symlinks ?? {})) { + const linkAbs = join(root, linkRel); + await mkdir(dirname(linkAbs), { recursive: true }); + await symlink(join(root, targetRel), linkAbs, "dir"); + } const resolveDir = resolve(root, c.resolveFrom ?? "."); if (c.config !== null) { await mkdir(join(resolveDir, ".metaobjects"), { recursive: true }); @@ -76,9 +86,17 @@ describe("source-resolution conformance", () => { } return; } + // A case with neither `expectFiles` nor `expectError` is a malformed corpus + // entry, not "expect zero files" — `?? []` here would silently pass such a + // case instead of failing loudly on it (this is the TS-runner-specific half + // of the "assert count, not just presence" family of fixes; see the C#/Java/ + // Python runners' analogous length assertions below the set comparison). + if (c.expectFiles === undefined) { + throw new Error(`corpus case "${c.name}" has neither expectFiles nor expectError`); + } const collection = await resolveCollection(resolveDir, { explicitDir: resolveDir }); const got = collection.files.map((f) => relative(root, f).split(sep).join("/")).sort(); - expect(got).toEqual([...(c.expectFiles ?? [])].sort()); + expect(got).toEqual([...c.expectFiles].sort()); }); } }); From 5a93e11db548220dd7441ebcde0451999ec1457e Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 19:19:21 -0400 Subject: [PATCH 40/44] fix(cli): meta init --config-only --print-only must not write MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `--config-only` branch returned above the `--print-only` guard the full-scaffold path checks further down, so `meta init --config-only --print-only` — a documented dry run — silently wrote the real .metaobjects/config.json. `--print-only` now wins outright for that branch too. `--docs-only --print-only` has the same shape (writeAgentContext has no print-only awareness either) but is not fixed here: its write set is derived dynamically from assemble()/planScaffold() rather than a fixed list, so skipping the writes while still reporting what WOULD be written is more than a trivial change. Co-Authored-By: Claude Opus 5 (1M context) --- server/typescript/packages/cli/src/commands/init.ts | 7 +++++++ server/typescript/packages/cli/test/init.test.ts | 10 ++++++++++ 2 files changed, 17 insertions(+) diff --git a/server/typescript/packages/cli/src/commands/init.ts b/server/typescript/packages/cli/src/commands/init.ts index 472543e7a..25134f17d 100644 --- a/server/typescript/packages/cli/src/commands/init.ts +++ b/server/typescript/packages/cli/src/commands/init.ts @@ -387,6 +387,13 @@ export async function init(opts: InitOptions): Promise { } if (opts.configOnly) { + // --print-only must win outright: a documented dry run must never write, and + // this branch used to return ABOVE the printOnly guard the full-scaffold path + // uses below, so `--config-only --print-only` silently wrote the real file. + if (opts.printOnly) { + result.created.push(".metaobjects/config.json"); + return result; + } // Config only: write/preserve .metaobjects/config.json and nothing else — no // metaobjects/ dir, no agent-context, no TypeScript scaffold. `agentDirExists` is // captured before the mkdir below so an existing valid config is still preserved. diff --git a/server/typescript/packages/cli/test/init.test.ts b/server/typescript/packages/cli/test/init.test.ts index fdbf5b01b..de0ba0e26 100644 --- a/server/typescript/packages/cli/test/init.test.ts +++ b/server/typescript/packages/cli/test/init.test.ts @@ -245,6 +245,16 @@ describe("init() --config-only", () => { } }); + test("--print-only writes nothing to disk", async () => { + // --config-only used to return ABOVE the --print-only guard the full-scaffold + // path checks below it, so this documented dry run silently wrote the real file. + const result = await init({ cwd, configOnly: true, printOnly: true }); + + expect(result.created).toContain(".metaobjects/config.json"); + expect(existsSync(join(cwd, ".metaobjects"))).toBe(false); + expect(existsSync(join(cwd, ".metaobjects", "config.json"))).toBe(false); + }); + test("leaves an existing valid config untouched", async () => { mkdirSync(join(cwd, ".metaobjects"), { recursive: true }); const existing = { schema_version: 1, sources: [{ path: "model" }] }; From 7626ffb9d361a74965cb5cda4b36d9f0f57d9a76 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 19:19:26 -0400 Subject: [PATCH 41/44] fix(csharp): a missing --out must win over the ladder's own ERR_COLLECTION_NOT_FOUND MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ResolveMetadataDirOrExit ran before the `outDir is null` usage check in both RunGen and RunDocs, so `dotnet meta gen`/`docs` with neither a metadataDir nor metadata to resolve printed the ladder's own "error: ERR_COLLECTION_NOT_FOUND" instead of the actionable "usage: ..." line — confusing on the common first-run case where both are missing at once. The outDir check, being an unconditional CLI-usage requirement, now runs first in both commands. Co-Authored-By: Claude Opus 5 (1M context) --- .../MetadataDirFallbackTests.cs | 17 +++++++++++++ server/csharp/MetaObjects.Cli/Program.cs | 24 ++++++++++++------- 2 files changed, 33 insertions(+), 8 deletions(-) diff --git a/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs b/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs index 6882cae1e..e56667c1b 100644 --- a/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs +++ b/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs @@ -66,6 +66,23 @@ public void Gen_with_no_positional_metadataDir_and_nothing_to_resolve_reports_th Assert.Contains("ERR_COLLECTION_NOT_FOUND", stderr); } + [Fact] + public void Gen_with_no_outDir_and_nothing_to_resolve_prints_usage_not_the_ladders_error() + { + // Usage-first: a missing --out is a plain CLI-usage mistake, independent of + // whether metadata can be found. Before the fix, ResolveMetadataDirOrExit ran + // FIRST and this empty project (no --out either) exited 2 with the ladder's + // own ERR_COLLECTION_NOT_FOUND instead of the actionable "usage: ..." line — + // confusing on the common first-run case where both are missing at once. + Directory.CreateDirectory(_tmp); + + var (exitCode, _, stderr) = RunCli(_tmp, "gen"); + + Assert.Equal(2, exitCode); + Assert.Contains("usage: dotnet meta gen", stderr); + Assert.DoesNotContain("ERR_COLLECTION_NOT_FOUND", stderr); + } + [Fact] public void Gen_with_no_positional_metadataDir_and_multiple_declared_sources_refuses_rather_than_picking_one() { diff --git a/server/csharp/MetaObjects.Cli/Program.cs b/server/csharp/MetaObjects.Cli/Program.cs index f643f0e5d..fa6a457e8 100644 --- a/server/csharp/MetaObjects.Cli/Program.cs +++ b/server/csharp/MetaObjects.Cli/Program.cs @@ -73,10 +73,12 @@ static int RunGen(string[] rest) return 0; } - // Rung 1 (explicit positional) is honored as-is; an omitted metadataDir - // falls back to the port-neutral .metaobjects/config.json ladder. - metadataDir = ResolveMetadataDirOrExit(metadataDir); - + // Usage-first: a missing --out is a plain CLI-usage error, unconditional on + // whether metadata can be found — it must win over ResolveMetadataDirOrExit + // below, which can itself terminate the process with an unrelated + // ERR_COLLECTION_NOT_FOUND. Checking outDir after resolution would show that + // confusing error instead of this actionable usage line on the (common) + // first-run case where BOTH are missing. if (outDir is null) { Console.Error.WriteLine("usage: dotnet meta gen --out [--namespace ] [--generators ] [--template-root ] [--template-spec ] [--emit-abstract-shapes]"); @@ -84,6 +86,10 @@ static int RunGen(string[] rest) return 2; } + // Rung 1 (explicit positional) is honored as-is; an omitted metadataDir + // falls back to the port-neutral .metaobjects/config.json ladder. + metadataDir = ResolveMetadataDirOrExit(metadataDir); + // Advisory: nudge a re-scaffold if the copied-in agent context predates this build. // Never throws, never changes the exit code (a missing/corrupt manifest is ignored). AgentContextStalenessCheck.WarnIfStale(Directory.GetCurrentDirectory()); @@ -125,16 +131,18 @@ static int RunDocs(string[] rest) else if (!rest[i].StartsWith('-')) metadataDir ??= rest[i]; } - // Rung 1 (explicit positional) is honored as-is; an omitted metadataDir - // falls back to the port-neutral .metaobjects/config.json ladder. - metadataDir = ResolveMetadataDirOrExit(metadataDir); - + // Usage-first — see the identical comment in RunGen above; a missing --out + // must win over ResolveMetadataDirOrExit's own possible ERR_COLLECTION_NOT_FOUND. if (outDir is null) { Console.Error.WriteLine("usage: dotnet meta docs --out [--namespace ] [--project ] [--model-base-url ]"); return 2; } + // Rung 1 (explicit positional) is honored as-is; an omitted metadataDir + // falls back to the port-neutral .metaobjects/config.json ladder. + metadataDir = ResolveMetadataDirOrExit(metadataDir); + // Default the project label to the input directory's leaf name (cosmetic — surfaces // in the AGENT-API header). Trailing-separator-safe. project ??= new DirectoryInfo(Path.TrimEndingDirectorySeparator(Path.GetFullPath(metadataDir))).Name; From 90c4816792bf3ae4bf5e626d7ff180916efa4f51 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 19:19:38 -0400 Subject: [PATCH 42/44] fix(java): mojo hands processSources an unambiguous URI; schema_version rejects a non-integral float MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit resolveNeutralSourcesIfPomIsSilent handed MetaDataLoader.processSources a bare Path::toString() for each resolved source. processSources decides how to wrap a bare source string by sniffing `s.indexOf(':') < 0` — an absolute path is not guaranteed colon-free (a Windows drive letter is the common case; a colon is also legal in a plain Unix directory name, which is how the regression test reproduces this without a Windows machine), so a path containing one skipped the wrapping entirely and was handed raw to URIHelper.toURI(), which throws. Since these are always fully-resolved absolute filesystem paths, there is nothing to sniff: each is now prefixed "model:file:" explicitly, the same shape processSources' own fallback branch already builds. Separately, NeutralConfig's `schema_version` check used Gson's JsonPrimitive#getAsInt(), which truncates a non-integral BigDecimal instead of raising — so `schema_version: 1.5` silently read as 1 and passed. Compared as a double instead (matching C#'s NeutralConfig.cs, which already does this and already has the regression test this change mirrors for Java); `1.0` is still accepted. Co-Authored-By: Claude Opus 5 (1M context) --- .../mojo/AbstractMetaDataMojo.java | 11 +++++- .../mojo/NeutralConfigMojoFallbackTest.java | 35 +++++++++++++++++++ .../com/metaobjects/config/NeutralConfig.java | 8 ++++- .../metaobjects/config/NeutralConfigTest.java | 19 ++++++++++ 4 files changed, 71 insertions(+), 2 deletions(-) diff --git a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java index 640b33ac1..a2969660a 100644 --- a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java +++ b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java @@ -513,10 +513,19 @@ protected List resolveNeutralSourcesIfPomIsSilent(LoaderParam loaderConf || (loaderConfig.getSources() != null && !loaderConfig.getSources().isEmpty()); if (pomNamesLocation) return List.of(); + // Prefixed "model:file:" explicitly rather than handed back as a bare + // Path::toString(): MetaDataLoader.processSources decides how to wrap a bare + // source string by checking `s.indexOf(':') < 0`, and an absolute path is not + // guaranteed colon-free — a Windows path (`C:\...`) fails that ambiguous + // sniff and is handed to URIHelper.toURI() unwrapped, which dies in + // validateUriType(). These are always fully-resolved absolute filesystem + // paths (SourceResolver.resolveCollection), so there is nothing to sniff: + // say "file" outright, the same shape processSources itself already builds + // for its own `new File(s).exists()` branch. return com.metaobjects.config.SourceResolver .resolveCollection(getProjectBaseDir().toPath()) .stream() - .map(java.nio.file.Path::toString) + .map(p -> "model:file:" + p.toString().replace(java.io.File.separatorChar, '/')) .toList(); } } diff --git a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java index 934f0fa18..1f1c5f6e9 100644 --- a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java +++ b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java @@ -109,6 +109,41 @@ public void createLoaderFallsBackToTheBuiltInDefaultDirectoryWhenNoNeutralConfig } } + @Test + public void createLoaderHandlesAColonInTheResolvedAbsolutePath() throws IOException { + // Regression: resolveNeutralSourcesIfPomIsSilent used to hand + // MetaDataLoader.processSources a bare Path::toString() — ambiguous + // whenever the absolute path contains a colon, since processSources + // sniffs `s.indexOf(':') < 0` to decide whether to wrap the string as a + // "model:file:" source. A Windows drive letter (`C:\...`) is the common + // case; reproduced here without a Windows machine via a colon in a plain + // Unix directory NAME (legal on ext4/most POSIX filesystems), which is + // just as ambiguous to that sniff. Before the fix this died with an + // uncoded IllegalArgumentException out of URIHelper.validateUriType. + Path root = Files.createTempDirectory("mo-mojo-neutral-colon-").toAbsolutePath().normalize(); + try { + Path weirdRoot = root.resolve("cache:v1"); + Files.createDirectories(weirdRoot); + Path metaDir = weirdRoot.resolve("custom-metadata"); + Files.createDirectories(metaDir); + Files.write(metaDir.resolve("meta.widget.json"), WIDGET_JSON.getBytes(StandardCharsets.UTF_8)); + + Path dotMo = weirdRoot.resolve(".metaobjects"); + Files.createDirectories(dotMo); + Files.write(dotMo.resolve("config.json"), + "{\"schema_version\":1,\"sources\":[{\"path\":\"custom-metadata\"}]}" + .getBytes(StandardCharsets.UTF_8)); + + MetaDataGeneratorMojo mojo = mojoWithSilentPom(weirdRoot); + MetaDataLoader loaded = mojo.createLoader(mojo.createProjectClassLoader()); + + assertEquals(1, loaded.getMetaObjects().size()); + assertEquals("Widget", loaded.getMetaObjects().get(0).getShortName()); + } finally { + deleteRecursive(root); + } + } + @Test(expected = MetaDataException.class) public void createLoaderRaisesWhenPomIsSilentAndNoCollectionExists() throws IOException { // Neither a neutral config nor a default "metaobjects/" directory — the final diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java index 0f083991f..391cb0d0c 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java +++ b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java @@ -108,8 +108,14 @@ public static Optional read(Path configDir) { JsonObject root = parsed.getAsJsonObject(); JsonElement version = root.get("schema_version"); + // Compared as a double, not getAsInt(): Gson's getAsInt() on a non-integral + // BigDecimal TRUNCATES rather than raising, so a typo'd `schema_version: 1.5` + // silently read as 1 and passed. getAsDouble() still accepts an + // integral-valued float like `1.0` (equal to 1, and valid JSON — every other + // port accepts it too, C#'s NeutralConfig.cs for the identical reason) while + // correctly rejecting a genuinely non-integral value. if (version == null || !version.isJsonPrimitive() || !version.getAsJsonPrimitive().isNumber() - || version.getAsInt() != SUPPORTED_SCHEMA_VERSION) { + || version.getAsDouble() != SUPPORTED_SCHEMA_VERSION) { throw new MetaDataException( path + ": unsupported schema_version (expected " + SUPPORTED_SCHEMA_VERSION + ")", ErrorCode.ERR_BAD_ATTR_VALUE); diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java index b9e9d9934..7dd719138 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java @@ -59,4 +59,23 @@ public void nonStringSourceValueRaises() throws IOException { MetaDataException ex = assertThrows(MetaDataException.class, () -> NeutralConfig.read(dir)); assertEquals(ErrorCode.ERR_BAD_ATTR_VALUE, ex.getCode().orElseThrow()); } + + @Test + public void schemaVersionAsFloatLiteralIsAcceptedLikeTheOtherThreePorts() throws IOException { + // `schema_version: 1.0` is valid JSON, equal to 1, and every other port + // accepts it (C#'s NeutralConfigTests.cs pins the identical case). + Path dir = writeConfig("{ \"schema_version\": 1.0, \"sources\": [ { \"path\": \"model\" } ] }"); + NeutralConfig cfg = NeutralConfig.read(dir).orElseThrow(); + assertEquals(1, cfg.getSources().size()); + } + + @Test + public void schemaVersionNonIntegralRaisesRatherThanTruncating() throws IOException { + // Regression: Gson's JsonPrimitive#getAsInt() TRUNCATES a non-integral + // BigDecimal instead of raising, so `schema_version: 1.5` used to read as + // 1 and pass silently. Comparing as a double (NeutralConfig.java) catches it. + Path dir = writeConfig("{ \"schema_version\": 1.5, \"sources\": [] }"); + MetaDataException ex = assertThrows(MetaDataException.class, () -> NeutralConfig.read(dir)); + assertEquals(ErrorCode.ERR_BAD_ATTR_VALUE, ex.getCode().orElseThrow()); + } } From 3448eb1362c9c5e26188257976380b0f357d7b67 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 19:19:45 -0400 Subject: [PATCH 43/44] refactor(python): remove resolve_metadata_location's dead rung-1 duplicate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit No command handler ever called resolve_metadata_location with a non-None `explicit` — rung 1 (the positional metadata_dir) is served by the pre-existing, independently-correct `_load_root(args.metadata_dir)` in every command handler (a plain MetaDataLoader.from_directory call, with no resolve-then-rejoin path of its own to regress). The dead branch duplicated rung 1 via resolve_sources instead, reachable from nothing, so it could silently drift from the real path with no test noticing — e.g. it would have picked up source_resolver.py's CLI-facing `_pending` exclusion while `_load_root`'s loader-level walk does not. Removed rather than wired: replumbing every metadataDir-taking command onto resolve_sources's from_uris-based load instead of from_directory is a materially larger, riskier change than this ladder needs. Its now-pointless regression test (pinning a relative-path defect that cannot occur in the code path actually reachable from the CLI) is removed with it. Co-Authored-By: Claude Opus 5 (1M context) --- server/python/src/metaobjects/cli.py | 31 +++++++++++++++------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index 31b7560b8..1a337d0eb 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -438,33 +438,36 @@ def gen_state_dir_for(metadata_dir: str) -> str: def resolve_metadata_location( - explicit: str | None, config: ProjectConfig | None, root: Path, ) -> list[str]: - """The precedence ladder for where metadata lives. First match wins. + """The precedence ladder for where metadata lives, rungs 2-4. First match wins. - 1. An explicit CLI argument (the positional ``metadata_dir``). 2. This port's native surface — ``metadata`` in ``metaobjects.config.yaml``. 3. ``sources`` in the port-neutral ``.metaobjects/config.json``. 4. The built-in default directory. + Rung 1 — an explicit CLI argument (the positional ``metadata_dir``) — is + NOT this function's concern: every command handler that accepts one already + loads it directly via ``_load_root(args.metadata_dir)`` (``MetaDataLoader + .from_directory``, a single-directory load with no resolve-then-rejoin step + of its own) BEFORE this function would even be reachable, so this function is + only ever called with rung 1 already having been tried and found absent. An + earlier revision accepted an ``explicit`` parameter and duplicated rung 1 + here via `resolve_sources` — reachable from no command handler, so it could + silently drift from the real rung-1 path (e.g. picking up + `source_resolver.py`'s CLI-facing `_pending` exclusion while + `_load_root`'s loader-level walk does not) with nothing to notice. Removed + rather than wired: replumbing every metadataDir-taking command onto + `resolve_sources`'s `from_uris`-based load instead of `from_directory` is a + materially different, higher-risk change than this ladder needs. + A file that EXISTS at any rung but is malformed raises rather than falling through to the next rung. See `docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §5. """ from metaobjects.config.source_resolver import resolve_collection, resolve_sources - if explicit is not None: - # Resolve to an absolute path FIRST and pass it as an absolute spec — - # `resolve_sources` takes an absolute `path` as-is, so the base is - # irrelevant. Joining a relative `explicit` onto its own already- - # absolute parent (the naive approach) resolves one level too deep. - return [ - str(p) - for p in resolve_sources(root, [{"path": str(Path(explicit).resolve())}]) - ] - if config is not None: # `config.metadata_dir()` is already resolved to an absolute path # (`ProjectConfig._resolve_under`), so the base passed here is @@ -487,7 +490,7 @@ def _resolve_metadata_location_or_print_error( the caller has nothing further to print and should return 1. """ try: - return resolve_metadata_location(explicit=None, config=config, root=root) + return resolve_metadata_location(config=config, root=root) except ParseError as exc: print(f"error: could not resolve metadata location: {exc}", file=sys.stderr) return None From c9dda9b393a233d089dd3b100fb1c8253ffafc16 Mon Sep 17 00:00:00 2001 From: Doug Mealing Date: Wed, 19 Aug 2026 19:19:51 -0400 Subject: [PATCH 44/44] docs: changelog entries for symlink-following and the Java mojo's behavior change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the symlink-following fix (Java/Python) and, per the un-flagged gap the review caught, the Java/Maven behavior change: a naming neither nor , with nothing else to resolve, now fails the build (ERR_COLLECTION_NOT_FOUND) instead of silently loading an empty model and passing — deliberate and tested, but previously undisclosed. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b440e56d..9429b41d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,23 @@ guide: [`docs/features/metadata-sources.md`](docs/features/metadata-sources.md). but which error is each port's own — verified empirically: TypeScript raises a raw `ZodError` with no code at all, Python raises `ERR_COLLECTION_NOT_FOUND`, C# and Java both raise `ERR_BAD_ATTR_VALUE`. +- **Directory expansion follows symlinked directories in all four ports** — + including when a declared `sources` path is itself a symlink, or a symlink + sits partway through a walked tree. TypeScript and C# already did; Java and + Python now match (a symlinked `sources` path previously resolved to zero + files in Java, silently, exit 0). A symlink cycle is a loud error rather + than a hang. Gated by two new `symlinks`-bearing corpus cases. +- **Behavior change (Java/Maven only): a `` naming neither + `` nor ``, with no `.metaobjects/config.json` `sources` + and no default `metaobjects/` directory, now FAILS the build** + (`ERR_COLLECTION_NOT_FOUND`) instead of silently producing an empty model + and passing. This is the one behavior change here that can break an + existing `mvn metaobjects:generate`/`:verify` — most likely to bite a + multi-module reactor where a parent pom configures `` and one child + module never adds its own ``. To restore the old outcome, declare + ``/`` explicitly in that module's pom, or give it a real + metadata source (a `metaobjects/` directory or a `.metaobjects/config.json` + `sources` entry). ### Changed — a committed migration chain must replay from empty, and `meta migrate` stops writing chains that cannot ([#313](https://github.com/metaobjectsdev/metaobjects/issues/313))