Skip to content

mdcode: publish relationships as semantic-relationship entries, and make many-to-many authorable with through #403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
libei wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

mdcode: publish relationships as semantic-relationship entries, and make many-to-many authorable with through #403

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
96d2038
mdcode: make many-to-many relationships authorable
libei Sep 8, 2026
f25b94b
mdcode: publish many-to-many relationships to Knowledge Catalog
libei Sep 8, 2026
146a4bf
mdcode: rename the M:N construct to `through` on the relationship
libei Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions toolbox/mdcode/docs/semantic-model/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,14 @@ Field and relationship names are the business vocabulary — `order_id`,
metric's `expression` may be a bare formula over the logical fields or the fuller
per-dialect form. `entities` may also be written `datasets` (the two are interchangeable under the `/google` version).

A relationship that pairs many rows on each side — a student takes many courses,
a course has many students — adds `through`, naming the table that holds the
pairs. `from_columns` and `to_columns` then reach each side from that table
rather than from the endpoints, and the relationship may carry a key of its own
plus any fields the pairing itself has (an enrollment's grade, say). Both graphs
deploy it as an edge table over that table. See
[Model spec §2.2.1](model_spec.md#221-many-to-many-through).

Entities can **extend** other entities (`extends: [Parent]`); push flattens the
supertype's fields down and expresses the hierarchy as graph labels, so a query
against the supertype gathers every subtype. See
Expand Down
70 changes: 40 additions & 30 deletions toolbox/mdcode/docs/semantic-model/fidelity.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ agree on every structural row and differ only where a Spanner target has no
| Primary key | `schema.primaryKey` | ✓ | `KEY(...)` on the node table | `KEY(...)` on the node table |
| Unique keys | `schema.uniqueConstraints` | ✓ | — dropped (only PK emitted) | — dropped (only PK emitted) |
| Metric | `semantic-metric` entry | name, entity, description, instructions, type⁵ | `MEASURE`⁴ | — dropped (no `MEASURE`) |
| Relationship (1:1 / 1:N) | `schema-join` link | ✓ (name normalized⁶) | `EDGE TABLE` | `EDGE TABLE` |
| Relationship (M:N / `association`) | — not stored | — | `EDGE TABLE` (via junction table) | `EDGE TABLE` (via junction table) |
| Relationship (foreign key) | `semantic-relationship` entry⁶ | ✓ | `EDGE TABLE` | `EDGE TABLE` |
| Relationship (`through` a table of pairs) | `semantic-relationship` entry⁶ | ✓¹³ | `EDGE TABLE` over that table | `EDGE TABLE` over that table |
| Entity `extends` | — not modelled | — | `LABEL` clauses + flattened fields | `LABEL` clauses + flattened fields |
| Action | `semantic-action` entry¹² | ✓¹² | — not represented (write-side) | — not represented (write-side) |
| `description` (entity / metric / field / relationship) | entry description / aspect | ✓ | `OPTIONS(description)` | — dropped |
Expand Down Expand Up @@ -63,14 +63,19 @@ agree on every structural row and differ only where a Spanner target has no
5. **Metric type.** A metric's expression is gated behind `--emit-expressions`;
its data type round-trips only for a concrete type (e.g. `Decimal`) — an
untyped, `String`, or `Opaque` metric comes back un-typed.
6. **Relationship name.** Relationship names come back lowercased/hyphenated
(`Places Order` → `places-order`) — the catalog stores the name only in the
link id. See [Writer-side follow-up](#writer-side-follow-up).
6. **Relationship storage.** Every relationship is stored as one
`semantic-relationship` entry under the model entry, holding both endpoints,
the join columns, and the relationship's `instructions`. A relationship
carried by a foreign key *also* gets a `schema-join` link, so Dataplex
surfaces that read joins still see it; the entry is the fidelity record and
the link the graph-shaped projection. Because the name rides the entry, it
comes back verbatim — a link could only ever return the lowercased,
hyphenated form of its id.
7. **Guidelines aspect.** The `guidelines` aspect exists only for the model,
entities, and metrics — not fields or relationships, so field- and
relationship-level `ai_context.instructions` has no Knowledge Catalog home (a
relationship's instructions still reach BigQuery, folded into the edge's
`OPTIONS(description)`).
entities, and metrics — not fields, so field-level
`ai_context.instructions` has no Knowledge Catalog home. A relationship's
instructions ride its own `semantic-relationship` aspect instead: that aspect
type is ours, so it has a field for them.
8. **Model-level metadata.** Neither graph has a home for statement-level
metadata — BigQuery silently drops graph-statement `OPTIONS`, and Spanner
carries no `OPTIONS` at all — so the model's `description` and
Expand All @@ -86,7 +91,8 @@ agree on every structural row and differ only where a Spanner target has no
10. **Logical (unbound) model.** A model with no bindings still publishes to
Knowledge Catalog: each entity's `source` is recorded empty (`resources: []`)
because there is no table behind it, and a relationship that carries no join
columns is skipped with a warning. When the same push also deploys a graph,
columns publishes as an entry with none (it gets no `schema-join` link, which
requires a column pair). When the same push also deploys a graph,
the catalog entries are first pruned to what the graph binds — see
[To Knowledge Catalog](#to-knowledge-catalog).
11. **Vendor-dialect fallback.** What you author is the `expression.dialects[]`
Expand All @@ -103,12 +109,22 @@ agree on every structural row and differ only where a Spanner target has no
scope: an action's `precondition` and `affects` are not modelled, so nothing
about them is stored either way. See
[Modeling write operations](actions.md).
13. **Relationships through a table of pairs.** A relationship with a `through`
table gets no `schema-join` link: that link holds exactly one source/target
column pair, which cannot describe an edge running through a third table. Its
entry carries the rest — the `through` table, its key, and the fields of the
pairing — so the whole relationship round-trips, the edge's own field
expressions included. Those are stored unconditionally: the aspect type is
ours, so it has fields for them, where the `--emit-expressions` gate exists
for the published system templates that do not. See
[Model spec §2.2.1](model_spec.md#221-many-to-many-through).

## To Knowledge Catalog

The catalog holds metadata rather than a full copy of your model. Every resource
type it uses is a built-in system type under `dataplex-types/global`, apart from
the custom `semantic-action` pair that `kcmd init` provisions — push references
the custom `semantic-relationship` and `semantic-action` pairs that `kcmd init`
provisions — push references
types, it never creates them (see
[Reference → What gets created in Knowledge Catalog](reference.md#what-gets-created-in-knowledge-catalog)).

Expand All @@ -122,7 +138,7 @@ authored model.

A logical model still produces complete entries. Each entity's `source` is
recorded empty (`resources: []`) because there is no table behind it, and a
relationship that carries no join columns is skipped with a warning.
relationship that carries no join columns publishes as an entry with none.

By default the catalog does **not** store the SQL expressions: the published
system-type templates do not yet carry a per-field `semantics` block or a
Expand All @@ -135,6 +151,18 @@ SQL (`importedExpression` — for example the MAQL or Snowflake form a metric wa
imported from). Those stay in your authored document; the vendor SQL and
expressions are still used when generating graph SQL.

**Relationships** get an entry rather than a link. A `schema-join` link holds one
source/target column pair, which cannot describe an edge running through a table
of pairs, and it has no field for the relationship's name; Knowledge Catalog has
no custom *link* types either — only custom entry and aspect types. So each
relationship becomes a `semantic-relationship` entry under the model entry,
carrying the two entities it pairs, the join columns, and, when there is one, the
`through` table with its key and fields. All of it round-trips through `pull`,
name included. A relationship carried by a foreign key also keeps its
`schema-join` link, for the Dataplex surfaces that read joins. The entry type is
custom, so `kcmd init` creates it; a model with no relationships never needs
it.

**Actions** follow the same one-entry-per-element rule as everything else: each
becomes a `semantic-action` entry under the model entry, carrying its executor
and typed parameters in a `semantic-action` aspect. They round-trip losslessly
Expand Down Expand Up @@ -205,9 +233,6 @@ returns that view. Two things about *how* it comes back:

**Normalized** — the content survives, the form changes:

- Relationship *names* come back lowercased/hyphenated (`Places Order` →
`places-order`); the catalog stores the name only in the link id. See
[Writer-side follow-up](#writer-side-follow-up).
- Field types round-trip except two collapses: a field authored with no type
comes back as `Opaque`, and a field authored as `String` comes back un-typed
(both store `dataType STRING`, kept distinct by a field's `metadataType` — see
Expand All @@ -226,21 +251,6 @@ returns that view. Two things about *how* it comes back:
pulled document as a faithful copy of the catalog metadata rather than of the
authored model, and keep the authored document as the source of truth.

## Writer-side follow-up

One reduction above is a limit of what push currently *writes* rather than of
what pull can recover. It is recorded here as a write-side follow-up; the reader
(pull) already returns everything the catalog holds.

- **Relationship names.** The `schema-join` aspect type's `metadataTemplate` has
no field for the relationship name, so push cannot store it and pull recovers
it from the link id — which is lowercased and hyphenated (the entry-link id
format forbids the original casing/underscores). Returning the name verbatim
requires adding a name field to the built-in `schema-join` aspect type in
Knowledge Catalog (server-side), after which the client write/read is trivial;
it is the same class of gap as the `semantics` field that gates
`--emit-expressions`.

(A non-canonical deployment target is **not** a pull gap: push rejects it at the
validation gate before any leg runs, so it is never written — see
[Validation](reference.md#validation).)
79 changes: 61 additions & 18 deletions toolbox/mdcode/docs/semantic-model/model_spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,16 +199,19 @@ A relationship is a directed edge between two datasets.
| `name` | string | required |
| `from` | string | required; a declared dataset name |
| `to` | string | required; a declared dataset name |
| `from_columns` | list of strings | join key on `from` |
| `to_columns` | list of strings | join key on `to` |
| `from_columns` | list of strings | join key reaching `from` |
| `to_columns` | list of strings | join key reaching `to` |
| `through` | string | `/google` only; a table holding the pairs · [§2.2.1](#221-many-to-many-through) |
| `keys` | list of strings | `/google` only; requires `through` |
| `fields` | list of [field](#211-field) | `/google` only; requires `through` |
| `description` | string | |
| `ai_context` | [ai_context](#24-ai_context) | |
| `custom_extensions` | list | [§6](#6-the-extension-mechanism) |

`from_columns` and `to_columns` are the edge's join keys. They MUST be given
together (a bound edge) or both omitted (a logical edge); one without the other is
rejected. When both are given they MUST have equal length. `from` and `to` MUST
name datasets declared in the same model.
rejected. Without `through` they MUST have equal length, because they pair up
positionally. `from` and `to` MUST name datasets declared in the same model.

Ossie **requires** both join-column lists; allowing both to be omitted — a
logical edge with no join keys — is a `kcmd` **relaxation** ([§4](#4-narrowings-and-relaxations))
Expand All @@ -217,10 +220,47 @@ requires them (see [§4](#4-narrowings-and-relaxations)). Unlike `source` and fi
join columns are declared on the logical model and are not profile-swappable
([§7](#7-the-binding-layer)).

> **Many-to-many is not yet authorable.** A junction-table (M:N) relationship
> exists in `kcmd`'s internal representation but has **no YAML syntax** in
> `0.2.0.dev0`. It cannot be authored today and is reserved for a future format
> extension. Direct-FK relationships (1:1, 1:N) are the authorable forms.
#### 2.2.1. Many-to-many (`through`)

A relationship whose two sides each match many of the other cannot be carried by
a foreign key: a foreign-key column holds one value, so it references at most one
row. The pairs live in a table of their own — one row per pair — and the
relationship names it with `through`.

```yaml
relationships:
- name: enrollment
from: students
to: courses
through: analytics.school.enrollment
keys: [enrollment_id]
from_columns: [student_id]
to_columns: [course_id]
fields:
- name: grade
expression: enrollment.grade
```

`through` changes where `from_columns` and `to_columns` live, not what they mean.
Without it they are on the two endpoints' own tables; with it both are on the
table named by `through`, one list reaching each endpoint's declared
`primary_key`. Because the two lists no longer pair up with each other, they need
not have equal length. Both are required when `through` is given: without them
nothing says which pairs the table holds.

`keys` is the edge's own key, on the `through` table. It defaults to the two
column lists combined and deduplicated, which is unique whenever a pair appears
at most once; give it explicitly for a surrogate key, or when a pair may
legitimately repeat (an enrollment per term).

`fields` are properties of the pairing rather than of either endpoint — a grade
belongs to the enrollment, not to the student or the course.

`keys` and `fields` both require `through`. An edge carried by a foreign key has
no table of its own, so it has nowhere to put a key or a property.

`through`, `keys`, and `fields` are native keys of the extended profile
([§5](#5-extensions)); vanilla Ossie has no syntax for a pairing table.

### 2.3. Metric

Expand Down Expand Up @@ -305,7 +345,7 @@ extension, [§5](#5-extensions)), or *rejected* / *not authorable* (excluded).
| `abstract` | — | added | supertype with no table; `/google` only · [§5](#5-extensions) |
| relationship `name`, `from`, `to` | defined | same | [§2.2](#22-relationship) |
| relationship `from_columns` / `to_columns` | required | optional | model before binding; none = logical edge · [§4.2](#42-relaxations-looser-than-ossie) |
| relationship M:N (`association`) | — | not authorable (reserved) | no M:N syntax yet · [§2.2](#22-relationship) |
| relationship M:N (`through`) | — | added | pairing table + its own key and fields; `/google` only · [§2.2.1](#221-many-to-many-through), [§5](#5-extensions) |
| `metrics`, metric `expression` | required | same; graph-bound stricter | a graph measure binds one node and aggregate · [§4.1](#41-narrowings-stricter-than-ossie) |
| `expression.dialects` | closed enum | any dialect string | tolerate imported / newer input · [§4.2](#42-relaxations-looser-than-ossie) |
| field `expression` (column binding) | required | optional | model before binding; unbound is pruned · [§4.2](#42-relaxations-looser-than-ossie), [§7](#7-the-binding-layer) |
Expand Down Expand Up @@ -353,8 +393,9 @@ Each rule and its reason:
measure.

- **A graph-bound relationship MUST have its join columns bound.** For any graph
target, a non-M:N relationship MUST supply both `from_columns` and `to_columns`
before deploy. *Why:* the edge table needs both keys.
target, a relationship MUST supply both `from_columns` and `to_columns` before
deploy, whether they sit on the endpoints' own tables or on a `through` table.
*Why:* the edge table needs both keys.

- **Unknown keys are rejected.** Every object is validated closed: an unrecognized
sibling key is a hard load error, not silently dropped. Combined with the version
Expand Down Expand Up @@ -453,10 +494,11 @@ reads the document ([§6](#6-the-extension-mechanism)).
bindings, so one logical model serves several stores. Not part of the Ossie
document; a `kcmd`-specific file alongside it ([§7](#7-the-binding-layer)).

Deliberately **not** extensions in `0.2.0.dev0`, to avoid the impression they
exist: there is **no `actions` block** and **no authorable M:N `association`
syntax**. Both are reserved for future consideration; neither is part of the
format today.
- **`through` (extended profile only).** A many-to-many relationship, backed
by a table of pairs with its own key and its own properties. Ossie's
relationship is a foreign key only, and the carrier cannot express one either:
a `custom_extensions` block holds opaque data, and this edge has to be read by
the graph generators. Grammar in [§2.2.1](#221-many-to-many-through).

## 6. The extension mechanism

Expand Down Expand Up @@ -605,9 +647,10 @@ The full merge behavior and worked examples are in
and constructs with no vanilla form (inheritance, the `entities` spelling) are
simply unavailable there — a model that needs them uses `0.2.0.dev0/google`.

- **Reserved constructs.** `association` (M:N) and any `actions`-like write-side
construct are reserved: recognized as future work, not authorable today. A
document MUST NOT rely on either in `0.2.0.dev0`.
- **Extensions are additive.** `through` and `actions` were both added to
the extended profile after `0.2.0.dev0/google` was first published, each as a
new optional key. A document that used neither is unaffected, which is the
shape any further extension takes.

## Appendix: annotated example

Expand Down
2 changes: 1 addition & 1 deletion toolbox/mdcode/docs/semantic-model/profiles.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ each field reads. A profile sets binding and leaves declaration alone.
| an entity's `source` (its store URI) | which entities, fields, relationships, or metrics exist, and what each means |
| a field's column (its `expression`, a bare column reference) | a field's `label`, `description`, `dimension`, `datatype` |
| whether a field is bound at all under this profile | the grain (`primary_key` / `unique_keys`) and graph shape (`from`/`to`, `from_columns`/`to_columns`) |
| the deployment target | a field `expression` that is arbitrary SQL, which changes the computation; any `metric` definition; any `ai_context` / synonyms; a relationship or its junction `source` |
| the deployment target | a field `expression` that is arbitrary SQL, which changes the computation; any `metric` definition; any `ai_context` / synonyms; a relationship or its `through` table |

An element's `name` is not overridden — it is the key that pairs a profile
element with the model element it binds. The grain and the join columns name
Expand Down
Loading