Skip to content

mdcode: publish relationships as semantic-relationship entries, and make many-to-many authorable with through - #403

Draft
libei wants to merge 3 commits into
GoogleCloudPlatform:mainfrom
libei:mn-relationships
Draft

libei wants to merge 3 commits into
GoogleCloudPlatform:mainfrom
libei:mn-relationships

Conversation

@libei

@libei libei commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Many-to-many relationships were unauthorable in the model format even though
both BigQuery Graph and Spanner Graph support the shape, and Knowledge Catalog
had nowhere to put one. Two parts: the format, then the catalog.

1. Make many-to-many authorable (through)

BigQuery Graph and Spanner Graph model one thing here — a relationship with a
name, two endpoints, join columns, and properties — and whether the edge is
carried by a foreign key or by a table of pairs changes only where those
columns live. So the authored form stays flat: a relationship gains through
(the table it runs through), plus the keys and fields that table makes
possible. from_columns / to_columns keep their meaning; through says
which table they are on.

relationships:
  - name: enrollment
    from: students
    to: courses
    through: analytics.school.enrollment  # the table of pairs
    keys: [enrollment_id]
    from_columns: [student_id]            # through -> students
    to_columns: [course_id]               # through -> courses
    fields:                               # the pairing's own fields
      - name: grade
        expression: enrollment.grade

Both graph legs already knew how to emit an edge table over a table of pairs;
this gives it a spelling. through is /google-only, and it is a deliberate
superset of released Apache OSI (whose relationship knows only the direct
foreign-key edge) — the schema guardrail tolerates exactly its three extra keys
and nothing else, so real drift still fails.

2. Publish every relationship as a semantic-relationship entry

A schema-join entry link holds exactly one source/target column pair and has
no field for the relationship's name, so it cannot describe an edge that runs
through a third table — and Dataplex has no custom entry-link types, only
custom entry and aspect types. So a relationship is published as an entry:
one semantic-relationship entry per relationship, parented to the model
entry, holding its name, its two endpoints, its join columns and fields, and
the through table when there is one.

This covers every relationship, not just a many-to-many one — the aspect fit
the foreign-key case with no new fields. A foreign-key relationship also keeps
its schema-join link, for the Dataplex surfaces that read links: the entry is
the fidelity record, the link the graph-shaped projection. Pull prefers the
entries and falls back to the links for a catalog written by an older kcmd.

Widening closes three fidelity losses the docs previously had to record:

  • a relationship's name came back lowercased and hyphenated from the link id;
    the entry carries it verbatim
  • relationship-level ai_context.instructions had no catalog home
  • a purely logical, column-less relationship was skipped entirely

The custom type is added exactly the way semantic-action was — one entry type
and one aspect type sharing an id, appended to CUSTOM_TYPES in
kc_custom_types.ts and provisioned by kcmd init --semantic-model in the
destination project at global. Provisioning, naming, and the init wiring
needed no changes; the tests that covered them are now generic over
CUSTOM_TYPES rather than hard-coding the one type.

The encoding lives in kc_relationships.ts, mirroring kc_actions.ts, with the
same "when a built-in type ships, move to it" note in the header. Two decisions
worth flagging:

  • The relationship's instructions and the edge's own fields ride the custom
    aspect
    , not the built-in guidelines / schema aspects, because a pull
    derives the aspect base from the entry type's project and the custom type
    lives in the destination project.
  • Field expression is stored unconditionally, unlike an entity's. The
    --emit-expressions gate exists because the published system templates have
    no field for expressions yet; this template is ours and does.

Also fixed

Two defects the flat shape exposed, both in code that assumed a relationship's
columns live on its endpoints:

  • the loader's equal-length check on from_columns / to_columns — a foreign
    key pairs the two lists positionally, but a through table's lists do not pair
    with each other at all, so a composite key on one side is valid there
  • pruneUnavailable keyed a relationship's join columns as Entity.column,
    so a same-named unbound field on an endpoint would have dropped a
    through-edge whose columns are on the through table

Tests and docs

  • New fixture school_manytomany.yaml plus OSI, Knowledge Catalog, and pull
    goldens, added to the OSI, KC, e2e, and symmetry corpora. Its BigQuery and
    Spanner DDL goldens were run against a live BigQuery instance and traversed
    with a GQL MATCH.
  • The two tests that asserted "many-to-many is skipped and warned" are replaced
    with real coverage: the round trip, a foreign-key and a many-to-many edge
    coexisting in one model, an unpublished endpoint, and a through table with no
    columns. Recovery is tested both ways — through the entry, and links-only for
    an older catalog.
  • npx tsc --noEmit clean; the full suite passes.
  • Docs: model-spec §2.2.1, and fidelity.md / reference.md / README.md /
    profiles.md describe through alongside the foreign-key edge.

A many-to-many edge is backed by a junction table rather than by a foreign
key on either endpoint. The IR has modelled that (`Association`) since the
BigQuery graph generator gained it, and the Spanner generator renders it too,
but there was no way to write one down: the format's relationship schema knew
only the direct foreign key, so the only models that reached those generators
were hand-built IR in tests.

Adds an `association` block on a relationship, a native key of the extended
profile ('0.2.0.dev0/google'). It names the junction table, the edge's own key,
the junction columns that reference each endpoint's key, and the properties of
the pairing itself. The relationship's own from_columns/to_columns must be
absent when it is present -- the two are alternative bindings, and neither
endpoint holds a foreign key.

Vanilla Ossie rejects the key: it has no junction-table syntax and no
custom_extensions encoding for one, so accepting it there would silently drop
the junction on load.

The serializer writes the block back, so a many-to-many model round-trips
through a pull whole instead of collapsing to a direct-FK view.

The two committed goldens (school_manytomany.{bigquery,spanner}.golden.sql,
run against a live BigQuery instance and traversed with a GQL MATCH) now
render from an authored document rather than hand-built IR, byte for byte --
so the format-to-DDL path is covered end to end.
A `schema-join` entry link holds exactly one source/target column pair, so
it cannot describe an edge that runs through a junction table, and Dataplex
has no custom entry-LINK types -- only custom entry and aspect types. So a
many-to-many relationship is published as an entry instead: one
`semantic-association` entry per relationship, parented to the model entry,
holding the two entities it pairs and the junction table with its keys, join
columns, and fields.

The custom type is added the same way `semantic-action` was: one entry type
and one aspect type sharing an id, appended to CUSTOM_TYPES in
kc_custom_types.ts and provisioned by `kcmd init --semantic-model` in the
destination project at global. Nothing in provisioning, naming, or init
wiring needed changing.

The encoding lives in kc_associations.ts, mirroring kc_actions.ts. Both the
relationship's `instructions` and the edge's own fields ride the custom
aspect rather than the built-in `guidelines` / `schema` aspects, because a
pull derives the aspect base from the entry type's project and the custom
type lives in the destination project. Field expressions are stored
unconditionally -- the `--emit-expressions` gate exists for the published
system templates that lack the fields; this template is ours.

Push, pull, and the round-trip goldens cover the new shape, and the docs
(model spec, fidelity, reference, README) describe it alongside the
foreign-key edge.
`association` named a concept the graph stores does not have. BigQuery
Graph and Spanner Graph both model one thing here -- a relationship with
a name, two endpoints, join columns, and properties -- and whether the
edge is carried by a foreign key or by a table of pairs changes only
where those columns live. So the authored form flattens: a relationship
gains `through` (the table it runs through), plus the `keys` and `fields`
that table makes possible. `from_columns`/`to_columns` keep their
meaning; `through` says which table they are on.

The Knowledge Catalog type follows the same reasoning, and widens.
`semantic-association` becomes `semantic-relationship`, and EVERY
relationship publishes as one such entry, not just a many-to-many one --
the aspect already fit the foreign-key case with no new fields. A
foreign-key relationship also keeps its `schema-join` link, for the
Dataplex surfaces that read links: the entry is the fidelity record, the
link the graph-shaped projection. Pull prefers the entries and falls back
to the links for a catalog written by an older kcmd.

Widening closes three losses the fidelity doc previously recorded:

  - a relationship's name came back lowercased and hyphenated from the
    link id; the entry carries it verbatim
  - relationship-level ai_context.instructions had no catalog home
  - a purely logical, column-less relationship was skipped entirely

Also fixes a defect the flattening exposed: pruneUnavailable keyed a
relationship's join columns to its endpoint entities, which is wrong for
a through-edge -- those columns are on the through table, so a same-named
unbound field on an endpoint would have dropped the edge.
@libei libei changed the title mdcode: support many-to-many relationships mdcode: publish relationships as semantic-relationship entries, and make many-to-many authorable with through Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant