Conversation
A many-to-many edge is backed by a junction table rather than by a foreign
key on either endpoint. The IR has modelled that (`Association`) since the
BigQuery graph generator gained it, and the Spanner generator renders it too,
but there was no way to write one down: the format's relationship schema knew
only the direct foreign key, so the only models that reached those generators
were hand-built IR in tests.
Adds an `association` block on a relationship, a native key of the extended
profile ('0.2.0.dev0/google'). It names the junction table, the edge's own key,
the junction columns that reference each endpoint's key, and the properties of
the pairing itself. The relationship's own from_columns/to_columns must be
absent when it is present -- the two are alternative bindings, and neither
endpoint holds a foreign key.
Vanilla Ossie rejects the key: it has no junction-table syntax and no
custom_extensions encoding for one, so accepting it there would silently drop
the junction on load.
The serializer writes the block back, so a many-to-many model round-trips
through a pull whole instead of collapsing to a direct-FK view.
The two committed goldens (school_manytomany.{bigquery,spanner}.golden.sql,
run against a live BigQuery instance and traversed with a GQL MATCH) now
render from an authored document rather than hand-built IR, byte for byte --
so the format-to-DDL path is covered end to end.
A `schema-join` entry link holds exactly one source/target column pair, so it cannot describe an edge that runs through a junction table, and Dataplex has no custom entry-LINK types -- only custom entry and aspect types. So a many-to-many relationship is published as an entry instead: one `semantic-association` entry per relationship, parented to the model entry, holding the two entities it pairs and the junction table with its keys, join columns, and fields. The custom type is added the same way `semantic-action` was: one entry type and one aspect type sharing an id, appended to CUSTOM_TYPES in kc_custom_types.ts and provisioned by `kcmd init --semantic-model` in the destination project at global. Nothing in provisioning, naming, or init wiring needed changing. The encoding lives in kc_associations.ts, mirroring kc_actions.ts. Both the relationship's `instructions` and the edge's own fields ride the custom aspect rather than the built-in `guidelines` / `schema` aspects, because a pull derives the aspect base from the entry type's project and the custom type lives in the destination project. Field expressions are stored unconditionally -- the `--emit-expressions` gate exists for the published system templates that lack the fields; this template is ours. Push, pull, and the round-trip goldens cover the new shape, and the docs (model spec, fidelity, reference, README) describe it alongside the foreign-key edge.
`association` named a concept the graph stores does not have. BigQuery
Graph and Spanner Graph both model one thing here -- a relationship with
a name, two endpoints, join columns, and properties -- and whether the
edge is carried by a foreign key or by a table of pairs changes only
where those columns live. So the authored form flattens: a relationship
gains `through` (the table it runs through), plus the `keys` and `fields`
that table makes possible. `from_columns`/`to_columns` keep their
meaning; `through` says which table they are on.
The Knowledge Catalog type follows the same reasoning, and widens.
`semantic-association` becomes `semantic-relationship`, and EVERY
relationship publishes as one such entry, not just a many-to-many one --
the aspect already fit the foreign-key case with no new fields. A
foreign-key relationship also keeps its `schema-join` link, for the
Dataplex surfaces that read links: the entry is the fidelity record, the
link the graph-shaped projection. Pull prefers the entries and falls back
to the links for a catalog written by an older kcmd.
Widening closes three losses the fidelity doc previously recorded:
- a relationship's name came back lowercased and hyphenated from the
link id; the entry carries it verbatim
- relationship-level ai_context.instructions had no catalog home
- a purely logical, column-less relationship was skipped entirely
Also fixes a defect the flattening exposed: pruneUnavailable keyed a
relationship's join columns to its endpoint entities, which is wrong for
a through-edge -- those columns are on the through table, so a same-named
unbound field on an endpoint would have dropped the edge.
semantic-relationship entries, and make many-to-many authorable with through
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Many-to-many relationships were unauthorable in the model format even though
both BigQuery Graph and Spanner Graph support the shape, and Knowledge Catalog
had nowhere to put one. Two parts: the format, then the catalog.
1. Make many-to-many authorable (
through)BigQuery Graph and Spanner Graph model one thing here — a relationship with a
name, two endpoints, join columns, and properties — and whether the edge is
carried by a foreign key or by a table of pairs changes only where those
columns live. So the authored form stays flat: a relationship gains
through(the table it runs through), plus the
keysandfieldsthat table makespossible.
from_columns/to_columnskeep their meaning;throughsayswhich table they are on.
Both graph legs already knew how to emit an edge table over a table of pairs;
this gives it a spelling.
throughis/google-only, and it is a deliberatesuperset of released Apache OSI (whose relationship knows only the direct
foreign-key edge) — the schema guardrail tolerates exactly its three extra keys
and nothing else, so real drift still fails.
2. Publish every relationship as a
semantic-relationshipentryA
schema-joinentry link holds exactly one source/target column pair and hasno field for the relationship's name, so it cannot describe an edge that runs
through a third table — and Dataplex has no custom entry-link types, only
custom entry and aspect types. So a relationship is published as an entry:
one
semantic-relationshipentry per relationship, parented to the modelentry, holding its name, its two endpoints, its join columns and fields, and
the
throughtable when there is one.This covers every relationship, not just a many-to-many one — the aspect fit
the foreign-key case with no new fields. A foreign-key relationship also keeps
its
schema-joinlink, for the Dataplex surfaces that read links: the entry isthe fidelity record, the link the graph-shaped projection. Pull prefers the
entries and falls back to the links for a catalog written by an older kcmd.
Widening closes three fidelity losses the docs previously had to record:
the entry carries it verbatim
ai_context.instructionshad no catalog homeThe custom type is added exactly the way
semantic-actionwas — one entry typeand one aspect type sharing an id, appended to
CUSTOM_TYPESinkc_custom_types.tsand provisioned bykcmd init --semantic-modelin thedestination project at
global. Provisioning, naming, and the init wiringneeded no changes; the tests that covered them are now generic over
CUSTOM_TYPESrather than hard-coding the one type.The encoding lives in
kc_relationships.ts, mirroringkc_actions.ts, with thesame "when a built-in type ships, move to it" note in the header. Two decisions
worth flagging:
instructionsand the edge's own fields ride the customaspect, not the built-in
guidelines/schemaaspects, because a pullderives the aspect base from the entry type's project and the custom type
lives in the destination project.
expressionis stored unconditionally, unlike an entity's. The--emit-expressionsgate exists because the published system templates haveno field for expressions yet; this template is ours and does.
Also fixed
Two defects the flat shape exposed, both in code that assumed a relationship's
columns live on its endpoints:
from_columns/to_columns— a foreignkey pairs the two lists positionally, but a through table's lists do not pair
with each other at all, so a composite key on one side is valid there
pruneUnavailablekeyed a relationship's join columns asEntity.column,so a same-named unbound field on an endpoint would have dropped a
through-edge whose columns are on the through table
Tests and docs
school_manytomany.yamlplus OSI, Knowledge Catalog, and pullgoldens, added to the OSI, KC, e2e, and symmetry corpora. Its BigQuery and
Spanner DDL goldens were run against a live BigQuery instance and traversed
with a GQL
MATCH.with real coverage: the round trip, a foreign-key and a many-to-many edge
coexisting in one model, an unpublished endpoint, and a through table with no
columns. Recovery is tested both ways — through the entry, and links-only for
an older catalog.
npx tsc --noEmitclean; the full suite passes.fidelity.md/reference.md/README.md/profiles.mddescribethroughalongside the foreign-key edge.