mdcode: store SQL expressions in the sql-expressions companion aspect

toolbox/mdcode/docs/semantic-model.md

@@ -13,8 +13,9 @@ model to two destinations at once:
 Both are generated from the same source document — you never author them
 separately, and a single `push` keeps them in sync.
 
-This guide covers authoring, deploying, and updating a model. For the Ossie
-document format itself, see [ossie.apache.org](https://ossie.apache.org/).
+This guide covers authoring, deploying, pulling back, and updating a model.
+For the Ossie document format itself, see
+[ossie.apache.org](https://ossie.apache.org/).
 
 ## Prerequisites
 
@@ -114,7 +115,7 @@ kcmd push --print              # also print the generated DDL / entry plan
 | `--validate-only` | Run every validation check and report pass/fail, but write nothing. |
 | `--print` | Print each destination's generated artifact (BigQuery DDL, Knowledge Catalog entry plan). Combine with `--validate-only` to preview without deploying. |
 | `--force-remove` | Delete models in the entry group that this push no longer includes (see [Updating and removing models](#updating-and-removing-models)). |
-| `--emit-expressions` | Also write the SQL-expression fields (per-field `schema.semantics` and `semantic-metric.expression`) to Knowledge Catalog. Off by default: the published system-type templates do not carry them yet. Knowledge Catalog push only. |
+| `--emit-expressions` | Also write the SQL expressions to Knowledge Catalog: a `sql-expressions` companion aspect (the field/metric expressions, both the primary GoogleSQL form and the imported vendor form) plus the schema field's `semantic` DIMENSION marker. Off by default: the published system-type templates do not carry them yet. Knowledge Catalog push only. |
 
 Destinations always deploy BigQuery-first and fail fast, so a rejected model
 never half-deploys.
@@ -137,6 +138,12 @@ statement runs in the right region; without that permission it falls back to
 BigQuery's own location inference and warns. Nothing runs under
 `--validate-only`; add `--print` to see the DDL.
 
+Push to BigQuery is **lossy**: the graph captures the queryable structure —
+node tables, edge tables, and measures — but not descriptive metadata
+(descriptions, `ai_context`, synonyms, labels), and a metric that does not
+reduce to a single MEASURE is dropped with a warning. It is a query surface,
+not a copy of your model.
+
 ### What gets created in Knowledge Catalog
 
 Each element of your model maps to one catalog resource. Every resource type
@@ -154,23 +161,37 @@ An entity entry carries its columns in the `schema` aspect (name, data type, and
 description per field); a `schema-join` link carries the relationship detail — the
 paired columns and foreign-key direction — in its aspect.
 
-> **Note — the catalog is not a full copy of your model.** By default the SQL
-> expressions are **not** written to Knowledge Catalog: the published system-type
-> templates do not yet carry a per-field `semantics` block or a
-> `semantic-metric.expression` field, so the default push omits them (pass
-> `--emit-expressions` to write them once the templates gain the fields). The
-> original vendor SQL (`importedExpression` — e.g. the MAQL or Snowflake form a
-> metric was imported from) is never written either. All of it stays in your
-> authored document and is still used when generating BigQuery SQL. Keep your
-> model document as the source of truth: a model reconstructed only from the
-> catalog would come back without its SQL.
+> **Note — push to Knowledge Catalog is lossy.** The catalog holds metadata,
+> not a full copy of your model. It **stores** names, descriptions, data
+> sources, field datatypes and roles, and 1:1 / 1:N relationships (as
+> `schema-join` links). By default it does **not** store the SQL expressions: the
+> core `schema` / `semantic-metric` aspects stay metadata-only, and the
+> executable SQL lives in a separate `sql-expressions` companion aspect the
+> published system-type templates do not carry yet, so the default push omits it.
+> Pass `--emit-expressions` (once that aspect type is provisioned) to write the
+> companion aspect — it stores **both** the canonical GoogleSQL/ANSI expression
+> and the original vendor form (`importedExpression` — e.g. the MAQL or Snowflake
+> form a metric was imported from), under an `imported` qualifier — plus the
+> schema field's DIMENSION marker. It never stores entity keys, `ai_context`,
+> field labels, the exact imported dialect label, or M:N relationships. Those stay
+> in your authored document (and, for the edges, in the BigQuery property graph).
+> Keep your model document as the source of truth.
 
 ## Validation
 
 `push` and `--validate-only` run the same checks, **before either destination is
 touched**, so a model that cannot deploy fails fast instead of half-deploying:
 
-* **Exactly one deployment target per model.** *(static)*
+* **Exactly one deployment target per model, and it must be a valid BigQuery
+  Graph URI.** A model with no target — or with more than one — is rejected, and
+  so is a single target whose URI does not match
+  `//bigquery.googleapis.com/projects/<p>/datasets/<d>/propertyGraphs/<g>` (for
+  example a `propertyGraph`/`propertyGraphs` typo, or a
+  `…/entryGroups/@bigquery/entries/…` entry form). The error names the offending
+  URI and the expected form. This gate runs before any destination leg and for
+  every `--target`, so a malformed target writes **nothing** — not to BigQuery
+  and **not to Knowledge Catalog**; the push aborts with a non-zero exit and no
+  entries are created. *(static)*
 * **Every metric on a BigQuery Graph model resolves to exactly one entity** —
   otherwise it would be dropped from the BigQuery Graph. Set the metric's attach
   entity, or scope its expression to a single entity. *(static)*
@@ -220,3 +241,113 @@ Every push prints one line per destination summarizing what it did. For a
 Deployed 1 BigQuery Graph(s).
 Wrote 5 new and 2 updated Knowledge Catalog entries; removed 1 orphaned entry; linked 2 relationships; unlinked 1 orphaned link.
 ```
+
+## Pull
+
+`kcmd pull` is the inverse of push's Knowledge Catalog leg: it reads the
+`semantic-*` entries back from the catalog and reconstructs local model
+documents at `catalog/EntryGroups/<entryGroupId>/<model>.yaml`. Use it to
+recover a workspace from a catalog someone else deployed, or to see what the
+catalog actually holds.
+
+```bash
+kcmd pull
+```
+
+Pull reads only from Knowledge Catalog (never BigQuery). Its coordinates come
+from the same scope you authored under (`<projectId>.<locationId>.<entryGroupId>`).
+
+| Flag | Effect |
+|------|--------|
+| `--dry-run` | Reconstruct from the catalog and report what would be written, but write no files. |
+| `--model <name>` | Pull a single model by name; other models in the entry group are left alone. |
+
+One entry group can hold **many models** — each `semantic-model` entry is a
+separate anchor, and pull reconstructs one document per anchor. `--model`
+narrows both the fetch and the write to a single anchor.
+
+Pull writes with the same last-write-wins policy as the core pull: a model that
+already exists locally is overwritten in place, and a local-only document (one
+with no matching catalog entry) is left untouched — pull never deletes.
+
+> **Note — pull reconstructs what the catalog holds, not your original file.**
+> Pull can only recover what push wrote (see the note under [What gets created in
+> Knowledge Catalog](#what-gets-created-in-knowledge-catalog)). What that means in
+> practice:
+>
+> **Recovered exactly** — these come back as authored:
+> - Model structure: the model, its entities, and each entity's fields.
+> - Field data source and data type.
+> - Metrics: name, data type, and attach entity.
+> - 1:1 / 1:N relationships: endpoints, foreign-key direction, and join columns
+>   (from the `schema-join` links).
+> - Deployment targets.
+>
+> **Recovered only if pushed with `--emit-expressions`** — the `sql-expressions`
+> companion aspect and the schema DIMENSION marker are omitted from the catalog
+> by default (see the note above), so pull returns them only when the push that
+> wrote them used `--emit-expressions`:
+> - Field expressions and metric expressions — **both** the canonical
+>   GoogleSQL/ANSI form and the original vendor form (`importedExpression`), the
+>   latter under the aspect's `imported` qualifier. Only the exact imported
+>   *dialect label* is not recovered.
+> - A field's dimension role, which comes back as a bare `dimension: {}` marker,
+>   without its detail (`is_time`, and so on). A default push drops the marker
+>   entirely.
+>
+> **Recovered, but normalized** — the content survives, the form changes:
+> - Relationship *names* come back lowercased/hyphenated (the catalog stores the
+>   name only in the link id, e.g. `Places Order` → `places-order`).
+> - A metric authored with no data type comes back as an explicit `Decimal`
+>   (push must write a type, and defaults it to `NUMERIC`).
+>
+> **Not recovered** — push never wrote these, so pull cannot return them:
+> - Entity keys / unique keys.
+> - `ai_context`.
+> - Field labels.
+> - The exact imported dialect label (`importedDialect`) — the vendor SQL itself
+>   comes back with `--emit-expressions`, but its dialect is not stored.
+> - M:N relationships (the edge lives only in the BigQuery property graph).
+>
+> **So: a push followed by a pull does not return your original file.** Treat a
+> pulled document as a faithful copy of the catalog metadata, not of the authored
+> model, and keep the authored document as the source of truth.
+
+> **Note — writer-side follow-up (not inherent to pull).** One reduction above is
+> a limit of what push currently *writes*, not of what pull can recover. It is
+> recorded here as a write-side follow-up; the reader (pull) already returns
+> everything the catalog holds.
+>
+> - **Relationship names.** The `schema-join` aspect type's `metadataTemplate` has
+>   no field for the relationship name, so push cannot store it and pull recovers
+>   it from the link id — which is lowercased and hyphenated (the entry-link id
+>   format forbids the original casing/underscores). Returning the name verbatim
+>   requires adding a name field to the built-in `schema-join` aspect type in
+>   Knowledge Catalog (server-side), after which the client write/read is trivial;
+>   it is the same class of gap as the `semantics` field that gates
+>   `--emit-expressions`.
+>
+> (A non-canonical deployment target is **not** a pull gap: push rejects it at the
+> validation gate before any leg runs, so it is never written — see
+> [Validation](#validation).)
+
+## Permissions
+
+`push` needs access to whichever destinations you deploy to.
+
+**BigQuery** — for `--target bq` or `all`, and for the validation pre-flight:
+
+* `bigquery.jobs.create` in the deployment-target project — to run the deploy's
+  `CREATE OR REPLACE PROPERTY GRAPH` and the validation dry-run query
+* read access on each entity's source table, so the dry-run can resolve it
+* `bigquery.datasets.get` on the target dataset (region detection; optional —
+  push degrades gracefully without it)
+
+**Knowledge Catalog / Dataplex** — for `--target kc` or `all`:
+
+* `dataplex.entryGroups.useSemanticModelAspect` on the destination entry group
+* `dataplex.entryGroups.useSchemaJoinEntryLink` and
+  `dataplex.entryGroups.useSchemaJoinAspect` when the model has relationships
+
+`kcmd pull` needs read access to the same entry group instead — to list its
+entries and fetch each `semantic-*` entry with its aspects.

toolbox/mdcode/src/libts/layouts/semantic-model.ts

@@ -91,6 +91,36 @@ export class SemanticModelLayout implements CatalogLayout {
     return docs;
   }
 
+  // True when a model document with this handle already exists on disk. `pull`
+  // uses it to report which files it would overwrite vs. create.
+  hasModel(name: string): boolean {
+    return fs.existsSync(this.modelPath(name));
+  }
+
+  // The absolute path a model document with this handle maps to:
+  // `<catalog>/EntryGroups/<entryGroup>/<name>.yaml`. Path separators in the
+  // model name are replaced so a name still yields a single flat file. Requires
+  // the layout to be scoped to an entry group (the semantic-model source always
+  // is).
+  modelPath(name: string): string {
+    if (!this._entryGroup) {
+      throw new Error(
+          'SemanticModel layout has no entry group; cannot resolve a model path.');
+    }
+    const file = `${name.replace(/[/\\]/g, '_')}.yaml`;
+    return path.join(this._catalogPath, 'EntryGroups', this._entryGroup, file);
+  }
+
+  // Writes a model's serialized document to its path, creating the EntryGroup
+  // directory if needed, and indexes it so a later modelDocuments() sees it.
+  // This is the sink `pull` writes reconstructed models to.
+  writeModelDocument(name: string, text: string): void {
+    const localPath = this.modelPath(name);
+    fs.mkdirSync(path.dirname(localPath), {recursive: true});
+    fs.writeFileSync(localPath, text);
+    this._index.set(name, localPath);
+  }
+
   // The Knowledge Catalog entry-level members are not applicable to this
   // push-only layout; the model is authored as a single Ossie document, not as
   // per-entry Knowledge Catalog files. These are wired when KC-resource emit

toolbox/mdcode/src/libts/semantic/deploy_knowledge_catalog.ts

@@ -57,10 +57,10 @@ export interface KcDeployOptions {
   systemTypeProject?: string;
   // Location the built-in system types are referenced from. Default `global`.
   systemTypeLocation?: string;
-  // Emit the SQL-expression fields not yet in the published system-type
-  // templates (per-field `schema.semantics` and `semantic-metric.expression`).
-  // Off by default so the push matches the live types; see
-  // KcGenerateOptions.emitExpressions.
+  // Emit the SQL expressions not yet in the published system-type templates: a
+  // `sql-expressions` companion aspect on entity/metric entries plus the schema
+  // field's DIMENSION marker. Off by default so the push matches the live
+  // types; see KcGenerateOptions.emitExpressions.
   emitExpressions?: boolean;
   // Compile and report only; never writes to the catalog (a dry run).
   validateOnly?: boolean;