Define one semantic model per document without a wrapper

[flyrain]

Summary

Change the core JSON/YAML specification so each standalone document contains exactly one semantic model, with model properties directly at the root. Remove both the semantic_model array and its wrapper. For example:

version: 0.2.0.dev0
name: sales_analytics
datasets:
  - name: orders
    source: sales.public.orders

A model still supports multiple datasets, relationships, and metrics. Bulk exchange uses separate documents; this PR does not introduce a bundle format.

Breaking change to the unreleased, mutable 0.2.0.dev0 schema: wrapped models (arrays or objects) are rejected. Migration moves a single model's properties to the root or splits multiple models into separate documents, preserving model contents and using the current specification version. Root-level dialects and vendors declarations remain removed, as established by #397.

Changes

• Update the core JSON Schema, Markdown specification, YAML reference, and documentation overview.
• Flatten the canonical TPC-DS example without changing model contents.
• Update validator traversal and add focused checks for the new shape and rejection of legacy/malformed input.
• Preserve $defs/SemanticModel and the ontology map's embedded semantic_model property.

Scope and merge coordination: The Python SDK and converter migration is implemented in PR #396, which also updates converter CI to run on shared schema/example changes. Review the spec and migration separately, but do not land this schema change until #396 is approved and its combined spec/consumer CI is green. Coordinate the two merges so the repository is not left with the new schema and old consumers. This PR alone does not migrate those consumers.

Applicable checklist

• Specification and reference examples updated in core-spec/.
• Breaking change and migration documented.
• Validator behavior and focused regression coverage updated.
• Ontology embedded model definition preserved.
• No dependencies added.
• Dev mailing list discussion (pending).
• Python SDK and converter adoption: #396, required for coordinated landing.

[djwaldo]

The shape change is fine by me. I.e. a document-boundary change, not a model-semantics one. I have
a bidirectional converter in flight (#364, ThoughtSpot); adopting it is small.

One case worth settling first, because it is where "one model per document" meets real vendor
topologies. Tested both directions rather than assumed.

A chasm trap: sales and inventory both joining a shared product, no fact-to-fact join.

Ossie → vendor is unaffected. ThoughtSpot resolves multi-fact chasm traps in its query engine,
so this stays one vendor model and cross-fact questions keep working. Wrapper or root makes no
difference.

Vendor → Ossie is the problem. A modeller may legitimately have split those into two
ThoughtSpot models. I.e. permissions, natural-language scope, grain separation. Converting back, I
get two Ossie documents, each with its own copy of product, and nothing records that
those are the same dataset or that the two models are related. The chasm trap is gone.

This PR does not create that, but it makes it permanent: one model per document and no bundle
means there is nowhere to say "these documents are one estate" or "this dataset is that
dataset."

Which is intended?

1. Documents are standalone, duplication is expected, dataset identity across documents is out
   of scope. This is fine, but worth stating so converter authors stop trying to preserve it.
2. It belongs to the deferred bundle work. I.e. in which case cross-document dataset identity is a
   requirement for that design, not just packaging.

Either is defensible. It decides whether a converter should silently duplicate the shared
dataset or report a declared loss, and right now I would be guessing.

[flyrain]

cc @jbonofre @khush-bhatia

[Copilot]

🔵 Needs a closer look

The breaking schema change conflicts with existing consumers, while SDK and converter migration remains deferred.

Pull request overview

This PR changes standalone documents to contain one root-level semantic model, updating the schema, validator, documentation, examples, and tests.

Changes:

• Removes the standalone semantic_model wrapper.
• Updates validation and regression coverage.
• Flattens specifications and the TPC-DS example while preserving embedded models.

A critical compatibility concern remains because existing SDK, converter, and consumer flows still expect the wrapped format.

File summaries

File	Description
validation/validate.py	Traverses and validates root-level models.
validation/tests/test_validate.py	Tests new document shapes and traversal.
validation/test_validate.py	Adds CLI regression coverage.
examples/tpcds_semantic_model.yaml	Flattens the canonical example.
docs/index.md	Updates the overview and migration guidance.
core-spec/spec.yaml	Updates the YAML reference structure.
core-spec/spec.md	Documents the schema and migration guidance.
core-spec/ossie-schema.json	Defines the flat standalone document shape.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[khush-bhatia]

@flyrain Could you fix this please ?
In https://github.com/apache/ossie/blob/main/converters/README.md, it says "Parse the Ossie model: Load the YAML file and iterate over the top-level semantic_model entries."

[khush-bhatia]

Could you create an issue to fix this in a follow up ?

[flyrain]

This guard prevents semantic checks from crashing on structurally invalid input. We already have #270 tracking that issue and #271 addressing it. Does that cover the follow-up you had in mind, or were you referring to something else in this condition?

[flyrain]

This PR does not create that, but it makes it permanent: one model per document and no bundle means there is nowhere to say "these documents are one estate" or "this dataset is that dataset."

@djwaldo, thanks for raising this. The existing array groups models in one file, but it doesn’t define shared dataset identity across those models. Two copies of product aren’t currently declared to be the same dataset just because they appear in the same document. This PR changes how the models are packaged while leaving that limitation unchanged.

Which is intended?

1. Documents are standalone, duplication is expected, dataset identity across documents is out
   of scope. This is fine, but worth stating so converter authors stop trying to preserve it.
2. It belongs to the deferred bundle work. I.e. in which case cross-document dataset identity is a
   requirement for that design, not just packaging.

I see your two options as compatible: cross-model dataset identity is outside the current core spec, and we can add support for it separately. One model per doc still leaves room for shared identifiers, references, or a manifest in the future. If we introduce a shared identity would need an explicit design beyond simply grouping files.

[flyrain]

@khush-bhatia thanks a lot for the quick review. Resolved all comments. Pls take another look.

[khush-bhatia]

@flyrain Could you please rebase to the latest and resolve conflicts.. We did remove dialects and vendors array from root for good.
Relevant PR
#397

[khush-bhatia]

vendors and dialects should be removed.

[khush-bhatia]

Please remove dialects, and vendors from here.

[khush-bhatia]

Instead we should update the ontology spec to point to the root of this spec(perhaps "https://github.com/apache/ossie/core-spec/ossie-schema.json") ?

Can be done in a follow up PR though.

[flyrain]

Agreed, let’s handle that in a follow-up PR. Referencing the root schema would also require version on each embedded semantic model, so we should update the ontology examples, converters, and tests together. I’ll leave the current reference in place for this PR.

[flyrain]

@khush-bhatia , fixed as suggested. PTAL.

[djwaldo]

Thanks for clarification.

[kayemkim]

Merging this branch (6470653) with current main (fc6c9df): #375 landed yesterday, after the last push here, and its validate_relationship_column_arity still iterates data.get("semantic_model", []). For a flat document that loop is empty, so the arity check never runs. A relationship with from_columns: [customer_id, region_id] and to_columns: [id] prints Validation PASSED on the merged tree, where the wrapper form fails on main with [Arity]. The two test_arity_rejects_mismatched_length_columns cases fail the same way, so Validation CI would go red on main after merge. There is no textual conflict and the PR's runs predate #375, which is why neither GitHub nor CI shows it.

Same fix this PR applies to the other three validators: drop the for model in loop and read relationships from the root. The other 37 pytest cases and the two remaining CI steps pass on the merged tree.

[flyrain]

Thanks @kayemkim , fixed it in the new commit. I'd suggest to merge this to avoid further rebase.