Skip to content
Merged
2 changes: 1 addition & 1 deletion converters/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ A converter should map `ai_context` when the target vendor supports equivalent c

1. **Validate input**: Use the [Ossie JSON Schema](../core-spec/ossie-schema.json) and the [validation script](../validation/validate.py) to ensure the source Ossie model is valid before conversion.

2. **Parse the Ossie model**: Load the YAML file and iterate over the top-level `semantic_model` entries.
2. **Parse the Ossie model**: Load the JSON or YAML document as one model.

3. **Map datasets**: For each dataset, translate the `name`, `source`, `primary_key`, `unique_keys`, and `fields` to the vendor's format. Parse the `source` string (typically `database.schema.table`) into the vendor's catalog structure.

Expand Down
30 changes: 22 additions & 8 deletions core-spec/ossie-schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,37 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://github.com/apache/ossie/core-spec/ossie-schema.json",
"title": "Apache Ossie Core Metadata Specification",
"description": "JSON Schema for validating Apache Ossie semantic model definitions",
"description": "JSON Schema for validating a single Apache Ossie semantic model document",
"type": "object",
"properties": {
"version": {
"type": "string",
"const": "0.2.0.dev0",
"description": "Apache Ossie specification version"
},
"semantic_model": {
"type": "array",
"description": "Collection of semantic model definitions",
"items": {
"$ref": "#/$defs/SemanticModel"
}
"name": {
"$ref": "#/$defs/SemanticModel/properties/name"
},
"description": {
"$ref": "#/$defs/SemanticModel/properties/description"
},
"ai_context": {
"$ref": "#/$defs/SemanticModel/properties/ai_context"
},
"datasets": {
"$ref": "#/$defs/SemanticModel/properties/datasets"
},
"relationships": {
"$ref": "#/$defs/SemanticModel/properties/relationships"
},
"metrics": {
"$ref": "#/$defs/SemanticModel/properties/metrics"
},
"custom_extensions": {
"$ref": "#/$defs/SemanticModel/properties/custom_extensions"
}
},
"required": ["version", "semantic_model"],
"required": ["version", "name", "datasets"],
"additionalProperties": false,
Comment thread
flyrain marked this conversation as resolved.
"$defs": {
"Dialect": {
Expand Down
242 changes: 140 additions & 102 deletions core-spec/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,12 +84,17 @@ ontology specification's built-in value types; `Time`, `DateTimeTz`, and

## Semantic Model

The top-level container that represents a complete semantic model, including datasets, relationships, and metrics.
Each JSON or YAML document represents exactly one semantic model.

A standalone document must contain `version`, `name`, and a non-empty `datasets`
array. For bulk exchange, use separate model documents. This specification does
not define a bundle format or cross-model references.

### Schema

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `version` | string | Yes | Apache Ossie specification version (`0.2.0.dev0`) |
| `name` | string | Yes | Unique identifier for the semantic model |
| `description` | string | No | Human-readable description |
| `ai_context` | string/object | No | Additional context for AI tools (e.g., custom instructions) |
Expand All @@ -101,21 +106,54 @@ The top-level container that represents a complete semantic model, including dat
### Example

```yaml
semantic_model:
- name: sales_analytics
description: Sales and customer analytics model
ai_context:
instructions: "Use this model for sales analysis and customer insights"
datasets:
- name: orders
source: sales.public.orders
relationships: []
metrics: []
custom_extensions:
- vendor_name: DBT
data: '{"project_name": "tpcds_analytics", "models_path": "models/semantic"}'
version: 0.2.0.dev0
name: sales_analytics
description: Sales and customer analytics model
ai_context:
instructions: "Use this model for sales analysis and customer insights"
datasets:
- name: orders
source: sales.public.orders
relationships: []
metrics: []
custom_extensions:
- vendor_name: DBT
data: '{"project_name": "tpcds_analytics", "models_path": "models/semantic"}'
```

The same document structure in JSON:

```json
{
"version": "0.2.0.dev0",
"name": "sales_analytics",
"datasets": [
{"name": "orders", "source": "sales.public.orders"}
]
}
```

### Migrating earlier document shapes

This is a breaking change in the unreleased `0.2.0.dev0` specification. Earlier
releases and earlier development snapshots use a `semantic_model` array. The
current schema accepts only the flat document shape; it does not accept the array
or an object-valued wrapper.

To migrate a document containing one model, move that model's properties to the
root and remove `semantic_model`. Use `version: 0.2.0.dev0` for the migrated
document. Remove any root-level `dialects` and `vendors` declarations; preserve
per-expression dialects and vendor information in `custom_extensions`. For
multiple models, create one document per model and validate each result. An empty
model array cannot produce a valid model document. Preserve model contents and
custom extensions; never silently select only the first model or overwrite a file
when splitting a document.

The reusable `$defs/SemanticModel` schema still describes model contents without
standalone document metadata. In particular, an ontology map continues to embed
those contents under its `semantic_model` property. This standalone document
change does not rename or flatten that ontology property.
Comment on lines +152 to +155

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead we should update the ontology spec to point to the root of this spec(perhaps "https://github.com/apache/ossie/core-spec/ossie-schema.json") ?

Can be done in a follow up PR though.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, let’s handle that in a follow-up PR. Referencing the root schema would also require version on each embedded semantic model, so we should update the ontology examples, converters, and tests together. I’ll leave the current reference in place for this PR.


---

## Datasets
Expand Down Expand Up @@ -507,101 +545,100 @@ Here's a complete semantic model example showing all components working together

```yaml
version: 0.2.0.dev0
semantic_model:
- name: ecommerce_analytics
description: E-commerce sales and customer analytics
ai_context:
instructions: "Use this model for analyzing sales trends, customer behavior, and product performance"

datasets:
- name: orders
source: sales.public.orders
primary_key: [order_id]
description: Customer orders
fields:
- name: order_id
expression:
dialects:
- dialect: ANSI_SQL
expression: order_id
description: Order identifier

- name: customer_id
expression:
dialects:
- dialect: ANSI_SQL
expression: customer_id
description: Customer identifier

- name: order_date
expression:
dialects:
- dialect: ANSI_SQL
expression: order_date
datatype: Date
dimension:
is_time: true
description: Order date

- name: amount
expression:
dialects:
- dialect: ANSI_SQL
expression: amount
description: Order amount

- name: customers
source: sales.public.customers
primary_key: [id]
description: Customer information
fields:
- name: id
expression:
dialects:
- dialect: ANSI_SQL
expression: id
description: Customer identifier

- name: email
expression:
dialects:
- dialect: ANSI_SQL
expression: email
description: Customer email

relationships:
- name: orders_to_customers
from: orders
to: customers
from_columns: [customer_id]
to_columns: [id]

metrics:
- name: total_revenue
name: ecommerce_analytics
description: E-commerce sales and customer analytics
ai_context:
instructions: "Use this model for analyzing sales trends, customer behavior, and product performance"

datasets:
- name: orders
source: sales.public.orders
primary_key: [order_id]
description: Customer orders
fields:
- name: order_id
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(orders.amount)
description: Total revenue from all orders
ai_context:
synonyms:
- "total sales"
- "revenue"

- name: customer_count
expression: order_id
description: Order identifier

- name: customer_id
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT customers.id)
description: Total number of customers
ai_context:
synonyms:
- "total customers"
- "customer base"
expression: customer_id
description: Customer identifier

custom_extensions:
- vendor_name: SNOWFLAKE
data: '{"warehouse": "ANALYTICS_WH"}'
- name: order_date
expression:
dialects:
- dialect: ANSI_SQL
expression: order_date
datatype: Date
dimension:
is_time: true
description: Order date

- name: amount
expression:
dialects:
- dialect: ANSI_SQL
expression: amount
description: Order amount

- name: customers
source: sales.public.customers
primary_key: [id]
description: Customer information
fields:
- name: id
expression:
dialects:
- dialect: ANSI_SQL
expression: id
description: Customer identifier

- name: email
expression:
dialects:
- dialect: ANSI_SQL
expression: email
description: Customer email

relationships:
- name: orders_to_customers
from: orders
to: customers
from_columns: [customer_id]
to_columns: [id]

metrics:
- name: total_revenue
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(orders.amount)
description: Total revenue from all orders
ai_context:
synonyms:
- "total sales"
- "revenue"

- name: customer_count
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT customers.id)
description: Total number of customers
ai_context:
synonyms:
- "total customers"
- "customer base"

custom_extensions:
- vendor_name: SNOWFLAKE
data: '{"warehouse": "ANALYTICS_WH"}'
```

---
Expand Down Expand Up @@ -643,6 +680,7 @@ ai_context:
## Version History

- **0.2.0.dev0** (Unreleased): In-development next minor release. Schema is mutable; do not depend on this version in production.
- Breaking: each standalone document contains one model directly at the root; the `semantic_model` array is removed.
- **0.1.1** (2025-12-11): Initial release
- Core semantic model structure
- Support for datasets, relationships, fields, and metrics
Expand Down
Loading
Loading