TDL T4 - Derived Source Tooling

[evomimic]

Derived Source Tooling

1. Summary (Required)

What is the enhancement?
Deliver T4 — Derived Source Tooling: enable code generation, semantic diffing, CI support, editor services, and contributor workflows to consume the stable LoaderRefRep boundary and immutable derived indexes.

Derived tooling must remain a consumer of source-toolchain output. It must not introduce a third semantic authority, own runtime descriptor validation, materialize runtime defaults, or bypass Holon Loading.

---

2. Problem Statement (Required)

Why is this needed?
T1–T3 establish TDL and MAP JSON as source syntaxes that converge on LoaderRefRep, with transitional source representations retired. The resulting source-toolchain boundary is useful beyond parsing and rendering:

• code generators need deterministic schema facts;
• reviewers and CI need semantic diffs and fidelity checks that ignore formatting-only changes;
• editor services need source diagnostics and read-only schema navigation;
• contributors need consistent, supported commands for checking, rendering, comparing, and regenerating source artifacts.

Without an explicit derived-tooling boundary, each consumer risks reconstructing schema semantics from source text, duplicating loader-graph state, or incorrectly embedding descriptor validation in host-side tooling.

---

3. Dependencies (Required)

Does this depend on other issues or features?

• T1 — Preserve Schema 2.0 Source Fidelity, delivered.
• T2 — Canonical Rendering and Round-Trip Fidelity, delivered.
• T3 — Retire Transitional Source Representations, delivered.
• The stable LoaderRefRep representation and its immutable derived-index/projection facilities.
• Existing source-toolchain substrate:
  • tools/map-schema
  • schema-src/
  • generated/json-imports/
  • host/crates/holons_loader_client
• Runtime descriptor semantics remain owned by HolonDescriptor, Holons Core, Holon Loading, and Descriptor-Aware Holon Validation.

---

4. Proposed Solution (Required)

How would you solve it?

Establish a documented, read-only derived-tooling surface over LoaderRefRep and immutable indexes derived from it.

The surface must support these consumer classes:

1. Code generation

   • Deterministically enumerate loader-graph facts needed by generators.
   • Produce generated output from explicit loader content and immutable derived indexes only.
   • Keep generator-specific output models outside the source-toolchain semantic boundary.

2. Diffing and fidelity

   • Compare two normalized LoaderRefRep projections.
   • Report additions, removals, and changes to explicit holons, properties, keyed relationships, ordering, literals, schema dependencies, and omissions.
   • Ignore formatting, JSON field order, and lossless TDL shorthand differences.

3. CI support

   • Provide repository-local commands that validate the Schema 2.0 corpus, generated-artifact freshness, and source-conversion fidelity.
   • Make the relevant checks available to GitHub Actions without requiring a separate semantic-runtime path.

4. Editor services

   • Expose bounded source provenance and syntax/lowering diagnostics where available.
   • Support read-only navigation and inspection based on source-toolchain facts.
   • Do not require full incremental descriptor-aware semantic validation for syntactically incomplete documents.

5. Contributor workflows

   • Document supported check, compile, decompile, round-trip, diff, and generation commands.
   • Define which outputs are checked in, generated temporarily, or intentionally excluded from version control.

Implementation must reuse the existing LoaderRefRep source boundary or immutable projections already derived from it. It must not create a mutable semantic IR, duplicate descriptor-kernel behavior, or add a parallel loader ingress path.

---

5. Scope and Impact (Required)

What does this impact?

In scope:

• tools/map-schema and any appropriately named native source-tooling crate/module.
• Deterministic immutable projections/indexes derived from LoaderRefRep.
• Source-to-source semantic diffing.
• Code-generation integration points.
• CI and contributor commands.
• Bounded editor-facing diagnostics and provenance.
• Documentation in README.md and tools/README.md.
• Relevant GitHub Actions workflow steps.

Out of scope:

• Runtime descriptor validation or DS-* rule execution.
• Default population, reference resolution, staged-holon construction, transaction behavior, or commit behavior.
• A new loader JSON format.
• A parallel host, hApp, or shared-crate runtime API.
• A mutable semantic model that competes with LoaderRefRep or Holons Core.
• Full incremental semantic validation of invalid or incomplete editor documents.

---

6. Testing Considerations (Required)

How will this enhancement be tested?

• Add focused unit tests for immutable index/projection construction and deterministic ordering.
• Add diff fixtures proving that source-format-only changes do not produce semantic changes.
• Add fixtures proving that explicit changes to properties, relationships, ordering, literals, dependencies, and omissions are reported.
• Add code-generation snapshot or golden-output tests where a generator is introduced.
• Add CI coverage for corpus validation, generated-artifact freshness, and round-trip fidelity.
• Add provenance/diagnostic tests for editor-facing source locations where source spans are available.
• Verify no derived-tooling operation invokes runtime descriptor validation, materializes defaults, or requires Holon Loading.
• Run the relevant map-schema tests and repository CI checks.

---

7. Definition of Done (Required)

When is this enhancement complete?

• Derived tooling consumes LoaderRefRep or immutable indexes derived from it; it does not maintain a mutable duplicate of holonic semantic state.
• A deterministic read-only tooling surface supports code-generation and inspection consumers.
• A semantic diff facility compares normalized loader-graph content and excludes formatting-only differences.
• CI can validate source syntax/lowering, generated-artifact freshness, and round-trip fidelity through repository-local commands.
• Contributor documentation describes supported source-tooling workflows and artifact ownership.
• Editor-facing tooling can return bounded syntax/lowering diagnostics and available source provenance without taking ownership of descriptor semantics.
• Runtime descriptor validation, default population, loader lifecycle behavior, and commit behavior remain unchanged and outside this implementation.
• No new third semantic authority, mutable semantic IR, or parallel loader ingress path is introduced.
• Relevant unit, fixture, and CI checks pass.

Optional Details (Expand if needed)

8. Alternatives Considered

What other solutions did you think about?

• Let each generator, editor integration, and CI job parse TDL or JSON independently. Rejected because it recreates source representations and invites semantic drift.
• Expose descriptor-runtime products directly to source tooling. Rejected because source conversion and runtime descriptor semantics have separate ownership and lifecycle boundaries.
• Treat TDL text as the semantic-diff source of truth. Rejected because fidelity is defined over normalized LoaderRefRep, not formatting or declaration shorthand.
• Combine source tooling with Holon Loading. Rejected because loading begins after the source-toolchain boundary.

---

9. Risks or Concerns

What could go wrong?

• A convenience index can become a de facto mutable semantic model unless it is explicitly immutable, derived, and bounded.
• Editor requirements can expand into runtime validation or incomplete-document semantic analysis; those capabilities are explicitly deferred.
• Code generators may infer descriptor semantics from names or declaration spelling; generated output must consume explicit loader facts and approved immutable indexes only.
• Diff output can become noisy unless ordering, omission, and formatting rules are defined consistently with existing fidelity comparison.
• This T4 unit contains multiple consumer categories. If implementation cannot preserve one coherent shared tooling boundary, split the work into the smallest dependency-ordered issues: immutable tooling surface, diff/CI workflow, code generation, then editor services.

---

10. Additional Context

Any supporting material?

Authoritative sources:

• map-dev-docs/docs/core/type-system/tdl/tdl-impl-plan-v2.md, T4 — Derived Source Tooling.
• map-dev-docs/docs/core/type-system/tdl/tdl-spec.md, especially the LoaderRefRep source boundary, source provenance, and semantic-diff/fidelity rules.
• docs/roadmap/desc-driven-impl-plan.md.
• .github/ISSUE_TEMPLATE/enhancement.md.

Core invariant:

Derived tooling may consume LoaderRefRep and immutable derived indexes, but must not introduce a third semantic authority or take ownership of runtime descriptor validation.