refactor(declarative): compile configuration and reference bindings into explicit internal models

[rspurgeon]

Problem and desired outcome

Separate decoded source declarations, normalized desired resources, remote
identity bindings, and executable plan bindings. Make phase transitions
explicit so later stages do not repeatedly infer resource structure and
reference meaning from mutable structs, map fields, and placeholder strings.

This is the largest migration in the assessment and should follow the resource
capability and shared reconciliation work. Preserve the public manifest and
saved-plan formats through compatibility adapters during initial migrations.

Additional evidence after #2190

Post-merge assessment baseline: ec6261aaa. This issue remains Todo,
after the current #2079 capability and #2080 lifecycle milestones.

The #2190 compatibility audit found existing map-order-dependent diagnostics
in ResourceSet.GetResourceByRef for deliberately ambiguous cross-kind
duplicate refs. Of 1,401 external comparison cases, 1,346 deterministic cases
matched unmodified main. The remaining 55 raw ambiguous cases reproduced
nondeterminism when rerunning the baseline itself; identical audit-only lookup
ordering made all 1,401 cases match. Production lookup ordering was unchanged.

Use this as evidence for explicit reference-index ownership and ambiguity
handling in the compiled model. It is not evidence of a #2190 regression or
authorization to change which declarations are accepted or which error wins.
Inspect and preserve the compatibility contract when choosing deterministic
diagnostics. This issue's implementation has not started.

The audit and retained artifact locations are recorded in #2079's merged
organization-loading milestone.

Evidence from the current code

Assessment baseline: 0fe92b40.

• ResourceSet carries typed
  declarations, flattened children, identity state, sync scope, and deferred
  source metadata.
• Nested extraction moves many
  children into root slices while retaining flattened API documents under
  their API. Accessors in
  API planning therefore
  reconcile both storage locations.
• Loader preparation decodes
  placeholder-preserving and resolved representations to protect phase and
  disclosure semantics.
• Reference resolution
  combines relationship descriptors with field-name and resource-specific
  inference.
• Execution binding fills
  parents/references from created-resource caches and resource-specific
  lookups.
• Dependency construction
  already persists explicit dependency edges and execution groups. Preserve
  that useful scheduler boundary.

Proposed internal representations

A compiled desired configuration should have one canonical indexed resource
entry per declaration, with stable local identity, explicit parentage,
namespace/ownership metadata, and explicit collection presence. Remote IDs
belong in a separate binding table rather than changing the meaning of the
source declaration over time.

Values should distinguish literals, local references, remote selectors,
deferred environment expressions, and secret expressions. Each expression
retains the phase in which it can be resolved and any required source/scope
information.

A compiled operation should carry explicit field/path bindings to known values
or outputs of predecessor operations. It should identify the destination
request field or routing slot and requested source field. Execution consumes
these bindings instead of rediscovering their resource type from field names.

Keep structural relationships and operation dependencies distinct. Delete
ordering, explicit-ID references, API serialization requirements, and decK
post-resolution outputs cannot all be derived by reversing a resource graph.

Phase and trust boundaries to preserve

• All input files must be merged before cross-file reference validation.
• Shape validation must not disclose resolved environment or secret values.
• Ordinary environment values can participate in planning but resolve again
  from the execution environment, as documented today.
• Secret sources remain deferred; secret material must not enter the persisted
  plan, changed-field report, progress reporter, or execution result.
• File-backed secrets are bound to the saved-plan directory at execution;
  do not trust a serialized source boundary.
• External lookups, same-plan creation outputs, and name-result references
  have distinct binding behavior.
• decK can create gateway entities whose identities become available only
  after its operation. Preserve these explicit post-resolution dependencies.
• Preserve supported current saved-plan compatibility and rejection guidance;
  a new public format requires a separately reviewed compatibility decision.

Suggested migration

1. Introduce canonical identity/parent indexes behind existing ResourceSet
   accessors without changing callers or serialization.
2. Define typed internal bindings and translate existing plan references at
   the boundary.
3. Migrate a narrow reference path end to end, including serialized plans.
4. Expand to arrays, nested fields, external targets, and name-result references.
5. Move source/deferred-expression representation once the binding contract
   is established.
6. Remove obsolete inference paths only after all consumers have migrated.

Retain old test-facing functions as delegates where needed. Avoid carrying two
independent sources of semantic truth indefinitely; each step needs a stated
authority and clear conversion direction.

Open design questions

• How are source presence and provenance retained without exposing secrets?
• Which identity indexes are immutable, and where are execution results stored?
• How are parent routing and inherited protection ancestry represented?
• How do typed bindings coexist with the current JSON plan contract?
• Which compatibility wrappers can be removed only after separate permission
  to update tests or public formats?

Acceptance criteria

• Migrated resources have one canonical internal storage location.
• Resolved IDs and operation outputs have explicit ownership and lifetime.
• Migrated reference paths no longer require repeated field-name inference.
• Existing plan JSON, dependency ordering, and execution groups are preserved.
• External, deferred-environment, secret, and decK phase semantics remain.
• Obsolete inference/storage paths are removed for migrated cases.
• Applicable unchanged serialization, reference, scope, and secret tests pass.

Related work

Build on the existing relationship descriptors and payload compatibility
contracts. #2051 tracks conformance coverage, including the external-portal
routing regression fixed by #2050; it is a distinct testing follow-up.

Compatibility and validation

This is a behavior-preserving refactor. Preserve accepted YAML, omitted versus
explicit-empty scope, namespace and protection behavior, identity precedence,
external resources, error propagation, secret redaction, and saved-plan
serialization. Preserve execution ordering and request counts where existing
tests specify them. Do not silently broaden support while moving code.

Existing test packages, test files, fixtures, mocks, and golden artifacts must
remain unchanged without explicit maintainer permission. Establish the
unchanged test baseline before implementation. Keep test-facing entry points
as delegating adapters when needed. New regression or conformance tests are a
separate follow-up requiring permission; their desirability does not authorize
test changes in this refactor.

Run production-only modernization/formatting, a CGO-disabled build, lint, the
unchanged unit suite, and applicable existing integration checks after each
migration. Avoid repository-wide rewrite commands that can modify tests.
Record baseline failures separately from regressions, and verify test-file
hashes and the final Git diff.

Refactoring program

• refactor(declarative): make resource capabilities authoritative across the pipeline #2079: resource capabilities and pipeline registration.
• refactor(declarative): centralize resource lifecycle reconciliation #2080: shared lifecycle reconciliation and observation boundaries.
• refactor(declarative): unify field contracts across diffing, payload mapping, and dump #2081: field contracts for comparison, requests, and export.
• refactor(declarative): compile configuration and reference bindings into explicit internal models #2082: compiled configuration and reference bindings.

Start with bounded #2079/#2080 migrations. #2081 and #2082 remain later design
tracks; the initial implementation does not close the whole program.

Implementation guide acceptance criteria

Treat docs/contributor/declarative-resource-implementation-guide.md as the authoritative implementation entry point for agents throughout this refactoring. Guide maintenance is part of the implementation, not a documentation follow-up.

• Document the implemented internal representations and phase boundaries, including reference binding, deferred values, saved-plan compatibility, and secret/file trust boundaries.
• Update the guide in the same change as the code; replace obsolete instructions and checklists instead of appending competing patterns or historical baseline sections.
• Keep the workflow concise, retain behavioral invariants, and link to current implementations instead of copying large implementation templates.
• Verify code references, local links, and section anchors. An agent should be able to follow a resource or field through loading, planning, execution, and export without guessing missing wiring.
• Preserve maintainer constraints, including the existing test freeze; documentation or golden-regeneration instructions do not authorize test changes.

[github-actions]

🎯 Agentic Issue Triage

This is the fourth and final track (#2082) in the declarative engine refactoring program (#2079 → #2080 → #2081 → #2082). It proposes compiling decoded source declarations, normalized desired resources, remote identity bindings, and executable plan bindings into separate explicit internal models — eliminating repeated inference of resource structure and reference meaning from mutable structs across pipeline phases.

Sequencing note: This issue is explicitly marked as a "later design track" that should follow the resource capability (#2079), shared reconciliation (#2080), and field contract (#2081) work. The initial implementation of #2079/#2080 does not close this issue.

Scope and key changes

The refactor targets these current pain points (baseline 0fe92b40):

• ResourceSet carries typed declarations, flattened children, identity state, sync scope, and deferred source metadata in one mutable struct.
• Nested extraction in the loader moves children into root slices while retaining flattened API documents, forcing planners to reconcile both storage locations.
• Reference resolution combines relationship descriptors with field-name and resource-specific inference that is repeated across phases.
• Execution binding fills parents/references from created-resource caches and resource-specific lookups, rediscovering resource type from field names.

The proposal introduces:

1. A canonical indexed resource entry per declaration with stable local identity, explicit parentage, and namespace/ownership metadata.
2. Typed value expressions distinguishing literals, local references, remote selectors, deferred environment expressions, and secret expressions — each retaining the phase in which it can be resolved.
3. Compiled operation bindings with explicit field/path bindings to known values or outputs of predecessor operations, eliminating runtime field-name inference during execution.

Phase and trust boundaries to preserve

Several critical invariants must be maintained:

• All input files merged before cross-file reference validation
• Shape validation must not disclose resolved environment or secret values
• Secret material must not enter persisted plans, changed-field reports, progress reporters, or execution results
• File-backed secrets bound to saved-plan directory at execution
• External lookups, same-plan creation outputs, and name-result references have distinct binding behavior
• decK post-resolution dependencies (gateway entity identities available only after its operation)
• Existing saved-plan compatibility and rejection guidance preserved

Suggested migration path

The issue proposes a 6-step incremental migration:

1. Introduce canonical identity/parent indexes behind existing ResourceSet accessors (no caller or serialization changes)
2. Define typed internal bindings and translate existing plan references at the boundary
3. Migrate a narrow reference path end-to-end, including serialized plans
4. Expand to arrays, nested fields, external targets, and name-result references
5. Move source/deferred-expression representation once binding contract is established
6. Remove obsolete inference paths only after all consumers migrate

Each step needs a stated authority and clear conversion direction — avoid carrying two independent sources of semantic truth indefinitely.

Open design questions for team discussion

These questions should be resolved before broad implementation:

• How are source presence and provenance retained without exposing secrets?
• Which identity indexes are immutable, and where are execution results stored?
• How are parent routing and inherited protection ancestry represented?
• How do typed bindings coexist with the current JSON plan contract?
• Which compatibility wrappers can be removed only after separate permission to update tests or public formats?

Key files and areas affected

Based on the issue's code references:

Area	File	Role
Types	internal/declarative/resources/types.go	ResourceSet — current multi-purpose mutable struct
Loader	internal/declarative/loader/loader.go	Nested extraction and placeholder-preserving decode
Planner	internal/declarative/planner/api_planner.go	API planning that reconciles both storage locations
Resolver	internal/declarative/planner/resolver.go	Reference resolution with field-name inference
Executor	internal/declarative/executor/executor.go	Execution binding from created-resource caches
Dependencies	internal/declarative/planner/dependencies.go	Dependency edges and execution groups (preserve)

Related issues and dependencies

Refactoring program (ordered):

• refactor(declarative): make resource capabilities authoritative across the pipeline #2079 — Resource capabilities and pipeline registration (start here)
• refactor(declarative): centralize resource lifecycle reconciliation #2080 — Shared lifecycle reconciliation and observation boundaries (start here)
• refactor(declarative): unify field contracts across diffing, payload mapping, and dump #2081 — Field contracts for comparison, requests, and export (later track)
• refactor(declarative): compile configuration and reference bindings into explicit internal models #2082 — Compiled configuration and reference bindings (this issue, later track)

Related work:

• Task(declarative): add payload-contract conformance coverage #2051 — Payload-contract conformance coverage (separate testing follow-up)
• fix(declarative): resolve external portal publication refs #2050 — External portal routing regression fix (referenced as prior fix)

Validation constraints

This is a behavior-preserving refactor with strict constraints:

• Existing test packages, test files, fixtures, mocks, and golden artifacts must remain unchanged without explicit maintainer permission
• Preserve accepted YAML, scope semantics, identity precedence, error propagation, secret redaction, and saved-plan serialization
• Preserve execution ordering and request counts where existing tests specify them
• Run production-only modernization/formatting, CGO-disabled build, lint, and unchanged unit suite after each migration
• Verify test-file hashes and final Git diff

Generated by Opened Issue Triage for #2082 · opus46 · 62.1 AIC · ⌖ 9.06 AIC · ⊞ 3.8K · ◷

_{Add this agentic workflow to your repo}

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/issue-triage.md@69b5e3ae5fa7f35fa555b0a22aee14c36ab57ebb