refactor(declarative): unify field contracts across diffing, payload mapping, and dump

[rspurgeon]

Problem and desired outcome

A declarative field can be accepted by the load schema while still requiring
independent create extraction, comparison, update mapping, and dump conversion.
Define its behavior once, with action-specific projections shared across these
paths, to reduce drift when the Konnect SDK and resource surface evolve.

This is a design proposal following the resource capability and shared
reconciliation milestones. The migration must retain existing behavior before
support for additional SDK fields is considered.

Evidence from the current code

Assessment baseline: 0fe92b40.

• fix(declarative): support portal sipr_enabled #2014 added sipr_enabled support through separate edits to the resource,
  planner,
  executor,
  and dump conversion.
• Payload validation
  detects fields dropped between a plan and its SDK request. It cannot detect
  a manifest field that the planner omitted before constructing that plan.
• AI Gateway comparisons
  maintain normalization, collection ordering, write-only filtering, and
  literal default rules separately from schema/default derivation.
• DCR comparison
  explicitly handles fields omitted from API responses and normalized issuers.
• Dump children
  contain substantial independent response-to-manifest conversion.

Proposed responsibilities

Derive mechanical field structure from existing typed declarations and SDK
request/response types where trustworthy. Supply explicit reviewed policies
for behavior that type reflection cannot determine:

• accepted declarative field paths and presence;
• create eligibility, update eligibility, and immutable fields;
• sparse update versus full replacement request requirements;
• observable response fields and unknown/unreturned values;
• comparison normalization and ordered versus unordered collections;
• literal documented defaults, with narrowly scoped overrides;
• write-only secrets and public vault references;
• routing/relationship fields versus API body fields;
• exportable fields and dump omission behavior.

The same resource-owned contract should provide separate create, update,
comparison, and export projections. These are different views of shared field
knowledge, not a single request struct reused everywhere.

Preserve the existing payload-loss checks as defense in depth. Keep custom
transformations explicit and reviewable; do not infer business semantics from
field names or permit arbitrary opaque maps to bypass validation.

Suggested migration

1. Inventory the existing projection paths for one ordinary resource.
2. Extract its current field policy without broadening field support.
3. Route create, compare, update, and dump through those projections.
4. Verify nil/empty and changed-field rendering behavior with existing tests.
5. Exercise a resource with asymmetric create/update or response shapes.
6. Expand through resource families after the exceptions are understood.

Any code generation should derive mechanical accessors or metadata. Assess it
against the existing reflection machinery before introducing a new generator
or a second manually maintained schema language.

Questions to resolve before broad implementation

• Which schema facts should come from the SDK Go types versus the authoritative
  OpenAPI schema work in Task: embed schema data for runtime usage #1599?
• How should absent, null, explicit empty, and defaulted fields be represented?
• Where do response observability and comparison-only normalization belong?
• Can ordinary sparse updates be derived while keeping full-body APIs explicit?
• How do union discriminators select action-specific field contracts?
• How is a future SDK field addition surfaced for review without silently
  changing the accepted declarative contract?

Acceptance criteria

• A migrated ordinary field has one authoritative behavioral declaration.
• Create, update, comparison, and export consume the shared contract.
• Existing accepted fields, omission semantics, and request bodies remain.
• Write-only values remain absent from plans, diffs, dumps, and diagnostics.
• SDK-derived defaults do not introduce hidden cross-field defaults.
• Unsupported mappings still fail explicitly before execution.
• Duplicated migrated field lists are removed.
• Applicable unchanged unit, integration, and dump tests pass.

Related work

Coordinate with #1599 and #1827 for schema authority and derivation. This issue
covers lifecycle projections beyond explain/load shape. #2051 covers
conformance tests and is not authorization to modify tests in this refactor.
#1039 addresses a specific API publication transport path.

Compatibility and validation

This is a behavior-preserving refactor. Preserve accepted YAML, omitted versus
explicit-empty scope, namespace and protection behavior, identity precedence,
external resources, error propagation, secret redaction, and saved-plan
serialization. Preserve execution ordering and request counts where existing
tests specify them. Do not silently broaden support while moving code.

Existing test packages, test files, fixtures, mocks, and golden artifacts must
remain unchanged without explicit maintainer permission. Establish the
unchanged test baseline before implementation. Keep test-facing entry points
as delegating adapters when needed. New regression or conformance tests are a
separate follow-up requiring permission; their desirability does not authorize
test changes in this refactor.

Run production-only modernization/formatting, a CGO-disabled build, lint, the
unchanged unit suite, and applicable existing integration checks after each
migration. Avoid repository-wide rewrite commands that can modify tests.
Record baseline failures separately from regressions, and verify test-file
hashes and the final Git diff.

Refactoring program

• refactor(declarative): make resource capabilities authoritative across the pipeline #2079: resource capabilities and pipeline registration.
• refactor(declarative): centralize resource lifecycle reconciliation #2080: shared lifecycle reconciliation and observation boundaries.
• refactor(declarative): unify field contracts across diffing, payload mapping, and dump #2081: field contracts for comparison, requests, and export.
• refactor(declarative): compile configuration and reference bindings into explicit internal models #2082: compiled configuration and reference bindings.

Start with bounded #2079/#2080 migrations. #2081 and #2082 remain later design
tracks; the initial implementation does not close the whole program.

Implementation guide acceptance criteria

Treat docs/contributor/declarative-resource-implementation-guide.md as the authoritative implementation entry point for agents throughout this refactoring. Guide maintenance is part of the implementation, not a documentation follow-up.

• Document how a field is carried through accepted YAML, comparison, action-specific SDK mapping, and dump, including omission/default and write-only-field contracts.
• Update the guide in the same change as the code; replace obsolete instructions and checklists instead of appending competing patterns or historical baseline sections.
• Keep the workflow concise, retain behavioral invariants, and link to current implementations instead of copying large implementation templates.
• Verify code references, local links, and section anchors. An agent should be able to follow a resource or field through loading, planning, execution, and export without guessing missing wiring.
• Preserve maintainer constraints, including the existing test freeze; documentation or golden-regeneration instructions do not authorize test changes.

[github-actions]

🎯 Agentic Issue Triage

Summary: Design proposal to unify per-resource field contracts so that create eligibility, update mapping, comparison normalization, and dump/export conversion are derived from a single authoritative declaration per field, rather than maintained independently in the planner, executor, and dump paths. Part 3 of 4 in the declarative refactoring program (#2079 → #2080 → #2081 → #2082), explicitly sequenced as a later design track after capabilities and lifecycle reconciliation land.

Context and relationship to prior work

• Predecessor (completed): [refactor] standardize declarative planner-to-executor API payload contracts #1905 standardized planner-to-executor payload contracts and was merged via refactor(declarative): standardize planner payload contracts #1924. That work established the PlannedChange.Fields / Parent / References / SecretWrites separation. This issue goes further by making the field-level behavioral declarations (create-eligible, update-eligible, immutable, write-only, comparison normalization, export omission) authoritative from a single source.
• Sibling issues: refactor(declarative): make resource capabilities authoritative across the pipeline #2079 (resource capabilities/pipeline registration) and refactor(declarative): centralize resource lifecycle reconciliation #2080 (shared lifecycle reconciliation) are the immediate priorities. refactor(declarative): compile configuration and reference bindings into explicit internal models #2082 (compiled configuration and reference bindings) is the final track.
• Related open issues: Task: derive custom Explain schemas from SDK types and detect drift #1827 (derive Explain schemas from SDK types) addresses schema authority for explain/scaffold but not lifecycle projections — coordinate on shared type-reflection machinery. Task(declarative): add payload-contract conformance coverage #2051 (payload-contract conformance tests) is separately authorized testing work.
• Closed related: Task: embed schema data for runtime usage #1599 (embed OpenAPI schema data) was closed as not-planned but its open questions about SDK-type vs OpenAPI authority are carried forward in this issue's design questions section.

Key design decisions to resolve

The issue identifies six open questions that should be answered before broad implementation. These are the critical path items:

1. Schema source authority: SDK Go types vs. OpenAPI spec (Task: embed schema data for runtime usage #1599 direction) — this determines whether field metadata is reflection-derived or generated from spec.
2. Absent/null/empty/default representation: Directly impacts comparison normalization and sparse-update eligibility. The current codebase handles this differently per resource (e.g., DCR provider's normalized issuers vs. AI Gateway's literal default rules).
3. Response observability boundaries: Where do comparison-only normalization rules live relative to the field contract? Currently scattered across *_payload_compare.go files.
4. Sparse vs. full-body update derivation: Can the contract declare PATCH-eligible resources and derive sparse updates, while keeping full-body PUT resources explicit?
5. Union discriminator handling: How do polymorphic resources (AI Gateway model providers, Event Gateway policies) select field contracts per variant?
6. Future SDK field surfacing: How to detect new SDK fields without silently accepting them into the declarative contract — the review gate.

Suggested approach for initial implementation

Based on the migration steps in the issue and the codebase structure:

1. Start with Portal as the inventory candidate — the sipr_enabled addition (fix(declarative): support portal sipr_enabled #2014) is cited as evidence, and portal has a moderate field surface across planner, executor, and dump with good existing test coverage.
2. Extract field policy as data, not code — define a struct/registry entry per field with create/update/compare/export eligibility flags, normalization rules, and default policy. Keep it co-located with the resource definition.
3. Route through projections incrementally — wire create-extraction first (smallest blast radius), then comparison, then update mapping, then dump conversion. Verify each with unchanged existing tests before proceeding.
4. Exercise an asymmetric resource second — pick a resource where create and update shapes differ (e.g., a resource with immutable fields or PATCH semantics) to validate the contract handles action-specific projections.
5. Preserve payload-loss checks — the existing payload_contract.go validation should remain as defense-in-depth, catching any field the contract accidentally omits.

Key source files to inventory

Based on the evidence cited in the issue (baseline 0fe92b40):

• internal/declarative/planner/portal_planner.go — portal field comparison
• internal/declarative/executor/portal_adapter.go — portal create/update mapping
• internal/cmd/root/verbs/dump/declarative.go — dump conversion entry point
• internal/cmd/root/verbs/dump/declarative_children.go — child resource dump conversion
• internal/declarative/planner/ai_gateway_payload_compare.go — AI Gateway normalization/ordering/defaults
• internal/declarative/planner/dcr_provider_planner.go — DCR omitted-field and issuer normalization
• internal/declarative/executor/payload_contract.go — existing payload-loss defense checks

Risk areas and constraints

• Test immutability: The issue explicitly requires that existing test packages, fixtures, mocks, and golden artifacts remain unchanged without maintainer permission. This is a hard constraint — the refactor must preserve all test-facing entry points as delegating adapters.
• No silent broadening: Moving field inventories must not accidentally accept new fields that the current code doesn't accept. The field contract should be extracted from what exists, not expanded.
• Write-only secret handling: Secret fields must remain absent from plans, diffs, dumps, and diagnostics. The field contract needs a clear write-only disposition that integrates with the existing SecretWrites mechanism.
• Ordering sensitivity: Comparison normalization currently handles ordered vs. unordered collections per resource. The unified contract must preserve these semantics.

Labels applied: declarative, refactoring, code-quality, medium-priority, triaged

Priority rationale: Medium priority — this is an important architectural improvement but is explicitly sequenced after #2079 and #2080 in the refactoring program. The issue itself states "remain later design tracks; the initial implementation does not close the whole program."

Generated by Opened Issue Triage for #2081 · opus46 · 59.3 AIC · ⌖ 9.08 AIC · ⊞ 3.8K · ◷

_{Add this agentic workflow to your repo}

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/issue-triage.md@69b5e3ae5fa7f35fa555b0a22aee14c36ab57ebb