Skip to content

refactor(declarative): centralize resource lifecycle reconciliation #2080

Description

@rspurgeon

Problem and desired outcome

Resource planners repeatedly implement desired/current matching, create and
update selection, delete mode, sync pruning, and protection decisions.
Centralize these lifecycle decisions so a policy correction applies once to
all resources using that lifecycle.

The engine should reconcile desired state against explicit observed state;
resource adapters should supply identity, comparison, payload construction,
and exceptional API behavior.

Current progress

Latest completed milestone: PR #2227 merged on 2026-09-18 as
8e4aee512. Policy and MCP-server identity/retention now use API names.
Policies join agents, models, and vaults on the shared name-matched child
reconciler. MCP ordering and dependencies remain explicit. The change removes
90 production Go lines and adds the authorized 18-case focused test file.

All seven merged files match reviewed head 6c2f84855. Final local gates
and CI/E2E passed; the review thread is resolved. The merged local worktree,
branch, and deleted remote-tracking ref are removed. Artifacts are preserved
under /home/rspurgeon/.cache/kongctl-ai-child-identity-2223. The clean
session anchor is detached at current main da8e0414f; shared main and
parallel worktrees were untouched.

Recommended next: AI Gateway model providers and auth strategies under
#2223. Their resource matchers use names, but planners still prefer IDs.
Both have detail-read/recreate lifecycles compatible with the existing
child strategy. Their ID-first test expectations need explicit maintainer
permission to change. Keep identity correction separate from extraction.
#2223 and this umbrella remain In Progress; further lifecycle families,
certificate identity, and required-name defaults remain.

Earlier milestones include #2218's first child strategy and all ten managed
roots sharing observation-cache policy (#2130/#2139). No tests were rerun for
this cleanup and read-only assessment.

PR #2085 merged on 2026-09-06 as 2edd247d. This issue remains
In Progress: the managed-root pilot is complete, while additional
lifecycle families and broader migrations remain.

The pilot's documented coverage limitations remain relevant. Additional
regression tests require maintainer permission; no tests changed in #2085.

Name-matched child reconciliation — PR #2218 (merged)

AI Gateway agents, models, and vaults share reconcileNameMatchedChildren.
The reconciler matches names within the caller's parent scope, refreshes
matched observations, chooses create/update, propagates read/comparison
errors, and prunes in observed order with fail-fast protection validation.
Absent detail responses still schedule creation.

The final contract follows the name-only direction established for gateway
roots and consumers in #2096. UUID-shaped refs and cached IDs cannot override
a different declared name or retain a differently named child during sync.
Resource-level matchers follow the same rule. An agent's matched remote ID
is bound before the detail read.

Typed adapters retain API reads, payload comparison, change construction,
and dependencies. Parent scope and new-parent creation stay with callers.
The final PR removes 159 production Go lines. The contributor guide and user
resource reference explain eligibility, name matching, pruning, and migration:
a changed name declares a different resource; sync may delete the former
resource when it is no longer declared in scope.

The correction is a separate commit after the refactor and initial test
commit, preserving a reviewable boundary between restructuring and behavior.
Implementation head: 0f99863ac0e05fc3084c98fdc3439225518c0113.
Base: main c831d87c4592fe36f56eddbfefbda0bd92a6390a.
Merge: e5d88e63b9bd971b44779ef61c9ae5778c5fa3b9.
The completed local worktree and branch have been removed.

Validation:

  • Unchanged resources/planner/loader suites pass on the rebased pre-fix
    checkout before the identity correction.
  • The approved focused test file was revised and renamed to
    name_matched_child_reconciler_test.go. It has six agent-planner cases
    and twelve resource-matcher cases spanning all three resource types.
  • Coverage checks UUID/cached-ID conflicts, missing IDs, new-name creation,
    absent-detail recreation, bound IDs after name matching, and exact sync
    retention/deletion order. The revised tests fail against the previous
    production implementation using temporary production overlays.
  • Read-only modernization, clean formatting, CGO-disabled build, and full
    make test-all pass, including lint, installer checks, race-enabled
    unit/integration suites, and E2E tooling checks.
  • Every other existing test, fixture, mock, and unrelated file is unchanged.
    All 86 local contributor-guide links resolve; changed Markdown wraps at
    80 columns.
  • The original review thread contains the corrected contract and test
    evidence and remains resolved. Final-head CI and all five E2E groups,
    replay/coverage verification, and E2E Required pass:
    https://github.com/Kong/kongctl/actions/runs/35306068769

Earlier 15,630 baseline/candidate comparisons and green CI established
compatibility for the original extraction, which preserved legacy ID-first
behavior. They do not establish correctness of that identity policy. Those
claims and the earlier ID-precedence test expectations are superseded by the
explicit name-only correction and regression coverage above.

Current validation evidence:
/home/rspurgeon/.cache/kongctl-child-reconciliation/name-matching-2218.
Original compatibility audit:
/home/rspurgeon/.cache/kongctl-child-reconciliation/dd6c2dceb.

#2223 tracks the seven remaining AI Gateway ID-first child planners,
certificate/title/SNI matcher consistency, and required-name defaults
(including the unchanged defaults for agents/models/vaults). That issue is
Todo; those additional behavior changes are outside PR #2218.
Keep #2080 In Progress after this bounded milestone: broader lifecycle
families still need separate assessments and migrations.

Observation-cache pilot merged — PR #2130

PR #2130 merged on 2026-09-10 as 4cda781b, including the focused
test follow-up at 4d44c06a. The Portal/API pilot was implemented on main
0e0ace99 in the isolated refactor/declarative-observation-cache
worktree. Its merged local branch and worktree have now been removed;
the clean session worktree is detached at main 4cda781b. Commit
7ab1d111 introduces observationCache[T] and replaces both resources'
duplicate cache state, listing policy, and namespace filters with thin typed
adapters. The guide documents the shared boundary and remaining migrations.

The unchanged focused suites, read-only modernization, production formatting,
CGO-disabled build, and full make test-all pass with the repository-pinned
golangci-lint 2.13.2. Temporary production build overlays compare 3,240
baseline/candidate sequences (19,440 results): request traces, context,
results, errors/retry, nil/empty slices, fanout, shared caches, and resets
match. This is bounded compatibility evidence. All 2,746 preexisting files
outside the two intentionally edited files retain their baseline hashes;
no test packages, files, fixtures, or helpers were added or modified.

After the maintainer authorized the focused test follow-up, commit
4d44c06a adds one 193-line observation_cache_test.go with 12 cases
asserting returned values and request traces. It detects the confirmed
nil-cache fanout regression and mutations that cache errors or treat empty
wildcard results as cache misses. All 2,749 preexisting files retain their
hashes; production code and existing tests/fixtures remain unchanged.
Modernization, formatting, build, and full make test-all pass again.
The review thread is resolved. Final-head CI, CodeQL, Code Quality,
security checks, all five E2E scenario groups, scenario-coverage verification,
and E2E Required passed before merge.

Managed-root observation-cache consolidation merged — PR #2139

PR #2139 merged on 2026-09-10 as 0304cc78, incorporating head
50d1673c. It was implemented in the isolated
refactor/declarative-root-observation-cache worktree based on main
4cda781b. Its merged local worktree and branch have been removed;
the clean session worktree is detached at current main 0304cc78.

Control planes, Event Gateway control planes, authentication strategies,
DCR providers, catalog services, AI gateways, dashboards, and organization
teams now use observationCache[T]. The existing listing entry points
remain thin typed fetch/namespace-access adapters. All ten managed roots
share the policy introduced in #2130. Eight duplicated listing policies and
eight namespace filters are removed, for 553 net lines removed.

The shared cache helper, Portal/API adapters, Portal child caches,
state-client methods, and lifecycle decisions remain unchanged. Namespace
normalization, cache lifetime/sharing, request counts/order, wildcard
behavior, nil/empty results, retry, and uncached fanout are preserved.
The authoritative guide now documents the completed managed-root boundary.

Validation:

  • Baseline and candidate make test-all pass with pinned golangci-lint 2.13.2: lint,
    installer checks, race-enabled unit tests, E2E metrics, and integration.
  • Read-only modernization, production formatting checks, and CGO-disabled
    make build-ci pass.
  • External production build overlays produce identical results and fetch
    traces for 3,240 sequences across all ten root kinds (97,200 results).
    These cover populated/nil/empty/failed/retried observations, namespace
    normalization, fanout, cache sharing, and resets. This is bounded
    compatibility evidence, not exhaustive API coverage.
  • All 2,758 preexisting files outside the production cache and guide retain
    their baseline hashes. No tests, fixtures, helpers, or test packages were
    added or changed. Guide links, anchors, and 80-column wrapping pass.

Final-head unit/integration CI, CodeQL, Code Quality, security, all five
E2E scenario groups, E2E Required, and independent approval passed before
merge. The earlier KSAI team-request dispatch failure remains in the check
history; it was a repository/team configuration error, not a refactor
failure. Do not describe the complete historical check list as all green.

The managed-root observation-cache milestone is complete. Keep #2080 open
and the project status In Progress: shared lifecycle reconciliation
still covers authentication strategies and DCR providers only. Portal child
caches also remain separate, but their small cache wrappers are a lower
priority than removing the next substantial manual resource inventory.

Recommended sequencing after #2227

  1. Correct AI Gateway provider/auth-strategy identity under fix(declarative): finish AI Gateway child identity and explicit-name contracts #2223, then
    reuse the existing child reconciler if fresh baseline coverage supports
    it. Both perform detail reads and recreate on absent detail. This corrects
    the earlier assessment that they lacked that lifecycle. Existing tests
    assert ID precedence and require permission to update.
  2. Assess stores, groups, and credentials separately for secret-child,
    membership, and replacement semantics. Keep certificate identity and
    required-name default removal as explicit compatibility changes.
  3. Continue refactor(declarative): make resource capabilities authoritative across the pipeline #2079 only where child traversal, export completeness,
    capability conflicts, or loader inventories duplicate substantial policy.
    Retain deliberate phase/grouping exceptions and MCP ordering.
  4. refactor(declarative): unify field contracts across diffing, payload mapping, and dump #2081's shared field contracts remains the next major architecture pilot:
    unify comparison/request/export policy without inventing a second schema.
    refactor(declarative): compile configuration and reference bindings into explicit internal models #2082's explicit configuration/reference bindings remains the later,
    larger track.

Use a fresh worktree on then-current main. Recheck unchanged-test baseline
and coverage before implementation; past test permissions do not authorize
future changes. Maintain the authoritative implementation guide in each PR.
No new implementation was started during merge cleanup.

Follow-up assessment during executor migration

The existing suite on 2a74716a reports 0% coverage for
planCatalogServiceChanges; its short name-based lifecycle is therefore not a
sound next migration under the unchanged-test constraint.
planDashboardChanges has 52.6% coverage and uses explicit-ID matching,
duplicate-name errors, and stable observed-order pruning, which differ from
the current name-based reconciler.

The certificate planners also need distinct sequencing policies: control-plane
data-plane certificates create replacements before sync deletion, with deletes
depending on all creates; AI Gateway certificates match by title and replace
a changed certificate by deleting it before creating its replacement. Those
behaviors must remain explicit in any shared immutable-resource strategy.
The executor registration migrations #2095/#2097/#2098 intentionally leave
these planner policies unchanged.

Reassessment after #2092 merged

The unchanged resource, loader, and planner package tests pass on merged main
38aaca3f. Fresh statement coverage confirms catalog reconciliation remains
at 0% and dashboard reconciliation at 52.6%. AI Gateway config-store
reconciliation reaches 90.9%, while both existing-gateway reconciliation and
new-gateway creation traversal for Event Gateway schema registries are at
0%; their tests primarily exercise field comparisons.

These are not equivalent child lifecycles: config stores support explicit
ID/name matching, immutable-name errors, nested-secret traversal, and
observed-order pruning; schema registries match by name and prune a map.
Select a second compatible, covered lifecycle before extracting a shared
child strategy. Keep scope ownership and replacement ordering explicit.

Observation caching remains a substantial companion opportunity:
planner/resource_cache.go repeats namespace selection, full-namespace cache
reuse, and filtering across ten managed root families. Coverage varies
considerably, so a migration should begin with covered families and preserve
request counts, error behavior, cache lifetime, and nil/empty semantics.

Historical assessment after #2118 and #2119e7278f05

The static namespace/scope milestones and first paired loader capability
migration are merged. #2119 also removes unused legacy operations and
simplifies helpers. No additional lifecycle or observation-cache migration
landed in #2092, #2102, #2103, #2108, #2118, or #2119. The shared lifecycle
pilot still covers authentication strategies and DCR providers only.

On 2026-09-09, the unchanged planner, loader, and resources package suites
pass on current main e7278f05. Fresh planner statement coverage reports:

  • listManagedPortals: 96.8%; listManagedAPIs: 74.2%.
  • Both corresponding namespace filters: 84.6%.
  • Catalog-service reconciliation: 0%; dashboard reconciliation: 52.6%.
  • AI Gateway config-store reconciliation: 90.9%.

These measurements describe the existing package suites, not exhaustive
behavioral coverage. They support candidate selection; the pilot must still
inspect the unexercised branches and preserve compatibility.

Recommended next implementation: shared typed observation caching, starting
with Portals and APIs.
planner/resource_cache.go repeats cache fields,
namespace-query selection, cache hits, wildcard reuse, and namespace filtering
across ten managed root families. Extract one shared policy with typed fetch
and namespace-access adapters, demonstrate it with the two covered families,
and remove their duplicate implementations. Migrate other compatible roots
as their validation evidence permits. Keep child caches and reconciliation
decisions outside the initial change.

Preserve all existing observation semantics:

  • Namespace normalization, cache keys, wildcard requests, and fanout behavior.
  • Cache hits, wildcard-result filtering, and request counts/order.
  • Cache lifetime and successful empty observations versus not-yet-loaded state.
  • Error propagation without caching failed observations.
  • Nil versus empty slices and current behavior when the cache is absent.

Keep this a behavior-preserving extraction; request optimizations and lifecycle
policy changes need their own scope. Update the authoritative implementation
guide in the same PR. Establish the unchanged baseline in a fresh worktree,
inspect uncovered behavior, and use external comparison evidence where useful.
Existing tests remain frozen without separate maintainer permission.

Keep #2080 open and In Progress. After observation policy is shared,
select a second compatible lifecycle using fresh coverage and the explicit
identity, pruning, replacement, and parent/child ordering constraints above.

Evidence from the current code

Assessment baseline: 0fe92b40.

Proposed design

Use a small set of explicit lifecycle strategies: managed roots, parent-scoped
collections, create/delete resources, singleton updates, and assignments.
Avoid one algorithm with many unrelated boolean switches.

A strategy consumes desired and observed records and invokes narrow typed
operations for matching, comparing, and constructing changes. Shared code owns
mode handling, common protection decisions, and traversal. Keep API-specific
semantics in adapters, including immutable-field errors, replacement ordering,
unobservable fields, and full-update requirements.

Observation is a separate responsibility from reconciliation decisions.
Existing state-client and cache methods can remain the first observation
adapters. A whole-organization eager snapshot is unnecessary; reads may be
demand-driven and namespace-aware. Preserve read behavior during extraction
before optimizing requests.

Initial actionable milestone

  1. Choose a managed-root implementation with substantial unchanged coverage.
    Extract its lifecycle traversal into a typed reusable reconciler while
    preserving existing field comparison and change-builder functions.
  2. Migrate a second compatible root implementation to demonstrate actual
    shared policy and remove duplicate lifecycle code.
  3. Route migrated implementations through the resource planning inventory.
  4. Exercise a different lifecycle using a separate narrow strategy, such as
    control-plane data-plane certificates or portal customization, before
    claiming the abstraction fits children and singletons.
  5. Stop the first migration at a complete, useful boundary. Broader family
    conversion stays tracked here.

Candidate selection must follow existing coverage. Do not choose a weakly
covered resource merely because its implementation is short.

Semantics that must stay explicit

  • Omitted sync collections do not request deletion; explicit empty scoped
    collections do. An external parent can have managed child operations while
    remaining outside the parent's managed lifecycle.
  • Managed root identity, child parent-plus-moniker identity, and explicit-ID
    matching have different rules.
  • Preserve protection-error accumulation and when child planning proceeds
    after a parent decision.
  • Create-only resources, update-only singletons, optional singleton deletion,
    and API replacement ordering remain separate capabilities.
  • Do not turn a failed observation into an empty successful observation.
    Preserve current missing-client behavior during migration.
  • Planning-time protection validation does not replace execution-time
    revalidation against current remote state.
  • Secret write intent selection stays separate from observable-field diffing.

Acceptance criteria

  • At least two migrated resources use a shared lifecycle algorithm.
  • Common mode/protection/pruning decisions are absent from their adapters.
  • Typed resource comparison and SDK-specific exceptions remain local.
  • Observation and lifecycle decisions have a clear boundary.
  • Scope handling and parent/child ordering remain unchanged.
  • Existing error aggregation and warnings are preserved.
  • Old duplicated lifecycle implementations are removed or reduced to
    delegating compatibility entry points.
  • Each migrated lifecycle passes the unchanged applicable suites.

Sequencing

Implement alongside the resource capability model. Defer universal field
mapping, a new saved-plan format, a new scheduler, and broad cache performance
changes. #2051 remains a separately authorized conformance-testing follow-up.

Compatibility and validation

This is a behavior-preserving refactor. Preserve accepted YAML, omitted versus
explicit-empty scope, namespace and protection behavior, identity precedence,
external resources, error propagation, secret redaction, and saved-plan
serialization. Preserve execution ordering and request counts where existing
tests specify them. Do not silently broaden support while moving code.

Existing test packages, test files, fixtures, mocks, and golden artifacts must
remain unchanged without explicit maintainer permission.
Establish the
unchanged test baseline before implementation. Keep test-facing entry points
as delegating adapters when needed. New regression or conformance tests are a
separate follow-up requiring permission; their desirability does not authorize
test changes in this refactor.

Run production-only modernization/formatting, a CGO-disabled build, lint, the
unchanged unit suite, and applicable existing integration checks after each
migration. Avoid repository-wide rewrite commands that can modify tests.
Record baseline failures separately from regressions, and verify test-file
hashes and the final Git diff.

Refactoring program

Start with bounded #2079/#2080 migrations. #2081 and #2082 remain later design
tracks; the initial implementation does not close the whole program.

Implementation guide acceptance criteria

Treat docs/contributor/declarative-resource-implementation-guide.md as the authoritative implementation entry point for agents throughout this refactoring. Guide maintenance is part of the implementation, not a documentation follow-up.

  • Document each implemented lifecycle strategy, the semantics that make a resource eligible for it, representative current implementations, and the behavior that remains resource-specific.
  • Update the guide in the same change as the code; replace obsolete instructions and checklists instead of appending competing patterns or historical baseline sections.
  • Keep the workflow concise, retain behavioral invariants, and link to current implementations instead of copying large implementation templates.
  • Verify code references, local links, and section anchors. An agent should be able to follow a resource or field through loading, planning, execution, and export without guessing missing wiring.
  • Preserve maintainer constraints, including the existing test freeze; documentation or golden-regeneration instructions do not authorize test changes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions