Skip to content

refactor(declarative): make resource capabilities authoritative across the pipeline #2079

Description

@rspurgeon

Problem and desired outcome

Adding a declarative resource currently requires repeated edits to shared
infrastructure beyond the resource's actual API behavior. Make an established
resource lifecycle available through one co-located capability definition and
typed adapters, so adding a kind does not require adding branches throughout
the loader, planner, executor, and export pipeline.

This is an architectural program, delivered in independently reviewable
migrations. The interfaces below describe responsibilities, not a final API.

Current progress

Latest merged milestone — PR #2281: merged as c5cfb23.
Portal child export now uses thirteen ordered collectors covering fifteen
exported kinds plus an explicit team-group-mapping omission. Shared adapters
handle collections, singletons, and named maps. Ownership guards preserve
nested team roles and combined assets. Fatal page errors identify the Portal
and wrap the original cause. Coverage tests select collectors by kind.
The authoritative implementation guide was updated in the same PR.

Together with #2269, #2273, and #2276, registered child export now covers
AI Gateways, Event Gateways, APIs, and Portals. The remaining control-plane and organization export assessment is complete;
PR #2289 closes managed coverage while preserving their different contracts.

Related correctness fix — PR #2283: merged as 44a7a47; #2282 is
Closed / Done. Model/provider reference dependencies now precede compatible
AI Gateway serialization. Live before/after reproduction and SDK runtime
ordering compatibility passed. The defect predated refactoring (#1361);
this was a separate correction, not a new lifecycle migration.

Latest merged planning milestone — PR #2287: merged as ae9bb99.
Portal child planning now shares one ordered traversal with explicit policies
for new, existing, and external parents. It removes 252 net production Go
lines while preserving scope, errors, order, and asset behavior. The guide and
14 focused parity/regression cases shipped with the change.

Latest merged export milestone — PR #2289: merged as f2de508.
Dump registration now accounts for every managed kind, including children
and selector assignments. Family coverage derives from actual child
inventories and explicit omissions. Control-plane certificates use the
shared adapter; gateway services retain their separate earlier export step.
Organization assignment builders preserve membership-derived selectors and
declare their coverage beside the custom traversal. Export-completeness
assessment is complete; this adds no new dump support.

All six merged files match reviewed head 9695f0c. Final PR checks report
35 successful checks and eight skips, with no failures. Local full gates,
all 20 unchanged replay scenarios, and 21 focused cases passed. The original
tests/fixtures remained unchanged; original-traversal parity and two mutation
checks passed. The authoritative guide was updated.

Completed session worktrees/branches for #2281, #2283, #2287, and #2289 were
removed after merge verification. The SDK compatibility scratch checkout was
also removed after archiving its patch and changed/untracked files. Validation
artifacts are preserved under
/home/rspurgeon/.cache/kongctl-post-2289-cleanup/, with earlier evidence
directories retained. Shared main and parallel worktrees were left untouched.
The session anchor is clean and detached at origin/main c5ea2ee.

Companion lifecycle milestones — #2293 through #2296 are merged.
#2080 now covers ten AI Gateway child kinds, nine Event Gateway child kinds,
three Portal singletons and two managed roots. Its next proposed unit is
managed roots owning children, beginning with control planes and Portals.
That work remains separate from the registration work tracked here.

Recommended next #2079 workstream: ordered cross-reference validation
participation. loader.validateCrossReferences still explicitly enumerates
Portals, nested API publications/implementations, flattened API children,
Portal IP allow lists, and Portal audit-log webhooks. Co-locate participation
with declarations and guard coverage/dispositions while retaining exactly the
current enrollment, nested/flattened visits, and first-error order. Registering
all relationship-bearing resources automatically would broaden validation;
treat any such behavior correction as separate scope. Grouped extraction
sequencing and capability conflicts remain follow-up audit targets.

This issue remains In Progress. Export completeness is complete. Remaining
work includes child-planner traversal beyond the completed families, capability
conflicts, and substantial loader enumeration. #2081/#2082 remain later design tracks.

PR #2085 merged on 2026-09-06 as 2edd247d. This issue remains
In Progress: the root-planning milestone is complete; the broader
cross-pipeline capability program still has work remaining.

PR #2092 merged to main on 2026-09-07 as 38aaca3f, including the
#2095/#2097/#2098 migrations. The executor-registration milestone is complete:
all 63 constructors, supported actions, and payload contracts are registered
together; 171 action-switch cases and all concrete per-resource fields are
removed from the engine. The authoritative guide describes every family and
the remaining manual integration points. This broader issue remains open
and In Progress.

Static metadata milestone — PR #2102

PR #2102 merged to main on 2026-09-07 as 329d7d6a. Its namespace
and initial scope-registration milestone is complete, including the
reviewed descriptor cache with defensive copies.

  • All 12 namespace participants now register beside their declarations.
    Ten resources reuse their existing slice accessors; the two organization
    selectors remain outside the ordinary resource inventory. The loader's
    independent namespace enumeration is removed.
  • All eight ordinary root collections use scope registration. API versions,
    publications, implementations, documents, and control-plane data-plane
    certificates share child ownership metadata between loader presence
    capture and planner fallback inference.
  • Root/nested YAML keys come from shared declaration structure used by
    explain; parent selectors come from relationship descriptors. The
    migrated scope-table entries and planner loops are removed.
  • The authoritative guide documents the new entry points and the remaining
    manual integration paths.

The unchanged focused suites and full local make test-all pass, including
lint 2.13.1, installer checks, race-enabled unit/integration suites, and
30 metrics tests. Read-only modernization and the CGO-disabled build pass.
All 1,590 existing test/fixture/helper files match baseline SHA-256 values.
No test files/packages were added or modified.

This does not finish the static capability program. Specialized child scope
(singletons, indirect ownership, assignments), grouped roots, namespace
inheritance/filtering, extraction, and broader capability completeness
remain. Keep this issue open and In Progress through the remaining
migrations.

Nested AI Gateway and Event Gateway scope — PR #2103

PR #2103 merged to main on 2026-09-07 as
b518225c, including the authorized coverage follow-up at 20cf3202.
This nested gateway scope milestone is complete. Implementation used the
isolated refactor/declarative-child-scopes worktree based on 329d7d6a;
parallel worktrees were left untouched.

  • All 16 AI Gateway and 11 Event Gateway children now register scope beside
    their declarations. Together with refactor(declarative): register namespace and collection scope capabilities #2102, registration covers eight roots
    and 32 children.
  • Child owners may themselves be children. Nested paths derive from the
    declared ownership chain and existing structural metadata, removing the
    two families' loader inventories, specialized presence walkers, and
    planner inference loops. The production migration removes 763 lines and adds 397.
  • WithEmptyRootCollectionError retains all 16 AI Gateway loader-stage
    empty errors, their exact messages, and diagnostic order. Event Gateway
    retains planner-stage empty-collection validation.
  • WithNestedScopeWithin preserves Event Gateway's existing policy
    placement boundary: nested capture under event_gateways, without adding
    it under root-declared listeners or virtual clusters. Root-level policy
    declarations and planner fallback still use their immediate owner.
    AI credentials and config-store secrets retain capture under both
    root-declared parents and parents nested under AI gateways.
  • Descriptor caching and defensive copying are retained for all path slices.
    The authoritative guide documents both compatibility options and the
    remaining manual paths.

Validation: unchanged focused suites, read-only modernization, production
formatting, CGO-disabled build, and full make test-all pass (lint 2.13.1,
installer checks, race-enabled unit tests, 30 metrics tests, integration).
A baseline audit verifies all 27 owners/root keys, 29 nested paths, 16
ordered errors, and unchanged Portal/organization inventories and specialized
scope entry points. All 1,590 protected test/fixture/helper files retain their
baseline SHA-256 values. CI, including 166 E2E scenarios, passed on the
production refactor at af3de038.

The maintainer subsequently authorized a narrow coverage expansion for the
review finding. Commit 20cf3202 adds one loader test with 36 cases covering
all four Event Gateway policy kinds and the nested/root placement boundary.
Exact child scopes and root-empty markers are asserted. A temporary build
overlay removing the filter makes the eight expected root-parent cases fail.
No existing test files or production files changed in that follow-up.
All local gates passed again. Final-head CI, CodeQL, security checks, and
E2E passed before merge; all three review threads are resolved.

Portal/organization scope registration follows in PR #2108 below.
Keep #2079 open and In Progress through the remaining migrations.

Portal and organization scope — PR #2108

PR #2108 merged on 2026-09-08 as 34885e29, including the authorized
loader coverage follow-up at 176d133e. It was rebased onto 662b953b
before the test addition. This completes the scope-registration milestone.
The implementation used an isolated worktree; its merged branch and worktree
have now been removed during session cleanup.

  • Register the remaining 16 Portal children, five organization assignment
    kinds, and the grouped organization-team/dashboard roots. Scope metadata
    now covers all ten managed roots and 53 child kinds. The two external-only
    resource kinds remain outside managed scope.
  • Derive grouped declaration paths and Portal direct child/singleton/map
    keys from resource declarations. Remove the loader's final 21-entry
    root-child, 12-entry nested Portal, and eight-key singleton inventories,
    along with the Portal/organization planner inference loops. Schema null
    diagnostics share the derived Portal keys.
  • Preserve Portal team roles' portal sync owner without changing their team
    structural parent. Registered policies retain team/role coupled scope,
    portal-owned group mappings, asset handling, singleton null diagnostics,
    and per-portal error order. Root portal_teams capture retains its distinct
    scope behavior.
  • Keep organization users/system accounts as group-scoped selectors outside
    the ordinary resource lifecycle. Flat assignment presence captures parent
    keys; fallback inference marks selector groups. Explicit scope, AI's 16
    ordered loader errors, and Event Gateway policy placement are preserved.
  • Update the authoritative implementation guide with the new registration
    options and remaining manual integration work.

Validation: unchanged focused suites, read-only modernization, production-only
formatting checks, CGO-disabled build, and full make test-all pass (lint
2.13.1, installer checks, race-enabled unit tests, 30 metrics tests, and
integration). An external Go build-overlay comparison produces identical
results for 271 baseline scope-capture/inference cases. This is bounded
compatibility evidence, not exhaustive coverage. All 1,591 protected
test/fixture/helper files retain their baseline SHA-256 values; no test files
or packages were added or changed. Guide links, anchors, and wrapping pass.

On 2026-09-08, rebase #2108 onto main at 662b953b, incorporating the
upstream E2E scenario updates without conflicts. Both production commits
replay unchanged (5589222c and 32847574); the latter uses full declaration
paths in empty-collection diagnostics, addressing the first review finding.

After the maintainer authorized the narrow loader coverage follow-up, commit
176d133e adds portal_team_sync_scope_test.go with four cases covering
nested empty teams, omitted teams, populated root teams, and empty root teams.
It asserts exact child scope and root-empty markers. Temporary production
build overlays demonstrate failures for missing nested role scope and
incorrectly broadened root role scope, each in the intended case.
All 2,749 preexisting files match the post-rebase baseline, including all
1,591 protected test/fixture/helper files. Only the authorized test file is
added; no existing tests, fixtures, E2E scenarios, or production files change
in this follow-up. Modernization, formatting, build, and full make test-all
pass again. Both review threads are resolved. Final-head CI, CodeQL,
security checks, and all five E2E scenario groups passed before merge;
the final scenario-coverage verification and E2E Required status passed.

The scope-registration migration is merged. Remaining #2079 work includes
loader extraction and validation
inventories, namespace inheritance/filtering, and cross-pipeline capability
completeness. #2079 remains open and In Progress; #2080's lifecycle and
observation work remains a separate active track.

AI Gateway child loading milestone — PR #2118

PR #2118 merged on 2026-09-08 as 7861ba3b. All 16 AI Gateway child
kinds now register paired typed extraction and resource-set validation beside
their declarations. Registration reuses root slice accessors and replaces the
loader's separate extraction and validation inventories for this family.

Extraction order and diagnostic order remain separate explicit policies.
Config-store secrets retain their specialized extraction/defaulting phases;
consumer credentials retain their later extraction phase. Parent assignment,
root-versus-nested behavior, errors, and ordering are preserved. The
implementation guide documents the capability boundary and remaining manual
loader phases.

The production refactor passed the unchanged suites and all local gates.
An external Go build-overlay comparison matched 324 baseline cases: 120
extraction, 136 diagnostic-order, and 68 plain-loader cases. This is bounded
compatibility evidence, not exhaustive coverage.

After maintainer approval, the only test change was a new 36-line
resources/child_load_test.go. It pins the complete ordered list of all 16
validators. Temporary production overlays confirmed that it fails when SNI
validation is omitted or validation order changes. All preexisting tests and
fixtures remained unchanged. Final-head CI and E2E passed; all review threads
were resolved before merge.

PR #2119 also merged as 811df7b0, removing unused legacy API child
operations and simplifying shared helpers (493 net lines removed). It closed
#2104 and #2114 without changing tests. The guide now points to the active
runtime API-parent reference path.

Desired namespace selection milestone — PR #2141

PR #2141 merged on 2026-09-10 as 23df22a1, incorporating reviewed
head 0c758f22. Implementation used the isolated
refactor/declarative-namespace-selection worktree, based on 5ced01a3.
The namespace-selection milestone is complete. This issue remains open and
In Progress for the remaining loader and capability migrations.

  • All ten managed roots now derive desired namespace selection from their
    existing WithNamespace metadata and registered flattened storage.
  • GetResourcesByNamespace[R] shares traversal; WithNamespaceFrom
    co-locates typed owner lookup for four API children and Portal pages.
    Fifteen retained getters delegate to these capabilities.
  • Remove 25 unused namespace getters and ten unused planner wrappers:
    the initial production refactor removes 579 net lines. Keep the existing
    Portal-pages test-facing adapter.
  • Preserve external/default namespace policy, exact/first-match parent
    lookup, missing-owner exclusion, nil results, source order, shallow
    copies, and grouped-collection exclusion. Organization assignment
    selection remains specialized.
  • Update the authoritative guide with the selection entry points and
    remaining manual rules.

Baseline and candidate local gates pass. After rebasing onto the E2E
infrastructure update in 5ced01a3, read-only modernization, production
formatting, CGO-disabled build, lint 2.13.2, installer checks, race-enabled
unit/integration suites, and all 57 E2E tooling tests pass again.
An external Go build-overlay comparison matches 123,000 selection results
and 26,112 namespace-participant observations against the original
implementation, including returned-copy isolation and metadata sharing.
This is bounded compatibility evidence.

The implementation patch is unchanged by rebase. SHA-256 comparison against
rebased main verifies all 2,759 preexisting files outside the intended ten
tracked edits are unchanged; only one new production file is added.
The initial production refactor changed no tests, helpers, or fixtures.
Its unit/integration CI, CodeQL, and E2E checks subsequently passed.
Guide wrapping and all 72 local links/anchors pass.

Following the maintainer's robustness finding, 0c758f22 records typed
namespace owners and adds the proposed 30-line registry contract test.
After all registrations finish, the contract verifies every selected kind's
owner chain reaches a registered namespace root without cycles or missing
selection capabilities. Direct generic calls for unsupported kinds remain
programming errors; this contract validates registration dependencies.

Temporary production overlays confirm failures for an unregistered owner,
an owner without selection, and an ownership cycle. All preexisting test
bodies and other test files, fixtures, and helpers remain unchanged.
The only test edit is the new function in registry_test.go.
Modernization, formatting, build, full local gates, and the original
123,000-selection/26,112-participant comparison pass on the final code.
Final-head CI is complete: 33 checks passed and five were skipped, with no
failures or pending checks. Unit/integration, CodeQL/Code Quality, all five
E2E scenario groups, Control Plane replay, coverage verification, E2E Required,
and security checks passed before merge. The complete merged change removes
543 net lines, including the contract and guide follow-up.

Post-merge cleanup verified that the reviewed branch integrated onto the merge
parent produces the exact merged tree. The clean implementation worktree,
local branch, and stale remote-tracking reference have been removed; GitHub
had already removed the remote branch. The retained session worktree is clean
and detached at 23df22a1. Other worktrees and the main checkout were left
untouched.

Portal child loading milestone — PR #2143

PR #2143 merged on 2026-09-11 as 11c67aa8, incorporating production
commit f01a8123 and authorized test follow-up 263fb0e8. Its isolated
refactor/declarative-portal-child-loading worktree and local branch have
been removed. The session worktree is clean and detached at current main
11c67aa8; other worktrees and the shared main checkout were left untouched.

  • All fourteen Portal child extraction paths and twelve existing validators
    register beside their declarations and reuse registered storage. The
    loader's separate Portal extraction and validation inventories are removed.
  • Shared child loading supports ordinary slices and typed custom extraction.
    Singleton copying, recursive page preorder, template key defaults, nested
    team ownership, diagnostic order, and both loader representations retain
    their existing behavior.
  • Team and team-role registrations explicitly record their validation
    omission. Root-declared teams retain their existing extraction boundary;
    Portal assets retain separate handling.
  • The authoritative implementation guide documents the registration forms,
    compatibility boundaries, and remaining manual integration points.

The production refactor preserved all existing tests and passed full local
gates. An external comparison matched 198 extraction, validation, and
environment-source cases. The authorized follow-up adds one 133-line
child_load_contract_test.go covering extraction conflicts, validation
omission rules, omitted-validator errors, and the exact Portal validator
membership/order. Eight production mutations fail in the intended tests;
all 2,771 preexisting tracked files retain their pre-follow-up hashes.

Final-head CI completed with 33 successful and ten skipped checks, including
unit/integration, CodeQL, Code Quality, security, replay, live E2E, and
E2E Required; there were no failed or pending checks. The review thread is
resolved. Cleanup verified that merging reviewed head 263fb0e8 onto the
merge parent produces the exact tree of 11c67aa8.

This completes the Portal child-loading milestone. The architectural program
remains open and In Progress.

API child loading milestone — PR #2148

PR #2148 merged on 2026-09-11 as 3de35ecb, incorporating head
af5fcf49. It was implemented in the isolated
refactor/declarative-api-child-loading worktree, based on 11c67aa8.
Post-merge cleanup verified all twelve PR files match the merged versions,
then removed the clean completed worktree and branch.

  • API versions, publications, implementations, and documents register
    extraction and validation beside their declarations. The loader's separate
    extraction, nested-child validation, and root-child validation inventories
    are removed.
  • apiChildLoad shares ordinary extraction and per-API/root validation.
    The shared child loader supports an optional typed per-parent validator,
    run in extraction order. API and ordinary Portal child ref checks reuse
    the same validation policy; the Portal test-facing adapter is preserved.
  • Document flattening moves beside its declaration. Nested documents remain
    under APIResource.Documents; root documents normalize separately.
    Parent selectors, explicit child API overrides, preorder, copying, and
    allocated empty document slices retain legacy behavior. Canonical storage
    remains separate refactor(declarative): compile configuration and reference bindings into explicit internal models #2082 work.
  • The authoritative guide documents the registration paths, both validation
    phases, storage exceptions, and remaining manual phases.
  • Production code and the guide remove 43 net lines; the focused test adds
    36 lines, for seven net lines removed overall.

Baseline and candidate make test-all pass: lint 2.13.2, installer checks,
race-enabled unit/integration suites, and 72 E2E tooling tests. Read-only
modernization, production-only formatting, and make build-ci pass.

An external Go build-overlay comparison matches all 199 baseline cases:
88 resource-validation cases, 16 cross-kind ref cases, 12 family/parent
ordering pairs, seven full-validation phase cases, six ordinary extraction
cases, 64 document flattening/re-extraction cases, and six parsing/environment
cases. This is bounded compatibility evidence, not exhaustive coverage.
The production refactor preserved all 1,614 preexisting test/fixture/helper
files, adding or changing no tests. Of 2,783 tracked baseline
files, only the eight intended production files and the guide changed;
two production helper files were added.

Review follow-up 2f2d39c2 clarifies the defensive pre-extraction validation
callbacks in a source comment and the guide. Full local gates pass again;
all 2,785 reviewed-head tracked files were checked, and only those two
documentation locations changed. Existing production loader coverage was
verified with three parent-assignment mutations, each failing its intended
unchanged assertion; the validator fixture helper was left intact.
The clarification thread is resolved. The API registration-order contract
finding is addressed by the authorized follow-up below.
The initial refactor passed CI unit/integration, CodeQL, security, replay,
and all five live E2E groups; the manually requested KSAI review completed.
After explicit maintainer authorization, af5fcf49 adds one 36-line
resources/api_child_load_test.go. It pins API versions, publications,
implementations, and documents in both the extraction and root-validation
registries; extraction order also controls per-parent validation.
Three production-only mutation overlays confirm that independent extraction
and validation reordering fail the corresponding subtest, and omitting the
document registration fails both. Full local gates pass again. All 2,785
preexisting tracked files retain their pre-follow-up hashes; no existing
production, documentation, tests, or fixtures changed. Both review threads
are resolved. Final-head unit/integration CI, code analysis, security,
all five E2E groups, coverage verification, E2E Required, and independent
approval passed before merge.

Event Gateway child extraction — PR #2155

PR #2155 merged as 85561602 on 2026-09-13 UTC, incorporating reviewed
head 730b3069. The isolated
refactor/declarative-event-gateway-child-loading worktree and local
branch have been removed. The session worktree is clean and detached at
merged main; other worktrees and branches were left untouched.

  • Static keys, TLS trust bundles, schema registries, backend clusters,
    listeners, and data-plane certificates register extraction beside their
    declarations. The loader's six-kind extraction inventory is removed.
  • eventGatewayChildLoad records the shared family and existing absence of
    family-level loader validation through the established omission disposition.
    The shared child-loading API and existing validators are unchanged.
  • Preserve extraction order, parent-selector overwrite, root-before-nested
    append order, shallow-copy behavior, nil/empty slices, repeated extraction,
    and deferred environment/secret attribution. Virtual clusters and all four
    policy kinds retain their existing placement and consumer behavior.
  • Update the authoritative guide's loading instructions and exceptions.

Baseline and candidate make test-all pass both before and after the
rebase onto 892b1ddc: lint 2.13.2, installer checks, race-enabled
unit/integration suites, and 76 E2E tooling tests. Read-only modernization,
scoped formatting, CGO-disabled make build-ci, and guide checks pass.

An external production build-overlay comparison matches 601 cases:
361 extraction cases and 240 parsing/validation cases, including 120 existing
Event Gateway manifests. The latter include 233 successful parses and seven
matching expected parse errors. The candidate audit also verifies the six
registration entries, extraction order, and validation dispositions.
This is bounded compatibility evidence, not exhaustive coverage.

After green production CI and explicit maintainer authorization, the focused
follow-up at 730b3069 adds one 25-line
resources/event_gateway_child_load_test.go. It pins the six extraction
kinds and their order. Two production-only build overlays confirm the test
fails when the static-key registration is omitted or extraction order is
reversed. Modernization, new-file formatting, CGO-disabled build, and full
make test-all pass again. All 2,800 preexisting files from the reviewed
production head retain their hashes. No preexisting tests, production files,
documentation, or fixtures changed in that follow-up.

The merged PR totals ten files, +111/-59. All ten files match the reviewed
head exactly. Final-head checks report 33 successes and eight skips, with
none failed or pending. The review thread is resolved.

Control-plane child loading — PR #2174

PR #2174 merged on 2026-09-14 as 96a1eee2, incorporating reviewed head
4f8feecd. All seven PR files match the reviewed head. Its isolated
refactor/declarative-control-plane-child-loading worktree and local
branch have been removed; the remote branch was already deleted.

  • Gateway services and control-plane data-plane certificates register
    extraction and collection validation beside their declarations.
    The loader's two extraction loops are removed.
  • The existing child-loading option is factored into withChildLoad.
    Gateway services compose it with their existing external registration,
    sharing the same storage accessor and preserving external lookup selectors,
    parent metadata, and materialization.
  • Both validator bodies move unchanged into their resource files. The loader
    retains individual registered validation calls around audit-log destination
    validation: services, audit-log destinations, then certificates.
    Parent lookup, external-placeholder handling, and certificate identity
    checks remain explicit.
  • Update the authoritative guide with the composition path and sequencing
    constraint. The production migration changes six files, +145/-113,
    without changing tests or fixtures.

Baseline and candidate make test-all pass: lint 2.13.2, installer checks,
race-enabled unit/integration suites, and 85 E2E tooling tests.
Read-only modernization, scoped formatting with no changes, CGO-disabled
make build-ci, and guide links/anchors/wrapping pass. All 2,795 preexisting
files outside the six intended edits retain their baseline hashes.

An external production build-overlay comparison matches 346 deterministic
cases: 120 extraction, 176 validation, six external capability/materialization,
and 44 parsing cases. It covers copying/aliasing, parent overwrite, repeated
extraction, nested/root declarations, accepted inline/external service forms,
certificate sources, parent checks, and validation precedence.

Forty synthetic cross-kind duplicate-ref cases are excluded from exact
comparison: repeated baseline runs already vary because ref lookup traverses
a map. Both moved validator bodies are byte-for-byte unchanged after removing
package qualifiers. This is bounded compatibility evidence, not exhaustive
coverage. The nondeterministic duplicate-ref diagnostic is preexisting behavior,
outside this migration's scope.

The review follow-up at b15a6c7f makes the guide explicitly require a
registered validation call for each new control-plane child. That head passed
unit/integration CI, CodeQL/security checks, and all E2E scenarios. The earlier
production-head E2E run timed out during an API cleanup read; the later run
passed with unchanged production code.

After explicit maintainer authorization, 4f8feecd adds one 33-line
resources/control_plane_child_load_test.go, pinning both extraction and
validation registration membership/order. Four temporary production-only
overlays demonstrate failures for missing registration, missing validation,
reversed extraction order, and reversed validation order. All 2,801 preexisting
files retain their hashes; no production files, documentation, existing tests,
fixtures, or mocks changed in this follow-up.

Read-only modernization, new-file formatting, CGO-disabled build, and full
make test-all pass again. Both original review findings have replies and
are resolved. Final-head CI reports 33 successful and ten skipped checks,
including successful E2E, with none failed or pending. The merged PR totals
seven files, +181/-113, including the authorized test. The existing test freeze
remains in force for further changes.

Organization assignment loading — PR #2190 (merged)

PR #2190 merged on 2026-09-16 UTC as ec6261aaa, including reviewed
head 713c7e10. Implementation was based on released main c63cb2304
in the isolated refactor/declarative-organization-loading worktree.
That completed worktree and local branch have now been removed. The remote
branch was deleted after merge and its tracking reference has been pruned.

  • Team roles, user memberships/roles, and system-account memberships/roles
    register extraction, root storage, and collection validation beside their
    declarations. The loader loses 309 net lines of independent assignment
    enumeration and retains phase sequencing.
  • Typed selector-loader handles bind user/system-account sources to assignment
    registration. These selectors remain outside the ordinary Resource registry.
    Assignment extraction shares the existing child-loading algorithm and its
    registration guards; it does not invent managed lifecycles for selectors.
  • Preserve grouped-team copy/append behavior, retained roles on already-flat
    teams, selector source order, membership-before-role extraction, and
    selector-before-assignment validation. Missing-group checks discover
    assignment collections through registration.
  • Preserve duplicate-ref rules, external team placeholders, role entity
    references, and normalization/sync-scope rebinding. Three test-facing loader
    helpers remain delegates to resource validation.
  • Update the authoritative guide in the same PR, documenting selector
    registration and all remaining phase boundaries.

The baseline and final candidate pass make test-all, including lint,
installer checks, race-enabled unit/integration suites, and E2E tooling checks.
Read-only modernization, formatting, and the CGO-disabled make build-ci pass.
All existing tests and fixtures remain unchanged. All 79 guide local reference
links resolve, with 80-column wrapping.

After green CI on the refactoring head, the maintainer authorized one focused
test follow-up. Commit 713c7e10 adds only the 193-line
resources/selector_load_contract_test.go: registration/order contracts,
duplicate owners, family/nested-validation rejection, parent-source guards,
and missing versus empty organization-group diagnostics for all four selector
assignment kinds. All 2,844 previously tracked files/links match their hashes.
Seven temporary production mutations each fail the intended focused test;
production source is unchanged. Both review findings are now resolved.
The authorization covers this focused addition only; the general test freeze
continues for future work. Follow-up artifacts are retained under
/home/rspurgeon/.cache/kongctl-organization-loading/focused-test-2190.

An external production-overlay audit compares 1,401 cases: 304 extraction,
1,070 validation/precedence, and 27 parsing cases. The 1,346 deterministic
cases match unmodified main exactly. The remaining 55 deliberately ambiguous
cross-kind duplicate-ref inputs expose existing map-order-dependent
GetResourceByRef diagnostics, reproduced by rerunning the baseline itself.
When both audit builds use identical lookup order, all 1,401 cases match.
Production lookup ordering is unchanged. Keep deterministic reference indexing
as a separate concern for #2082; this migration does not expand that scope.

Validation artifacts are retained outside the repository under
/home/rspurgeon/.cache/kongctl-organization-loading/implementation-c63cb2304.
Post-merge verification confirms that all 13 changed files are byte-identical
between reviewed head 713c7e10 and merge ec6261aaa, which is on
origin/main. Final-head unit/integration CI, CodeQL, security, all five
E2E scenario groups, replay/coverage verification, and E2E Required passed.
Both review threads are resolved. This milestone is complete; #2079 remains
open and In Progress for the remaining capability work. No source or test
files were changed during post-merge cleanup.

Ordered collection validation — PR #2197 (merged)

PR #2197 merged on 2026-09-16 UTC as a13a91130, incorporating reviewed
head 8124f8ce. All 22 changed files match that head at the merge commit.
All three review threads are resolved. The completed isolated worktree and
branch have been removed; shared main and parallel worktrees are untouched.

  • Nine managed-root collection validators and the grouped audit-log
    destination validator register beside resource declarations, using
    existing storage visitors. Seven roots share type-wide name validation;
    namespace uniqueness, external policies, audit-log ref checks, and nested
    API validation retain their specific rules.
  • Dispatch derives all 51 steps from resource and selector capabilities:
    ten root/grouped collections, 39 child collections, and two selector
    prevalidation steps. The loader no longer maintains the invocation list.
    Children inherit a family phase or supply an explicit phase; an explicit
    phase does not require a family default. Certificate validation preserves
    services, audit-log destinations, certificates. Selectors retain their own
    required phase and run before their memberships and roles.
  • Validator factories allocate fresh uniqueness state per pass and visit
    registered resource pointers. Both loader representations retain the same
    entry point. Three directly tested loader methods delegate to registered
    collection rules; unused root helpers are removed.
  • Managed roots require validation or an explicit omission. Guards reject
    incompatible typed validators, invalid phases, root/child validation
    conflicts, missing inherited/selector phases, and duplicate positions.
    Dependencies resolve after init, independently of file order; dispatch is
    cached.
  • Preserve Event Gateway's omission from this loader collection pass and
    existing child-validation omissions. Organization normalization remains
    first; cross-references and namespace checks remain last. Grouped
    extraction, saved plans, SDK payloads, and reference lookup ordering are
    unchanged.
  • Update the authoritative guide in the same PR with phase registration,
    explicit exceptions, and the focused contract test. All 82 local reference
    links and 80-column wrapping validate.

The final PR touches 22 files: 631 insertions and 467 deletions, including
217 lines in the authorized new contract-test file. The loader loses 424 net
lines. All 1,650 existing test/fixture files remain unchanged; 2,828
preexisting files outside the production/guide changes retain their hashes.

Validation:

  • Baseline and final candidate make test-all pass: lint 2.13.2, installer
    checks, race-enabled unit/integration suites, and all 99 E2E tooling tests.
    Final-head CI at 8124f8ce passed, including all five API-backed E2E
    groups, replay coverage, E2E Required, code/security analysis, and
    independent approval. Other recorded checks were skipped.
  • Read-only go fix -diff ./..., formatting, and CGO-disabled
    make build-ci pass. Existing raw SDK struct tags remain unchanged.
  • External production overlays compare 2,270 cases, each validated twice
    (4,540 validation passes per build). The final candidate matches main
    byte for byte for dispatch traces, errors, and serialized resource state.
    Coverage includes invalid root-kind pairs, uniqueness/external cases,
    nested APIs, audit-log destinations, selector precedence, and repeated
    validation. This is bounded evidence, not exhaustive API coverage.
  • Nine external misregistrations trigger their expected guards.
  • The authorized collection_validation_contract_test.go assembles actual
    registrations directly and pins all 51 kinds in diagnostic order without
    freezing numeric gaps. It covers duplicate positions, missing inherited
    and selector phases, root/child disposition conflicts, missing root
    dispositions, nonpositive phases, and mismatched typed validators.
    Negative cases use copied registries and leave the cached dispatch intact.
  • The explicit-phase regression test fails against the previous dispatcher
    and passes with the fix. Five additional production-overlay mutations
    confirm the tests detect a phase permutation and removal of the duplicate,
    inherited-phase, selector-phase, and root-disposition guards.

Artifacts remain outside the repository:
/home/rspurgeon/.cache/kongctl-validation-dispatch/ec6261aaa and
/home/rspurgeon/.cache/kongctl-validation-dispatch/findings-2197.

This milestone is complete. Keep #2079 open and the project status
In Progress for the remaining cross-pipeline work. The test permission
covered the completed focused file; future test changes require separate
maintainer permission.

Managed pipeline coverage and dump registration — PR #2208 (merged)

PR #2208 merged on 2026-09-17 UTC as dd6c2dceb, incorporating reviewed
head 0b076088. All seven changed files match that head at the merge commit.
Final-head CI, security/code analysis, independent approval, scenario replay,
all five E2E scenario groups, and E2E Required passed; the review thread is
resolved.

The isolated refactor/declarative-pipeline-contracts worktree and local
branch have been removed. GitHub had already deleted the remote branch;
its remote-tracking ref is pruned. The clean session worktree is detached
at origin/main (dd6c2dceb); shared main and parallel work are untouched.

Audit results, derived from actual initialized production registrations:

  • 65 resource declarations. Of those, 63 opt into managed sync scope and
    exactly those 63 have runtime executors. Gateway services retain
    decK/external handling; audit-log destinations are external-only. Both
    remain outside managed SDK execution scope.
  • All ten managed roots have ordered planner entry points.
  • Dump supports nine roots. Catalog-service dump remains intentionally
    unsupported. Organization user/system-account selectors remain outside the
    resource registry, while their assignments and Portal assets are registered
    managed resources with executors.
  • No missing current managed counterpart was found. The new guards prevent
    future drift; they do not add resource operations or public dump support.

Implementation:

  • ValidateManagedRootCoverage and ValidateManagedResourceCoverage
    derive expected sets from existing scope registrations. Consumers reject
    missing, unexpected, and repeated kinds with deterministic diagnostics.
    Planner assembly checks all roots without changing order; executor
    construction checks all managed kinds without changing actions/contracts.
  • One dump inventory registers supported selectors and collectors, plus an
    explicit catalog-service omission. CLI validation, help, and dispatch
    derive from it. Registration rejects duplicate kinds/selectors, missing
    collectors, and contradictory omission/implementation dispositions.
  • Eight roots share typed collect/populate/append behavior. Organization
    export retains its explicit team/role and user/system-account sequence.
    Existing API mapping, child traversal, SDK client setup, root request order,
    aliases, repeated selectors, grouped empty output, Portal child errors,
    other families' warnings, defaults, and serialization remain unchanged.
  • The authoritative guide documents the new registration workflow and exact
    boundary. All 84 local reference links and 80-column wrapping validate.

The production refactor touches six files: 340 insertions and 135 deletions.
The authorized review follow-up adds one focused 126-line test file. Its value is
removing independent wiring obligations and making missing registrations
detectable, rather than reducing net source lines.

Validation:

  • Baseline and candidate make test-all pass: lint 2.13.2, installer checks,
    race-enabled unit/integration suites, and 100 E2E tooling tests.
  • Read-only CGO_ENABLED=0 go fix -diff ./..., production formatting with
    raw SDK tags preserved, and CGO-disabled make build-ci pass.
  • 2,160 baseline/candidate dump orchestration cases match byte for byte for
    YAML output, errors, help, and SDK-getter/collector/population traces.
    Controlled results at unchanged collector boundaries cover nil/empty/
    populated collections, child inclusion and errors, aliases, repeated
    selectors, filters, namespaces, default omission, and grouped output.
    There are 1,284 successful cases and 876 expected errors, with no panics.
    This checks moved orchestration, not exhaustive remote API behavior.
  • The initialized resource inventory, planner order, runtime action map,
    and supported dump selectors match baseline exactly.
  • Ten deliberately broken production overlays trigger intended guards:
    missing/duplicate/unexpected root planners; missing/unexpected executors;
    missing dump disposition; duplicate selector; conflicting omission;
    missing collector; and duplicate dump kind.
  • All 2,854 preexisting files outside the four intentional production/guide
    edits retain their hashes. All 1,664 protected existing test/mock/fixture
    paths remain unchanged. The initial refactor added no tests or fixtures.
  • Authorized follow-up 0b076088 adds capability_coverage_test.go covering
    both wrappers, missing/duplicate/unexpected kinds, root/child filtering,
    deterministic diagnostics, and preservation of the input slice. The focused
    test reaches every statement in capability_coverage.go.
  • Follow-up make test-all, CGO-disabled build, modernization, and formatting
    pass. All 2,860 preexisting tracked files retain their hashes; only the new
    test file was added.

Artifacts remain outside the repository:
/home/rspurgeon/.cache/kongctl-pipeline-contracts/a13a91130;
follow-up evidence is in the sibling findings-2208 directory.

Keep #2079 open and In Progress. The contract covers resources opted into
managed scope; it does not establish per-action planner reachability, every
child planner/export path, or field/reference semantics.

Remaining sequencing after #2208 is reviewed

  1. Review refactor(declarative): validate pipeline coverage and register dump collectors #2208 and its final CI results, then merge when approved. Do not
    repeat the completed coverage audit or collection-dispatch migration.
  2. Resume refactor(declarative): centralize resource lifecycle reconciliation #2080 with a sufficiently covered lifecycle and a second compatible
    consumer. Dashboard identity/protection and parent-scoped AI config stores
    remain candidates with distinct semantics. Reassess current unchanged
    coverage before selecting a strategy.
  3. Retain explicit grouped extraction, selector normalization, and exceptional
    defaulting phases. Further refactor(declarative): make resource capabilities authoritative across the pipeline #2079 work should remove substantial duplicate
    wiring; child-planner traversal and child export remain separate surfaces.
  4. refactor(declarative): unify field contracts across diffing, payload mapping, and dump #2081 field contracts and refactor(declarative): compile configuration and reference bindings into explicit internal models #2082 compiled identity/bindings remain later
    design tracks. Refactor: namespace iterator #1154 retains selector-only CLI emptiness compatibility.
    None of the broader program issues is complete.

Keep implementation in fresh worktrees based on then-current main, maintain
the authoritative guide in each change, and preserve the standing test
restriction. This PR does not extend earlier narrow test permissions.

Historical namespace selection assessment — 0304cc78

This issue remains open and In Progress. Root planner construction,
executor dispatch, namespace participation, all managed sync-scope
registration, and paired loading for the 16 AI Gateway child kinds are
complete. Other loader families, desired namespace selection, and
cross-pipeline capability completeness still need work.

The companion #2080 observation-cache migration is now complete. #2130
introduced the Portal/API pilot, and #2139 merged on 2026-09-10 as
0304cc78, incorporating 50d1673c. All ten managed roots share
observation policy; #2139 removes 553 net lines and updates the guide.
Baseline/candidate local gates and 97,200 compared observation results and
fetch traces pass. No tests or fixtures changed in #2139. Final-head
unit/integration CI, code analysis, E2E, and independent approval passed.
The earlier KSAI team-request dispatch failure remains historical.
The completed local branch/worktree have been removed; the clean session
worktree is detached at current main.

Recommended next implementation: consolidate desired-resource namespace
selection through resource capabilities.

A fresh caller audit of resources/types.go finds 45 namespace getters
spanning 707 lines of function definitions. Of these, 16 have no production
references and another ten are reached only through unused BasePlanner
wrappers. These 26 are not active production selection paths; the Portal
pages getter still has a direct existing test caller. Avoid building new
capabilities for obsolete paths merely to preserve their inventories.

Proposed first PR, in a fresh worktree based on current main:

  • Remove confirmed-unused AI Gateway and Portal getter paths and their
    unused planner wrappers. Retain the Portal pages test-facing entry point
    as a delegating compatibility adapter; preserve all existing tests.
  • Share the active managed-root and API-child namespace selection policy.
    Reuse registered storage, namespace metadata, and typed ownership
    adapters. Co-locate necessary policy with resource declarations instead
    of introducing another central resource list. Portal pages can exercise
    the same child-owner boundary through its retained adapter.
  • Preserve NamespaceExternal, default namespaces, exact parent lookup
    behavior, missing-parent results, nil/empty slices, result order, and
    copying behavior. Namespace ownership is not universally the structural
    parent or sync owner; leave organization assignments and more complex
    ownership cases explicit until separately assessed.
  • Keep planner-facing methods as thin adapters where useful. Remove the
    superseded loops, update the authoritative guide in the same PR, and
    demonstrate compatibility with unchanged suites and external comparison
    evidence. No test changes are authorized by this recommendation.

The unchanged resource, loader, and planner suites pass on 0304cc78.
Coverage measured across all three packages gives 100% for Portal, control
plane, AI Gateway, and Event Gateway root namespace getters, API
publications/implementations, and Portal pages. API-root coverage is 66.7%;
API versions and documents are 50% and 60%. These measurements guide
validation; unexercised behavior still needs inspection and comparison.

After that milestone, assess the remaining active organization ownership
paths, another loader family, and capability completeness. API/Portal
loading has recursive documents, singletons/assets, and diagnostic ordering.
Event Gateway loading does not currently use the paired AI validation
dispatch; copying that registration must not silently add earlier validation.

The four program issues remain open: #2079 and #2080 are In Progress;
#2081 and #2082 are Todo. #1154 retains only its separate selector-only CLI
emptiness compatibility work. Namespace selection is now implemented in
#2141; see the milestone status above.

Historical static metadata assessment — 38aaca3f

Prioritize static resource metadata shared across namespace processing,
loader sync-scope capture, and planner scope inference. Namespace
participation still enumerates 12 kinds, and loader namespace validation
maintains an additional inventory. Loader and planner scope code separately
repeat root/child ownership and external-parent knowledge.

Start by co-locating namespace metadata/accessors with resource registration,
preserving the existing participant entry point and all consumers. Then
share declaration-location and explicit sync-owner metadata between loader
presence capture and planner scope inference. Reuse the structural facts
already derived for explain/load schemas. A namespace-only extraction is
preparatory; the larger value is removing duplicated facts across consumers.

Preserve structural parent versus sync-owner distinctions: Portal team
roles have a team parent but portal-owned sync scope. Also preserve
omission/empty semantics, singleton null handling, early-versus-late scope
errors, nested/flattened storage, traversal order, and each caller's external
namespace policy. Migrate ordinary families before exceptional groupings,
singletons, or assignments; typed extraction/validation dispatch follows
once the shared metadata expresses its requirements.

The unchanged resource, loader, and planner package tests pass on
38aaca3f. Fresh statement coverage is 95.8% for loader scope capture,
93.8% for root-child capture, 93.1% for Portal scope capture, and 100% for
loader namespace validation. These are package-test measurements, not
exhaustive behavior coverage. The combined executor refactor already passed
full CI/E2E and a local merge preview with the then-current main.

The assessment is isolated in branch
refactor/declarative-capability-assessment, based on 38aaca3f.
No implementation or test files were changed for this assessment.
Existing test constraints and guide-maintenance requirements continue.

Evidence from the current code

Assessment baseline: 0fe92b40 on astra-refactor.

Proposed boundaries

  1. Keep static declaration metadata independent of runtime API clients:
    resource kind, canonical/root/nested locations, parent selectors,
    namespace/protection participation, and external lookup support.
  2. Register runtime planning and execution capabilities alongside their
    resource implementations, using typed constructors/adapters.
  3. Compose capabilities at an explicit boundary. The engine should depend on
    contracts rather than import every concrete SDK request type.
  4. Derive ordinary enumeration, dispatch, and scope participation from the
    capability model. Avoid replacing scattered switches with scattered maps.
  5. Keep semantic distinctions explicit: managed roots, parent-scoped
    children, selectors/groupings, singletons, assignments, and external-tool
    operations are not interchangeable.
  6. Validate registrations for duplicate kinds, missing required behavior, and
    inconsistent declarations. Unsupported capabilities need an explicit
    disposition rather than an accidental omission.

Resource-specific validation, SDK calls, selectors, and API exceptions remain
typed adapter behavior. A runtime plugin system is not required.

Initial actionable milestone

  • Inventory current root planner ordering and scope predicates.
  • Replace duplicated root planner construction and dispatch with one runtime
    planning inventory, preserving order, HTTP log components, namespace error
    context, and organization assignment scope.
  • Retain public constructors and any test-facing compatibility entry points.
  • Couple a bounded resource migration to the shared reconciliation work.
  • Document precisely which inventories remain manual after this milestone;
    do not describe a planning-only registry as completion of this whole issue.

Subsequent milestones can move executor routing, static scope/parent metadata,
and loader enumeration onto the same capability boundaries, family by family.

Acceptance criteria

  • A kind with an established lifecycle can be introduced through a
    co-located definition and typed adapter without editing engine control
    flow in each pipeline stage.
  • Root and nested declaration locations have one authoritative definition.
  • Runtime registration detects missing or conflicting capabilities.
  • Namespace processing, plan order, and logging/error context are preserved.
  • Existing registry-driven explain/scaffold and load-schema behavior remain.
  • Migrated kinds are removed from superseded infrastructure inventories.
  • Remaining migration work is explicit, with no unused framework APIs.
  • All applicable unchanged validation gates pass.

Related work and sequencing

#1602 and #2102 implement the iterator portion of #1154. That older
ticket remains open for selector-only CLI emptiness semantics; reuse the
existing participant helper rather than repeating the iterator refactor. #1827 and #2051 address schema
and payload conformance and remain separate work.

Pair the first implementation milestone with shared lifecycle reconciliation.
Field contracts and compiled representations are later design tracks.

Compatibility and validation

This is a behavior-preserving refactor. Preserve accepted YAML, omitted versus
explicit-empty scope, namespace and protection behavior, identity precedence,
external resources, error propagation, secret redaction, and saved-plan
serialization. Preserve execution ordering and request counts where existing
tests specify them. Do not silently broaden support while moving code.

Existing test packages, test files, fixtures, mocks, and golden artifacts must
remain unchanged without explicit maintainer permission.
Establish the
unchanged test baseline before implementation. Keep test-facing entry points
as delegating adapters when needed. New regression or conformance tests are a
separate follow-up requiring permission; their desirability does not authorize
test changes in this refactor.

Run production-only modernization/formatting, a CGO-disabled build, lint, the
unchanged unit suite, and applicable existing integration checks after each
migration. Avoid repository-wide rewrite commands that can modify tests.
Record baseline failures separately from regressions, and verify test-file
hashes and the final Git diff.

Refactoring program

Start with bounded #2079/#2080 migrations. #2081 and #2082 remain later design
tracks; the initial implementation does not close the whole program.

Implementation guide acceptance criteria

Treat docs/contributor/declarative-resource-implementation-guide.md as the authoritative implementation entry point for agents throughout this refactoring. Guide maintenance is part of the implementation, not a documentation follow-up.

  • Document the root/runtime registration entry points, what registration derives automatically, and every integration inventory that remains manual after each migration.
  • Update the guide in the same change as the code; replace obsolete instructions and checklists instead of appending competing patterns or historical baseline sections.
  • Keep the workflow concise, retain behavioral invariants, and link to current implementations instead of copying large implementation templates.
  • Verify code references, local links, and section anchors. An agent should be able to follow a resource or field through loading, planning, execution, and export without guessing missing wiring.
  • Preserve maintainer constraints: additional coverage is authorized after
    the existing suite validates a refactoring; do not alter tests to make
    the initial refactor pass. Documentation does not authorize regeneration.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions