fix: qualify beta setup, Copilot transport, witness and upgrade safety together

[Pal Lakatos-Toth (pallakatos)]

Summary

Recover the public-beta setup defects and qualify the requested follow-ups as one coherent candidate:

• Verify supported AKS private-root Pod admission against both a constrained derivation and strict server-side dry-run, preserving the original root template, retirement binding, identities and publication checks.
• Serialize Copilot device polling, honor cumulative GitHub slowdown, preserve expiry, handle cancellation and transport/protocol failures, and preflight the legitimate credential store before token exchange. Unresolved storage outcomes are explicit, not authorization success.
• Show the current managed-MCP controller explanation in the actual capabilities page. Stale status and older BFFs cannot claim verified readiness. Document the namespace-local pull-credential requirement that blocked Playwright.
• Keep an otherwise exact, witnessed Recreate restoration pending while its rollout revision propagates, even if Kubernetes has already observed the generation. Completion still requires the exact next revision and all existing authority/retirement evidence.
• Publish the requested public-beta Helm installation guide, with honest installation failures and upgrade limitations.
• Constrain existing Copilot credential transport to fixed HTTPS endpoints with no redirects, preserving credential authority and explicit outcomes.
• Recognize exact non-root QoS admission using the existing root eligibility rule; retain only fixed boolean comparisons on rejected admission so the native failure can be diagnosed without exposing values.
• Refuse unsafe qualified-root controller upgrades before mutation while preserving read-only schema checks and original recovery.
• Include the optional default-off witness from feat: add optional operator-owned datapath witness Helm switch #568, its independently closed report-consistency/licensing repairs, and the bounded governed-service rollout prerequisite.

The current installed cluster remains on public 3bc7ff58; these changes have not been deployed. Target is kars-bridge, not main.

#568 is retained as the original witness review/evidence record; its corrected source is now included here so the final fixes are qualified together rather than against incompatible intermediate bases. It will be closed as superseded only after the combined source is successfully merged.

Current candidate qualification

Current public candidate: a79b5438feb03b32f11f6d6ac939d685f37d4345.
All follow-up source reviews are closed, including the root-selection repair
and its direct-owner safety correction, and the test-only fixed-localhost
fixture routing. The actual native and CodeQL alert verdicts must be established
again on this head; neither is inferred from source review.

Fresh runs: core,
native are in progress.
CodeQL passed all three
analysis jobs, and its separate alert verdict
also passed on this exact head. The remaining alert 833 now reports fixed on
the PR merge ref, with no manual dismissal or configuration suppression.
Bridge passed.
Actual BFF logs confirm all sixteen Copilot tests, including the new
fixed-localhost/pinned-socket network regression, executed successfully.
BFF totals are 284 unit tests plus four and two additional groups.
Repository/source-audit gates,
Dependency Review and
Secret Scanning passed.
Analysis-job success was not substituted for the separate alert verdict.

The final local join passed 736 CLI tests across 23 files, types/build/focused
lint and fifteen native diagnostic tests. All 58 exact Rust registration
guards and the witness parity step remain, including sixteen Copilot tests.
The final native lifecycle and complete core workflow still require acceptance.
No earlier run was cancelled to publish this update.

Retained e6908eb5 qualification

Core CI
completed successfully, including real Kind, chaos and Criterion gates.
Native qualification
failed the observer preview; all three cold-install API lanes passed. The new
diagnostic shows a non-root consumer with all twelve normalized execution
sections matching, yet it was sent through root-only admission verification.
The unconditional Workload Identity opt-in selector is the reproduced defect:
local regressions fail in both same-root-namespace and separate-namespace cases.
A narrow pinned-root selection repair is now in the current candidate.
Independent review caught a direct Pod-to-root-Deployment shortcut in the
initial follow-up; it was repaired and re-reviewed before publication.
The existing verifier still rejects that unsupported lineage, and root WI
admission remains mandatory. This is not yet a successful native outcome.

All three CodeQL analysis jobs
completed successfully, but the separate CodeQL alert verdict
failed with one high cleartext-transmission alert at copilot_transport.rs:65.
The three original alerts (830-832) now report fixed on this PR's merge ref.
The implementation investigation traces the remaining alert to test-only
loopback URL construction rather than a production credential interpolation.
A fixed-localhost URL/socket-routing separation is now independently
source-reviewed and included in the current candidate. It still requires fresh
hosted execution and alert analysis. Successful
analysis jobs are not being substituted for a passing verdict; no alert is waived.
Bridge CI passed all
11 jobs. Actual logs confirm all 15 Copilot and eight witness Rust regressions
executed successfully; BFF totals were 283 unit tests plus four and two further
test groups, with locked Clippy passing. Locked web production image/startup,
the pinned witness image and real guarded add-on Helm lifecycle also passed.
Repository/source-audit gates,
Dependency Review
and Secret Scanning
have passed. This is not a final merge verdict.

All bounded source reviews are now closed, including the final value-free admission diagnostics. Separate audit records disclose exact source ranges, independent AI contexts, findings/repairs, maintainer delegation and execution limits; no broad independent assembly review or second human reviewer is implied.

The initial combined candidate preserved exact implementation slices and 57 regression-registration guards plus parity. Its combined local results were 733 CLI tests across 23 files, CLI types/build/focused lint; 94 web tests, full web types/focused lint; 13 producer, seven chart, three parity, twelve governed-readiness and fifteen native-diagnostic tests, plus Rust formatting. The follow-up expands the CLI evidence to 736 tests and registration to 58 as recorded above. Caches remain provisional and no local Rust execution is claimed.

Fresh final-head core, Bridge, native and CodeQL qualification is required. Earlier runs below are retained evidence, not substitutes for that result. Native diagnosis is still open: PodTolerationRestriction is not default-enabled on Kubernetes 1.31, so the QoS reproduction is not asserted as the proven cause of the earlier Kind failure.

Review and qualification

Independent, bounded AI source reviews covered the AKS repair, MCP/Copilot contracts and composition, and the later writer-settlement change. Two Copilot review findings were fixed and re-reviewed: uncertain in-flight expiry and nested/unregistered Rust tests. The existing BFF inventory gate now requires both formerly nested tests by exact name.

New audit records disclose actual participants, source bounds, the maintainer's explicit delegation, unresolved deployment limitations and unexecuted checks. They do not imply a second human review or waive technical gates.

Actually executed locally:

• 233 targeted CLI tests across writer settling, private continuity, AKS admission and credential-grant commands; CLI types/build/focused lint.
• 90 combined web tests, full TypeScript check and focused lint.
• Rust formatting and fresh audit/whitespace checks.
• Two writer-settlement regression cases failed before the production fix and passed afterward.

Local dependency caches are provisional (YAML 2.9.0 versus locked 2.8.3; Next 16.2.9 versus locked 16.3.3). They are not substitutes for hosted qualification.

Combined-head evidence

On exact 8952689bf571d1334f0b0c6d053fbcd936f1daa5, Bridge CI 34990200325 passed all 11 jobs. All eight Copilot Rust tests actually registered and executed, including both formerly nested cases; BFF Clippy passed. The locked Next 16.3.3 production image compiled and started successfully.

Core CI 34990200422 completed successfully, including Kind, chaos and benchmark gates. Native qualification 34990200439 failed at the initial controller preview; the later writer/observer cases did not execute. Its three cold-install API lanes passed. CodeQL separately reported three high cleartext-transmission alerts in Copilot login/provider routes; these are being addressed and are not waived. No repair deployment has occurred.

The investigation confirmed that the Kind fixture inherited Azure Workload Identity opt-in despite installing no Azure identity webhook. Four-file test-harness follow-up 616cbdb1 explicitly selects the non-Azure setting and retains only closed admission-stage diagnostics. Production CLI/admission checks, AKS values and chart defaults are unchanged. Two regressions failed before that follow-up; all 18 targeted fixture/diagnostic tests and actual fixture Helm rendering passed afterward.

Follow-up candidate 3d1ea2ba996d534c5f3cf19cf9ec477e6be6b736 adds only that harness correction and the verified live MCP recovery record. It is published on isolated qualify/bridge-beta-3d1ea2ba; native qualification 34992902236 failed later, at the private-observer preview with the closed root-pod-admission-identity diagnostic. The initial controller preview and intervening lifecycle cases passed. That isolated run preserved the useful old-head core run rather than cancelling it; the PR remained at 8952689b until that core run completed. These are historical outcomes, not final-head qualification.

The subsequent HTTPS/no-redirect transport, exact non-root QoS comparison and value-free diagnostic repairs have completed their bounded independent source reviews and are included in the current composition. The QoS change reproduces the same identity-error category with a reviewed non-root Pod; five regressions failed before repair and 399 targeted CLI tests passed afterward. The retained native artifact lacks the full rejected Pod spec, so this is not asserted as the proven native cause or a successful native repair yet.

Verified live MCP recovery

On the unchanged installed public 3bc7ff58, only the original approved, configured managed-MCP pull credential was restored to its correctly claimed namespace, using strict server dry-run and CREATE-only semantics. The existing controller reconciled Playwright to current-generation Ready and discovered 24 tools; its actual Pod is Ready on the pinned image. Parent read-only follow-up independently confirmed those observations and unchanged protected model, alias and cluster namespace identities. No controller/ServiceAccount/MCP spec or manual readiness override was needed. General node image-pull access and the other installation blockers remain separate.

After that recovery, a separate explicit operator request removed GPT-OSS through AI Runway to free the H100. Normal provider cleanup removed only the model workloads; the node remained Ready with zero GPU memory use and no compute processes. Local GPT-OSS inference is now intentionally unavailable. This later operation does not change the earlier MCP recovery evidence or constitute deployment of this PR.

Retained earlier-head evidence

• MCP-only a9a35bba: Bridge component run 34983443655 passed, including real BFF and locked web production execution.
• Its core run 34983443516 also passed, including the existing Kind, chaos and benchmark jobs; the run was allowed to finish before this PR update.
• The same source's native run 34983443416 failed the private-observer writer-transition case; the three cold-install API lanes passed. The failing runtime result is preserved, not relabeled as success.
• Native booleans show a rollout-revision mismatch, but do not identify its exact old/unexpected value. The later fix addresses a reproduced legitimate Kubernetes ordering; it is not yet proof that this failed native execution is repaired.

No earlier-head success qualifies the new combined source.

Explicitly not claimed or included

• No complete authenticated Home-to-Team/recurring-repository acceptance claim.
• No qualified-root template-upgrade mechanism. Changing a sealed controller image/environment or using rollout restart remains blocked pending the reviewed migration path tracked in Support qualified-root upgrades without stranding private credential continuity #567.
• No arbitrary credential copying, private-metadata reset, grant recreation, or readiness patch.
• No GPU, AKS/node-pool or CNI changes from this PR. The separately authorized model-only removal is documented above.
• No live optional eBPF witness installation or legacy observer adoption. The operator-owned feature is included here but remains default-off; successful kernel capture is not claimed.
• No inference-budget/TLS, reproducible node image-pull or seccomp qualification based merely on Helm success.

Merge and matching-source image deployment remain conditional on actual final-head qualification and the separately documented operational prerequisites.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details
pip/PyYAML	6.0.3	Unknown	Unknown

Scanned Files

• deploy/ebpf-witness/tests/requirements.txt