Skip to content

Fix Kubernetes locker ownership with client-go Lease election - #983

Open
pood1e wants to merge 2 commits into
openconfig:mainfrom
pood1e:fix/k8s-lease-lifecycle
Open

Fix Kubernetes locker ownership with client-go Lease election#983
pood1e wants to merge 2 commits into
openconfig:mainfrom
pood1e:fix/k8s-lease-lifecycle

Conversation

@pood1e

@pood1e pood1e commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Fixes #982.

The Kubernetes locker rejects valid runtime target identities and can delay collection shutdown after Lease expiry when coordination API requests block. This replaces the custom acquisition/renewal loop with client-go leader election and LeaseLock, adds a renewal deadline and reports ownership loss through the collector's stop/retry path.

Lease names use a SHA-256 digest of the original key; annotations preserve the key and instance name. Each acquisition has a unique holder identity. Release checks ownership, uses UID/resourceVersion preconditions and has a timeout. The collector cancels the subscription context before cleanup and performs release I/O outside its operational mutex. A shared Lease informer serves ownership queries; QPS/burst settings make the API request budget explicit.

Upgrade: stop members using the previous Kubernetes locker before upgrading because Lease key encoding changes. Replace renew-period/retry-timer with renew-deadline/retry-period, and grant Lease watch permission. Configuration and migration guidance are in docs/user_guide/ha_kubernetes_locker.md.

Local validation at 2ba66e42ce82979aab28f3266add4122fb1687a4:

  • ./tests/run_tests.sh: passed.
  • go test -race ./pkg/lockers/k8s_locker ./pkg/app: passed.
  • Regression tests cover identity constraints, original metadata, cached ownership changes, renewal deadlines, blocked API calls, reacquisition, cancellation and safe release.
  • Blocked-release regressions fail before the app fix and pass afterward.
  • Staticcheck: no new findings against base b796d09d.

Pod replacement and failed-Lease-write recovery were also exercised with #980. GitHub Actions requiring maintainer approval remain pending. #980 handles EndpointSlice discovery independently.

@pood1e

pood1e commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Local runtime checks passed for 2ba66e42ce82979aab28f3266add4122fb1687a4, combined with the EndpointSlice implementation from #980 at 92a5d658fa2cdb4b418e5e2ab0bdbaf7e67bdd0b:

  • Target identities are preserved when mapped to Lease names.
  • Pod replacement recovers active target ownership and fresh Subscribe responses.
  • Failed Lease writes cause collection to stop; collection recovers after write access is restored.
  • Sampled active owners agree with Lease annotations at converged checkpoints.

The stop check uses removal of active targets from the runtime API and unchanged receive counters, since UP gauges update asynchronously. Blocked API requests and release I/O are covered by the regression tests in this PR. These observations describe the exercised cases, not an exactly-once guarantee.

@pood1e

pood1e commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Additional local integration checks passed for target ownership reconciliation, subscription recovery and Remote Write with this PR combined with #980. The PR's local unit and race checks also passed.

GitHub Actions requiring maintainer approval remain pending; these results describe local validation.

@pood1e

pood1e commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Final combined deployment validation is now complete for this Lease lifecycle change together with #980:

  • Kubernetes v1.36.2+k3s1, 3 nodes, 6 gNMIc replicas distributed 2/2/2.
  • 641 target Leases and 4,487 subscriptions remained healthy through a 7,204-second steady-state run.
  • Active owners remained unique; Remote Write failures, dropped messages, subscription failures, and container restarts had zero increase.
  • Dedicated member/leader replacement recovery passed (11.458s and 25.639s business-sample recovery, both below the existing 60s limit).
  • Lease write denial still stopped active collection and recovery restored all owners in the earlier fault test; the final run completed with no cleanup errors.

The PR is mergeable and all GitHub checks are green.

@karimra, could you review this together with #980 when available?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kubernetes locker cannot safely maintain target ownership

1 participant