Fix Kubernetes locker ownership with client-go Lease election - #983
Fix Kubernetes locker ownership with client-go Lease election#983pood1e wants to merge 2 commits into
Conversation
|
Local runtime checks passed for
The stop check uses removal of active targets from the runtime API and unchanged receive counters, since UP gauges update asynchronously. Blocked API requests and release I/O are covered by the regression tests in this PR. These observations describe the exercised cases, not an exactly-once guarantee. |
|
Additional local integration checks passed for target ownership reconciliation, subscription recovery and Remote Write with this PR combined with #980. The PR's local unit and race checks also passed. GitHub Actions requiring maintainer approval remain pending; these results describe local validation. |
|
Final combined deployment validation is now complete for this Lease lifecycle change together with #980:
The PR is mergeable and all GitHub checks are green. @karimra, could you review this together with #980 when available? |
Fixes #982.
The Kubernetes locker rejects valid runtime target identities and can delay collection shutdown after Lease expiry when coordination API requests block. This replaces the custom acquisition/renewal loop with client-go leader election and LeaseLock, adds a renewal deadline and reports ownership loss through the collector's stop/retry path.
Lease names use a SHA-256 digest of the original key; annotations preserve the key and instance name. Each acquisition has a unique holder identity. Release checks ownership, uses UID/resourceVersion preconditions and has a timeout. The collector cancels the subscription context before cleanup and performs release I/O outside its operational mutex. A shared Lease informer serves ownership queries; QPS/burst settings make the API request budget explicit.
Upgrade: stop members using the previous Kubernetes locker before upgrading because Lease key encoding changes. Replace
renew-period/retry-timerwithrenew-deadline/retry-period, and grant Leasewatchpermission. Configuration and migration guidance are indocs/user_guide/ha_kubernetes_locker.md.Local validation at
2ba66e42ce82979aab28f3266add4122fb1687a4:./tests/run_tests.sh: passed.go test -race ./pkg/lockers/k8s_locker ./pkg/app: passed.b796d09d.Pod replacement and failed-Lease-write recovery were also exercised with #980. GitHub Actions requiring maintainer approval remain pending. #980 handles EndpointSlice discovery independently.