Skip to content

Add a K8s readiness check for device pods. - #747

Open
kraney wants to merge 11 commits into
openconfig:mainfrom
kraney:cisco-readiness
Open

Add a K8s readiness check for device pods.#747
kraney wants to merge 11 commits into
openconfig:mainfrom
kraney:cisco-readiness

Conversation

@kraney

@kraney kraney commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Cisco pods may have a running process, but fail to come to "Router up" state. Prior to this change, KNE just hangs forever waiting for Router up even for permanent failures.

This resolves this in two ways (belt and suspenders.) First, for Cisco pods we add the check for "Router up" as a readiness check. That way, a failure ends up reflected in K8s' reported pod status. The pod will not transition to Ready state until that message is logged. And if it doesn't show up within the threshold, the pod will report as failed.

Second, KNE explicitly checks for failure in the logs and treats it as such, rather than just waiting indefinitely. KNE may notice first, since the failure message will likely arrive before the readiness check exceeds its threshold.

Similar weaknesses exist broadly across vendors - KNE typically waits for Running, not Ready. This means it waits for the internal process to start, not for it to be ready - and many of the device types take an appreciable time to actually be ready. The pods themselves appear to be running even if configuration has failed.

This sets up readiness checks broadly, so that startup failures are reported as startup failures and on a paricular single pod, rather than being silently ignored and letting the test fail at a later stage, often based on a timeout trying to connect to gnmi, or something similar.

Cisco pods may have a running process, but fail to come to "Router up"
state. Prior to this change, KNE just hangs forever waiting for Router
up even for permanent failures.

This resolves this in two ways (belt and suspenders.) First, for
Cisco pods we add the check for "Router up" as a readiness check.
That way, a failure ends up reflected in K8s' reported pod status.
The pod will not transition to Ready state until that message is logged.
And if it doesn't show up within the threshold, the pod will report as
failed.

Second, KNE explicitly checks for failure in the logs and treats it
as such, rather than just waiting indefinitely. KNE may notice first,
since the failure message will likely arrive before the readiness check
exceeds its threshold.
The log check isn't working. This is a more typical approach for K8s.
@kraney kraney changed the title Add a K8s readiness check for Cisco pods. Add a K8s readiness check for device pods. Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant