Add a K8s readiness check for device pods. - #747
Open
kraney wants to merge 11 commits into
Open
Conversation
Cisco pods may have a running process, but fail to come to "Router up" state. Prior to this change, KNE just hangs forever waiting for Router up even for permanent failures. This resolves this in two ways (belt and suspenders.) First, for Cisco pods we add the check for "Router up" as a readiness check. That way, a failure ends up reflected in K8s' reported pod status. The pod will not transition to Ready state until that message is logged. And if it doesn't show up within the threshold, the pod will report as failed. Second, KNE explicitly checks for failure in the logs and treats it as such, rather than just waiting indefinitely. KNE may notice first, since the failure message will likely arrive before the readiness check exceeds its threshold.
The log check isn't working. This is a more typical approach for K8s.
kraney
force-pushed
the
cisco-readiness
branch
from
September 3, 2026 16:43
298d529 to
6279b5a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cisco pods may have a running process, but fail to come to "Router up" state. Prior to this change, KNE just hangs forever waiting for Router up even for permanent failures.
This resolves this in two ways (belt and suspenders.) First, for Cisco pods we add the check for "Router up" as a readiness check. That way, a failure ends up reflected in K8s' reported pod status. The pod will not transition to Ready state until that message is logged. And if it doesn't show up within the threshold, the pod will report as failed.
Second, KNE explicitly checks for failure in the logs and treats it as such, rather than just waiting indefinitely. KNE may notice first, since the failure message will likely arrive before the readiness check exceeds its threshold.
Similar weaknesses exist broadly across vendors - KNE typically waits for Running, not Ready. This means it waits for the internal process to start, not for it to be ready - and many of the device types take an appreciable time to actually be ready. The pods themselves appear to be running even if configuration has failed.
This sets up readiness checks broadly, so that startup failures are reported as startup failures and on a paricular single pod, rather than being silently ignored and letting the test fail at a later stage, often based on a timeout trying to connect to gnmi, or something similar.