[Feature]: Configurable max retry/timeout for validators with automatic node tainting on failure

[dezmodue]

Problem

Several validators in nvidia-validator (e.g. the driver validator, and any other component run with WITH_WAIT=true) retry indefinitely on failure with a fixed sleep interval (SLEEP_INTERVAL_SECONDS, default 5s).

This becomes especially problematic during cluster autoscaling. If a newly scaled-up node fails installation for any reason or happens to land on an instances that is malfunctioning, the GPU Operator validator will simply retry forever. The node never becomes "Ready" for GPU workloads, but nothing signals back to the cluster that this node is unusable. As a result:

• The node sits around consuming resources.
• The cluster autoscaler doesn't realize the node is broken and won't replace it until it reaches its timeout.

Feature Request

The proposed approach is two fold, first is to define either a configurable max retries or a time limit for validation, second signal the cluster autoscaler (or Karpenter) that this node is to be considered out of service and it should immediately spin up a new one. Something like:

1. Add a configurable maximum retry count and/or maximum wait duration for validation loops (e.g. via a MAX_RETRIES / MAX_WAIT_SECONDS environment variable on the validator, configurable through the ClusterPolicy CR).
2. When the configured maximum is reached without a successful validation, the GPU Operator should taint the node.
   • Suggested taint: status-taint.cluster-autoscaler.kubernetes.io/validation=failed
   • This is a taint that Cluster Autoscaler already recognizes as marking a node as unavailable/broken, so it can trigger scale down + replacement of the node without requiring any custom logic.
3. This behavior should obviously be opt-in/configurable so existing clusters relying on indefinite retries are not unexpectedly affected (e.g. a ClusterPolicy field such as validator.failurePolicy.maxRetries, validator.failurePolicy.maxWaitSeconds, and validator.failurePolicy.taintOnFailure).
4. The operator has already access to taint nodes so this approach would add no extra "permissions"

Motivation

This affects autoscaling reliability for GPU node pools. Without this, a single misconfigured/broken node can silently stall autoscaling for a long time, since neither the operator nor the autoscaler has visibility into a "stuck" validation state.

We would like to contribute this feature if the approach sounds reasonable, thanks