Add opt-in /health check for GPU collector presence

[gengwg]

Problem

A DCGM hostengine that starts before the NVIDIA driver is ready initialises without NVML. It listens normally, but reports no GPUs. An exporter that connects to it in that state finds nothing to collect and builds its registry with no GPU collector:

Attempting to initialize NVML library.
Initializing system entities of type 'GPU'
Not collecting GPU metrics; error retrieving DCGM MIG hierarchy: Cannot perform
  the requested operation because NVML doesn't exist on this system.
Registry built successfully  collector_count=1

A healthy start on the same node logs collector_count=2.

The registry is only rebuilt on reload or a GPU bind event, so from here the exporter serves an empty metrics page for the life of the process. /health returns 200 the whole time, because it only checks that the registry pointer is non-nil. The pod stays 1/1 Running, a liveness probe on /health never fires, and the node exports no GPU metrics until a human restarts it.

Fixing the hostengine underneath does not help: the exporter does not re-enumerate. I confirmed that on a live node — restarting the hostengine took dcgmi discovery -l from 0 GPUs found to 8 GPUs found, and metrics stayed absent until the exporter was restarted as well.

What this changes

Adds --health-require-gpus / DCGM_EXPORTER_HEALTH_REQUIRE_GPUS. When set, /health returns 503 if the registry holds no collector for dcgm.FE_GPU, so an existing httpGet liveness probe on /health restarts the pod and it re-enumerates on its own.

• Off by default. The exporter cannot tell "this node has no GPUs" from "the hostengine reported no GPUs", so the operator declares the intent. No existing deployment changes behaviour.
• Reload takes precedence. The registry is legitimately empty mid-reload; returning 503 there would restart the pod on every reload. Guarded by IsReloadInProgress().
• Registry.CollectorCount(entityGroup) reports the collector count for an entity group under the existing read lock.

Where this was seen

A production H100 cluster running gpu-operator. Nodes were being rebooted one at a time for unrelated maintenance while a gpu-operator chart upgrade was in flight. Nodes that rebooted before the upgrade came back sighted; after it, 8 of 9 came back blind and stayed blind, one for two days before anyone noticed. There was no signal in Kubernetes to find them by: every pod read 1/1 Running.

Relationship to gpu-operator#2855

NVIDIA/gpu-operator#2855 proposes an exec liveness probe on the nvidia-dcgm container so a deaf hostengine restarts itself. That covers the hostengine; this covers the exporter in front of it. Both are needed, because recovering the hostengine alone leaves the exporter holding its empty registry.

Doing the exporter half here rather than as a probe in gpu-operator is deliberate: the dcgm-exporter image ships no HTTP client (no curl, wget or nc), so an exec probe would have to hand-roll a request over bash /dev/tcp. The httpGet /health probe already exists in the gpu-operator asset and starts working once the endpoint reports the truth.

Testing

go test ./internal/pkg/server/... ./internal/pkg/registry/... ./pkg/cmd/... passes. Five new server cases cover: default off with no GPU collectors (200), required and absent (503 plus X-GPU-Collectors: 0), required with only non-GPU collectors — the incident shape — (503), required and present (200), and reload in progress with the flag on (200). A registry unit test pins CollectorCount to the entity group, and a CLI table test covers the flag, its explicit-false form and both env values.

I verified the tests fail when the production code is broken, rather than only that they pass:

• removing the 503 branch fails TestHealthReturnsUnavailableWhenGPUCollectorsRequiredButAbsent
• removing the reload guard fails TestHealthReturnsOKDuringReloadEvenWhenGPUCollectorsRequired

Alternatives considered

• Re-enumerate on hostengine reconnect, so no restart is needed at all. Cleaner, and it would make this flag unnecessary, but a much larger change; restart-to-recover is already the documented remediation.
• Make 503 the default. Rejected as a breaking change for any deployment where an exporter legitimately sees no GPUs.

---

Validation

• make check-format — clean.
• make test-main — all packages pass except internal/pkg/nvmlprovider and internal/pkg/integration_test, which fail identically on the unmodified base commit on this machine (no NVIDIA driver / NVML present). Verified by stashing the change and re-running those two packages.
• GPU, Docker and Kubernetes gates are skipped: no GPU or DCGM available here. Recording the skip per .codex/validation.md. The new behaviour is a registry-state predicate and is provable at package level, which is what the added tests do.

Mutation-checked rather than only run green — with CollectorCount changed to ignore its entity-group argument (return len(r.collectorGroups)), TestHealthReturnsUnavailableWhenOnlyNonGPUCollectorsRegistered and TestRegistryCollectorCountIsKeyedByEntityGroup both fail; removing the 503 branch or the reload guard each fail their own test.

Open question for maintainers: the predicate

CollectorCount(dcgm.FE_GPU) == 0 is a proxy for "no GPUs visible", and it is not exact in either direction:

• False 503: the GPU collector is skipped when the entity watch list has no device fields, so a counters file contributing no FE_GPU fields reports 503 on a node that does have GPUs. The default CSV ships all six DCGM_EXP_* commented out, so nothing else registers under FE_GPU in that case.
• False 200: an enabled DCGM_EXP_* counter registers under FE_GPU, so a node where the watch list exists but DCGM reports no GPUs would read healthy.

The sharper predicate is the topology the server already holds — the FE_GPU watch list's DeviceInfo().GPUCount(), the same accessor logTopologyInfo uses — which is zero in the first case and non-zero in the second.

I have not switched to it because I cannot verify here how GPUCount() behaves under MIG partitioning or -d i / -d g:0-3 device selection; if -d i yields a GPU watch list with no physical GPUs it would false-positive where the current predicate does not. Happy to switch if a maintainer can confirm that behaviour, or to leave it as-is given the flag is opt-in and both caveats are now documented beside it.