dcgm-exporter 4.4.2 memory leak causing periodic OOM kills

[inyohh]

What is the version?

4.4.2-4.7.1-ubuntu22.04

What happened?

OOM Killed

Summary

dcgm-exporter 4.4.2 has a memory leak that causes the container to be OOM-killed periodically. The process memory (anon-rss) grows slowly until it hits the cgroup memory limit, triggering an OOM kill and container restart. This has been observed across multiple nodes in our cluster, with very consistent memory usage at the time of the kill (~517MB against a 512Mi limit).

Symptoms

• dcgm-exporter pods restart periodically due to OOM kills
• Memory usage grows steadily from ~100MB to the 512Mi limit over ~24-48 hours
• All OOM-killed processes show very similar memory at time of kill (anon-rss ~517MB)

OOM Kill Evidence

Kernel logs from multiple nodes show consistent behavior:

Memory cgroup out of memory: Killed process 3833171 (dcgm-exporter) total-vm:4478432kB, anon-rss:517656kB, file-rss:47148kB, shmem-rss:0kB, UID:0 pgtables:1636kB oom_score_adj:-997

Observed OOM kills across nodes:

Node	OOM Time	anon-rss at kill	Container Restarts
idc-z790-ud-ax-2	2026-06-28 05:45	518MB	4
idc-z790-ud-ax-3	2026-06-28 16:50	517MB	2
idc-z790-ud-ax-5	2026-06-26 14:51	518MB	1
idc-z790-ud-ax-7	2026-06-28 23:19	516MB	6
idc-z790-ud-ax-8	2026-06-27 13:32	517MB	1

Resource Configuration

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 200m
    memory: 512Mi

Current Workaround

Increasing the memory limit to 1Gi reduces the frequency of OOM kills but does not fix the underlying leak. The memory still grows, just takes longer to hit the limit.

Actual Behavior

anon-rss grows steadily from ~100MB (initial) to 512MB (limit) over 24-48 hours, causing periodic OOM kills and container restarts.

What did you expect to happen?

Expected Behavior

dcgm-exporter should maintain stable memory usage during normal operation. The anon-rss should not continuously grow over time.

What is the GPU model?

NVIDIA GeForce RTX 4090 (24GB)

What is the environment?

Environment

• Kubernetes version: v1.33.0
• OS: Ubuntu 24.04.3 LTS (kernel 6.14.0-37-generic)
• GPU nodes: NVIDIA GeForce RTX 4090 (24GB)
• Driver: 580.126.09
• CUDA: 13.0
• MIG mode: N/A
• Container runtime: containerd 2.1.5
• Helm chart: dcgm-exporter
• Image: nvcr.io/nvidia/k8s/dcgm-exporter:4.4.2-4.7.1-ubuntu22.04

How did you deploy the dcgm-exporter and what is the configuration?

Helm Chart

How to reproduce the issue?

How to reproduce

1. Deploy dcgm-exporter as a DaemonSet with the default resource limits (memory: 512Mi):

resources:
  limits:
    cpu: 200m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 128Mi

2. Use the default counters configuration:

args:
  - "-f"
  - "/etc/dcgm-exporter/default-counters.csv"

3. Deploy on nodes with NVIDIA GPUs (we observed this on RTX 4090 and RTX A2000).

4. Monitor the container's memory usage (anon-rss) over time. It will grow steadily from ~100MB to the 512Mi limit within 24-48 hours, regardless of GPU workload (our GPUs were mostly idle, 0% utilization).

5. Once the memory reaches the cgroup limit, the kernel OOM killer terminates the process:

Memory cgroup out of memory: Killed process 3833171 (dcgm-exporter) total-vm:4478432kB, anon-rss:517656kB

6. Kubelet restarts the container, and the cycle repeats.

Anything else we need to know?

No response

[nccurry]

This should be fixed in the next release 4.6.0-4.5.3

[inyohh]

This should be fixed in the next release 4.6.0-4.5.3

May I ask when it is expected to be released?

[jaeeyoungkim]

this pr fix issue
@nccurry @inyohh
#626