Skip to content

Add optional cgroup-direct process attribution for non-MIG shared GPUs #725

Description

@SMS-Derfflinger

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

Please provide a clear description of the problem this feature solves

#594 added per-process metrics for shared GPUs by selecting candidate pods from kubelet PodResources and then mapping NVML PIDs to those pods through cgroups.
Our some non-MIG MPS device plugins expose logical device IDs such as:

GPU-6a16b9c2-...::5
GPU-6a16b9c2-...::19

Our shared-GPU device plugin exposes 48 logical slots per physical GPU. A workload pod usually requests 6 or 7 slots, while its processes run on one physical GPU.
These logical assignments are allocation records and may not reliably describe runtime physical GPU ownership. They can also cause the same pod-level FB_USED value to be emitted once per logical slot:

DCGM_FI_DEV_FB_USED{UUID="GPU-6a16b9c2-...",pod_uid="1843d150-...",vgpu="5",...} 10056
DCGM_FI_DEV_FB_USED{UUID="GPU-6a16b9c2-...",pod_uid="1843d150-...",vgpu="19",...} 10056

This request only covers non-MIG DCGM_FI_DEV_FB_USED.

Feature Description

As an operator using a MPS(Multi-Process Service) shared-GPU device plugin, I want an optional process attribution mode based directly on NVML PIDs and cgroups, so that pod-level GPU memory reflects actual runtime ownership instead of logical slot assignments.

Describe your ideal solution

Add an optional cgroup-direct attribution mode:

physical GPU UUID -> NVML PID -> /proc/<pid>/cgroup -> pod UID

Aggregate DCGM_FI_DEV_FB_USED by:

physical GPU UUID + pod UID

The expected pod-level output is:

DCGM_FI_DEV_FB_USED{UUID="GPU-6a16b9c2-...",pod_uid="1843d150-...",pod="workload-1",namespace="default",...} 10056

PodResources and the Kubernetes informer should not determine physical GPU ownership in this mode. The existing PodResources-based mode should remain the default. Non-FB_USED metrics should remain unchanged.

Additional context

Tested with:

dcgm-exporter: 4.5.3-4.8.2
GPU: NVIDIA RTX 6000D
Driver: 580.126.09
Environment: Kubernetes with an MPS shared-GPU device plugin

This is related to #587 and #594. I can submit a pull request if this approach is acceptable.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions