Missing DRA metrics in case of dynamically created MIG devices

[guptaNswati]

Description

dcgm-exporter:4.6.0-4.8.3 cannot attribute metrics to pods using Dynamic MIG devices allocated by dra-driver-nvidia-gpu.

In case of Dynamically created MIG devices, NVIDIA DRA ResourceSlices only contain an abstract device (which has ParentUUID and Profile), not the live MIG UUID and GI created during device preparation. The exporter currently requires that MIG UUID to resolve the GI key and therefore drops the mapping. https://github.com/NVIDIA/dcgm-exporter/blob/main/internal/pkg/transformation/kubernetes.go#L758

I think we missed addressing it properly earlier #664 (comment)

kubectl logs "$DCGM_POD" -n nvidia-gpu-operator --since=5m -c nvidia-dcgm-exporter | grep DRA
time=2026-07-23T18:57:37.605Z level=DEBUG msg="Skipping DRA MIG mapping without MIG UUID" parentUUID=GPU-a15aaed3-d4e7-dde0-adcd-4432a8caf752
time=2026-07-23T18:57:37.605Z level=DEBUG msg="Completed toDeviceToPodsDRA transformation" totalMappings=0 deviceToPodsMap=map[]
time=2026-07-23T18:57:37.606Z level=DEBUG msg="Device to pod mapping for DRA: map[]"

An example ResourceSlice entry of an abstract mig device that can be created when a claim requests it:

  "name": "gpu-0-mig-1g12gb-19-0",
  "attributes": {
    "uuid": null,
    "parentUUID": {
      "string": "GPU-a15aaed3-d4e7-dde0-adcd-4432a8caf752"
    },
    "profile": {
      "string": "1g.12gb"
    },
    "type": {
      "string": "mig"
    }
  },
  "capacity": {
    "memory": {
      "value": "11904Mi"
    }
  },
  "consumesCounters": [
    {
      "counterSet": "gpu-0-counter-set",
      "counters": {
        "copy-engines": {
          "value": "1"
        },
        "decoders": {
          "value": "1"
        },
        "encoders": {
          "value": "0"
        },
        "jpeg-engines": {
          "value": "1"
        },
        "memory": {
          "value": "11904Mi"
        },
        "memory-slice-0": {
          "value": "1"
        },
        "multiprocessors": {
          "value": "16"
        },
        "ofa-engines": {
          "value": "0"
        }
      }
    }
  ]
}

Claim with allocated device

status:
  allocation:
    allocationTimestamp: "2026-07-23T18:55:23Z"
    devices:
      results:
      - device: gpu-0-mig-1g12gb-19-0
        driver: gpu.nvidia.com
        pool: lego-cg1-qct-046
        request: mig
    nodeSelector:
      nodeSelectorTerms:
      - matchFields:
        - key: metadata.name
          operator: In
          values:
          - lego-cg1-qct-046
  reservedFor:
  - name: dcgm-dynamic-mig

The ResourceSlice intentionally has no live MIG UUID or GI ID. These identities exist only after the dra plugin creates the partition during ResourceClaim preparation.

The ResourceClaim also has no concrete identity, just the device name instatus.devices.

For Static MIG, exporter maps MIG UUID → parent UUID + GI ID + CI ID via GetMIGDeviceInfoByID()
but for Dynamic MIG, we need a new mapping something like: parent UUID + profile ID + placement → live GI ID

Using only parentUUID + profile would not be safe because multiple live instances with the same profile may exist at different placements.

Potential fixes:
Option 1 (DCGM only change) Parse device-name:

• Parse profile and placement start from the allocated DRA device name. The device name encodes:
  gpu-0-mig-1g12gb-19-0
  └─ placement start: 0
  └──── profile ID: 19
  https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/blob/main/cmd/gpu-kubelet-plugin/mig.go#L113
• Enumerate live MIG devices under parentUUID through NVML.
• Inspect each device's GI profile ID and placement.
• Match exactly parentUUID + profileID + placementStart.
• Use the resulting runtime GI ID with the existing GetGPUInstanceIdentifier() path.

No change is needed for Static MIG where if MIG UUID present use GetMIGDeviceInfoByID()

Any issue on relying on parsing the device-name?

Option 2 (DRA change): Should we add placementStart in the ResourceSlice attributes as we already have the profile. With profileID + placementStart, dcgm wont have to parse the device-name.

Option 3 (DRA change): Concrete identity in ResourceClaim device status
After creating the MIG device, the DRA driver could populate ResourceClaim.status.devices[].data with MIGUUID, GI or CI. This may require additional cleanup on the driver side during tear down.

[guptaNswati]

Since there are a few different ways to close the Dynamic MIG mapping gap, let's first walk through how the current per-pod → DRA → DCGM metric labeling actually works.

DCGM already knows every GPU and MIG device on the node by its NVML UUID — that's where its raw metrics come from. What it does not know on its own is which pod currently owns each device. That's the job of the PodResources + DRA translation layer inside dcgm-exporter.

There are two inputs:

Input 1 — Kubelet PodResources gRPC (local, on-node)
For every pod running on this node, PodResources returns the DRA resources allocated to it as a list of (driver, pool, device) tuples https://github.com/NVIDIA/dcgm-exporter/blob/main/internal/pkg/transformation/kubernetes.go#L850:

drInfo := DynamicResourceInfo{
    ClaimName:      dr.GetClaimName(),
    ClaimNamespace: dr.GetClaimNamespace(),
    DriverName:     draDriverName,
    PoolName:       draPoolName,
    DeviceName:     draDeviceName,
}

Input 2 — ResourceSlice informer (cluster, K8s API)
The exporter filters and only watches the ResourceSlices published by NVIDIA DRA ("gpu.nvidia.com") driver. Using the (pool, device) tuple from Input 1, the exporter looks up the matching device in the ResourceSlice and gets four attributes — type, uuid, parentUUID, profile:

func buildDeviceMapping(deviceType, uuid, parentUUID, profile string) (string, *DRAMigDeviceInfo) {
    switch deviceType {
    case "gpu":
        // Full GPU: the slot's UUID *is* the concrete GPU UUID.
        return uuid, nil
    case "mig":
        // MIG device: we know the parent GPU and the profile string.
        return parentUUID, &DRAMigDeviceInfo{
            MIGDeviceUUID: uuid,
            Profile:       profile,
            ParentUUID:    parentUUID,
        }
    }
}

That resolved UUID is then attached as a label to DCGM's per-device metrics streaming from NVML — so a metric like DCGM_FI_DEV_GPU_UTIL{UUID="GPU-abc…"} gets enriched with pod, namespace, container and DRAinfo.

Some more details to already listed options:

Option A — Parse the canonical device name in the exporter
The (pool, device) tuple the exporter already gets from PodResources contains everything: the device string encodes parentMinor, profileID, and placementStart by construction. Reuse the driver's regex in the exporter.

Why this is safe today, grounded in the driver:

The name schema is not a convention or a "helpful hint" — it's constructed by MigSpecTuple.ToCanonicalName and parsed back by NewMigSpecTupleFromCanonicalName
The driver itself relies on the round-trip in unpreparePartiallyPrepairedClaim — the reclaim-on-partial-prepare path has no other source of truth for (parentMinor, profileID, placementStart) than the device name string.
Any future rename would already require the driver to migrate its own reclaim logic. An external consumer parsing the same schema does not add a new constraint on the driver; it depends on a constraint the driver already carries.

Pros:
Zero driver change. The information is already in the wire format the exporter receives from PodResources.
No ResourceSlice attribute additions, no status.devices[].data schema, no informer/watch change on the exporter side.

Cons:

• The parse contract is not "documented as public API". May be expose NewMigSpecTupleFromCanonicalName as an importable api that exporter imports internally and use.
• Still requires the NVML side-channel to resolve (parentMinor, profileID, placementStart) → MIG UUID — same as Option A. No help for future per-allocation fields (sharing / time-slicing / IMEX).
• Full-GPU names (gpu-) don't yield a UUID by parse — for those the exporter still needs uuid from the ResourceSlice attributes (which is fine; no change from today).

Option B — Add placementInfo (Start + Size) attributes to the ResourceSlice
Keep the ResourceSlice as the exporter's single source of truth. Add Placement info to each Dynamic MIG device entry, alongside parentUUID and profile which exporter can use to resolve the concrete MIG UUID with a local NVML lookup similar to what it does for Static MIG.

Pros: additive driver change; no new watch on the exporter side; no timing coupling to NodePrepareResources.

Cons: the exporter grows an NVML side; every future per-allocation field (sharing group, time-slice partition) may also need new attributes meaning this wont work out of box for different types and sharing modes.

Option C — Populate ResourceClaim.status.devices[].data with concrete identity
Move concrete post-prepare identity out of the ResourceSlice (a static catalog) into ResourceClaim.status (per-allocation, post-prepare). The driver writes it in Prepare() once the MIG device actually exists.

The exporter switches from watching ResourceSlice to watching ResourceClaim and reads the concrete identity directly for all device types.

devices:
    - conditions:
      - lastTransitionTime: "2025-11-13T22:22:56Z"
        message: ""
        reason: GPUDeviceReady
        status: "True"
        type: Ready
      data:
        driverVersion:
          version: 1.0.0
        uuid:
          string: gpu-18db0e85-99e9-c746-8531-ffeb86328b39
      device: gpu-0

Example implementation is here kubernetes-sigs/dra-example-driver#128

Pros: concrete identity lives where it belongs (post-prepare, per-allocation); direct correlation, no NVML side-channel; naturally handles MIG destroy+recreate on the same claim slot; future-proof for sharing / time-slicing / IMEX per-allocation state.

Cons: two-sided contract (driver publishes, exporter consumes) so both sides must ship compatible versions; exporter changes the object it watches; must handle "data not yet populated" (older driver, prepare in flight).

[guptaNswati]

Implementing Option A as NVIDIA DRA driver also relies on it for cleanup.