[Interactive Drive] Add Vulkan Support

[ArielG-NV]

The new Vulkan backend (via very non-rigorous testing) is significantly faster than CUDA (hits lows of 2ms, highs of ~10ms; CUDA has lows of around 10ms)

The design was rushed on my end, so I do not have a proper PR description to provide accompanying the code added.

At a high-level the goal was: minimize syncs (versus last-time); minimize use external memory for a 'zero copy' abstraction over CUDA; do not recreate buffers every time we enter vk-ludus. It seems these design changes had massive perf-benefits when put together.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

The PR restores an optional Vulkan rendering backend for Interactive Drive while retaining CUDA as the default renderer.

• Adds Vulkan/CUDA external-memory and timeline-semaphore interop with reusable output and staging slots.
• Adds Vulkan task, mesh, fragment, and export-compute shaders plus extension build support.
• Exposes Vulkan renderer selection through the Ludus API, Interactive Drive configuration, CLI, manifests, and documentation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported output-slot stream-lifetime and staging-dimension issues are addressed at the current head.

Important Files Changed

Filename	Overview
integrations/omnidreams/ludus-renderer/ludus_renderer/_cpp/bindings/torch_rasterize_vk.cpp	Adds PyTorch Vulkan bindings and now safely quarantines returned export slots until a device-wide drain permits reuse.
integrations/omnidreams/ludus-renderer/ludus_renderer/_cpp/render/ludus_timestamped_vk.cpp	Implements the timestamped Vulkan renderer, per-slot staging metadata, synchronization, resource management, and CUDA handoff.
integrations/omnidreams/ludus-renderer/ludus_renderer/_cpp/common/vkutil.cpp	Implements Vulkan device setup, CUDA device pairing, external resource import, and timeline-semaphore utilities.
integrations/omnidreams/ludus-renderer/ludus_renderer/_ops/context_vk.py	Adds the Python Vulkan timestamped context and packs render inputs onto the configured CUDA device.
integrations/omnidreams/omnidreams/interactive_drive/rasterizer.py	Integrates backend selection into Interactive Drive rasterizer construction.

Sequence Diagram

sequenceDiagram
  participant Python as Interactive Drive
  participant CUDA as CUDA stream
  participant Vulkan as Vulkan renderer
  participant Slot as Export slot
  Python->>CUDA: Upload queries and camera poses
  CUDA->>Vulkan: Signal interop timeline
  Vulkan->>Vulkan: Rasterize and compute-export RGBA
  Vulkan->>CUDA: Signal completion timeline
  CUDA->>Slot: Map exported linear buffer
  Slot-->>Python: Return PyTorch tensor
  Python->>Slot: Drop final tensor reference
  Slot->>Slot: Mark PendingRelease
  Python->>CUDA: Request another frame under pool pressure
  CUDA->>CUDA: Synchronize consumer streams
  CUDA->>Slot: Reclaim slot

Loading

_{Reviews (7): Last reviewed commit: "more robustness" | Re-trigger Greptile}

[ArielG-NV]

Note it looks like there is a race-condition in the code (rotating buffers without checking 'frame-inflight'), this needs to be solved before code is RFR

[ArielG-NV]

/ok to test ce5b00a

[ArielG-NV]

/ok to test 6602176

[ArielG-NV]

/ok to test 7a72bf0