XQA NVRTC JIT passes no CUDA include path: cuda_fp16.h cannot find vector_types.h on x86 CUDA 12.x

[filipemartinsubrobotics]

Summary

On an x86-64 Linux build, every model whose decoder uses the XQA JIT path fails to load, because
the runtime NVRTC compile cannot find the CUDA toolkit headers. The failure surfaces during ONNX
parsing as a plugin creation error, which points away from the actual cause.

buildNvrtcOptions() in cpp/kernels/decodeAttentionKernels/decoderXQAJitCompiler.cpp deliberately
passes no -I:

// No -I flags needed: all headers are passed as virtual includes to nvrtcCreateProgram.

That holds for the project's own headers, but the kernel includes cuda_fp16.h, which in turn
includes vector_types.h from the toolkit. NVRTC on CUDA 12.x for x86 does not carry that as a
builtin, so the compile dies.

This is not architecture-specific. The option builder is the same for every SM, so any x86
CUDA 12.x build reaching this path is affected, including SM80 and SM86 on the listed developer
tier.

Reproduction

Building examples/llm/llm_build against a Cosmos3-Edge INT4-AWQ ONNX export:

[ERROR] [attentionPlugin.cpp:2183:createPlugin] Failed to create AttentionPlugin:
        Failed to NVRTC compile XQA kernel for SM89, dtype=1, kv_dtype=1, head_dim=128,
        q_heads_per_kv=2, tokens_per_page=128, sliding_window=0, spec_decode=0
cuda_fp16.h(129): catastrophic error: cannot open source file "vector_types.h"
  #include "vector_types.h"
                           ^
1 catastrophic error detected in the compilation of "mha.cu".

[ERROR] ModelImporter.cpp:150: ERROR: onnxOpImporters.cpp:8706 In function addPluginWithCreator:
[8] Assertion failed: plugin: Could not create the plugin.

Environment:

• tag: v0.10.1 (e8b2952)
• host: x86-64, Ubuntu 22.04 on WSL2
• CUDA: 12.8 toolkit
• TensorRT: 11.2.1.2
• GPU: GeForce RTX 4060 Ti, SM89

Configured natively for x86 (no toolchain file, no EMBEDDED_TARGET), with TRT_PACKAGE_DIR
pointing at a TensorRT tree and -DCMAKE_CUDA_COMPILER pinned to the 12.8 nvcc.

Suggested fix

Pass one include path. The build system already knows it: CUDA_DIR is resolved in
CMakeLists.txt, so it can be baked in as a compile definition, with an environment variable kept
as an override for relocatable installs. Happy to open a PR with that shape if you would like it,
or with whatever mechanism you prefer.

We are running locally with a ten-line version of this and the kernel then compiles for SM89 in
about a second.

Three related things found on the same build, reported here rather than as separate issues

Each cost an hour, and each presents as something other than what it is. Happy to split them out
if you would rather track them separately.

1. CUDA 12.8 is a hard floor, not a default. The generated CuTe DSL headers use cudaLibrary_t,
   which does not exist before 12.8. With 12.5 the whole project builds and both engines build; the
   failure appears only when a kernel group that needs those headers is enabled, as
   'cudaLibrary_t' does not name a type. set_ifndef(CUDA_CTK_VERSION 12.8) reads like a
   preference rather than a requirement.

2. The decoder builds and runs with ENABLE_CUTE_DSL=OFF, which looks like success. Two things
   then fail well downstream: visual_build cannot build the vision encoder, because
   ViTAttentionPlugin has only CuTe DSL backends and reports "Cannot implement the
   ViTAttentionPlugin configuration" without mentioning CuTe DSL; and inference fails inside a
   plugin with a bare pluginUtils::isSuccess(status) assertion, because
   int4GroupwiseGemmPluginV2 also has CuTe DSL backends. A note in the docs that a vision-language
   model needs fmha and int4_fp16_gemm would have saved both.

3. ENABLE_CUTE_DSL silently rejects a comma-separated list. -DENABLE_CUTE_DSL=fmha,int4_fp16_gemm
   configures and then reports "matched no groups in ...", which reads as a missing artifact. It is
   a CMake list, so it needs a semicolon. The help string says "semicolon-separated group list", but
   the failure message does not say that is the problem.

Would SM89 be worth adding to the support matrix?

The x86-64 developer row lists SM80, SM86, SM100 and SM120. SM89 is absent, which reads as
unsupported, yet CMAKE_CUDA_ARCHITECTURES already defaults to 80;86;89;90 on x86, the FMHA
cubin list includes 89, the XQA JIT names it, and TensorRT ships an SM89 builder resource.

With the include-path fix and both CuTe DSL groups built for sm_89, a Cosmos3-Edge INT4-AWQ
checkpoint runs end to end on a GeForce RTX 4060 Ti: decoder engine 840 MiB built in 23 s, vision
encoder 942 MiB, 94.4 tokens/s generating and 68.9 end to end including prefill, 494 image tokens
per frame, correct output on examples/multimodal/pics/red_panda.jpeg.

Happy to share the full recipe if useful.