fix https://github.com/NVIDIA/TensorRT-Edge-LLM/issues/204: pass the CUDA include path to the XQA NVRTC JIT - #205
Conversation
buildNvrtcOptions() passed no -I, on the grounds that every header reaches nvrtcCreateProgram as a virtual include. That holds for the kernel's own headers, but mha.cu includes cuda_fp16.h, which includes vector_types.h from the CUDA toolkit. NVRTC on CUDA 12.x for x86 does not carry that as a builtin, so every model whose decoder takes the XQA JIT path fails to load: cuda_fp16.h(129): catastrophic error: cannot open source file "vector_types.h" It surfaces during ONNX parsing as "Could not create the plugin", which points away from the cause. Pass one include path, taken from CUDA_DIR, which CMake already resolves for this build. EDGELLM_NVRTC_INCLUDE overrides it at runtime for installs relocated away from the toolkit they were built against, and the conventional install path is the fallback when the definition is absent. The option builder is shared by every SM, so this is not architecture specific. Verified on x86-64 with CUDA 12.8 against an SM89 GPU: the XQA kernel compiles in about a second, and a Cosmos3-Edge INT4-AWQ checkpoint then runs end to end with the environment variable unset, so the build-system path is the only source of the include directory. Fixes NVIDIA#204 Signed-off-by: Filipe Martins <293984334+filipemartinsubrobotics@users.noreply.github.com>
sylvesterkaczmarek
left a comment
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
The embedded include path comes from ${CUDA_DIR}/include, but FindNVRTC.cmake explicitly supports selecting NVRTC through NVRTC_ROOT_DIR or CUDA_TARGET_DIR and resolves the actual headers as NVRTC_INCLUDE_DIR. In those supported configurations this can link against one NVRTC toolkit while the runtime JIT searches another header tree. Could the definition use the resolved NVRTC_INCLUDE_DIR (or equivalent resolved CUDA include path) and cover a non-default NVRTC root?
…UDA_DIR Review on NVIDIA#205: FindNVRTC.cmake supports selecting NVRTC through NVRTC_ROOT_DIR or CUDA_TARGET_DIR, so baking ${CUDA_DIR}/include could link one toolkit's NVRTC while the runtime JIT searched another's headers. Resolving NVRTC_INCLUDE_DIR instead would not be right either: that is found by locating nvrtc.h, and a standalone NVRTC package ships nvrtc.h alone. The nvidia-cuda-nvrtc wheel's include/ holds exactly one header, so pointing NVRTC_ROOT_DIR at it and baking that directory would hand the JIT a tree with neither cuda_fp16.h nor vector_types.h — the failure this series fixes. So resolve the header that is actually included, across the roots FindNVRTC already honours and in the same precedence. A non-default NVRTC root wins when it can serve the compile and is declined when it cannot, with a warning and the CUDA_DIR fallback rather than a silent mismatch. Checked against four real configurations on one box: an nvidia-cu13 wheel with a full header set resolves to itself; NVRTC_ROOT_DIR=/usr with apt CUDA 11.5 there resolves to /usr/include; CUDA_TARGET_DIR at a second toolkit resolves to that toolkit; and the nvidia-cuda-nvrtc wheel falls back to CUDA_DIR with a warning. A cross build still bakes a path correct on the build host, which may not be where the toolkit sits on the target. EDGELLM_NVRTC_INCLUDE covers that, and a comment now says so. Signed-off-by: Filipe Martins <293984334+filipemartinsubrobotics@users.noreply.github.com>
|
You are right that the two can diverge. I have pushed a change, though not quite the suggested one.
So the definition now resolves the header that is actually included, searching the roots find_path(
EDGELLM_JIT_CUDA_INCLUDE_DIR vector_types.h
HINTS ${NVRTC_INCLUDE_DIR} ${NVRTC_ROOT_DIR}/include ${CUDA_TARGET_DIR}/include
${CUDA_DIR}/include
NO_DEFAULT_PATH)A non-default NVRTC root wins when it has the headers and is skipped when it does not, with a warning and the
One case the definition cannot cover on its own is the case when a cross build bakes a path correct on the build host, and the toolkit may sit elsewhere on the target. That is what Rebuilt and re-verified on SM89 with the environment variable unset, so the baked path is the only source: the XQA kernel compiles and the Cosmos3-Edge INT4-AWQ checkpoint runs end to end. |
sylvesterkaczmarek
left a comment
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Rechecked 4e5edfff. The JIT include path now resolves the runtime header tree through the same supported NVRTC/CUDA roots instead of assuming CUDA_DIR, and the fallback/override behavior covers standalone NVRTC and cross-target layouts. This resolves my earlier concern.
2 participants
Addresses the first of the four findings in #204. The other three, and the
support-matrix question, are untouched by this PR and should outlive it — hence a
plain reference rather than a closing keyword.
Why
buildNvrtcOptions()passes no-Ito NVRTC, on the stated grounds that every header reachesnvrtcCreateProgramas a virtual include. That holds for the kernel's own headers, butmha.cuincludes
cuda_fp16.h, which includesvector_types.hfrom the CUDA toolkit. NVRTC on CUDA 12.xfor x86 does not carry that as a builtin, so the XQA kernel never compiles:
Every model whose decoder takes the XQA JIT path then fails to load, and it surfaces during ONNX
parsing as
Could not create the plugin, which points away from the cause. The option builder isshared by every SM, so this is not architecture specific.
What
One include path, taken from
CUDA_DIR, whichCMakeLists.txtalready resolves for the build.It reaches the source as a
PRIVATEcompile definition onedgellmPluginJit, the target thatcompiles
decoderXQAJitCompiler.cpp.EDGELLM_NVRTC_INCLUDEoverrides it at runtime for installsrelocated away from the toolkit they were built against, and the conventional install path is the
fallback when the definition is absent, so the translation unit still compiles in any build that
does not set it.
Impact and risk
No behaviour change for builds that already work: the option is additive, and an include path that
resolves to the toolkit the build was configured against cannot shadow a virtual include, since
NVRTC resolves those first. No API or ABI change. No new dependency. The only build-system change
is one
target_compile_definitionscall.If you would rather the path came from somewhere else,
CUDAToolkit_INCLUDE_DIRSor a cachevariable, say so and I will reshape it.
Verification
x86-64 Ubuntu 22.04 under WSL2, CUDA 12.8, TensorRT 11.2.1.2, GeForce RTX 4060 Ti (SM89),
configured natively with both CuTe DSL groups built for
sm_89.llm_inferencefails at plugin creation with the error above.EDGELLM_NVRTC_INCLUDEunset, so the compile definition is the only source ofthe path: the SM89 XQA kernel compiles, and a Cosmos3-Edge INT4-AWQ checkpoint runs end to end
against
examples/multimodal/pics/red_panda.jpegwith a correct description, 94.4 tokens/sgenerating.
git-clang-formatandcmake-formatboth report clean on the change.Related
Issue #204 also lists three other things found on the same build, which are not touched here: the
CUDA 12.8 floor,
ENABLE_CUTE_DSL=OFFfailing well downstream rather than at configure time, andENABLE_CUTE_DSLsilently rejecting a comma-separated list. Happy to send separate PRs for any ofthem.
Found during the NVIDIA / OpenHackathons / Oracle Open Models Codefest 2026,
while bringing up an INT4-AWQ Cosmos3-Edge reasoner on a DGX B300 for an
offline-first search-and-rescue robotics entry (Team UBR Stack). The production
target is a Jetson Orin Nano; the B300 is a bench machine used for evaluation.