[CUDA] Allow selecting nvcc's host compiler, and invalidate on change

[GoodOlClint]

Proposed changes

A follow-up to #413 ("Allowing SPM to compile on Linux with CUDA"): make nvcc's host compiler selectable, and make --incremental notice when it changes.

The problem

Plugins/CudaBuild/plugin.swift resolves the host compiler as context.tool(named: "clang++") — the Swift toolchain's own clang — and passes it to encuda as --clangpp, which encuda turns into -ccbin= on the nvcc command line. There is no way to point it elsewhere:

• CudaBuild.json has no key for it (CUDA_ARCH is the plugin's only environment knob).
• NVCC_PREPEND_FLAGS cannot win, because -ccbin= arrives later on the command line. Only NVCC_APPEND_FLAGS beats it, which is not a reasonable thing to ask of a consumer.

When the toolchain's clang falls outside the range the installed CUDA release accepts, nvcc refuses to compile anything:

crt/host_config.h:151:2: error: -- unsupported clang version! clang version must be
less than 21 and greater than 3.2

This is reachable with current releases: Swift 6.3.3 ships clang 21.0.0, and CUDA 13.0 requires clang < 21. It is specific to that toolkit version — CUDA 13.1 raises the ceiling to < 22 and accepts clang 21 — but 13.0 is what a current CUDA 13 install may well be, and the only escape today is to not use Swift 6.3.

The change

Resolve the host compiler in priority order:

1. MLX_CUDA_HOST_COMPILER environment variable
2. a new hostCompiler key in CudaBuild.json
3. the toolchain's clang++, exactly as before

so an unconfigured build is unchanged.

It must still be a clang: nvcc preprocesses with the host compiler and SwiftPM then compiles the resulting .cpp, so pointing -ccbin at g++ produces a translation unit full of GCC-only _Float32 declarations that clang rejects. That constraint is noted at the resolver rather than left to be rediscovered.

Why the invalidation half is in the same PR

encuda's --incremental check compares only input/output modification times, which cannot observe a changed compiler. Shipping an overridable host compiler without fixing that would be actively unsafe: switching host compilers silently reuses .cpp files preprocessed by the previous one, and the failure surfaces as a flood of errors from glibc headers with nothing pointing at the real cause. (That is exactly how this was found.)

So the compilation configuration — nvcc path, -ccbin, -std, CUDA_ARCH, include roots — is stamped in a <output>.encuda-stamp beside each generated .cpp, and a changed signature counts as out of date. It also fixes stale output after an arch or include-path change, which had the same blind spot.

Verification

On macOS (Xcode 26.6, Swift 6.3.3) — inert, since the resolver runs only after the plugin's isCudaEnabled() guard:

• pre-commit run --all-files — clean.
• scripts/verify-docs.sh — passes.
• xcodebuild build-for-testing -scheme mlx-swift-Package -destination 'platform=macOS' — builds; CmlxTests and MLXTests pass.

On a real CUDA device (DGX Spark / GB10, sm_121, CUDA 13.0.88, Swift 6.3.3, aarch64, Ubuntu 24.04):

• swift build completed cleanly in 466.89s with MLX_CUDA_HOST_COMPILER=/usr/bin/clang++-18 and no NVCC_APPEND_FLAGS and no -Xcxx overrides — where before this change the build could not be completed on that toolkit/toolchain pair at all.
• All 95 .cu files recompiled and 95 .encuda-stamp files were written, each recording ccbin=/usr/bin/clang++-18 — i.e. the override is what nvcc actually used, not a stale artifact being reused.
• Decoded Llama-3.2-1B-Instruct-4bit at 196.4 tok/s and gemma-4-12B-it-4bit at 23.0 tok/s, matching the throughput of an ad-hoc flag-pile workaround it replaces (194–195 / 22.9 tok/s), with identical output.

Note CI does not cover the SwiftPM CUDA lane — linux_build_cmake_cuda builds via CMake, on cuda-12.9 with swift-6.2.3 — so this permutation (CUDA 13.0 + Swift 6.3) is untested there.

Related

Two other follow-ups to #413 are open alongside this one, and all three touch Plugins/CudaBuild/plugin.swift. They are functionally independent, but whichever merges first will leave the others needing a trivial rebase — happy to reorder or stack them however you prefer:

• [CUDA] Build the CUDA argument lists in steps #442 — building the CUDA argument lists in steps (readability / type-check cost).
• [CUDA] Make the CUDA include roots configurable #444 — making the CUDA include roots configurable (CCCL / cudnn-frontend / CUTLASS).

Checklist

Put an x in the boxes that apply.

• I have read the CONTRIBUTING document
• I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
• I have added tests that prove my fix is effective or that my feature works — no test added: both halves are build-plugin behaviour that only exists in a CUDA-enabled build, which has no CI lane here (the CUDA job builds via CMake and does not use this plugin). Evidenced on-device instead, per the stamp contents above. Glad to add a test if you can point me at the right harness for plugin behaviour.
• I have updated the necessary documentation (if needed) — the two new knobs are documented at the resolver; happy to add them to a README section if you'd prefer them surfaced there.

[davidkoski]

Change looks good to me. Thank you!

[davidkoski]

Conflict needs resolving (vs other PR). It looks like it may collide with #444 as well.