uniform_1d: cudaEventElapsedTime raises cudaErrorNotReady as an untyped RuntimeError

[rickpr]

Summary

A uniform_1d kernel launch raises a bare RuntimeError from its own event-timing call. The failure is not in the math — it is the elapsed-time query on a CUDA event that is not ready:

RuntimeError: CUDA error encountered at:
  file=/builds/clara-discovery/kernelcatcher/cuequivariance_ops/cuda/equivariance/uniform1d/../common/kernel.cuh
  line=887
  call='cudaEventElapsedTime(&elapsed_ms, start, stop)'
  Reason=cudaErrorNotReady: device not ready
  ...
  #5 kernelcatcher::equivariance::uniform_1d::run_uniform_1d_cuda(...)

cudaErrorNotReady from cudaEventElapsedTime means the recorded work had not completed when the elapsed time was queried. Per the CUDA docs, cudaEventElapsedTime returns cudaErrorNotReady if either event has not yet been completed — the call requires a preceding synchronisation on the end event, which does not appear to be happening here.

Impact on callers

It surfaces as a plain RuntimeError, not a torch.AcceleratorError, so nothing downstream can classify it as a CUDA fault. In our case it aborted a 9h40m training run at roughly epoch 60. Callers that want to retry or retire a worker on CUDA faults have to string-match the Reason=cudaErrorXxx field out of the message, which is fragile across releases.

Two asks, either of which resolves this

1. Synchronize (or remove) the timing call so it cannot raise. If the timing is diagnostic only, gating it behind an opt-in would also work.
2. If it can raise, raise something typed that callers can classify programmatically, rather than a bare RuntimeError whose only machine-readable signal is a substring.

Ask 2 is useful even if ask 1 lands, since it applies to every other check_cuda-style site in the ops layer.

Environment

cuequivariance-ops-torch-cu13	0.10.0
cuequivariance-torch / cuequivariance	0.9.1
torch	2.13.0+cu130
GPU	H100 80GB HBM3
Platform	Linux x86_64

Not yet retested on cuequivariance-ops-torch-cu13 0.11.1; we will report back if it reproduces there.

Reproduction

We have no minimal reproducer — this is a race, and we have seen it once. Reported because the failure is cheap for you to rule out at the call site and expensive for callers to diagnose.

Shape of the workload: a MACE-style equivariant model built on cuet.SegmentedPolynomial(..., method="uniform_1d") and cuet.SymmetricContraction(..., method="uniform_1d"), evaluated as roughly 1,430 independent single-GPU forward passes per epoch across a long multi-epoch run. One pass out of that population raised; the surrounding passes on other GPUs were fine.

That frequency is consistent with a missing synchronisation that is normally masked by the work happening to have completed by the time the timing query runs.

Related

Same class of defect as NVIDIA/warp#1896 ("Raise on array_scan device failures"), which we reported and which NVIDIA merged: a CUDA failure inside a library that the caller cannot see or classify. That one was silent corruption; this one is an untyped raise. Both come down to the caller not being given a usable signal.

Question

The CUDA sources for cuequivariance_ops are not in this repository (the wheels are prebuilt), so we cannot open a PR against kernel.cuh the way we did for warp. If a patch would be useful and there is a path for contributing to the ops tree, we are happy to write it — please let us know.