Skip to content

Clear CUDA errors once reported and raise eval errors in buffer protocol - #4523

Open
aleroot wants to merge 1 commit into
ml-explore:mainfrom
aleroot:sticky-error_abort
Open

aleroot wants to merge 1 commit into
ml-explore:mainfrom
aleroot:sticky-error_abort

Conversation

@aleroot

@aleroot aleroot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Investigating the CI build failure I realized that : a CUDA error that MLX has already reported stays set on the calling thread, and Event::wait() re reads it with cudaPeekAtLastError().
After one failed allocation every later eval on that thread throws cudaPeekAtLastError() failed: out of memory, even though nothing is wrong. When the read comes through the buffer protocol the exception escapes the extern "C" getbuffer slot and the process aborts with exit 134.
This is what took down the Linux (cuda-*, x86_64) jobs after an OOM in test_fast_sdpa. see: #4522

  • ☑️ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants