Find more `rapids doctor` checks from other RAPIDS projects

[jacobtomlinson]

Now that we have a framework for running smoke tests we should start locating existing code in other RAPIDS projects that would make a useful check. We should avoid writing code from scratch where possible and reuse existing benchmarks, smoke tests, unit tests, etc.

[jayavenkatesh19]

I have been looking into this and there are a couple of design decisions that need to be made. Looking at RAPIDS libraries like cuDF and cuML, the repositories themselves have unit tests and existing benchmarks that we can use as a RAPIDS doctor check.

However, these tests are not packaged with the library wheels, so that would mean the only way to use them in rapids doctor is to make a copy of the relevant tests into the checks folder and add an entrypoint for these checks to be discoverable. Which then brings up the possibility of these tests being out of date with changes upstream in the RAPIDS libraries and the need for constant maintenance for these checks.

Another aspect to think about is that some developers might not install the entire RAPIDS suite (by building their own custom Docker image for example). In that scenario, having checks across RAPIDS might cause some to fail due to the libraries not being present. I wonder if we can default to a subset of these checks and add a filtering option for other checks based on user requirements.

[jacobtomlinson]

Which then brings up the possibility of these tests being out of date with changes upstream in the RAPIDS libraries and the need for constant maintenance for these checks.

Yeah I want to avoid this. I wonder if @bdice has any thoughts on code that does get included in the cudf wheel that we could use to exercise as a smoke test here?

having checks across RAPIDS might cause some to fail due to the libraries not being present

The design of the checks entrypoints means that each project registers their own checks. So any combination of RAPIDS packages can be installed and RAPIDS doctor will just gather and run the checks that are available. So you don't need to worry about this.

[betatim]

If you use entrypoints to discover checks, then it is up to each package to implement the checks no? This avoids having to copy code to this repository, means the checks match the installed version (they come from that version) and the checks are specially made for the purpose of rapids doctor.

In particular that latter part is important I think. Running unit tests from cuml (because I know it) seems like a weird and imprecise way of determining that "cuml works properly" in an amount of time that a user would want to wait. At least my expectation as a user of rapids doctor would be to get an answer within a few seconds. Maybe with the option of getting more detailed feedback that I can share with a developer, but takes maybe a minute or two to generate.

If you assume people have several RAPIDS libraries installed, having to wait "a few seconds" for each one can quickly add up.

Is that about right? Totally missing the mark?

[jacobtomlinson]

If you use entrypoints to discover checks, then it is up to each package to implement the checks no?

Yeah exactly, each check should live in the repo it belongs to.

I don't think we should run full unit tests, but some tiny smoke tests to verify imports. We can contribute these tests, but it's best done in collaboration with each project and reusing existing code if it's available.

There's nothing stopping us parallelising checks, so I'm not concerned about it adding up. I would expect each check to be milliseconds rather than seconds though, they are just smoke tests.

[ncclementi]

I don't think we should run full unit tests, but some tiny smoke tests to verify imports. We can contribute these tests, but it's best done in collaboration with each project and reusing existing code if it's available.

+1 to these. But I think the question remains of what kind of test do we think would be valuable to check. I personally don't have a mental model, of what are we looking to check here. Any thoughts?

[jacobtomlinson]

Ideally I want to defer to each project on what they feel would be valuable smoke tests.

At a bare minimum I would want to check for three things:

• Can we import the library?
• Can we allocate GPU memory?
• Can we execute a CUDA kernel?

Sometimes it might make sense to roll up multiple things into a single check, but also it would be nice to have the granularity so that if a step fails it's clear in the output what failed.

So for cudf I think something like this exercises all of those things.

def cudf_import_check():
    import cudf

def cudf_allocate_and_compute_check():
    import cudf
    gdf = cudf.DataFrame({"values": [10, 20, 30, 40, 50, 60]})
    gdf["values"].mean()

[betatim]

NVIDIA/cuml#7314 is the kind of thing that the doctor could check for and then give a sane error message to the user (and maybe even a suggestion on how to fix it).

The problem in the issue is, I think, that there was some CUDA stuff installed already on the system, that install is outside the reach of the package manager the user used to install something, and the versions of the "CUDA stuff" and what the user asked to install don't match. The resulting error message is hard to understand for the human.

The package manager can't really help the human because the problem is stuff beyond what it manages.

The human can't really fix it because they just ran a command that worked (the install command) and now things are broken. Potentially even worse is that this exact command used to work. Making the connection to an action a few days ago, if they themselves upgraded CUDA, is hard. Most likely a sys admin upgraded the CUDA stuff, or a newer version of the OS uses a different CUDA version, etc.

TL;DR: the doctor can help spot mismatching CUDA versions and tell the user

[jacobtomlinson]

I agree this would be an excellent check to add. I expect with the release of CUDA 13 we will see a lot of 12/13 mismatched environments for a while.

[ncclementi]

xref some PR that have been added towards this issues:

• cuml: Implement health checks. NVIDIA/cuml#7854
• cudf: Add rapids-doctor basic health checks NVIDIA/cudf#21928
• nvlink related: update nvlink checks #143
• cuda-toolkit related: Add CUDA toolkit check to rapids doctor #141