[Story]: Instrument I/O concurrency and partition-sizing signals across cudf-polars, rapidsmpf, kvikio

[Matt711]

Problem

Tuning cudf-polars' I/O and memory-admission settings today is mostly guesswork. max_concurrent_io_tasks (#23552) and target_partition_size (#23549) are what prompted this, but the same blind spot applies to num_py_executors, kvikio_nthreads, pinned memory, and any future auto-tuning work. We audited cudf-polars, rapidsmpf, and kvikio and found the same pattern in all three: the raw signal usually already exists somewhere in the C++ layer, but is discarded, folded into an aggregate that's too coarse, or never wired up. Most of this list is "turn on a light that's already wired," not new sensor design.

Goal: give sweeps, benchmark runs, and future tuning work a way to tell what's actually causing a result, instead of guessing.

Related

• [PERF]: Remove max_io_threads from the public API and derive it automatically #23552 max_concurrent_io_tasks default
• [PERF]: Remove 1.5 GB cap from default target partition size #23549 target_partition_size default
• Develop an Nsight System plugin to profile network bandwidth rapidsai/kvikio#999, kvikio_nic: an nsys plugin that samples per-interface NIC bandwidth and places it on the nsys timeline alongside CUDA/NVTX traces.
• Add an I/O observation facility rapidsai/kvikio#1033, Add an I/O observation facility, and Add I/O statistics: totals of what a run read and wrote rapidsai/kvikio#1036, Add I/O statistics: totals of what a run read and wrote: the Observation/Summary/SummaryMonitor machinery the kvikio sub-issues in this story build on.
• Add Quent Resource and IR-node tracing to cudf-polars #23179 and Use Quent bindings in cudf-polars #22949: an active Quent tracing effort adding resource-level instrumentation (CPU, device memory, filesystem memory, network) to cudf-polars. The signals this story adds may be useful to surface in Quent.
• Report kvikio I/O statistics per rank in the streaming engines #23738, Report kvikio I/O statistics per rank in the streaming engines: recent cudf-polars work surfacing kvikio's Summary per rank, the current baseline the kvikio sub-issues in this story go beyond.
• Introduce Statistics::record_copy and consolidate copy/spill statistics rapidsai/rapidsmpf#875, Introduce Statistics::record_copy and consolidate copy/spill statistics, and Stream ordered timings rapidsai/rapidsmpf#882, Stream ordered timings: the most recent rapidsmpf statistics work, and the direct predecessor of Wire real spill instrumentation into SpillManager rapidsai/rapidsmpf#1166.
• cudf-polars 1k ndsh runs show malformed spill stats rapidsai/rapidsmpf#869 (open), cudf-polars 1k ndsh runs show malformed spill stats: a live report of the same gap Wire real spill instrumentation into SpillManager rapidsai/rapidsmpf#1166 is about.
• [Discussion] Implement backpressure when allocating buffers for incoming payloads on a rank rapidsai/rapidsmpf#1155 (open), [Discussion] Implement backpressure when allocating buffers for incoming payloads on a rank: related to Instrument the reservation admission / forced-progress path rapidsai/rapidsmpf#1165, though scoped to the shuffler specifically rather than I/O reads generally.