Right now there's no way to tell, per I/O task, whether it was admitted immediately or blocked waiting on reserve_memory (and for how long), versus some other cause of delay. Add that as a field alongside the "IO Task" log entry from #23811: something like a backpressure_reason (memory_reserve, channel_full, or similar) and a wait duration, recorded at the reserve_memory call sites (streaming/actor_graph/io.py, streaming/actor_graph/join.py).
This is the cudf-polars-side complement to rapidsai/rapidsmpf#1165. Without it, a task waiting on memory_reserve looks identical to a task waiting on the network, which is exactly the ambiguity that makes latency-only tuning unreliable.
Right now there's no way to tell, per I/O task, whether it was admitted immediately or blocked waiting on
reserve_memory(and for how long), versus some other cause of delay. Add that as a field alongside the"IO Task"log entry from #23811: something like abackpressure_reason(memory_reserve,channel_full, or similar) and a wait duration, recorded at thereserve_memorycall sites (streaming/actor_graph/io.py,streaming/actor_graph/join.py).This is the cudf-polars-side complement to rapidsai/rapidsmpf#1165. Without it, a task waiting on
memory_reservelooks identical to a task waiting on the network, which is exactly the ambiguity that makes latency-only tuning unreliable.