read_chunk (streaming/actor_graph/io.py) already logs start/admitted/stop timestamps, estimated_output_bytes, and reservation_bytes per I/O task. Missing one field: actual_output_bytes. Needed to calibrate target_partition_size against real outcomes and to check the fixed 2x memory-reservation multiplier against real usage. The value is already computed once the chunk is materialized, this is just plumbing it into the existing log call.
Longer term, #20780 (cudf::memory_resources, separating output allocations from temporary/intermediate ones in libcudf APIs) would let us measure the temporary allocation a parquet read actually uses instead of assuming the fixed 2x, once the IO module is migrated to it.
read_chunk(streaming/actor_graph/io.py) already logsstart/admitted/stoptimestamps,estimated_output_bytes, andreservation_bytesper I/O task. Missing one field:actual_output_bytes. Needed to calibratetarget_partition_sizeagainst real outcomes and to check the fixed2xmemory-reservation multiplier against real usage. The value is already computed once the chunk is materialized, this is just plumbing it into the existing log call.Longer term, #20780 (
cudf::memory_resources, separating output allocations from temporary/intermediate ones in libcudf APIs) would let us measure the temporary allocation a parquet read actually uses instead of assuming the fixed2x, once the IO module is migrated to it.