Scan/IO task assignment is fully static. scan_node (streaming/actor_graph/io.py) builds a fixed producer_id = task_idx % num_producers assignment once at graph-build time, no shared queue, no work-stealing. If one producer draws a large or slow file, other producers that finish early can't help. Log when a producer finishes its assigned list while others are still working. Quantifies the imbalance concretely and is useful evidence for (or against) a future queue-based scheduler change.
The actual shape of this metric depends on whatever queue-based scheduler redesign eventually replaces the static assignment, so hold off on locking in its exact form until that's picked.
On its own this metric can't say why a producer was slow. See #23814, a producer stuck waiting for a free thread there looks identical to one stuck on a genuinely large file.
Scan/IO task assignment is fully static.
scan_node(streaming/actor_graph/io.py) builds a fixedproducer_id = task_idx % num_producersassignment once at graph-build time, no shared queue, no work-stealing. If one producer draws a large or slow file, other producers that finish early can't help. Log when a producer finishes its assigned list while others are still working. Quantifies the imbalance concretely and is useful evidence for (or against) a future queue-based scheduler change.The actual shape of this metric depends on whatever queue-based scheduler redesign eventually replaces the static assignment, so hold off on locking in its exact form until that's picked.
On its own this metric can't say why a producer was slow. See #23814, a producer stuck waiting for a free thread there looks identical to one stuck on a genuinely large file.