What happens
When a rank stops responding, the other ranks of a JACCL group wait forever. No
timeout, no error, and not idly: each survivor spins a full core.
Four M4 Pro minis, full mesh over Thunderbolt RDMA, all four in a loop of 16 MB
all_sum inside try/except. Rank 2 suspended with SIGSTOP:
- the other three froze at the same iteration and stayed there, silent, indefinitely
- no exception, no abort, nothing on stderr
ps showed all three at 99 to 100% CPU in state Rs+, so they are busy-waiting
rather than blocked on a descriptorSIGCONT and all four resumed and completed normally, no corruption, no lost work
So a transient stall is survivable, which is good, and a death is indistinguishable
from it, which is not. Nothing ever reports that a peer is gone.
Why SIGSTOP rather than killing a rank
mlx.launch kills the surviving ranks the moment any rank exits, so it hides
whatever the survivors would have done. Suspending one rank leaves the process alive,
so the launcher stays out of the way while the peer stops servicing RDMA entirely.
Cutting the victim off at the network layer does not work as
a test: with ifconfig en2/en3/en4 down, the interfaces showing neither UP nor
RUNNING and IP ping over the point to point link at 100% loss, the four rank
collective kept running at full speed. RDMA over Thunderbolt does not go through
the IP stack once queue pairs are established. The /30 addresses only bootstrap the
side channel.
Comparison with ring
Ring had the same shape and it was addressed: #4060 made a lost peer fail the pending
tasks, and #3742 turns that into a catchable error. On the three revisions, a two
rank ring peer loss gives: v0.32.0 hangs, current main aborts with exit 134, #3742
raises a catchable RuntimeError. JACCL has none of that.
What would help
Anything that ends the wait. A timeout on the completion queue poll, or a liveness
check on the side channel, with the group reporting the loss instead of spinning.
The recovery behaviour after SIGCONT suggests the wait itself is sound and only the
unbounded part is the problem.
What happens
When a rank stops responding, the other ranks of a JACCL group wait forever. No
timeout, no error, and not idly: each survivor spins a full core.
Four M4 Pro minis, full mesh over Thunderbolt RDMA, all four in a loop of 16 MB
all_suminsidetry/except. Rank 2 suspended withSIGSTOP:psshowed all three at 99 to 100% CPU in stateRs+, so they are busy-waitingrather than blocked on a descriptor
SIGCONTand all four resumed and completed normally, no corruption, no lost workSo a transient stall is survivable, which is good, and a death is indistinguishable
from it, which is not. Nothing ever reports that a peer is gone.
Why SIGSTOP rather than killing a rank
mlx.launchkills the surviving ranks the moment any rank exits, so it hideswhatever the survivors would have done. Suspending one rank leaves the process alive,
so the launcher stays out of the way while the peer stops servicing RDMA entirely.
Cutting the victim off at the network layer does not work as
a test: with
ifconfig en2/en3/en4 down, the interfaces showing neitherUPnorRUNNINGand IP ping over the point to point link at 100% loss, the four rankcollective kept running at full speed. RDMA over Thunderbolt does not go through
the IP stack once queue pairs are established. The /30 addresses only bootstrap the
side channel.
Comparison with ring
Ring had the same shape and it was addressed: #4060 made a lost peer fail the pending
tasks, and #3742 turns that into a catchable error. On the three revisions, a two
rank ring peer loss gives: v0.32.0 hangs, current main aborts with exit 134, #3742
raises a catchable
RuntimeError. JACCL has none of that.What would help
Anything that ends the wait. A timeout on the completion queue poll, or a liveness
check on the side channel, with the group reporting the loss instead of spinning.
The recovery behaviour after
SIGCONTsuggests the wait itself is sound and only theunbounded part is the problem.