Ring backend: recv fails with errno 14 (EFAULT) during multi-node pipeline inference, fabric-independent (distinct from #3862)

[SystemsTiki]

Summary

During multi-node pipeline inference over the ring backend (mx.distributed, backend="ring"), the receiving rank fails with errno 14 (EFAULT) on recv, repeats until the 10-error threshold, then aborts:

[ring] Receiving from socket 51 failed with errno 14
[ring] Receiving from socket 51 failed with errno 14
... (x10 within milliseconds)
[ring] Too many send/recv errors. Aborting...

The peer rank then logs errno 32 (EPIPE) echoes when the aborting side's destructor shuts the sockets.

We believe this is distinct from #3862: that report's trigger is errno 54 (ECONNRESET — a network-layer reset), while EFAULT from recv(2) indicates the buffer pointer passed to the syscall is invalid — a memory-side fault, not a transport drop. Consistent with that reading, the failure reproduces identically across different physical fabrics (1 GbE ethernet and a direct Thunderbolt bridge link), which a network-layer cause would not predict.

Environment

• mlx 0.32.0 (PyPI wheel), Python 3.13, macOS (Apple Silicon)
• Nodes tested in 2-node rings, all combinations of: Mac15,14 (96 GB), 2× Mac14,8 M2 Ultra (64 GB), Mac16,11 M4 Pro (48 GB)
• libmlx.dylib md5 identical on 3 of 4 machines; failures reproduced on pairs with identical hashes, so binary skew is not the trigger
• Invoked via exo v0.3.70's MlxRing pipeline runner (hostfile with 2 ranks); model mlx-community/Llama-3.3-70B-Instruct-4bit (fits each pair's memory comfortably; shards sized ~13–26 GB per rank)

Reproduction pattern (three controlled runs)

1. Pair A–B over 1 GbE: ranks init (Group size: 2), weights load, warmup starts → receiving rank EFAULT burst → abort.
2. Pair A–C over direct Thunderbolt bridge: identical signature — sender EPIPE + receiver EFAULT.
3. Pair B–D (excluding node A): got further — prefill completed across the ring (~1.8 s), decode started — then wedged silently: one rank pinned at ~100 % CPU in the ring wait loop with GPU at 0 MHz / 0 mW (powermetrics), no error lines. This quiet wedge matches the abandoned-promises behavior described in [BUG] distributed ring: SocketThread dies silently on transient connection reset — all ranks wedge forever in Event::wait #3862 (and ring: fail on peer disconnect instead of hanging forever #4060's fix would at least convert it to a raised error — though we note ring: fail on peer disconnect instead of hanging forever #4060 is merged but not yet in any release).

After the abort, runners remain alive but stuck (pre-#4060 behavior), holding partially-loaded weights.

What we ruled out

• Memory pressure (placement sized under measured free memory; also reproduced with a model at ~35 % of aggregate RAM)
• Version skew (0.32.0 everywhere; PyPI has no newer release as of 2026-08-15)
• Reachability/firewall (all inter-rank TCP ports verified open both directions)
• A single bad machine (three different pairs, four machines, same class of failure)

Ask

Is EFAULT-on-recv a known failure mode of the ring backend on Metal? Candidate mechanisms we can't distinguish from outside: recv into a buffer whose Metal backing was freed/migrated mid-transfer, or an offset/pointer bug in the chunked recv path. Happy to run MLX_RING_VERBOSE=1, a minimal mlx.launch -n2 + all_sum repro, or a main-build test if that helps — the hardware and failure are readily available to us.

[erwinzhang7]

I have the same hardware class and could not reproduce this from the send/recv path
alone. Posting the negative because it rules out a fair amount of the space.

What I ran

4x Mac16,11 M4 Pro 48 GB, mlx 0.32.0 from PyPI (matching yours), Python 3.12,
macOS 26.5.1. Two ranks, mlx.launch --backend ring, over each of your two fabric
types in turn: 1 GbE-class LAN, and a direct Thunderbolt 5 link with a point to point
/30 on each end.

send/recv rather than all_sum, since that is the path in your traceback. To be
clear about what it is and isn't: rank 0 sends a tensor, rank 1 receives it and
sends the same tensor back. That is a round trip between two ranks, not a real
pipeline. There is no model, no stage to stage forward flow and no KV cache, so if
the fault needs the traffic pattern of an actual pipeline this would not find it.

EFAULT is a bad pointer, so everything below is aimed at the receive buffer rather
than at moving bytes quickly:

• buffers are GPU-produced, mx.random.normal(...) * 2 rather than mx.ones, so
  the pointer handed to the transport is the output of a Metal kernel
• the receive target is freshly allocated every round, so the allocator is recycling
  underneath the transport
• sizes alternate 4096 / 1,000,003 / 16,777,213 / 67,108,859 elements, none of them
  round, so the segment arithmetic never divides evenly
• a --free-early mode drops every python reference and calls mx.clear_cache()
  immediately after each collective
• a --ballast-gb mode holds a large resident allocation to stand in for a shard

500 rounds across both fabrics, plus 200 rounds holding 20 GB resident (~42% of
RAM), no EFAULT and no errors of any kind. So on this hardware a two rank
send/recv round trip does not fault, even when the receive buffer is released and the
cache cleared the instant the collective returns.

Script: https://github.com/erwinzhang7/mlx/blob/repro/4276/pipeline_repro_4276.py

The quiet wedge in your run 3 is 0.32.0 specific

eparating from the EFAULT, because it has a known cause and a fix that has
not shipped. In v0.32.0 the socket thread's error path is:

log_info(true, "Too many send/recv errors. Aborting...");
return;

It returns without satisfying the pending promises, so every future::get waiting on
that socket blocks forever. That is the ~100% CPU wait loop with the GPU at 0 MHz you
saw, and it is why the runners stay alive holding weights.

Current main replaces that with set_exception on the pending tasks, so the same
event raises instead of wedging. I checked the three revisions separately on a two
rank ring while testing #3742: v0.32.0 hangs, current main aborts with exit 134,
and #3742 turns it into a catchable RuntimeError. None of that addresses EFAULT,
but it means the symptom you get after the first failure changes completely
depending on which of the three you are on.

One thing to rule in or out from your side

SocketTask holds a raw void* into the array and nothing keeps the array alive, and
send/recv split the payload across sockets.size() segments, waiting with:

for (auto& f : recvs) { f.get(); }

On main, where f.get() can now throw, the first throw exits that loop and the
remaining segment futures are never awaited while their tasks still point into the
buffer. That needs more than one socket per neighbour to matter, and mlx.launch
defaults --connections-per-ip to 1, which is why my runs would not have exercised it.

If exo's MlxRing hostfile gives more than one address per node, it would be worth
running the same failure with exactly one and seeing whether EFAULT survives. If it
disappears, that is a strong signal. If it persists at one connection per neighbour,
the fault is somewhere the buffer pointer is wrong from the start rather than
abandoned partway.

Offer

Four M4 Pro minis, currently cabled as a full mesh and reconfigurable as a ring or
chain, RDMA enabled, and I can build any revision from source. I can run a
specific hostfile, a --connections-per-ip sweep, or a patch with instrumentation on
the recv path you point at.