ArraysCache metadata graph grows during sustained Qwen3.8-27B batch decoding

[marksmeltzer]

ArraysCache metadata growth during sustained Qwen3.8-27B batch decoding

Proposed fix: PR #1872

Summary

While enabling mlx-community/Qwen3.8-27B-8bit and
mlx-community/Qwen3.8-27B-bf16 in EXO, we found a lazy-evaluation gap involving
ArraysCache length and left-padding metadata during sustained decoding.

Both artifacts use the existing qwen3_5 implementation. This is a cache-behavior
report, not a request for a new Qwen3.8 architecture implementation.

This appears related to #1845. Please let me know whether the reproduction and
downstream evidence belong there rather than in a separate issue.

Observed behavior

A retained Q8 generation run failed on two of four EXO workers approximately
264 seconds into decoding:

[metal::malloc] Resource limit (499000) exceeded

The exception surfaced at mx.async_eval while evaluating token and logprob
outputs. No equivalent historical BF16 crash was captured.

A focused reproduction shows that repeated ArraysCache.advance() calls can
grow the metadata computation graph when the caller evaluates a populated cache
tensor without evaluating the complete cache.state.

This demonstrates metadata retention. It does not establish that every Metal
object involved in the historical crash originated from these arrays.

Reproduction and environment

The tested baseline was rltakashige/mlx-lm at
6a3df6cd6b00a347ee40f12d97a182aaf86ea599, reporting version 0.31.3,
with MLX 0.32.0.dev20260522. This was not an unmodified PyPI release.
A red/green comparison against current upstream main remains pending.

The standalone reproduction uses real MLX and ArraysCache, without weights
or a running EXO service. It initializes both cache slots, advances length and
padding metadata 256 times, and evaluates cache[0] rather than complete state.

Standalone reproduction source

On the tested baseline, the metadata graph assertion fails at 1,280 edges.
With the candidate correction, both metadata graphs satisfy the at-most-eight-edge
bound, and the final state and metadata values are correct.

Candidate correction and related work

The candidate attaches non-null metadata to the first populated cache tensor
using mx.depends after advancing the metadata. On the reviewed Qwen path, this
tensor is consumed by the next decode step.

This adapts the advance() portion of
Pierre Lamy's bb615eb;
it does not include that commit's separate extract() change.
Related work includes #1632, #1641, #1642, and the alternative discussed in #1845.
I am not claiming this is a new mechanism or the only viable solution.

Downstream evidence and scope

EXO issue #2302 records the exact
model revisions, environment, symptoms, and validation limits.
EXO PR #2301 carries the candidate
dependency patch and public reproduction instructions.

A combined development build completed sustained Q8 and BF16 generation on four
Mac Studio M3 Ultra nodes using Tensor/JACCL. It also included a separate EXO
processor correction, so those results do not establish current-upstream MLX-LM
qualification. Four-node execution is not a demonstrated prerequisite for the
cache defect.

AI assistance was used for investigation, candidate code, regression preparation,
hardware test execution, etc.