The finding
Spreading linear layers between softmax "anchors" costs far less quality than clustering them in a contiguous block, at an identical layer budget:
- contiguous 8 layers: −4.7 MMLU
- spread 8 layers: −0.7 MMLU
The interpretation is cross-layer error compounding: the interleaved softmax layers re-anchor the residual stream and stop the drift from accumulating. We verified the effect in both regimes we could measure — untrained perplexity and post-co-training TurkishMMLU.
The open question
This was measured on one architecture: Erk-14B, i.e. Qwen3-14B (40 layers, GQA). We do not know whether the rule is a property of linear-attention hybrids in general or of this specific depth / head configuration.
What would settle it
Run the placement sweep (contiguous vs spread, same layer count) on a different base — a different depth, a different GQA ratio, or a non-Qwen family — and report the degradation for both placements under one protocol. scripts/04_placement.py does this with saved GDN initializations and requires no co-training, so it is comparatively cheap.
If the effect holds, "spread, don't cluster" becomes a transferable design rule for attention hybrids rather than an observation about one model.
The finding
Spreading linear layers between softmax "anchors" costs far less quality than clustering them in a contiguous block, at an identical layer budget:
The interpretation is cross-layer error compounding: the interleaved softmax layers re-anchor the residual stream and stop the drift from accumulating. We verified the effect in both regimes we could measure — untrained perplexity and post-co-training TurkishMMLU.
The open question
This was measured on one architecture: Erk-14B, i.e. Qwen3-14B (40 layers, GQA). We do not know whether the rule is a property of linear-attention hybrids in general or of this specific depth / head configuration.
What would settle it
Run the placement sweep (contiguous vs spread, same layer count) on a different base — a different depth, a different GQA ratio, or a non-Qwen family — and report the degradation for both placements under one protocol.
scripts/04_placement.pydoes this with saved GDN initializations and requires no co-training, so it is comparatively cheap.If the effect holds, "spread, don't cluster" becomes a transferable design rule for attention hybrids rather than an observation about one model.