The limitation
Our needle-in-a-haystack control was run only at ≤4K context, and both models scored 100% across all 45 trials. The difference is therefore zero by definition and the confidence interval is degenerate — the test does not discriminate at these lengths. We report it as a control, not as evidence of long-context parity.
This matters because long context is exactly where linear attention is supposed to pay off, and where a fixed-size recurrent state is most likely to lose information. We also measured a real (not bf16-artefact) perplexity penalty at 2048 vs 512, which is consistent with a state-capacity effect that a ≤4K needle test would never surface.
The open question
Where does the 20% hybrid actually break under long-context recall?
What would settle it
A discriminating long-context evaluation:
- ≥16K, ideally 32–64K context
- multi-needle rather than single-needle
- ordered / positional recall (which of several facts came first)
- needle placed at varying depths, including early in a long context
Report oracle and hybrid under one protocol with per-trial results so a CI can be computed. Turkish-language haystacks are preferred but any language is informative.
The KV-memory advantage we measured grows with context (−13.6% @4k → −19.6% @64k), so knowing the quality cost at those same lengths is the missing half of the trade-off.
The limitation
Our needle-in-a-haystack control was run only at ≤4K context, and both models scored 100% across all 45 trials. The difference is therefore zero by definition and the confidence interval is degenerate — the test does not discriminate at these lengths. We report it as a control, not as evidence of long-context parity.
This matters because long context is exactly where linear attention is supposed to pay off, and where a fixed-size recurrent state is most likely to lose information. We also measured a real (not bf16-artefact) perplexity penalty at 2048 vs 512, which is consistent with a state-capacity effect that a ≤4K needle test would never surface.
The open question
Where does the 20% hybrid actually break under long-context recall?
What would settle it
A discriminating long-context evaluation:
Report oracle and hybrid under one protocol with per-trial results so a CI can be computed. Turkish-language haystacks are preferred but any language is informative.
The KV-memory advantage we measured grows with context (−13.6% @4k → −19.6% @64k), so knowing the quality cost at those same lengths is the missing half of the trade-off.