What we measured
Co-trained, spread configurations on TurkishMMLU (oracle vs hybrid, identical protocol, paired bootstrap 95% CI):
| Linear |
MMLU Δ |
95% CI |
Verdict |
| 20% (8 layers) |
−0.7 |
[−2.9, +1.7] |
not significant |
| 22.5% (9 layers) |
−1.8 |
[−4.2, +0.7] |
not significant (limited power) |
| 25% (10 layers) |
−4.0 |
[−6.4, −1.4] |
significant degradation |
Our diagnosis (input teacher-forcing + cosine analysis) attributes the loss roughly half to cross-layer compounding and half to a single-layer capacity gap, which is consistent with a genuine capacity wall in the middle band.
The open question
Is 25% failing because those layers genuinely cannot be represented by a fixed-size state — or because our co-training recipe is too thin? We used KL distillation on the GDN parameters only. We did not try:
- MLP-LoRA on the downstream feed-forward blocks
- hidden-state matching in addition to logit KL
- substantially more co-training data or steps
- multiple seeds (all main comparisons are single-seed)
In one earlier experiment a fuller recipe closed roughly 40% of the gap (−9.3 → −5.3) and then plateaued — suggesting the wall is real but that our current number may be pessimistic.
What would settle it
Co-train the 25% configuration with an expanded recipe and report MMLU with a paired bootstrap CI on the held-out set. Even a partial result is useful. If the gap closes below significance, the ceiling is a recipe artifact; if it plateaus again, the capacity interpretation stands.
Protocol and raw predictions: REPRODUCE.md, results/raw_predictions.json.
What we measured
Co-trained, spread configurations on TurkishMMLU (oracle vs hybrid, identical protocol, paired bootstrap 95% CI):
Our diagnosis (input teacher-forcing + cosine analysis) attributes the loss roughly half to cross-layer compounding and half to a single-layer capacity gap, which is consistent with a genuine capacity wall in the middle band.
The open question
Is 25% failing because those layers genuinely cannot be represented by a fixed-size state — or because our co-training recipe is too thin? We used KL distillation on the GDN parameters only. We did not try:
In one earlier experiment a fuller recipe closed roughly 40% of the gap (−9.3 → −5.3) and then plateaued — suggesting the wall is real but that our current number may be pessimistic.
What would settle it
Co-train the 25% configuration with an expanded recipe and report MMLU with a paired bootstrap CI on the held-out set. Even a partial result is useful. If the gap closes below significance, the ceiling is a recipe artifact; if it plateaus again, the capacity interpretation stands.
Protocol and raw predictions:
REPRODUCE.md,results/raw_predictions.json.