Add Laguna XS.2 - #1223
Add Laguna XS.2 #1223Blaizzy wants to merge 12 commits into
Conversation
|
Running the 4-bit on my M5 Max: 6000-token generation @ 116 tok/s, 19.9 GB peak. Nice work! |
|
My pleasure John! 🙌🏽 |
|
Thanks for this — got 1. def make_cache(self):
caches = []
for lt in self.args.layer_types:
if lt == "sliding_attention" and self.args.sliding_window:
caches.append(RotatingKVCache(max_size=self.args.sliding_window))
else:
caches.append(KVCache())
return caches(with 2. Loading VLM-style repacks. Some 8-bit repacks (e.g. AtomicChat's) wrap every tensor under a if any(k.startswith("language_model.") for k in weights):
weights = {(k[len("language_model."):] if k.startswith("language_model.") else k): v
for k, v in weights.items()}Happy to open a PR against your branch if that's easier. Thanks again! |
|
Running the 8-bit repack ( 1. 2. Both are tested on our end and independent of the core model code here. Happy to push them to this branch or open a small follow-up once this lands — whichever you'd prefer. |
|
Turned the two additions into a ready-to-merge follow-up onto your branch: Blaizzy#26 (windowed |
|
any updates! |
…a XS 2.1 Architecture table was wrong for three models. The "-A3B" and "-a4b" suffixes in the model IDs mean activated parameters, so these are all MoE, not dense: qwen3.6-35b Qwen3.6-35B-A3B 35B total / 3B active, 256 experts, top-8 +1 qwen3.5-35b Qwen3.5-35B-A3B 35B total / 3B active gemma-4-26b gemma-4-26b-a4b 25.2B total / 3.8B active, 128 experts ornith (36B-A3B) and both Lagunas were missing entirely and are now included. The correction strengthens the section's own thesis rather than weakening it: laguna-xs.2 turns out to have the best tok/s per active billion of anything tested, which is the same model the cost table independently ranks cheapest. deepseek-r1 32b moves to a new "kept for the record, not in the running" section. A dense 32B that reasons at length costs more wall clock than its score justifies. Its per-task detail is still published; it is only removed from the main table, the cost table, the architecture table and the nav, so a ranking is not led by a model nobody would pick. Dropped the phantom devstral-small entry: it named the same checkpoint as devstral/devstral2, was never in bench.py, and produced no row — it only ever surfaced as a misleading name in the cost table's exclusion note. Laguna XS 2.1 prepared but not yet run: bench target, runner, and a thinking budget. Served with mlx_vlm.server because stock mlx-lm has no laguna class (ml-explore/mlx-lm#1223); the GGUF route needs two unmerged llama.cpp PRs and returns empty text on Metal. The runner smoke-tests for that empty-content bug and refuses to score it as 126 model failures. Co-authored-by: Cursor <cursoragent@cursor.com>
|
closed via #1704 |
6 participants
No description provided.