Skip to content

Add support for Talkie (talkie-1930-13b) - #1492

Merged
zcbenz merged 3 commits into
ml-explore:mainfrom
ethanelasky:add-talkie-model
Sep 3, 2026
Merged

zcbenz merged 3 commits into
ml-explore:mainfrom
ethanelasky:add-talkie-model

Conversation

@ethanelasky

@ethanelasky ethanelasky commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Adds the Talkie architecture — talkie-lm's 13B models trained entirely on pre-1931 text (model card). llama.cpp already supports it as LLM_ARCH_TALKIE. The official repos are raw PyTorch checkpoints, but the transformers port at lewtun/talkie-1930-13b-it-hf converts directly with mlx_lm.convert and ships a normal tokenizer.json + chat template, so this is just a model file and a test entry. No tokenizer changes, no new dependencies.

Mostly a standard 40-layer MHA decoder (40 heads x 128, SwiGLU, vocab 65540), with a few quirks:

  • every RMSNorm is weightless (no learned scale), fp32 eps
  • RoPE rotates with the opposite sign to NeoX. Implemented with mx.fast.rope and negated freqs, same trick as nanochat.py; checked against the PyTorch reference (max diff ~5e-7). llama.cpp gets the same effect by folding the sign flip into the q/k weights at conversion time.
  • per-head RMSNorm on q and k applied after RoPE, plus a learned per-head gain on q only
  • learned scalar gains on the attention and MLP residuals, and a per-block skip connection from the normalized token embeddings
  • lm_head is a bare tensor scaled by a learned scalar (what the GGUFs export as logit_scale)

The offset goes straight from the cache into mx.fast.rope (int or per-sequence array) and there's no custom make_cache, so batch_generate and the server's continuous batching work as-is.

Re: the two earlier talkie PRs (#1220, #1231) — both build RoPE from a precomputed cos/sin table indexed by a scalar cache[0].offset and pin make_cache to KVCache, so they don't work with BatchKVCache. #1220 also targets the original checkpoint layout rather than the -hf one, and adds a tiktoken-based tokenizer adapter plus a tiktoken dependency that turns out to be unnecessary: the -hf repo's tokenizer.json tokenizes identically to llama.cpp's GGUF tokenizer on everything I threw at it, special tokens included.

Testing:

  • converted lewtun/talkie-1930-13b-it-hf to bf16 and q8 (group size 64); both load strictly with no missing/unexpected keys
  • q8 vs bf16 last-position logits on 12 prompts: argmax matches 12/12, max logit delta under 0.9 on 11 of them (one prompt reaches 3.7 with argmax unchanged — this model has unusually large activation outliers and some prompts are sensitive)
  • greedy decode vs llama.cpp (CPU, Q8_0): exact 64-token agreement on 4/12 prompts; every divergence is a near-tie between the two different q8 grids (our token sits at rank 2 in llama.cpp's logprobs)
  • batched generation matches single-stream greedy at prompt lengths 1/8/9/500 in one batch, apart from single-token flips at exact bf16 ties. Measured with Fix float32 promotion in BatchKVCache/BatchRotatingKVCache extend() #1491 applied — without that fix, sequences joining an in-flight batch get fp32-promoted K/V, which this model is sensitive to. Not a dependency, just affects that particular comparison.
  • added a talkie entry to test_all_models; full tests/test_models.py passes and pre-commit is clean

To try it:

mlx_lm.convert --hf-path lewtun/talkie-1930-13b-it-hf --mlx-path talkie-q8 \
    -q --q-bits 8 --q-group-size 64
mlx_lm.generate --model talkie-q8 --temp 0 \
    --prompt "What is the proper way to address a letter to a duchess?"

which answers: "To the Duchess of —, Belgrave Square, London, W."

@zcbenz
zcbenz merged commit 173d49d into ml-explore:main Sep 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants