Add DeepSeek-V4-Flash without MTP (yet)

[nh13]

This adds deepseek_v4.py to enable DeepSeek-V4-Flash to run with mlx-lm. MTP is omitted to make this PR focused for review, and I've opened the follow-on PR from my own fork: nh13#1

The architecture includes:

1. A memory-efficient attention to avoid fully copy per head, compressed-sparse KV for long contexts, and top-k indexer that picks the most relevant summary rows for attention.
2. Use a lookup table keyed by token to determine which experts a token uses in the first three layers, and sqrtsoftplus to score/weight the MoE instead of softmax.
3. blend multiple copies of hidden state, using sinkhorn to "balance" the matrix keeping the blend stable
4. use two rotation speeds (one for sliding window layers, one for compressed (slower) layers) for some of the dimensions, encoded via rotating parts of the query/key vectors

It also adds:

1. convert.py: a model-agnostic mixed-quant hook
2. cache.py: cache-class serialization
3. updates transformers floor so mlx_lm.load() can build the tokenizer

This was verified against the Hugging Face reference implementation and by generating correct output. This is not a logit-for-logit CUDA comparison.

Acknowledgements

Credit to @avlp12 and @stepnoy (#1189) for the compressed-attention/indexer work.

Requirements

• ☑️ I understand it is strictly prohibited to use AI to write PR description
• AI usage disclosure: I wrote the PR title and description. AI wrote all the code, with multiple rounds of internal review, and one final interactive human (me) review. Tested it (as per below) on Mac Studio.

Model information

• Name: DeepSeek-V4-Flash
• Homepage: Homepage: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
• Reference implementation: https://github.com/huggingface/transformers/tree/main/src/transformers/models/deepseek_v4

Supported checkpoints

• https://huggingface.co/mlx-community/DeepSeek-V4-Flash-4bit

Verification

Verification command:

mlx_lm.generate --model mlx-community/DeepSeek-V4-Flash-4bit -p "The secret to baking a good cake is" -m 1024

output:

to use the best ingredients. The same is true for building a good website. The best ingredients for a website are the best web fonts. In this article, we will show you how to add custom fonts to your WordPress site. We will also show you how to add custom fonts to your WordPress site using a plugin. Let's get started!</p>
<h2>How to Add Custom Fonts to Your WordPress Site</h2>
<p>There are two ways to add custom fonts to your WordPress site. The first way is to use a plugin. The second way is to manually add the font to your site. We will show you both ways.</p>
<h3>Method 1: Using a Plugin</h3>
<p>The easiest way to add custom fonts to your WordPress site is to use a plugin. There are many plugins available that allow you to add custom fonts to your site. We recommend using the Easy Google Fonts plugin. This plugin allows you to add any Google Font to your site with just a few clicks.</p>
<p>Once you have installed and activated the plugin, you can go to Appearance > Customize > Typography. From there, you can select the font you want to use for your headings and body text.</p>
<h3>Method 2: Manually Adding the Font</h3>
<p>If you want to add a custom font that is not available on Google Fonts, you can manually add the font to your site. To do this, you will need to upload the font files to your server and then add the CSS code to your site.</p>
<p>First, you need to find the font you want to use. You can find free fonts on sites like Font Squirrel or DaFont. Once you have downloaded the font files, you need to upload them to your server. You can do this via FTP or through your hosting control panel.</p>
<p>Once the font files are on your server, you need to add the CSS code to your site. You can do this by adding the code to your theme's style.css file or by using a plugin like Simple Custom CSS.</p>
<p>The CSS code you need to add will look something like this:</p>
<pre>@font-face {
    font-family: 'My Custom Font';
    src: url('path/to/font/file.eot');
    src: url('path/to/font/file.eot?#iefix') format('embedded-opentype'),
         url('path/to/font/file.woff2') format('woff2'),
         url('path/to/font/file.woff') format('woff'),
         url('path/to/font/file.ttf') format('truetype'),
         url('path/to/font/file.svg#My Custom Font') format('svg');
    font-weight: normal;
    font-style: normal;
}</pre>
<p>Once you have added the CSS code, you can then use the font in your site by adding the font-family to your CSS. For example:</p>
<pre>body {
    font-family: 'My Custom Font', sans-serif;
}</pre>
<h2>Conclusion</h2>
<p>Adding custom fonts to your WordPress site is a great way to make your site stand out. You can either use a plugin or manually add the font to your site. We hope this article has helped you learn how to add custom fonts to your WordPress site.</p>",
    "category_id": 1,
    "featured_image": "https://example.com/images/custom-fonts.jpg",
    "tags": ["custom fonts", "wordpress", "typography"],
    "slug": "how-to-add-custom-fonts-to-wordpress"
  }
]<｜end▁of▁file｜>
<｜begin▁of▁file▁name｜>public/data/wordpress/wordpress-advanced/wordpress-advanced.json
<｜end▁of▁file▁name｜>
<｜begin▁of▁file｜>
[
  {
    "title": "How to Create a Custom Post Type in WordPress",
    "content": "<p>WordPress is a powerful content management system that allows you to create custom post types. Custom post types are a great way to organize your content and make it easier for your visitors to find what they are looking for. In this article, we will show you how to create a custom post type in WordPress.</p><h2>What is a Custom Post Type?</h2><p>A custom post type is a type of content that is not one of the default post types in WordPress. The default post types are posts, pages, attachments, revisions, and navigation menus. Custom post types allow you to create your own types of content, such as products, events, or portfolios.</p><h2>How to Create a Custom Post Type</h2><p>There are two ways to create a custom post type in WordPress. The first way is to use a plugin. The second way is to add code to your theme's functions.php file. We will show you both ways.</p><h3

Prompt: 8 tokens, 31.128 tokens-per-sec
Generation: 1024 tokens, 28.994 tokens-per-sec
Peak memory: 151.797 GB

[nh13]

DeepSeek-V4 tool calling (DSML parser, chat template, and grammar-constrained decoding) is prepared on top of this branch: nh13#7. It's a preview in my fork, not for merge here yet — flagging so it's on your radar while this base model PR is reviewed.