MLX LM

MLX LM is a Python package for generating text and fine-tuning large language models on Apple silicon with MLX.

This is a fork of ml-explore/mlx-lm. It adds long-context inference with the KV cache split across several Macs, and fast decoding of several questions about the same text at once. It is not part of the upstream project. See Context sharding below.

Why a fork. The features of this fork are not meant to go into the upstream repository, and they do not need to. MLX already has good low-level, mid-level and high-level building blocks, so with an AI coding assistant it is fast to adapt the library to your own needs. This fork is an example of that approach. The matrix-instruction attention kernel and the distributed attention solve a narrow, special task and are unlikely to be useful in MLX itself. The main task (the sharded KV cache with distributed attention) was done in about 1.4 hours of work with an AI coding agent. The weekend went into testing and packaging, and into extra experiments with optimization and batch processing (the matrix kernel, batched questions, continuous batching and the server). Those were not part of the main task. It is also common to keep several such libraries next to the standard one on the same machine and to run them as parts of a specialized solution.

Concept illustration of a local Apple silicon cluster. The numbers on the screen are not measurements.

Some key features include:

• Integration with the Hugging Face Hub to easily use thousands of LLMs with a single command.
• Support for quantizing and uploading models to the Hugging Face Hub.
• Low-rank and full model fine-tuning with support for quantized models.
• Distributed inference and fine-tuning with mx.distributed
• Fork only: context sharding: prepare a long text once, keep its KV cache split between several Macs, and ask many short questions against it, alone or in batches.

The easiest way to get started is to install the mlx-lm package:

With pip:

pip install mlx-lm

With conda:

conda install -c conda-forge mlx-lm

Context sharding (this fork)

Install this fork instead of the upstream package:

pip install git+https://github.com/danxn/mlx-lm.git

The idea: a large document is prepared once. Every machine keeps the whole model and only its own part of the KV cache, so the context can be longer than one Mac can hold. Short questions then run against that cache, and nothing is written to it, so the prepared text stays as it was. Machines exchange one small message per layer and step, so plain TCP over Ethernet or Thunderbolt is enough.

Large model                       Large context
──────────────                    ────────────────
weights > RAM of one Mac          KV > RAM of one Mac
        │                                │
        ▼                                ▼
weight sharding                   context sharding
        │                                │
 Mac1: W₁                         Mac1: model + KV₁
 Mac2: W₂                         Mac2: model + KV₂
 Mac3: W₃                         Mac3: model + KV₃
 Mac4: W₄                         Mac4: model + KV₄
        │                                │
        └───────────┬────────────────────┘
                    ▼
             one inference

The left side is what mx.distributed already offers in MLX LM. The right side is what this fork adds. Using both together was not tested.

# On every machine, same text file. Two local processes for a first try:
mlx.launch --hosts 127.0.0.1 -n 2 --backend ring -- \
    python mlx_lm/examples/sharded_context.py prepare \
    --model mlx-community/Llama-3.2-1B-Instruct-4bit \
    --file document.txt --cache-dir /tmp/context_cache --block-size 256

mlx.launch --hosts 127.0.0.1 -n 2 --backend ring -- \
    python mlx_lm/examples/sharded_context.py ask \
    --model mlx-community/Llama-3.2-1B-Instruct-4bit \
    --cache-dir /tmp/context_cache --batch \
    --question "What is the vault code?" --question "Who wrote the letter?"

Without --batch the questions run one after another. With --batch they are decoded together and each machine reads its part of the cache once for all of them. The cache can hold 16-bit values or be quantized to 8 or 4 bits (--kv-bits).

For questions that arrive at different times there is an OpenAI-compatible server (python -m mlx_lm.sharded_server): a new question joins the running batch at the next step, see mlx_lm/CONTEXT_SHARDING.md.

Time of one decoding step, Llama 3.2 1B, 32768 tokens in the cache, M3 Max (each question gets one new token per step):

Questions	One by one	Batched, 16-bit cache	Batched, 8-bit cache
1	10.6 ms	10.6 ms	10.7 ms
4	42 ms	12.0 ms	12.3 ms
8	85 ms	16.5 ms	16.6 ms

Works with Llama and Mistral, Qwen 2 and 3, and Gemma 1 to 4 (including Gemma 3 and 4 image features). It was tested on two local processes, three local processes and a pair of Macs over Thunderbolt Bridge. The RDMA backend (JACCL) was not tested. Details, limits and more measurements are in mlx_lm/CONTEXT_SHARDING.md, and the tests are in tests/sharded_context_tests.py.

This code was written with the help of an AI assistant (Claude).

Quick Start

To generate text with an LLM use:

mlx_lm.generate --prompt "How tall is Mt Everest?"

To chat with an LLM use:

mlx_lm.chat

This will give you a chat REPL that you can use to interact with the LLM. The chat context is preserved during the lifetime of the REPL.

Commands in mlx-lm typically take command line options which let you specify the model, sampling parameters, and more. Use -h to see a list of available options for a command, e.g.:

mlx_lm.generate -h

The default model for generation and chat is mlx-community/Llama-3.2-3B-Instruct-4bit. You can specify any MLX-compatible model with the --model flag. Thousands are available in the MLX Community Hugging Face organization.

Python API

You can use mlx-lm as a module:

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")

prompt = "Write a story about Einstein"

messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True,
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

To see a description of all the arguments you can do:

>>> help(generate)

Check out the generation example to see how to use the API in more detail. Check out the batch generation example to see how to efficiently generate continuations for a batch of prompts.

The mlx-lm package also comes with functionality to quantize and optionally upload models to the Hugging Face Hub.

You can convert models using the Python API:

from mlx_lm import convert

repo = "mistralai/Mistral-7B-Instruct-v0.3"
upload_repo = "mlx-community/My-Mistral-7B-Instruct-v0.3-4bit"

convert(repo, quantize=True, upload_repo=upload_repo)

This will generate a 4-bit quantized Mistral 7B and upload it to the repo mlx-community/My-Mistral-7B-Instruct-v0.3-4bit. It will also save the converted model in the path mlx_model by default.

To see a description of all the arguments you can do:

>>> help(convert)

Streaming

For streaming generation, use the stream_generate function. This yields a generation response object.

For example,

from mlx_lm import load, stream_generate

repo = "mlx-community/Mistral-7B-Instruct-v0.3-4bit"
model, tokenizer = load(repo)

prompt = "Write a story about Einstein"

messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True,
)

for response in stream_generate(model, tokenizer, prompt, max_tokens=512):
    print(response.text, end="", flush=True)
print()

Sampling

The generate and stream_generate functions accept sampler and logits_processors keyword arguments. A sampler is any callable which accepts a possibly batched logits array and returns an array of sampled tokens. The logits_processors must be a list of callables which take the token history and current logits as input and return the processed logits. The logits processors are applied in order.

Some standard sampling functions and logits processors are provided in mlx_lm.sample_utils.

Command Line

You can also use mlx-lm from the command line with:

mlx_lm.generate --model mistralai/Mistral-7B-Instruct-v0.3 --prompt "hello"

This will download a Mistral 7B model from the Hugging Face Hub and generate text using the given prompt.

For a full list of options run:

mlx_lm.generate --help

To quantize a model from the command line run:

mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q

For more options run:

mlx_lm.convert --help

You can upload new models to Hugging Face by specifying --upload-repo to convert. For example, to upload a quantized Mistral-7B model to the MLX Hugging Face community you can do:

mlx_lm.convert \
    --model mistralai/Mistral-7B-Instruct-v0.3 \
    -q \
    --upload-repo mlx-community/my-4bit-mistral

Models can also be converted and quantized directly in the mlx-my-repo Hugging Face Space.

Long Prompts and Generations

mlx-lm has some tools to scale efficiently to long prompts and generations:

• A rotating fixed-size key-value cache.
• Prompt caching
• A configurable prefill step size

To use the rotating key-value cache pass the argument --max-kv-size n where n can be any integer. Smaller values like 512 will use very little RAM but result in worse quality. Larger values like 4096 or higher will use more RAM but have better quality.

Long prompts are read in steps, and the step size sets how many prompt tokens are processed at once. Smaller steps use less peak memory while the prompt is read, at some cost to prompt processing speed. To change it pass --prefill-step-size n. The default is 2048, and mlx_lm.generate, mlx_lm.chat, mlx_lm.server and mlx_lm.benchmark all accept it.

Caching prompts can substantially speedup reusing the same long context with different queries. To cache a prompt use mlx_lm.cache_prompt. For example:

cat prompt.txt | mlx_lm.cache_prompt \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --prompt - \
  --prompt-cache-file mistral_prompt.safetensors

Then use the cached prompt with mlx_lm.generate:

mlx_lm.generate \
    --prompt-cache-file mistral_prompt.safetensors \
    --prompt "\nSummarize the above text."

The cached prompt is treated as a prefix to the supplied prompt. Also notice when using a cached prompt, the model to use is read from the cache and need not be supplied explicitly.

Prompt caching can also be used in the Python API in order to avoid recomputing the prompt. This is useful in multi-turn dialogues or across requests that use the same context. See the example for more usage details.

Supported Models

mlx-lm supports thousands of LLMs available on the Hugging Face Hub. If the model you want to run is not supported, file an issue or better yet, submit a pull request. Many supported models are available in various quantization formats in the MLX Community Hugging Face organization.

For some models the tokenizer may require you to enable the trust_remote_code option. You can do this by passing --trust-remote-code in the command line. If you don't specify the flag explicitly, you will be prompted to trust remote code in the terminal when running the model.

Tokenizer options can also be set in the Python API. For example:

model, tokenizer = load(
    "qwen/Qwen-7B",
    tokenizer_config={"eos_token": "<|endoftext|>", "trust_remote_code": True},
)

Large Models

Note

This requires macOS 15.0 or higher to work.

Models which are large relative to the total RAM available on the machine can be slow. mlx-lm will attempt to make them faster by wiring the memory occupied by the model and cache. This requires macOS 15 or higher to work.

If you see the following warning message:

[WARNING] Generating with a model that requires ...

then the model will likely be slow on the given machine. If the model fits in RAM then it can often be sped up by increasing the system wired memory limit. To increase the limit, set the following sysctl:

sudo sysctl iogpu.wired_limit_mb=N

The value N should be larger than the size of the model in megabytes but smaller than the memory size of the machine.