Adding model for local inference

[elijahrenner]

Hi,

I'm using MLXLLM for inference on Llama 3.1 in my swift app. I want to use a higher-parameter model, but the ModelRegistry class in LLMModelFactory.swift doesn't have it. How can I add a model from mlx-community?

In my app I have

let modelConfiguration = ModelRegistry.llama3_1_8B_4bit

which retrieves the model from

static public let llama3_1_8B_4bit = ModelConfiguration(
    id: "mlx-community/Meta-Llama-3.1-8B-Instruct-4bit",
    defaultPrompt: "What is the difference between a fruit and a vegetable?"
)

How do I add more models?

[awni]

If the model type is supported you can make a new ModelConfiguration for the model you want to use. If it's not supported you have to port the model.. that is a lot more work.

[elijahrenner]

And how do I know if it's supported? (Thank you for the quick reply)

[awni]

You can see a list of supported model types here.

Those model types correspond to the model_type field in the config.json of the Hugging Face repo. See here for example.

[elijahrenner]

Looking to use Llama-4. I saw you recently tweeted about inference with some of the larger models on M4 Max. Do you think https://huggingface.co/mlx-community/Llama-4-Maverick-17B-128E-Instruct-6bit would work on my M3 Pro? And, are these Llama-4 models supported by the "llama" in model_type?

[awni]

No that's not supported in Swift yet. You can check the model type in the config.json. In this case it's llama4.

[elijahrenner]

And it would be complex to make it supported? Or, would I just have to configure it in LLMModelFactory.swift?

[awni]

It would require a port from Python which is not as simple as a configuration change