MacProvider — making mlx-lm endpoints addressable over the internet, with verifiable inference

[Augustas11]

Hi all — sharing a project built on top of mlx-lm that some of you might find interesting (or want to poke holes in).

The premise: a lot of the most interesting LLM applications — long-running personal agents, privacy-sensitive tooling, dev workflows that hammer a model thousands of times a day — don't really belong in a cloud datacenter. But the moment you want your Mac's MLX endpoint to be reachable from somewhere that isn't localhost, you fall off a cliff (auth, tunneling, multi-tenant routing, observability, none of it exists out of the box).

MacProvider is an attempt to fill that gap as a thin layer over mlx-lm:

• Provider side: a small daemon on any Apple Silicon Mac that runs mlx-lm and registers with a coordinator. No code changes to mlx-lm — it wraps the existing OpenAI-compatible server.
• Coordinator + gateway: routes requests to whichever provider can serve them, holds auth/rate-limit/quotas, exposes a single OpenAI-compatible URL to callers.
• Verifiable inference (in progress): signed receipts that bind (prompt hash, model id, output, provider identity, timestamp) so a caller can later prove which Mac actually ran their request. This is the piece I'm most curious to get feedback on — it's what makes "inference outside a datacenter" something you can build on rather than just hope works.

What this is not: a replacement for JACCL/tensor-parallel across a TB-linked cluster. Each Mac stays an independent endpoint; the coordinator just picks one per request. Different trade-off from exo — no model sharding, but also no cluster bring-up.

Why I think it matters for this community: a lot of MLX is being used today for on-device personal experiments that hit a ceiling the moment they need to be addressable from a phone, a teammate, or a cron job running elsewhere. Making MLX endpoints first-class internet citizens opens up the design space for apps that are local-first by default rather than cloud-first by inertia.

Repo: https://github.com/Augustas11/macprovider
Install (single Mac, one-liner): curl -fsSL get.streamvc.live/install.sh | sh

Specifically would love feedback on:

1. The receipts design — is the (prompt-hash, model-id, output-hash, signature) tuple the right shape, or am I missing a primitive?
2. mlx-lm integration — currently a wrapper. Would a tighter hook into mlx-lm (e.g. native signed responses) be welcome upstream, or is this firmly out of scope?
3. Anything obviously wrong with the coordinator/gateway split for this use case.

Thanks for mlx and mlx-lm — none of this exists without them.

[NovaRouteAI]

"MacProvider making mlx-lm endpoints addressable over the internet, with verifiable inference" is the kind of thread where I would make raw usage and normalized cost visible before trying to optimize model choice.

I would keep provider, model, request type, prompt/output tokens, cached-token behavior, latency, retry count, and final billing bucket together in one event. That makes cost debugging and future routing decisions much easier.

I am testing a multi-model OpenAI-compatible API layer around official Chinese models, so the routing and usage-accounting parts of this are directly relevant to me. For mlx, would the first win be user-facing cost transparency, internal spend debugging, or routing cheaper calls to cheaper models?