MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.
-
Updated
Sep 6, 2026 - Python
#
expert-caching
Here are 3 public repositories matching this topic...
Run AI models too large for your Mac's memory — at near-full speed. Intelligent expert caching, speculative execution, and 15+ research techniques for MoE inference on Apple Silicon.
python macos rust machine-learning inference moe quantization mlx speculative-execution mixture-of-experts memory-optimization apple-silicon llm metal-gpu ssd-streaming expert-caching
-
Updated
May 11, 2026 - Python
Run large MLX models on Apple Silicon with flash weight streaming, using native precision beyond RAM limits
menubar clustering optimization transformer quantization speculative-execution swiftui bpe memory-optimization apple-silicon bpe-encoder llm-training ssd-streaming weight-streaming expert-caching
-
Updated
Sep 7, 2026 - Python
Add this topic to your repo
To associate your repository with the expert-caching topic, visit your repo's landing page and select "manage topics."