feat: Add fixed-slot GPU expert streaming for PhiMoE

[DevElCuy]

Summary

Adds a FreeToken-inspired, opt-in fixed-slot GPU expert-streaming runtime, initially supporting PhiMoE.

On an AMD Radeon RX 6400, generation improved from 366.74 seconds to 2.64 seconds—approximately 139× faster—with identical token IDs. This is an independent AirLLM implementation; no FreeToken code is reused.

The runtime:

• Builds a normalized CPU host bank for expert weights.
• Keeps dense weights resident on the GPU.
• Reuses preallocated GPU slots through LRU replacement.
• Uses scratch slots during multi-expert prefill.
• Eliminates generation-time expert disk reads and allocator churn.

Usage

model = AutoModel.from_pretrained(
    model_path,
    expert_cache_backend="slot",
    expert_cache_gb=0.75,
)

The slot backend is currently supported by the new PhiMoE adapter and requires a CUDA- or ROCm-compatible device.

Changes

• Add AirLLMPhiMoE and automatic architecture selection.
• Add the fixed-slot expert runtime and CPU host bank.
• Preserve AirLLM’s existing streaming behavior by default.
• Add cache lifecycle statistics and a reproducible benchmark.
• Add CPU and GPU coverage for slot reuse, eviction, prefill, and failure recovery.

Validation

Benchmarked with Microsoft Phi-tiny-MoE-instruct at revision 2fe50e8.

Test system

• GPU: AMD Radeon RX 6400 (gfx1034, 4 GiB VRAM)
• CPU: AMD Ryzen 5 3600, 6 cores / 12 threads
• Host memory: 31 GiB
• Storage: NVMe, ext4
• PyTorch: 2.12.0+rocm7.14.0
• Runtime dtype: float16

Resource usage

• Hugging Face checkpoint: approximately 7.0 GiB
• AirLLM split checkpoint: approximately 7.0 GiB
• Measured disk usage with both retained: approximately 14 GiB
• CPU expert host bank: 5.25 GiB for 512 experts
• Peak process RAM: 7.60 GiB
• Dense GPU weights: 1.74 GiB
• Expert-slot budget: 0.75 GiB
• GPU slots: 73 resident slots plus 2 scratch slots
• Peak reserved VRAM: 2.56 GiB
• Expert disk reads during generation: 0

Generation improved from 366.74 seconds to 2.64 seconds—approximately 139× faster—while producing identical token IDs.

The final smoke test generated 8 tokens in 2.73 seconds at 2.93 tokens/second. The ROCm suite discovered 37 tests: 36 passed and one optional BitsAndBytes test was skipped.