Bound Whisper VRAM across worker replicas, gte on CUDA, multi-worker GPU docs

[hischampion]

AS-IS

Running several audiomuse-ai-worker replicas on one GPU stops scaling early, for three independent reasons:

1. Every replica that reaches the Whisper-small fallback loads its own pipeline (~1.5 GB + activations). Three replicas transcribing at once fill a 12 GB card; a fourth OOMs. Worker count is bounded by Whisper even though CLAP/MusiCNN would leave room for many more.
2. Without a cgroup CPU limit cpu_budget.py sees the whole host, so every replica opens ONNX thread pools as wide as the machine. Six replicas on a 20-thread host = 600+ threads, load average ~50, and the replicas mostly thrash each other.
3. The gte lyrics-embedding model is hard-wired to CPUExecutionProvider, so a transformer pass per lyric runs on the cores that audio decoding needs, while the GPU sits at 10-20%.

TO-BE

All opt-in, defaults unchanged:

• LYRICS_ASR_LOCK_DIR / LYRICS_ASR_LOCK_SLOTS — an N-slot semaphore made of flock files in a directory every replica mounts (new asr-locks volume in docker-compose-nvidia.yaml). At most N replicas run Whisper concurrently; the rest wait. The slot is taken before the ASR alarm starts so queueing is never charged against the 300 s timeout, and the pipeline is unloaded before the slot is released so two resident copies never overlap. A crashed worker releases its slot via the kernel. POSIX only — the Windows native build skips it (guarded fcntl import).
• LYRICS_GTE_USE_GPU — resolve_providers(label='gte', cpu_only_default=not LYRICS_GTE_USE_GPU). Off by default because the int8 graph picks up Memcpy nodes on CUDA; on a shared GPU it still wins.
• Compose / docs — commented examples for the env and a per-replica cpus: quota (which cpu_budget.py already honors, it just is never set), and a "Several workers on one GPU" section in docs/GPU.md with the measurements below.

Test

Measured on an RTX 5070 (12 GB), i7-12700K (20 threads), 32 GB RAM, Docker in WSL2, latest-nvidia image v3.6.1, Navidrome 0.61.2, 5.7k-track library, ~25% of tracks reaching the Whisper fallback:

replicas	Whisper slots	cpus / replica	tracks/min	VRAM	load avg
3	unlimited	none	10	11.5 GB (ceiling)	13
6	3	none	19	6 GB	49
6	3	3	29	7.6 GB	8
8	3	3	38	9 GB	8

Full library analyzed end to end with 8 replicas (5,724 tracks, no CUDA OOM, no worker restarts), then a lyrics-only re-run of 828 tracks. Whisper pass latency also dropped (median 20 s → 4-6 s) once the semaphore stopped 3-4 decoders contending for the GPU at the same time.

Semaphore unit check inside the image: 6 threads / 2 slots → peak concurrency 2; with LYRICS_ASR_LOCK_DIR empty _acquire_asr_slot() returns None and _release_asr_slot(None) is a no-op.

To reproduce: docker-compose-nvidia.yaml, uncomment LYRICS_ASR_LOCK_DIR / LYRICS_ASR_LOCK_SLOTS / cpus, drop container_name on the worker, docker compose up -d --scale audiomuse-ai-worker=8, watch nvidia-smi and docker stats.

Other useful information

The same tuning applies to Kubernetes: a shared RWX volume for the lock dir and a CPU limit on the worker pod. Not tested there.

The gte-on-CUDA session logs the ONNX Runtime "N Memcpy nodes are added" warning — expected for this int8 graph, harmless.

Checklist

Type of change:

• New feature
• Documentation

Tested on architecture:

• Intel
• ARM
• -noavx2
• NVIDIA image

Tested on media server:

• Navidrome
• Jellyfin
• Emby
• Lyrion

Updated:

• Documentation
• Unit test
• Integration test

Other:

• CONTRIBUTING.md read and accepted
• Checked performance on a big library (> 150k songs) works without issues

Not tested on ARM, the non-NVIDIA images, or Jellyfin/Emby/Lyrion — the change is media-server independent (lyrics stage only) and no-op unless the new env vars are set, but flagging it per CONTRIBUTING.md. The lyrics-stage code path is the same on ARM; the fcntl guard covers the Windows native build.

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[hischampion]

This sped up my analysis 4x on my machine. I am not sure if we should have something set sane defaults to avoid thrashing or if we should leave it for a followup.