feat(enhancement): LocalVQE AEC with safe streaming and benchmark validation (beta)

[Alex-Wengg]

Why is this change needed?

Adds LocalVQE acoustic echo cancellation, noise suppression and dereverberation for 16 kHz speech, addressing the LocalVQE request in #49. Includes the Core ML consumer, CLI, benchmark corrections and safe streaming lifecycle. Beta: live capture/playback integration on target devices remains unvalidated.

Companion conversion and evaluation PR: mobius #96. Models: FluidInference/localvqe-coreml.

Implementation

• LocalVqeManager for files/whole clips and LocalVqeStream for arbitrary buffer sizes; v1.3/v1.2, fp32, 16 ms or 256 ms model chunks, CPU default.
• Sample-aligned output, length preserved after flushing, and explicit recurrent state.
• Per-stream FIFO serialization spans asynchronous inference: overlapping pushes, flushes and resets cannot overwrite input/state buffers. Independent streams remain concurrent.
• Queued cancellation leaves the active clip untouched; active cancellation or prediction failure clears the unfinished clip before reuse. Reset waits for earlier operations.
• enhance CLI plus the exploratory enhance-benchmark; fixes the shared WER scorer's swapped insertion/deletion labels and adds regression tests.

Benchmark findings and fixes

The main evaluation covers 800 real AEC-Challenge recordings. Direct reference comparisons support port fidelity, but the complete Hugging Face table is not reproduced. The legacy protocol reproduces the unprocessed baseline exactly; v1.2 far-end echo remains unresolved. The model card declares plain whole-signal ERLE, while its reported values resemble a separately reconstructed gated metric, so the two are now reported separately instead of treating gated ERLE as the confirmed upstream protocol.

The companion PR now includes a standard-library verifier that checks all 13 saved result-file hashes, exact 800-clip coverage, scenario membership, metric availability and every stored aggregate without rerunning inference. The published model-card targets and source revision are pinned alongside it.

The separate training-shard ASR study remains exploratory: 167/200 examples scored after excluding 33 empty reference transcripts. Corrected v1.3 recall is 77.6% versus 44.1% unprocessed; leakage is 1.1% versus 34.0%. Its dataset revision, archive and metadata hashes are now pinned and verified. Malformed metadata, duplicate IDs and missing audio fail loudly; JSON output records exact selected/excluded IDs, machine transcripts and raw metric numerators/denominators.

Benchmark data and investigation retain all tested configurations, the historical 300-recording follow-up, evidence verifier and unresolved differences.

Automated LocalVQE benchmark

The new LocalVQE Benchmark workflow runs both variants over the pinned 200-example study on relevant PR changes, split into 5 contiguous shards (enhance-benchmark --shard i/5) on parallel macOS runners because a single runner exceeded the 60-minute job limit; a final job merges the shard reports in order and verifies them as one 200-example report. Protocol localvqe-asr-v2 fixes Parakeet v3 int8 scoring to CPU, records model/audio SHA256 fingerprints, per-file S/D/I counts and timing, and rejects incomplete or invalid runs. An independent Python verifier recomputes every score and aggregate and checks all selected waveforms against the archive-derived 600-file manifest. Both variants must improve recall and reduce leakage over the same-run unprocessed baseline.

The workflow uploads the JSON report, logs and environment and publishes a job summary. The first cloud run is here (still running as of 2026-09-20 03:19 UTC). All 14 other PR checks passed, including the full macOS test suite.

The full local 200-example real-model run passed on revision de4c0c03: 167 examples scored, 33 empty-reference transcripts excluded. The independent verifier passed coverage, audio/model fingerprints, recomputed scores and improvement gates; all 12 verifier tests passed on the full report, including deliberate corruption of report coverage, counts and fingerprints.

Condition	Recall	WER	Leakage
Unprocessed	42.47%	112.71%	34.03%
LocalVQE v1.3	77.08%	29.48%	0.95%
LocalVQE v1.2	73.34%	33.78%	1.09%

These use protocol localvqe-asr-v2, Parakeet v3 int8, CPU-only and 256ms enhancement exports. They remain exploratory training-shard results with machine transcripts, not a reproduction of the upstream 800-recording table. Historical exploratory figures above are not a frozen baseline for the new explicit CPU protocol.

The macOS suite's cold-cache Sortformer download race is addressed by running speaker-enrollment tests serially before the remaining parallel tests; assertions are retained.

Validation

• Release build, strict Swift formatting and diff checks passed.
• The committed evidence verifier passed for 13 runs × 800 unique clips.
• Standalone harness linked to actual FluidAudio and real v1.3 models passed 13 audio comparisons: overlap ordering, cancellation, reset/reuse, independent streams, buffer sizes and chunk parity.
• Release streaming and whole-clip output were bit-identical, finite and length-preserving on three predetermined real mic/reference recordings.
• Added XCTest regression methods; explicit FLUIDAUDIO_LOCALVQE_MODEL_DIR enables real-model tests in CI and missing supplied bundles fail.
• Local XCTest execution was blocked by this Mac's missing Xcode/XCTest framework. The PR's Xcode CI executes the test suite.

Streaming validation report records coverage and timing limits. Target-device reference timing, route changes, audio callbacks and sustained live-call performance still need application-level validation.

[github-actions]

PocketTTS Smoke Test ✅

Check	Result
Build	✅
Model download	✅
Model load	✅
Synthesis pipeline	✅
Output WAV	✅ (157.5 KB)

_{Runtime: 0m8s}

_{Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon.}

[github-actions]

Parakeet EOU Benchmark Results ✅

Status: Benchmark passed
Chunk Size: 320ms
Files Tested: 100/100

Performance Metrics

Metric	Value	Description
WER (Avg)	7.03%	Average Word Error Rate
WER (Med)	4.17%	Median Word Error Rate
RTFx	5.33x	Real-time factor (higher = faster)
Total Audio	470.6s	Total audio duration processed
Total Time	88.6s	Total processing time

Streaming Metrics

Metric	Value	Description
Avg Chunk Time	0.089s	Average chunk processing time
Max Chunk Time	0.177s	Maximum chunk processing time
EOU Detections	0	Total End-of-Utterance detections

_{Test runtime: 1m41s • 09/20/2026, 01:36 AM EST}

_{RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O}

[github-actions]

Offline VBx Pipeline Results

Speaker Diarization Performance (VBx Batch Mode)

Optimal clustering with Hungarian algorithm for maximum accuracy

Metric	Value	Target	Status	Description
DER	10.4%	<20%	✅	Diarization Error Rate (lower is better)
RTFx	8.41x	>1.0x	✅	Real-Time Factor (higher is faster)

Offline VBx Pipeline Timing Breakdown

Time spent in each stage of batch diarization

Stage	Time (s)	%	Description
Model Download	20.409	16.4	Fetching diarization models
Model Compile	8.747	7.0	CoreML compilation
Audio Load	0.127	0.1	Loading audio file
Segmentation	30.985	24.8	VAD + speech detection
Embedding	124.385	99.7	Speaker embedding extraction
Clustering (VBx)	0.137	0.1	Hungarian algorithm + VBx clustering
Total	124.724	100	Full VBx pipeline

Speaker Diarization Research Comparison

Offline VBx achieves competitive accuracy with batch processing

Method	DER	Mode	Description
FluidAudio (Offline)	10.4%	VBx Batch	On-device CoreML with optimal clustering
FluidAudio (Streaming)	17.7%	Chunk-based	First-occurrence speaker mapping
Research baseline	18-30%	Various	Standard dataset performance

Pipeline Details:

• Mode: Offline VBx with Hungarian algorithm for optimal speaker-to-cluster assignment
• Segmentation: VAD-based voice activity detection
• Embeddings: WeSpeaker-compatible speaker embeddings
• Clustering: PowerSet with VBx refinement
• Accuracy: Higher than streaming due to optimal post-hoc mapping

_{🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 155.5s processing • Test runtime: 2m 39s • 09/20/2026, 01:46 AM EST}

[github-actions]

Sortformer High-Latency Benchmark Results

ES2004a Performance (30.4s latency config)

Metric	Value	Target	Status
DER	30.3%	<35%	✅
Miss Rate	28.2%	-	-
False Alarm	0.9%	-	-
Speaker Error	1.2%	-	-
RTFx	18.1x	>1.0x	✅
Speakers	4/4	-	-

_{Sortformer High-Latency • ES2004a • Runtime: 2m 24s • 2026-09-20T05:40:14.453Z}

[github-actions]

VAD Benchmark Results

Performance Comparison

Dataset	Accuracy	Precision	Recall	F1-Score	RTFx	Files
MUSAN	94.0%	89.3%	100.0%	94.3%	363.7x faster	50
VOiCES	94.0%	89.3%	100.0%	94.3%	372.8x faster	50

Dataset Details

• MUSAN: Music, Speech, and Noise dataset - standard VAD evaluation
• VOiCES: Voices Obscured in Complex Environmental Settings - tests robustness in real-world conditions

✅: Average F1-Score above 70%

[github-actions]

ASR Benchmark Results ✅

Status: All benchmarks passed

Parakeet v3 (multilingual)

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	0.57%	0.00%	4.97x	✅
test-other	1.19%	0.00%	2.93x	✅

Parakeet v2 (English-optimized)

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	0.80%	0.00%	4.21x	✅
test-other	1.00%	0.00%	2.61x	✅

Streaming (v3)

Metric	Value	Description
WER	0.00%	Word Error Rate in streaming mode
RTFx	0.47x	Streaming real-time factor
Avg Chunk Time	1.959s	Average time to process each chunk
Max Chunk Time	2.910s	Maximum chunk processing time
First Token	2.461s	Latency to first transcription token
Total Chunks	31	Number of chunks processed

Streaming (v2)

Metric	Value	Description
WER	0.00%	Word Error Rate in streaming mode
RTFx	0.39x	Streaming real-time factor
Avg Chunk Time	2.319s	Average time to process each chunk
Max Chunk Time	3.047s	Maximum chunk processing time
First Token	2.409s	Latency to first transcription token
Total Chunks	31	Number of chunks processed

_{Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming}

_{25 files per dataset • Test runtime: 8m11s • 09/20/2026, 01:47 AM EST}

_{RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time
Processing time includes: Model inference on Apple Neural Engine, audio preprocessing, state resets between files, token-to-text conversion, and file I/O
Example: RTFx of 2.0x means 10 seconds of audio processed in 5 seconds (2x faster than real-time)}

Expected RTFx Performance on Physical M1 Hardware:

• M1 Mac: ~28x (clean), ~25x (other)
• CI shows ~0.5-3x due to virtualization limitations

_{Testing methodology follows HuggingFace Open ASR Leaderboard}

[github-actions]

Supertonic3 Smoke Test ✅

Check	Result
Build	✅
Model download (incl. VectorEstimatorVariants/ int4 buckets)	✅
Model load	✅
Synthesis pipeline (--ve-variant int4)	✅
Output WAV	✅ (364.7 KB)

_{Runtime: 0m18s}

_{Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf.}

[github-actions]

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Evaluating "who spoke when" detection accuracy

Metric	Value	Target	Status	Description
DER	15.1%	<30%	✅	Diarization Error Rate (lower is better)
JER	24.9%	<25%	✅	Jaccard Error Rate
RTFx	18.23x	>1.0x	✅	Real-Time Factor (higher is faster)

Diarization Pipeline Timing Breakdown

Time spent in each stage of speaker diarization

Stage	Time (s)	%	Description
Model Download	15.100	26.2	Fetching diarization models
Model Compile	6.471	11.2	CoreML compilation
Audio Load	0.162	0.3	Loading audio file
Segmentation	17.255	30.0	Detecting speech regions
Embedding	28.759	50.0	Extracting speaker voices
Clustering	11.504	20.0	Grouping same speakers
Total	57.564	100	Full pipeline

Speaker Diarization Research Comparison

Research baselines typically achieve 18-30% DER on standard datasets

Method	DER	Notes
FluidAudio	15.1%	On-device CoreML
Research baseline	18-30%	Standard dataset performance

Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:

• M2 MacBook Air (2022): Runs at 150 RTFx real-time
• Performance scales with Apple Neural Engine capabilities

_{🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 57.5s diarization time • Test runtime: 3m 0s • 09/20/2026, 01:37 AM EST}