feat(enhancement/localvqe): Core ML export and benchmark evaluation - #96
Open
Alex-Wengg wants to merge 11 commits into
Open
Alex-Wengg wants to merge 11 commits into
Alex-Wengg wants to merge 11 commits into
Conversation
…/v1.2 (AEC + NS + dereverb) Explicit-state re-formulation of the upstream whole-clip PyTorch graph (33 in_*/out_* tensors: conv time histories, AlignBlock delay windows, S4D hidden state, CCM history, DCT PCM history and OLA tail) so one Core ML call processes N consecutive 256-sample hops. Exports 16 ms and 256 ms chunk sizes per checkpoint; the audio is identical, only per-call overhead differs. fp32 only: fp16 drops parity vs the PyTorch reference from 102 dB to 5 dB (CPU) / 33 dB (ANE) because the power-law front-end epsilons underflow and the S4D recurrence accumulates error. Output level follows the shipped GGML engine (no OLA divisor, 2x the PyTorch reference), and the 256-sample output delay is documented in the model metadata. Verified against the upstream GGML regression fixture (2.3e-6) and the GGML CLI on the upstream double-talk demo (80 dB SNR from Swift). Requested in FluidInference/FluidAudio#49. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015S4u7dmbe4skFjUNMpodud
…MOS / ERLE / DNSMOS) + GGML control Renders the 800-clip ICASSP 2022 blind set through the Swift or upstream GGML CLI and scores it with Microsoft's local AECMOS model under the challenge trimming rules, blind ERLE and DNSMOS OVRL. compare-renders.py does an aligned A/B of two engines on identical samples. Core ML == GGML: per-scenario echo/deg/ERLE identical to two decimals, per-clip echo delta mean -0.0001 (p95 0.02) once the GGML CLI's 256-sample output delay, tail truncation and 16-bit writer (quantisation of ~1e-4 RMS residuals, wrap-around above full scale) are normalised. v1.2 reproduces the upstream table's single-talk rows; double-talk rows use a segment we could not reproduce (baseline 2.17 vs 2.67), and the table's v1.3 far-end echo MOS is ~1 above what the published v1.3 weights give under this protocol. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015S4u7dmbe4skFjUNMpodud
- render-blind.sh: atomic temp+mv outputs, propagate per-clip failures (exit 1, count check) instead of swallowing them - compare-renders.py: compare whole hops only (GGML zero-fills the trailing partial hop); median aligned SNR 49 -> 84 dB, deg p95 delta 0.0063 -> 0.0004 - README: numerical equivalence not "bit-matched"; single-talk agreement stated with the 38.2 vs 40.6 dB ERLE exception; double-talk mismatch described as an unresolved upstream protocol discrepancy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015S4u7dmbe4skFjUNMpodud
…ction, gated ERLE, rated-region DNSMOS --protocol upstream selects the legacy AECMOS model (Run_1663829550, no scenario marker) over the first 20 s of each recording: this reproduces the LocalVQE README / HF model-card unprocessed baseline exactly (2.67 / 2.56 / 1.90 / 2.13 / 5.00) and the model rows to within ~0.05 echo MOS from the GGML CLI's raw output. The default --protocol challenge keeps the scenario-marker model with the AECMOS README trimming rules. Adds the technical-report blind-ERLE definition (energy pooled over 512/256 frames with loopback RMS above its 75th percentile and mic RMS below 2x loopback) alongside the plain energy ratio, and scores DNSMOS on the rated segment by default (--dnsmos-region whole for the old behaviour); the rated region reproduces the published doubletalk OVRL (2.89 / 2.85). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015S4u7dmbe4skFjUNMpodud
…etract the 'not from published weights' claim Challenge-protocol reference table (gated ERLE, rated-segment DNSMOS) plus an upstream-protocol reproduction table against the HF card for v1.3 and v1.2, Core ML and GGML rows separate. Baseline, all doubletalk and near-end cells, ERLE (~1 dB) and OVRL (0.06) reproduce; v1.3 far-end echo within 0.04 from raw GGML output; v1.2 far-end echo rows remain 0.2-0.4 above the card from either runtime and are reported as unexplained. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015S4u7dmbe4skFjUNMpodud
… protocol; per-metric agreement claims --protocol upstream set the AECMOS start to 0 and DNSMOS 'rated' then scored the whole recording (2.40 instead of 3.13 on a doubletalk clip); the rated segment is now computed from the challenge rules regardless of the AECMOS protocol. Per-clip OVRL under --protocol upstream now equals the challenge run's. README: replace 'ERLE ~1 dB / OVRL 0.06 everywhere' with per-metric numbers; v1.2 far-end differs on echo (+0.29/+0.15), gated ERLE (+1.9/+0.7 dB) and OVRL (+0.09/+0.05), not only echo MOS. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015S4u7dmbe4skFjUNMpodud
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015S4u7dmbe4skFjUNMpodud
…ains ERLE/deg, echo MOS cells remain open Adds render-torch.py (PyTorch-reference renders with --dmax / --no-fold / --arch-version diagnostics). Rendering v1.2 at the pre-v1.2 delay window (dmax 32, which the reference config left on the day the row was published) reproduces the card's far-end ERLE and deg; temperature 1.0, ReLU6, GGML output emulation and every scorer/segment variation are rejected. The far-end echo MOS cells stay unexplained. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015S4u7dmbe4skFjUNMpodud
Alex-Wengg
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streaming Core ML exports and evaluation tooling for LocalVQE v1.3/v1.2 acoustic echo cancellation, noise suppression and dereverberation. Companion Swift consumer and streaming hardening: FluidAudio #930. Published models: FluidInference/localvqe-coreml.
Conversion
in_*/out_*state tensors; v1.3 4.8M and v1.2 1.3M checkpoints, 16 ms and 256 ms chunks, fp32, iOS 17/macOS 14 targets.Evaluation and evidence
The challenge protocol remains the quality reference; the legacy first-20-second AECMOS protocol is a separate model-card reproduction check. DNSMOS uses the rated segment independently of AECMOS protocol.
The model card explicitly defines ERLE as the plain whole-signal energy ratio, but its reported values resemble a separately reconstructed gated metric. Both values are now kept separately labelled; the gated value is not claimed as the card's confirmed protocol. The model-card targets and source revision are pinned in
upstream-model-card.json.The validation index publishes 13 saved runs, each containing the same 800 real recordings, including five full-corpus PyTorch configuration diagnostics. Per-recording CSVs, their hashes and the common manifest are included. These are existing measurements; this update did not rerun the benchmark. The saved upstream runs skipped DNSMOS and explicitly mark it unavailable; separate challenge files contain rated DNSMOS.
verify-benchmark-evidence.pyindependently verifies all 13 CSV hashes, exact manifest coverage, scenario membership, DNSMOS availability and every stored aggregate using only the Python standard library.The reproduction report also records:
Status: port fidelity validated; published table only partially reproduced. The baseline matches exactly. The v1.2 far-end echo cells and the public ERLE protocol conflict remain unresolved. Results from different configurations and metric definitions are kept separate.
Scorer readiness
The blind scorer now enforces the exact committed 800-stem manifest, fails on missing loopbacks/renders and invalid audio, and records selected stems, sample counts, scorer/model hashes and package versions. Diagnostic subsets are explicitly marked incomplete. Unscored DNSMOS and ineligible gated ERLE serialize as JSON null; model-load errors are checked before spawning workers.
Five fixed existing Core ML renders (first sorted stem per scenario) reproduced the saved AECMOS, degradation, plain/gated ERLE and DNSMOS measurements exactly. Five manifest regression tests passed, and all 13 stored 800-clip runs still pass the evidence verifier. A second check on those same five clips validated the no-DNSMOS report path. This is a bounded scorer check, not a fresh full-corpus enhancement run.
No model weights, recordings, local plans or machine-specific build products are added. The Swift consumer remains beta pending live call-pipeline validation; its queue/cancellation fixes and real-model smoke results are documented in the companion PR.