feat(eval): support Common Voice gender classification

[ssss141414]

Summary

This dependent contribution adds WinML CPU FP32 and FP16 support for prithivMLmods/Common-Voice-Gender-Detection, a Wav2Vec2 audio classifier that emits female or male voice labels. Effort L2 ships model recipes plus a class-wide scalar-string label-mapping extension; Outcome L2 is complete, and the highest tested Goal is L3 PASS. The branch is stacked on Draft #1326 at exact head 1bd919b583aa8ccd974e35fa6a1a132ac7e1c9f8; that PR owns the generic audio-classification evaluator and raw-streaming implementation included in this PR's diff against main.

Model metadata

What the model does

This Wav2Vec2 sequence-classification checkpoint accepts 16 kHz speech waveforms and emits two logits interpreted as female or male voice labels (verified). Evidence: pinned checkpoint prithivMLmods/Common-Voice-Gender-Detection@ebbf41293c8dbea45c5bb0de6e3ef77bf89cc2ee, its audio-classification model card, and the pinned Wav2Vec2ForSequenceClassification implementation.

Primary user stories

• A user supplies a speech recording to obtain the checkpoint's female-or-male voice classification for audio analysis (verified from the checkpoint model card and id2label metadata).

Supported tasks

• audio-classification across the checkpoint, Transformers, Optimum ONNX, and WinML surfaces (verified from the checkpoint pipeline tag, Wav2Vec2ForSequenceClassification, the Optimum Wav2Vec2 ONNX task registry, and WinML inspect/config behavior).

Model architecture

Wav2Vec2ForSequenceClassification
|-- Input waveform [batch, samples] at 16 kHz
|-- Wav2Vec2Model
|   |-- Convolutional feature encoder x 7
|   |-- Feature projection (conv width -> 768)
|   |-- Encoder prelude: positional convolution + LayerNorm
|   `-- Transformer encoder layer x 12
|       |-- Self-attention (12 heads, 768 hidden)
|       `-- Feed-forward + residual + normalization
|-- Projector (768 -> 256)
|-- Mean pooling over time
`-- Classifier (256 -> 2 logits)

• Source/confidence: pinned checkpoint config and pinned Wav2Vec2 source structure, with final ONNX regions mapped from node/tensor scopes and topology (mapped).

Validation and support evidence

1. Baseline

The frozen baseline is microsoft/winml-cli main commit 0876e5ae1c98a169a6137e092e0d7b30bf9cee33, WinML 0.3.0. Recipe-free FP32 CPU build passed in 46.1 s, producing an approximately 360.9 MB model with 619/619 tagged ONNX nodes and input_values float32[1,16000] -> logits float32[1,2]. CPU perf over three measured iterations was 59.565 ms mean, 57.39 ms p50, 16.79 samples/s, and +63.55 MB RSS total delta. Eval failed before dataset loading because current-main WinML did not register audio-classification. Generated config already selected export.compatibility.transformers_attention=eager. Optimum exposed audio-classification and feature-extraction both before and after WinML registration, so the probe verdict was VENDOR-ONLY with no WinML-added task.

2. Goal

• Effort: L2.
• Goal ceiling: L3, with a floor of L1.
• Outcome: L2.
• Success definition: march L0-L3 plus mandatory perf, functional smoke, Analyze, compatibility, and CI parity for CPU FP32 and FP16.
• Ceiling history: charter-r2 reissued only the planner-owned final-artifact semantic mapping after technical PASS; it did not change or downgrade the L3 ceiling.

3. Outcome

The shipped tier is L2 and the highest Goal verdict is L3 PASS, with full technical coverage. Required tuples CPUExecutionProvider/cpu/fp32 and CPUExecutionProvider/cpu/fp16 both passed; no tuples are deferred. The contribution adds two CPU recipes and a generic, schema-driven scalar Value('string') mapping path. The model branch deliberately depends on Draft #1326 at 1bd919b583aa8ccd974e35fa6a1a132ac7e1c9f8; #1326 retains ownership of evaluator registration, the base audio-classification implementation, raw Audio(decode=False) plus SoundFile decoding, fixed-window aggregation, and existing ClassLabel/sequence behavior. This PR adds only exact scalar-string mapping validation/resolution, its focused tests, schema help, and the model recipes.

Lane A knowledge was appended separately in gim-home/ModelKitArtifacts#254 at commit 0c896ff8e9eff8e26f41c3ee40058e960f149e85, with published wav2vec2.json SHA-256 14d78c8ae620e646f72a9255a924767c3b273633844e3219c7a6a9e0dcb279c0. No methodology friction observed.

4. Per-EP/device/precision results and Functional smoke Eval

Tier	EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM total delta
L0	CPUExecutionProvider / cpu	fp32	PASS; build 54.5 s	-	-	-	-
L0	CPUExecutionProvider / cpu	fp16	PASS; build 66.8 s	-	-	-	-
L1	CPUExecutionProvider / cpu	fp32	PASS	51.024 ms	51.123 ms	19.6 samples/s	+63.52 MB
L1	CPUExecutionProvider / cpu	fp16	PASS	61.586 ms	62.633 ms	16.24 samples/s	+41.48 MB

L2 parity passed against pinned PyTorch Wav2Vec2ForSequenceClassification@ebbf41293c8dbea45c5bb0de6e3ef77bf89cc2ee using deterministic float32 input [1,16000]. FP32 cosine was 0.9999999999999206, max absolute error 8.821487426757812e-06, mean absolute error 8.225440979003906e-06, finite output, and matching argmax. FP16 cosine was 0.9999999999953266, max absolute error 0.0008280277252197266, mean absolute error 0.0007888078689575195, finite output, and matching argmax.

Functional smoke Eval: L3 PASS on final candidate e4d19ba1d4af532ba4cf0b047ff2fb395ac9d868, FP32 CPU, using AsmaaQ/gender_audio_1080@47f6563cf1b819cbf1cbc8569a8dfe8fc426bab3, config default, split test, streaming, no shuffle. Exact mapping was female -> 0, male -> 1, matching the checkpoint. Selection took the first eligible row for each exact mapped label in source order. Accounting was requested/selected/processed/rejected = 2/2/2/0, with no rejection reasons; both classes were represented. The female row was mono 16 kHz PCM16, 55,872 frames (3.492 s), four 16,000-sample windows; the male row was mono 16 kHz PCM16, 81,216 frames (5.076 s), six windows. Fan-out caps were 2 rows, 2 classes, 10 total windows, 1 beam, and 0 other expansion. Accuracy was 1.0; macro-F1 was 1.0; represented classes were 2/2 and class coverage was 1.0.

This is bounded real-data end-to-end operability evidence only. Two rows are not representative accuracy and are not a benchmark-quality claim. Only FP32 CPU Eval was measured; no FP16 or other-EP accuracy is implied. The former blocker was rejection of scalar datasets.Value('string') targets before inference; the new capability accepts them only through a complete, exact, explicit mapping to checkpoint class IDs.

5. Delta

Recipes added:

• examples/recipes/prithivMLmods_Common-Voice-Gender-Detection/cpu/cpu/audio-classification_fp32_config.json: /export/compatibility/transformers_attention is eager, identical to the generated baseline value.
• examples/recipes/prithivMLmods_Common-Voice-Gender-Detection/cpu/cpu/audio-classification_fp16_config.json: /export/compatibility/transformers_attention is eager, identical to baseline; /quant is the existing FP16 quantization object, while baseline has no /quant.

Parsed FP32 recipe content equals the baseline exactly. Parsed FP16 content equals baseline after substituting only the FP16 /quant object. Recipe-free Eval acceptance passed through the final built FP32 model. examples/recipes/README.md remains untouched.

Code delta relative to dependency #1326:

• src/winml/modelkit/eval/audio_classification_evaluator.py: _validate_target_schema, _target_for_row, and _validate_scalar_string_mapping add exact scalar-string mapping support.
• src/winml/modelkit/utils/eval_utils.py: _AUDIO_CLASSIFICATION_SCHEMA documents explicitly mapped scalar strings.
• tests/unit/eval/test_audio_classification_evaluator.py: focused class-wide regressions cover the new and preserved paths.

Bug fix explanation:

1. Symptom and trigger: a valid audio-classification dataset with scalar datasets.Value('string') targets, such as female and male, was rejected during target-schema validation before inference even when the caller supplied an explicit mapping.
2. Root cause: the inherited evaluator accepted scalar ClassLabel and sequence-valued label forms but had no scalar-string validation/resolution branch, so schema validation rejected the feature before exact mapping semantics could be applied.
3. Changed symbols and mechanism: _validate_target_schema recognizes scalar string features only when _validate_scalar_string_mapping proves a non-empty, one-to-one, complete mapping to contiguous checkpoint IDs; _target_for_row resolves each observed value by exact key lookup. _AUDIO_CLASSIFICATION_SCHEMA exposes that contract.
4. General rule: behavior is derived from dataset feature type, explicit caller mapping, and checkpoint class inventory. Production code contains no model ID, dataset ID, female/male special case, inferred ordinal, fuzzy match, or case folding.
5. Compatibility and blast radius: scalar ClassLabel, sequence ClassLabel/string multi-label behavior, metrics, raw streaming decode=False, SoundFile decoding, and fixed-window aggregation are preserved. Intentional change is limited to explicitly mapped scalar strings. Missing, empty, non-integer, duplicate-destination, out-of-range, incomplete, and unmapped observed values fail closed before inference.
6. Regression evidence: the focused evaluator suite passed 18 tests in 11.18 s; the previously failing scalar-string mapping test passed; related eval/metric/recipe/command coverage passed 198 tests in producer verification. Tester CI passed 3,648 tests with 9 skipped in the commands group and 1,538 tests with 6 skipped and 2 expected failures in the models group. Ruff, full-package mypy, and license checks all passed.

6. Analyze summary - component level and op level

Analyze completed as ANALYZE-PARTIAL-SUCCESS with exit code 1 because six targets have no populated static rules. Static rule classification is not accelerator runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Explicit gaps
fp32	feature encoder; feature projection; encoder prelude; 12x encoder layers; projector; pooling; classifier	392/397 mapped; planner-frozen with explicit gaps	5 cross-component optimizer transition nodes
fp16	same seven regions	392/399 mapped; planner-frozen with explicit gaps	same 5 transitions plus 2 graph-boundary Casts

The five shared gaps are the final feature-extractor Gelu, feature-projection output Reshape, first encoder-layer Gemm input Reshape, projector Gemm input Reshape, and projector output Reshape. They remain unattributed because adjacency alone is not accepted as semantic ownership evidence. FP16 additionally has input/output precision-adapter Casts. Every source component is mapped; resolving transition ownership would require exporter provenance or an instrumented unoptimized export with stable correspondence, and requires no product repair.

Op-level summary

Artifact	Graph	Dominant ops	Rule-backed EP roll-up
fp32	397 ops / 14 types	Reshape 126; Gemm 75; Transpose 51; Add 38; LayerNormalization 26; MatMul 24	Fully supported: NvTensorRTRTX/GPU, QNN/NPU, QNN/GPU, OpenVINO/NPU, OpenVINO/GPU, OpenVINO/CPU
fp16	399 ops / 15 types	same dominant counts, plus 2 Casts	Fully supported: NvTensorRTRTX/GPU, QNN/NPU, QNN/GPU, OpenVINO/NPU, OpenVINO/GPU, OpenVINO/CPU

Rule-less targets CUDA/GPU, MIGraphX/GPU, TensorRT/GPU, DML/GPU, CPU/CPU, and VitisAI/NPU have no populated classifications. These are rules-availability gaps, not runtime failures or measured support claims.

7. Reproduce commands

Prerequisites: Windows PowerShell, Git, uv, internet access to the public Hugging Face checkpoint/dataset, and sufficient disk/RAM for approximately 361 MB FP32 and 180 MB FP16 external-weight payloads. The commands clone the exact dependent branch, verify its SHA and dependency ancestry, install the locked environment, clone the public static-rule repository, and reproduce build, perf, Analyze, bounded Eval, and quality checks.

$CandidateRef = 'ssss141414/add-common-voice-gender-dependent'
$ExpectedCandidate = 'e4d19ba1d4af532ba4cf0b047ff2fb395ac9d868'
$ExpectedDependency = '1bd919b583aa8ccd974e35fa6a1a132ac7e1c9f8'

git clone https://github.com/microsoft/winml-cli.git winml-cli-common-voice-gender
Set-Location winml-cli-common-voice-gender
git fetch origin $CandidateRef
$ResolvedCandidate = (git rev-parse 'FETCH_HEAD^{commit}').Trim()
if ($ResolvedCandidate -ne $ExpectedCandidate) { throw "remote ref resolved to $ResolvedCandidate, expected $ExpectedCandidate" }
git merge-base --is-ancestor $ExpectedDependency $ResolvedCandidate
if ($LASTEXITCODE -ne 0) { throw 'candidate does not descend from the exact dependency SHA' }
git checkout --detach $ResolvedCandidate
if ((git rev-parse HEAD).Trim() -ne $ExpectedCandidate) { throw 'wrong candidate SHA after checkout' }
if (git status --porcelain) { throw 'checkout is not clean' }

uv sync --locked --all-extras --all-groups
$OUT = Join-Path $PWD 'temp/common-voice-gender-repro'
$FP32 = Join-Path $OUT 'fp32'
$FP16 = Join-Path $OUT 'fp16'
New-Item -ItemType Directory -Force -Path $OUT | Out-Null

uv run --no-sync winml build -c examples/recipes/prithivMLmods_Common-Voice-Gender-Detection/cpu/cpu/audio-classification_fp32_config.json -m prithivMLmods/Common-Voice-Gender-Detection -o $FP32 --rebuild
uv run --no-sync winml build -c examples/recipes/prithivMLmods_Common-Voice-Gender-Detection/cpu/cpu/audio-classification_fp16_config.json -m prithivMLmods/Common-Voice-Gender-Detection -o $FP16 --rebuild --precision fp16
uv run --no-sync winml perf -m (Join-Path $FP32 'model.onnx') --device cpu --ep cpu --iterations 10 --warmup 3 --memory --format json -o (Join-Path $OUT 'perf-fp32.json')
uv run --no-sync winml perf -m (Join-Path $FP16 'model.onnx') --device cpu --ep cpu --iterations 10 --warmup 3 --memory --format json -o (Join-Path $OUT 'perf-fp16.json')

git clone https://github.com/gim-home/ModelKitArtifacts.git (Join-Path $OUT 'ModelKitArtifacts')
$env:WINMLCLI_RULES_DIR = (Resolve-Path (Join-Path $OUT 'ModelKitArtifacts/rules')).Path
uv run --no-sync winml analyze --model (Join-Path $FP32 'model.onnx') --ep all --device all --output (Join-Path $OUT 'analyze-fp32.json')
uv run --no-sync winml analyze --model (Join-Path $FP16 'model.onnx') --ep all --device all --output (Join-Path $OUT 'analyze-fp16.json')
Remove-Item Env:WINMLCLI_RULES_DIR

$Mapping = Join-Path $OUT 'gender-label-mapping.json'
'{"female":0,"male":1}' | Set-Content -Path $Mapping -Encoding ascii
uv run --no-sync winml eval -m (Join-Path $FP32 'model.onnx') --model-id prithivMLmods/Common-Voice-Gender-Detection --task audio-classification --ep cpu --device cpu --dataset AsmaaQ/gender_audio_1080 --dataset-revision 47f6563cf1b819cbf1cbc8569a8dfe8fc426bab3 --split test --streaming --samples 2 --no-shuffle --column input_column=path --column label_column=gender --label-mapping $Mapping -o (Join-Path $OUT 'functional-smoke.json')

uv run --no-sync pytest tests/unit/commands tests/unit/config tests/unit/build tests/unit/compiler tests/unit/session tests/unit/eval --tb=short --no-cov -m "not e2e and not npu and not gpu"
uv run --no-sync pytest tests/unit/models tests/unit/loader tests/unit/datasets tests/unit/export --tb=short --no-cov -m "not e2e and not npu and not gpu"
uv run --no-sync ruff check src/ tests/
uv run --no-sync mypy -p winml.modelkit
uv run --no-sync pre-commit run insert-license --all-files

The Eval command is a two-row functional smoke only, not representative accuracy. Analyze is static rule classification, not accelerator runtime evidence.

Dependency and provenance

• Dependency: Draft microsoft/winml-cli#1326, exact head 1bd919b583aa8ccd974e35fa6a1a132ac7e1c9f8, owns the generic audio evaluator implementation described above. This dependent PR does not duplicate or rewrite that ownership.
• Candidate: ssss141414/add-common-voice-gender-dependent, exact head e4d19ba1d4af532ba4cf0b047ff2fb395ac9d868; the dependency head is its verified ancestor.
• Lane A: Draft gim-home/ModelKitArtifacts#254, exact commit 0c896ff8e9eff8e26f41c3ee40058e960f149e85, published knowledge-file SHA-256 14d78c8ae620e646f72a9255a924767c3b273633844e3219c7a6a9e0dcb279c0.