feat(eval): add shared CTC ASR support for facebook/mms-1b-all

[ssss141414]

Summary

This follow-up to merged PR #1177 adds shared, metadata-gated CTC automatic speech recognition evaluation for facebook/mms-1b-all, fixes generic HF --no-optimize stage control, and retains verified CPU FP32/FP16 recipes. The contribution ships at Effort L2 / Outcome L2 and reaches the committed Goal L3 with CPU builds, performance, PyTorch parity, and bounded two-row FLEURS functional smokes. The final successor applies DatasetConfig.shuffle and seed before the sample cap with deterministic bounded-buffer semantics, selects identical original source indices in streaming and non-streaming modes for a fixed seed, preserves shuffle=false first-N Polish behavior, and retains the complete source-index/dataset-ID/redacted-media composite identity boundary before inference. The exact D6 candidate is 4b528133fef8cbeed4e9a3aba9be42d22a3c0221 on current main and merge base e28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0; it is a byte-equivalent replay of the accepted D5 series.

Model metadata

What the model does

facebook/mms-1b-all is a 1 billion-parameter multilingual automatic speech recognition checkpoint. It accepts mono speech waveforms sampled at 16 kHz and emits frame-level CTC token logits that the checkpoint processor greedily decodes into text; published language adapters cover 1,162 languages.

• Evidence: pinned model card and AutoConfig/AutoProcessor metadata at revision 3d33597edbdaaba14a8e858e2c8caa76e3cec0cd, plus current-main WinML export metadata.
• Confidence: verified.

Primary user stories

• A user supplies a 16 kHz speech recording to obtain a text transcription in the checkpoint's selected language. Evidence: the pinned model-card inference example uses AutoProcessor, argmax over logits, and processor.decode. Confidence: verified.
• A user selects a published MMS language adapter and supplies speech in that language to obtain multilingual transcription without replacing the shared acoustic backbone. Evidence: the pinned model card documents tokenizer.set_target_lang and model.load_adapter across 1,162 supported languages. Confidence: verified.

Supported tasks

• automatic-speech-recognition across the checkpoint, Transformers, and WinML surfaces. Evidence: the checkpoint pipeline tag, Wav2Vec2ForCTC architecture, metadata-derived WinML registration for wav2vec2, and recipe-free winml inspect/build resolution. Confidence: verified.

Model architecture

Wav2Vec2ForCTC
+-- Wav2Vec2Model
|   +-- Feature encoder: 7 x Conv1d + LayerNorm + GELU (waveform -> 512 features; total stride 320)
|   +-- Feature projection: LayerNorm + Linear 512 -> 1280 + dropout
|   `-- Stable-layer-norm encoder
|       +-- Positional Conv1d (kernel 128, groups 16) + GELU
|       `-- Encoder layer x 48 (hidden 1280)
|           +-- Self-attention (16 heads; Q/K/V/out 1280 -> 1280)
|           +-- Feed-forward (1280 -> 5120 -> 1280, GELU)
|           `-- Attention adapter (1280 -> 16 -> 1280, ReLU)
+-- Final dropout
`-- CTC language-model head (1280 -> 154 logits per frame)

• Source/confidence: pinned AutoConfig, Transformers 5.14.1 Wav2Vec2ForCTC source and meta-device module tree, and current-main export hierarchy metadata (verified).

Validation and support evidence

1. Baseline

• Measurement baseline: microsoft/winml-cli main at 0876e5ae1c98a169a6137e092e0d7b30bf9cee33, WinML 0.3.0. L0/L2 values retain execution provenance cac2526b620df08fdbcc26ef60fe590479e55a0e; L1 Perf and Analyze retain C5 execution provenance ac8acd690b48828eb7dfdd38c4aa5de9e3bcb14d with artifacts from cac2526b620df08fdbcc26ef60fe590479e55a0e; L3 retains artifact execution cac2526b620df08fdbcc26ef60fe590479e55a0e and evaluator execution 3440c5401d0111f06456b00662469daad2a6f0f6. Shipment identity is current main and merge base e28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0, exact D6 head 4b528133fef8cbeed4e9a3aba9be42d22a3c0221, parent 34bc2a0e6181e8c9f0cc27fddf9ebfaf2d1dbfe7, and tree b9b8745bb942e2f359cf583a642c3d8cc0112282; current main is its ancestor.
• Historical context: merged PR recipe(mms-1b-all): add verified CPU fp32/fp16 configs #1177 (b91152fb07a8a769a4756e94006bf83ede7073c9 on base 8079522ee692645569508d11abd4a0ea45a40987) established recipe-only CPU FP32/FP16 support and explicitly did not establish WER or benchmark accuracy. This is a new follow-up branch and PR; recipe(mms-1b-all): add verified CPU fp32/fp16 configs #1177 is not reopened or modified.
• Optimum probe: vendor tasks were empty; after WinML registration, audio-classification, audio-frame-classification, audio-xvector, automatic-speech-recognition, and feature-extraction resolved. Verdict: WINML-ONLY.
• Recipe-free baseline build: PASS in 392.6 s (427.13 s wall), with Export 229.2 s and Optimize 137.7 s. It resolved AutoModelForCTC / automatic-speech-recognition and produced input_values float32 [1,16000] -> logits float32 [1,49,154]. It also exposed the generic defect that Optimize ran despite explicit --no-optimize because the HF pipeline dropped skip_optimize.
• Baseline CPU perf: PASS; mean 330.30 ms, p50 326.50 ms, p90 348.65 ms, throughput 3.03 samples/s, total RAM delta +112.1 MB.
• Baseline Eval: exit 1 before dataset loading with Task 'automatic-speech-recognition' is not supported. This set the Goal floor at L1 and established the shared CTC evaluator gap.
• Starting auto-config used Wav2Vec2ForCTC; the shipped recipes use metadata-derived AutoModelForCTC and add the bounded Eval contract.

2. Goal

• Committed axes: Effort L2, Goal ceiling L3, Outcome L2; no ceiling downgrade or re-issued charter.
• Target coverage: CPUExecutionProvider / cpu, required precisions FP32 and FP16.
• Success definition: both recipes build and run on CPU; identical-input PyTorch comparisons pass; one final-candidate FP32 CPU FLEURS functional smoke emits meaningful ASR WER/CER with bounded sample and window fan-out.

3. Outcome

• Final tester verdict: PASS, technical state CLOSED, and dependency public acquisition state CLOSED; all 14 fresh D6 public rows are closed and the independently rewalked 49-file seal matches byte-for-byte. Highest Goal verdict: L3 PASS.
• Coverage: full; deferred tuples: none.
• Shipped recipes: examples/recipes/facebook_mms-1b-all/cpu/cpu/automatic-speech-recognition_fp32_config.json and examples/recipes/facebook_mms-1b-all/cpu/cpu/automatic-speech-recognition_fp16_config.json.
• Shipped shared code: src/winml/modelkit/eval/ctc_asr_evaluator.py, evaluator registration/defaults/schema and generic wrapper registration, plus HF build stage-control forwarding in src/winml/modelkit/commands/build.py.
• Quality: fresh exact-D6 public runs passed 50/50 CTC evaluator tests, 116/116 build compatibility tests, 15/15 targeted VED export tests, the owning models partition with 1,538 passed / 6 skipped / 2 xfailed, full Ruff, scoped Ruff, insert-license, and mypy over 440 source files. Scoped format checked the VED implementation/test plus the accepted CTC evaluator/test and build/test pairs: the two VED files would reformat on both exact main and D6, while the four D6-owned files are formatted. Full-tree format reports the same 96 inherited failure paths on exact main and D6, with no D6 regression. Unaffected r5 quality rows retain C5 execution provenance ac8acd690b48828eb7dfdd38c4aa5de9e3bcb14d; optim retains 3440c5401d0111f06456b00662469daad2a6f0f6. Earlier full Eval and five-partition results remain retained evidence under their original provenance and are not relabeled as fresh D6 execution.
• Model knowledge: existing wav2vec2-027 now records DatasetConfig.shuffle/seed before bounded selection, deterministic streaming/non-streaming original-index equivalence, bounded finite-buffer consumption, pre-shuffle original-source provenance, and preserved composite source-index/dataset-ID/redacted-media identity. No duplicate finding was created.
• Lane A binding: draft PR gim-home/ModelKitArtifacts#254, exact head b7efe2b77c59eafbd13c48c4cb9beb9336df8305, label model-scale-by-skill, check GitOps/AdvancedSecurity: COMPLETED/SUCCESS on that exact head. Refined _meta-112 binds the paired tester/reviewer contracts to the same shuffle-before-cap, fixed-seed mode-equivalence, bounded-consumption, original-provenance, and preserved composite-identity requirements.
• Methodology declaration: no new methodology friction was introduced by the byte-equivalent D6 current-main rebase. The sole main increment is confined to VisionEncoderDecoder/TrOCR behavior, has no owned-path overlap or CTC dispatch/evaluator reach, and passed its fresh targeted and owning-partition checks. Existing wav2vec2-027 and _meta-112 remain the durable binding for shuffle-before-cap, deterministic streaming/non-streaming original-index equivalence, bounded buffering, and composite row/media provenance; D6 refreshes candidate, VED-compatibility, public-acquisition, and seal provenance only, with no duplicate finding or new verdict shape.

Final closure rows

Row	State	Verdict	Exact evidence
Seeded bounded selection	CLOSED	PASS	Seed 123 selected [7, 4, 0, 2] in both modes; seed 456 selected [4, 6, 3, 2]; 1,200-row selection matched at [15, 682, 592, 53]; infinite-style source yielded 1,007 rows under ceiling 1,100
Composite identity before inference	CLOSED	PASS	Missing source provenance raised DatasetValidationError with 0 transcription calls and 0 model calls
Pinned Polish shuffle=false	CLOSED	PASS	Both modes selected IDs [1525, 1657], original indices [0, 1], and basenames 10018492969996036091.wav, 10288018704489549018.wav
Focused selection/identity tests	CLOSED	PASS	19 passed, 31 deselected
Focused compatibility tests	CLOSED	PASS	9 passed, 41 deselected
Full CTC evaluator	CLOSED	PASS	50 passed
Full Eval	CLOSED	PASS	686 passed, 1 warning
Fresh D6 CTC/build compatibility	CLOSED	PASS	50 CTC tests and 116 build compatibility tests passed from exact public D6 imports
VED no-impact boundary	CLOSED	PASS	The sole main increment is vision_encoder_decoder.py; no owned-path overlap or CTC reach; 15 targeted VED tests and the models partition (1,538 passed, 6 skipped, 2 xfailed) passed
Quality and static gates	CLOSED	PASS	Full/scoped Ruff, insert-license, and mypy for 440 source files passed
Scoped format main/D6 parity	CLOSED	PASS	The same two VED files would reformat on main and D6; the four scoped D6-owned CTC/build files are formatted
Full-format current-main/D6 parity	CLOSED	PASS	Exact main and D6 report the same 96 inherited failure paths; no D6 regression
Byte-equivalent r6 replay	CLOSED	PASS	8/8 commits range-diff equal; 8/8 ordered canonical stable patch IDs equal; 10/10 changed path statuses and 10/10 final blobs equal; no extra paths
Reused stage provenance	CLOSED	PASS	L0/L2 at cac2526b; L1/Analyze at C5 ac8acd69 with artifacts at cac2526b; L3 at cac2526b + evaluator 3440c540; unaffected quality at C5/3440c540; none relabeled as fresh D6 execution
Public acquisition, commands, and seal	CLOSED	PASS	Fresh empty public clone advertised and detached exact D6; tree, parent, merge base, 8 commits, 10 paths, canonical patch IDs, clean state, and locked audio initialization verified; all 14 fresh rows closed; independent 49-file rewalk matched
Canonical stable patch authority was recomputed with a byte-preserving git show --pretty=format: --no-ext-diff --binary <commit> to git patch-id --stable pipeline. Ordered IDs are 2e0f73d16f95a28fce9ea36da84c2cc7fa68850e, e84028b1e13c416b39a9133ff930159e79689560, 902bae581a224965e269eb3546a0bb9ad565049a, 6e1039c7d747d5c39658d3601864a9ddf0312098, 1661be1ec70cd0b1232133b14b912a99ee84e4db, 6dfe21004adfba20ec0fe86966160008bd57ea1e, a09c8752c26cfb1609c68ef107ad003c0c0b3a51, and 1fb43c8e1bb54e3fceca1c490d27fa9d9b49fa59. They match the accepted D5 and exact D6 series in order. Earlier serialized patch-ID fields had no generating command and are noncanonical historical evidence; they are not used for acceptance.

4. Per-EP/device/precision results and Functional smoke Eval

Tier	EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM delta	VRAM delta
L0	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-	-
L0	CPUExecutionProvider / cpu	fp16	PASS	-	-	-	-	-
L1	CPUExecutionProvider / cpu	fp32	PASS	347.594 ms	350.13 ms	2.88 samples/s	+142.06 MB	0.0 MB
L1	CPUExecutionProvider / cpu	fp16	PASS	421.026 ms	418.761 ms	2.38 samples/s	+70.61 MB	0.0 MB

FP16 is slower on this CPU; no speedup is claimed. Both runs used 20 iterations and 3 warmups with the same named float32 [1,16000] input.

L2 precision	Verdict	Cosine similarity	SQNR	Max absolute difference
fp32	PASS	0.9999999999981509	110.60827500976687 dB	0.00014126300811767578
fp16	PASS	0.9999999214712589	64.86593539245807 dB	0.023023605346679688

Functional smoke Eval: PASS with original public execution provenance at accepted head 3402b7b4b1990a0932551a3d69d3838df0e70bda, carried to exact D6 4b528133fef8cbeed4e9a3aba9be42d22a3c0221 by the byte-equivalent accepted series and 10/10 final-blob equality; it was not freshly re-executed on D6. The FP32 CPU smoke used a fresh exact-head MMS FP32 build from that accepted public checkout and google/fleurs revision 70bb2e84b976b7e960aa89f1c648e09c59f894dd, config en_us, validation split. Deterministic no-shuffle source-order selection processed source IDs [1548, 1620] at indices [0, 1], exactly 2 rows with 0 skipped and 0 rejected; emitted redacted row provenance, schema, label semantics, prediction semantics, and accounting were verified. Fan-out was capped at 2 dataset rows, batch size 1, at most 64 fixed waveform windows per utterance, no candidate-label/prompt expansion, no beams, and one full CTC decode across accepted windows. The processor-selected adapter language remained eng, independently of the FLEURS locale en_us. Raw metrics were WER 0.8541666666666666 and CER 0.3684210526315789.

This is functional-smoke-only accounting: it proves real-audio decode, complete-utterance preprocessing, ONNX inference, CTC decoding, and metric emission. It is not representative accuracy or benchmark quality, and it does not claim accuracy for FP16 or other EPs. The former blocker was the absent ASR task path; the shared evaluator now supplies SoundFile bytes/path decoding, metadata-derived resampling/language/vocabulary handling, static/dynamic waveform processing, processor-owned CTC decode, WER/CER, and fail-closed row accounting.

5. Delta

Recipe delta from frozen auto-config

Recipe	JSON pointer	Baseline	Shipped
FP32	/loader/model_class	Wav2Vec2ForCTC	AutoModelForCTC
FP32	/eval	null	Pinned google/fleurs@70bb2e84b976b7e960aa89f1c648e09c59f894dd, en_us, validation, two rows, deterministic ordering, audio / transcription
FP16	/loader/model_class	Wav2Vec2ForCTC	AutoModelForCTC
FP16	/quant	null	mode=fp16, fp16_keep_io_types=true
FP16	/eval	null	Pinned google/fleurs@70bb2e84b976b7e960aa89f1c648e09c59f894dd, en_us, validation, two rows, deterministic ordering, audio / transcription

The delta is reducible to metadata-derived CTC behavior plus the checkpoint's bounded dataset/precision recipe intent. Recipe-free inspect acceptance passed with wav2vec2 -> automatic-speech-recognition -> AutoModelForCTC -> Wav2Vec2OnnxConfig -> WinMLModelForGenericTask. The production recipe README remains untouched. FP16 requires the explicit public --precision fp16 build contract because CPU auto-precision otherwise overrides the recipe's FP16 mode.

Shared CTC evaluator capability

• Symptom/trigger: winml eval for metadata-resolved CTC ASR failed before dataset loading because automatic-speech-recognition had no evaluator path.
• Root cause: the registry, schema/default dataset, binary audio decode/preprocessing, CTC output interpretation, and WER/CER aggregation were absent as a shared task capability.
• Changed mechanism: WinMLCTCASREvaluator and its decode/resample/language/error-rate helpers implement SoundFile bytes/path decode, mono conversion, processor-rate resampling, deterministic static/dynamic utterance handling, vocabulary checks, processor-owned CTC decode, and fail-closed WER/CER accounting; registry/schema/default-dataset/generic-wrapper mappings route only metadata-resolved CTC models.
• Generality: dispatch is keyed by task plus CTC architecture/loader metadata, never checkpoint ID. Processor sampling rate, target language, blank ID, and vocabulary are metadata-derived; en_us is not guessed to equal eng.
• Compatibility: existing evaluator schemas/defaults remain unchanged; non-CTC ASR fails closed instead of entering CTC decode; audio-classification and binary-audio registration remain unchanged.
• Regression evidence: 50 CTC evaluator tests, 686 eval unit tests, 716 loader/build compatibility tests, and all five CI partitions passed.

Generic CTC processor and language compatibility repair

• Symptom/trigger: the pinned Polish checkpoint selected Wav2Vec2ProcessorWithLM and failed before greedy CTC evaluation when optional pyctcdecode was absent; separately, ordinary Wav2Vec2CTCTokenizer(target_lang=None) was rejected as though it were a misconfigured MMS adapter checkpoint.
• Root cause: processor initialization treated the optional LM wrapper as mandatory once advertised, and language initialization used target_lang truthiness instead of positive adapter metadata to distinguish ordinary Wav2Vec2 from MMS-style language adapters.
• Changed mechanism: _load_ctc_processor preserves any successfully loaded processor and falls back to a composed AutoFeatureExtractor + AutoTokenizer Wav2Vec2Processor only for the exact missing-pyctcdecode Wav2Vec2ProcessorWithLM import failure. _configure_processor_language accepts null language for non-adapter checkpoints, while positive adapter metadata requires a valid active or configured language and fails closed otherwise.
• Generality: the rule is driven by processor class, dependency error, model/tokenizer adapter metadata, and tokenizer validation; there is no checkpoint-ID hardcoding.
• Compatibility: available LM processors remain intact; unrelated import failures and incompatible processor components still propagate; ordinary single-language Wav2Vec2 accepts target_lang=None; MMS preserves active eng, validates requested adapters, and rejects invalid or missing language. Vocabulary-width validation, 64-window fan-out, blank-boundary insertion, and empty-hypothesis WER/CER accounting are unchanged.
• Regression evidence: the pinned Polish probe produced Wav2Vec2Processor, 16 kHz, vocab 40, blank 0, decode -a, and no LM decoder; the pinned MMS probe preserved eng and vocab 154 and failed closed for invalid/missing adapters. Thirteen focused compatibility/empty-hypothesis cases, all 50 CTC tests, 686 eval tests, 716 loader/build tests, and the five CI partitions passed.

Generic seeded bounded selection, composite identity, provenance, and accounting repair

• Symptom/minimal trigger: WinMLCTCASREvaluator.prepare_data() ignored DatasetConfig.shuffle and seed; with shuffle=true, seed=123, and a bounded sample cap, both modes silently evaluated first-N rows and could emit valid WER/CER for the wrong corpus.
• Root cause: both branches truncated before any seeded reordering. Their bounded-selection path therefore had no deterministic shuffle contract and could not preserve each row's original position through reordering.
• Changed mechanism: every row receives its original source index before reordering. shuffle=true applies the configured seed before take(samples) through the same finite buffered-shuffle semantics in streaming and non-streaming modes; shuffle=false retains native first-N order. The selected original index, valid scalar dataset ID, and stable redacted media key/basename continue through output and are validated as a composite identity before transcription/model inference.
• Generality: the rule is driven only by dataset configuration, seed, finite buffer, source position, and row/media metadata. It contains no checkpoint, language, FLEURS ID, or recipe-specific branch.
• Compatibility and boundedness: fixed seeds select identical original indices in both modes below and beyond the shuffle-buffer size, different seeds discriminate, and an unbounded source is consumed only through the finite buffer plus requested prefix rather than materialized. Legitimate duplicate semantic IDs remain valid when source/media identities differ. shuffle=false Polish selection remains IDs [1525, 1657], indices [0, 1], with the same two redacted basenames; accounting, WER/CER, MMS eng, ordinary Wav2Vec2 optional-LM fallback and null target_lang, and successful empty-hypothesis deletion scoring are preserved.
• Exact regression evidence: seed 123 selected [7, 4, 0, 2] in both modes and seed 456 selected [4, 6, 3, 2]; both modes selected [15, 682, 592, 53] across 1,200 rows; the infinite-style source yielded 1,007 rows under a 1,100 ceiling. The focused selection/identity matrix passed 19/19, full CTC passed 50/50, full Eval passed 686/686, the five CI partitions passed 8,543 tests, and current-main/D5 full-format parity remained PASS_NO_REGRESSION with the same 95 inherited files.

Empty CTC hypothesis accounting repair

• Symptom/trigger: after successful inference and decode, a blank-only CTC output normalized to "" and was rejected, removing its deletion errors from corpus WER/CER.
• Root cause: WinMLCTCASREvaluator.compute used a post-decode truthiness guard even though actual decode/validation failures were already represented separately by _RejectedSampleError; the guard therefore conflated output quality with sample failure.
• Changed mechanism: the falsey-prediction rejection was removed, so a successful empty hypothesis remains in the prediction/reference arrays and the existing corpus edit-distance calculation counts its word and character deletions. skipped_samples is explicitly 0.
• Generality: the rule is driven by CTC decode success and normalized values, not by checkpoint identity or language.
• Compatibility: empty or invalid normalized references remain rejected fail-closed; explicit decode failures remain rejected with their reason; unexpected inference/decode errors still propagate. The intentional change is only that successful empty hypotheses are processed.
• Regression evidence: references ["hello world", "good day"] with predictions ["", "good day"] produce WER 0.5, CER 11/19 (0.5789473684210527), 2 processed, 0 skipped, and 0 rejected. Two empty hypotheses produce WER/CER 1.0 with 2 processed and 0 skipped/rejected. Focused edge tests: 4 passed; full CTC evaluator: 25 passed; full eval unit suite: 661 passed.

Generic --no-optimize bug fix

• Symptom/trigger: an HF build with explicit --no-optimize still ran graph optimization.
• Root cause: _build_hf_pipeline dropped the CLI skip_optimize state before the optimize stage.
• Changed mechanism: src/winml/modelkit/commands/build.py::_build_hf_pipeline forwards explicit stage-control state while preserving independent quantize and compile behavior.
• Generality: the rule is model-agnostic CLI state, not checkpoint hardcoding.
• Compatibility: normal MMS FP32/FP16 builds still optimize; explicit skip no longer misclassifies raw ONNX as pre-quantized or suppresses separately requested quantization/compilation.
• Regression evidence: four focused --no-optimize tests passed, and a real tiny Wav2Vec2 HF probe completed with Optimize 0.0 s while export remained enabled.

D6 head 4b528133fef8cbeed4e9a3aba9be42d22a3c0221 is the complete eight-commit accepted D5 series replayed byte-equivalently onto current main e28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0; its final commit changes only ctc_asr_evaluator.py and its unit test relative to parent 34bc2a0e6181e8c9f0cc27fddf9ebfaf2d1dbfe7. The r6 proof is 8/8 equal commits, 8/8 equal ordered canonical stable patch IDs, 10/10 equal changed path statuses, and 10/10 equal final blobs. The sole main increment changes src/winml/modelkit/models/hf/vision_encoder_decoder.py, has no owned-path overlap or CTC reach, and is closed by fresh exact-D6 VED, models-partition, CTC/build, Ruff, mypy, license, and format-parity evidence. L0/L1/L2/L3/Analyze and unaffected quality rows retain their recorded original execution SHAs and are not relabeled as fresh D6 execution. The final GitHub snapshot is 9/9 completed-success checks; zero review threads are unresolved; all 14 fresh public rows and the independently rewalked 49-file seal are closed.

CodeQL superclass-initialization repair

• Symptom/trigger: normal construction of WinMLCTCASREvaluator(config, model) left the inherited pipe attribute absent, and CodeQL reported py/missing-call-to-init at the evaluator constructor.
• Root cause: the constructor duplicated assignments for model, config, and data instead of invoking WinMLEvaluator.__init__; that omission skipped the base-owned pipe initialization.
• Changed mechanism: WinMLCTCASREvaluator.__init__ initializes processor-specific state and calls super().__init__(config, model), while prepare_pipeline() returns None to preserve the evaluator's intentional direct-model CTC path.
• Generality: the repair follows the shared evaluator lifecycle and is independent of checkpoint identity.
• Compatibility: the public constructor signature is unchanged; direct CTC model invocation is preserved; build, export, perf, compare, and Analyze do not import this evaluator and were dependency-preserved.
• Regression evidence: the constructor regression remains covered, all 50 CTC evaluator tests passed, a fresh exact-candidate two-row FLEURS run passed, full mypy passed for 440 source files, and all five CI partitions passed with 8,543 tests.

6. Analyze summary - component level and op level

ANALYZE-PARTIAL-SUCCESS: Analyze returned accepted exit code 1 in 70.28099999995902 s because six requested EP/device rows had no shipped rule data; all 12 rows were present and parseable, with no errors or warnings. This is static rule analysis, not runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	Feature encoder/projection; positional convolution; 48 encoder attention/feed-forward/adapter blocks; CTC head	1,235 mapped; 686 unmapped; mapped by retained optimized scope names with fusion-generated gaps explicit	None

Mapping gaps: 686 fusion-generated nodes lack retained component scope and remain unmapped rather than being assigned by adjacency. Final dropout is disabled in evaluation mode and has no ONNX node.

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32	1,921 operators / 13 unique types	Reshape 676; Gemm 386; Transpose 207; Add 193; LayerNormalization 153	All 13 types supported by available rules for NvTensorRTRTX/GPU, QNN/NPU, QNN/GPU, and OpenVINO NPU/GPU/CPU; no actionable partial/unsupported types

Rule-less rows: CUDA/GPU, MIGraphX/GPU, Tensorrt/GPU, DML/GPU, CPU/CPU, and VitisAI/NPU. Six of 12 records had rule-declared support; records with errors: 0; records with warnings: 0.

7. Reproduce commands

The following is the exact public acquisition, environment, data preparation, quality, FP32/FP16 build, perf, L2 compare, L3 functional-smoke, rules acquisition, and Analyze sequence. It requires a new or empty $Out and first verifies that the public branch advertises the exact candidate SHA.

param(
    [Parameter(Mandatory = $true)]
    [string]$Out,
    [string]$Remote = 'https://github.com/ssss141414/winml-cli.git',
    [string]$Branch = 'ssss141414/add-facebook-mms-1b-all-recipe',
    [string]$SourceRepository = ''
)

$ErrorActionPreference = 'Stop'
$Head = '4b528133fef8cbeed4e9a3aba9be42d22a3c0221'
$Parent = '34bc2a0e6181e8c9f0cc27fddf9ebfaf2d1dbfe7'
$Tree = 'b9b8745bb942e2f359cf583a642c3d8cc0112282'
$Base = 'e28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0'
$Model = 'facebook/mms-1b-all'
$DatasetRevision = '70bb2e84b976b7e960aa89f1c648e09c59f894dd'
$Repo = Join-Path $Out 'checkout'
$Artifacts = Join-Path $Out 'artifacts'
$Results = Join-Path $Out 'results'
$Helpers = Join-Path $Out 'helpers'

if (Test-Path -LiteralPath $Out) {
    if (@(Get-ChildItem -LiteralPath $Out -Force).Count -ne 0) {
        throw "Output root must be new or empty: $Out"
    }
} else {
    New-Item -ItemType Directory -Path $Out | Out-Null
}

$Advertised = (git ls-remote $Remote "refs/heads/$Branch" | Out-String).Trim()
if (-not $Advertised) { throw "Explainer publication pending: $Branch is not advertised" }
$AdvertisedHead = ($Advertised -split '\s+')[0]

if ($SourceRepository) {
    git clone --no-local $SourceRepository $Repo
} else {
    if ($AdvertisedHead -ne $Head) {
        throw "Explainer publication pending: $Branch advertises $AdvertisedHead, expected $Head"
    }
    git clone --branch $Branch --single-branch $Remote $Repo
}

git -C $Repo checkout --detach $Head
if ((git -C $Repo rev-parse HEAD).Trim() -ne $Head) { throw 'HEAD mismatch' }
if ((git -C $Repo rev-parse 'HEAD^').Trim() -ne $Parent) { throw 'Parent mismatch' }
if ((git -C $Repo rev-parse 'HEAD^{tree}').Trim() -ne $Tree) { throw 'Tree mismatch' }
if ((git -C $Repo merge-base HEAD $Base).Trim() -ne $Base) { throw 'Base mismatch' }
if ((git -C $Repo status --porcelain | Out-String).Trim()) { throw 'Checkout dirty at start' }
New-Item -ItemType Directory -Path $Artifacts, $Results, $Helpers | Out-Null

Push-Location $Repo
try {
    Remove-Item Env:PYTHONPATH -ErrorAction SilentlyContinue
    uv sync --locked --extra audio
    uv run --no-sync pytest -q tests/unit/eval/test_ctc_asr_evaluator.py
    uv run --no-sync pytest -q tests/unit/eval --tb=short --no-cov -m 'not e2e and not npu and not gpu'
    uv run --no-sync pytest -q tests/unit/loader tests/unit/build tests/unit/commands/test_build.py --tb=short --no-cov -m 'not e2e and not npu and not gpu'
    uv run --no-sync pytest -q tests/unit/export/test_vision_encoder_decoder_onnx_config.py
    uv run --no-sync ruff format --check src/winml/modelkit/models/hf/vision_encoder_decoder.py tests/unit/export/test_vision_encoder_decoder_onnx_config.py src/winml/modelkit/eval/ctc_asr_evaluator.py tests/unit/eval/test_ctc_asr_evaluator.py src/winml/modelkit/commands/build.py tests/unit/commands/test_build.py
    uv run --no-sync pre-commit run insert-license --all-files
    uv run --no-sync ruff check src/ tests/
    uv run --no-sync mypy -p winml.modelkit
    uv run --no-sync pytest tests/unit/analyze --tb=short --no-cov -m 'not e2e and not npu and not gpu'
    uv run --no-sync pytest tests/unit/models tests/unit/loader tests/unit/datasets tests/unit/export --tb=short --no-cov -m 'not e2e and not npu and not gpu'
    uv run --no-sync pytest tests/unit/optim --tb=short --no-cov -m 'not e2e and not npu and not gpu'
    uv run --no-sync pytest tests/unit/commands tests/unit/config tests/unit/build tests/unit/compiler tests/unit/session tests/unit/eval --tb=short --no-cov -m 'not e2e and not npu and not gpu'
    uv run --no-sync pytest tests/unit/core tests/unit/onnx tests/unit/cache tests/unit/utils tests/unit/test_helpers tests/unit/sysinfo tests/unit/inspect tests/unit/optracing tests/unit/serve tests/regression tests/cli --tb=short --no-cov -m 'not e2e and not npu and not gpu'

    uv run --no-sync winml build -c examples/recipes/facebook_mms-1b-all/cpu/cpu/automatic-speech-recognition_fp32_config.json -m $Model -o (Join-Path $Artifacts 'fp32') --rebuild --no-color
    uv run --no-sync winml build -c examples/recipes/facebook_mms-1b-all/cpu/cpu/automatic-speech-recognition_fp16_config.json -m $Model -o (Join-Path $Artifacts 'fp16') --precision fp16 --rebuild --no-color

    $InputHelper = @'
from io import BytesIO
from pathlib import Path
import numpy as np
import soundfile as sf
from datasets import Audio, load_dataset
from transformers import AutoProcessor

out = Path(r"__HELPERS__")
dataset = load_dataset(
    "google/fleurs",
    name="en_us",
    split="validation",
    revision="__DATASET_REVISION__",
).cast_column("audio", Audio(decode=False)).sort("id")
audio = dataset[0]["audio"]
raw = audio.get("bytes") or Path(audio["path"]).read_bytes()
waveform, rate = sf.read(BytesIO(raw), dtype="float32", always_2d=False)
waveform = np.asarray(waveform, dtype=np.float32)
if waveform.ndim == 2:
    waveform = waveform.mean(axis=1, dtype=np.float32)
if rate != 16000:
    raise RuntimeError(f"expected 16000 Hz, got {rate}")
clip = waveform[:16000]
if clip.size < 16000:
    clip = np.pad(clip, (0, 16000 - clip.size))
processor = AutoProcessor.from_pretrained("facebook/mms-1b-all", revision="__MODEL_REVISION__")
processor.tokenizer.set_target_lang("eng")
inputs = processor(clip, sampling_rate=16000, return_tensors="np")
np.savez(out / "named-input.npz", input_values=np.asarray(inputs["input_values"], dtype=np.float32))
'@
    $InputHelper = $InputHelper.Replace('__HELPERS__', $Helpers.Replace('\', '\\'))
    $InputHelper = $InputHelper.Replace('__DATASET_REVISION__', $DatasetRevision)
    $InputHelper = $InputHelper.Replace('__MODEL_REVISION__', '3d33597edbdaaba14a8e858e2c8caa76e3cec0cd')
    $InputHelperPath = Join-Path $Helpers 'make-pinned-input.py'
    Set-Content -LiteralPath $InputHelperPath -Value $InputHelper -Encoding utf8
    uv run --no-sync python $InputHelperPath
    $NamedInput = Join-Path $Helpers 'named-input.npz'

    uv run --no-sync winml perf -m (Join-Path $Artifacts 'fp32/model.onnx') --ep cpu --device cpu --iterations 20 --warmup 3 --input-data $NamedInput --memory --format json --output (Join-Path $Results 'perf-fp32.json') --no-color
    uv run --no-sync winml perf -m (Join-Path $Artifacts 'fp16/model.onnx') --ep cpu --device cpu --iterations 20 --warmup 3 --input-data $NamedInput --memory --format json --output (Join-Path $Results 'perf-fp16.json') --no-color
    uv run --no-sync winml eval --mode compare -m (Join-Path $Artifacts 'fp32/model.onnx') --model-id $Model --input-data $NamedInput --ep cpu --device cpu --format json --output (Join-Path $Results 'parity-fp32.json') --no-color
    uv run --no-sync winml eval --mode compare -m (Join-Path $Artifacts 'fp16/model.onnx') --model-id $Model --input-data $NamedInput --ep cpu --device cpu --format json --output (Join-Path $Results 'parity-fp16.json') --no-color
    uv run --no-sync winml eval -m (Join-Path $Artifacts 'fp32/model.onnx') --model-id $Model --task automatic-speech-recognition --dataset google/fleurs --dataset-name en_us --dataset-revision $DatasetRevision --split validation --samples 2 --no-shuffle --column input_column=audio --column label_column=transcription --ep cpu --device cpu --format json --output (Join-Path $Results 'functional-smoke.json') --no-color

    git clone --depth 1 https://github.com/gim-home/ModelKitArtifacts.git (Join-Path $Out 'rules-source')
    $env:WINMLCLI_RULES_DIR = Join-Path $Out 'rules-source/rules'
    uv run --no-sync winml analyze --model (Join-Path $Artifacts 'fp32/model.onnx') --ep all --device all --output (Join-Path $Results 'analyze-fp32.json') --no-color
    if ($LASTEXITCODE -notin @(0, 1)) { throw "Analyze failed with exit $LASTEXITCODE" }
    Remove-Item Env:WINMLCLI_RULES_DIR -ErrorAction SilentlyContinue
    if ((git status --porcelain | Out-String).Trim()) { throw 'Checkout dirty at end' }
} finally {
    Pop-Location
}

[ssss141414]

Independent reviewer terminal verdict: REQUEST_CHANGES

Reviewed candidate

• PR: microsoft/winml-cli#1335
• Head: 48661b15ded2380c9d8b292cc663dd48bf5f2a17
• Base: 0876e5ae1c98a169a6137e092e0d7b30bf9cee33

Blocking finding: valid empty CTC hypotheses are excluded from WER/CER

src/winml/modelkit/eval/ctc_asr_evaluator.py:242 raises _RejectedSampleError("decoded prediction is empty") after successful inference and decoding. An empty hypothesis is a valid model output, not an invalid dataset row; rejecting it removes its deletion errors from the corpus denominator and can make a poor model look perfect.

Independent probe:

• references: ["hello world", "good day"]
• predictions: ["", "good day"]
• candidate result: WER 0.0, CER 0.0, processed_samples=1, rejected_samples=1, reason decoded prediction is empty
• required result: WER 2/4 = 0.5, CER 11/19 ~= 0.5789473684, processed_samples=2, rejected_samples=0

The existing _corpus_error_rate implementation already scores an empty prediction correctly; the blocker is the rejection branch before aggregation.

Owner handoff

• Producer: retain "" in predictions whenever inference and decoding succeed. Keep rejection accounting for actual sample/data failures, but do not classify model output quality as a rejected row.
• Tester: add a regression with one empty decoded hypothesis and one exact hypothesis. Assert WER 0.5, CER 11/19, two processed samples, zero rejected samples, and no rejection reason.
• Explainer/release owner: regenerate the final-head focused/L3 evidence and integrity references after the repair, then request independent re-review on the new SHA.

Independent validation on this exact head

• Focused CTC evaluator suite: 21 passed.
• CI-equivalent local partitions: analyze 1,526 passed; models 1,538 passed; optim 848 passed; commands 3,652 passed / 9 skipped; remaining 926 passed / 2 skipped / 1 deselected. Checkout remained clean.
• Format, license, Ruff, full mypy, and all 9 GitHub checks passed.
• One fully paginated review thread exists and is resolved; no unresolved review thread was hidden by pagination.
• Sealed evidence independently rehashed with no missing or mismatched files: successor 32/32 (f7c5e8c44750f4da491f36935178b9374a4ba6109f475a1ebc244d4bcdbd156c), prior 35/35 (d890f509c7f6c076359bb2397988962df8ac3d008cfb7362ad2d19b4c0125fe4), public reproduction 29/29 (1927ecda88eb0590626958a5c284f6642e550a5e88672107f7b1540d1b453991).
• Historical PR recipe(mms-1b-all): add verified CPU fp32/fp16 configs #1177, Lane A PR refactor: move remaining network-dependent tests to integration #254, generic ASR routing, processor language handling, SoundFile decoding, fixed-window reconstruction, --no-optimize forwarding, pinned recipes, FP16 shipment evidence, and the CodeQL superclass-init repair were reviewed without another actionable blocker.

Approval is blocked until the empty-hypothesis metric path is repaired and the replacement head is independently revalidated.

[ssss141414]

Addressed on exact head 8eafeb5f9d7513c9f178ad5ca978071f9272323a (base 0876e5ae1c98a169a6137e092e0d7b30bf9cee33).

Root cause and fix

WinMLCTCASREvaluator.compute used a post-decode truthiness guard even though inference, decode, and validation failures were already represented separately by _RejectedSampleError. A successful blank-only CTC decode normalizes to "", so the guard incorrectly classified model output quality as sample failure and removed its deletion errors from corpus WER/CER. The repair removes only that falsey-prediction rejection: successful empty hypotheses remain in the prediction/reference arrays and the existing corpus edit-distance calculation scores their deletions. Empty normalized references still fail closed, explicit decoder failures remain rejected with their reason, and unexpected inference/decode errors still propagate.

Edge regression and real L3 evidence

• Reviewer case, references ["hello world", "good day"], predictions ["", "good day"]: WER 0.5, CER 11/19 (0.5789473684210527), processed_samples=2, skipped_samples=0, rejected_samples=0.
• All-empty hypotheses: WER 1.0, CER 1.0, 2 processed, 0 skipped, 0 rejected.
• Empty reference remains rejected as normalized transcription is empty; explicit decoder failure remains rejected as decode failed.
• Final-head FP32 CPU FLEURS smoke is unchanged: WER 0.75, CER 0.37037037037037035, 2 processed, 0 skipped, 0 rejected on google/fleurs@70bb2e84b976b7e960aa89f1c648e09c59f894dd/en_us/validation.

Validation and public closure

• Focused edge regressions: 4 passed; full CTC evaluator: 25 passed; full eval unit suite: 661 passed.
• Format, insert-license, full Ruff, and full-package mypy passed (439 source files).
• CI-equivalent partitions: analyze 1,526; models 1,538; optim 848; commands 3,656; remaining 926 (8,494 passed total), and GitHub reports 9/9 successful checks on the exact head.
• The exact public sequence now acquires the published head, runs the edge/full eval and quality suites, rebuilds FP32/FP16, reacquires perf/parity, reruns the real FLEURS L3 smoke, and completes all-EP Analyze. The terminal public seal is PASS, state CLOSED, closure 12/12.

The canonical PR body has been regenerated with this final-head evidence, exact public sequence, and Lane A #254 binding at 3ecb82ce22ea4bdbda89ed55b79e17a515209037. Please independently re-review the repaired candidate.

[ssss141414]

Independent successor reviewer terminal verdict: APPROVE

Reviewed candidate

• PR: microsoft/winml-cli#1335
• Head: 8eafeb5f9d7513c9f178ad5ca978071f9272323a
• Base / merge base: 0876e5ae1c98a169a6137e092e0d7b30bf9cee33
• State: OPEN, DRAFT, label model-scale-by-skill

Prior blocker: CLOSED

• A successful empty CTC hypothesis is now retained as a processed prediction and contributes deletions to corpus metrics.
• Independent fresh reviewer case: references ["hello world", "good day"], predictions ["", "good day"] -> WER 0.5, CER 11/19 (0.5789473684210527), 2 processed, 0 skipped, 0 rejected.
• All-empty hypotheses -> WER 1.0, CER 1.0, 2 processed, 0 skipped, 0 rejected.
• Empty normalized references remain rejected as normalized transcription is empty; explicit decode failures remain rejected as decode failed; unexpected inference/decode errors still propagate.
• Fresh targeted regressions: 4 passed, 21 deselected. Full CTC evaluator suite: 25 passed.

Engineering review

• The evaluator is generic: dispatch is by automatic-speech-recognition plus *ForCTC architecture metadata, with processor-derived sampling rate, language, blank ID, and vocabulary. No checkpoint/model-ID branch was added.
• The repair removes only the post-decode truthiness rejection. Existing fail-closed data/decode handling and corpus edit-distance semantics remain intact.
• Shared ASR schema/defaults, lazy evaluator registration, generic WinML wrapper routing, and HF skip_optimize forwarding are appropriately generalized and regression-covered. The two CPU recipes are pinned, precision-correct, and the production recipe README is untouched.
• Final-head FP32 CPU FLEURS L3 remains a bounded two-row functional smoke: WER 0.75, CER 0.37037037037037035, 2 processed, 0 skipped, 0 rejected. It is correctly described as operability evidence, not representative accuracy.

Independent validation

• Fresh detached checkout at the exact head remained clean.
• Affected CI partition: 3,656 passed, 9 hardware/design skips, 1 known warning; exit 0.
• Format: 2 files already formatted. Insert-license: passed. Full Ruff: passed. Full mypy: no issues in 439 source files. git diff --check: passed.
• No expensive rebuild was repeated: the final repair changes only evaluator logic and its tests, while the terminal public run already rebuilt FP32/FP16 and reacquired perf, parity, L3, and Analyze on this exact head. Artifact equivalence and integrity were independently rehashed.

Public and evidence closure

• PR body hierarchy and exact public command sequence are complete; canonical remote body matches the sealed local body.
• GitHub checks: 9/9 COMPLETED/SUCCESS on the exact reviewed head.
• Review threads: 1 total, 1 resolved, 0 unresolved; both thread and comment pagination are complete.
• Normal comments before this verdict: exactly prior REQUEST_CHANGES comment 5383666732 and repair reply 5384214844.
• Evidence rehash: terminal integrity 40/40, public reproduction 36/36, terminal seal 5/5, shipment 7/7, publication files 2/2, learner references 4/4, charter profile 1/1; 50 unique files / 23.16 GB read, 0 missing, 0 mismatched.
• Terminal public seal: PASS, state CLOSED, closure 12/12.
• Lane A gim-home/ModelKitArtifacts#254: exact head 3ecb82ce22ea4bdbda89ed55b79e17a515209037, DRAFT, required label present, 1/1 check successful.

No actionable blocker remains. This is a skill-level reviewer opinion posted as a normal issue comment only; it does not change GitHub Review state or authorize merge/readiness changes.

[ssss141414]

Successor independent review requested for exact head 512338dd56080e97efab81da0cf59245ecc02ccf (base 0876e5ae1c98a169a6137e092e0d7b30bf9cee33). The prior independent approval covered dependency head 8eafeb5f9d7513c9f178ad5ca978071f9272323a; the head changed because this successor adds generic optional-Wav2Vec2ProcessorWithLM fallback and metadata-driven ordinary target_lang=None versus MMS adapter handling.

Sealed public evidence is PASS/CLOSED 12/12, all 9/9 GitHub checks succeeded, and Lane A knowledge/methodology is bound to draft PR #254 head 15ba207dfea4fefc9de8692170054f4d6dde1ad4. The compatibility probe technically clears the two #1207 dependency blockers, but it is not #1207 L3 acceptance evidence. Please independently review this exact successor head; this PR remains draft.

[ssss141414]

Independent successor reviewer terminal verdict: APPROVE

Reviewed candidate

• PR: microsoft/winml-cli#1335
• Head: 512338dd56080e97efab81da0cf59245ecc02ccf
• Base / merge base: 0876e5ae1c98a169a6137e092e0d7b30bf9cee33
• State: OPEN, DRAFT, label model-scale-by-skill

Engineering review

• The successor fallback is narrow: it runs only when AutoProcessor raises an ImportError naming both Wav2Vec2ProcessorWithLM and missing pyctcdecode. A successfully loaded processor, including an LM processor, is returned unchanged; unrelated loader errors propagate.
• Ordinary Wav2Vec2CTCTokenizer(target_lang=None) remains valid without adapter metadata. Positive MMS adapter metadata requires a valid active or configured language; invalid and missing language fail closed.
• The complete shared-code diff contains no target checkpoint identifiers or model-ID conditionals. The behavior is metadata-driven and preserves the public constructor, CTC vocabulary checks, bounded windows, blank-boundary insertion, and error propagation.
• The prior empty-hypothesis repair remains intact: successful empty decodes contribute deletion errors, while empty references and explicit decode failures remain rejected.
• The bounded Polish probe clears only DEP-CTC-PROCESSOR-LM and DEP-CTC-TARGET-LANG. It is correctly labeled as not establishing recipe(wav2vec2): add jonatasgrosman/wav2vec2-large-xlsr-53-polish ASR recipes #1207 L3 acceptance.

Independent validation

• Fresh detached checkout: exact head and merge base verified; checkout remained clean.
• Focused processor/language/empty-hypothesis regressions: 13 passed, 19 deselected.
• Full CTC evaluator suite: 32 passed.
• Affected commands CI partition: 3,663 passed, 9 hardware/design skips, 1 known warning.
• Ruff format: 2 files already formatted. Full Ruff: passed. Full mypy: no issues in 439 source files. git diff --check: passed.
• No expensive rebuild was repeated: the successor changes only evaluator initialization and tests relative to approved head 8eafeb5f9d7513c9f178ad5ca978071f9272323a; all 10 changed files match the sealed public acquisition, and dependency-equivalent L0/L1/L2/Analyze evidence was rehashed.

Public and evidence closure

• Canonical PR body hash: 521364efc89e1b7ce81bf1296cec0f7827c8698c99d4406b59b4a17c1493b790; hierarchy and all 12 closure rows verified.
• Sealed evidence rehash: successor shipment 7/7; public tester integrity 48/48; fresh-checkout changed-file equivalence 10/10; no missing or mismatched files.
• GitHub checks: 9/9 COMPLETED/SUCCESS on the exact reviewed head.
• Review threads: 1 total, 1 resolved, 0 unresolved; thread and comment pagination complete.
• Lane A gim-home/ModelKitArtifacts#254: exact head 15ba207dfea4fefc9de8692170054f4d6dde1ad4, DRAFT, required label present, 1/1 check successful, body hash 30dd87b4b055464d3afb4437f944a42a235254e50592cfb988cb1aa409014e6c.
• Prior approval 5384294373, successor request 5384964725, producer/tester public closure, learner finding wav2vec2-026, and frozen recipe(wav2vec2): add jonatasgrosman/wav2vec2-large-xlsr-53-polish ASR recipes #1207 blocked charter were read in full.

No actionable blocker remains. This is a skill-level reviewer opinion posted as one normal issue comment only; it does not change GitHub Review state, labels, threads, readiness, or merge state.

[ssss141414]

Successor evidence update for the prior exact-head review at #1335 (comment).

The reviewed 512338dd56080e97efab81da0cf59245ecc02ccf compatibility behavior remains preserved, but head e633f2e5c33d1784e311116f0e86a2bd0bd8a074 adds the generic dependency repair required by the frozen #1207 charter-r3 investigation:

• streaming and non-streaming bounded selection now preserve pinned dataset source order;
• repeated semantic IDs remain distinct rows and are neither sorted nor deduplicated;
• results emit requested/selected counts, source IDs/indices, basename-only audio provenance, selected rows, and processed/rejected/skipped accounting;
• missing, malformed, boolean, or duplicate source indices fail before inference (zero inference calls in all four probes);
• MMS target_lang=eng, ordinary Wav2Vec2 target_lang=null/optional-LM fallback, and successful empty-hypothesis deletion scoring remain unchanged.

The pinned Polish compatibility probe now agrees across modes at IDs [1525, 1657], indices [0, 1], and the two expected audio basenames. This verifies the shared dependency behavior only; it does not claim #1207 model acceptance or #1207 L3 acceptance.

Exact-head public closure is PASS/CLOSED 15/15. Candidate-sensitive probes, MMS L3, focused/full Eval, quality gates, and all five CI partitions were freshly run from a clean public checkout; L0/L1/L2/Analyze retain their original e9180e05... execution provenance and are explicitly reused/rehashed, not relabeled fresh. MMS L3 emitted WER 0.8541666666666666, CER 0.3684210526315789, 2 processed, 0 rejected, and 0 skipped. All 9 GitHub checks are COMPLETED/SUCCESS on e633f2e5....

The canonical body now records Methodology friction observed: _meta-112 added and binds the finding plus paired tester/reviewer contract edits to Draft Lane A PR #254 at exact head c936f6ccccc107a2d771d9f7a562ac6574ef8d2b (GitOps/AdvancedSecurity: COMPLETED/SUCCESS). This is an explainer evidence update only; the PR remains open and draft.

[ssss141414]

Independent reviewer terminal verdict: APPROVE

Reviewed candidate

• PR: microsoft/winml-cli#1335
• Exact head: e633f2e5c33d1784e311116f0e86a2bd0bd8a074
• Base / merge base: 0876e5ae1c98a169a6137e092e0d7b30bf9cee33
• State: OPEN, DRAFT, label model-scale-by-skill

Engineering findings

• No actionable blocker found. Streaming and non-streaming paths take the same bounded first rows in pinned source order. Repeated semantic IDs remain distinct rows; no semantic sort or deduplication remains.
• Source provenance is validated before the inference loop. Missing, string, boolean, and duplicate source indices each failed with zero inference calls. Successful output emits requested/selected/processed/rejected/skipped accounting, aligned predictions/references, source IDs/indices, selected rows, and basename-only audio provenance.
• WER/CER corpus semantics are unchanged. Successful empty hypotheses remain processed deletion errors; empty references and explicit decode failures remain rejected.
• MMS metadata still resolves target_lang=eng with vocabulary 154. Ordinary Wav2Vec2 still falls back from unavailable optional pyctcdecode to plain Wav2Vec2Processor, with target_lang=null and no LM decoder. Shared source contains no checkpoint-ID branch.
• The independently recomputed pinned Polish compatibility probe matched in both modes: source indices [0, 1], IDs [1525, 1657], audio basenames 10018492969996036091.wav and 10288018704489549018.wav. This is dependency compatibility evidence only, not a recipe(wav2vec2): add jonatasgrosman/wav2vec2-large-xlsr-53-polish ASR recipes #1207 model or L3 acceptance claim.

Independent validation

• Fresh detached no-LFS checkout verified exact head, merge base, complete 10-file diff, and clean state.
• Focused CTC evaluator: 35 passed.
• Affected commands CI partition: 3666 passed, 9 skipped, 1 known ConstantInputWarning.
• Workflow gates: full Ruff passed; full-package mypy passed for 439 source files; insert-license passed; git diff --check passed.
• Sealed tester rewalk: 42/42 files matched exact size and SHA-256. Frozen recipe(wav2vec2): add jonatasgrosman/wav2vec2-large-xlsr-53-polish ASR recipes #1207 charter-r3 and producer dependency deliverable matched their declared hashes.
• Public _meta-111 commands use locked all-extras/all-groups initialization, create their helpers/assets, use relative output paths, and preserve the FP16 --precision fp16 contract. The advertised public branch resolves to the exact reviewed head.

Live closure

• GitHub checks: 9/9 COMPLETED/SUCCESS on the exact reviewed head.
• Review threads: 1 total, 1 resolved, 0 unresolved; pagination complete.
• Lane A gim-home/ModelKitArtifacts#254: DRAFT at exact head c936f6ccccc107a2d771d9f7a562ac6574ef8d2b, labeled and green. Exact commit contains wav2vec2-027, _meta-112, and the paired tester/reviewer contract edits with bound file hashes.

This is a skill-level reviewer opinion posted as one normal PR comment only. It does not submit GitHub Review state, edit the PR, resolve threads, mark ready, merge, or claim #1207 acceptance.

[ssss141414]

Exact-head dependency closure update for 0d242174c25c062a21d40dceb645eace21a3f888.

This successor closes the incomplete selected-row identity validation defect reported by the independent reviewer on stacked Draft PR #1343: all selected rows are now fully validated before transcription/model inference as a nonnegative integer source index, a valid JSON-stable scalar semantic dataset ID, and a stable redacted audio key or basename. Source indices and normalized media identities must be unique; semantic dataset IDs are deliberately not uniqueness keys, so legitimate duplicate IDs remain valid when source and media identities differ. The 19 malformed/duplicate discriminators all recorded zero inference calls, while the valid duplicate-semantic-ID case processed both rows with preserved source order, redacted provenance, and requested/selected/processed/rejected/skipped accounting.

Public exact-head evidence is PASS/CLOSED 20/20: a fresh MMS FP32 build produced model.onnx SHA-256 c1a21c6685f7f5b51474213b95481e37c407e5c5fc1ddf0c75379c3c7cd7830f; fresh bounded MMS L3 produced WER 0.8541666666666666, CER 0.3684210526315789, 2 processed, 0 rejected, and 0 skipped; focused CTC is 43 passed and full Eval is 679 passed; all five CI partitions total 8,512 passed tests; and full-tree format comparison is PASS_NO_REGRESSION with the same 91 inherited files on parent and candidate. L0/L1/L2/Analyze retain execution provenance e9180e05332cae9db3724de824c0183a16cb0043; 9 files were rehashed through the dependency head and are not represented as fresh.

The canonical body now binds the refined existing _meta-112 and wav2vec2-027 findings plus paired tester/reviewer contract updates to Draft Lane A PR gim-home/ModelKitArtifacts#254 at exact head 8a1f41a73d8c62d3b5ec6e2fc9d39155a72b4d49. All 9 checks on this #1335 head are COMPLETED/SUCCESS. This dependency closure does not claim that #1343 itself is accepted, approved, or ready; #1335 remains Draft with model-scale-by-skill.

[ssss141414]

Independent review of exact head 0d242174c25c062a21d40dceb645eace21a3f888 found one blocking issue.

[P1] Honor --shuffle before bounded ASR sampling

WinMLCTCASREvaluator.prepare_data() selects the first samples rows directly in both branches (src/winml/modelkit/eval/ctc_asr_evaluator.py:315-318) and never reads DatasetConfig.shuffle or seed. This conflicts with the public winml eval contract, whose default --shuffle option is documented as "Shuffle dataset before sampling," and with WinMLEvaluator.prepare_data(), which applies the seeded shuffle before truncation. My exact-head discriminator configured shuffle=True, seed=123, and three source rows; it observed shuffle_call_count=0 and selected IDs [1, 2]. The command therefore silently evaluates first-N rows instead of the requested reproducible sample, changing the selected corpus and potentially WER/CER while still returning valid metrics.

Please apply the seeded shuffle before selection for both streaming and non-streaming datasets while retaining each row's original source index, then add focused tests that distinguish shuffle=True from shuffle=False in both modes and verify reproducibility for a fixed seed.

All other reviewed exact-head gates passed: composite identity rejection before inference (10 passed), the full CTC evaluator (43 passed), models/loader/datasets/export (1,538 passed, 6 skipped, 2 xfailed), commands/config/build/compiler/session/eval (3,674 passed, 9 skipped), Ruff, targeted format, and mypy over 439 source files. The sealed evidence rewalk had 55/55 matching files, the pinned Polish source-order/composite-identity probe and MMS functional smoke were internally consistent, Lane A PR #254 remained bound to 8a1f41a73d8c62d3b5ec6e2fc9d39155a72b4d49 with its required label and successful check, and the live PR checks/threads were green/closed at the review snapshot.

REQUEST_CHANGES

[ssss141414]

Evidence reply to reviewer comment 5389680910 for exact head 3402b7b4b1990a0932551a3d69d3838df0e70bda.

The blocking seeded-shuffle gap reported against 0d242174c25c062a21d40dceb645eace21a3f888 is addressed in its direct successor 3402b7b4b1990a0932551a3d69d3838df0e70bda (tree 01e8fe52c4cd2d9e3ae17dbcdbd0b309ccad2ed1). WinMLCTCASREvaluator.prepare_data() now attaches each row's original source index before reordering, applies DatasetConfig.shuffle with the configured seed before take(samples), and uses the same finite buffered-shuffle semantics in streaming and non-streaming modes. shuffle=false remains native first-N order.

Exact public-checkout discriminators:

• 10-row source, seed 123: both modes selected original indices [7, 4, 0, 2]; repeating the seed was stable.
• 10-row source, seed 456: both modes selected [4, 6, 3, 2], discriminating the seed.
• 1,200-row source beyond the shuffle buffer: both modes selected [15, 682, 592, 53].
• Infinite-style source: selected [15, 682, 592, 53] after 1,007 yields, below the sealed ceiling of 1,100; the source was not materialized.
• Pinned google/fleurs@70bb2e84b976b7e960aa89f1c648e09c59f894dd/pl_pl/validation, shuffle=false, 2 rows: both modes preserved source indices [0, 1], dataset IDs [1525, 1657], and redacted basenames 10018492969996036091.wav, 10288018704489549018.wav.
• Composite identity remains before inference: missing source provenance raised DatasetValidationError with 0 transcription calls and 0 model calls. Duplicate semantic dataset IDs remain accepted when original source/media identities differ.

Fresh sealed closure from the public fork ref and detached exact SHA:

• focused seeded/bounded/composite/pinned matrix: 19 passed, 31 deselected
• focused compatibility: 9 passed, 41 deselected
• full CTC evaluator: 50 passed
• full Eval: 686 passed, 1 warning
• five CI partitions: 8,519 passed, 78 skipped, 3 xfailed, 1 deselected (1,526 + 1,538 + 848 + 3,681 + 926)
• insert-license, Ruff, targeted format, and mypy over 439 source files: passed
• full-format comparison: PASS_NO_REGRESSION; parent and candidate each retain the same 91 inherited files, with empty parent-only, candidate-only, and changed-file overlap sets
• public closure: all 22 fresh rows plus the retained initial missing-soundfile environment incident are CLOSED; the incident closed after locked uv sync --locked --extra audio

Dependency-aware L0/L1/L2/Analyze evidence was reused without being relabeled fresh: execution provenance cac2526b620df08fdbcc26ef60fe590479e55a0e, 31 files verified. The two fresh two-row functional smokes remained bounded and processed 2/2 with 0 skipped/rejected: Polish WER/CER 0.7115384615384616 / 0.17794486215538846, and MMS WER/CER 0.8541666666666666 / 0.3684210526315789; these are operability smokes, not representative accuracy claims.

Lane A was refined in place rather than duplicated: draft gim-home/ModelKitArtifacts#254 at exact head b7efe2b77c59eafbd13c48c4cb9beb9336df8305, required label present, GitOps/AdvancedSecurity COMPLETED/SUCCESS. Existing wav2vec2-027 and _meta-112 now bind shuffle-before-cap, fixed-seed streaming/non-streaming original-index equivalence, bounded consumption, pre-shuffle provenance, and preserved composite identity.

The canonical PR body has been refreshed with these exact values. At this snapshot, all 9 exact-head checks are completed/successful and the sole review thread is resolved. The PR remains Draft with model-scale-by-skill; this reply does not mark ready, approve, merge, or state a terminal review opinion.

[ssss141414]

Independent reviewer terminal verdict: APPROVE

Reviewed candidate

• PR: microsoft/winml-cli#1335
• Exact head: 3402b7b4b1990a0932551a3d69d3838df0e70bda
• Base / merge base: 0876e5ae1c98a169a6137e092e0d7b30bf9cee33
• State: OPEN, DRAFT, label model-scale-by-skill

Engineering findings

• The prior seeded-shuffle blocker is closed. Original source indices are attached before reordering; shuffle and seed are applied before the sample cap through one bounded-buffer iterable shuffle path in both configured modes.
• Fixed seed 123 reproduced the same selected original indices across streaming and non-streaming sources; seed 456 discriminated; equality held for a 1,200-row source beyond the 1,000-row buffer. An infinite-style source yielded only 1,007 rows for four selected samples, so selection does not materialize an unbounded stream.
• shuffle=false retains pinned first-N source order. Selected-row provenance remains composite and pre-inference: source index, JSON-stable scalar dataset ID, and redacted media basename/key are validated before transcription. Malformed or duplicate row/media identity produces zero inference calls; repeated semantic IDs remain valid when source and media differ.
• Compatibility remains intact: MMS adapter language is metadata-driven; ordinary Wav2Vec2 accepts null target language and narrowly falls back when optional pyctcdecode is unavailable; successful empty hypotheses contribute deletion errors while empty references and explicit decode failures remain rejected.
• The two CPU recipes remain pinned and precision-correct. Shared evaluator registration, generic WinML ASR routing, and generic --no-optimize forwarding remain generalized; no checkpoint-ID branch was introduced.

Independent validation

• Fresh detached checkout at the exact head remained clean.
• Focused seeded/bounded/composite/pinned selection matrix: 19 passed.
• Focused MMS/ordinary/empty-hypothesis compatibility matrix: 9 passed.
• Complete affected workflow partition (commands, config, build, compiler, session, eval): 3,681 passed, 9 expected skips, 1 known ConstantInputWarning.
• Full Ruff passed; package mypy passed; targeted format reported both files already formatted; git diff --check passed.
• The tester's terminal public integrity manifest independently rehashed 67/67 files with zero missing or mismatched files. Exact-head producer, tester, learner, and explainer handoffs were independently hashed and read.

Live closure

• PR body hierarchy is complete and ordered; exact-head GitHub checks are 9/9 SUCCESS; unresolved review threads: 0.
• Draft Lane A gim-home/ModelKitArtifacts#254 is bound to exact head b7efe2b77c59eafbd13c48c4cb9beb9336df8305, carries the required label, has 1/1 SUCCESS, and has 0 unresolved threads.
• The prior REQUEST_CHANGES comment 5389680910 and its exact-head evidence reply were read; the requested repair and regression coverage are present.

No actionable blocker remains. This is a skill-level reviewer opinion posted as one normal PR comment only; it does not submit GitHub Review state, edit metadata or threads, mark ready, merge, or authorize a non-Draft transition.

[ssss141414]

APPROVE

Independent reviewer verdict for exact published D5 02e51c2782e9d8dd4cc38ea934eee6fef5d2d073 against main 774dd6e0811f6947eaf2bd8fb721d5f6e718b0c0.

• Identity and scope: fresh detached public checkout; parent 29afdeeae3753a8e6a126a2473f49b4f30f4772c, tree f0302ec98cbc76e60680a0cd1e86f6a8ac3079bc, merge base exact main, clean tracked/untracked state, and all imports bound to the fresh checkout. The 10-path L2 diff is appropriate and git diff --check passes.
• Moved-main authority: independently recomputed byte-preserving canonical patch IDs and range-diff. Accepted history, D4, and D5 are 8/8 ordered commits equal; changed path/status rows are 10/10 equal; final blobs are 10/10 equal. All four current-main Perf/GenAI/runtime-constant blobs are unchanged in D5.
• Engineering review: seeded shuffle occurs after original source-index attachment and before bounded take; streaming and non-streaming paths share bounded iterable selection. Composite source index, dataset ID, and redacted audio identity are validated before transcription/inference. Ordinary Wav2Vec2 permits no adapter language, MMS metadata selects or rejects language fail-closed, and successful empty CTC hypotheses remain in corpus WER/CER as deletions. No checkpoint-ID branching or unrelated scope was found.
• Independent local gates: CTC 50 passed; build/Perf compatibility 220 passed; Ruff passed; mypy passed for 440 source files; all 8 D5-owned Python files formatted. Current workflow partitions passed: Analyze 1,529; Models 1,538; Optim 848; Commands 3,692; Remaining 936, totaling 8,543 passed.
• Public evidence: tester public acquisition is CLOSED/PASS and its seal independently rewalked 65/65 files with zero mismatch. The PR body is byte-equal to the D5 explainer body. Retained L0/L1/L2/Analyze execution is explicitly preserved under original provenance rather than relabeled as fresh D5 execution.
• GitHub gate: exact head remained unchanged after tests; PR is OPEN/Draft with model-scale-by-skill; all 9 checks are COMPLETED/SUCCESS. Review threads enumerated with pagination: 1 total, 1 resolved, 0 unresolved. Lane A gim-home/ModelKitArtifacts#254 remains OPEN/Draft at b7efe2b77c59eafbd13c48c4cb9beb9336df8305, labeled correctly, with its check successful.
• Findings: none.
• Blockers: none.

This is a normal skill-level reviewer comment only. It does not authorize or perform a GitHub Review, body/thread/metadata mutation, ready transition, merge, or branch mutation.

[ssss141414]

APPROVE

• Reviewed exact D6 head 4b528133fef8cbeed4e9a3aba9be42d22a3c0221 against exact main e28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0 from a fresh public checkout. The eight-commit D5-to-D6 replay is byte-equivalent: 8/8 range-diff entries, ordered stable patch IDs, 10/10 path statuses, and 10/10 final blobs match; no extra path or diff-check issue was found.
• Engineering review found the metadata-gated CTC evaluator, deterministic seeded selection and source-identity validation, bounded static-window inference, processor-owned decode, corpus WER/CER accounting, evaluator/registry/model routing, recipes, and skip_optimize forwarding coherent and covered. The current-main VisionEncoderDecoder/TrOCR change has no CTC path overlap or dispatch reach; its 15 focused tests and the full models partition pass.
• Fresh bounded validation on this SHA: CTC 50 passed; build 116 passed; VED 15 passed. All five non-hardware CI partitions pass: analyze 1,529; models 1,538; optim 848; commands 3,692; remaining 936 (8,543 total; 78 skipped, 3 xfailed, 1 marker-deselected). Source imports resolve inside the fresh checkout, and every command preserved the exact clean HEAD.
• Static gates pass: license hook; full Ruff check; mypy for 440 source files. All 8 PR-changed Python files pass Ruff format. The 96 global format failures, including the two VED files, match exact main path-for-path and are not introduced by D6.
• Public evidence closes: the 49-file seal independently rewalks byte-for-byte; retained L0/L1/L2/L3/Analyze/quality rows remain attributed to their original execution SHAs rather than being relabelled as fresh D6 execution. Lane A gim-home/ModelKitArtifacts#254 remains Open/Draft at b7efe2b77c59eafbd13c48c4cb9beb9336df8305 with its required label and successful check.
• Final GitHub gate: PR is Open/Draft with model-scale-by-skill; all 9 exact-head checks are terminal-success. Enumerated 1 review thread across complete pagination: 0 open, 1 resolved (the CodeQL superclass-initialization finding is repaired and exact-head CodeQL succeeds).

No blocking finding remains. Hardware/model-scale measurements were not re-executed in this public terminal review; approval relies on the independently sealed artifacts under their explicitly preserved original commit provenance, with fresh exact-D6 evaluator, focused, static, and complete non-hardware partition validation above.