Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2

[ssss141414]

Summary

This PR adds canonical WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2, a cross-encoder passage reranker whose checkpoint metadata is authoritative for text-ranking semantics while export compatibility remains on the existing sequence-classification path. The shipped outcome is L3 with full CPU coverage: CPU fp32 and fp16 recipes build and perf cleanly, reranking parity preserves raw single-logit ordering, and the final candidate SHA completes a CPU fp32 functional smoke eval on a provenance-preserved MS MARCO fixture. The highest goal verdict reached is L3 PASS on candidate 030e79fa6482acb08e51338ab4172ac4387b387d.

Model metadata

What the model does

This checkpoint is a cross-encoder reranker that tokenizes a query-document pair together with a MiniLM-based BERT encoder and emits a single relevance logit used to order candidate passages for search and retrieval workloads.

• Evidence/confidence: Hugging Face model metadata reports pipeline_tag=text-ranking, architectures=[BertForSequenceClassification], and a single-label output mapping (verified).

Primary user stories

• A user supplies a search query and candidate passages to obtain a relevance ordering for retrieval or reranking (mapped).
• A user supplies a prompt and candidate evidence snippets to obtain the best supporting passages before downstream answer generation or ranking display (inferred).

Supported tasks

• text-ranking on checkpoint metadata surfaces (verified).
• text-classification on transformers, optimum-onnx, and WinML export/inference surfaces (mapped).

Model architecture

BertForSequenceClassification
Γö£ΓöÇΓöÇ BertEmbeddings (word + position + token-type embeddings, hidden_size=384)
Γö£ΓöÇΓöÇ BertEncoder x 6
Γöé   Γö£ΓöÇΓöÇ BertSelfAttention (12 heads)
Γöé   Γö£ΓöÇΓöÇ BertSelfOutput (residual + LayerNorm)
Γöé   Γö£ΓöÇΓöÇ BertIntermediate (384 -> 1536, GELU)
Γöé   ΓööΓöÇΓöÇ BertOutput (1536 -> 384, residual + LayerNorm)
Γö£ΓöÇΓöÇ BertPooler
ΓööΓöÇΓöÇ Classifier head (single logit)

• Source/confidence: pinned checkpoint config plus BertForSequenceClassification structure in the frozen model breakdown (verified).

Validation and support evidence

Baseline

• Pinned baseline main commit: 02179d185f0e3ac23518236b5ca1f61fd37ef680.
• Baseline WinML CLI environment audit recorded winml-cli 0.3.0.
• Fresh-main inspect, config, and build all resolved the checkpoint through the default text-classification path.
• Baseline auto-config behavior preserved the wrong task semantics for evaluation and reporting: the checkpoint builds, but authoritative checkpoint metadata says the single output logit is a reranking relevance score rather than MRPC-style classification output.
• Baseline config auto-detected AutoModelForSequenceClassification and text-classification with opset 17 and the standard BERT [1, 512] int32 tensor inputs.
• Baseline build PASS: BertForSequenceClassification exported cleanly as a 196-node ONNX graph.
• Baseline eval floor stayed below reranking support: default winml eval attempted nyu-mll/glue / MRPC under text-classification and failed offline, which confirmed the missing reranking evaluator and dataset path rather than any recipe/build defect.
• Optimum/export boundary on baseline already worked for this checkpoint through sequence classification; the missing piece was canonical reranking detection, evaluation, and reporting.

Goal

• Effort: L2.
• Goal ceiling: L3.
• Planned outcome from charter: L2, with success defined as CPU fp32/fp16 builds, measured CPU perf, preserved reranking raw-logit ordering, and a final-SHA fp32 CPU reranking eval on 1-2 real MS MARCO queries with authoritative pinned provenance.
• No ceiling downgrade was issued.

Outcome

• Final shipped tier: L3.
• Highest goal verdict: L3 PASS.
• Coverage: full.
• Deferred tuples: none.
• Candidate checkout state handed to explainer/tester: branch producer/ms-marco-minilm-l6-v2-reranking, final candidate SHA 030e79fa6482acb08e51338ab4172ac4387b387d, tester repo-state verdict PASS, tracked tree clean, and no PR-attributable blockers remaining.
• Shared reranking compatibility/regression scope stayed bounded to reranking surfaces. Final validation keeps earlier L0/L1/analyze artifacts unchanged and adds exact-head quality confirmation: changed-file mypy PASS in a clean environment, uv lock --check PASS, reranking evaluator 10 passed, focused reranking regression 265 passed, changed-path Ruff PASS, and preserved L2/L3 PASS evidence.
• Durable findings appended by the learner: authoritative offline grouped dataset provenance is now cleared; lock/bootstrap is reclassified as baseline or environment debt rather than candidate failure; model-breakdown mapping is frozen; and the eval command authority now explicitly uses the grouped fixture and raw single-logit reranking metrics.

Per-EP/device/precision results and Functional smoke Eval

Tier	EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM Δ
L0	CPUExecutionProvider / cpu	fp32	PASS	ΓÇö	ΓÇö	ΓÇö	ΓÇö
L0	CPUExecutionProvider / cpu	fp16	PASS	ΓÇö	ΓÇö	ΓÇö	ΓÇö
L1	CPUExecutionProvider / cpu	fp32	PASS	64.14 ms	63.23 ms	15.59 samples/sec	+79.8 MB
L1	CPUExecutionProvider / cpu	fp16	PASS	92.11 ms	91.65 ms	10.86 samples/sec	+88.4 MB

• L0 fp32 evidence: single-logit ONNX artifact with output shape [1, 1], initializer dtypes FLOAT=108 BOOL=1 INT64=32, graph node count 196, and model size 90927104 bytes.
• L0 fp16 evidence: single-logit ONNX artifact with output shape [1, 1], initializer dtypes FLOAT16=108 BOOL=1 INT64=32, graph node count 197, and model size 45499992 bytes.
• L2 parity PASS on fp32 CPU: pair_count=4, cosine=0.9999999999999976, max_abs_diff=9.5367431640625e-07, and identical descending rank order between PyTorch and ONNX.

Functional smoke Eval

• Final candidate SHA: 030e79fa6482acb08e51338ab4172ac4387b387d.
• Path and precision: CPUExecutionProvider on cpu, fp32 only.
• Dataset identity and scope: a provenance-preserved grouped fixture built from orgrctera/msmarco_passage_ranking@a7388b9efd4dd4b87a0db91314e5b3f0e4b0d9e6 and joined against the public MS MARCO queries.tar.gz, qrels.dev.tsv, and top1000.dev.tar.gz sources recorded in provenance.json.
• Dataset bounds: exactly 2 real grouped queries, 8 processed pairs, 8 expanded pairs, 0 skipped groups.
• Fan-out caps: max_queries=2, max_negatives=3, beam cap 1, sequence length 512, --samples 2, --no-shuffle, grouped candidates with candidate_id_key=id and candidate_text_key=text.
• Semantics verification: schema, label semantics, and prediction semantics were all verified for grouped reranking with a single raw relevance logit.
• Final functional smoke metrics: MRR@10=1.0, Recall@1=1.0, Recall@10=1.0 on exactly 2 real groups / 8 pairs.
• Provenance statement: those metrics come from the tester-owned final l3_cpu_eval_r7.json plus fixture provenance that pins the Hugging Face revision and published source checksums.
• Bound on claim: this is operability evidence only. It proves final-SHA end-to-end reranking on the bounded grouped CPU fixture and is not a benchmark-quality accuracy claim.
• Capability closure versus the earlier blocker: the prior host-blocked default-dataset path was replaced by a reproducible grouped fixture materializer plus explicit CPU-pinned reranking eval, which removed the authoritative offline-dataset blocker without widening task semantics.

Delta

• Recipe authority:
  • examples/recipes/cross-encoder_ms-marco-MiniLM-L6-v2/cpu/cpu/reranking_fp32_config.json changes /loader/task from text-classification to reranking.
  • examples/recipes/cross-encoder_ms-marco-MiniLM-L6-v2/cpu/cpu/reranking_fp16_config.json changes /loader/task from text-classification to reranking.
  • All export tensors, shapes, and the model-class/export boundary remain on the existing sequence-classification path; the semantic delta is intentionally the surfaced canonical task.
  • The production recipe README remains untouched.
• Code-path delta for the main reranking contribution:
  • src/winml/modelkit/loader/resolution.py: auto-detected text-classification checkpoints now surface reranking when authoritative Hub metadata reports text-ranking, while the Optimum/export boundary still resolves to text-classification.
  • src/winml/modelkit/eval/reranking_evaluator.py: adds grouped reranking evaluation that scores raw single-logit relevance directly, with deterministic tie handling and grouped metrics.
  • src/winml/modelkit/eval/metrics/ranking.py: defines the ranking metrics used by the reranking evaluator.
  • src/winml/modelkit/eval/base_evaluator.py: honors explicit split selection when --dataset points at a locally saved DatasetDict, so the grouped fixture can be consumed reproducibly.
  • src/winml/modelkit/utils/eval_utils.py: registers reranking schema and column conventions.
  • src/winml/modelkit/inference/tasks.py, src/winml/modelkit/inference/pipeline.py, src/winml/modelkit/loader/task.py, src/winml/modelkit/eval/evaluate.py, src/winml/modelkit/datasets/__init__.py, and src/winml/modelkit/models/winml/__init__.py: wire the canonical reranking surface through shared task resolution, inference, and evaluation.
  • scripts/e2e_eval/datasets/build_msmarco_reranking_fixture.py: materializes a tiny grouped MS MARCO fixture by joining the pinned Hugging Face dev rows against the official public query, qrels, and top1000 sources while preserving provenance.
• Reviewer-loop repair delta (final exact head):
  • Grouped fixture selection and emission now preserve authoritative top1000 traversal order instead of positive-first materialization.
  • The bounded selected groups keep positives at emitted index 3 with original authoritative ranks 553 (query 1048585) and 936 (query 2), while the model still ranks both positives first at inference time.
  • Added tie-order regression coverage verifies stable ties respect original authoritative candidate order, and tester r7 manual metric audit confirms MRR@10, Recall@1, and Recall@10 exactly match evaluator output.
• Reducibility consistency with charter: true.
• No-recipe acceptance: the evaluator path now accepts a public model ID plus a materialized grouped fixture for reranking smoke evaluation; no additional semantic override is required for recipe-owned build commands.

Bug fix explanation for the final repair commit

• Symptom and minimal trigger: grouped fixture construction emitted positives before earlier-ranked negatives, which could bias stable-tie metrics away from authoritative candidate order.
• Root cause: fixture materialization separated positives and negatives and then emitted them as positive-first candidates, while tie breaking in ranking metrics is defined by original candidate order.
• Changed symbols and fix mechanism: scripts/e2e_eval/datasets/build_msmarco_reranking_fixture.py now preserves authoritative top1000 order through bounded selection and emitted candidate ordering, and tests/unit/eval/test_reranking_evaluator.py adds tie-order and fixture-order regressions.
• Compatibility and blast radius: the fix is constrained to fixture materialization plus reranking evaluator tests; build/export/perf paths and model/runtime artifacts are unchanged.
• Exact regression evidence: tester r10 preserved L2 and L3 PASS plus reranking quality evidence (10 passed full reranking evaluator, 265 passed focused regression slice, changed-path Ruff PASS), and tester r11 re-ran the two environment-sensitive gates on the exact head with clean-env changed-file mypy PASS and uv lock --check PASS.

Analyze summary ΓÇö component level and op level

Static rule analysis completed with ANALYZE-PARTIAL-SUCCESS; this is compatibility analysis, not runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	embeddings; encoder stack x 6; pooler; single-logit classifier head	Frozen model-breakdown architecture is available, but independent component-to-ONNX grouping remained partial in the earlier tester run because no component mapping artifact had been supplied at that time. The later frozen model breakdown resolved the architecture description, and no ONNX/analyze surfaces changed after that reuse point.	No additional component-specific runtime blocker was introduced by the reranking work; the actionable compatibility detail remains at op level.

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32	196 ops / 16 types	Reshape 61; Gemm 38; Transpose 24; Add 20	QNN: 15 supported operator types, 1 partial, 1 unknown. OpenVINO: 4 supported operator types, 12 unknown. NvTensorRTRTX: all 16 operator types unknown on this host snapshot. CUDA, MIGraphX, Tensorrt, and DML were all-unknown groups in this host snapshot.

• Mapping gap preserved honestly: the earlier standalone analyze JSON captured exhaustive op-level counts, but the first tester analyze pass did not receive a component mapping artifact for independent grouping. The final contribution did not alter build/export/analyze artifacts, so the reused analysis stayed valid rather than being regenerated.

Reproduce commands

The public reproduce path below uses the public model ID, the checked-in grouped fixture materializer, and the pinned dataset provenance recorded by the materialized provenance.json.

$OUT='temp/model-support-repro/ms-marco-MiniLM-L6-v2'
$FIXTURE_DIR="$OUT/msmarco_dev_reranking_fixture"
$CACHE_DIR="$OUT/cache"
# Fixture source pin: orgrctera/msmarco_passage_ranking@a7388b9efd4dd4b87a0db91314e5b3f0e4b0d9e6
# Public source checksums are recorded by the builder in $FIXTURE_DIR/provenance.json.
winml build -c examples/recipes/cross-encoder_ms-marco-MiniLM-L6-v2/cpu/cpu/reranking_fp32_config.json -m cross-encoder/ms-marco-MiniLM-L6-v2 -o $OUT/cpu_fp32
winml build -c examples/recipes/cross-encoder_ms-marco-MiniLM-L6-v2/cpu/cpu/reranking_fp16_config.json -m cross-encoder/ms-marco-MiniLM-L6-v2 -o $OUT/cpu_fp16 --precision fp16
python scripts/e2e_eval/datasets/build_msmarco_reranking_fixture.py --output $FIXTURE_DIR --cache-dir $CACHE_DIR --queries 2 --max-negatives 3
winml perf -m $OUT/cpu_fp32/model.onnx --device cpu --ep cpu
winml perf -m $OUT/cpu_fp16/model.onnx --device cpu --ep cpu
winml analyze --model $OUT/cpu_fp32/model.onnx --ep all --output $OUT/cpu_fp32/analyze_all.json
winml eval -m $OUT/cpu_fp32/model.onnx --model-id cross-encoder/ms-marco-MiniLM-L6-v2 --task reranking --dataset $FIXTURE_DIR --split dev --samples 2 --no-shuffle --column query_column=input --column expected_output_column=expected_output --column metadata_column=metadata --column candidates_column=candidates --column candidate_id_key=id --column candidate_text_key=text --device cpu --ep cpu

[ssss141414]

REQUEST_CHANGES for head 207d9abcce3a21481511d34968d374fa82d87723

Scope and metadata re-check:

• PR Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 is still Draft, base main, head producer/ms-marco-minilm-l6-v2-reranking, and the model-scale-by-skill label is present.
• Exact-head GitHub checks for 207d9abcce3a21481511d34968d374fa82d87723 are not green yet: license/cla is COMPLETED/SUCCESS, while Analyze (Python), lint, test (analyze), test (models), test (optim), test (commands), and test (remaining) are still queued.
• Public conversation/review-thread scrape at verdict time shows no external review comments or open threads.

Owner-tagged issues:

1. PRODUCER: scripts/e2e_eval/datasets/build_msmarco_reranking_fixture.py biases grouped-eval tie handling by reordering candidates so positives always precede negatives. The metric explicitly says ties are broken by original candidate order, and RerankingMetric.update() implements that stable ordering in src/winml/modelkit/eval/metrics/ranking.py. But the fixture builder collects positives and negatives separately and then materializes candidates as (*positives, *negatives), while also preserving the original source ranks separately in candidate_ranks. That means an equal-score tie in grouped eval will prefer the positive passage even when the original authoritative top1000 order had negatives first, inflating MRR@K / Recall@K on the grouped fixture. Please preserve authoritative candidate order in the emitted candidates list, then add a regression test that proves grouped-inline ties do not gain a positive-first advantage from fixture construction.

Independent validation performed:

• Local product checkout points at refs/heads/producer/ms-marco-minilm-l6-v2-reranking with remote origin=https://github.com/microsoft/winml-cli.git.
• Focused loader regression slice passed: 2 passed in 19.63s.
• Focused shared regression slice passed: 262 passed in 26.14s.
• Changed-path Ruff passed.
• uv lock --check evidence shows Resolved 256 packages in 6ms.
• Final-SHA CPU fp32 L3 eval evidence reports MRR@10=1.0, Recall@1=1.0, Recall@10=1.0 on 2 grouped queries / 8 pairs, but the grouped fixture ordering issue above means the tie-handling claim is not yet trustworthy enough to approve.

Re-review gate:

• After the fixture-order fix lands, rerun the focused reranking/loader regression slice and the bounded final-SHA CPU eval, then wait for every required exact-head GitHub check to reach COMPLETED/SUCCESS before requesting another reviewer pass.

[ssss141414]

Addressed on exact head 0286a7d6fcee4ffc103748aa9a85f7c93c407091.

Fix summary:

• Changed files: scripts/e2e_eval/datasets/build_msmarco_reranking_fixture.py, tests/unit/eval/test_reranking_evaluator.py.
• Behavior: grouped fixture selection/emission now preserves authoritative top1000 order (no positive-first materialization), while keeping deterministic bounded selection with required positive coverage.
• Evidence details: both selected groups keep positives at emitted index 3 (original authoritative ranks 553 and 936), and model inference still ranks both positives first.

Validation:

• Tie-order regressions: 3 passed.
• Full reranking evaluator module: 10 passed.
• Focused seven-path slice: 265 passed.
• L3 CPU rerun on grouped fixture: MRR@10=1.0, Recall@1=1.0, Recall@10=1.0.
• Manual metric audit: exact match with evaluator metrics and accounting.

Current review-thread snapshot:

• 1 thread found: Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 (comment)
• Unresolved threads: 1

Please re-review this exact head when convenient.

[ssss141414]

REQUEST_CHANGES for head 0286a7d6fcee4ffc103748aa9a85f7c93c407091

Exact-head gate re-check:

• PR Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 is still Draft, base main, head producer/ms-marco-minilm-l6-v2-reranking, and the model-scale-by-skill label is present.
• Exact-head GitHub checks on 0286a7d6fcee4ffc103748aa9a85f7c93c407091: CodeQL, license/cla, Analyze (Python), test (commands), test (optim), test (remaining), test (analyze), and test (models) are COMPLETED/SUCCESS; lint is COMPLETED/FAILURE.
• Current thread gate: user-provided state is unresolved threads = 0.

Owner-tagged issue:

1. PRODUCER: the failing check is not Ruff. In workflow Lint run 32369963611, job 96428006621, step Type check (required) failed while running the exact required command uv run mypy -p winml.modelkit. In the same job, Check license headers passed and the Lint step (uv run ruff check src/ tests/) passed, so this is a candidate-owned type-check failure on the exact PR head rather than infrastructure. Please fix the mypy diagnostics on this exact head and rerun until lint is green.

Evidence used for this verdict:

• Live PR API confirms head 0286a7d6fcee4ffc103748aa9a85f7c93c407091, draft state, and label.
• Live check-runs API confirms only lint failed on this head.
• Live job metadata API confirms the failing step is Type check (required).
• .github/workflows/lint.yml at this exact head defines that step as uv run mypy -p winml.modelkit.
• The public Actions surface available here did not expose the raw mypy lines from job 96428006621; the public job-logs endpoint returned HTTP 403, so the actionable owner route is to fix or surface the exact mypy diagnostics from the authenticated log and rerun the required gate.

[ssss141414]

ssss141414 Thanks for the re-check on #issuecomment-5356216204.

Addressed on exact head 030e79fa6482acb08e51338ab4172ac4387b387d:

• Fixed the two original mypy diagnostics in src/winml/modelkit/eval/reranking_evaluator.py.
• Fixed the replacement-signature mypy diagnostic by aligning _RawRerankingPipeline.postprocess to the locked TextClassificationPipeline/Pipeline postprocess signature surface while preserving raw ModelOutput passthrough behavior.

Current evidence status on this exact head:

• Clean-env changed-file mypy: PASS.
• uv lock --check: PASS.
• Full reranking evaluator tests: 10 passed.
• Focused reranking regression slice: 265 passed.
• Changed-path Ruff: PASS.
• L2 parity: PASS.
• L3 functional smoke eval: PASS.

Please re-review PR #1322 at this head.

[ssss141414]

REQUEST_CHANGES for head 030e79fa6482acb08e51338ab4172ac4387b387d

Exact-head status re-check:

• PR Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 is still Draft, base main, head producer/ms-marco-minilm-l6-v2-reranking, and the model-scale-by-skill label is present.
• Exact-head GitHub checks for 030e79fa6482acb08e51338ab4172ac4387b387d are all green: Analyze (Python), CodeQL, lint, test (analyze), test (commands), test (models), test (optim), test (remaining), and license/cla all reached COMPLETED/SUCCESS.
• Focused local retained evidence also passed: changed-path Ruff, the 3 tie-order regression tests, the 265-test eval/loader/commands slice, the module-targeted mypy check for winml.modelkit.eval.reranking_evaluator, and uv lock --check.
• The package-wide mypy -p winml.modelkit command fails on this host, but the same error family reproduces from the clean base-main-02179d185f0e snapshot under PYTHONPATH=<base-main>/src, so that is a pre-existing/environmental gate issue rather than a PR regression.

Owner-tagged issue:

1. EXPLAINER (or PRODUCER if you prefer the cleanup path): fresh post-CI thread enumeration still shows 1 unresolved review thread on this exact head, which blocks approval by contract.
   • Thread: Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 (comment)
   • Claim re-verified: the cited TYPE_CHECKING import of torch in src/winml/modelkit/eval/reranking_evaluator.py is unused, while runtime behavior still comes from the local import torch inside _score_pair() and the remaining cast("torch.Tensor", logits) is only a string forward reference.
   • Required action: either remove the unused TYPE_CHECKING import in a follow-up commit, or post an explicit rationale reply for this new thread and resolve it. I cannot post APPROVE while reviewThreads.isResolved == false for any thread.

Independent verification completed on this head:

• Final diff scope contains 22 tracked source/test/recipe files and no temp/scratch payloads or large binary artifacts in the PR diff.
• The tie-order fix is present in scripts/e2e_eval/datasets/build_msmarco_reranking_fixture.py, and targeted tests prove grouped inline ties preserve authoritative candidate order.
• Metric semantics are consistent: RerankingMetric still uses stable score ordering and excludes groups without positives from the scoring denominator; the retained manual metric audit matches evaluator output exactly.
• Fixture provenance is pinned to the stated HF and official MS MARCO sources, and the retained validation evidence confirms the selected candidate IDs preserve authoritative top1000 order with positives not forced first.
• The PR body claims inspected at verdict time are consistent with the verified CPU recipes, reranking evaluator support, and the retained final-SHA CPU fp32 functional-smoke evidence.

Re-review gate:

• After the unresolved thread count reaches zero on head 030e79fa6482acb08e51338ab4172ac4387b387d, rerun the final GitHub thread enumeration and exact-head metadata check, then the PR is ready for an approval-only final pass.

[ssss141414]

APPROVE for head 030e79fa6482acb08e51338ab4172ac4387b387d

Exact-head authoritative reconfirmation:

• PR Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 metadata is unchanged and valid for shipment gating: OPEN, Draft=true, base main, head producer/ms-marco-minilm-l6-v2-reranking, label model-scale-by-skill present.
• Remote branch head matches reviewed SHA exactly: refs/heads/producer/ms-marco-minilm-l6-v2-reranking -> 030e79fa6482acb08e51338ab4172ac4387b387d.
• Checks gate is fully green on this exact SHA: 9/9 SUCCESS (Analyze (Python), CodeQL, lint, test (analyze), test (commands), test (models), test (optim), test (remaining), license/cla), with zero pending and zero failed.
• Thread gate is closed: reviewThreads(first:100) reports totalCount=2, unresolved=0, with resolved thread IDs PRRT_kwDORaJc5s6azQYG and PRRT_kwDORaJc5s6a2T_d.
• Latest existing issue comment remains the prior reviewer REQUEST_CHANGES for this same head; its blocking conditions are now satisfied by green exact-head CI and resolved review thread state.

Structured verification evidence:

• CPU coverage confirmed for both required tuples:
  • CPUExecutionProvider/cpu/fp32: L0 build PASS and L1 perf PASS (mean 64.14 ms, p50 63.23 ms, 15.59 samples/sec, RAM +79.8 MB).
  • CPUExecutionProvider/cpu/fp16: L0 build PASS and L1 perf PASS (mean 92.11 ms, p50 91.65 ms, 10.86 samples/sec, RAM +88.4 MB).
• L3 functional smoke evidence is present and consistent on bounded real grouped data: MRR@10=1.0, Recall@1=1.0, Recall@10=1.0 on exactly 2 groups / 8 pairs.
• Tie-order and metric-semantics integrity is explicitly re-verified: stable tie ordering preserves authoritative candidate order, and manual metric audit matches evaluator output exactly.

Reviewer verdict:

• APPROVE as the independent final authoritative pass for this head.
• Posted as a normal PR comment only (no review API state mutation).

[ssss141414]

REQUEST_CHANGES for head 207d9abcce3a21481511d34968d374fa82d87723

Scope and metadata re-check:

• PR Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 is still Draft, base main, head producer/ms-marco-minilm-l6-v2-reranking, and the model-scale-by-skill label is present.
• Exact-head GitHub checks for 207d9abcce3a21481511d34968d374fa82d87723 are not green yet: license/cla is COMPLETED/SUCCESS, while Analyze (Python), lint, test (analyze), test (models), test (optim), test (commands), and test (remaining) are still queued.
• Public conversation/review-thread scrape at verdict time shows no external review comments or open threads.

Owner-tagged issues:

1. PRODUCER: scripts/e2e_eval/datasets/build_msmarco_reranking_fixture.py biases grouped-eval tie handling by reordering candidates so positives always precede negatives. The metric explicitly says ties are broken by original candidate order, and RerankingMetric.update() implements that stable ordering in src/winml/modelkit/eval/metrics/ranking.py. But the fixture builder collects positives and negatives separately and then materializes candidates as (*positives, *negatives), while also preserving the original source ranks separately in candidate_ranks. That means an equal-score tie in grouped eval will prefer the positive passage even when the original authoritative top1000 order had negatives first, inflating MRR@K / Recall@K on the grouped fixture. Please preserve authoritative candidate order in the emitted candidates list, then add a regression test that proves grouped-inline ties do not gain a positive-first advantage from fixture construction.

Independent validation performed:

• Local product checkout points at refs/heads/producer/ms-marco-minilm-l6-v2-reranking with remote origin=https://github.com/microsoft/winml-cli.git.
• Focused loader regression slice passed: 2 passed in 19.63s.
• Focused shared regression slice passed: 262 passed in 26.14s.
• Changed-path Ruff passed.
• uv lock --check evidence shows Resolved 256 packages in 6ms.
• Final-SHA CPU fp32 L3 eval evidence reports MRR@10=1.0, Recall@1=1.0, Recall@10=1.0 on 2 grouped queries / 8 pairs, but the grouped fixture ordering issue above means the tie-handling claim is not yet trustworthy enough to approve.

Re-review gate:

• After the fixture-order fix lands, rerun the focused reranking/loader regression slice and the bounded final-SHA CPU eval, then wait for every required exact-head GitHub check to reach COMPLETED/SUCCESS before requesting another reviewer pass.