recipe(ms-marco-MiniLM-L2-v2): add CPU reranking recipes

[ssss141414]

Summary

This adds CPU fp32 and fp16 reranking recipes for cross-encoder/ms-marco-MiniLM-L2-v2, an English cross-encoder that emits one raw relevance logit per query-document pair. The shipped Effort and Outcome are L0, stacked on the generic reranking capability owned by dependency PR #1322. Candidate evidence reaches L3 PASS with full CPU fp32/fp16 coverage through L2 and a final-SHA fp32 CPU SciDocs L3 functional smoke.

Model metadata

What the model does

An English cross-encoder passage reranker that jointly encodes one query-document pair and emits one raw scalar relevance logit; candidates are ordered by decreasing logit without softmax, sigmoid, or instruction prefixes.

• Evidence: the pinned checkpoint model card describes MS MARCO passage reranking and decreasing-score sorting; its pinned configuration declares BertForSequenceClassification, num_labels=1, and Identity as the SentenceTransformers cross-encoder activation; the exact-main ONNX build exposes float32 logits with shape [1,1]. Confidence: verified.

Primary user stories

• A user supplies a search query and a bounded candidate passage list to obtain one relevance score per pair and reorder retrieval results. Evidence: pinned model card usage and retrieve-and-rerank description. Confidence: verified.
• A search or RAG pipeline supplies first-stage retrieval candidates to obtain a stronger final ordering before presenting or generating an answer. Evidence: the pinned model card links the checkpoint to information retrieval and retrieve-and-rerank usage. Confidence: mapped.

Supported tasks

• reranking on the checkpoint and Transformers surfaces. Evidence: Hugging Face pipeline_tag=text-ranking, the pinned model-card pair-scoring example, and AutoModelForSequenceClassification loading the pinned single-logit checkpoint. Confidence: verified.
• text-classification on the Transformers, Optimum ONNX, and WinML surfaces. Evidence: the Optimum probe lists BERT text-classification as vendor support, and exact-main WinML inspect and recipe-free build resolve text-classification. Confidence: verified.

Model architecture

BertForSequenceClassification
|-- BertEmbeddings (vocab 30,522 -> hidden 384; positions 512)
|-- BertEncoder layer x 2
|   |-- Bidirectional self-attention (12 heads)
|   |-- Feed-forward (384 -> 1,536 -> 384, GELU)
|   `-- Residual + LayerNorm
|-- BertPooler (first-token hidden state)
`-- Sequence-classification head (384 -> 1 raw relevance logit)

• Source/confidence: pinned checkpoint configuration and BertForSequenceClassification architecture, plus exact-main HTP metadata establishing 15,615,745 parameters, 51 modules, 19 traced modules, two BertLayer children, and 131/131 tagged ONNX nodes (verified).

Validation and support evidence

1. Baseline

• Pinned main: 0876e5ae1c98a169a6137e092e0d7b30bf9cee33.
• WinML version: 0.0.1.dev0.
• Recipe-free build: PASS for BertForSequenceClassification / text-classification, opset 17, with three INT32 [1,512] inputs and FLOAT logits [1,1]; 15,615,745 parameters and 131/131 HTP nodes tagged. Build completed in 12.8s (8.2s export; the CLI also reported 3.6s optimize despite the requested stage-bypass flag) and produced a 62,495,600-byte model.
• CPU perf: PASS over 100 measured iterations after 10 warmups: mean 22.97 ms, p50 22.38 ms, p90 28.67 ms, p95 30.83 ms, p99 36.60 ms, min 16.53 ms, max 36.60 ms, throughput 43.54 samples/s; RAM 665.9 MB before and +79.2 MB inference/total.
• Starting auto-config: inspect, default config, and explicit text-classification resolve identically and pass. Explicit text-ranking and reranking both exit 2 as unsupported on main. On dependency PR Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322, both aliases are byte-equivalent and differ from main auto-config only at /loader/task.
• Baseline Eval: UNSUPPORTED-TASK. reranking Eval exits 1 on main and emits no metric.
• Optimum probe: VENDOR-ONLY. Vendor support includes feature-extraction, fill-mask, multiple-choice, question-answering, text-classification, and token-classification; WinML adds no task. BERT sequence-classification export is vendor-covered, while PR Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 supplies canonical reranking semantics.
• Goal floor: L1.

2. Goal

• Committed Effort: L0.
• Committed Goal ceiling: L3.
• Committed Outcome: L0.
• Success definition: both CPU fp32 and true-fp16 recipes build and load with the exact named I/O contract, pass bounded CPU perf and raw scalar-logit parity/order checks, and the final fp32 candidate completes the pinned bounded public SciDocs reranking smoke with grouped metrics and accounting.
• No ceiling change or re-issued charter occurred.

3. Outcome

The shipped Outcome is L0: exactly two model-specific CPU reranking recipes declare verified fp32 and fp16 support. The highest reached Goal verdict is L3 PASS on final candidate 518dbc8c270e47ef3dfcfbd4c8218ba2d067ddcc, with full required-tuple coverage and no deferred tuples. Both CPU precision tuples pass L0-L2; L3 is the final-SHA fp32 CPU functional smoke only.

The candidate is exactly one two-recipe commit whose parent is dependency SHA 3708969b731425b0c6d4b97920d1b5e6519bb013, the verified head of draft PR #1322. PR #1322 owns generic reranking task resolution, inference/evaluator dispatch, grouped dataset handling, and ranking metrics. This contribution is stacked on that dependency branch and does not claim to be based directly on current main; it must be restacked if #1322 changes.

Exact-candidate quality gates passed: license PASS; Ruff All checks passed!; mypy Success: no issues found in 438 source files; focused regressions 111 passed; analyze partition 1526 passed, 45 skipped; models partition 1534 passed, 6 skipped, 2 xfailed; optim partition 848 passed, 16 skipped, 1 xfailed; commands partition 3641 passed, 9 skipped, 1 warning; remaining partition 871 passed, 2 skipped, 1 deselected.

Learner findings bert-017 through bert-023 retain the resolved configuration, exact HTP profile, 76-operator/16-type analysis, artifact identities and I/O, CPU perf, raw-logit parity/order, and final-SHA SciDocs smoke. These are separate Lane A knowledge changes in e365af419859f287fd37e9a8631637f57437f46b (compare); no Lane A file is included in this model PR. The learner audit reports no methodology friction, no new contract gap, no new verdict shape, and no methodology finding or additional Lane A methodology commit required.

4. Per-EP/device/precision results and Functional smoke Eval

Tier	EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM delta
L0	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-
L0	CPUExecutionProvider / cpu	fp16	PASS	-	-	-	-
L1	CPUExecutionProvider / cpu	fp32	PASS	41.288 ms	37.767 ms	24.22 samples/s	+78.19 MB
L1	CPUExecutionProvider / cpu	fp16	PASS	51.337 ms	49.55 ms	19.48 samples/s	+87.83 MB

• L0 fp32: 62,493,437 bytes; 44 FLOAT, 1 BOOL, and 16 INT64 initializers; input_ids, attention_mask, and token_type_ids are INT32 [1,512]; FLOAT logits are [1,1]; sample logit -11.001389503479004.
• L0 fp16: 31,262,053 bytes; 44 FLOAT16, 1 BOOL, and 16 INT64 initializers; the same INT32 named inputs and FLOAT logits are preserved; sample logit -11.003010749816895; fp16/fp32 size ratio 0.5002453777666285.
• L1 fp32: p95 55.468 ms in addition to the table values.
• L1 fp16: p95 84.476 ms in addition to the table values.
• L2 fp32: ONNX raw logits [8.510398864746094, -4.8600850105285645, -11.681105613708496]; cosine 0.999999999999931; max absolute difference 0.000005245208740234375; reference and ONNX descending order [0,1,2].
• L2 fp16: ONNX raw logits [8.50883960723877, -4.857270240783691, -11.681852340698242]; cosine 0.9999999796576001; max absolute difference 0.0028095245361328125; reference and ONNX descending order [0,1,2].
• L2 reference raw scalar logits: [8.510400772094727, -4.860079765319824, -11.681106567382812] for the three frozen query-document pairs. Both precisions used named attention_mask, input_ids, and token_type_ids, applied no score transform, and preserved descending relevance order.

Functional smoke Eval: L3 PASS on final candidate 518dbc8c270e47ef3dfcfbd4c8218ba2d067ddcc, FP32 CPU, using mteb/scidocs-reranking, split test, pinned revision 56a6d0140cf6356659e2a7c1413286a774468d44. Deterministic first-N streaming selection used no shuffle (recorded seed 42) and selected/processed 2/2 groups, skipped 0, expanded/processed 20/20 pairs, and found 0 groups without a positive. Caps were 2 groups, 10 candidates per group, 20 total pairs, and sequence length 512; the candidate-label/prompt cap was 10, while beam and frame/crop caps were not applicable. The query, positive[], and negative[] schema was verified; positives map to relevance 1 and negatives to 0; exactly one raw float logit is emitted per pair and higher logits rank first without softmax, sigmoid, label mapping, normalization, or instruction transforms. MRR@10 = 0.75, Recall@1 = 0.5, and Recall@10 = 1.0. This proves end-to-end operability only and is not representative accuracy or benchmark quality. No fp16 or accelerator Eval claim is made. The former blocker was main's unsupported reranking task; PR #1322 supplies canonical task normalization, paired-input inference, the raw scalar-logit evaluator, grouped-text adaptation, and MRR/Recall metrics.

5. Delta

The candidate diff against its exact dependency parent contains only these two added files:

• examples/recipes/cross-encoder_ms-marco-MiniLM-L2-v2/cpu/cpu/reranking_fp16_config.json
• examples/recipes/cross-encoder_ms-marco-MiniLM-L2-v2/cpu/cpu/reranking_fp32_config.json

Recipe	JSON pointer	Baseline/dependency auto-config value	Shipped value
fp32	/loader/task	text-classification	reranking
fp32	/quant	null	null
fp16	/loader/task	text-classification	reranking
fp16	/quant/mode	absent in default fp32 auto-config	fp16
fp16	/quant/fp16_keep_io_types	absent in default fp32 auto-config	true

Parsed JSON and WinMLBuildConfig schema semantics pass. Dependency text-ranking and reranking generated configs are byte-equivalent; dependency auto-config equals current-main default after changing only /loader/task from text-classification to reranking. The fp16 precision block realizes true FP16 while preserving FLOAT I/O. Omitted compatibility and optimization fields materialize to the established defaults, including opset 17, hierarchy tags, GELU fusion, and MatMul-Add fusion.

The recipes are identical to the approved L6/L12 family shape apart from the model path and precision realization: AutoModelForSequenceClassification, model type bert, reranking task, three named INT32 [1,512] inputs with ranges [0,30522], [0,2], and [0,2], and one logits output. Recipe-free dependency-head acceptance for both task aliases passes. The delta is reducibility-consistent with the charter, changes no source or tests, and does not duplicate dependency-owned generic behavior. The production recipe README is untouched.

6. Analyze summary - component level and op level

Static rule analysis completed as ANALYZE-PARTIAL-SUCCESS with analyze exit code 1; the summarizer ran afterward and semantic validation succeeded. This is static compatibility analysis, not runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	embeddings; 2x encoder attention/FFN; pooler; scalar classifier	49 mapped, 0 partial, 27 explicitly unmapped; confidence mapped	Root/helper nodes remain explicitly unmapped; no component-level partial or unsupported EP issue

The unresolved mapping gap is 27 root/helper nodes not attributable to one frozen semantic component; repeated encoder layers are collapsed above.

Op-level summary

Artifact	Graph	Dominant ops	Actionable finding
fp32	76 operators / 16 types	Reshape 21; Gemm 14; Add 8; Transpose 8; LayerNormalization 5; Gather 4	QNN partial: OP/ai.onnx/Gather; no unsupported type reported

Complete seven-EP static classifications:

EP	Supported classification	Partial	Unsupported	Unknown
NvTensorRTRTXExecutionProvider	Gather, Add, LayerNormalization, Reshape, Gemm, Transpose, MatMul, Mul, Cast, Flatten, And, Expand, Softmax, Gelu, Tanh	none	none	Where
CUDAExecutionProvider	no rule classifications	none	none	none recorded
MIGraphXExecutionProvider	no rule classifications	none	none	none recorded
QNNExecutionProvider	Gather, Add, LayerNormalization, Reshape, Gemm, Transpose, MatMul, Mul, Cast, Flatten, And, Expand, Softmax, Gelu, Tanh	Gather	none	Where
OpenVINOExecutionProvider	Gather, Add, LayerNormalization, Reshape, Gemm, Transpose, MatMul, Mul, Cast, Flatten, And, Expand, Softmax, Gelu, Tanh	none	none	Where
TensorrtExecutionProvider	no rule classifications	none	none	none recorded
DmlExecutionProvider	no rule classifications	none	none	none recorded

These are complete static classifications, not non-CPU runtime support claims.

7. Reproduce commands

$OUT='temp/model-support-repro/cross-encoder-ms-marco-MiniLM-L2-v2'
winml build -c examples/recipes/cross-encoder_ms-marco-MiniLM-L2-v2/cpu/cpu/reranking_fp32_config.json -m cross-encoder/ms-marco-MiniLM-L2-v2 -o $OUT/fp32
winml build -c examples/recipes/cross-encoder_ms-marco-MiniLM-L2-v2/cpu/cpu/reranking_fp16_config.json -m cross-encoder/ms-marco-MiniLM-L2-v2 -o $OUT/fp16 --precision fp16
winml perf -m $OUT/fp32/model.onnx --ep cpu --device cpu
winml analyze --model $OUT/fp32/model.onnx --ep all --output $OUT/analyze-all.json
winml eval -m $OUT/fp32/model.onnx --model-id cross-encoder/ms-marco-MiniLM-L2-v2 --task reranking --dataset mteb/scidocs-reranking --dataset-revision 56a6d0140cf6356659e2a7c1413286a774468d44 --split test --streaming --no-shuffle --samples 2 --column query_column=query --column positive_column=positive --column negative_column=negative --column max_candidates=10 --ep cpu --device cpu -o $OUT/scidocs.json --overwrite

[ssss141414]

Independent reviewer verdict: APPROVE

Reviewed SHA: 518dbc8c270e47ef3dfcfbd4c8218ba2d067ddcc

Stacked dependency: PR #1322 branch producer/ms-marco-minilm-l6-v2-reranking at 3708969b731425b0c6d4b97920d1b5e6519bb013; the candidate is one commit whose parent is that exact SHA.

Coverage and engineering judgment

• Coverage is full. Reachable verified tuples are CPUExecutionProvider/cpu/fp32 and CPUExecutionProvider/cpu/fp16, both PASS through L0-L2. The representative final-SHA L3 smoke is fp32 CPU on pinned SciDocs, 2 groups / 20 pairs. There are no deferred tuples.
• The dependency-relative diff is exactly the two precision-suffixed CPU recipe JSONs. examples/recipes/README.md is unchanged; no source, tests, skill files, or unrelated paths are present.
• I read both recipes line-by-line. They match the approved L6 family templates: BERT sequence classification exported as canonical reranking, three correctly named INT32 [1,512] inputs with checkpoint-consistent ranges, scalar logits, opset 17, fp32 quant:null, and fp16 mode:fp16 with preserved float I/O. The baseline/config delta is reducible to dependency-owned task normalization plus the explicit precision realization; no model-id branching or hidden shared-code fix is present.
• The PR is draft, has model-scale-by-skill, and its body has the required Summary -> Model metadata -> Validation hierarchy with source-faithful profile, baseline, goal, outcome, tuple rows, delta, analyze summary, and reproduction commands. The baseline SHA equals current upstream main.

Independent evidence checks

• Fresh detached checkout: HEAD 518dbc8c, parent 3708969b, clean status. Independent recipe/template/schema checks passed.
• Independent ONNX rehash/inspection passed: fp32 155247984f6cf514f37aaa320b605c296a69e487f97178dc715ae80728706b6c at 62,493,437 bytes with 44 FLOAT initializers; fp16 08945efdcc246f057ddc0cd995b5419ffd1e91e06580a5cf01e15ba306fd9c51 at 31,262,053 bytes with 44 FLOAT16 initializers. Ratio 0.5002453778; both expose the exact named INT32 inputs and FLOAT [1,1] logits.
• Build logs contain Build complete; the exact hosted workflow builds fp16 with --precision fp16 and no --no-quant. Perf realizes CPU fp32/fp16 at 41.288/51.337 ms mean and reports memory. Raw-logit parity/order passes for both precisions. L3 reports MRR@10 0.75, Recall@1 0.5, Recall@10 1.0 with explicit operability-only scope.
• Analyze is honestly ANALYZE-PARTIAL-SUCCESS: analyze exit 1, wrapper exit 0, 76 operators / 16 types, 49 mapped / 27 explicit gaps, and all seven requested EP rows are parseable. QNN Gather is partial; static rows are not presented as non-CPU runtime proof.
• Tester r12 is immutable and semantically consistent: 81 sealed files, independent rehash 81/81 with zero mismatches, completed/PASS/exit-0 identities aligned, and candidate-local environment provenance tied to the exact lock and checkout.
• Learner findings bert-017 through bert-023 are present and bounded honestly. Lane A branch resolves remotely to e365af419859f287fd37e9a8631637f57437f46b; the no-friction declaration does not conceal candidate/model failures or a new contract gap.

Quality, checks, and threads

• Exact-candidate hosted run 32668856421 checks out ssss141414/winml-cli@518dbc8c, hydrates the exact lock, and completed all seven jobs SUCCESS: license PASS; Ruff PASS; whole-package mypy PASS (438 files); analyze 1526 passed; models 1534 passed; optim 848 passed; commands 3641 passed; remaining 871 passed; focused dependency regressions 111 passed. I inspected the workflow source and candidate-local provenance. A local duplicate hydration attempt was blocked before gates by a host TLS handshake failure fetching markdown; no candidate failure was inferred.
• Final exact-SHA PR rollup: license/cla COMPLETED/SUCCESS. No queued, in-progress, failed, cancelled, or missing triggered check is present.
• Pre-score and final pagination both found 0 conversation comments, 0 line comments, 0 review threads / 0 open.

Blockers: none.

This approval is full reviewer-contract coverage for the two reachable fp32/fp16 CPU tuples. It does not authorize changing the PR from draft.