recipe(mmarco-mMiniLMv2-L12-H384-v1): add CPU reranking recipes

[ssss141414]

Summary

This adds CPU fp32 and fp16 reranking recipes for cross-encoder/mmarco-mMiniLMv2-L12-H384-v1, a multilingual cross-encoder that emits one raw relevance logit per query-passage pair. The shipped Effort and Outcome are L0, layered on the generic reranking capability owned by dependency PR #1322. Candidate evidence reaches L3 PASS with full CPU fp32/fp16 coverage; exact current-main hosted evidence establishes the recipe-free fp32 baseline while confirming that current-main reranking config and Eval are unsupported.

Model metadata

What the model does

A multilingual cross-encoder reranker that jointly tokenizes a query and candidate passage, then emits one raw relevance logit used to sort passages in descending order.

• Evidence: the pinned checkpoint model card documents paired-query usage and decreasing-score ordering; its configuration selects XLMRobertaForSequenceClassification with one label and Identity activation; Transformers v5.14.1 supplies the one-logit sequence-classification path. Confidence: verified.

Primary user stories

• A user supplies a search query and retrieved candidate passages to obtain relevance scores for second-stage passage reranking. Evidence: pinned model card Information Retrieval and retrieve-and-rerank usage. Confidence: verified.
• A user supplies multilingual query-passage pairs to rank candidates in any of the 14 mMARCO languages advertised by the checkpoint. Evidence: pinned model card training statement and checkpoint language tags. Confidence: verified.

Supported tasks

• reranking on the checkpoint and Transformers surfaces. Evidence: checkpoint pipeline_tag=text-ranking and the paired-input raw-logit model-card example. Confidence: verified.
• text-classification on the Transformers and WinML surfaces. Evidence: checkpoint Transformers metadata selects AutoModelForSequenceClassification; the existing exact-model text-classification fp16 recipe remains unchanged. Confidence: mapped.

Model architecture

XLMRobertaForSequenceClassification
|-- XLM-R embeddings (250002 vocab, 384 hidden, learned absolute positions)
|-- Encoder stack x 12
|   |-- Self-attention (12 heads x 32 dimensions)
|   |-- Feed-forward (384 -> 1536 -> 384, GELU)
|   `-- Residual connections + LayerNorm
`-- Classification head on <s> (384 -> 384, tanh -> 1 raw logit)

• Source/confidence: pinned checkpoint configuration and Transformers v5.14.1 XLMRobertaForSequenceClassification source (verified).

Validation and support evidence

1. Baseline

• Pinned main: 0876e5ae1c98a169a6137e092e0d7b30bf9cee33.
• WinML version: winml, version 0.3.0. Hosted recovery supplied an exact-lock current-main environment after the earlier local hydration attempt was blocked.
• Recipe-free build: PASS in 111.313 seconds with auto-config, CPU, no analyze/optimize/quantize/compile, producing fp32 opset 17 with input_ids and attention_mask INT32 [1,512], FLOAT logits [1,1], 435 nodes, and external data.
• CPU perf: PASS over 3 iterations after 1 warmup: mean 210.972 ms, p50 211.587 ms, throughput 4.74 samples/s, and RSS total delta +80.81 MB.
• Starting auto-config resolves AutoModelForSequenceClassification, xlm-roberta, and text-classification. Explicit text-ranking and reranking config requests both exit 2. Current-main reranking Eval is UNSUPPORTED-TASK, exits 1, and emits no metrics.
• Optimum statically advertises vendor text-classification, and WinML adds no task: VENDOR-ONLY. The exact-lock live probe could not import Optimum's ONNX model-config module because the locked Optimum/Transformers pair is incompatible (_CAN_RECORD_REGISTRY import failure), so its evidence remains STATIC-MAPPED; LIVE-PROBE-LOCK-INCOMPATIBILITY; this does not become a claim that WinML added reranking support.
• Goal floor: L0 is the first required tier. The hosted run is actual current-main baseline evidence, not candidate evidence.

2. Goal

• Committed Effort: L0.
• Committed Goal ceiling: L3.
• Committed Outcome: L0.
• Success definition: CPU fp32/fp16 reranking build, perf, multilingual raw-logit parity and ordering, and a pinned bounded smoke through the shipped reranking evaluator.
• No ceiling change or re-issued charter occurred. Candidate evidence reached L3 PASS with full coverage and no deferred tuples; this does not change the shipped L0 Outcome.

3. Outcome

The shipped Outcome is L0: exactly two model-specific CPU reranking recipes declare verified fp32 and fp16 coverage while generic reranking remains owned by dependency PR #1322. The highest reached Goal verdict is L3 PASS on candidate bc7377782d96a4788e565ff3acf0a6c2f06ae56d, with full required-tuple coverage and no deferred tuples.

Learner findings xlm-roberta-005 through xlm-roberta-009 retain the resolved autoconf/HTP facts, complete static analysis, fp32/fp16 structure and bounded perf, raw-logit parity/order, and pinned Chinese functional-smoke evidence. They are separate Lane A knowledge changes, as is methodology finding _meta-113 (Lane A commit 113413ae, compare); none is included in this model PR.

Exact-candidate quality: Ruff passed, mypy reported no issues in 438 source files, and the five candidate test partitions passed an aggregate 8420 tests: models 1534 passed / 6 skipped / 2 xfailed; optim 848 passed / 16 skipped / 1 xfailed; commands 3641 passed / 9 skipped / 1 warning; analyze 1526 passed / 45 skipped; remaining 871 passed / 2 skipped / 1 deselected / 1 warning. Aggregate non-pass accounting was 78 skipped, 3 xfailed, and 1 deselected.

The original exact-head validation workflow concluded failure: its model job stopped when winml analyze returned exit 1 for partial/unknown static findings, so the wrapper skipped its summarizer even though complete JSON had already been emitted and was independently validated. This was a harness false negative and a non-candidate blocker, not a candidate change or a claim that the original workflow was green.

Methodology finding _meta-113 is now mechanism confirmed by validation-only run 32660811255 on unchanged candidate bc7377782d96a4788e565ff3acf0a6c2f06ae56d, through closed, unmerged draft PR #1341. The repaired control flow captured analyze exit 1, continued to the summarizer, validated the complete ANALYZE-PARTIAL-SUCCESS result with 387 operators, 20 unique types, and all 7 requested EP rows, then returned wrapper exit 0 with both validator and final job successful. This exercises the repair without changing the candidate.

4. Per-EP/device/precision results and Functional smoke Eval

Tier	EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM delta
L0	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-
L0	CPUExecutionProvider / cpu	fp16	PASS	-	-	-	-
L1	CPUExecutionProvider / cpu	fp32	PASS	234.276 ms	227.901 ms	4.27 samples/s	+80.53 MB
L1	CPUExecutionProvider / cpu	fp16	PASS (bounded: 1 iteration, 0 warmups)	282.673 ms	282.673 ms	3.54 samples/s	+44.33 MB

• L0 fp32: input_ids and attention_mask INT32 [1,512]; FLOAT logits [1,1]; 204 FLOAT initializers; artifact size 470724506 bytes.
• L0 fp16: input_ids and attention_mask INT32 [1,512]; FLOAT logits [1,1]; 204 FLOAT16 initializers; artifact size 235435418 bytes.
• L2 fp32: cosine 0.9999999999999137, max absolute difference 0.0000057220458984375; reference and ONNX descending order [0,1,2].
• L2 fp16: cosine 0.9999999857145679, max absolute difference 0.0023097991943359375; reference and ONNX descending order [0,1,2].
• L2 scope: exactly three pairs at sequence length 512, with English, German, and unrelated Chinese passages. Scores were Identity-activated raw scalar logits with no postprocessing.

Functional smoke Eval: L3 PASS on final candidate bc7377782d96a4788e565ff3acf0a6c2f06ae56d, FP32 CPU, using C-MTEB/Mmarco-reranking, configuration default, split dev, pinned revision 8e0c766dbe9e16e1d221116a3f36795fbade07f6. Deterministic first-N selection with no shuffle retained source candidate order and processed/scored 2/2 groups, 20/20 expanded pairs, with zero skipped groups and zero groups without a positive. Caps were 2 groups, 10 candidates per group, 20 total pairs, and sequence length 512. Schema (query, positive, negative), positive/negative relevance labels, and descending raw-logit prediction semantics were verified. MRR@10 = 0.333333, Recall@1 = 0.0, and Recall@10 = 1.0. This Chinese run is functional smoke only: it proves end-to-end evaluator operability, not benchmark accuracy or multilingual quality. No fp16 or accelerator Eval claim is made. The former blocker was the absence of a WinML reranking evaluator; dependency PR #1322 supplies generic task resolution, paired-input inference, raw-logit evaluation, grouped relevance metrics, and dataset adaptation.

5. Delta

The candidate diff contains exactly these two model-specific recipes:

• examples/recipes/cross-encoder_mmarco-mMiniLMv2-L12-H384-v1/cpu/cpu/reranking_fp32_config.json
• examples/recipes/cross-encoder_mmarco-mMiniLMv2-L12-H384-v1/cpu/cpu/reranking_fp16_config.json

Recipe	JSON pointer	Existing legacy value	Shipped value
fp32	/loader/task	text-classification	reranking
fp32	/quant	fp16 quantization block	null
fp32	/eval	absent	Pinned C-MTEB default/dev plan; revision 8e0c766dbe9e16e1d221116a3f36795fbade07f6; samples 2; shuffle false; streaming true; query/positive/negative columns; max candidates 10
fp16	/loader/task	text-classification	reranking
fp16	/quant/task	text-classification	reranking
fp16	/eval	absent	Pinned C-MTEB default/dev plan; revision 8e0c766dbe9e16e1d221116a3f36795fbade07f6; samples 2; shuffle false; streaming true; query/positive/negative columns; max candidates 10

The current-main generated auto-config was recovered and compared field by field after stripping note-only fields:

Shipped recipe	Identical	Different	Precision-specific
fp32	27	14	0
fp16	26	14	22

• The 14 differences for each recipe are complete: 12 shipped /eval leaves absent from auto-config (task; dataset path/name/revision/split/samples/shuffle/streaming; query/positive/negative/max-candidates mappings), /loader/task changes text-classification to reranking, and auto-config-only /export/compatibility/transformers_attention=eager is omitted. The eval block encodes the Producer-owned pinned bounded smoke plan; the loader delta selects dependency PR Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322's paired-input reranking semantics; omitting the compatibility hint preserves the existing exact-model recipe contract.
• The common identical fields cover /compile; /optim/clamp_constant_values; loader model class/type; export batch, cleanup, folding, dynamo, hierarchy, parameter, verbosity and opset settings; both named INT32 input shapes and value ranges; and the logits output name. fp32 also has identical /quant=null.
• The 22 fp16 precision-specific pointers are complete: /quant itself plus /quant/{activation_symmetric,activation_type,calibration_load_path,calibration_method,calibration_save_path,distribution,fp16_keep_io_types,fp16_op_block_list,mode,model_id,model_type,nodes_to_exclude,op_types_to_quantize,per_channel,samples,save_calibration,seed,symmetric,task,weight_symmetric,weight_type}. They have no like-for-like value because current-main auto-config was generated without a precision flag; this is a precision realization, not an auto-config regression.

Relative to the existing exact-model legacy recipe, export, optimization, and compile configuration are identical; the recipes preserve AutoModelForSequenceClassification, xlm-roberta, opset 17, INT32 [1,512] named inputs, no token_type_ids, and one logits output. The existing text-classification recipe is unchanged. No source, tests, workflow files, or production recipe README are changed. Current-main recipe-free fp32 acceptance passes, while current-main reranking config/Eval remain unsupported; generic reranking behavior and acceptance remain dependency PR #1322 ownership. The two-recipe delta is reducibility-consistent with the charter and contains no checkpoint-specific shared-code behavior.

6. Analyze summary - component level and op level

Static rule analysis completed as ANALYZE-PARTIAL-SUCCESS with exit code 1; complete JSON was emitted and independently validated. This is static compatibility analysis, not accelerator runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	embeddings 16; 12-layer self-attention 192; feed-forward 48; classifier 4	260 mapped, 127 explicitly unmapped, 387 total; mapped with explicit unmapped bucket	QNN partial Gather/GatherElements affects embeddings; Gather also affects classifier; Where is in the unmapped bucket

The unresolved mapping gap is 127 optimizer-generated/helper nodes in fp32; repeated encoder layers are collapsed above.

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32	387 operators / 20 types	Reshape 121; Gemm 74; Transpose 48; Add 39; MatMul 24	NvTensorRTRTX and OpenVINO: 19 supported types, Where unknown; QNN: 17 supported types, Gather/GatherElements partial, Where unknown

No unsupported operator type was reported. CUDA, MIGraphX, TensorRT, and DML have no rule classifications. These are complete per-EP static findings, not runtime support claims.

7. Reproduce commands

$OUT='temp/mmarco-mminilmv2-l12-h384-v1-test'
uv run winml build -c examples/recipes/cross-encoder_mmarco-mMiniLMv2-L12-H384-v1/cpu/cpu/reranking_fp32_config.json -m cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 -o $OUT/fp32
uv run winml build -c examples/recipes/cross-encoder_mmarco-mMiniLMv2-L12-H384-v1/cpu/cpu/reranking_fp16_config.json -m cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 -o $OUT/fp16 --precision fp16
uv run winml analyze --model $OUT/fp32/model.onnx --ep all --output $OUT/analyze-all.json
uv run winml perf -m $OUT/fp32/model.onnx --device cpu --ep cpu --iterations 3 --warmup 1
uv run winml eval -m $OUT/fp32/model.onnx --model-id cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 --task reranking --dataset C-MTEB/Mmarco-reranking --dataset-name default --dataset-revision 8e0c766dbe9e16e1d221116a3f36795fbade07f6 --split dev --samples 2 --no-shuffle --streaming --column query_column=query --column positive_column=positive --column negative_column=negative --column max_candidates=10 --ep cpu --device cpu

[ssss141414]

REQUEST_CHANGES

Reviewed exact candidate bc7377782d96a4788e565ff3acf0a6c2f06ae56d on parent/base 3708969b731425b0c6d4b97920d1b5e6519bb013.

Blocking changes

• Planner / Explainer: refresh the missing current-main baseline and recipe-vs-auto-config delta. The current Reviewer checklist requires: "Baseline probe ran + recipe-vs-winml config diff present"; Planner Step 1b further says every recipe PR must cite the baseline build command/output, winml --version, current/evidence commit, and the actual recipe-vs-winml config diff. The frozen charter instead records config/build/perf/eval as NOT-RUN, has no generated config, and the Producer correctly reports recipe_comparison: NOT-COMPARABLE. Candidate evidence on the stacked Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 state cannot establish current origin/main behavior. Current main is still the charter SHA 0876e5ae1c98a169a6137e092e0d7b30bf9cee33, and exact-lock hydration succeeded in run 32655153048, so issue a replacement charter after running the main-only baseline/config in a clean exact-lock environment, then refresh dependent handoffs and the PR body. Preserve an exact failure if the baseline itself fails; do not infer it from candidate success.
• Tester / Learner: exercise the _meta-113 agent-contract change end to end. Lane A is separately pushed: 97f5f65a156f4d956d3b23941e280d38ea6663fd adds the paired Tester/Reviewer rules and 8d517867d8c2e9d4418098b8c2cb722191f3aa5b adds the finding/model knowledge. However, _meta-113 explicitly records mechanism_confirmed: false and says the repaired wrapper has not been rerun. The Reviewer skill_meta gate requires new agent-file content to be exercised once end to end. Run a repaired non-terminating analyze wrapper that reaches the completeness validator after exit 1, seals the resulting ANALYZE-PARTIAL-SUCCESS, and update/push the finding with that evidence.

Verified checks

• PR recipe(mmarco-mMiniLMv2-L12-H384-v1): add CPU reranking recipes #1339 is OPEN, DRAFT, model-scale-by-skill, MERGEABLE/CLEAN, exact head/base, and changes exactly the two CPU reranking recipe JSON files. No source, tests, README, workflow, or skill leakage.
• Both recipes preserve the existing XLM-R contract: input_ids + attention_mask INT32 [1,512], no token_type_ids or instruction prefix, scalar [1,1] logits, fp32 quant:null, and true fp16 mode with retained IO types. Existing text-classification recipe semantics are unchanged.
• Sealed Tester root passed pre-seal semantic validation, 80-file acceptance seal plus independent rehash, terminal 87-file rehash, and all seven downloaded Actions ZIPs matched GitHub-published SHA-256 digests.
• Exact candidate checkout/assert and candidate-local .venv/lock/import provenance are present. Ruff and mypy passed; all five test partitions passed: 8420 passed, 78 skipped, 3 xfailed, 1 deselected.
• Run 32655153048 model phases build fp32/fp16, structure/L2, perf fp32/fp16, L3, and rules all succeeded. Analyze emitted complete seven-EP JSON with 387 operators/20 types, QNN partial Gather/GatherElements, and unknown Where; exit 1 alone skipped the summarizer. This is conclusively a harness-only _meta-113 false negative, not a candidate quality failure. PR body does not claim all jobs green.
• L3 is full CPU fp32/fp16 coverage with no deferred tuples. FP32 CPU shipped-Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 evaluator smoke used pinned C-MTEB/Mmarco-reranking default/dev revision 8e0c766dbe9e16e1d221116a3f36795fbade07f6: 2/2 groups, 20 pairs, 0 skipped, MRR@10 0.333333, Recall@1 0.0, Recall@10 1.0; functional-smoke only.
• Dependency PR Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 remains OPEN/DRAFT at 3708969b731425b0c6d4b97920d1b5e6519bb013 with 9/9 checks successful.
• Threads enumerated: 0 total, 0 open. Pre-verdict line comments: 0; conversation comments: 0; reviews: 0.
• PR recipe(mmarco-mMiniLMv2-L12-H384-v1): add CPU reranking recipes #1339 status rollup is only license/cla: SUCCESS; it is not being mistaken for exact-head evidence.

Leave the PR draft.

[ssss141414]

APPROVE

Reviewed the full PR afresh at exact candidate bc7377782d96a4788e565ff3acf0a6c2f06ae56d on exact parent/base 3708969b731425b0c6d4b97920d1b5e6519bb013.

Prior blockers resolved

• Current-main baseline: run 32658426722 succeeded after exact checkout/assert of current main 0876e5ae1c98a169a6137e092e0d7b30bf9cee33 and a main-local exact-lock environment. Raw records show recipe-free fp32 build Build complete, opset 17, named INT32 [1,512] inputs and FLOAT [1,1] logits; CPU perf mean/p50 210.972/211.587 ms, 4.74 samples/s, +80.81 MB; current-main reranking config exits 2 and Eval exits 1 as unsupported with no metrics. Charter v2 explicitly supersedes v1, preserves Effort L0 / Goal L3 / Outcome L0 and the full per-model recipe fix classification, and has blocking_questions: []. The recovered auto-config is sealed and compared field-by-field against both candidate recipes: fp32 27 identical / 14 different; fp16 26 identical / 14 different / 22 precision-specific, with every delta accounted for.
• _meta-113 exercise: run 32660811255 and closed-unmerged draft PR #1341 used exact candidate checkout/assert plus candidate-local exact-lock provenance. Analyze exit 1 was captured without fail-fast termination; the summarizer then ran, the validator accepted complete 387 operators / 20 types / 7 requested EP rows, wrapper exit was 0, and the job succeeded. Learner records mechanism_confirmed: true; paired Tester/Reviewer edits are in pushed Lane A commit 97f5f65a156f4d956d3b23941e280d38ea6663fd, and confirmation is pushed at 113413ae5580e1c78c0afce20c5b14b0436f8d14.

Full review

• PR is OPEN, DRAFT, model-scale-by-skill, MERGEABLE/CLEAN. The diff is exactly the two CPU reranking recipe JSON files; no source, test, workflow, production README, or skill leakage. Both strict-parse with the committed hashes and preserve the XLM-R contract: input_ids/attention_mask INT32 [1,512], scalar logits, fp32 quant:null, and true fp16 mode with retained I/O types.
• Body hierarchy and all seven evidence sections are complete, values are source-owned, commands are portable, and no internal absolute/run-root paths appear. Model metadata matches the frozen breakdown.
• Goal ladder is coherent with no short-circuit: L0 fp32/fp16 structural PASS; L1 CPU fp32/fp16 PASS; L2 raw-logit cosine/order PASS; representative final-SHA fp32 CPU L3 PASS. Required tuple coverage is full for CPU fp32/fp16 with no deferred tuples. The pinned C-MTEB/Mmarco-reranking smoke processed 2/2 groups and 20/20 pairs, skipped 0, retained explicit fan-out/sequence caps and relevance semantics, and correctly limits its claim to operability rather than benchmark accuracy.
• Analyze publication is complete at component and op levels: 387 operators / 20 types, mapped architecture counts plus an explicit unmapped bucket, QNN partial Gather/GatherElements, unknown Where, and no fabricated accelerator runtime claim.
• Exact-head quality evidence satisfies _meta-112: candidate-local checkout/environment/interpreter/lock/import roots, Ruff PASS, mypy PASS in 438 files, and all five non-hardware partitions PASS (8420 passed, 78 skipped, 3 xfailed, 1 deselected). I independently recomputed the original Tester 87-file seal and meta-113 42-file seal with zero mismatches, and rehashed the hosted baseline ZIP plus critical checkout/provenance/command/comparison records against its external seal.
• Learner model entries xlm-roberta-005 through 009 preserve concrete HTP, analysis, true-fp16/perf, L2, and L3 findings. Dependency PR Add WinML reranking support for cross-encoder/ms-marco-MiniLM-L6-v2 #1322 remains OPEN/DRAFT at exact base 3708969b731425b0c6d4b97920d1b5e6519bb013 with 9/9 checks successful.
• Threads enumerated: 0 total, 0 open, pagination complete. Line comments: 0. The stacked base suppresses normal PR workflows, so PR recipe(mmarco-mMiniLMv2-L12-H384-v1): add CPU reranking recipes #1339's lone license/cla: SUCCESS was not treated as quality evidence; the sealed exact-head run above supplies the required validation.

No residual blockers. Leave the PR draft; this comment does not authorize readiness or any other metadata change.