recipe(ms-marco-MiniLM-L12-v2): add CPU reranking recipes - #1334
Conversation
REQUEST_CHANGESReviewed Blocking owner actions
Verified gates
Reachable vs deferred
|
07a7ee4 to
8dae90f
Compare
|
Addressed the three requested changes on candidate
The PR remains draft with |
REQUEST_CHANGESReviewed head Blocking owner action
Re-entry verification
This is an internal evidence-integrity correction only; no product-code or recipe defect was found, and the PR must remain draft. |
|
R4 addressed in a fresh immutable Tester root: the controller was corrected, the bounded true-fp16 CPU inference reran at the unchanged |
APPROVEReviewed head Verified outcome
Gates
Blockers: none. This is a skill-level opinion comment only; it does not alter GitHub review state or authorize readiness changes. |
Summary
This adds CPU fp32 and fp16 reranking recipes for
cross-encoder/ms-marco-MiniLM-L12-v2, a cross-encoder that scores query-document pairs for ranking. The contribution has Effort L2 and Outcome L2 because its independent delta remains recipe-only while generic L2 reranking support is owned by stacked dependency PR #1322. The highest Goal verdict reached is L3 PASS with full coverage across both required CPU precision tuples.Model metadata
What the model does
Cross-encoder reranker that scores a query-document pair with a single scalar relevance logit for ranking. Evidence: Hugging Face model metadata pipeline tag and model card task tags; pinned checkpoint configuration identifies
BertForSequenceClassificationwithnum_labels=1. Confidence:verified.Primary user stories
mapped.inferred.Supported tasks
text-rankingacross the checkpoint, Transformers, Optimum ONNX, and WinML support surfaces. Evidence: pipeline tag, WinML inspect pipeline task, and Optimum probe tasks. Confidence:mapped.Model architecture
cross-encoder/ms-marco-MiniLM-L12-v2configuration and Transformers BERT class structure (mapped).Validation and support evidence
Baseline
Current
mainand the frozen baseline remain0876e5ae1c98a169a6137e092e0d7b30bf9cee33with WinML0.0.1.dev0. Recipe-free auto-resolution usedtext-classification: build PASS in 49.9s, followed by perf PASS at 98.69 ms mean, 98.21 ms p50, 10.13 samples/s, and +79.4 MB RAM. The build output reported an Optimize stage despite--no-optimize. Canonicaltext-rankingconfig was unsupported by TasksManager (exit 2), and canonical text-ranking eval was unsupported (exit 1). The Optimum probe wasVENDOR-ONLY: BERT export was vendor-covered for text-classification, with no task added by WinML at baseline.The dependency baseline for this stacked contribution is PR #1322 at
3708969b731425b0c6d4b97920d1b5e6519bb013, advanced by fast-forward from030e79fa6482acb08e51338ab4172ac4387b387d. The current L12 candidate is8dae90fbd75c30c79d0bd5ec60f7817d23d5bd6d, with that exact dependency commit as its parent.Goal
The L2 context is stacked: PR #1322 owns generic reranking task resolution, inference/evaluator dispatch, grouped ranking data handling, and ranking metrics. This independent commit adds model recipes only and does not claim to add generic source support.
Outcome
L3 PASS was reached with
fullcoverage. Both required tuples,CPUExecutionProvider/cpu/fp32andCPUExecutionProvider/cpu/fp16, pass L0, L1, and L2; the bounded fp32 public evaluator smoke passes L3. Deferred tuples:[]. Unresolved tuples:[]. The shipped Outcome remains L2 because this PR's independent delta contains exactly two CPU recipes and relies on PR #1322 for generic L2 reranking code.Learner findings v4 retain
bert-011,bert-013,bert-014,bert-015, andbert-016, and update:bert-012: L12 reranker CPU fp16 completes a bounded one-run probe despite historical multi-iteration timeouts. The current tuple is PASS; scaled-run completion and a stable latency distribution remain unmeasured.The methodology audit is not a no-friction result: reviewer R4 exposed a hash-valid root with contradictory terminal-state semantics, so learner finding
_meta-111now requires semantic agreement across terminal-state and identity fields in addition to immutable roots and hashes. Lane A pushed the implementation in2c5700a047fc85e5fb3405d361f8fb06e1c846fdand exact-SHA metadata finalization ine70be83aefd621dbb4170aa647fe6344a6d4884c; review it at https://github.com/gim-home/ModelKitArtifacts/compare/main...ssss141414/learner-l12-bert-knowledge-20260823?expand=1. These knowledge and skill edits remain separate from this model PR.Per-EP/device/precision results and Functional smoke Eval
[1,512]int32 inputs;[1,1]float logits[1,512]int32 inputs;[1,1]float logitsHistorical context only: prior candidate
07a7ee4fa097210a43e2ca9ce7f68b659a2cedb0exceeded 2700 seconds at 2743.499 seconds for a 30-iteration/5-warmup run, and a 10-iteration/3-warmup attempt reached 600 seconds after setup without a result. Those bounded prior attempts are superseded as the tuple verdict by the current one-sample PASS and are not stable benchmark evidence.R4 terminal-state integrity is repaired in a fresh immutable Tester v3 root: the corrected controller records result
PASS, phasecompleted, and outer exit0. Cross-file validation reports semantic agreement, and the 26-entry final seal reports zero hash mismatches. The old sealed root is preserved only as incident provenance and is not authoritative for the current fp16 result.L2 named-input parity PASS for both artifacts over three pairs: fp32 cosine
0.9999999999999853, max absolute difference0.000002384185791015625; fp16 cosine0.9999990532003638, max absolute difference0.03248405456542969. L0 and L2 were reused only after range-diff and blob comparison proved both recipes byte-identical across the restack; the dependency delta is Eval-only.Functional smoke Eval
L3 PASS on final candidate
8dae90fbd75c30c79d0bd5ec60f7817d23d5bd6d, FP32CPUExecutionProvider/cpu, through the publicwinml evalcommand andwinml.modelkit.eval.reranking_evaluator.WinMLRerankingEvaluator. The pinned dataset ismteb/scidocs-rerankingrevision56a6d0140cf6356659e2a7c1413286a774468d44, splittest, streamed without shuffle. The deterministic first 2 groups were selected and processed with 0 skipped groups, producing 20 query-document pairs. Positive and negative passage text was materialized before scoring; positives are relevant, negatives are non-relevant, and each pair produces one scalar relevance logit ranked in descending order. Caps were 10 candidates per group and sequence length 512.Results: MRR@10
1.0, Recall@11.0, and Recall@101.0; 2 scored groups and 0 groups without a positive. This is a bounded functional operability smoke only, not representative accuracy or benchmark quality, and it does not claim fp16 Eval accuracy. Baseline main could not run canonical text-ranking Eval; PR #1322 at3708969b731425b0c6d4b97920d1b5e6519bb013supplies the generic public evaluator capability exercised here.Delta
The independent PR #1334 delta adds exactly:
examples/recipes/cross-encoder_ms-marco-MiniLM-L12-v2/cpu/cpu/reranking_fp32_config.jsonexamples/recipes/cross-encoder_ms-marco-MiniLM-L12-v2/cpu/cpu/reranking_fp16_config.jsonThere are no modified or deleted files and no source, test, or README changes; the production recipe README remains untouched. The canonical baseline recipe is
NOT-COMPARABLE: baseline TasksManager could not emit canonical text-ranking config, so there are no baseline JSON-pointer old/new values. Both recipe blobs are byte-identical to old candidate07a7ee4fa097210a43e2ca9ce7f68b659a2cedb0after restacking on dependency3708969b731425b0c6d4b97920d1b5e6519bb013. Recipe-free acceptance PASSes with resolved taskreranking,AutoModelForSequenceClassification, and canonical paired BERT inputs. The delta remains reducibility-consistent with the charter.Bug fix explanation (dependency PR #1322)
winml eval --task reranking --samples 2selected an ID-only MS MARCO dataset whose rows lacked candidate passage text, soWinMLRerankingEvaluatorrejected them before scoring._DEFAULT_DATASETS['reranking'],_RERANKING_SCHEMA,RerankingDatasetMode,detect_reranking_dataset_mode, andWinMLRerankingEvaluatorgroup materialization. The default is pinned streaming SciDocs; groupedquery/positive/negativetext is detected and materialized with deterministic candidate IDs, positive retention, andmax_candidates=10.candidates_columnpaths remain supported; materialized candidate order remains unchanged and is not capped. The intentional changes are the default streaming SciDocs dataset and bounded grouped positive/negative text support.This generic bug fix belongs to dependency PR #1322. PR #1334's own diff remains exactly the two recipes listed above.
Analyze summary — component level and op level
ANALYZE-PARTIAL-SUCCESS(exit 1): OpenVINO CPU rule analysis completed, while CPUExecutionProvider has no runtime rule data. This is static compatibility analysis, not runtime execution.Component-level summary
Final optimization removes hierarchy metadata and renames nodes; semantic components use the preserved tagged export hierarchy or native ONNX scopes, while op/EP counts come from the final graph.
Op-level summary
CPUExecutionProvider/CPU has no rule data and therefore an empty classification; this is not a runtime support claim.
Reproduce commands