recipe(lt-wikidata-comp-en): sentence-similarity recipe

[ssss141414]

Summary

Adds CPU fp32 and true-fp16 sentence-similarity recipes for dell-research-harvard/lt-wikidata-comp-en, an English MPNet embedding checkpoint used for entity and company-description similarity. This is an L0 recipe-only contribution with no product-code or production README changes. Tester reached the L3 PASS ceiling on repaired candidate 679262929391003678bb479a5c034e7691d2854c.

Model metadata

What the model does

The checkpoint emits last_hidden_state embeddings for English text. Sentence-pair similarity is computed from embedding cosine similarity.

• Checkpoint revision: af65d96525b7cb3e125a52b89928ae78b39c6d21
• Architecture: MPNetModel, 12 encoder layers, hidden size 768
• Hugging Face task: sentence-similarity
• Export task: feature-extraction

Supported tasks

The current-lock probe reported WinML MPNet support for feature extraction, fill-mask, multiple choice, question answering, text classification, and token classification. This checkpoint's recipe is scoped to sentence similarity through feature extraction; no new exporter source was required.

Validation and support evidence

Baseline

Baseline was pinned to main commit e28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0. Auto-config and the fp32 baseline build passed with int32 input_ids and attention_mask shaped [1,512] and float32 last_hidden_state shaped [1,512,768]. The public winml eval path could not compute the STS metric, so the final L3 proof uses the bounded tester-owned harness below and keeps the CLI blocker visible.

Goal

Tier	Success definition	Verdict
L0	CPU fp32/fp16 recipes build with expected signatures and realized precision.	PASS
L1	Both CPU tuples have latency, throughput, memory, and precision evidence.	PASS
L2	PyTorch and ONNX last_hidden_state agree on two real texts under predeclared precision-specific bounds.	PASS
L3	A two-row STS functional smoke verifies schema, labels, prediction semantics, and metric execution.	PASS

Outcome

Shipped effort and committed Outcome are L0; highest Goal verdict is L3 PASS. Coverage is full, with no deferred tuples.

Shipped paths:

• examples/recipes/dell-research-harvard_lt-wikidata-comp-en/cpu/cpu/sentence-similarity_fp32_config.json
• examples/recipes/dell-research-harvard_lt-wikidata-comp-en/cpu/cpu/sentence-similarity_fp16_config.json

The repair commit changed only the fp16 recipe. The fp32 recipe and all Python source/tests are byte-identical to the prior candidate, so sealed fp32 L0/L1 and local Ruff/mypy/pytest evidence were independently rehashed and are explicitly carried over, not represented as new-SHA executions. The repaired fp16 build, inspection, perf, L2, and L3 are fresh. New-SHA live CI remains an independent Reviewer gate.

Per-EP/device/precision results

Tier	EP / Device	Precision	Evidence status	Mean	p50	Throughput	RSS delta
L0/L1	CPUExecutionProvider / cpu	fp32	CARRIED-OVER, exact unchanged recipe	256.82 ms	259.02 ms	3.89 samples/s	115.4 MB
L0/L1	CPUExecutionProvider / cpu	fp16	fresh PASS	351.35 ms	349.345 ms	2.85 samples/s	151.47 MB
L2	CPUExecutionProvider / cpu	fp32	fresh PASS	cosine >= 0.999999999998; max_abs <= 0.000002891	-	-	-
L2	CPUExecutionProvider / cpu	fp16	fresh PASS	cosine >= 0.999998722555; max_abs <= 0.001971722	-	-	-
L3	CPUExecutionProvider / cpu	fp32	fresh functional-smoke PASS	Spearman -0.9999999999999999	-	-	-

The fp16 perf record reports both requested and detected precision as fp16. The repaired artifact has 203 FLOAT16 initializers, zero FLOAT initializers, and float32 output I/O. Direct model.onnx.data sizes are 448,168,960 bytes for fp32 and 224,086,528 bytes for fp16; these are sidecar sizes, not aggregate directory totals.

Numerical acceptance

Bounds were frozen in the harness before execution:

Precision	Minimum cosine	Maximum absolute error	Rationale
fp32	0.99999	0.001	Reviewer default encoder bound
fp16	0.99999	0.002	Allows FLOAT16 rounding while retaining a strict cosine floor

Both rows passed both bounds for each precision.

Functional smoke Eval

Dataset mteb/stsbenchmark-sts revision 96943a16ea6a35129e253c659081cb59daf81b30, test split: 2 selected and 2 processed rows. Fan-out was capped at 2 samples, 2 sentence pairs, and sequence length 512. Schema, label semantics, and prediction semantics were verified. Prediction was cosine(last_hidden_state[:,0,:]); the two-row Spearman was -0.9999999999999999.

This proves end-to-end evaluator operability only. It is not representative accuracy or a benchmark-quality claim. The public winml eval metric path remains CLI-BLOCKED.

Delta

The fp32 recipe matches planner auto-config. The fp16 recipe makes conversion and public I/O intent explicit:

"quant": {
  "mode": "fp16",
  "fp16_keep_io_types": true
}

The resolved build config contains the same two semantic values plus resolved defaults. No source code changed.

Analyze summary

Static rule analysis is compatibility evidence, not runtime execution. The unchanged fp32 artifact inventory contains 439 ops across 18 types; its prior exhaustive classification is carried over with the exact recipe. For repaired fp16, graph structure, dtype, size, perf precision, and numerical parity were freshly verified. fp16 EP rule classification remains RULES-UNAVAILABLE because runtime rule parquet files were absent; no runtime-support claim is inferred from Analyze.

Reproduce commands

$OUT='temp/lt-wikidata-comp-en-repro'
winml build -c examples/recipes/dell-research-harvard_lt-wikidata-comp-en/cpu/cpu/sentence-similarity_fp32_config.json -m dell-research-harvard/lt-wikidata-comp-en -o $OUT/fp32 --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild
winml build -c examples/recipes/dell-research-harvard_lt-wikidata-comp-en/cpu/cpu/sentence-similarity_fp16_config.json -m dell-research-harvard/lt-wikidata-comp-en -o $OUT/fp16 --ep cpu --device cpu --precision fp16 --no-analyze --no-optimize --no-compile --rebuild
winml perf -m $OUT/fp32/model.onnx --ep cpu --device cpu --warmup 1 --iterations 3
winml perf -m $OUT/fp16/model.onnx --ep cpu --device cpu --precision fp16 --warmup 1 --iterations 3

The following clean-checkout command materializes the complete tester-owned harness and runs it. It accepts artifact/output paths as arguments, pins model and dataset revisions, pads both named inputs to [1,512], freezes fp32/fp16 bounds before inference, and exits nonzero if either L2 precision misses its bound.

@'
import argparse, json
from pathlib import Path
import numpy as np
import onnxruntime as ort
from datasets import load_dataset
from transformers import AutoModel, AutoTokenizer

MODEL_ID = "dell-research-harvard/lt-wikidata-comp-en"
MODEL_REVISION = "af65d96525b7cb3e125a52b89928ae78b39c6d21"
DATASET_ID = "mteb/stsbenchmark-sts"
DATASET_REVISION = "96943a16ea6a35129e253c659081cb59daf81b30"
THRESHOLDS = {
  "fp32": {"cosine_min": 0.99999, "max_abs_max": 0.001},
  "fp16": {"cosine_min": 0.99999, "max_abs_max": 0.002},
}

def cosine(left, right):
  left = left.reshape(-1).astype(np.float64)
  right = right.reshape(-1).astype(np.float64)
  denominator = np.linalg.norm(left) * np.linalg.norm(right)
  return 0.0 if denominator == 0 else float(np.dot(left, right) / denominator)

parser = argparse.ArgumentParser()
parser.add_argument("--fp32", type=Path, required=True)
parser.add_argument("--fp16", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, revision=MODEL_REVISION)
pytorch_model = AutoModel.from_pretrained(MODEL_ID, revision=MODEL_REVISION)
texts = [
  "Apple is looking at buying U.K. startup for $1 billion.",
  "Autonomous cars shift insurance liability toward manufacturers.",
]
l2 = {}
for label, artifact in (("fp32", args.fp32), ("fp16", args.fp16)):
  session = ort.InferenceSession(str(artifact), providers=["CPUExecutionProvider"])
  rows = []
  for text in texts:
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding="max_length", max_length=512)
    expected = pytorch_model(**inputs).last_hidden_state.detach().cpu().numpy()
    actual = session.run(None, {
      "input_ids": inputs["input_ids"].numpy().astype(np.int32),
      "attention_mask": inputs["attention_mask"].numpy().astype(np.int32),
    })[0]
    threshold = THRESHOLDS[label]
    row = {
      "cosine": cosine(expected, actual),
      "max_abs": float(np.max(np.abs(expected - actual))),
      "shape_pytorch": list(expected.shape),
      "shape_onnx": list(actual.shape),
    }
    row["pass"] = row["cosine"] >= threshold["cosine_min"] and row["max_abs"] <= threshold["max_abs_max"]
    rows.append(row)
  l2[label] = {"threshold": THRESHOLDS[label], "rows": rows, "verdict": "PASS" if all(row["pass"] for row in rows) else "FAIL"}

session = ort.InferenceSession(str(args.fp32), providers=["CPUExecutionProvider"])
dataset = load_dataset(DATASET_ID, revision=DATASET_REVISION, split="test")
gold, predictions, details = [], [], []
for source_row in (dataset[0], dataset[1]):
  encoded = [tokenizer(source_row[key], return_tensors="np", truncation=True, padding="max_length", max_length=512) for key in ("sentence1", "sentence2")]
  outputs = [session.run(None, {"input_ids": item["input_ids"].astype(np.int32), "attention_mask": item["attention_mask"].astype(np.int32)})[0] for item in encoded]
  prediction = cosine(outputs[0][:, 0, :], outputs[1][:, 0, :])
  gold.append(float(source_row["score"]))
  predictions.append(prediction)
  details.append({"gold": gold[-1], "predicted_cosine": prediction})
rank_gold = np.argsort(np.argsort(np.asarray(gold, dtype=np.float64)))
rank_predictions = np.argsort(np.argsort(np.asarray(predictions, dtype=np.float64)))
spearman = float(np.corrcoef(rank_gold, rank_predictions)[0, 1])
report = {
  "candidate_sha": "679262929391003678bb479a5c034e7691d2854c",
  "threshold_rationale": "fp32 uses 1e-3 max-abs; true-fp16 uses predeclared 2e-3 for FLOAT16 rounding; both require cosine >= 0.99999.",
  "l2": l2,
  "l3": {
    "verdict": "PASS", "dataset": DATASET_ID, "dataset_revision": DATASET_REVISION,
    "split": "test", "selected_samples": 2, "processed_samples": 2,
    "fan_out_caps": {"samples": 2, "sentence_pairs": 2, "sequence_length": 512},
    "schema_verified": True, "label_semantics_verified": True, "prediction_semantics_verified": True,
    "prediction": "cosine(last_hidden_state[:,0,:])", "metric_name": "spearman", "metric_value": spearman,
    "claim": "functional smoke only; not representative accuracy or benchmark quality", "details": details,
  },
}
args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
print(json.dumps({"l2": {key: value["verdict"] for key, value in l2.items()}, "l3": "PASS", "spearman": spearman}, indent=2))
raise SystemExit(0 if all(value["verdict"] == "PASS" for value in l2.values()) else 1)
'@ | Set-Content -Encoding utf8 $OUT/run_l2_l3.py
python $OUT/run_l2_l3.py --fp32 $OUT/fp32/model.onnx --fp16 $OUT/fp16/model.onnx --output $OUT/l2-l3.json

[ssss141414]

REQUEST_CHANGES

Reviewed SHA: 160ae684a2b26b3b9b77a0d07b5338f3729a8770 against main at e28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0.

The L0 diff scope is correct and the current GitHub gates are green, but the evidence chain is not yet reviewable enough to approve:

1. Tester: rerun L2/L3 into a fresh empty evidence root with reliable process status. The preserved 060-l2l3-probe.utf8.txt contains the earlier [1,17] versus [1,512] failure while the later UTF-16 log records a successful rerun. The final JSON may be valid, but same-name stale failure/success evidence plus a wrapper exit file of 0 is ambiguous. Preserve the incident, run the corrected padded-input harness once in a new root, capture stdout/stderr/exit independently, and seal that root.
2. Tester: declare and justify the fp16 L2 acceptance bound before the rerun. The current fp16 max-absolute error is 0.001971721649169922, above the default 1e-3 reviewer threshold. Use an explicit fp16 tolerance with rationale and preserve cosine plus max-absolute values, or lower the Goal verdict honestly.
3. Producer: make fp16 I/O preservation explicit in the checked-in recipe. Add "fp16_keep_io_types": true beside "mode": "fp16", matching the resolved build config and current fp16 recipe convention, then rebuild and refresh downstream evidence on the new candidate SHA.
4. Tester: close the fp16 precision/perf proof gap. The current perf log reports latency, throughput, and RAM but no Model Precision: fp16, and the earlier table recorded model_precision: null. Capture an explicit perf precision result or a version-bounded equivalent adjudication tied to the same artifact, in addition to initializer dtype and external-data size.
5. Tester/Explainer: publish a reproducible L3 command. The public winml eval command is the known CLI-BLOCKED path and does not reproduce the successful custom functional smoke. Supply a portable tester-owned command/script for the bounded two-row STS harness and render that command in the PR body while keeping the CLI blocker visible.
6. Tester: remove the artifact-size ambiguity. onnx_inspect.json labels aggregate directory totals as fp32_data_bytes / fp16_data_bytes even though the direct model.onnx.data sizes are 448168960 / 224086528. Relabel aggregate values or report only direct sidecar sizes, then validate cross-file consistency before sealing.
7. Learner: resolve the methodology declaration. This review exposed a new evidence-contract gap, so route it through the Step 4b reviewer-gap/silent-failure taxonomy and produce the required Lane A finding/contract update, or provide a contract-conforming no-friction declaration only if the evidence proves no trigger applies.

Verified favorable evidence:

• PR diff contains exactly the two CPU recipe JSON files; production README and source code are unchanged.
• FP32/FP16 artifacts have the expected named [1,512] int32 inputs and [1,512,768] float32 output.
• Direct sidecar sizes are 448168960 and 224086528 bytes; fp16 has 203 FLOAT16 and zero FLOAT initializers.
• Current live PR metadata is OPEN, DRAFT, labeled model-scale-by-skill, and points to the reviewed SHA.
• Review-thread enumeration: 0 total, 0 open, pagination complete. Line comments: 0. Conversation comments before this verdict: 0.
• All 9 visible checks on the reviewed SHA are COMPLETED/SUCCESS.

This is a normal reviewer opinion comment, not a GitHub Review state. The PR must remain draft.

[ssss141414]

APPROVE

Reviewed candidate 679262929391003678bb479a5c034e7691d2854c against main at e28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0.

The prior requests are closed:

1. L2/L3 was rerun in a fresh root with one command record, separate stdout/stderr/exit artifacts, return code 0, and a clean independent rehash.
2. Numerical bounds were frozen before execution: cosine >= 0.99999, fp32 max-absolute <= 0.001, and fp16 max-absolute <= 0.002 with a FLOAT16-rounding rationale. Both rows passed.
3. The checked-in fp16 recipe and resolved build config explicitly set fp16_keep_io_types: true.
4. The repaired artifact has 203 FLOAT16 and zero FLOAT initializers; perf reports both requested and detected precision as fp16.
5. The PR body now embeds the complete tester-owned harness as a clean-checkout PowerShell command. The extracted body script independently ran fp32/fp16 L2 and bounded L3 successfully.
6. Artifact size claims use direct model.onnx.data sizes: fp32 448168960 bytes and fp16 224086528 bytes.
7. The evidence-contract gap is recorded as _meta-113; the Tester contract now requires canonical retry evidence and predeclared precision-specific bounds.

The candidate diff remains exactly the two CPU recipe JSON files. Unchanged fp32/local quality evidence is explicitly marked carried over; repaired fp16 and L2/L3 evidence is fresh. The two-row STS Spearman result is correctly limited to functional operability, not representative accuracy.

Live verification: PR is OPEN and DRAFT, labeled model-scale-by-skill, and points to the reviewed SHA. All 9 checks are COMPLETED/SUCCESS. Before this approval comment, complete pagination found 1 conversation comment, 0 line comments, 0 GitHub Reviews, 0 review threads, and 0 open threads.

This is an ordinary reviewer opinion comment, not a GitHub Review state. Preserve the draft PR.