Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)

[ssss141414]

Summary

This PR adds stock LayoutLM v1 document-question-answering support for impira/layoutlm-invoices, including metadata-derived export/input handling, a bounded document-QA evaluator with ANLS, and verified CPU recipes for fp32 and fp16. The committed scope is Effort L2 / Outcome L2, and tester validation at candidate 29f169eae33406bd4072a9d67d3a7d4b28985c53 reached the Goal L3 ceiling with L3 PASS and full planned tuple coverage. Support is intentionally limited to the stock LayoutLMForQuestionAnswering contiguous start/end span head; a custom non-consecutive-token classifier head is not implemented or claimed.

Model metadata

What the model does

This English LayoutLM v1 checkpoint scores extractive answers in invoices and other documents. A caller supplies OCR-derived token IDs, normalized token bounding boxes, an attention mask, and token-type IDs; the concrete Transformers class returns per-token start and end logits. Image/PDF decoding, OCR, token-box alignment, and answer decoding are outside this neural forward path.

• Evidence/confidence: the model card and config pinned at checkpoint revision fdf56765f8bdf236b2bd45d23347066475aa730a identify invoice/document question answering, DocVQA/SQuAD2.0 fine-tuning, model_type=layoutlm, and LayoutLMForQuestionAnswering; Transformers 5.14.1 exposes the four-input forward signature and QuestionAnsweringModelOutput (verified).

Primary user stories

• Supply an invoice question plus OCR words and normalized page boxes to extract a field such as the invoice number. The pinned model-card widget asks, "What is the invoice number?", and recommends the checkpoint for invoices and other documents (verified).
• Supply a document question plus OCR text/layout to extract a purchase amount or other document value. The pinned model-card widget asks, "What is the purchase amount?" (verified).

Supported tasks

• document-question-answering: declared by the checkpoint's Hugging Face metadata and model-card frontmatter (verified on the checkpoint surface).
• question-answering: Transformers selects LayoutLMForQuestionAnswering; WinML maps LayoutLM QA to the canonical exporter and emits start_logits / end_logits while the contribution supplies document-aware preprocessing and evaluation (verified on the Transformers and WinML surfaces).

Model architecture

LayoutLMForQuestionAnswering
|-- LayoutLMModel
|   |-- Embeddings: word + 1D position + x/y/h/w 2D position + token type
|   |-- Encoder layer x 12
|   |   |-- Self-attention (12 heads, hidden size 768)
|   |   |-- Feed-forward (768 -> 3072 -> 768, GELU)
|   |   `-- Residual + LayerNorm
|   `-- Pooler
`-- QA span head (768 -> 2)
    |-- start_logits
    `-- end_logits

• Source/confidence: pinned checkpoint dimensions (hidden_size=768, 12 layers, 12 heads, intermediate_size=3072, GELU) and Transformers 5.14.1 LayoutLM constructors (verified).

Validation and support evidence

1. Baseline

• Current-main baseline: 2b9ec0e9e57bba8003d25eaff85f8c7c9e30d75a, WinML 0.3.0.
• Refresh decision: FULL-RERUN. Main had moved from the previous validated baseline, and changes reached runtime/configuration/dependency, Eval, Analyze, and quality-gate surfaces, so the planner refreshed the complete baseline instead of reusing stale measurements.
• Optimum probe: vendor Optimum exposed LayoutLM for feature extraction, fill mask, text classification, and token classification. WinML additionally exposed canonical question answering; verdict WINML-ONLY for that task.
• Auto-config/build floor: automatic resolution selected AutoModelForNextSentencePrediction; Transformers rejected LayoutLMConfig. The build exited 2 after 26.502 s and produced no ONNX model. A diagnostic explicit-QA build also failed after 33.68 s during input generation/export with index out of range in self.
• Perf floor: BLOCKED-NO-ARTIFACT; neither baseline build produced a complete model to benchmark.
• Eval floor: document-question-answering was unsupported and exited 2. The existing question-answering schema exited 0 but required text-only SQuAD question/context/id/answers columns and was incompatible with OCR words, boxes, and images.
• Starting auto-config behavior: next-sentence-prediction task/class, sequence length 514, bbox high-exclusive bound 1, token-type high-exclusive bound 2, and a single logits output. The shipped recipes replace these values with the metadata-derived document-QA contract detailed below.

2. Goal

• Effort: L2.
• Goal ceiling: L3.
• Outcome: L2.
• Success definition: march L0 through L3; require both CPU fp32/fp16 build and structural validation, concrete perf, named-input PyTorch parity, and one bounded semantically verified document-QA functional smoke. A low ANLS value remains PASS when execution and semantics succeed.
• The current-main impact gate required FULL-RERUN; no downstream role changed the committed E/G/O tiers.

3. Outcome

• Shipped outcome: L2.
• Highest Goal verdict: L3 PASS.
• Coverage: full; no planned (EP, device, precision) tuple was deferred.
• Final semantic-binding candidate: 29f169eae33406bd4072a9d67d3a7d4b28985c53.
• Recipes: examples/recipes/impira_layoutlm-invoices/cpu/cpu/question-answering_fp32_config.json and examples/recipes/impira_layoutlm-invoices/cpu/cpu/question-answering_fp16_config.json.
• Product code covers task resolution, LayoutLM metadata-derived I/O, safe bbox/range generation, the QA runtime wrapper, document-QA schema/evaluation/ANLS, and registrations under src/winml/modelkit/.
• Regression coverage is under tests/unit/commands/, tests/unit/eval/, tests/unit/export/, and tests/unit/loader/.
• Model findings layoutlm-009 through layoutlm-014 record separate fp32/fp16 CPU build/perf/parity evidence, complete component mapping, operator/rule classification, fallback task and input-contract behavior, and the bounded strict-boundary ANLS 0.0 smoke. They remain in the separate skill knowledge lane, not this product PR.
• Methodology finding: none; the learner reported no new methodology trigger or Lane A methodology change.

4. Per-EP/device/precision results and Functional smoke Eval

Goal ladder

Tier	Scope	Verdict	Evidence
L0	CPU / fp32 and fp16	PASS	Both artifacts have the required four inputs, start/end outputs, opset 17, complete external data, and realized precision.
L1	CPU / fp32 and fp16	PASS	Concrete latency, throughput, and RAM measurements are retained below.
L2	CPU / fp32 and fp16	PASS	Named-input PyTorch parity passed for both output logits.
L3	FP32 CPU functional smoke	PASS	One real document-QA row completed at the final SHA with verified schema, target, prediction, accounting, strict ANLS semantics, and ANLS output.

L0 build and structure

EP / Device	Precision	Verdict	Build	Graph	Inputs / outputs	Realized storage
CPUExecutionProvider / cpu	fp32	PASS	90.16 s	394 nodes, opset 17	input_ids [1,512], bbox [1,512,4], attention_mask [1,512], token_type_ids [1,512] -> start_logits [1,512], end_logits [1,512]	206 FLOAT initializers; 508,803,072-byte external data
CPUExecutionProvider / cpu	fp16	PASS	99.39 s	396 nodes, opset 17	input_ids [1,512], bbox [1,512,4], attention_mask [1,512], token_type_ids [1,512] -> start_logits [1,512], end_logits [1,512]	206 FLOAT16 initializers; 254,401,536-byte external data

L0/L1/L2 and component/op Analyze were executed at be919c416b51949a77aacdb16725d6cec8f33360 and reused with verified no-impact provenance. The tester verified 95 source-integrity entries with zero mismatches and proved that the changes through final candidate 29f169eae33406bd4072a9d67d3a7d4b28985c53 did not touch graph, config, recipe, or runtime-producing paths. L3, strict ANLS boundary evidence, affected evaluator regressions, current-SHA CI partitions, Ruff, mypy, license, and the exact public pinned-rules Analyze sequence were rerun or revalidated for the final semantic binding.

L1 perf

EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM base	RAM load delta	RAM inference delta	RAM total delta	VRAM
CPUExecutionProvider / cpu	fp32	PASS	253.36 ms	253.27 ms	3.95 samples/s	968.3 MB	0.2 MB	114.9 MB	115.0 MB	N/A
CPUExecutionProvider / cpu	fp16	PASS	310.38 ms	309.22 ms	3.22 samples/s	990.8 MB	0.2 MB	125.2 MB	125.5 MB	N/A

L2 named-input parity

Precision	start logits cosine / max abs	end logits cosine / max abs	Verdict
fp32	0.9999999999999476 / 0.0000209808349609375	0.9999999999999456 / 0.00001621246337890625	PASS
fp16	0.9999999788805163 / 0.011110305786132812	0.9999999806844648 / 0.010725021362304688	PASS

Parity used named input_ids, bbox, attention_mask, and token_type_ids inputs for pinned DocVQA row 0, question ID 9951.

Functional smoke Eval

Verdict: PASS. This is one-row end-to-end operability evidence only; it is not representative accuracy and is not a benchmark-quality claim.

• Final candidate: 29f169eae33406bd4072a9d67d3a7d4b28985c53; FP32 on CPUExecutionProvider through ONNX Runtime.
• Dataset: pixparse/docvqa-single-page-questions, revision 33136ef456fa5a3fe68568d6e31dda4eeff95b9b, default config, train split, file data/train-00000-of-00036.parquet, row group 0, row 0, question ID 9951.
• Question/target/prediction: "When is the contract effective date?" / 7 - 1 - 99 / addition; prediction logit score -13.570242881774902.
• Metric: ANLS=0.0. The evaluator, document inputs, contiguous OCR-span prediction, and strict-boundary metric path operated end to end; the low value is model output quality on this single row, not a failed smoke.
• Strict ANLS rule: normalized edit distance < 0.5 retains similarity; distance >= 0.5 scores 0.0. Revalidated cases were distance 0.3333333333333333 -> direct similarity 0.6666666666666667 / aggregate 0.6667; distance 0.5 -> 0.0; and distance 0.75 -> 0.0.
• Row and OCR semantics: schema, label/target, and prediction semantics verified; 483 OCR words, 83 lines, aligned words/boxes, and a decodable 1000x1400 grayscale PNG.
• Input contract: sequence length 512; input_ids [1,512], bbox [1,512,4], attention_mask [1,512], token_type_ids [1,512].
• Fan-out caps: requested samples 1; available windows 2; max_windows=1; windows processed 1; doc_stride=128; top_k=1; max_answer_words=64.
• Accounting: requested 1, processed 1, n_samples=1, skipped 0, windows processed 1.
• The former blocker was the absence of a document-aware evaluator/schema: text-only SQuAD inputs could not consume nested OCR and boxes. This contribution adds strict OCR extraction/box normalization, subword box alignment, bounded contiguous-span decoding, ANLS, and explicit processed/skipped accounting.

5. Delta

The recipes were structurally compared with the frozen generated baseline after removing _note fields. They are changed, not identical. examples/recipes/README.md remains untouched.

Recipes	JSON pointer	Baseline	Candidate	Purpose
fp32, fp16	/loader/task	next-sentence-prediction	document-question-answering	Preserve the public document task while using canonical QA export behavior.
fp32, fp16	/loader/model_class	AutoModelForNextSentencePrediction	LayoutLMForQuestionAnswering	Select the stock contiguous span head from architecture metadata.
fp32, fp16	/export/input_tensors/*/shape/1	514	512	Derive usable sequence length from position metadata and padding offset.
fp32, fp16	/export/input_tensors/1/value_range/1	1	1001	Use the metadata-bounded high-exclusive normalized bbox range.
fp32, fp16	/export/input_tensors/3/value_range/1	2	1	Honor type_vocab_size=1; only token type 0 is legal.
fp32, fp16	/export/output_tensors	logits	start_logits, end_logits	Describe the stock contiguous QA outputs.
fp32, fp16	/export/compatibility/transformers_attention	eager	eager	Explicitly retain current-main compatibility behavior.
fp32, fp16	/eval	missing	pinned one-row document-QA contract	Own dataset revision, split, sample count, and all fan-out caps in the recipe.
fp16	/quant	null	fp16 conversion config	Declare and independently realize fp16 conversion in the fp16 recipe.

Recipe-free acceptance also passed: the generalized code path produced a four-input, two-output fp32 ONNX model without relying on a checkpoint-specific recipe override. The recipes remain exact evidence artifacts for the two verified CPU precision tuples.

Bug fix explanation: task resolution and task-boundary normalization

1. Symptom/trigger: recipe-free impira/layoutlm-invoices resolution selected next-sentence prediction and failed before export; an early broad QA suffix implementation also regressed four sibling task-resolution cases.
2. Root cause: Optimum does not register LayoutLM question answering, so its inference could not identify the concrete QA architecture. The initial fallback was allowed to outrank established explicit/current architecture, modality, and pipeline-tag decisions.
3. Changed symbols/mechanism: loader/resolution.py adds fallback-only ForQuestionAnswering suffix inference with architecture-suffix provenance and restores the existing string return contract; loader/task.py maps public document-question-answering to canonical question-answering only at loader/export boundaries. The final repair also narrows evaluator tokenizer typing without runtime behavior change.
4. General rule: inference is based on architecture suffix and registered task metadata, not impira/layoutlm-invoices or any checkpoint ID.
5. Compatibility/blast radius: explicit task/model-class overrides, supported Optimum QA siblings, pipeline-tag fallback, modality upgrades, image/text classification, image feature extraction, and existing LayoutLM non-QA registrations keep precedence. Only unresolved stock *ForQuestionAnswering heads use the fallback.
6. Regression evidence: 8 focused repair regressions, 59 affected resolver/evaluator tests, and 320 charter-compatibility tests passed; the final probe resolved question-answering / LayoutLMForQuestionAnswering / architecture-suffix; all final quality partitions passed.

Bug fix explanation: LayoutLM metadata-derived export and runtime inputs

1. Symptom/trigger: the explicit baseline QA build failed with index out of range in self; generated sequence length 514, token type 1, and unusable bbox bounds violated this checkpoint's embedding constraints.
2. Root cause: generic defaults did not account for LayoutLM's padding-offset position capacity, type_vocab_size=1, max_2d_position_embeddings, or concrete QA output contract.
3. Changed symbols/mechanism: models/hf/layoutlm.py derives usable sequence length, token-type range, bbox range, QA I/O, and concrete model-class mapping; the QA runtime wrapper forwards bbox only when the ONNX graph declares it; registration modules expose the mapping.
4. General rule: values come from normalized checkpoint configuration and concrete architecture metadata, never a model-ID condition.
5. Compatibility/blast radius: existing LayoutLM feature-extraction, fill-mask, text-classification, and token-classification registrations remain unchanged; text-only QA models do not receive an undeclared bbox; only the stock contiguous start/end span contract is supported.
6. Regression evidence: both L0 artifacts passed exact four-input/two-output structure, opset, external-data, and realized-precision checks; fp32/fp16 PyTorch parity passed; model/loader/dataset/export and charter compatibility suites passed.

Bug fix explanation: bbox and integer-range generation

1. Symptom/trigger: independent random bbox coordinates could produce reversed or zero-area boxes, while high-exclusive config bounds could be shifted incorrectly at inclusive random-generator call sites.
2. Root cause: generic integer tensor generation did not recognize bbox ordering/area invariants, and the boundary between high-exclusive schema ranges and inclusive APIs was not adapted exactly once.
3. Changed symbols/mechanism: core/model_input_generator.py and onnx/io.py generate ordered positive-area coordinates for declared rank-3 bbox tensors ending in dimension 4; commands/perf.py preserves high-exclusive semantics when calling inclusive generators.
4. General rule: detection is based on tensor declaration, rank, final dimension, and declared range, not LayoutLM or checkpoint identity.
5. Compatibility/blast radius: non-bbox rank-3/4 integer tensors retain prior behavior; generated boxes remain within declared bounds; export/config boundaries remain high-exclusive.
6. Regression evidence: exact bbox validity passed in L0/L2 inputs; the 320-test charter compatibility bundle covers bbox ordering/ranges and non-bbox siblings; full current-SHA model, command, analyze, optimization, and remaining partitions passed.

Bug fix explanation: document-QA evaluation and strict ANLS boundary

1. Symptom/trigger: document-question-answering had no WinML evaluator/schema, and the first ANLS implementation incorrectly awarded 0.5 at exact normalized edit distance 0.5 for prediction ab and reference a.
2. Root cause: the registry, document adapter, preprocessing, decoder, metric, and accounting contract existed only for text QA; within the new metric, the old similarity >= 0.5 condition made the canonical strict distance boundary inclusive.
3. Changed symbols/mechanism: the document evaluator flattens Pixparse OCR, normalizes polygons, aligns token boxes, caps windows, decodes contiguous spans, and reports accounting. normalized_levenshtein_similarity now retains similarity only for normalized edit distance < 0.5 (equivalently similarity > 0.5), while evaluator/schema/default-dataset registrations expose the first-class task.
4. General rule: behavior is driven by the public document-QA schema, nested OCR structure, model-declared inputs, configurable caps, and canonical ANLS distance semantics, not a checkpoint ID.
5. Compatibility/blast radius: text-only SQuAD evaluation remains unchanged when document mode is absent; malformed/misaligned OCR is rejected; fan-out and answer length remain bounded; custom non-consecutive-token heads remain out of scope. The intentional metric change affects only exact/above-boundary document-QA comparisons, which now score 0.0 canonically.
6. Regression evidence: test_anls_uses_strict_normalized_distance_threshold covers distance 0.3333333333333333 -> 0.6666666666666666, exact distance 0.5 -> 0.0, and distance 0.75 -> 0.0; tester boundary evidence passed the same three cases. The affected document-QA file passed 12 tests, the Eval partition passed 637 tests with one non-failing warning, and the final one-row smoke remained ANLS=0.0 at 29f169eae33406bd4072a9d67d3a7d4b28985c53.

6. Analyze summary - component level and op level

ANALYZE-PARTIAL-SUCCESS: the retained fp32 and fp16 central-rule analyses exited 1 because four emitted GPU rows had no rule data; both JSON payloads remained parseable and complete for all seven emitted rows. This is static rule-backed compatibility analysis, not runtime execution. The exact public r4 sequence independently verified pinned v0.3.0 rules acquisition and produced 12 complete result rows with exit 1 under the same partial-success semantics.

Component-level summary

Artifact	Architecture coverage	Mapping	Confidence	Actionable EP findings
fp32	document embeddings; 12x attention; 12x feed-forward; pruned pooler; QA span head	394 mapped, 0 unmapped	mapped; empty parent/pooler rows explicitly mapped-empty	None
fp16	document embeddings; 12x attention; 12x feed-forward; pruned pooler; QA span head plus output casts	396 mapped, 0 unmapped	mapped; empty parent/pooler rows explicitly mapped-empty	None

Op-level summary

Artifact	Graph	Dominant ops	Rule-backed EP roll-up
fp32	394 operators / 15 types	Reshape 122; Gemm 73; Transpose 48; Add 44	NvTensorRTRTX GPU, OpenVINO CPU/GPU/NPU, and QNN GPU/NPU fully supported; no partial/unsupported operator types
fp16	396 operators / 15 types	Reshape 122; Gemm 73; Transpose 48; Add 44	NvTensorRTRTX GPU, OpenVINO CPU/GPU/NPU, and QNN GPU/NPU fully supported; no partial/unsupported operator types

Rule-less GPU rows for CUDA, MIGraphX, TensorRT, and DML had no rule data; they are not runtime failures and are the reason the retained two-artifact Analyze status remains ANALYZE-PARTIAL-SUCCESS with exit code 1 for each artifact.

7. Reproduce commands

$ErrorActionPreference = 'Stop'

$REPO = (git rev-parse --show-toplevel).Trim()
if ($LASTEXITCODE -ne 0 -or -not $REPO) {
    throw 'Run this sequence from a winml-cli checkout.'
}

$OUT = Join-Path ([System.IO.Path]::GetTempPath()) 'winml-layoutlm-invoices-repro'
if (Test-Path $OUT) {
    throw "Output already exists: $OUT"
}

$recipe = Join-Path $REPO 'examples\recipes\impira_layoutlm-invoices\cpu\cpu\question-answering_fp32_config.json'
uv run winml build -c $recipe -m impira/layoutlm-invoices -o $OUT
if ($LASTEXITCODE -ne 0) {
    throw "winml build failed with exit code $LASTEXITCODE"
}

$TAG = 'v0.3.0'
$ASSET = 'rules-v0.3.0.zip'
$EXPECTED_RULES_SHA256 = '32030e3169a837b84798d6e321e07f418416ec3d3070d9af049385bf619a8b1d'
$RULES_ROOT = Join-Path ([System.IO.Path]::GetTempPath()) 'winml-cli-v0.3.0-rules'
$rulesArchive = Join-Path $RULES_ROOT $ASSET
$RULES = Join-Path $RULES_ROOT 'expanded'
$analyzeJson = Join-Path $OUT 'analyze.json'
$releaseUrl = "https://github.com/microsoft/winml-cli/releases/download/$TAG/$ASSET"
$hadPreviousRules = Test-Path Env:WINMLCLI_RULES_DIR
$previousRules = $env:WINMLCLI_RULES_DIR

if (Test-Path $RULES_ROOT) {
    throw "Rules root already exists: $RULES_ROOT"
}

try {
    New-Item -ItemType Directory -Path $RULES_ROOT -ErrorAction Stop | Out-Null
    Invoke-WebRequest -Uri $releaseUrl -OutFile $rulesArchive -ErrorAction Stop

    $rulesHash = (Get-FileHash -Path $rulesArchive -Algorithm SHA256 -ErrorAction Stop).Hash.ToLowerInvariant()
    if (-not [string]::Equals($rulesHash, $EXPECTED_RULES_SHA256, [System.StringComparison]::Ordinal)) {
        throw "Rules archive SHA256 mismatch: expected $EXPECTED_RULES_SHA256, got $rulesHash"
    }

    Expand-Archive -Path $rulesArchive -DestinationPath $RULES -ErrorAction Stop
    $parquetFiles = @(Get-ChildItem $RULES -Recurse -File -Filter '*.parquet' -ErrorAction Stop)
    if ($parquetFiles.Count -eq 0) {
        throw "No parquet rules found after extracting $releaseUrl"
    }
    $emptyParquetFiles = @($parquetFiles | Where-Object Length -eq 0)
    if ($emptyParquetFiles.Count -ne 0) {
        throw "Found $($emptyParquetFiles.Count) empty parquet rule files in $RULES"
    }

    $env:WINMLCLI_RULES_DIR = $RULES
    uv run winml analyze --model (Join-Path $OUT 'model.onnx') --ep all --device all --output $analyzeJson
    $analyzeExit = $LASTEXITCODE
    if ($analyzeExit -notin @(0, 1)) {
        throw "winml analyze failed with unexpected exit code $analyzeExit"
    }
    if (-not (Test-Path $analyzeJson -PathType Leaf)) {
        throw "winml analyze emitted no JSON: $analyzeJson"
    }

    $analysisRows = Get-Content $analyzeJson -Raw -ErrorAction Stop | ConvertFrom-Json -ErrorAction Stop
    $analysisRows = @($analysisRows)
    if ($analysisRows.Count -eq 0) {
        throw 'winml analyze emitted an empty JSON result.'
    }
    $expectedTargets = @(
        'NvTensorRTRTXExecutionProvider/GPU',
        'CUDAExecutionProvider/GPU',
        'MIGraphXExecutionProvider/GPU',
        'QNNExecutionProvider/NPU',
        'QNNExecutionProvider/GPU',
        'OpenVINOExecutionProvider/NPU',
        'OpenVINOExecutionProvider/GPU',
        'OpenVINOExecutionProvider/CPU',
        'TensorrtExecutionProvider/GPU',
        'DmlExecutionProvider/GPU',
        'CPUExecutionProvider/CPU',
        'VitisAIExecutionProvider/NPU'
    )
    $actualTargets = @()
    foreach ($analysisRow in $analysisRows) {
        if ($null -eq $analysisRow.metadata -or [int]($analysisRow.metadata.total_operators) -le 0 -or [int]($analysisRow.metadata.unique_operator_types) -le 0) {
            throw 'winml analyze JSON is missing nonzero operator metadata.'
        }
        if (@($analysisRow.results).Count -ne 1 -or $null -eq $analysisRow.results[0].classification) {
            throw 'winml analyze JSON has an incomplete result classification.'
        }
        $actualTargets += "$($analysisRow.results[0].ep_type)/$($analysisRow.results[0].device_type)"
    }
    $missingTargets = @($expectedTargets | Where-Object { $_ -notin $actualTargets })
    $unexpectedTargets = @($actualTargets | Where-Object { $_ -notin $expectedTargets })
    if ($missingTargets.Count -ne 0 -or $unexpectedTargets.Count -ne 0) {
        throw "winml analyze JSON target mismatch; missing=[$($missingTargets -join ', ')]; unexpected=[$($unexpectedTargets -join ', ')]"
    }

    $rulelessRows = @()
    foreach ($analysisRow in $analysisRows) {
        $classification = $analysisRow.results[0].classification
        if (@($classification.supported).Count -eq 0 -and
            @($classification.partial).Count -eq 0 -and
            @($classification.unsupported).Count -eq 0 -and
            @($classification.unknown).Count -eq 0) {
            $rulelessRows += $analysisRow
        }
    }
    if ($rulelessRows.Count -eq 0) {
        throw 'Expected at least one complete ruleless EP result for ANALYZE-PARTIAL-SUCCESS semantics.'
    }
    $analysisStatus = 'ANALYZE-PARTIAL-SUCCESS'

    [pscustomobject]@{
        RulesTag = $TAG
        RulesAsset = $ASSET
        RulesArchiveSha256 = $rulesHash
        RulesParquetCount = $parquetFiles.Count
        AnalyzeStatus = $analysisStatus
        AnalyzeExitCode = $analyzeExit
        AnalyzeResultCount = $analysisRows.Count
        RulelessResultCount = $rulelessRows.Count
        AnalyzeJson = $analyzeJson
    } | ConvertTo-Json
}
finally {
    if ($hadPreviousRules) {
        $env:WINMLCLI_RULES_DIR = $previousRules
    }
    else {
        Remove-Item Env:WINMLCLI_RULES_DIR -ErrorAction SilentlyContinue
    }
    Remove-Item $RULES -Recurse -Force -ErrorAction SilentlyContinue
    Remove-Item $rulesArchive -Force -ErrorAction SilentlyContinue
    Remove-Item $RULES_ROOT -Recurse -Force -ErrorAction SilentlyContinue
}

[ssss141414]

APPROVE — PR #1096 independent final review

Reviewed state

• Repository / PR: microsoft/winml-cli#1096.
• Exact reviewed PR head: 19177bc4e3d92eb11d3071d9cafe0eb1f85869d3.
• Independent checkout: C:\repo\winml-cli-1096-review-19177; local HEAD was rechecked and equals the exact PR head.
• Current-main / PR base SHA: 5deebd422e95f28fe8fd912ee50ce874710187b3.
• Live public main was rechecked on 2026-07-23 and still equals 5deebd422e95f28fe8fd912ee50ce874710187b3.
• Diff: exactly 12 files, 653 insertions, 27 deletions.
• Verdict: APPROVE. No blocking findings remain.

12-file engineering assessment

1. examples/recipes/impira_layoutlm-invoices/cpu/cpu/question-answering_fp16_config.json — correct nested CPU/cpu placement; sequence length 512; token_type_ids high-exclusive range [0,1); bbox high-exclusive range [0,1001); explicit fp16 quantization; backed by an exact fp16 artifact and runtime evidence.
2. examples/recipes/impira_layoutlm-invoices/cpu/cpu/question-answering_fp32_config.json — same validated LayoutLM input contract without an fp16 quantization claim; backed by the exact fp32 artifact and runtime evidence.
3. src/winml/modelkit/commands/perf.py — propagates configured input ranges into generated perf inputs; integer high-exclusive bounds are adapted exactly once to the legacy inclusive generator, while float and unspecified-range behavior remain preserved.
4. src/winml/modelkit/core/model_input_generator.py — shared integer bbox generation now rejects insufficient coordinate domains and emits ordered, positive-area boxes rather than four unrelated scalars.
5. src/winml/modelkit/loader/resolution.py — architecture-class routing is registry-driven, exact, and ambiguity-safe; it preserves non-QA LayoutLM routing and contains no checkpoint/model-ID branch.
6. src/winml/modelkit/models/hf/__init__.py — imports and aggregates the LayoutLM module so its registrations execute through the central registry.
7. src/winml/modelkit/models/hf/layoutlm.py — the per-architecture override is the correct abstraction; it registers LayoutLMForQuestionAnswering, derives sequence/token-type/bbox bounds from model metadata, and supplies valid PyTorch/NumPy bbox inputs. The implementation follows existing per-architecture registration patterns rather than hardcoding the contributed checkpoint.
8. src/winml/modelkit/onnx/io.py — explicit recipe InputTensorSpec bbox generation enforces the same ordered positive-area invariant as auto-config/default perf, closing the alternate input-generation surface.
9. tests/unit/commands/test_perf_cli.py — covers integer high-exclusive adaptation, float ranges, unspecified defaults, and structured bbox generation for default perf inputs.
10. tests/unit/export/test_onnx_config_overrides.py — covers metadata-derived LayoutLM sequence, token-type, and bbox contracts, ordered positive-area generation, and invalid coordinate domains.
11. tests/unit/loader/test_hf_model_class_mapping.py — covers class-wide LayoutLM registration/mapping without model-ID branching.
12. tests/unit/loader/test_resolve_task.py — covers exact, unambiguous architecture routing and preservation of alternate task behavior.

The full diff was read line-by-line. The design is generalized, minimally scoped for Outcome L1, and consistent across auto-config, explicit recipes, and perf. No production examples/recipes/README.md change, unsupported accelerator claim, skill-file leakage, dead temp reference, or unrelated tracked change is present.

Structured bbox invariants

Independent final-SHA runtime verification regenerated default inputs and ran CPU ONNX inference against exact fp32, exact fp16, and recipe-free fp32 artifacts.

For all three paths:

• bbox shape: [1,512,4].
• Observed coordinate range: 0..1000 under declared high-exclusive [0,1001) bounds.
• Minimum width: 1.
• Minimum height: 1.
• Valid boxes: 512/512.
• Contract: 0 <= x1 < x2 < 1001 and 0 <= y1 < y2 < 1001.
• Inference provider: CPUExecutionProvider.
• Outputs: start_logits and end_logits, each [1,512].
• Verdict: PASS for fp32, fp16, and recipe-free.

The tester's six-path repair validation additionally covers auto-config/Optimum, exact fp32 InputTensorSpec, exact fp16 InputTensorSpec, default perf fp32, default perf fp16, and default perf recipe-free. Every path produced ordered positive-area boxes and passed generated-input inference.

Independent local quality gates

All commands were actually executed from the isolated exact-SHA checkout with the repository environment:

• Ruff: PASS — All checks passed!.
• mypy: PASS — Success: no issues found in 409 source files.
• Analyze partition: PASS — 1,441 passed, 45 skipped.
• Models/loader/datasets/export partition: PASS — 1,360 passed, 6 skipped, 1 xfailed.
• Optim partition: PASS — 547 passed, 16 skipped, 1 xfailed.
• Commands/config/build/compiler/session/eval partition: PASS — 2,378 passed, 8 skipped, 2 deselected, 1 expected warning.
• Remaining ordinary partition: PASS — 873 passed, 1 skipped, 1 deselected.
• Total ordinary non-hardware tests: 6,599 passed across all five CI partitions.

The exact invocations were the current workflow-parity Ruff/mypy commands and each complete non-hardware pytest partition. The checkout retained no tracked modification; only ignored reviewer evidence was created.

Artifact and runtime rerun evidence

Artifact structure and precision:

• fp32: ONNX checker PASS; IR 8 / opset 17; named INT32 inputs [1,512], [1,512,4], [1,512], [1,512]; FLOAT outputs [1,512]; external data 508,803,072 bytes.
• fp16: ONNX checker PASS; 206 FLOAT16 initializers; external data 254,401,536 bytes, exactly 0.5x fp32.
• Final exact builds, recipe-free build/check/inference, and identical-input L2 validation passed. L2 cosine is approximately 1.0 for both precisions.
• Supplementary document-QA L3 remains honestly CLI-BLOCKED; it is above and outside the committed L2 ceiling.

Independent bounded semantic-input artifact perf reruns:

Artifact	Mean	p50	Throughput	RAM delta	Verdict
exact fp32	240.358 ms	236.686 ms	4.16 samples/s	+437.77 MB	PASS
exact fp16	312.089 ms	310.154 ms	3.20 samples/s	+469.73 MB	PASS

The slower fp16 CPU result is reported without a speedup claim; fp16's validated benefit here is the halved external weight data.

Exact-SHA GitHub checks

The live PR headRefOid was rechecked and remained 19177bc4e3d92eb11d3071d9cafe0eb1f85869d3. All nine visible checks on that exact SHA are COMPLETED/SUCCESS:

1. Analyze (Python) — COMPLETED / SUCCESS.
2. lint — COMPLETED / SUCCESS.
3. test (analyze) — COMPLETED / SUCCESS.
4. test (models) — COMPLETED / SUCCESS.
5. test (optim) — COMPLETED / SUCCESS.
6. test (commands) — COMPLETED / SUCCESS.
7. test (remaining) — COMPLETED / SUCCESS.
8. CodeQL — COMPLETED / SUCCESS.
9. license/cla — COMPLETED / SUCCESS.

No exact-SHA check is queued, in progress, cancelled, skipped, or failed.

Threads, comments, and reviews

The GitHub conversation gates were executed and then rechecked live after the final-SHA checks:

• GraphQL review threads: 0 total, 0 open; hasNextPage=false.
• REST line/review comments: 0.
• REST issue/conversation comments: 0.
• Submitted reviews: 0.

There is no unresolved external feedback to route.

PR body, draft, and label state

• Body hierarchy: complete and ordered as Summary -> Model metadata -> Validation and support evidence.
• Validation body includes Baseline, Goal, Outcome, per-EP/device/precision evidence, Delta, component/op Analyze summary, Reproduce commands, and quality gates.
• Body encoding: ASCII-readable; the prior mojibake is absent.
• Coverage: full for required CPU fp32 and fp16 tuples; no deferred tuples and no accelerator support claim.
• Draft state: true.
• Required label: model-scale-by-skill present.
• Production recipe index: unchanged.

Lane A methodology commit verification

• Exact Lane A commit: 35027f7a662b38a9761616dc5794d5229649ead3.
• Subject: fix(skill): validate structured bbox inputs.
• Parent: 1cfa2269af312acac9ef5383dafc590eb9799364.
• Repository remote: https://github.com/gim-home/ModelKitArtifacts.
• Remote-tracking verification: the exact commit is contained by origin/skill/pr1096-layoutlm-finding-v2.
• The PR body cites the same exact commit and describes the matching layoutlm-003, layoutlm-006, _meta-086, tester, and reviewer gate updates.
• No Lane A files are mixed into the WinML CLI Lane B diff.

Final disposition

APPROVE. The exact PR SHA is green locally and on GitHub, the 12-file change is engineering-sound, every applicable bbox-generation path now enforces the real relational input contract, exact fp32/fp16 runtime evidence passes, the PR remains a correctly labeled draft, all discussion counts are zero, and the Lane A methodology correction is present at the cited exact commit.

[ssss141414]

REQUEST_CHANGES - PR #1096 independent final review

Reviewed state

• Exact reviewed PR head: 37805dc35587d86bdd187dae5ab632aad03eda3d.
• Exact fetched origin/main / base: 2b9ec0e9e57bba8003d25eaff85f8c7c9e30d75a.
• Fresh detached checkout: C:\repo\winml-cli-pr1096-review-final; clean, fetched from refs/pull/1096/head, LFS smudge disabled.
• Scope: 24 changed files reviewed line-by-line; examples/recipes/README.md unchanged; no checkpoint-ID hardcode found in src/.
• Verdict: REQUEST_CHANGES. The implementation and evidence are otherwise strong, but the two actionable issues below block approval.

Actionable findings

1. Producer - ANLS incorrectly awards 0.5 at the canonical cutoff. src/winml/modelkit/eval/metrics/document_qa.py:27 returns the similarity when similarity >= 0.5. Canonical DocVQA ANLS awards similarity only when normalized Levenshtein distance is strictly below 0.5; an exact distance of 0.5 scores 0. Independent reproduction at this SHA with prediction ab and reference a produced normalized_distance=0.5, canonical score 0.0, implementation score 0.5. Change the cutoff to preserve the strict-distance rule and add an exact-boundary regression in tests/unit/eval/test_document_question_answering_evaluator.py (plus keep the existing below/above-threshold and multiple-reference coverage).

2. Explainer - the public Analyze reproduction command is not executable or portable. In PR body section 7. Reproduce commands, $env:WINMLCLI_RULES_DIR='<central-rules>' is an unresolved placeholder. Running that command literally against the verified fp32 artifact exited 2, produced no JSON, and reported No runtime rule parquet files were found, searching ...\<central-rules>. The reported rule-backed EP rows depend on external ModelKitArtifacts parquet rules, while the body gives neither an obtainable pinned rules source nor an honest prerequisite/non-public limitation. Replace the placeholder with complete portable acquisition/setup instructions for the exact rules snapshot, or clearly separate the non-public rules-backed analysis from a public command that can actually run and state what evidence it can reproduce.

Independent validation

• Environment: uv sync --locked --all-extras --all-groups passed with Python 3.11.9.
• Workflow gates: license passed with no diff; uv run ruff check src/ tests/ passed; uv run mypy -p winml.modelkit passed (438 source files).
• Exact non-hardware CI partitions all passed: analyze 1526; models/loader/datasets/export 1547; optim 714; commands/config/build/compiler/session/eval 3628; remaining 871. Total: 8286 passed, 0 failed (expected skips/xfails/deselection retained).
• Focused document/resolver/input/metric suite: 110 passed, 158 deselected.
• Fresh recipe-free winml config -m impira/layoutlm-invoices passed in 28.39 s and emitted one QA config with the expected four inputs, sequence 512, metadata bounds, span outputs, and eager attention. winml eval --schema --task document-question-answering --no-color passed.
• Exact recipe parity: fp32/fp16 are identical outside quant; fp32 has quant:null, fp16 has mode:fp16 and fp16_keep_io_types:true. Fresh auto-config graph equals the checked-in recipe graph; intentional recipe deltas are the public document-QA task and bounded Eval contract.
• Artifact reuse proof: be919c41..37805dc3 changes only resolver logic, a tokenizer type cast, and resolver tests; graph-producing paths and recipes have a zero-line diff. All sealed hashes matched.
• L0 structure: fp32/fp16 ONNX inputs are input_ids, bbox, attention_mask, token_type_ids with expected int32 shapes; outputs are start_logits/end_logits; 394/396 nodes; fp32 has 206 FLOAT initializers and 508,803,072-byte data, fp16 has 206 FLOAT16 initializers and 254,401,536-byte data.
• Independent L2 spot: named-input harness passed; fp32 start/end cosine 0.9999999999999476 / 0.9999999999999456; fp16 0.9999999788805163 / 0.9999999806844648; bbox valid.
• Independent L3 spot: pinned DocVQA question 9951, 483 aligned OCR words, 2 available/1 processed window, target 7 - 1 - 99, prediction addition, ANLS 0.0, requested/processed/skipped 1/1/0.
• Analyze/knowledge audit: HTP reports 127,792,898 parameters and 89/234 traced modules; optimized component mapping is 394/394 and 396/396 nodes; operator totals/dominant counts reconcile; knowledge findings layoutlm-009 through layoutlm-014 preserve tuple scope, low-metric interpretation, static-vs-runtime distinctions, and no-methodology-friction declaration.

GitHub gate

• PR remains open and draft with label model-scale-by-skill.
• Exact-head status rollup: 9/9 COMPLETED/SUCCESS (Analyze (Python), lint, five CI partitions, CodeQL, license/cla).
• Paginated discussion enumeration: 1 line comment, 1 issue comment, 1 review thread total, 0 unresolved; thread pagination complete (hasNextPage=false).
• Head and base were refetched and unchanged after checks and thread enumeration.

Owner routing: finding 1 -> producer (then tester reruns metric/focused/commands evidence and learner/explainer refresh downstream claims as needed); finding 2 -> explainer with tester-owned Analyze command/provenance retained.

[ssss141414]

Addressed both blocking items in commit 29f169eae33406bd4072a9d67d3a7d4b28985c53.

• ANLS now uses the canonical strict boundary: normalized edit distance < 0.5 retains similarity and distance >= 0.5 scores 0.0. Regression coverage verifies below-boundary 0.3333333333333333 -> 0.6666666666666666, exact-boundary 0.5 -> 0.0, and above-boundary 0.75 -> 0.0; the final one-row smoke remains ANLS=0.0.
• The canonical PR body now carries the tester r4 command block verbatim: pinned v0.3.0 rules acquisition, SHA-256 fail-closed verification against 32030e3169a837b84798d6e321e07f418416ec3d3070d9af049385bf619a8b1d before expansion, parquet validation, WINMLCLI_RULES_DIR setup, Analyze result validation, environment restoration, and cleanup. Tester r4 exact-block verification passed with 2,110 parquet files, ANALYZE-PARTIAL-SUCCESS / exit 1, 12 result rows, 6 ruleless rows, preserved JSON, restored environment, and removed transient rules/archive paths.

[ssss141414]

APPROVE - PR #1096 independent reviewer iteration 2

Reviewed state

• Repository / PR: microsoft/winml-cli#1096.
• Exact reviewed head: 29f169eae33406bd4072a9d67d3a7d4b28985c53.
• Exact fetched origin/main / base: 2b9ec0e9e57bba8003d25eaff85f8c7c9e30d75a.
• Fresh detached, LFS-disabled checkout: C:\repo\winml-cli-pr1096-review-r2; clean before and after review.
• Scope: 24 changed files, 1,360 insertions / 36 deletions; production recipe README unchanged; no checkpoint-ID branch, private path, unresolved placeholder, or unrelated scope found.
• Verdict: APPROVE. Both prior blocking findings are closed and no new blocking findings remain.

Prior findings closure

1. ANLS boundary - closed. The implementation now retains similarity only when normalized edit distance is strictly below 0.5. Independent exact cases passed: 0.3333333333333333 -> 0.6666666666666667, 0.5 -> 0.0, and 0.75 -> 0.0; aggregate ANLS was 0.6667, 0.0, and 0.0. The focused ANLS tests passed 4 with 8 deselected, and the complete document-QA evaluator file passed 12.
2. Portable Analyze command - closed. The live PR body PowerShell block is byte-equivalent after newline normalization to tester r4. Executed literally from a fresh redirected portable temp root, it downloaded pinned v0.3.0 rules-v0.3.0.zip, fail-closed verified SHA-256 32030e3169a837b84798d6e321e07f418416ec3d3070d9af049385bf619a8b1d, found 2,110 nonempty parquet files, accepted expected Analyze exit 1, and validated 12 result rows with 6 complete ruleless rows. WINMLCLI_RULES_DIR was restored, transient rules/archive paths were removed, JSON was preserved for validation, and the reviewer-created portable output was then cleaned.

Independent validation

• Environment: uv sync --locked --all-extras --all-groups passed; Python 3.11.9; WinML 0.3.0.
• Workflow gates: license passed with no diff; uv run ruff check src/ tests/ passed; uv run mypy -p winml.modelkit passed for 438 source files.
• Exact non-hardware CI partitions all passed: Analyze 1,526; Models/Loader/Datasets/Export 1,547; Optim 714; Commands/Config/Build/Compiler/Session/Eval 3,631; Remaining 871. Total 8,289 passed, 0 failed, with expected skips/xfails/deselection and two non-failing warnings.
• Focused resolver/model-I/O/perf-input/evaluator bundle: 295 passed.
• Fresh recipe-free winml config and document-QA schema both passed. The generated export graph equals the fp32 recipe; intentional recipe deltas are loader.task=document-question-answering plus bounded Eval, while fp16 declares quant.mode=fp16 and fp16_keep_io_types=true.
• Fresh public build structure: 394 nodes; int32 input_ids [1,512], bbox [1,512,4], attention_mask [1,512], token_type_ids [1,512]; float start_logits [1,512], end_logits [1,512]; 508,803,072-byte external data.
• Artifact integrity: all 95 + 50 + 20 = 165 sealed files matched byte counts and SHA-256. Charter r2, producer r3, tester r4, and prior reviewer verdict hashes matched their cross-handoff references.
• Goal/value fidelity: exact-main baseline is a verified FULL-RERUN; L0/L1/L2/L3 are PASS; Outcome L2, highest Goal L3 PASS, coverage full, deferred tuples []. Reused L0/L1/L2/Analyze evidence has verified no graph/config/recipe/runtime impact, while L3 and ANLS semantics are bound to the exact reviewed head.
• Engineering assessment: architecture-suffix inference remains fallback-only; LayoutLM sequence/token/bbox bounds derive from model metadata; shared range adaptation preserves high-exclusive semantics; bbox generation enforces positive area; document QA bounds windows and answer spans and reports explicit accounting; no regression or maintainability blocker found.
• Live PR body equals frozen pr-body-r2.md, preserves required hierarchy and source-owned values, and contains no contributor-local path, old candidate SHA, or unresolved placeholder.

GitHub gate

• PR is OPEN, remains DRAFT, and has label model-scale-by-skill.
• Exact-head checks: 9/9 COMPLETED/SUCCESS: Analyze (Python), lint, test (analyze), test (models), test (optim), test (commands), test (remaining), CodeQL, license/cla.
• Paginated discussion enumeration after checks: 3 issue comments, 1 line comment, 1 review thread, 0 unresolved; thread and nested-comment pagination complete.
• The prior REQUEST_CHANGES comment and explainer reply were read in full. The sole line thread is resolved and not outdated.
• Head and remote main were refetched after checks and discussion enumeration and remained exactly 29f169eae33406bd4072a9d67d3a7d4b28985c53 and 2b9ec0e9e57bba8003d25eaff85f8c7c9e30d75a.

No owner actions remain. This is a skill-level normal-comment approval only; it does not change GitHub review state or authorize moving the PR out of draft.

[Qiong Wu (qiowu) (DingmaomaoBJTU)]

load_native_hf_model() returns a plain Hugging Face module, which does not expose io_config. For --runtime pytorch, this therefore produces an empty set and every document-QA evaluation fails at the bbox check even though LayoutLMForQuestionAnswering.forward accepts bbox. Please fall back to inspecting the native model's forward signature when io_config is unavailable (the keypoint evaluator already uses this ONNX-config-or-forward-signature pattern).

[ssss141414]

Fixed in 4513fee267ad15d8244619d1ef308d302a057366. Native document-QA input discovery now falls back to forward signature inspection when io_config is absent, filters optional training/alternative arguments, and retains evaluator-provided bbox and fallback token_type_ids.

Independent final-SHA smoke with impira/layoutlm-invoices processed one real DocVQA row: prediction DORAL, ANLS 1.0; the native forward received input_ids, bbox, attention_mask, and token_type_ids. The focused evaluator suite and full CI partitions also passed.

[Qiong Wu (qiowu) (DingmaomaoBJTU)]

The assembled tensors need to be moved to self.config.pipeline_device before this call. With --runtime pytorch --device gpu, the native loader places the model on CUDA, but the tokenizer outputs, bbox, and fallback token_type_ids created above remain on CPU, so inference fails with a device-mismatch error. Moving each assembled tensor (or constructing the new tensors directly on the target device) preserves CPU/WinML behavior and enables GPU evaluation.

[ssss141414]

Fixed in 4513fee267ad15d8244619d1ef308d302a057366. All assembled inputs are moved to config.pipeline_device after bbox and fallback token_type_ids are created. The regression covers a non-CPU (meta) target, and the independent real CPU smoke observed all four forwarded tensors on CPU.

Validation: 6,154 ordinary CI tests passed; Ruff passed; mypy passed across 443 source files; sealed independent evidence reverified 53/53 files.