recipe(hubert): refresh Russian emotion CPU configs - #1333
Conversation
REQUEST_CHANGES - independent reviewer verdictPR: #1333 Blocking finding
Independently verified passes
Ownership routing
|
|
Repair response for issue comment #5381789642 The canonical PR body has been updated without changing code. It remains bound to candidate The repaired body no longer points to Tester successor root authority bindings under
The tester executed that exact public sequence from a new empty root for 2,112.807 seconds: 32 recorded commands, zero unaccepted exits, no prior measurement reuse, and 10/10 rows CLOSED: Prerequisites, L0 FP32, L0 FP16, L1 FP32, L1 FP16, L2, L3, Analyze FP32, Analyze FP16, and Quality. The refreshed body transcribes the successor metrics, DUSHA source/license statements, stack guards, Lane A binding, and GitHub state after publication: PR remains OPEN/DRAFT with |
APPROVE - successor independent reviewer verdictPR: #1333 Prior blocker closure
Candidate and evidence
GitHub gate
Terminal skill verdict: APPROVE. The prior blocker is closed and no actionable blocker remains. The PR remains draft. |
Summary
Refreshes the existing CPU FP32 and FP16 recipes for
xbgoose/hubert-large-speech-emotion-recognition-russian-dusha-finetuned, a five-class Russian speech-emotion classifier, for the current Transformers 5 export path. This is an L0 recipe-only outcome stacked on the audio-classification evaluator work in Draft PR #1329, which contains Draft PR #1326. The exact candidate reached Goal L3 PASS with full CPU FP32/FP16 coverage; its five-row DUSHA result is functional-smoke evidence only, not representative accuracy or benchmark quality.Model metadata
What the model does
Classifies Russian speech audio into exactly five checkpoint classes: neutral, angry, positive, sad, and other. The pinned processor normalizes mono waveform samples at 16 kHz; the exported graph consumes one FP32
[1,16000]window and emits FP32[1,5]logits.verified.Primary user stories
id2label. Confidence:verified.verified.Supported tasks
audio-classification: checkpoint, Transformers, and WinML surfaces. Evidence:HubertForSequenceClassification, WinMLHubertOnnxConfigregistration, and successful FP32/FP16 builds. Confidence:verified.feature-extraction: Transformers and WinML surfaces. Evidence: WinML-registered HuBERT task lookup. Confidence:mapped.automatic-speech-recognition: Transformers and WinML surfaces. Evidence: WinML-registered HuBERT task lookup. Confidence:mapped.Model architecture
verified).Validation and support evidence
1. Baseline
The frozen baseline is
microsoft/winml-climain commit0876e5ae1c98a169a6137e092e0d7b30bf9cee33, verified again as the currentorigin/mainbefore shipment. The locked environment used WinML CLI 0.3.0, Transformers 5.14.1, datasets 5.0.0, SoundFile 0.14.0, PyTorch 2.13.0, andonnxruntime-windowsml1.24.5.202604171637.Inspection passed for
audio-classificationwithAutoModelForAudioClassification,HubertOnnxConfig, opset 17, input[1,16000], andlogitsoutput. Native Optimum vendor lookup had no HuBERT task registrations; WinML registration supplied feature extraction, automatic speech recognition, and audio classification. On exact main,winml eval --schema --task audio-classificationexited 1 because audio classification was not yet supported without the dependency stack.Recipe-free FP32 build passed in 134.7 s (80.1 s export, 36.5 s optimize). Its 100-iteration CPU run after 10 warmups measured mean 138.84 ms, p50 136.49 ms, p90 146.535 ms, 7.2 samples/s, peak RSS 1782.46 MB, and inference RSS delta 90.53 MB. The historical FP16 recipe also built on current main in 198.4 s (78.6 s export, 79.9 s optimize, 37.2 s FP16 conversion), producing 773 nodes, 420 FLOAT16 initializers, FP32 input/output, and a 631,400,642-byte external-data file. Its CPU run measured mean 159.563 ms, p50 160.209 ms, p90 165.296 ms, 6.27 samples/s, and total peak RSS delta 43.51 MB. FP16 was slower than FP32 on this CPU baseline and is retained as required supported precision, not as a performance recommendation.
Historical PR #1211 originally added the two CPU recipes and merged at
2cb8b003e8af5b7354aa9118e6e399e41da6dea2; this follow-up does not rewrite or reuse that historical branch.2. Goal
No ceiling change or re-issued charter event occurred.
3. Outcome
Candidate
e2667d973ca9f55437f8a5604d3ee2b4b42ba8f3(treefef6e2bc539ef39360c737fab31c83d5cc92bc01) reachedL3 PASS. Coverage isfullfor the chartered CPU FP32 and FP16 tuples, with no deferred tuple or blocker. The shipped L0 paths are:examples/recipes/xbgoose_hubert-large-speech-emotion-recognition-russian-dusha-finetuned/cpu/cpu/audio-classification_fp32_config.jsonexamples/recipes/xbgoose_hubert-large-speech-emotion-recognition-russian-dusha-finetuned/cpu/cpu/audio-classification_fp16_config.jsonThe candidate is stacked directly on Draft PR #1329 exact head
8010667eb20d0eb08e916157a6cc5080c91d385d, which contains Draft PR #1326 exact head1bd919b583aa8ccd974e35fa6a1a132ac7e1c9f8. #1329 supplies strict scalar-string-to-checkpoint-ID evaluation needed by the authoritative DUSHA dataset. Its guards require a non-empty mapping, exact-string keys, integer checkpoint IDs, unique destinations, known and complete checkpoint-ID coverage, and rejection of unmapped observed values before inference. Both dependency heads had 9/9 successful checks and an exact-head structured APPROVE verdict at charter issuance.Lane A knowledge is separately bound to Draft PR gim-home/ModelKitArtifacts#254, exact commit
01021859c4bfbbc1f200cad2bde4259dbd1fa88b, with its check successful andmodel-scale-by-skillpresent. It addshubert-002throughhubert-005while preservinghubert-001; no skill or knowledge file is included in this product PR.No methodology friction observed. No new methodology finding was required because existing
_meta-111already governs the public-command closure exercised by this run.4. Per-EP/device/precision results and Functional smoke Eval
Both L0 builds preserved FP32
input_values [1,16000]and FP32logits [1,5]. FP32 produced 771 nodes and 420 FLOAT initializers. Explicit FP16 produced 773 nodes and 420 FLOAT16 initializers; its external-data size ratio to FP32 was 0.4999995945525307.L2 named-input PyTorch parity passed. For deterministic FP32
[1,16000]input_values, FP32 cosine was 1.0 with max absolute difference1.0132789611816406e-05and mean absolute difference6.186962127685547e-06; FP16 cosine was0.9999966025352478with max absolute difference0.01157236099243164and mean absolute difference0.006389713287353516. Reference, FP32, and FP16 prediction ID was 4.Functional smoke Eval: L3 PASS on candidate
e2667d973ca9f55437f8a5604d3ee2b4b42ba8f3, FP32 CPU, usingxbgoose/dusha@e3e309de7723f4dc4a0230b4b9d725061d561044, configdefault, splittest. Selection used the first authoritativecrowd_testrow per exact checkpoint class: rows 0, 1, 6, 7, and 20. Requested/selected/processed was 5/5/5 with 0 skipped or rejected; all 5/5 classes were represented. Schema, label semantics, and prediction semantics were verified.The exact checkpoint mapping was
neutral=0,angry=1,positive=2,sad=3,other=4; no synonym, translation, case-folding, positional, or inferred mapping was allowed. Each selected row used one forward over one 16,000-sample window, for five total forwards, with no multi-window fan-out. Long clips were truncated to the first 16,000 samples and short clips were right-padded.All five selected WAVs were byte-identical to exact official
crowd_test/wavs/archive members and had zero Podcast overlap. Source attribution is DUSHA Crowd fromxbgoose/dushaat the pinned revision, with the original archive athttps://cdn.chatwm.opensmodel.sberdevices.ru/dusha/crowd.tarand license/provenance authority athttps://github.com/salute-developers/golos/tree/5c5c5f87044803fcccdf7e149ef5384c95cff107. No audio was modified; no endorsement is implied; no additional restrictive terms are applied. The official terms permit the bounded local functional evaluation and scratch retention subject to retaining creator identification, copyright/license/disclaimer notices, and supplied URI or hyperlink.Accuracy was
0.2; represented-class macro-F1 was0.08. These are explicitly non-representative functional-smoke outputs only, proving end-to-end evaluator/model-path operability. They are not checkpoint-quality, population-accuracy, or benchmark-quality claims.5. Delta
Relative to parent
8010667eb20d0eb08e916157a6cc5080c91d385d, each shipped recipe adds exactly one JSON pointer:/export/compatibility/transformers_attentionnulleager/export/compatibility/transformers_attentionnulleagerThe FP32 recipe is otherwise identical to the planner's canonical generated baseline; FP16 is otherwise identical outside its retained recipe-owned
quant.mode=fp16block. Reducibility is consistent with the L0 charter. No source code changed, recipe-free acceptance was not required, andexamples/recipes/README.mdremains untouched.The user-visible symptom was that the historical PR #1211 recipes predated Transformers 5 and omitted the generated eager-attention compatibility override required by the current export path. The root cause was stale checkpoint-specific recipe metadata, while current auto-config normalized the effective configuration and injected
export.compatibility.transformers_attention=eager. The fix changes only that field in the two existing CPU recipes; it is generated compatibility metadata rather than checkpoint-conditional source logic. The blast radius is limited to this checkpoint's recipes and preserves task, loader, model type, opset 17, fixed FP32 I/O shapes, and FP16 weights with FP32 I/O. Exact FP32/FP16 recipe builds, graph verification, performance, parity, functional smoke, compatibility tests, Analyze, quality partitions, andgit diff --checkall passed.6. Analyze summary - component level and op level
ANALYZE-PARTIAL-SUCCESS: both standalone--ep all --device allruns emitted complete 12-row JSON. Exit code 1 is retained because some provider groups were unavailable or unruled. This is static rule analysis, not runtime execution evidence.Component-level summary
Optimization rewrote 252 FP32 and 254 FP16 nodes to root-level generated names; these remain explicitly unmapped rather than inferred.
Op-level summary
CPU, CUDA, DirectML, MIGraphX, TensorRT, and VitisAI groups were runtime-unavailable or unruled in this static analysis and carry no support claim here.
7. Reproduce commands
The complete tester-owned public runner is embedded below, so it is available directly from this PR body rather than an uncommitted local file. It self-materializes the exact candidate, locked environment, pinned model and five-row dataset, exact mapping and attribution, validation helpers, and official pinned rules archive. Its exact UTF-8 source is 24,099 bytes with SHA-256
f3b8bde064da98b6ab7f7ffef5950bfcef389f95a50ec61b81da2c18d35aca56.Save the following block as
public-reproducible-commands.ps1, then invoke it from a clean public environment with a caller-selected path that does not yet exist:The tester executed this exact 32-command sequence from a new empty root in 2,112.807 seconds with zero unaccepted exits. The fresh run closed all ten tester-owned rows:
Fresh successor outcomes: FP32 model SHA-256
62802015e81c64710829d681107aa56c350e3b2aa4388ea51c99e49022a707be; FP16 model SHA-2568de091be816b5913646745158cf4b709d5b298bd25ab5199f55f56e4c4ee75cb; FP16/FP32 external-data ratio0.4999995945525307. FP32 mean/p50/p90 was143.148/140.517/155.327 msat6.99 samples/s; FP16 was172.447/171.031/184.086 msat5.8 samples/s. L2 cosine was FP321.0and FP160.9999966025352478, with reference, FP32, and FP16 prediction ID4. L3 requested/selected/processed5/5/5, rejected0, represented all five classes, and reported accuracy0.2and macro-F10.08as functional-smoke evidence only. Both Analyze runs emitted metadata and all 12 rows; their expected partial-success exit code1was accepted.Quality evidence: license hook PASS; Ruff PASS; mypy found no issues in 439 source files; Analyze partition 1,526 passed/45 skipped; models partition 1,538 passed/6 skipped/2 xfailed; optimization partition 848 passed/16 skipped/1 xfailed; commands partition 3,648 passed/9 skipped; remaining partition 926 passed/2 skipped/1 deselected; candidate
git diff --checkPASS. The non-workflow diagnosticruff format --check src/ tests/reported 92 pre-existing files that would be reformatted, changed no files, and is not a current PR-triggered workflow gate.