Publish audio.transcribe-speech@1.0.0 (Whisper-tiny, greedy English ASR)

[enricopiovesan]

Summary

Publishes audio.transcribe-speech@1.0.0 -- a real Whisper-tiny (encoder-decoder transformer) speech-to-text forward pass, greedy English-forced decoding. Closes #473, the fifth and final model from this session's AI-WASM-model review (following #455/#460/#465/#469).

Why this capability

Natural complement to audio.detect-speech-segments (#460): an app that already gates on "is there speech in this clip" (e.g. Callweave) is a short step from "what did it say," and free-text transcription feeds directly into text.detect-entities (#465) and text.detect-language (#469) as a pipeline.

Architecture / decoding

80-channel log-mel spectrogram front end -> 2-conv1d/4-layer/384-dim/6-head transformer encoder -> 4-layer decoder (causal self-attention + cross-attention to the full 1500-frame encoder output) -> greedy (argmax) autoregressive decoding, seeded with a fixed <|startoftranscript|><|en|><|transcribe|><|notimestamps|> prompt. This is an officially-sanctioned minimal decoding path, not an invented shortcut -- HF's own generation_config.json ships exactly this forced-token configuration for openai/whisper-tiny. No KV-cache: each step recomputes the full decoder stack over the sequence so far (matching the non-cached decoder_model.onnx variant this was verified against), which is correctness-first and tractable at this model's scale.

Verification

• Built a from-scratch pure-numpy encoder+decoder implementation reading weights directly from the real ONNX graphs (Xenova/whisper-tiny) -- not just calling onnxruntime. This caught a real bug before it shipped: a forgotten v_proj.bias in the weight dict (silently defaulting to zero), producing a ~17-magnitude hidden-state divergence invisible without this independent cross-check.
• Verified end to end on real speech: openai/whisper's own tests/jfk.flac fixture (the famous JFK inaugural excerpt). The numpy reference matched onnxruntime token-for-token, exactly.
• Quantization (int8, including the tied 51865x384 embed_tokens table) tested empirically on the same real audio through the full greedy autoregressive loop, not assumed safe: matched the float32 baseline's transcript exactly except one comma token (arguably the more accurate rendering of the real quote) -- unlike audio.detect-speech-segments' recurrent Silero VAD (audio.detect-speech-segments@1.0.0 — Silero VAD, generic speech-risk scorer #460), where int8 measurably broke the LSTM's output, autoregressive token decisions here proved robust to per-weight quantization noise.
• Compiled wasm32-unknown-unknown --features full-model artifact run under wasmtime (CLI) on the same real audio (11s clip and a 1s excerpt): output matched the quantized Python reference exactly, byte for byte.
• capability-src/audio-transcribe-speech: 14 #[test] cases against a synthetic fixture; cargo llvm-cov functions=100%, lines=98.46%, regions=98.59%.

Shared-crate change: heap size 64 -> 192 MiB

Building the wasm artifact against real ~176,000-sample (11s) audio input surfaced a genuine bump-allocator OOM (exit(2)) at the prior 64 MiB shared heap ceiling. Root cause was not this capability's own ~41 MiB of audio/attention scratch buffers (which are allocated once and reused, never per-layer/step) -- it was the shared JSON parser's Vec<Value> being grown by repeated reallocation for a long input array (parser doesn't know the final length up front), with every abandoned smaller buffer from each doubling staying allocated forever in a bump allocator that never frees.

Fixed by raising capability-src/wasi-capability-runtime's shared HEAP_SIZE constant 64 -> 192 MiB -- same allocator, same "zero-cost headroom" rationale documented at the prior 16 -> 64 MiB raise (registry#384). Re-ran cargo test for text.detect-entities, audio.detect-speech-segments, text.detect-language, and report.translate-fr-semantic after the change to confirm no regression from this shared-crate edit -- all green.

License

Dual-attested, logged as-is rather than picking one: openai/whisper's own GitHub LICENSE file is MIT (verified directly); the openai/whisper-tiny Hugging Face model card's own metadata (cardData.license) says apache-2.0. Both primary sources, both permissive.

Governing Spec

• 001-registry-foundation
• 014-extraction-compatibility
• 018-capability-test-coverage

Project Item

https://github.com/orgs/traverse-framework/projects/3 -- #473

Definition of Done

• capability-src/audio-transcribe-speech/ #![no_std]; ABI-clean (imports exactly {fd_read, fd_write, proc_exit}); #[test] coverage functions=100%/lines&regions>=95%.
• capabilities/audio/audio.transcribe-speech/1.0.0/contract.json -- full I/O schema, ai.model_backed: true, spec-024 risk.determinism_class: model_derived, spec-023 authoring, provenance citing audio.transcribe-speech@1.0.0 — Whisper-tiny ASR, fifth/final model from the AI-WASM-model list #473.
• Data table generation script (scripts/model/prepare_whisper_tiny_int8.py); digest-pinned release asset (weight table too large to commit inline, unlike text.detect-language@1.0.0 — CLD2 language ID, FR-017 status TBD #469's small trigram table).
• Signed .wasm, pre_pr_check.sh green, human merge.

Validation

• bash scripts/ci/pre_pr_check.sh <this body> with BASE/HEAD wired as CI wires them.

🤖 Generated with Claude Code