diff --git a/README.md b/README.md index 11f624c..2ffccb5 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ A research repo for the [WaveKat](https://github.com/wavekat) project — intera ``` wavekat-lab/ ├── tools/ -│ ├── audio-lab/ Real-time VAD + Turn Detection comparison app (Rust + React) +│ ├── audio-lab/ Real-time VAD + Turn Detection + ASR comparison app (Rust + React) │ └── cv-explorer/ Mozilla Common Voice dataset browser (Cloudflare Workers + React) ├── notebooks/ Jupyter notebooks (training, validation, dataset splits) └── docs/ Plans and design docs @@ -30,9 +30,9 @@ Each tool is self-contained — its own Makefile, lockfiles, and build setup liv ### [Audio Lab](tools/audio-lab/) — `tools/audio-lab/` -Web app for testing and comparing WaveKat library backends side by side in real time. Live mic capture, WAV upload, multi-config fan-out, VAD-gated pipeline mode, waveform + spectrogram + probability timelines. +Web app for testing and comparing WaveKat library backends side by side in real time. Live mic capture, WAV upload, multi-config fan-out, VAD-gated pipeline mode, live ASR transcripts, waveform + spectrogram + probability timelines. -Backends: webrtc-vad, silero-vad, ten-vad, firered-vad, pipecat smart-turn. [Details →](tools/audio-lab/README.md) +Backends: webrtc-vad, silero-vad, ten-vad, firered-vad, pipecat smart-turn, sherpa-onnx ASR. [Details →](tools/audio-lab/README.md) ### [Common Voice Explorer](tools/cv-explorer/) — `tools/cv-explorer/` diff --git a/tools/audio-lab/Makefile b/tools/audio-lab/Makefile index bc7098f..740784b 100644 --- a/tools/audio-lab/Makefile +++ b/tools/audio-lab/Makefile @@ -1,5 +1,5 @@ SHELL := /bin/bash -NVM := source $$NVM_DIR/nvm.sh && nvm use --silent && +NVM := source $$NVM_DIR/nvm.sh && .PHONY: help \ install install-backend install-frontend \ @@ -38,7 +38,7 @@ install-backend: cargo install cargo-watch install-frontend: - $(NVM) cd frontend && npm install + $(NVM) cd frontend && nvm use --silent && npm install # ─── Development ────────────────────────────────────────────────────────────── @@ -46,7 +46,7 @@ dev-backend: cargo watch -x 'run -p audio-lab' dev-frontend: - $(NVM) cd frontend && npm run dev + $(NVM) cd frontend && nvm use --silent && npm run dev dev: @echo "Run 'make dev-backend' and 'make dev-frontend' in separate terminals" diff --git a/tools/audio-lab/README.md b/tools/audio-lab/README.md index 1b92e5b..8df4f42 100644 --- a/tools/audio-lab/README.md +++ b/tools/audio-lab/README.md @@ -1,6 +1,6 @@ # Audio Lab -A web-based experimentation tool for testing and comparing [WaveKat](https://github.com/wavekat) library backends — VAD, turn detection, and more — side by side in real time. +A web-based experimentation tool for testing and comparing [WaveKat](https://github.com/wavekat) library backends — VAD, turn detection, ASR, and more — side by side in real time. > [!WARNING] > Early development. Things may change. @@ -10,6 +10,7 @@ A web-based experimentation tool for testing and comparing [WaveKat](https://git - **Live recording** — capture audio from your microphone server-side, stream results to the browser in real time - **File analysis** — upload a WAV file and run multiple configs against it at full speed - **Side-by-side comparison** — fan out audio to N configurations simultaneously and compare outputs +- **Live transcripts** — stream partial + final ASR transcripts per config as audio plays - **Preprocessing exploration** — apply high-pass filters, RNNoise denoising, or normalization per-config - **Interactive visualization** — waveform, spectrogram, and probability timelines with synchronized zoom, pan, and hover @@ -57,6 +58,17 @@ Each config can also enable per-config preprocessing: high-pass filter, RNNoise |---------|-------------|-------| | **pipecat** | Pipecat Smart Turn v3 — audio-based EOU detection | 16 kHz PCM audio | +### ASR + +Streaming speech-to-text via [`wavekat-asr`](https://github.com/wavekat/wavekat-asr). Each ASR config gets a per-config transcript card stacked under the timelines: committed finals with `[mm:ss.s–mm:ss.s]` timestamps, a dimmed trailing line for the live partial, and footer stats (last confidence, count, average segment duration). + +| Backend | Description | Preset (`preset` param) | +|---------|-------------|-------------------------| +| **sherpa-onnx** | Local streaming Zipformer / Paraformer via [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) | `bilingual` (default, ZH+EN) · `en` · `zh` · `paraformer-zh-en` | + +> [!NOTE] +> The first time you record or load a file with an ASR config enabled, sherpa-onnx downloads the chosen model from HuggingFace (~75 MB for `bilingual`) into `$HF_HOME` (default `~/.cache/huggingface/hub/`). The transcript card shows `loading model…` until the model is ready; subsequent runs are instant. + ## Architecture The Rust backend handles all audio capture and processing; the React frontend is embedded in the binary and handles visualization only.