fix(tts/luxtts): remove spurious mid-phrase pauses and chunk long text (#937) - #942
Open
Alex-Wengg wants to merge 2 commits into
Open
Alex-Wengg wants to merge 2 commits into
Alex-Wengg wants to merge 2 commits into
Conversation
#937) The reported pause is model-inherent, not a conversion or host bug: the PyTorch reference renders the same 101/138-character texts with the same gap at speed 1.0. It is also not a token-count threshold. Sweeping the reference across speeds 1.0–1.3 (i.e. across generated-frame counts) the pause appears and disappears non-monotonically (1.0 and 1.15 pause, 1.05, 1.1, 1.2, 1.3 clean), so each (length, seed) is effectively a fresh draw and longer text simply has more chances to land one. A 72-token continuation span reproduced a 290 ms pause, which rules out "≤ 102 target tokens is a stable regime" as the fix. Host-side handling in LuxTtsManager.synthesize: - Text that fits one pass (≤ 102 tokens, ≤ 555 generated frames, ≤ 1024 total) is synthesized exactly as before; the 97-character control is byte-identical to the pre-fix output. - Longer text is split into balanced spans at word/punctuation boundaries and continuation-prompted: each span's prompt is the previous span's untrimmed audio + tokens. Span size is bounded by the prompt's frames-per-token ratio so no span exceeds 468 generated frames (5 s), which keeps it untruncated as the next prompt. `speed` is applied to the first span only; later spans inherit the pace from their prompt (re-applying it compounded 0.8 → 0.64 → … and blew the 1024-frame graph on span 2). - Every pass is scanned for gaps ≥ 80 ms between runs of sustained speech (≥ 50 ms above −45 dB relative to peak). Gaps beyond the span's pause punctuation trigger a re-draw with the next seed, up to 3 times; the last two attempts also compress duration by 3 % / 6 % (surplus estimated frames are the other pause source), and the following span is prompted back at the original pace. The cleanest pass is kept, so text-driven breaks the model insists on are left alone rather than erroring. - Seams: the vocoder opens every pass with a ~20 ms click followed by 150–300 ms of silence. Anchoring onset/tail trimming and pause detection on sustained speech (not the first sample above threshold) brought the inter-span gap from ~190 ms down to ~80 ms; spans ending in punctuation keep their tail as the sentence break. Verification (M-series, gpu variant, seed 42, quick-brown-fox 5 s prompt): 101-char and 138-char texts render with no gap ≥ 80 ms at −45 dB and verbatim Parakeet transcripts; a 305-char two-sentence paragraph (4 spans) keeps only its comma/period pauses plus one plausible break at "downstream | toward" that survives all re-draws; speed 1.2 propagates without compounding (339/305/338/333 frames per span); speed 0.8 no longer errors. Each re-draw is one extra pass (~60–100 ms on GPU).
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 2m 7s • 2026-09-20T16:43:27.319Z |
Supertonic3 Smoke Test ✅
Runtime: 0m23s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 0m57s • 09/20/2026, 12:35 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
PocketTTS Smoke Test ✅
Runtime: 0m8s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 135.8s processing • Test runtime: 2m 21s • 09/20/2026, 12:48 PM EST |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 6m19s • 09/20/2026, 12:54 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 56.5s diarization time • Test runtime: 3m 5s • 09/20/2026, 12:44 PM EST |
…peline - Token bucket: fitsSinglePass and maxSpanTokens now leave room for the prompt transcript in the 256-token TextEncoder bucket (prompt + span + pad ≤ 256), so a long prompt transcript no longer throws inputTooLong on span 1. - Pathological prompt ratios are rejected in the chunking path instead of turning into dozens of passes: frames-per-token > 12 (transcript covering only part of the clip) or spans under 8 tokens throw inputTooLong with guidance. Single-pass behaviour is unchanged. - Re-draw budget is a parameter (`maxRedraws`, default 3; CLI `--redraws`), and the result reports `redraws` / `residualPauses`. LuxTtsE2ETests pins the raw pass with maxRedraws: 0 — the fixture text draws one breath beyond its two commas and would otherwise be re-drawn (verified: 838/432/220672 exact with 0, 825/419 without). - Duration compression on re-draws removed. On the 101-char text at seed 42 the ×1.03 pass removed the pause but squeezed the final word into the last frames (tail 120 ms only 3 dB under body vs 6–9 dB for clean passes; ASR read an extra word), and it did not fix the two persistent cases. Re-draws are pure re-seeds; the pace bookkeeping is gone. - Task.checkCancellation() before every pass. - Pause allowance: hyphen token added; the text API credits ellipses and quotes/brackets the G2P renders silently (`textPauseAllowance`). - Prompt clips longer than 5 s log a warning (audio is capped, the transcript is not, and the ratio now drives every span). - One sustained-speech scan per kept pass (vDSP peak / mean-square); leading and trailing trims are a single slice into the crossfade. - Resample failure on a continuation prompt is wrapped in LuxTtsError.inferenceFailed; single AudioConverter; maxPromptSamples constant; result-field docs describe the summed multi-span semantics; the "ratio the model produced" comment corrected (it is host-fixed). - The 102-token single-pass cap is kept, now measured: the 106-token issue text paused on 6 of 8 raw single-pass draws and the ladder still failed one of six seeds, while as two ~53-token spans all six seeds were clean with two re-draws total. Not changed: re-draws redo the text-encoder and prompt-mel stages (5–12 % of a pass); caching them needs a synthesizer split. The leading "word" ASR sometimes reports at 0.00–0.08 s (conf < 0.25) is the vocoder's opening click and appears on pre-PR renders too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #937.
The pause is model-inherent and stochastic, not a conversion or token-count bug: the upstream PyTorch reference renders the same texts with the same gap at speed 1.0, and sweeping speed 1.0–1.3 makes it come and go non-monotonically. Each (length, seed) is a fresh draw; longer text just has more chances to land one. Upstream hides it via a silent speed×1.3.
Host-side handling in
LuxTtsManager.synthesize:speedapplies to the first span only; later spans inherit pace. Implausible prompt ratios (> 12 frames/token, or spans < 8 tokens) throwinputTooLonginstead of rendering dozens of passes.maxRedraws(default 3, CLI--redraws, 0 pins the raw pass). Cleanest pass kept; never errors; the result reportsredraws/residualPauses. Cancellation is checked before every pass.Verification (gpu variant, quick-brown-fox 5 s prompt): reporter's 101- and 138-char texts render with no gap ≥ 80 ms at −45 dB and verbatim Parakeet transcripts across seeds 42–47; a 305-char paragraph keeps only comma/period pauses plus one text-driven break the ladder cannot remove; speed 1.2 propagates without compounding; speed 0.8 no longer errors; E2E fixture parity exact with
maxRedraws: 0. Each re-draw is one extra pass (~60–100 ms on GPU).Reviewer notes: second commit addresses the code-review findings (details in its message).
LuxTtsContinuationTestscovers chunking, token/frame budgets, single-pass fit, pause detection incl. the onset click, trims, and the text pause allowance.