From ee7c515611dd8ed9ce23f2ea9b0606ac76f0d3a4 Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Sat, 19 Sep 2026 21:47:04 -0400 Subject: [PATCH 1/2] fix(tts/luxtts): remove spurious mid-phrase pauses and chunk long text (#937) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reported pause is model-inherent, not a conversion or host bug: the PyTorch reference renders the same 101/138-character texts with the same gap at speed 1.0. It is also not a token-count threshold. Sweeping the reference across speeds 1.0–1.3 (i.e. across generated-frame counts) the pause appears and disappears non-monotonically (1.0 and 1.15 pause, 1.05, 1.1, 1.2, 1.3 clean), so each (length, seed) is effectively a fresh draw and longer text simply has more chances to land one. A 72-token continuation span reproduced a 290 ms pause, which rules out "≤ 102 target tokens is a stable regime" as the fix. Host-side handling in LuxTtsManager.synthesize: - Text that fits one pass (≤ 102 tokens, ≤ 555 generated frames, ≤ 1024 total) is synthesized exactly as before; the 97-character control is byte-identical to the pre-fix output. - Longer text is split into balanced spans at word/punctuation boundaries and continuation-prompted: each span's prompt is the previous span's untrimmed audio + tokens. Span size is bounded by the prompt's frames-per-token ratio so no span exceeds 468 generated frames (5 s), which keeps it untruncated as the next prompt. `speed` is applied to the first span only; later spans inherit the pace from their prompt (re-applying it compounded 0.8 → 0.64 → … and blew the 1024-frame graph on span 2). - Every pass is scanned for gaps ≥ 80 ms between runs of sustained speech (≥ 50 ms above −45 dB relative to peak). Gaps beyond the span's pause punctuation trigger a re-draw with the next seed, up to 3 times; the last two attempts also compress duration by 3 % / 6 % (surplus estimated frames are the other pause source), and the following span is prompted back at the original pace. The cleanest pass is kept, so text-driven breaks the model insists on are left alone rather than erroring. - Seams: the vocoder opens every pass with a ~20 ms click followed by 150–300 ms of silence. Anchoring onset/tail trimming and pause detection on sustained speech (not the first sample above threshold) brought the inter-span gap from ~190 ms down to ~80 ms; spans ending in punctuation keep their tail as the sentence break. Verification (M-series, gpu variant, seed 42, quick-brown-fox 5 s prompt): 101-char and 138-char texts render with no gap ≥ 80 ms at −45 dB and verbatim Parakeet transcripts; a 305-char two-sentence paragraph (4 spans) keeps only its comma/period pauses plus one plausible break at "downstream | toward" that survives all re-draws; speed 1.2 propagates without compounding (339/305/338/333 frames per span); speed 0.8 no longer errors. Each re-draw is one extra pass (~60–100 ms on GPU). --- Documentation/TTS/LuxTts.md | 32 ++- .../TTS/LuxTts/LuxTtsConstants.swift | 36 +++ .../TTS/LuxTts/LuxTtsContinuation.swift | 229 ++++++++++++++++++ .../FluidAudio/TTS/LuxTts/LuxTtsManager.swift | 148 ++++++++++- .../TTS/LuxTts/LuxTtsSynthesizer.swift | 4 +- .../TTS/LuxTts/LuxTtsContinuationTests.swift | 190 +++++++++++++++ 6 files changed, 626 insertions(+), 13 deletions(-) create mode 100644 Sources/FluidAudio/TTS/LuxTts/LuxTtsContinuation.swift create mode 100644 Tests/FluidAudioTests/TTS/LuxTts/LuxTtsContinuationTests.swift diff --git a/Documentation/TTS/LuxTts.md b/Documentation/TTS/LuxTts.md index 6832868f..99ed232a 100644 --- a/Documentation/TTS/LuxTts.md +++ b/Documentation/TTS/LuxTts.md @@ -71,8 +71,31 @@ The ANE path is ~0.5 dB softer, not bit-identical, but the Parakeet round-trip transcript matches the input text exactly. The much smaller jetsam-visible footprint is why iOS uses it. -All shapes are fixed: ≤ 255 tokens (+1 pad slot), ≤ 1024 mel frames -total, ≤ 555 generated frames (~5.9 s per call; chunking is phase 2). +All graph shapes are fixed: ≤ 255 tokens (+1 pad slot), ≤ 1024 mel frames +total, ≤ 555 generated frames (~5.9 s) per flow-matching pass. + +Longer text is handled by `LuxTtsManager` with continuation prompting +rather than a bigger graph: target tokens are split into balanced spans at +word/punctuation boundaries, each span is generated with the previous +span's audio and tokens as its prompt, the vocoder's onset/tail padding +is trimmed around sustained speech (a span ending in punctuation keeps its +tail), and spans are joined with a 30 ms crossfade. Spans are capped at +102 target tokens and bounded so no span generates more than ~5 s (468 +frames), which keeps every span short enough to prompt the next one +untruncated. `speed` is applied to the first span only; later spans +inherit the rate from their prompt. Text that fits one pass is synthesized +exactly as before. + +The model also drops the occasional spurious mid-phrase pause (issue #937): +where it lands depends on the exact length/noise draw — the PyTorch +reference does it too, and a small `speed` change moves or removes it — +so longer text simply has more chances to draw one. Every pass is scanned +for gaps ≥ 80 ms between runs of sustained speech (≤ −45 dB relative to +peak; natural stop closures stay ≤ 60 ms); when they outnumber the span's +pause punctuation the span is re-drawn with the next seed, up to 3 times +(the last two also 3 % / 6 % shorter, since surplus estimated frames are the +other source of pauses; the following span is prompted back at the original +pace), keeping the cleanest pass. ## Quick Start @@ -204,8 +227,9 @@ corpus-level gate is scored with `luxtts-g2p-dump` + `validate.py score` ## Remaining TODOs -- Long-input chunking across multiple vocoder windows (> 555 generated - frames currently errors; mel truncation is not allowed). +- Continuation spans are bounded by the *first* prompt's frames-per-token + ratio; a prompt whose pace differs a lot from the model's own pace can + still push a later span past the 555-frame vocoder bucket and error. - Optional VAD-based automatic prompt-silence trimming. - Non-English text (the G2P is `en-us` only; Mandarin pinyin tokens exist in `tokens.txt` but have no frontend). diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift index 6a7aa666..c800cad4 100644 --- a/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift @@ -54,4 +54,40 @@ public enum LuxTtsConstants { /// Default synthesis noise seed (matches the Python reference scripts). public static let defaultSeed: UInt64 = 42 + + /// Largest target-token span sent through one flow-matching pass. Spans + /// this size keep the per-pass odds of a spurious mid-phrase pause low + /// (issue #937) and fit the frame budget for typical prompts; longer text + /// is continuation-prompted so callers never see chunk seams. + public static let maxSinglePassTextTokens = 102 + + /// Re-seed attempts for a pass whose mid-speech silences outnumber the + /// span's pause punctuation. The model drops such pauses stochastically + /// (position depends on the exact length/noise draw; the PyTorch + /// reference does the same), so a fresh seed is the fix. One full pass + /// each. + public static let spuriousPauseRetries = 3 + /// Duration compression applied on each re-seed (indexed by attempt, + /// attempt 0 is the original pass). A surplus of estimated frames is the + /// other way the model ends up with a pause to fill, so later attempts + /// also shorten the span a little. The next span is prompted at the + /// original pace, so the nudge does not propagate. + public static let spuriousPauseRetrySpeedFactors: [Float] = [1.0, 1.0, 1.03, 1.06] + /// Silence floor (dB relative to the pass's peak) below which audio + /// counts as padding or pause, and the minimum gap between sustained + /// speech that counts as a pause. Natural stop closures stay ≤ 60 ms at + /// this floor; the reported pauses measure 100–160 ms. + public static let pauseFloorDb: Float = -45 + public static let pauseMinimumSeconds = 0.08 + /// Seed stride between continuation spans; leaves room for re-seeds. + public static let continuationSeedStride: UInt64 = 64 + + /// Generated-frame budget per continuation span. A span must fit inside + /// `maxPromptSeconds` so its untruncated audio can prompt the next span + /// with a transcript that still matches it. + public static let continuationSpanFrameBudget = + Int(maxPromptSeconds * Double(melSampleRate)) / hopLength + + /// Overlap used when joining continuation-prompted spans. + public static let continuationCrossfadeSeconds = 0.03 } diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsContinuation.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsContinuation.swift new file mode 100644 index 00000000..a2f4d706 --- /dev/null +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsContinuation.swift @@ -0,0 +1,229 @@ +import Foundation + +/// Host-side long-utterance helpers for LuxTTS. +enum LuxTtsContinuation { + + private static let analysisWindowsPerSecond = 100 + /// Activity shorter than this is a click or breath, not speech. Every + /// vocoder pass opens with a ~20 ms transient followed by 150–300 ms of + /// silence; anchoring on sustained speech keeps both trim and pause + /// detection from latching onto it. + private static let sustainedSpeechSeconds = 0.05 + private static let onsetPrerollSeconds = 0.03 + private static let tailPostrollSeconds = 0.03 + + /// Whether the whole text can go through one pass unchanged: within the + /// span cap, the 1024-frame graph, and the largest vocoder bucket. + static func fitsSinglePass( + textTokenCount: Int, promptFrames: Int, promptTokenCount: Int, speed: Double + ) -> Bool { + guard textTokenCount <= LuxTtsConstants.maxSinglePassTextTokens else { return false } + guard promptFrames > 0, promptTokenCount > 0, speed > 0 else { return true } + let featuresLength = LuxTtsSolver.featuresLength( + promptFrames: promptFrames, + promptTokenCount: promptTokenCount, + textTokenCount: textTokenCount, + speed: speed) + return featuresLength <= LuxTtsConstants.maxFrames + && featuresLength - promptFrames <= (LuxTtsConstants.vocoderBuckets.max() ?? 0) + } + + /// Largest span (in target tokens) that both stays inside the model's + /// stable regime and generates at most `continuationSpanFrameBudget` + /// frames for this prompt's frames-per-token ratio at `speed`. + static func maxSpanTokens(promptFrames: Int, promptTokenCount: Int, speed: Double) -> Int { + let cap = LuxTtsConstants.maxSinglePassTextTokens + guard promptFrames > 0, promptTokenCount > 0, speed > 0 else { return cap } + let framesPerToken = Double(promptFrames) / Double(promptTokenCount) + let budgetTokens = Int( + (Double(LuxTtsConstants.continuationSpanFrameBudget) * speed / framesPerToken) + .rounded(.down)) + return max(1, min(cap, budgetTokens)) + } + + /// Split a token sequence into balanced spans, preferring word/pause + /// boundaries nearest each ideal split point. + static func chunks( + tokenIds: [Int], maxTokens: Int, boundaryTokenIds: Set + ) -> [[Int]] { + precondition(maxTokens > 0, "maxTokens must be positive") + guard tokenIds.count > maxTokens else { return tokenIds.isEmpty ? [] : [tokenIds] } + + let chunkCount = (tokenIds.count + maxTokens - 1) / maxTokens + var chunks: [[Int]] = [] + chunks.reserveCapacity(chunkCount) + + var start = 0 + for chunkIndex in 0..<(chunkCount - 1) { + let remainingChunks = chunkCount - chunkIndex + let remainingTokens = tokenIds.count - start + let idealLength = Int( + (Double(remainingTokens) / Double(remainingChunks)).rounded()) + let idealEnd = start + idealLength + let minimumEnd = max( + start + 1, tokenIds.count - (remainingChunks - 1) * maxTokens) + let maxEnd = min(start + maxTokens, tokenIds.count - (remainingChunks - 1)) + let end = nearestBoundaryEnd( + in: tokenIds, + minimumEnd: minimumEnd, + idealEnd: idealEnd, + maxEnd: maxEnd, + searchRadius: max(1, maxTokens / 4), + boundaryTokenIds: boundaryTokenIds) + + chunks.append(Array(tokenIds[start.., boundaryTokenIds: Set + ) -> Int { + var end = tokenIds.count + while end > 0, boundaryTokenIds.contains(tokenIds[end - 1]) { end -= 1 } + return tokenIds[.., boundaryTokenIds: Set + ) -> Bool { + let spaceTokenIds = boundaryTokenIds.subtracting(pauseTokenIds) + guard let last = tokenIds.last(where: { !spaceTokenIds.contains($0) }) else { return false } + return pauseTokenIds.contains(last) + } + + /// Window ranges (in `windowSize`-sample windows) of sustained speech: + /// runs of at least `sustainedSpeechSeconds` above `pauseFloorDb` + /// relative to the clip's peak. + static func speechRuns(_ samples: [Float], sampleRate: Int) -> (runs: [Range], windowSize: Int) { + guard sampleRate > 0, !samples.isEmpty else { return ([], 1) } + let windowSize = max(1, sampleRate / analysisWindowsPerSecond) + let windowCount = samples.count / windowSize + guard windowCount > 0 else { return ([], windowSize) } + + var peak: Float = 0 + for sample in samples { peak = max(peak, abs(sample)) } + guard peak > 0 else { return ([], windowSize) } + let floorMeanSquare = peak * peak * powf(10, LuxTtsConstants.pauseFloorDb / 10) + let minimumWindows = max( + 1, Int((sustainedSpeechSeconds * Double(sampleRate) / Double(windowSize)).rounded(.up))) + + var runs: [Range] = [] + var runStart: Int? + for window in 0...windowCount { + var active = false + if window < windowCount { + var squareSum: Float = 0 + let start = window * windowSize + for sample in samples[start..<(start + windowSize)] { + squareSum += sample * sample + } + active = squareSum / Float(windowSize) > floorMeanSquare + } + if active { + if runStart == nil { runStart = window } + } else if let start = runStart { + if window - start >= minimumWindows { runs.append(start.. Int { + let (runs, windowSize) = speechRuns(samples, sampleRate: sampleRate) + guard runs.count > 1 else { return 0 } + let minimumWindows = max( + 1, + Int((LuxTtsConstants.pauseMinimumSeconds * Double(sampleRate) / Double(windowSize)).rounded(.up))) + return zip(runs, runs.dropFirst()).reduce(0) { count, pair in + count + (pair.1.lowerBound - pair.0.upperBound >= minimumWindows ? 1 : 0) + } + } + + /// Remove the vocoder's onset padding from a continuation span, keeping a + /// short preroll so unvoiced consonants are not clipped. + static func trimmingLeadingPadding(_ samples: [Float], sampleRate: Int) -> [Float] { + let (runs, windowSize) = speechRuns(samples, sampleRate: sampleRate) + guard let onset = runs.first?.lowerBound else { return samples } + let preroll = Int(onsetPrerollSeconds * Double(sampleRate)) + let trimStart = max(0, onset * windowSize - preroll) + return trimStart == 0 ? samples : Array(samples[trimStart...]) + } + + /// Remove trailing padding from a span that another span will follow, + /// keeping a short postroll. Spans that end in pause punctuation keep + /// their tail: that silence is the sentence break the text asked for. + static func trimmingTrailingPadding(_ samples: [Float], sampleRate: Int) -> [Float] { + let (runs, windowSize) = speechRuns(samples, sampleRate: sampleRate) + guard let end = runs.last?.upperBound else { return samples } + let postroll = Int(tailPostrollSeconds * Double(sampleRate)) + let trimEnd = min(samples.count, end * windowSize + postroll) + return trimEnd == samples.count ? samples : Array(samples[.. 0 else { + output.append(contentsOf: next) + return + } + + let outputStart = output.count - overlap + if overlap == 1 { + output[outputStart] = (output[outputStart] + next[0]) * 0.5 + } else { + for index in 0.. + ) -> Int { + var bestEnd: Int? + var bestDistance = Int.max + let searchStart = max(minimumEnd, idealEnd - searchRadius) + let searchEnd = min(maxEnd, idealEnd + searchRadius) + guard searchStart <= searchEnd else { + return min(max(idealEnd, minimumEnd), maxEnd) + } + + for end in searchStart...searchEnd { + guard boundaryTokenIds.contains(tokenIds[end - 1]) else { continue } + let distance = abs(end - idealEnd) + if distance < bestDistance { + bestEnd = end + bestDistance = distance + } + } + return bestEnd ?? min(max(idealEnd, minimumEnd), maxEnd) + } +} diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsManager.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsManager.swift index 786d53bc..dea3c07a 100644 --- a/Sources/FluidAudio/TTS/LuxTts/LuxTtsManager.swift +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsManager.swift @@ -161,7 +161,7 @@ public actor LuxTtsManager { speed: Float = LuxTtsConstants.defaultSpeed, seed: UInt64 = LuxTtsConstants.defaultSeed ) async throws -> LuxTtsSynthesisResult { - guard let synthesizer = synthesizer else { throw LuxTtsError.notInitialized } + guard let store, let synthesizer else { throw LuxTtsError.notInitialized } let prompt24k: [Float] do { @@ -173,12 +173,146 @@ public actor LuxTtsManager { "cannot load \(promptAudio.path): \(error.localizedDescription)") } - return try await synthesizer.synthesize( - promptTokenIds: promptTokenIds, - textTokenIds: tokenIds, - promptAudio24k: prompt24k, - speed: speed, - seed: seed) + let tokenizer = try await store.tokenizer() + let pauseTokens = Set([",", ".", ";", ":", "!", "?"].compactMap { tokenizer.tokenToId[$0] }) + let boundaryTokens = pauseTokens.union([tokenizer.tokenToId[" "]].compactMap { $0 }) + let maxPromptSamples = Int( + LuxTtsConstants.maxPromptSeconds * Double(LuxTtsConstants.melSampleRate)) + let promptFrames = LuxTtsMelExtractor().frameCount( + sampleCount: min(prompt24k.count, maxPromptSamples)) + + let spans: [[Int]] + if LuxTtsContinuation.fitsSinglePass( + textTokenCount: tokenIds.count, + promptFrames: promptFrames, + promptTokenCount: promptTokenIds.count, + speed: Double(speed)) + { + spans = [tokenIds] + } else { + let maxSpanTokens = LuxTtsContinuation.maxSpanTokens( + promptFrames: promptFrames, + promptTokenCount: promptTokenIds.count, + speed: Double(speed)) + spans = LuxTtsContinuation.chunks( + tokenIds: tokenIds, + maxTokens: maxSpanTokens, + boundaryTokenIds: boundaryTokens) + logger.info( + "LuxTTS continuation synthesis: \(tokenIds.count) target tokens in " + + "\(spans.count) balanced spans (≤ \(maxSpanTokens) tokens each)") + } + + let converter = AudioConverter(sampleRate: Double(LuxTtsConstants.melSampleRate)) + let crossfadeSamples = Int( + LuxTtsConstants.continuationCrossfadeSeconds + * Double(LuxTtsConstants.outputSampleRate)) + var currentPromptAudio = prompt24k + var currentPromptTokens = promptTokenIds + var samples: [Float] = [] + var originalPromptFrames = 0 + var totalGeneratedFrames = 0 + var previousSpeedFactor: Float = 1 + + for (index, span) in spans.enumerated() { + // A continuation prompt already speaks at the requested rate; + // applying `speed` again would compound it on every span. If the + // previous span was compressed by a retry, undo that here. + let (result, speedFactor) = try await synthesizeSpan( + synthesizer, + textTokenIds: span, + promptTokenIds: currentPromptTokens, + promptAudio24k: currentPromptAudio, + speed: index == 0 ? speed : 1 / previousSpeedFactor, + seed: seed &+ UInt64(index) &* LuxTtsConstants.continuationSeedStride, + allowedPauses: LuxTtsContinuation.expectedPauseCount( + in: span, pauseTokenIds: pauseTokens, boundaryTokenIds: boundaryTokens), + label: "span \(index + 1)/\(spans.count)") + if spans.count == 1 { return result } + + previousSpeedFactor = speedFactor + if index == 0 { originalPromptFrames = result.promptFrames } + totalGeneratedFrames += result.generatedFrames + var spanSamples = result.samples + if index > 0 { + spanSamples = LuxTtsContinuation.trimmingLeadingPadding( + spanSamples, sampleRate: result.sampleRate) + } + let hasNextSpan = index + 1 < spans.count + if hasNextSpan, + !LuxTtsContinuation.endsWithPausePunctuation( + span, pauseTokenIds: pauseTokens, boundaryTokenIds: boundaryTokens) + { + spanSamples = LuxTtsContinuation.trimmingTrailingPadding( + spanSamples, sampleRate: result.sampleRate) + } + LuxTtsContinuation.appendWithCrossfade( + spanSamples, to: &samples, crossfadeSamples: crossfadeSamples) + + guard hasNextSpan else { continue } + // Prompt with the untrimmed span so its frames-per-token ratio is + // the one the model actually produced for these tokens. + currentPromptAudio = try converter.resample( + result.samples, from: Double(result.sampleRate)) + currentPromptTokens = span + } + + return LuxTtsSynthesisResult( + samples: samples, + sampleRate: LuxTtsConstants.outputSampleRate, + promptFrames: originalPromptFrames, + generatedFrames: totalGeneratedFrames, + featuresLength: originalPromptFrames + totalGeneratedFrames) + } + + /// One flow-matching pass with a bounded re-seed ladder. The model can + /// drop a spurious mid-phrase pause whose position depends on the exact + /// (length, noise) draw (issue #937; the PyTorch reference does the + /// same), so a pass whose silences exceed the span's punctuation is + /// re-drawn with the next seed, later attempts also slightly compressed + /// in duration. The cleanest attempt is kept. Returns the pass and the + /// speed factor it was rendered with. + private func synthesizeSpan( + _ synthesizer: LuxTtsSynthesizer, + textTokenIds: [Int], + promptTokenIds: [Int], + promptAudio24k: [Float], + speed: Float, + seed: UInt64, + allowedPauses: Int, + label: String + ) async throws -> (LuxTtsSynthesisResult, Float) { + let factors = LuxTtsConstants.spuriousPauseRetrySpeedFactors + func render(_ attempt: Int) async throws -> (LuxTtsSynthesisResult, Float, Int) { + let factor = factors[min(attempt, factors.count - 1)] + let result = try await synthesizer.synthesize( + promptTokenIds: promptTokenIds, + textTokenIds: textTokenIds, + promptAudio24k: promptAudio24k, + speed: speed * factor, + seed: seed &+ UInt64(attempt)) + let pauses = LuxTtsContinuation.innerPauseCount( + result.samples, sampleRate: result.sampleRate) + return (result, factor, max(0, pauses - allowedPauses)) + } + + var (best, bestFactor, bestSpurious) = try await render(0) + guard bestSpurious > 0 else { return (best, bestFactor) } + + for attempt in 1...max(1, LuxTtsConstants.spuriousPauseRetries) { + logger.info( + "LuxTTS \(label): \(bestSpurious) spurious pause(s) beyond the " + + "\(allowedPauses) the text allows; re-drawing (attempt \(attempt))") + let (candidate, factor, spurious) = try await render(attempt) + if spurious < bestSpurious { + (best, bestFactor, bestSpurious) = (candidate, factor, spurious) + } + if spurious == 0 { return (candidate, factor) } + } + logger.warning( + "LuxTTS \(label): \(bestSpurious) spurious pause(s) remain after " + + "\(LuxTtsConstants.spuriousPauseRetries) re-draws; keeping the cleanest pass") + return (best, bestFactor) } public func cleanup() async { diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift index 9736d374..88deb293 100644 --- a/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift @@ -119,8 +119,8 @@ struct LuxTtsSynthesizer { throw LuxTtsError.tokenizerFailed( "estimated only \(genFrames) generated frames — text too short") } - // TODO(phase 2): chunk long inputs across multiple vocoder windows - // instead of erroring; mel truncation is NOT allowed. + // Long inputs are split into continuation-prompted spans by + // `LuxTtsManager` before reaching here; mel truncation is NOT allowed. guard let bucket = LuxTtsConstants.vocoderBuckets.first(where: { $0 >= genFrames }) else { throw LuxTtsError.inputTooLong( "generated frames \(genFrames) exceed the largest vocoder bucket " diff --git a/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsContinuationTests.swift b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsContinuationTests.swift new file mode 100644 index 00000000..8759ec59 --- /dev/null +++ b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsContinuationTests.swift @@ -0,0 +1,190 @@ +import XCTest + +@testable import FluidAudio + +final class LuxTtsContinuationTests: XCTestCase { + + func testChunksStayBalancedAndBreakAtBoundaries() { + let space = 0 + let tokens = Array(repeating: [1, 2, 3, 4, space], count: 22).flatMap { $0 } + let chunks = LuxTtsContinuation.chunks( + tokenIds: tokens, maxTokens: 102, boundaryTokenIds: [space]) + + XCTAssertEqual(chunks.count, 2) + XCTAssertEqual(chunks.flatMap { $0 }, tokens) + XCTAssertTrue(chunks.allSatisfy { $0.count <= 102 }) + XCTAssertTrue(chunks.dropLast().allSatisfy { $0.last == space }) + XCTAssertLessThanOrEqual(abs(chunks[0].count - chunks[1].count), 5) + } + + func testChunksKeepStableIssue937BoundarySinglePass() { + XCTAssertEqual( + LuxTtsContinuation.chunks( + tokenIds: Array(0..<102), maxTokens: 102, boundaryTokenIds: [] + ).count, + 1) + XCTAssertEqual( + LuxTtsContinuation.chunks( + tokenIds: Array(0..<106), maxTokens: 102, boundaryTokenIds: [] + ).map(\.count), + [53, 53]) + } + + func testChunksNeverLeaveAnOversizedRemainder() { + let tokens = Array(0..<205) + let chunks = LuxTtsContinuation.chunks( + tokenIds: tokens, maxTokens: 102, boundaryTokenIds: [0, 1]) + + XCTAssertEqual(chunks.flatMap { $0 }, tokens) + XCTAssertTrue(chunks.allSatisfy { !$0.isEmpty && $0.count <= 102 }) + XCTAssertEqual(chunks.map(\.count), [68, 69, 68]) + } + + func testMaxSpanTokensIsBoundedByStableRegimeAndFrameBudget() { + // Fast prompt: the 102-token stable-regime cap wins. + XCTAssertEqual( + LuxTtsContinuation.maxSpanTokens(promptFrames: 300, promptTokenCount: 100, speed: 1.0), + 102) + // Issue #937 prompt (100 tokens / 5 s): 468 frames ÷ 4.69 frames/token. + XCTAssertEqual( + LuxTtsContinuation.maxSpanTokens(promptFrames: 469, promptTokenCount: 100, speed: 1.0), + 99) + // Slower speech needs proportionally fewer tokens per span. + XCTAssertEqual( + LuxTtsContinuation.maxSpanTokens(promptFrames: 469, promptTokenCount: 100, speed: 0.5), + 49) + // Degenerate prompts fall back to the cap (the synthesizer rejects them). + XCTAssertEqual( + LuxTtsContinuation.maxSpanTokens(promptFrames: 0, promptTokenCount: 0, speed: 1.0), + 102) + } + + func testSpanFrameBudgetFitsInsidePromptCap() { + // A span at the budget yields (budget - 1) * hop48k samples at 48 kHz; + // at 24 kHz that must fit under the synthesizer's prompt cap so the + // next span's prompt is not truncated away from its transcript. + let budget = LuxTtsConstants.continuationSpanFrameBudget + let samples24k = (budget - 1) * LuxTtsConstants.hop48k / 2 + XCTAssertLessThanOrEqual( + samples24k, + Int(LuxTtsConstants.maxPromptSeconds * Double(LuxTtsConstants.melSampleRate))) + XCTAssertLessThanOrEqual(budget, LuxTtsConstants.vocoderBuckets.max() ?? 0) + } + + func testFitsSinglePassUsesIssue937PromptGeometry() { + // Reporter's prompt: 100 tokens over 5 s (469 frames), speed 1.0. + XCTAssertTrue( + LuxTtsContinuation.fitsSinglePass( + textTokenCount: 102, promptFrames: 469, promptTokenCount: 100, speed: 1.0)) + XCTAssertFalse( + LuxTtsContinuation.fitsSinglePass( + textTokenCount: 106, promptFrames: 469, promptTokenCount: 100, speed: 1.0)) + // Half speed doubles the frame estimate past the 555 bucket. + XCTAssertFalse( + LuxTtsContinuation.fitsSinglePass( + textTokenCount: 102, promptFrames: 469, promptTokenCount: 100, speed: 0.5)) + } + + func testExpectedPauseCountIgnoresTrailingBoundaryTokens() { + let space = 0 + let comma = 1 + let period = 2 + let pauses: Set = [comma, period] + let boundaries: Set = [space, comma, period] + XCTAssertEqual( + LuxTtsContinuation.expectedPauseCount( + in: [5, 6, comma, space, 7, 8, period, space], + pauseTokenIds: pauses, boundaryTokenIds: boundaries), + 1) + XCTAssertEqual( + LuxTtsContinuation.expectedPauseCount( + in: [5, 6, 7], pauseTokenIds: pauses, boundaryTokenIds: boundaries), + 0) + } + + func testInnerPauseCountFindsOnlyMidSpeechSilences() { + let sampleRate = 1_000 + let speech = Array(repeating: Float(0.5), count: 300) + let lead = Array(repeating: Float.zero, count: 200) + let shortClosure = Array(repeating: Float.zero, count: 50) + let pause = Array(repeating: Float.zero, count: 150) + // The vocoder's onset click: loud but far shorter than sustained speech. + let click = Array(repeating: Float(1), count: 20) + + XCTAssertEqual( + LuxTtsContinuation.innerPauseCount(lead + speech + lead, sampleRate: sampleRate), 0) + XCTAssertEqual( + LuxTtsContinuation.innerPauseCount(click + lead + speech + lead, sampleRate: sampleRate), + 0) + XCTAssertEqual( + LuxTtsContinuation.innerPauseCount( + lead + speech + shortClosure + speech + lead, sampleRate: sampleRate), + 0) + XCTAssertEqual( + LuxTtsContinuation.innerPauseCount( + lead + speech + pause + speech + shortClosure + speech + pause + speech, + sampleRate: sampleRate), + 2) + XCTAssertEqual(LuxTtsContinuation.innerPauseCount([], sampleRate: sampleRate), 0) + } + + func testCrossfadePreservesEdgesAndExpectedLength() { + var output: [Float] = [1, 1, 1, 1] + LuxTtsContinuation.appendWithCrossfade( + [0, 0, 0, 0], to: &output, crossfadeSamples: 3) + + XCTAssertEqual(output.count, 5) + XCTAssertEqual(output.first, 1) + XCTAssertEqual(output.last, 0) + XCTAssertEqual(output[1], 1) + XCTAssertEqual(output[2], 0.5) + XCTAssertEqual(output[3], 0) + } + + func testLeadingPaddingTrimKeepsOnsetPreroll() { + let samples = + Array(repeating: Float.zero, count: 50) + + Array(repeating: Float(1), count: 50) + + let trimmed = LuxTtsContinuation.trimmingLeadingPadding(samples, sampleRate: 1_000) + + XCTAssertEqual(trimmed.count, 80) + XCTAssertEqual(Array(trimmed.prefix(30)), Array(repeating: Float.zero, count: 30)) + XCTAssertEqual(trimmed[30], 1) + } + + func testLeadingPaddingTrimSkipsOnsetClick() { + let click = Array(repeating: Float(1), count: 20) + let padding = Array(repeating: Float.zero, count: 200) + let speech = Array(repeating: Float(0.5), count: 100) + + let trimmed = LuxTtsContinuation.trimmingLeadingPadding( + click + padding + speech, sampleRate: 1_000) + + // 30 ms preroll before the sustained onset at sample 220. + XCTAssertEqual(trimmed.count, 130) + XCTAssertEqual(trimmed[30], 0.5) + } + + func testTrailingPaddingTrimKeepsPostroll() { + let speech = Array(repeating: Float(1), count: 50) + let padding = Array(repeating: Float.zero, count: 200) + + let trimmed = LuxTtsContinuation.trimmingTrailingPadding(speech + padding, sampleRate: 1_000) + + XCTAssertEqual(trimmed.count, 80) + XCTAssertEqual(trimmed[49], 1) + XCTAssertEqual(trimmed[50], 0) + } + + func testEndsWithPausePunctuationIgnoresTrailingSpaces() { + let space = 0 + let comma = 1 + XCTAssertTrue( + LuxTtsContinuation.endsWithPausePunctuation( + [5, 6, comma, space], pauseTokenIds: [comma], boundaryTokenIds: [space, comma])) + XCTAssertFalse( + LuxTtsContinuation.endsWithPausePunctuation( + [5, comma, 6, space], pauseTokenIds: [comma], boundaryTokenIds: [space, comma])) + } +} From a9c9880768fa14abd75ac1d5aeeeff33183e2e3e Mon Sep 17 00:00:00 2001 From: Alex-Wengg Date: Sun, 20 Sep 2026 12:29:29 -0400 Subject: [PATCH 2/2] fix(tts/luxtts): address review of the #937 continuation + re-draw pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Token bucket: fitsSinglePass and maxSpanTokens now leave room for the prompt transcript in the 256-token TextEncoder bucket (prompt + span + pad ≤ 256), so a long prompt transcript no longer throws inputTooLong on span 1. - Pathological prompt ratios are rejected in the chunking path instead of turning into dozens of passes: frames-per-token > 12 (transcript covering only part of the clip) or spans under 8 tokens throw inputTooLong with guidance. Single-pass behaviour is unchanged. - Re-draw budget is a parameter (`maxRedraws`, default 3; CLI `--redraws`), and the result reports `redraws` / `residualPauses`. LuxTtsE2ETests pins the raw pass with maxRedraws: 0 — the fixture text draws one breath beyond its two commas and would otherwise be re-drawn (verified: 838/432/220672 exact with 0, 825/419 without). - Duration compression on re-draws removed. On the 101-char text at seed 42 the ×1.03 pass removed the pause but squeezed the final word into the last frames (tail 120 ms only 3 dB under body vs 6–9 dB for clean passes; ASR read an extra word), and it did not fix the two persistent cases. Re-draws are pure re-seeds; the pace bookkeeping is gone. - Task.checkCancellation() before every pass. - Pause allowance: hyphen token added; the text API credits ellipses and quotes/brackets the G2P renders silently (`textPauseAllowance`). - Prompt clips longer than 5 s log a warning (audio is capped, the transcript is not, and the ratio now drives every span). - One sustained-speech scan per kept pass (vDSP peak / mean-square); leading and trailing trims are a single slice into the crossfade. - Resample failure on a continuation prompt is wrapped in LuxTtsError.inferenceFailed; single AudioConverter; maxPromptSamples constant; result-field docs describe the summed multi-span semantics; the "ratio the model produced" comment corrected (it is host-fixed). - The 102-token single-pass cap is kept, now measured: the 106-token issue text paused on 6 of 8 raw single-pass draws and the ladder still failed one of six seeds, while as two ~53-token spans all six seeds were clean with two re-draws total. Not changed: re-draws redo the text-encoder and prompt-mel stages (5–12 % of a pass); caching them needs a synthesizer split. The leading "word" ASR sometimes reports at 0.00–0.08 s (conf < 0.25) is the vocoder's opening click and appears on pre-PR renders too. --- Documentation/TTS/LuxTts.md | 40 ++-- .../TTS/LuxTts/LuxTtsConstants.swift | 30 ++- .../TTS/LuxTts/LuxTtsContinuation.swift | 157 +++++++++----- .../FluidAudio/TTS/LuxTts/LuxTtsManager.swift | 196 ++++++++++++------ .../TTS/LuxTts/LuxTtsSynthesizer.swift | 20 +- .../FluidAudioCLI/Commands/TTSCommand.swift | 20 +- .../TTS/LuxTts/LuxTtsContinuationTests.swift | 33 ++- .../TTS/LuxTts/LuxTtsE2ETests.swift | 6 +- 8 files changed, 349 insertions(+), 153 deletions(-) diff --git a/Documentation/TTS/LuxTts.md b/Documentation/TTS/LuxTts.md index 99ed232a..4d08c1d1 100644 --- a/Documentation/TTS/LuxTts.md +++ b/Documentation/TTS/LuxTts.md @@ -74,17 +74,23 @@ jetsam-visible footprint is why iOS uses it. All graph shapes are fixed: ≤ 255 tokens (+1 pad slot), ≤ 1024 mel frames total, ≤ 555 generated frames (~5.9 s) per flow-matching pass. -Longer text is handled by `LuxTtsManager` with continuation prompting -rather than a bigger graph: target tokens are split into balanced spans at -word/punctuation boundaries, each span is generated with the previous -span's audio and tokens as its prompt, the vocoder's onset/tail padding -is trimmed around sustained speech (a span ending in punctuation keeps its -tail), and spans are joined with a 30 ms crossfade. Spans are capped at -102 target tokens and bounded so no span generates more than ~5 s (468 -frames), which keeps every span short enough to prompt the next one -untruncated. `speed` is applied to the first span only; later spans -inherit the rate from their prompt. Text that fits one pass is synthesized -exactly as before. +Text that fits one pass (≤ 102 target tokens, token bucket, 1024 frames, +555-frame vocoder bucket) is rendered in one pass; the 102 cap is measured, +not a graph limit — see `LuxTtsConstants.continuationSpanTokens`. Longer text is handled by `LuxTtsManager` +with continuation prompting rather than a bigger graph: target tokens are +split into balanced spans of ≤ 102 tokens at word/punctuation boundaries, +each span is generated with the previous span's audio and tokens as its +prompt, the vocoder's onset/tail padding is trimmed around sustained +speech (a span ending in punctuation keeps its tail), and spans are joined +with a 30 ms crossfade. Spans are also bounded by the prompt's +frames-per-token ratio so none generates more than ~5 s (468 frames), +which keeps every span short enough to prompt the next one untruncated; +a prompt whose ratio exceeds 12 frames per token (transcript covering only +part of the clip) or would force spans under 8 tokens is rejected with +`inputTooLong`. `speed` is applied to the first span only; later spans +inherit the rate from their prompt. For multi-span output +`generatedFrames`/`featuresLength` are sums over spans and `samples.count` +is shorter than `(generatedFrames − 1) × 512`. The model also drops the occasional spurious mid-phrase pause (issue #937): where it lands depends on the exact length/noise draw — the PyTorch @@ -92,10 +98,14 @@ reference does it too, and a small `speed` change moves or removes it — so longer text simply has more chances to draw one. Every pass is scanned for gaps ≥ 80 ms between runs of sustained speech (≤ −45 dB relative to peak; natural stop closures stay ≤ 60 ms); when they outnumber the span's -pause punctuation the span is re-drawn with the next seed, up to 3 times -(the last two also 3 % / 6 % shorter, since surplus estimated frames are the -other source of pauses; the following span is prompted back at the original -pace), keeping the cleanest pass. +pause punctuation (`, . ; : ! ? -`, plus ellipses and quotes/brackets the +G2P renders silently when going through the text API) the span is re-drawn +with the next seed, up to `maxRedraws` times (default 3), keeping the +cleanest pass. Re-draws keep the duration estimate: compressing it (speed +× 1.03–1.06) removed some pauses but squeezed the final word into the last +frames. This applies to single-pass text too, so a given seed is +only bit-reproducible with `maxRedraws: 0` (CLI `--redraws 0`); the result +reports `redraws` and `residualPauses`. Each re-draw is one extra pass. ## Quick Start diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift index c800cad4..615942b4 100644 --- a/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsConstants.swift @@ -45,6 +45,8 @@ public enum LuxTtsConstants { /// Prompt duration cap in seconds. Frames beyond this would eat too much /// of the 1024-frame bucket (~10.9 s total at 93.75 frames/s). public static let maxPromptSeconds: Double = 5.0 + /// `maxPromptSeconds` in 24 kHz samples. + public static let maxPromptSamples = Int(maxPromptSeconds * Double(melSampleRate)) /// Published fixed-shape vocoder buckets (generated frames). public static let vocoderBuckets = [282, 555] @@ -55,11 +57,22 @@ public enum LuxTtsConstants { /// Default synthesis noise seed (matches the Python reference scripts). public static let defaultSeed: UInt64 = 42 - /// Largest target-token span sent through one flow-matching pass. Spans - /// this size keep the per-pass odds of a spurious mid-phrase pause low - /// (issue #937) and fit the frame budget for typical prompts; longer text - /// is continuation-prompted so callers never see chunk seams. - public static let maxSinglePassTextTokens = 102 + /// Largest target-token span sent through one flow-matching pass; longer + /// text is continuation-prompted in balanced spans of at most this size. + /// Measured on the issue #937 text (106 tokens, six seeds): as one pass + /// it paused on 6 of 8 raw draws and the re-draw ladder still failed one + /// seed; as two ~53-token spans every seed was clean with two re-draws + /// in total. Shorter passes draw pauses far less often. + public static let continuationSpanTokens = 102 + /// Smallest span worth rendering. A prompt whose frames-per-token ratio + /// forces spans below this (silence-heavy clip, transcript that does not + /// match) is rejected instead of turning into dozens of passes. + public static let minimumSpanTokens = 8 + /// Highest plausible prompt frames-per-token ratio for continuation + /// synthesis. Natural speech sits around 4–6 (93.75 frames/s); a ratio + /// beyond this means the transcript covers only part of the clip (or + /// the clip is mostly silence), and every span would inherit the error. + public static let maxPromptFramesPerToken = 12.0 /// Re-seed attempts for a pass whose mid-speech silences outnumber the /// span's pause punctuation. The model drops such pauses stochastically @@ -67,17 +80,12 @@ public enum LuxTtsConstants { /// reference does the same), so a fresh seed is the fix. One full pass /// each. public static let spuriousPauseRetries = 3 - /// Duration compression applied on each re-seed (indexed by attempt, - /// attempt 0 is the original pass). A surplus of estimated frames is the - /// other way the model ends up with a pause to fill, so later attempts - /// also shorten the span a little. The next span is prompted at the - /// original pace, so the nudge does not propagate. - public static let spuriousPauseRetrySpeedFactors: [Float] = [1.0, 1.0, 1.03, 1.06] /// Silence floor (dB relative to the pass's peak) below which audio /// counts as padding or pause, and the minimum gap between sustained /// speech that counts as a pause. Natural stop closures stay ≤ 60 ms at /// this floor; the reported pauses measure 100–160 ms. public static let pauseFloorDb: Float = -45 + /// Minimum gap between sustained-speech runs that counts as a pause. public static let pauseMinimumSeconds = 0.08 /// Seed stride between continuation spans; leaves room for re-seeds. public static let continuationSeedStride: UInt64 = 64 diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsContinuation.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsContinuation.swift index a2f4d706..36314489 100644 --- a/Sources/FluidAudio/TTS/LuxTts/LuxTtsContinuation.swift +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsContinuation.swift @@ -1,3 +1,4 @@ +import Accelerate import Foundation /// Host-side long-utterance helpers for LuxTTS. @@ -12,12 +13,21 @@ enum LuxTtsContinuation { private static let onsetPrerollSeconds = 0.03 private static let tailPostrollSeconds = 0.03 - /// Whether the whole text can go through one pass unchanged: within the - /// span cap, the 1024-frame graph, and the largest vocoder bucket. + /// Runs of sustained speech, in `windowSize`-sample analysis windows. + struct SpeechRuns { + let runs: [Range] + let windowSize: Int + } + + /// Whether the whole text can go through one pass unchanged: at most + /// `continuationSpanTokens`, and within the token bucket, the 1024-frame + /// graph, and the largest vocoder bucket. static func fitsSinglePass( textTokenCount: Int, promptFrames: Int, promptTokenCount: Int, speed: Double ) -> Bool { - guard textTokenCount <= LuxTtsConstants.maxSinglePassTextTokens else { return false } + guard textTokenCount <= LuxTtsConstants.continuationSpanTokens, + promptTokenCount + textTokenCount + 1 <= LuxTtsConstants.maxTokens + else { return false } guard promptFrames > 0, promptTokenCount > 0, speed > 0 else { return true } let featuresLength = LuxTtsSolver.featuresLength( promptFrames: promptFrames, @@ -28,12 +38,15 @@ enum LuxTtsContinuation { && featuresLength - promptFrames <= (LuxTtsConstants.vocoderBuckets.max() ?? 0) } - /// Largest span (in target tokens) that both stays inside the model's - /// stable regime and generates at most `continuationSpanFrameBudget` - /// frames for this prompt's frames-per-token ratio at `speed`. + /// Largest span (in target tokens) that fits the token bucket next to + /// this prompt's transcript and generates at most + /// `continuationSpanFrameBudget` frames for its frames-per-token ratio + /// at `speed`, capped at `continuationSpanTokens`. static func maxSpanTokens(promptFrames: Int, promptTokenCount: Int, speed: Double) -> Int { - let cap = LuxTtsConstants.maxSinglePassTextTokens - guard promptFrames > 0, promptTokenCount > 0, speed > 0 else { return cap } + let cap = min( + LuxTtsConstants.continuationSpanTokens, + LuxTtsConstants.maxTokens - 1 - promptTokenCount) + guard promptFrames > 0, promptTokenCount > 0, speed > 0 else { return max(1, cap) } let framesPerToken = Double(promptFrames) / Double(promptTokenCount) let budgetTokens = Int( (Double(LuxTtsConstants.continuationSpanFrameBudget) * speed / framesPerToken) @@ -88,6 +101,17 @@ enum LuxTtsContinuation { return tokenIds[.. Int { + let ellipses = text.components(separatedBy: "...").count - 1 + let silentBreaks = text.reduce(0) { count, character in + count + ("…\"()[]«»".contains(character) ? 1 : 0) + } + return ellipses + silentBreaks + } + /// Whether the span's last spoken token is pause punctuation (trailing /// spaces ignored). static func endsWithPausePunctuation( @@ -98,84 +122,102 @@ enum LuxTtsContinuation { return pauseTokenIds.contains(last) } - /// Window ranges (in `windowSize`-sample windows) of sustained speech: - /// runs of at least `sustainedSpeechSeconds` above `pauseFloorDb` + /// Runs of at least `sustainedSpeechSeconds` above `pauseFloorDb` /// relative to the clip's peak. - static func speechRuns(_ samples: [Float], sampleRate: Int) -> (runs: [Range], windowSize: Int) { - guard sampleRate > 0, !samples.isEmpty else { return ([], 1) } + static func speechRuns(_ samples: [Float], sampleRate: Int) -> SpeechRuns { + guard sampleRate > 0, !samples.isEmpty else { return SpeechRuns(runs: [], windowSize: 1) } let windowSize = max(1, sampleRate / analysisWindowsPerSecond) let windowCount = samples.count / windowSize - guard windowCount > 0 else { return ([], windowSize) } + guard windowCount > 0 else { return SpeechRuns(runs: [], windowSize: windowSize) } var peak: Float = 0 - for sample in samples { peak = max(peak, abs(sample)) } - guard peak > 0 else { return ([], windowSize) } + vDSP_maxmgv(samples, 1, &peak, vDSP_Length(samples.count)) + guard peak > 0 else { return SpeechRuns(runs: [], windowSize: windowSize) } let floorMeanSquare = peak * peak * powf(10, LuxTtsConstants.pauseFloorDb / 10) let minimumWindows = max( 1, Int((sustainedSpeechSeconds * Double(sampleRate) / Double(windowSize)).rounded(.up))) var runs: [Range] = [] var runStart: Int? - for window in 0...windowCount { - var active = false - if window < windowCount { - var squareSum: Float = 0 - let start = window * windowSize - for sample in samples[start..<(start + windowSize)] { - squareSum += sample * sample + samples.withUnsafeBufferPointer { buffer in + guard let base = buffer.baseAddress else { return } + for window in 0...windowCount { + var active = false + if window < windowCount { + var meanSquare: Float = 0 + vDSP_measqv(base + window * windowSize, 1, &meanSquare, vDSP_Length(windowSize)) + active = meanSquare > floorMeanSquare + } + if active { + if runStart == nil { runStart = window } + } else if let start = runStart { + if window - start >= minimumWindows { runs.append(start.. floorMeanSquare - } - if active { - if runStart == nil { runStart = window } - } else if let start = runStart { - if window - start >= minimumWindows { runs.append(start.. Int { - let (runs, windowSize) = speechRuns(samples, sampleRate: sampleRate) - guard runs.count > 1 else { return 0 } + static func innerPauseCount(_ speech: SpeechRuns, sampleRate: Int) -> Int { + guard speech.runs.count > 1 else { return 0 } let minimumWindows = max( 1, - Int((LuxTtsConstants.pauseMinimumSeconds * Double(sampleRate) / Double(windowSize)).rounded(.up))) - return zip(runs, runs.dropFirst()).reduce(0) { count, pair in + Int( + (LuxTtsConstants.pauseMinimumSeconds * Double(sampleRate) / Double(speech.windowSize)) + .rounded(.up))) + return zip(speech.runs, speech.runs.dropFirst()).reduce(0) { count, pair in count + (pair.1.lowerBound - pair.0.upperBound >= minimumWindows ? 1 : 0) } } - /// Remove the vocoder's onset padding from a continuation span, keeping a - /// short preroll so unvoiced consonants are not clipped. + static func innerPauseCount(_ samples: [Float], sampleRate: Int) -> Int { + innerPauseCount(speechRuns(samples, sampleRate: sampleRate), sampleRate: sampleRate) + } + + /// Sample range to keep from a span: the vocoder's onset padding is cut + /// (leaving a short preroll so unvoiced consonants are not clipped) and + /// the tail padding likewise (leaving a short postroll). Clips without + /// sustained speech are kept whole. + static func speechSlice( + _ speech: SpeechRuns, sampleCount: Int, sampleRate: Int, trimLeading: Bool, trimTrailing: Bool + ) -> Range { + guard let first = speech.runs.first, let last = speech.runs.last else { return 0.. [Float] { - let (runs, windowSize) = speechRuns(samples, sampleRate: sampleRate) - guard let onset = runs.first?.lowerBound else { return samples } - let preroll = Int(onsetPrerollSeconds * Double(sampleRate)) - let trimStart = max(0, onset * windowSize - preroll) - return trimStart == 0 ? samples : Array(samples[trimStart...]) + let slice = speechSlice( + speechRuns(samples, sampleRate: sampleRate), sampleCount: samples.count, + sampleRate: sampleRate, trimLeading: true, trimTrailing: false) + return Array(samples[slice]) } - /// Remove trailing padding from a span that another span will follow, - /// keeping a short postroll. Spans that end in pause punctuation keep - /// their tail: that silence is the sentence break the text asked for. static func trimmingTrailingPadding(_ samples: [Float], sampleRate: Int) -> [Float] { - let (runs, windowSize) = speechRuns(samples, sampleRate: sampleRate) - guard let end = runs.last?.upperBound else { return samples } - let postroll = Int(tailPostrollSeconds * Double(sampleRate)) - let trimEnd = min(samples.count, end * windowSize + postroll) - return trimEnd == samples.count ? samples : Array(samples[.., to output: inout [Float], crossfadeSamples: Int ) { guard !output.isEmpty else { - output = next + output = Array(next) return } guard !next.isEmpty else { return } @@ -187,19 +229,26 @@ enum LuxTtsContinuation { } let outputStart = output.count - overlap + let nextStart = next.startIndex if overlap == 1 { - output[outputStart] = (output[outputStart] + next[0]) * 0.5 + output[outputStart] = (output[outputStart] + next[nextStart]) * 0.5 } else { for index in 0.. LuxTtsSynthesisResult { // Fail fast before the (potentially expensive) G2P lexicon load and // phonemization; the phonemes path guards on the same store below. @@ -110,7 +111,9 @@ public actor LuxTtsManager { promptAudio: promptAudio, promptPhonemes: g2p.phonemize(text: promptText), speed: speed, - seed: seed) + seed: seed, + maxRedraws: maxRedraws, + extraPauseAllowance: LuxTtsContinuation.textPauseAllowance(text)) } /// The bundled espeak-parity English G2P (loaded lazily; ~4 MB of @@ -135,12 +138,18 @@ public actor LuxTtsManager { /// - speed: Speech-rate divisor for the generated span. Keep 1.0 /// (upstream's hidden 1.3 clips sentence onsets). /// - seed: Noise seed for the flow-matching init. + /// - maxRedraws: Re-draw budget per span for the spurious-pause + /// detector (see `synthesize(tokenIds:...)`); 0 pins the raw pass. + /// - extraPauseAllowance: Pauses the phonemes call for beyond their + /// punctuation tokens (e.g. ellipses the G2P dropped). public func synthesize( phonemes: String, promptAudio: URL, promptPhonemes: String, speed: Float = LuxTtsConstants.defaultSpeed, - seed: UInt64 = LuxTtsConstants.defaultSeed + seed: UInt64 = LuxTtsConstants.defaultSeed, + maxRedraws: Int = LuxTtsConstants.spuriousPauseRetries, + extraPauseAllowance: Int = 0 ) async throws -> LuxTtsSynthesisResult { guard let store = store else { throw LuxTtsError.notInitialized } let tokenizer = try await store.tokenizer() @@ -149,37 +158,52 @@ public actor LuxTtsManager { promptAudio: promptAudio, promptTokenIds: tokenizer.tokenIds(phonemes: promptPhonemes), speed: speed, - seed: seed) + seed: seed, + maxRedraws: maxRedraws, + extraPauseAllowance: extraPauseAllowance) } /// Synthesize from pre-computed token ids (callers running their own /// espeak frontend against `tokens.txt`). + /// + /// Text that fits one flow-matching pass is rendered in one pass; longer + /// text is split into continuation-prompted spans (see + /// `Documentation/TTS/LuxTts.md`). Every pass is checked for mid-phrase + /// pauses beyond the text's punctuation (issue #937) and re-drawn up to + /// `maxRedraws` times; pass 0 to keep the raw pass for a given seed. The + /// result reports `redraws` and `residualPauses`. public func synthesize( tokenIds: [Int], promptAudio: URL, promptTokenIds: [Int], speed: Float = LuxTtsConstants.defaultSpeed, - seed: UInt64 = LuxTtsConstants.defaultSeed + seed: UInt64 = LuxTtsConstants.defaultSeed, + maxRedraws: Int = LuxTtsConstants.spuriousPauseRetries, + extraPauseAllowance: Int = 0 ) async throws -> LuxTtsSynthesisResult { guard let store, let synthesizer else { throw LuxTtsError.notInitialized } + let converter = AudioConverter(sampleRate: Double(LuxTtsConstants.melSampleRate)) let prompt24k: [Float] do { - let converter = AudioConverter( - sampleRate: Double(LuxTtsConstants.melSampleRate)) prompt24k = try converter.resampleAudioFile(promptAudio) } catch { throw LuxTtsError.invalidPromptAudio( "cannot load \(promptAudio.path): \(error.localizedDescription)") } + if prompt24k.count > LuxTtsConstants.maxPromptSamples { + let seconds = Double(prompt24k.count) / Double(LuxTtsConstants.melSampleRate) + logger.warning( + "LuxTTS prompt is \(String(format: "%.1f", seconds)) s; only the first " + + "\(LuxTtsConstants.maxPromptSeconds) s condition the model while the whole " + + "transcript sets the duration ratio — trim the clip and transcript to match") + } let tokenizer = try await store.tokenizer() - let pauseTokens = Set([",", ".", ";", ":", "!", "?"].compactMap { tokenizer.tokenToId[$0] }) + let pauseTokens = Set([",", ".", ";", ":", "!", "?", "-"].compactMap { tokenizer.tokenToId[$0] }) let boundaryTokens = pauseTokens.union([tokenizer.tokenToId[" "]].compactMap { $0 }) - let maxPromptSamples = Int( - LuxTtsConstants.maxPromptSeconds * Double(LuxTtsConstants.melSampleRate)) let promptFrames = LuxTtsMelExtractor().frameCount( - sampleCount: min(prompt24k.count, maxPromptSamples)) + sampleCount: min(prompt24k.count, LuxTtsConstants.maxPromptSamples)) let spans: [[Int]] if LuxTtsContinuation.fitsSinglePass( @@ -194,6 +218,17 @@ public actor LuxTtsManager { promptFrames: promptFrames, promptTokenCount: promptTokenIds.count, speed: Double(speed)) + let ratio = Double(promptFrames) / Double(max(1, promptTokenIds.count)) + guard ratio <= LuxTtsConstants.maxPromptFramesPerToken, + maxSpanTokens >= LuxTtsConstants.minimumSpanTokens + else { + throw LuxTtsError.inputTooLong( + "prompt yields \(String(format: "%.1f", ratio)) mel frames per token (plausible " + + "≤ \(Int(LuxTtsConstants.maxPromptFramesPerToken))) at speed \(speed), " + + "leaving spans of \(maxSpanTokens) tokens (minimum " + + "\(LuxTtsConstants.minimumSpanTokens)); trim prompt silence or check that " + + "the transcript matches the clip") + } spans = LuxTtsContinuation.chunks( tokenIds: tokenIds, maxTokens: maxSpanTokens, @@ -203,7 +238,6 @@ public actor LuxTtsManager { + "\(spans.count) balanced spans (≤ \(maxSpanTokens) tokens each)") } - let converter = AudioConverter(sampleRate: Double(LuxTtsConstants.melSampleRate)) let crossfadeSamples = Int( LuxTtsConstants.continuationCrossfadeSeconds * Double(LuxTtsConstants.outputSampleRate)) @@ -212,48 +246,69 @@ public actor LuxTtsManager { var samples: [Float] = [] var originalPromptFrames = 0 var totalGeneratedFrames = 0 - var previousSpeedFactor: Float = 1 + var totalRedraws = 0 + var totalResidualPauses = 0 for (index, span) in spans.enumerated() { // A continuation prompt already speaks at the requested rate; - // applying `speed` again would compound it on every span. If the - // previous span was compressed by a retry, undo that here. - let (result, speedFactor) = try await synthesizeSpan( + // applying `speed` again would compound it on every span. + let pass = try await synthesizeSpan( synthesizer, textTokenIds: span, promptTokenIds: currentPromptTokens, promptAudio24k: currentPromptAudio, - speed: index == 0 ? speed : 1 / previousSpeedFactor, + speed: index == 0 ? speed : 1.0, seed: seed &+ UInt64(index) &* LuxTtsConstants.continuationSeedStride, - allowedPauses: LuxTtsContinuation.expectedPauseCount( - in: span, pauseTokenIds: pauseTokens, boundaryTokenIds: boundaryTokens), + maxRedraws: maxRedraws, + allowedPauses: extraPauseAllowance + + LuxTtsContinuation.expectedPauseCount( + in: span, pauseTokenIds: pauseTokens, boundaryTokenIds: boundaryTokens), label: "span \(index + 1)/\(spans.count)") - if spans.count == 1 { return result } + totalRedraws += pass.redraws + totalResidualPauses += pass.residualPauses + let result = pass.result + if spans.count == 1 { + return LuxTtsSynthesisResult( + samples: result.samples, + sampleRate: result.sampleRate, + promptFrames: result.promptFrames, + generatedFrames: result.generatedFrames, + featuresLength: result.featuresLength, + redraws: totalRedraws, + residualPauses: totalResidualPauses) + } - previousSpeedFactor = speedFactor if index == 0 { originalPromptFrames = result.promptFrames } totalGeneratedFrames += result.generatedFrames - var spanSamples = result.samples - if index > 0 { - spanSamples = LuxTtsContinuation.trimmingLeadingPadding( - spanSamples, sampleRate: result.sampleRate) - } + + // Cut onset padding on continuation spans and tail padding on + // spans another span follows, unless the span ends in pause + // punctuation: that tail is the sentence break the text asked for. let hasNextSpan = index + 1 < spans.count - if hasNextSpan, - !LuxTtsContinuation.endsWithPausePunctuation( + let keepTail = + !hasNextSpan + || LuxTtsContinuation.endsWithPausePunctuation( span, pauseTokenIds: pauseTokens, boundaryTokenIds: boundaryTokens) - { - spanSamples = LuxTtsContinuation.trimmingTrailingPadding( - spanSamples, sampleRate: result.sampleRate) - } + let slice = LuxTtsContinuation.speechSlice( + pass.speech, + sampleCount: result.samples.count, + sampleRate: result.sampleRate, + trimLeading: index > 0, + trimTrailing: !keepTail) LuxTtsContinuation.appendWithCrossfade( - spanSamples, to: &samples, crossfadeSamples: crossfadeSamples) + result.samples[slice], to: &samples, crossfadeSamples: crossfadeSamples) guard hasNextSpan else { continue } - // Prompt with the untrimmed span so its frames-per-token ratio is - // the one the model actually produced for these tokens. - currentPromptAudio = try converter.resample( - result.samples, from: Double(result.sampleRate)) + // Prompt with the untrimmed span: its frame count is exactly what + // the host allotted for these tokens, so every span keeps the + // first prompt's frames-per-token ratio instead of drifting. + do { + currentPromptAudio = try converter.resample( + result.samples, from: Double(result.sampleRate)) + } catch { + throw LuxTtsError.inferenceFailed( + stage: "continuation prompt resample", underlying: "\(error)") + } currentPromptTokens = span } @@ -262,16 +317,25 @@ public actor LuxTtsManager { sampleRate: LuxTtsConstants.outputSampleRate, promptFrames: originalPromptFrames, generatedFrames: totalGeneratedFrames, - featuresLength: originalPromptFrames + totalGeneratedFrames) + featuresLength: originalPromptFrames + totalGeneratedFrames, + redraws: totalRedraws, + residualPauses: totalResidualPauses) } - /// One flow-matching pass with a bounded re-seed ladder. The model can + private struct SpanPass { + let result: LuxTtsSynthesisResult + let speech: LuxTtsContinuation.SpeechRuns + let redraws: Int + let residualPauses: Int + } + + /// One flow-matching pass with a bounded re-draw ladder. The model can /// drop a spurious mid-phrase pause whose position depends on the exact /// (length, noise) draw (issue #937; the PyTorch reference does the /// same), so a pass whose silences exceed the span's punctuation is - /// re-drawn with the next seed, later attempts also slightly compressed - /// in duration. The cleanest attempt is kept. Returns the pass and the - /// speed factor it was rendered with. + /// re-drawn with the next seed. The cleanest attempt is kept. Re-draws + /// keep the duration: compressing it (speed × 1.03–1.06) traded the pause + /// for a clipped final word. private func synthesizeSpan( _ synthesizer: LuxTtsSynthesizer, textTokenIds: [Int], @@ -279,40 +343,48 @@ public actor LuxTtsManager { promptAudio24k: [Float], speed: Float, seed: UInt64, + maxRedraws: Int, allowedPauses: Int, label: String - ) async throws -> (LuxTtsSynthesisResult, Float) { - let factors = LuxTtsConstants.spuriousPauseRetrySpeedFactors - func render(_ attempt: Int) async throws -> (LuxTtsSynthesisResult, Float, Int) { - let factor = factors[min(attempt, factors.count - 1)] + ) async throws -> SpanPass { + func render(_ attempt: Int) async throws -> SpanPass { + try Task.checkCancellation() let result = try await synthesizer.synthesize( promptTokenIds: promptTokenIds, textTokenIds: textTokenIds, promptAudio24k: promptAudio24k, - speed: speed * factor, + speed: speed, seed: seed &+ UInt64(attempt)) - let pauses = LuxTtsContinuation.innerPauseCount( - result.samples, sampleRate: result.sampleRate) - return (result, factor, max(0, pauses - allowedPauses)) + let speech = LuxTtsContinuation.speechRuns(result.samples, sampleRate: result.sampleRate) + let pauses = LuxTtsContinuation.innerPauseCount(speech, sampleRate: result.sampleRate) + return SpanPass( + result: result, speech: speech, redraws: attempt, + residualPauses: max(0, pauses - allowedPauses)) } - var (best, bestFactor, bestSpurious) = try await render(0) - guard bestSpurious > 0 else { return (best, bestFactor) } + var best = try await render(0) + guard best.residualPauses > 0, maxRedraws > 0 else { return best } - for attempt in 1...max(1, LuxTtsConstants.spuriousPauseRetries) { + var redraws = 0 + for attempt in 1...maxRedraws { logger.info( - "LuxTTS \(label): \(bestSpurious) spurious pause(s) beyond the " + "LuxTTS \(label): \(best.residualPauses) spurious pause(s) beyond the " + "\(allowedPauses) the text allows; re-drawing (attempt \(attempt))") - let (candidate, factor, spurious) = try await render(attempt) - if spurious < bestSpurious { - (best, bestFactor, bestSpurious) = (candidate, factor, spurious) + let candidate = try await render(attempt) + redraws = attempt + if candidate.residualPauses < best.residualPauses { + best = candidate } - if spurious == 0 { return (candidate, factor) } + if candidate.residualPauses == 0 { break } + } + if best.residualPauses > 0 { + logger.warning( + "LuxTTS \(label): \(best.residualPauses) spurious pause(s) remain after " + + "\(redraws) re-draws; keeping the cleanest pass") } - logger.warning( - "LuxTTS \(label): \(bestSpurious) spurious pause(s) remain after " - + "\(LuxTtsConstants.spuriousPauseRetries) re-draws; keeping the cleanest pass") - return (best, bestFactor) + return SpanPass( + result: best.result, speech: best.speech, redraws: redraws, + residualPauses: best.residualPauses) } public func cleanup() async { diff --git a/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift b/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift index 88deb293..324e2d31 100644 --- a/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift +++ b/Sources/FluidAudio/TTS/LuxTts/LuxTtsSynthesizer.swift @@ -11,10 +11,22 @@ public struct LuxTtsSynthesisResult: Sendable { public let sampleRate: Int /// Prompt conditioning length in 24 kHz mel frames. public let promptFrames: Int - /// Generated mel frames (`featuresLength - promptFrames`). + /// Generated mel frames. For a single pass this is + /// `featuresLength - promptFrames` and `samples.count == (generatedFrames - 1) * 512`; + /// for continuation-prompted text it is the sum over spans, whose + /// trimmed padding and crossfades make `samples.count` shorter. public let generatedFrames: Int - /// Total flow-matching sequence length (prompt + generated frames). + /// Flow-matching sequence length (prompt + generated frames), summed + /// over spans for continuation-prompted text (then it can exceed the + /// per-pass 1024-frame graph). public let featuresLength: Int + /// Extra flow-matching passes spent re-drawing spans the pause detector + /// flagged (0 = every span was clean on its first pass). See + /// `LuxTtsManager.synthesize(tokenIds:...maxRedraws:)`. + public let redraws: Int + /// Pauses beyond the text's punctuation still present in the kept + /// passes after the re-draw budget was spent. + public let residualPauses: Int } /// Drives the LuxTTS (ZipVoice-Distill) CoreML stages end-to-end, mirroring @@ -295,7 +307,9 @@ struct LuxTtsSynthesizer { sampleRate: LuxTtsConstants.outputSampleRate, promptFrames: promptFrames, generatedFrames: genFrames, - featuresLength: featuresLength) + featuresLength: featuresLength, + redraws: 0, + residualPauses: 0) } // MARK: - Helpers diff --git a/Sources/FluidAudioCLI/Commands/TTSCommand.swift b/Sources/FluidAudioCLI/Commands/TTSCommand.swift index 8d7a41d2..42ab3383 100644 --- a/Sources/FluidAudioCLI/Commands/TTSCommand.swift +++ b/Sources/FluidAudioCLI/Commands/TTSCommand.swift @@ -106,6 +106,7 @@ public struct TTS { var luxttsPromptText: String? = nil var luxttsSpeed: Float = LuxTtsConstants.defaultSpeed var luxttsSeed: UInt64 = LuxTtsConstants.defaultSeed + var luxttsRedraws: Int = LuxTtsConstants.spuriousPauseRetries var neuttsSeed: UInt64 = 1234 var neuttsEmotion = NeuTtsConstants.defaultEmotion var chatterboxSeed: UInt64 = 42 @@ -235,6 +236,11 @@ public struct TTS { luxttsPromptText = arguments[i + 1] i += 1 } + case "--redraws": + if i + 1 < arguments.count, let v = Int(arguments[i + 1]), v >= 0 { + luxttsRedraws = v + i += 1 + } case "--temperature": if i + 1 < arguments.count, let v = Float(arguments[i + 1]) { pocketTemperature = v @@ -408,7 +414,7 @@ public struct TTS { promptAudioPath: luxttsPromptAudioPath, promptText: luxttsPromptText, treatAsPhonemes: treatAsPhonemes, - speed: luxttsSpeed, seed: luxttsSeed, + speed: luxttsSpeed, seed: luxttsSeed, redraws: luxttsRedraws, metricsPath: metricsPath) case .neuTts: await runNeuTts( @@ -519,7 +525,7 @@ public struct TTS { text: String, output: String, promptAudioPath: String?, promptText: String?, treatAsPhonemes: Bool, - speed: Float, seed: UInt64, + speed: Float, seed: UInt64, redraws: Int, metricsPath: String? ) async { guard let promptAudioPath else { @@ -570,14 +576,16 @@ public struct TTS { promptAudio: promptURL, promptPhonemes: resolvedPromptText, speed: speed, - seed: seed) + seed: seed, + maxRedraws: redraws) } else { result = try await manager.synthesize( text: text, promptAudio: promptURL, promptText: resolvedPromptText, speed: speed, - seed: seed) + seed: seed, + maxRedraws: redraws) } let tSynth1 = Date() @@ -610,6 +618,7 @@ public struct TTS { logger.info( " Frames: prompt=\(result.promptFrames) " + "generated=\(result.generatedFrames) total=\(result.featuresLength)") + logger.info(" Re-draws: \(result.redraws), residual pauses: \(result.residualPauses)") logger.info(" RMS: \(String(format: "%.5f", rms))") logger.info(" RTFx: \(String(format: "%.2f", rtfx))x") logger.info(" Total: \(String(format: "%.3f", totalS))s") @@ -1549,6 +1558,9 @@ public struct TTS { --prompt-text are espeak IPA (en-us) --speed 1.0 speech-rate divisor (default 1.0) --seed N flow-matching noise seed (default 42) + --redraws N re-draw budget per span for the + mid-phrase pause detector (default 3; + 0 = raw pass for the given seed) --lexicon, -l Custom pronunciation lexicon file (KokoroAne --variant zh only): word pinyin1 pinyin2 (e.g. zi4 jie2) word @bopomofo1 (escape: @-prefixed, diff --git a/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsContinuationTests.swift b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsContinuationTests.swift index 8759ec59..85a4639e 100644 --- a/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsContinuationTests.swift +++ b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsContinuationTests.swift @@ -40,11 +40,15 @@ final class LuxTtsContinuationTests: XCTestCase { XCTAssertEqual(chunks.map(\.count), [68, 69, 68]) } - func testMaxSpanTokensIsBoundedByStableRegimeAndFrameBudget() { - // Fast prompt: the 102-token stable-regime cap wins. + func testMaxSpanTokensIsBoundedBySpanSizeTokenBucketAndFrameBudget() { + // Fast prompt: the 102-token span size wins. XCTAssertEqual( LuxTtsContinuation.maxSpanTokens(promptFrames: 300, promptTokenCount: 100, speed: 1.0), 102) + // Long transcript: the 256-token encoder bucket (minus pad slot) wins. + XCTAssertEqual( + LuxTtsContinuation.maxSpanTokens(promptFrames: 300, promptTokenCount: 200, speed: 1.0), + 55) // Issue #937 prompt (100 tokens / 5 s): 468 frames ÷ 4.69 frames/token. XCTAssertEqual( LuxTtsContinuation.maxSpanTokens(promptFrames: 469, promptTokenCount: 100, speed: 1.0), @@ -57,6 +61,11 @@ final class LuxTtsContinuationTests: XCTestCase { XCTAssertEqual( LuxTtsContinuation.maxSpanTokens(promptFrames: 0, promptTokenCount: 0, speed: 1.0), 102) + // Pathological ratios floor at 1; the manager rejects anything under + // `minimumSpanTokens` instead of rendering dozens of passes. + XCTAssertEqual( + LuxTtsContinuation.maxSpanTokens(promptFrames: 469, promptTokenCount: 4, speed: 1.0), + 3) } func testSpanFrameBudgetFitsInsidePromptCap() { @@ -71,18 +80,36 @@ final class LuxTtsContinuationTests: XCTestCase { XCTAssertLessThanOrEqual(budget, LuxTtsConstants.vocoderBuckets.max() ?? 0) } - func testFitsSinglePassUsesIssue937PromptGeometry() { + func testFitsSinglePassUsesSpanCapAndGraphLimits() { // Reporter's prompt: 100 tokens over 5 s (469 frames), speed 1.0. XCTAssertTrue( LuxTtsContinuation.fitsSinglePass( textTokenCount: 102, promptFrames: 469, promptTokenCount: 100, speed: 1.0)) + // 106 tokens would fit the graph (498 frames) but exceed the span cap. XCTAssertFalse( LuxTtsContinuation.fitsSinglePass( textTokenCount: 106, promptFrames: 469, promptTokenCount: 100, speed: 1.0)) + // 102 tokens at a slow prompt → 613 generated frames, past the 555 bucket. + XCTAssertFalse( + LuxTtsContinuation.fitsSinglePass( + textTokenCount: 102, promptFrames: 600, promptTokenCount: 100, speed: 1.0)) // Half speed doubles the frame estimate past the 555 bucket. XCTAssertFalse( LuxTtsContinuation.fitsSinglePass( textTokenCount: 102, promptFrames: 469, promptTokenCount: 100, speed: 0.5)) + // Prompt transcript + text must leave the pad slot in the 256 bucket. + XCTAssertFalse( + LuxTtsContinuation.fitsSinglePass( + textTokenCount: 60, promptFrames: 300, promptTokenCount: 196, speed: 1.0)) + XCTAssertTrue( + LuxTtsContinuation.fitsSinglePass( + textTokenCount: 59, promptFrames: 300, promptTokenCount: 196, speed: 1.0)) + } + + func testTextPauseAllowanceCountsSilentBreaksOnly() { + XCTAssertEqual(LuxTtsContinuation.textPauseAllowance("He paused... then said \"no\"."), 3) + XCTAssertEqual(LuxTtsContinuation.textPauseAllowance("Wait… (really)"), 3) + XCTAssertEqual(LuxTtsContinuation.textPauseAllowance("don't, won't; can't."), 0) } func testExpectedPauseCountIgnoresTrailingBoundaryTokens() { diff --git a/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsE2ETests.swift b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsE2ETests.swift index 5464e113..879a3675 100644 --- a/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsE2ETests.swift +++ b/Tests/FluidAudioTests/TTS/LuxTts/LuxTtsE2ETests.swift @@ -46,7 +46,10 @@ final class LuxTtsE2ETests: XCTestCase { promptAudio: promptURL, promptPhonemes: fixtures.prompt.phonemeString, speed: Float(fixtures.e2e.speed), - seed: UInt64(fixtures.e2e.seed)) + seed: UInt64(fixtures.e2e.seed), + // Pin the raw pass: the pause detector would otherwise re-draw + // this fixture (one breath beyond its two commas) with another seed. + maxRedraws: 0) // Frame accounting must be identical to Python (pure integer math // over fixture-pinned token ids and mel frame counts). @@ -55,6 +58,7 @@ final class LuxTtsE2ETests: XCTestCase { XCTAssertEqual(result.featuresLength, fixtures.e2e.featuresLen) XCTAssertEqual(result.generatedFrames, fixtures.e2e.genFrames) XCTAssertEqual(result.samples.count, fixtures.e2e.wavSamples) + XCTAssertEqual(result.redraws, 0) // Loudness within 1 dB of the Python CoreML pipeline, and non-silent. let sumSquares = result.samples.reduce(Double(0)) { $0 + Double($1) * Double($1) }