Surface CTC vocabulary rescoring decisions on the public ASR results - #929
Open
kevin-nous wants to merge 1 commit into
Open
kevin-nous wants to merge 1 commit into
kevin-nous wants to merge 1 commit into
Conversation
The rescorer already records a RescoringResult per decision — the original word, both scores, and the reason — but every call site kept only `.text`. Callers could see which vocabulary terms were applied (`ctcAppliedTerms`) but not which decoded word each one displaced, or on what evidence. That makes it impossible to tell a recognizer error from a bad vocabulary replacement, or to show a user what boosting changed. Add `ctcReplacements` to `ASRResult` and `SlidingWindowTranscriptionUpdate`, aligned 1:1 with `ctcAppliedTerms`, and populate it from the sliding-window path. `RescoringResult` gains `Codable` since `ASRResult` is `Codable`. The Unified managers had nowhere to put it: `transcribe` returns a `String` and `finish` returns a `String`, so the field would be unreachable for their callers. Add `UnifiedAsrManager.transcribeDetailed`, which returns the whole `ASRResult`, and `StreamingUnifiedAsrManager.consumeVocabularyReplacements`, which drains the decisions the way `consumeTokenTimings` drains timings. Additive: the new fields default to nil and sit last in their memberwise init, `withRescoring` gains a defaulted parameter rather than a new overload, and the existing transcribe entry points keep their signatures and behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With vocabulary boosting configured,
VocabularyRescoreralready builds aRescoringResultfor every word it decides on — the original word, theoriginal and replacement scores, and the reason. Nothing reaches the caller.
The public results expose
ctcDetectedTermsandctcAppliedTerms, so an appcan see which vocabulary terms went in, but not which decoded word each one
displaced or the scores behind the decision.
That makes two things impossible downstream:
transcript reads wrong.
The decisions are dropped at the call sites:
SlidingWindowAsrManagerflattensthem to
[String]viacompactMap { $0.replacementWord }, and everything elsekeeps
.text.Change
ASRResultgainsctcReplacements: [VocabularyRescorer.RescoringResult]?,aligned 1:1 with
ctcAppliedTerms.SlidingWindowTranscriptionUpdategains the same field, so the streaming pathexposes what the batch path does.
withRescoring(text:detected:applied:)gains a defaultedreplacements:parameter rather than a parallel helper.
VocabularyRescorer.RescoringResultgainsCodable(all stored propertiesare
String/Float/Bool), becauseASRResultisCodable.TextNormalizer.normalize(result:)carries the new field through.The Unified path needed its own accessor:
UnifiedAsrManager.transcribe(_:)returns a
StringandtranscribeWithTimings(_:)returns text plus timings, soneither has anywhere to put the CTC metadata — a field on
ASRResultalonewould be unreachable for callers of the offline manager. So:
UnifiedAsrManager.transcribeDetailed(_:)returns the wholeASRResult—text, confidence, duration, processing time, token timings, and all three CTC
fields. Naming follows the existing
synthesizeDetailedpairs inKokoroAneManager/PocketTtsManager.StreamingUnifiedAsrManager.consumeVocabularyReplacements()drains thedecisions applied since the previous call, mirroring the existing
consumeTokenTimings()/consumeWordTimings()accessors; itsfinish()returns a
Stringand cannot carry them.rescoreIfConfigurednow returns the rescorer'sRescoreOutputinstead ofdiscarding everything but
.text.Compatibility
Additive and source-compatible. Both new fields default to
niland are thelast parameter of their memberwise init; the new
withRescoringparameter isdefaulted, so existing three-argument calls compile unchanged.
transcribe(_:),transcribe(_:AVAudioPCMBuffer)andtranscribeWithTimings(_:)keep theirsignatures and their behaviour, including not building token timings when
boosting is not configured.
ctcDetectedTermsandctcAppliedTermsareuntouched.
Testing
Tests/FluidAudioTests/ASR/Parakeet/VocabularyRescoringSurfacingTests.swiftcovers the batch path (
withRescoringcarrying the decisions, the defaultedcall leaving them
nil, aCodableround trip, and the pureUnifiedAsrManager.applying(_:to:)/meanConfidence(of:isEmpty:)helperstranscribeDetailedis built from) and the streaming path(
SlidingWindowTranscriptionUpdatecarrying and defaulting them, and theStreamingUnifiedAsrManagerdrain). The rescoring mapping is factored intopure statics for the same reason
tokenTimingsalready is: it keeps the ruletestable without loading a 600M parameter model or mocking one.
swift test: 2492 tests, 57 skipped, 1 failure. The failure isLuxTtsG2pTests.testFixtureResourcesAreProcessedAtBundleRoot, which fails thesame way on an unmodified
maincheckout here and is unrelated to this change.swift format lintreports nothing new on the touched files.