Skip to content

Surface CTC vocabulary rescoring decisions on the public ASR results - #929

Open
kevin-nous wants to merge 1 commit into
FluidInference:mainfrom
kevin-nous:surface-vocabulary-rescoring-results
Open

kevin-nous wants to merge 1 commit into
FluidInference:mainfrom
kevin-nous:surface-vocabulary-rescoring-results

Conversation

@kevin-nous

Copy link
Copy Markdown

Problem

With vocabulary boosting configured, VocabularyRescorer already builds a
RescoringResult for every word it decides on — the original word, the
original and replacement scores, and the reason. Nothing reaches the caller.
The public results expose ctcDetectedTerms and ctcAppliedTerms, so an app
can see which vocabulary terms went in, but not which decoded word each one
displaced
or the scores behind the decision.

That makes two things impossible downstream:

  • Telling a recognizer error apart from a bad vocabulary replacement when a
    transcript reads wrong.
  • Showing a user what boosting actually changed ("codecs" → "Codex").

The decisions are dropped at the call sites: SlidingWindowAsrManager flattens
them to [String] via compactMap { $0.replacementWord }, and everything else
keeps .text.

Change

  • ASRResult gains ctcReplacements: [VocabularyRescorer.RescoringResult]?,
    aligned 1:1 with ctcAppliedTerms.
  • SlidingWindowTranscriptionUpdate gains the same field, so the streaming path
    exposes what the batch path does.
  • withRescoring(text:detected:applied:) gains a defaulted replacements:
    parameter rather than a parallel helper.
  • VocabularyRescorer.RescoringResult gains Codable (all stored properties
    are String/Float/Bool), because ASRResult is Codable.
  • TextNormalizer.normalize(result:) carries the new field through.

The Unified path needed its own accessor: UnifiedAsrManager.transcribe(_:)
returns a String and transcribeWithTimings(_:) returns text plus timings, so
neither has anywhere to put the CTC metadata — a field on ASRResult alone
would be unreachable for callers of the offline manager. So:

  • UnifiedAsrManager.transcribeDetailed(_:) returns the whole ASRResult
    text, confidence, duration, processing time, token timings, and all three CTC
    fields. Naming follows the existing synthesizeDetailed pairs in
    KokoroAneManager / PocketTtsManager.
  • StreamingUnifiedAsrManager.consumeVocabularyReplacements() drains the
    decisions applied since the previous call, mirroring the existing
    consumeTokenTimings() / consumeWordTimings() accessors; its finish()
    returns a String and cannot carry them.
  • rescoreIfConfigured now returns the rescorer's RescoreOutput instead of
    discarding everything but .text.

Compatibility

Additive and source-compatible. Both new fields default to nil and are the
last parameter of their memberwise init; the new withRescoring parameter is
defaulted, so existing three-argument calls compile unchanged. transcribe(_:),
transcribe(_:AVAudioPCMBuffer) and transcribeWithTimings(_:) keep their
signatures and their behaviour, including not building token timings when
boosting is not configured. ctcDetectedTerms and ctcAppliedTerms are
untouched.

Testing

Tests/FluidAudioTests/ASR/Parakeet/VocabularyRescoringSurfacingTests.swift
covers the batch path (withRescoring carrying the decisions, the defaulted
call leaving them nil, a Codable round trip, and the pure
UnifiedAsrManager.applying(_:to:) / meanConfidence(of:isEmpty:) helpers
transcribeDetailed is built from) and the streaming path
(SlidingWindowTranscriptionUpdate carrying and defaulting them, and the
StreamingUnifiedAsrManager drain). The rescoring mapping is factored into
pure statics for the same reason tokenTimings already is: it keeps the rule
testable without loading a 600M parameter model or mocking one.

swift test: 2492 tests, 57 skipped, 1 failure. The failure is
LuxTtsG2pTests.testFixtureResourcesAreProcessedAtBundleRoot, which fails the
same way on an unmodified main checkout here and is unrelated to this change.
swift format lint reports nothing new on the touched files.

The rescorer already records a RescoringResult per decision — the original
word, both scores, and the reason — but every call site kept only `.text`.
Callers could see which vocabulary terms were applied (`ctcAppliedTerms`)
but not which decoded word each one displaced, or on what evidence. That
makes it impossible to tell a recognizer error from a bad vocabulary
replacement, or to show a user what boosting changed.

Add `ctcReplacements` to `ASRResult` and `SlidingWindowTranscriptionUpdate`,
aligned 1:1 with `ctcAppliedTerms`, and populate it from the sliding-window
path. `RescoringResult` gains `Codable` since `ASRResult` is `Codable`.

The Unified managers had nowhere to put it: `transcribe` returns a `String`
and `finish` returns a `String`, so the field would be unreachable for their
callers. Add `UnifiedAsrManager.transcribeDetailed`, which returns the whole
`ASRResult`, and `StreamingUnifiedAsrManager.consumeVocabularyReplacements`,
which drains the decisions the way `consumeTokenTimings` drains timings.

Additive: the new fields default to nil and sit last in their memberwise
init, `withRescoring` gains a defaulted parameter rather than a new overload,
and the existing transcribe entry points keep their signatures and behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant