Parakeet TDT v3: spontaneous Spanish decoded through an English lexicon (upstream model behaviour, not the CoreML port)

[anibal]

Important

Correction (2026-08-09) — this report was wrong on two counts, and is now closed.
The failure is not specific to the CoreML port (upstream nvidia/parakeet-tdt-0.6b-v3 in
fp32 fails worse on the same clips), and it is not translation (English tokens are
frame-aligned 1:1 onto the Spanish words they replace, with Spanish word order preserved).
It is upstream model behaviour: Parakeet TDT v3 has no language conditioning and falls back to
its English prior on spontaneous non-English speech, driven by conversational speech rate.
See the follow-up comment
for the full evidence. The original report is preserved unedited below. Apologies for the
noise.

---

Summary

Running conversational Latin-American Spanish audio through the Parakeet TDT v3 CoreML backend produces output that is substantially translated into English (mixed with Spanish fragments), rather than transcribed. On our test segments, 12.8–24.4% of output words are English function words and ~20% of source words are dropped. The Nemotron 3.5 multilingual CoreML backend, via the same SDK on the same audio, transcribes the Spanish correctly.

Reproduced character-for-character with FluidAudio's own fluidaudiocli at tag v0.15.5, default settings, on a 14 s clip that fits a single 15 s model window — so long-form chunking is not involved.

Environment

• macOS 26.5.2 (25F84), Apple M1 Pro
• FluidAudio v0.15.5 (clean checkout, swift build -c release --product fluidaudiocli)
• Model cache freshly re-downloaded to rule out stale assets

Reproduction

$ ./.build/release/fluidaudiocli transcribe modelos-14s.wav

modelos-14s.wav: 14 s of conversational Latin-American Spanish (podcast speech), 16 kHz mono PCM, −19 dBFS RMS.

Observed (Parakeet TDT v3, 14 s clip, single window)

When one is models of the great, in my circle extended amigos and family, sativa is a very important idea.

Observed (same episode, 60 s clip)

When one is models of the people automatically, the great circle extended and family think in model chino. Saptiva is a un modelo ajustado a la cultura y a las leyes mexicanas andamos cal, KAL. Recuerdo una discusión in LinkedIn when alguien decía, but it's not sober, trayendo un modello chino for Mexico and yo le decía, Carlos, Carlos, vale la pena responderle, but I decía no, but if you want to respond, we're going to ensech that Cal is based on a model franc, Mistral. […]

Expected (same 60 s audio, Nemotron 3.5 multilingual CoreML via the same SDK)

cuando uno dice modelos de pesos abiertos la gente automáticamente la gran mayoría al menos en mi círculo extendido de amigos y familiares piensan ese modelos chinos de hecho Saptiva entre un modelo ajustado a la cultura y a las leyes mexicanas y le llama cal KAL y recuerdo una discusión en LinkedIn cuando alguien decía pero esto no es soberanía si estamos trayendo un modelo chino para México […]

Ruled out

• Long-form chunking (14 s clip fits one window; same behavior)
• Audio level (−19 dBFS RMS) and format (16 kHz mono PCM)
• Domain jargon (failure includes everyday words: "amigos and family", "circle extended")
• Stale model cache (fresh re-download)
• Our own harness (reproduces with fluidaudiocli at the same tag, identical output)

Since upstream nvidia/parakeet-tdt-0.6b-v3 benchmarks Spanish at ~3.5% WER (FLEURS), this looks specific to the CoreML conversion or the decoding path rather than the base model. Happy to share the WAV clips and full side-by-side transcripts if useful — we have 10-minute comparisons of Parakeet vs Nemotron vs Apple SpeechTranscriber on the same audio.

[anibal]

Correction: this is upstream model behaviour, not a FluidAudio bug — and it isn't translation

We went back and tried to break our own report. It broke. Two of the three claims in the original
issue are wrong, and I'd rather correct them here than leave them sitting in your tracker. Sorry
for the noise.

1. It is not the CoreML port

We ran the same audio through upstream nvidia/parakeet-tdt-0.6b-v3 in fp32 via
parakeet-mlx (mlx-community/parakeet-tdt-0.6b-v3, model.safetensors 2.3 GB, no
quantization), on the same machine (M1 Pro, macOS 26.5.2).

clip	FluidAudio CoreML (6-bit encoder, no hint)	upstream fp32 MLX
14 s reproducer	"When one is models of the great, in my circle extended amigos and family…"	"When one says models of the people automatically, the great circle extended and familiar, think in models chino."
60 s	"When one is modelled of the people automatically… pero esto no es soberanía si estamos trayendo un modelo chino para México y demás…"	"When one says model of the people automatically… it's sovereign, we are a model chin for Mexico and I said, Carlos, Carlos…"
30 s, different episode/speakers/mic	"But there are unit de identidad and no anclarse a tu rol y a tu cajita … deberían empezar a resolver ellos problemas que esas otras empresas no están resolviendo…" — largely usable Spanish	"But I think there's a thing of identity and your role and your cajita and the position…" — almost none
clean TTS Spanish	correct Spanish, conf 0.999	correct Spanish, character-identical

Upstream fp32 fails the same way — and measurably worse. Same clips both sides, no hint,
counting English-exclusive function words as a share of unambiguous EN+ES function words:

clip	FluidAudio CoreML (conf)	CoreML EN%	upstream fp32 EN%
14 s reproducer	0.687	100.0	100.0
60 s	0.811	53.2	97.7
30 s, elsewhere in ep. 1	0.830	17.9	95.2
30 s, different episode	0.886	21.6	100.0
clean TTS Spanish	0.999	0.0	0.0

So the 6-bit palettized encoder is not implicated, and on three of four real clips your decode path
returns substantially more correct Spanish than the unquantized original does. Our original
title pointed at the wrong component. Apologies.

Two smaller notes that may be useful to you:

• Confidence may be a usable detector for this. Across those five clips, aggregate confidence
  tracks English intrusion at r = −0.92, and a threshold near 0.9 separates the failing from
  the correct clips. Small sample (n = 5, only two correct clips, one of them TTS) — suggestive,
  not calibrated, and we would not ship a guard on it without a proper corpus.

2. It is not translation — it's decoding Spanish through an English lexicon

This matters because "translation" suggests a semantic path that isn't there. From the word
timestamps your API already returns:

t (s)	emitted	actually spoken
0.16	When	cuando
0.64	is	dice
1.04	modelled	modelos
6.32	circle	círculo
7.28	extended	extendido
10.16	model	modelo
10.48	chino.	chinos

Every token is frame-aligned 1:1 to the Spanish word it replaced; Spanish word order survives
(circle extended, model chino, and elsewhere intelligence artificial); non-words appear
(modelled, recuered); and Spanish words are left untranslated mid-sentence. A translation layer
would produce none of those. It's acoustic decoding under an English lexical prior.

3. What actually triggers it: spontaneous-speech acoustics

Tightest control we ran, all on upstream fp32:

• macOS say -v Paulina (es_MX, same LatAm dialect) reading the exact sentence the model
  fails on — including "Saptiva", "Cal", "modelos de pesos abiertos" — transcribes
  character-perfect Spanish.
• The same words spoken spontaneously on a podcast mic collapse to ~100% English.

So it is not the dialect, and it is not the anglicisms/code-switching. It's the acoustics of
spontaneous speech — and more specifically, speech rate. Time-stretching the same reproducer
clip (ffmpeg -filter:a atempo=…, upstream fp32, nothing else changed):

tempo	English %	output
0.60×	11	"When one dice model of pesos abiertos, la gente automáticamente, la gran mayoría, al menos en mi círculo extendido de amigos… "
0.75×	62	"When one dice model of the gente automatically, the great, almost in my circular extended amigos and familiares…"
1.00×	100	"When one says models of the people automatically, the great circle extended and familiar…"
1.25×	100	"When one says of the people automatically, the great extended people and family…"

Monotonic, and it recovers most of the Spanish at 0.60× on a recording that is 100% English at
1.00×. Time-stretch artifacts should hurt recognition, so the direction of that effect seems
safe. This looks like a decode collapse driven by conversational speech rate / coarticulation
rather than by anything about the language or the channel.

Two further probes:

• Loudness normalization (loudnorm=I=-16:TP=-1.5) changes nothing — not a gain problem.
• Prepending 6 s of clean Spanish does not rescue the hard audio; the hard audio corrupts the
  clean lead-in. Two controls: that 6 s cut transcribes correctly alone (so it isn't a
  duration or seam artifact), and reversing the order — hard audio first, clean Spanish second
  — collapses the clean segment just the same. So the contamination is position-independent and
  bidirectional, which looks like a window-global language state rather than left-to-right drift.
  Practical consequences: there is no "prime it with Spanish" workaround, and
  shorter/utterance-bounded windows should make this more likely, not less.

4. On languageHint — we were wrong twice, in opposite directions

We first implied the hint was being ignored. It isn't: TokenLanguageFilter partitions by Unicode
script, which for Spanish-vs-English (both Latin) genuinely is a no-op — as designed, since it was
built for #512's Cyrillic case — while applyEnglishBlocklist does fire.

We then briefly read our own numbers as the blocklist helping Spanish (38.5% → 28.9% English
tokens). We withdraw that. Our metric counts English tokens, and the blocklist's job is to delete
English tokens, so the metric is circular with respect to the mitigation — it can't tell recovered
Spanish from destroyed genuinely-English words, and this audio is full of real English terms
(LinkedIn, Mistral, benchmarks, GPT). #840's measurement is the one to trust: Spanish code-switch
median +15pp worse under --language es. Our own spot-check agrees in direction — the big English
terms survived, but the proper nouns Cal/KAL were lost under the hint. We are not recommending
--language es as a workaround, and would gently support #840's proposal to gate the blocklist to
French.

Credit where it's due — your comment on that function already had a better diagnosis than our issue
did:

"It targets the spontaneous-speech translation phenomenon where the model falls back to its
English prior on acoustically ambiguous frames."

That matches everything we measured. One datum that may still be useful for #840's discussion: we
decoded the residual English function words in our --language es run against englishBlocklistIds:

residual token	count	vocab id	in blocklist
of	29	592	no
is	16	508	no
to	11	366	no
in	9	334	no
it	9	869	yes — leaked anyway
those	6	—	no
for	4	509	no

86% of the residue is tokens the blocklist never covered; only 14% leaked despite being listed.
And the gap looks structural rather than accidental: the uncovered tokens are exactly the short,
low-id, high-frequency ones ( of 592, is 508, to 366, in 334, for 509) that also occur
as word-initial fragments of legitimate Spanish words — blocking in would break
internacional, to would break todo, is would break isla. So a space-prefixed whole-token
blocklist can't safely be widened to cover them, which caps how far this mitigation can go in any
Latin-script language — the same structural point #840 makes from the damage side.

5. This was not new, and we should have found that first

The general phenomenon was already described in your codebase — the applyEnglishBlocklist comment
quoted above — and in PR #630, which states it plainly: "Parakeet TDT v3 has no language
conditioning token. On spontaneous non-English speech, the model falls back to its English training
prior… This is a known architectural limitation." We should have read the decode path before
opening #842.

Two other prior reports we should have found, both of which corroborate the read-vs-spontaneous
axis better than our original issue did:

• PR fix(asr): reduce English drift on French recordings via token blocklist #630's own French benchmark is the cleanest published version of this: 0–7% English drift
  on read/archival recordings vs 31.3% on a spontaneous 57-minute interview. That is exactly
  the axis we ended up demonstrating in Spanish.
• Short utterances in Latin-script languages transcribed as Cyrillic [Parakeet TDT v3] #512 contains the same TTS-vs-real experiment we ran, in Polish — say -v Zosia synthetic
  Polish transcribes correctly while real human recordings of the same phrase come out as
  Fpisch Google CrockerCon.

For the record on one thing we don't claim: we looked at #594 and it appears to be a distinct
bug — a chunk-boundary mel-context regression from #264, closed by #596. Ours reproduces inside a
single 14 s window with melChunkContext off, and upstream in MLX where the flag doesn't exist. So
we're not proposing this as a duplicate of #594.

6. Suggested disposition

Worth adding, in fairness to the model: v3's published 3.45% Spanish WER is FLEURS es_419 —
which is read speech, and (per the FLEURS paper) Latin-American Spanish. So NVIDIA's Spanish
benchmark already is our dialect; it just isn't our speech type. Every clip that succeeded in our
testing was read speech. The model may be behaving exactly as documented, and the error may have
been our assumption that a FLEURS number transfers to spontaneous podcast audio.

We're closing this ourselves as upstream / not-a-FluidAudio-bug, and retitling it so it stops
misleading anyone who finds it by search. The one thing we'd still genuinely ask for is a
documentation note: a line in the Parakeet v3 docs saying the model
is unreliable for spontaneous non-English speech, that its published non-English WERs are
read-speech benchmarks, and that no inference-time language control exists for it — NVIDIA has
confirmed this twice (nithinraok, NeMo #14799:
"parakeet-v3 model doesn;t recieve or output language id"; and on HF, language prompting is
"Only with canary model"). Contrast Nemotron 3.5 in your own SDK, which has real language
conditioning and handled all of this audio correctly with detectedLanguage = es-ES. That
distinction would have saved us this investigation, and it's the one thing we'd genuinely ask for —
especially with Canary landing in #833.

If any of it is useful to you, the things we have that don't seem to be on file anywhere yet:

• the fp32-vs-CoreML comparison above (your port is the less-affected one);
• a speech-rate dose-response on a single clip — 0.60× recovers most of the Spanish, 1.25×
  stays pinned at English — which is the only manipulable demonstration of the driver we found in
  any language;
• an es_MX TTS control reading the exact failing sentence, which separates acoustics from
  dialect and vocabulary;
• confidence as a candidate detector (r = −0.92 against English intrusion, though only n = 5).

Happy to attach the run artifacts or the control clips. Thanks for a well-instrumented codebase —
the word timestamps and the top-K plumbing are what made this diagnosable at all, and #630's
French benchmark was the thing that told us we were looking at a known limitation rather than a
port bug.

[anibal]

Closing as upstream model behaviour rather than a FluidAudio defect — details and evidence in the correction above. Thanks for the well-instrumented codebase; the word timestamps are what made this diagnosable.