feat(tn/en/serial): alphanumeric serial codes (8→27) - #62
Merged
Merged
Conversation
Add a `serial` tagger (sentence-mode priority 65, below cardinal, above word)
for mixed letter/digit/symbol codes:
- "B2A23C" → "B two A twenty three C"
- "133-ABC" → "one hundred thirty three-ABC"
- "$12@12%" → "dollar twelve at twelve percent"
Letter runs are kept; digit runs read as cardinals (digit-by-digit with a
leading zero or beyond four digits); "-" and "/" are kept literal and glue,
other symbols spell out. Requires a digit (pure-symbol tokens stay with the
`word` tagger) and skips ARPABET phonemes ("AH0").
Also tighten telephone vanity detection to 3+ groups with an upper-case
mnemonic so short serials ("133-ABC") no longer take the phone path.
en TN serial 8→27; normalize_with_audio 31→39, punctuation 34→35 as side wins.
The five unreproduced cases are NeMo's irregular forms (hyphen spaced/dropped
in "1-8090"/"7-eleven", "1/f"→"one per F", 4-digit "9453" digit-by-digit). No
regressions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
serialtagger (sentence-mode priority 65, below cardinal, above word) for mixed letter/digit/symbol codes:B2A23C→ "B two A twenty three C"133-ABC→ "one hundred thirty three-ABC"$12@12%→ "dollar twelve at twelve percent"t-0t25d12-f→ "t-zero t twenty five d twelve-f"Letter runs are kept; digit runs read as cardinals (digit-by-digit with a leading zero or beyond four digits);
-and/are kept literal and glue their neighbours, other symbols spell out. Requires a digit (pure-symbol tokens stay with thewordtagger, which reads/as "slash"), and skips ARPABET phonemes (AH0).Also tightens the telephone vanity detection (from #61) to 3+ groups with an upper-case mnemonic, so short mixed serials (
133-ABC) and lower-case codes (1-413-te-b) no longer take the phone path.en TN serial 8→27; bonus side wins normalize_with_audio 31→39, punctuation 34→35. No regressions.
The five unreproduced cases are NeMo's internally irregular forms (hyphen spaced in
1-8090, dropped in7-eleven;1/f→"one per F"; 4-digit9453digit-by-digit while2000is cardinal) — documented in the module header.cargo test/fmt/clippy green; baseline regenerated (only increases).🤖 Generated with Claude Code