Release 0.0.4a8 - #79
Open
github-actions[bot] wants to merge 27 commits into
Open
Conversation
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Updated descriptions for arbtok and espyak, and removed some entries.
* feat: add East-Slavic stress and European-Portuguese sense diacritics backends Extends add_diacritics with two pronunciation-disambiguation backends in the same category as Hebrew niqqud and Arabic tashkeel: - ru/uk/be word stress via stressonnx: East-Slavic stress is free and mobile and not written in ordinary orthography, and unstressed vowels reduce (Russian о→[ɐ]/[ə]), so a wrong or missing stress mark corrupts vowel quality across the whole word, not just prosody. Routes on exact primary subtag equality (not prefix matching) so Berber (ber) can't false-match Belarusian (be). - pt/pt-PT heterophonic-homograph sense diacritics via bifonia: rewrites homographs like sede (thirst → closed sêde, seat → open séde) with a non-canonical open/closed-vowel mark a rule-based Portuguese G2P reads. Scoped to European Portuguese only; pt-BR is excluded because Brazilian Portuguese's vowel system differs. Both backends are lazily imported and raise a named ImportError with a pip install hint when their optional dependency is missing, matching the existing phonikud/tashkeel pattern. Adds the `stress` and `pt` extras; stressonnx is not yet published to PyPI so it is not added to the `test` extra, and stress routing is covered by a stub rather than the real package. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: clarify bifonia diacritics are ordinary orthography, not rule-based-only The European-Portuguese sense diacritics from bifonia are standard Portuguese orthographic marks (open/closed vowel), not a rule-based-G2P convention — they are chosen to be read correctly by any downstream G2P, rule-based, neural, or espeak alike. Reword the add_diacritics docstring, _sense_diacritics_pt docstring, and docs/phonemizers.md accordingly; the pt-BR exclusion and its European/Brazilian vowel-system rationale are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: expand stress routing to all 26 stressonnx language tags stressonnx supports word-stress restoration for East Slavic (ru, uk, be), Bulgarian/Macedonian/Slovene (bg, mk, sl), Latvian (lv), Armenian (hy), Georgian (ka), and a set of Turkic/Caucasian languages (az, ba, cv, kbd, kjh, kk, ky, mdf, myv, sah, tg, tt, udm, uz, xal) — 24 primary subtags covering 26 BCP-47 tags once Azerbaijani and Uzbek's Cyrillic/Latin script variants are counted. STRESS_LANGS now covers the full set instead of just ru/uk/be; routing still keys on the primary subtag while the full tag (including script variant, e.g. az-Latn) is forwarded to stressonnx so it can pick the right script. Stress is free and unwritten or under-marked across this whole language set. East Slavic remains the sharpest example because its stress is also mobile and unstressed vowels reduce, corrupting more than prosody when the mark is wrong or missing; the other families need the mark for correct stress placement even without reduction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * build: pin stress extra to published stressonnx floor stressonnx is now on PyPI at 0.0.2; pin the [stress] extra to that floor instead of an unbounded stressonnx dependency. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…rs (#70) * feat: model diacritization as a graph extension parallel to phonemizers Add scriptconv.diacritics.register, an opt-in Edge("text", "text-diacritized", ...) that wraps BasePhonemizer.add_diacritics (Arabic tashkeel, Hebrew niqqud, East-Slavic/Turkic/Caucasian stress, European-Portuguese sense marks) as a first-class graph transform. phonemizers.register now also consumes the diacritized node via a "text-diacritized" -> "ipa" edge, sharing the same phonemization function. Both edges are model-based (lossless=False), so routing "text" -> "ipa" keeps taking the direct edge by default — enabling either extension never changes phonemization output, and DEFAULT_GRAPH stays untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: add gated strip edge for overlay diacritics Register a text-diacritized -> text edge (strip_diacritics) alongside the existing restore edge in diacritics.register. Stripping every combining mark is a lossless inverse only for overlay diacritics — Arabic/Hebrew vocalization and East-Slavic/Turkic/Caucasian stress — whose bare canonical form carries no marks. Languages whose diacritics are native orthography (European Portuguese via bifonia) raise ValueError instead of silently corrupting the spelling. The strip edge is lossless and cheap while restore is model-based and expensive, so routing text -> ipa still prefers not to diacritize unless asked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: strip only per-backend overlay codepoints, not all combining marks strip_diacritics filtered every NFD combining mark, which decomposed and destroyed precomposed native letters (Cyrillic й/ё, Latvian macrons, Azerbaijani ç/ö, Arabic hamza carriers) in languages it claims to strip losslessly. It now removes only the specific overlay codepoints each backend adds: combining acute/grave for stress, tashkeel for Arabic, niqqud for Hebrew. _supports_strip and add_diacritics used lang.startswith("he"/"ar"), which false-matched Aragonese (arg), Herero (her), and Mapudungun (arn). Both now gate on exact primary-subtag match. diacritize() now reuses a cached default GraphemePhonemizer instead of constructing one per call, so the graph edge doesn't reload the ONNX model on every conversion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…cs tests (#72) * fix: lazy-import requests in DeepPhonemizer, friendly ImportErrors for he/fa requests was imported at module scope in en.py despite being a network dep used only inside DeepPhonemizer; move it (and os, which is likewise only used there) into the method bodies that need them, and add requests to the en-phonemizers extra. he.py/fa.py's lazy imports of phonikud/persian_phonemizer had no error handling; wrap them like the sibling backends (ar/gl/eu/pt/mwl) with a friendly ImportError naming the pip extra. Also export mantoq_to_ipa from notation.__all__ and scriptconv's top-level re-exports, matching the other *_to_ipa functions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: cover diacritics strip/overlay helpers strip_diacritics/_overlay_marks/_supports_strip are pure-python and need no optional backend, but had no direct test coverage. Assert overlay-only stripping (combining stress marks, tashkeel, niqqud) preserves precomposed native letters (Cyrillic й/ё, Arabic hamza carriers), raises ValueError for native-orthography langs (pt/en/es/ arg/her/arn), and that the exact-subtag gate never misreads Aragonese/ Herero/Mapudungun as Arabic/Hebrew. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: harden zh pinyin backends and dedupe notation/registry logic - fold full-width digits to ASCII before zh_num lookup, avoiding a KeyError crash on non-ASCII digit tokens - replace a bare assert in pinyin _retone with a graceful fallback for unanticipated combining marks - wrap the four pinyin backend imports in friendly ImportErrors naming the correct install extra, matching readings.py's pattern - reuse base._primary_subtag in registry.phonemizer_for_lang instead of reimplementing the primary-subtag rule inline - run graph.route once in notation.can_convert instead of twice Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * perf: avoid quadratic joins and repeated table parsing - accumulate unmapped runs in to_cangjie via a list buffer instead of repeated string concatenation (O(n) instead of O(n^2)) - memoize the parsed kog2p rulebook so runKoG2P no longer re-reads and re-parses rulebook.txt on every call - skip re-parsing the hangul2ipa CSV tables when the same tables_dir has already been initialized Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: annotate a few load-bearing invariants - explain why _STRESS_MARKS covers combining grave alongside acute - note that any BasePhonemizer subclass works as add_diacritics's vessel, and why GraphemePhonemizer was picked - document that graph.Edge.fn must accept **kwargs for context to pass through opaquely - document the _LOSSY_COST tie caveat and why shallow graphs make it moot - document the non-overlapping-ranges invariant char_script's binary search relies on, plus a regression test guarding it Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: default CotoviaPhonemizer to the Cotovia phoneset CotoviaPhonemizer only emits Cotovia notation in default registry routing (registry._EMITS gates it to Alphabet.COTOVIA), so defaulting the class to Alphabet.IPA was inconsistent — a bare CotoviaPhonemizer() advertised an IPA default it is never selected to provide. Default to Alphabet.COTOVIA, the binary-parity-tested native phoneset; IPA remains available on explicit request. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: single source of truth for diacritizer lang routing base.add_diacritics and diacritics._overlay_marks each hand-coded the he/ar/stress/pt lang dispatch, so adding a language to one and forgetting the other would silently make strip disagree with diacritize. Both now resolve through one _diacritizer_family(lang) helper, making that divergence structurally impossible; the strip side treats only overlay families (he/ar/stress) as removable and pt (native orthography) as not. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make hangul2ipa table initialisation thread-safe hangul2ipa() calls initialize_conversion_tables() on every invocation, and the rule functions read the resulting module globals (CT_*, CONSONANTS, VOWELS, …) without locking. Concurrent first-time calls could both enter the population block, letting one thread observe a half-initialised set of tables. Guard the one-time init with double-checked locking under a module lock; the globals are written exactly once and published (via _initialized_tables_dir, assigned last) only after every table is populated, so steady-state reads stay lock-free. kog2p needs no change — it declares no module globals, its data is read-only, and _cachedReadRules uses a thread-safe lru_cache. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Diacritization is a text->text graph transform, not phonemization. scriptconv/diacritics.py now owns lang->backend routing (STRESS_LANGS, _diacritizer_family, _is_european_portuguese), backend loading/caching (phonikud, text2tashkeel, stressonnx, bifonia) as module-level functions, and the public diacritize() dispatch. BasePhonemizer no longer carries any diacritization state or methods (phonikud/tashkeel properties, add_diacritics, diacritizer_model/phonikud_model constructor params). registry.py drops the phonikud_model passthrough accordingly. No backward-compat shim; downstream repos adopt scriptconv.diacritics.diacritize directly. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
phonikud is a small, public, unencumbered ONNX model, unlike the large/licensed ByT5/Charsiu models. Relax the "never download" policy for it specifically: when phonikud_model is omitted, scriptconv now downloads and caches it itself via stdlib urllib, atomically, under SCRIPTCONV_CACHE/XDG_CACHE_HOME. Explicit path/callable overrides are still honored unchanged. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…er_model) (#78) The two model-bearing edges now each read their own routing-context key: - text -> ipa (phonemizer edge) reads phonemizer_model and forwards it to model-backed phonemizers (ByT5/Charsiu/…). Previously it forwarded no model at all, so those backends could not receive one through the graph. - text -> text-diacritized (diacritizer edge) reads diacritizer_model, routed to whichever backend the language uses (Hebrew phonikud path, Arabic text2tashkeel model, stressonnx model). Hebrew no longer needs a separate phonikud_model parameter — it rides in the one diacritizer_model knob. diacritize()'s pre-unification kwargs (phonikud_model, model) are folded in so existing callers keep working during migration. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Human review requested!