feat(abbr)!: V2 single-pass abbreviation engine + tech-debt paydown (major, breaking) - #78
Merged
Merged
Conversation
Captures the architectural analysis from the latency investigation: the abbreviation engine inherits pySBD's global-re.sub-rewrite model, which makes the per-period boundary decision O(distinct-abbreviations x text-length), order-dependent, and re-implemented six ways across languages. Argues that the linguistically-essential task is a local per-period classification that a single-pass design would make O(text), order-independent, and far simpler. Includes: essential-vs-accidental complexity analysis, the proposed classifier design and per-language policy hooks, the full preservation spec distilled from surveying every lang/ override (suffix-decision patterns, flags, and the genuinely hard parts), a phased English-first implementation plan, the guardrails (differential oracle, Golden-Rule anchor, all-26-language diff, fuzz, CodSpeed), acceptance criteria, and an honest risk/reward recommendation (default: leave it; v2 only as a deliberate major-version effort). No code changes.
Build the differential oracle (debugging aid, not a gate) and the curated English correctness corpus that gate the V2 single-pass period classifier, per analysis/ABBREVIATION_ENGINE_V2_PLAN.md. - tests/v2/oracle.py: legacy_protect_positions() replays the legacy per-line abbreviation-protection step (search_for_abbreviations_in_string) and returns original-text offsets whose '.' became '∯', mapping back across the upstream single-letter rules and the '??' -> '&ᓷ&&ᓷ&' placeholder expansion. classifier_protect_positions() + diff_positions() are stubs that fail loudly (ClassifierUnavailable) until USE_PERIOD_CLASSIFIER lands. - tests/v2/corpus_en.py + test_corpus_en.py: 41 hand-labeled boundary cases (38 green that must stay green, 3 strict-xfail Phase-2 correctness targets the legacy engine gets wrong: Ph.D.+surname, a.m.+timezone). - tests/v2/test_oracle.py: self-tests for the oracle mechanics across 12 base-class and override languages. - analysis/: the revised plan, the RFC evaluation, and the captured green baseline (phase_profile + differential_profile --size medium). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement the single-pass period classifier that replaces ONLY the per-line abbreviation-protection step inside AbbreviationReplacer.search_for_abbreviations_in_string. The classifier classifies each candidate period ONCE from the ORIGINAL line text (never from a sentinel left by a prior decision) into PROTECT / BOUNDARY / PLACEHOLDER, then realizes each decision GLOBALLY per (abbr, follower-char) unit — mirroring the legacy global re.sub semantics — and rebuilds the line in one pass. Everything else in replace() (replace_multi_period_abbreviations, compact-ampm, uppercase-initialism, allcaps-imprint, ampm, standalone-I) is unchanged and still sees the same ∯/'.' substrate. Routing is a per-language feature flag: USE_PERIOD_CLASSIFIER (default False -> legacy path byte-exact). en/en_legal set it True and ride BASE_POLICY with zero policy code (they already set the class flags + STARTER_AWARE data the classifier reads via its replacer back-reference). The AbbrPolicy classify_special / candidate_filter / pre_stages / post_stages seams exist for Phase 4/5 but are inert for BASE_POLICY. The three branches and their suffix patterns are ported character-for-character (regular, prepositive, number incl. upper-join/upper-split/lower/QQ-placeholder and the multi-char number -> regular fallthrough). The reachability gate is reproduced exactly by reusing the SAME _AbbreviationData (automaton + sets + boundary_class + elision_chars) — the automaton/keys are never rebuilt, so the U+0130 İ bare-key exception and the publish-after-build thread-safety invariant are preserved. The global realization scopes IGNORECASE to the abbreviation lookbehind only (via (?i:...)) so a capital follower like "Ltd. She" does not match the case-sensitive lowercase follower class. Verification: a per-line differential (classifier vs forced-legacy) over the curated corpus + adversarial extras across all three split modes is byte-exact (0 mismatches / 360 lines, en + en_legal); the differential oracle reports ([],[]) on the whole English corpus (the Phase-2 equality TARGET). Full suite, English Golden Rules, ruff, zero-dep, and span round-trip all green. Perf is within noise (~3.5% on the densest abbreviation-heavy short sample, ~0 on normal prose) — this is a correctness+maintainability refactor, not a perf project. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spanish overrides zero scan methods and uses no elision, so it rides the base PeriodClassifier (BASE_POLICY) directly via a nested AbbreviationReplacer that only flips USE_PERIOD_CLASSIFIER = True. It is not one of the five CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE languages, so that flag stays off. Gates: tests/lang/test_spanish.py (94 passed), full suite (2052 passed, 9 pre-existing xfail), ruff check + format, zero-dep + span round-trip (332 passed). Differential oracle finds 0 protect-position divergences and segment() output is byte-identical to the legacy path across the full Spanish corpus (clean + non-clean). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Danish overrides zero scan methods and uses no elision, so it rides the base PeriodClassifier (BASE_POLICY) directly via its existing nested AbbreviationReplacer, which only needs USE_PERIOD_CLASSIFIER = True. It is one of the five CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE languages and the classifier already reads that flag off the replacer back-reference, so no new policy hook is required; PROTECT_ALLCAPS_IMPRINT_SUFFIXES runs in a later replace() pass that V2 leaves untouched. Gates: tests/lang/test_danish.py (50 passed), full suite (2052 passed, 9 pre-existing xfail; the unrelated tests/test_corpus_compare_segmenters.py collection error pre-exists at HEAD due to an untracked partial benchmarks/corpus_compare package), ruff check + format, zero-dep + span round-trip (332 passed), English gate (278 passed, 4 pre-existing xfail). Differential oracle finds 0 protect-position divergences and segment() output is byte-identical to the legacy path across the full Danish corpus (golden + clean + PDF + adversarial prepositive/number/regular cases) in both default and clean modes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Greek overrides zero scan methods and uses no elision, so it rides the base PeriodClassifier (BASE_POLICY) directly. The classifier reads the CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE flag off the replacer back-ref, so no policy hook is needed. The other Greek flags (PROTECT_ALLCAPS_IMPRINT_SUFFIXES, NON_LATIN_CAPITAL_STARTS_SENTENCE) and the Unicode MULTI_PERIOD_ABBREVIATION_REGEX drive only the later passes (replace_multi_period_abbreviations, the all-caps imprint / uppercase-initialism restores) that V2 leaves untouched. Differential oracle: 0 position divergences and 0 segment() divergences vs the legacy path across a 14-sentence Greek abbreviation corpus (regular, prepositive, number, multi-period, capital/lowercase followers). Full suite + English + zero-dep + span round-trip + ruff all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dutch is a base-class language: it overrides zero scan methods and uses no elision, so it rides the base PeriodClassifier (BASE_POLICY) directly. Set USE_PERIOD_CLASSIFIER = True and leave CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE off (Dutch capital followers flow through the split-mode ambiguity dial, matching legacy per-line protection). UPPERCASE_INITIALISM_SPLIT_MIN_RANK = 2 is retained; it drives a later pass the classifier leaves untouched. Differential oracle over 802 real Dutch inputs (596 UD-nl alpino sentences + 206 nl Wikipedia paragraphs) plus the 9 Golden-Rule cases shows ZERO protected- position divergences from the legacy path. Dutch tests, English gate, full suite, ruff, zero-dep, and span round-trip all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Italian overrides zero scan methods; its only language-specific hook is
elision ("l'Ing.", "l'Avv."), which flows automatically through the
existing _AbbreviationData wiring (ELISION_CHARACTERS -> boundary_class +
elision_chars, both read by the PeriodClassifier off the same data). So
Italian rides the base PeriodClassifier (BASE_POLICY) directly with the
flag flipped on; no policy hook is needed. It is not a
CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE language, so that flag stays off and
capital followers flow through the split-mode ambiguity dial, matching
the legacy path.
Gates: tests/lang/test_italian.py (41 passed), full suite, ruff, zero-dep,
span round-trip all green. Oracle: zero protected-position divergences
across the italian test corpus plus 1102 real Italian blocks (Wikipedia +
UD-ISDT test set) — legacy == V2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
French overrides zero abbreviation scan methods; its only language-specific
hook is elision ("l'art.", "d'env."), which already flows through
_AbbreviationData.boundary_class / elision_chars and is read by the
PeriodClassifier off the SAME data. So french rides the base PeriodClassifier
(BASE_POLICY) directly with USE_PERIOD_CLASSIFIER flipped on — identical shape
to italian. French is not a CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE language, so
capital followers keep flowing through the split-mode ambiguity dial.
Oracle adjudication on a 26-line French corpus (regular/prepositive/number
abbrs, elision, multi-period initialisms, capital followers) shows zero
legacy-vs-classifier divergences, and end-to-end segmentation is byte-identical
across all three split modes. French tests (14), full suite, ruff, zero-dep,
and span round-trip all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Polish overrides zero scan methods, uses no elision, and is not one of the CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE languages (it capitalizes proper nouns mid-sentence), so it rides the base PeriodClassifier (BASE_POLICY) directly — identical shape to Dutch, minus the initialism exception. The only hook needed is flipping USE_PERIOD_CLASSIFIER = True on a nested AbbreviationReplacer. Gates: tests/lang/test_polish.py (6 passed), full suite (2055 passed, 9 xfailed), ruff check + format clean, zero-dep + span round-trip (332 passed). The differential oracle reports NO position-level diffs and NO segment() diffs vs the legacy path across a 24-line Polish corpus covering regular (np./itd./itp./ łac./niem.), multi-period (p.n.e/n.e/p.o/sp. z o.o), and capital-follower shapes; Polish defines no number or prepositive abbreviations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hindi overrides zero scan methods, uses no elision, and is not a CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE language, so it rides the base PeriodClassifier (BASE_POLICY) directly with only the USE_PERIOD_CLASSIFIER opt-in flag — identical shape to Dutch/Spanish/Polish. It inherits Standard.Abbreviation (the English-derived lists), and capital followers flow through the split-mode ambiguity dial. Self-gate green: hindi tests, full suite (2055 passed, 9 pre-existing xfail), ruff check+format, zero-dep, span round-trip. Oracle differential on a mixed Hindi/English abbreviation corpus (Dr./Mr./U.S.A./No./fig./p./vol./pp./Prof./ St./Inc.) shows zero legacy-vs-V2 protected-position divergences, and end-to-end segment() output is byte-identical across all three split modes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Marathi overrides zero scan methods and uses no elision, so it rides the base PeriodClassifier (BASE_POLICY) directly — identical shape to Hindi. It inherits Standard.Abbreviation (the English-derived lists) and is NOT a CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE language, so capital followers keep flowing through the split-mode ambiguity dial, matching the legacy per-line protection on Marathi text. Gates green: marathi tests (8 passed), full suite (2055 passed, 9 pre-existing xfail), ruff check/format, zero-dependency import, span round-trip. Differential oracle shows zero legacy-vs-classifier divergence across a Marathi corpus (native danda/double-danda terminators plus mixed Marathi/English abbreviation, number-abbr, prepositive, and initialism cases); segment() output is identical to the legacy path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tagalog overrides zero scan methods and uses no elision, so it rides the base PeriodClassifier (BASE_POLICY) directly — identical shape to Hindi/ Marathi. The only hook needed is flipping USE_PERIOD_CLASSIFIER = True on a nested AbbreviationReplacer. Tagalog's prepositive titles (G./Bb./Gng./Dr./ Engr./Sr./Sta./Kgg./Ma.) and number abbreviations (No./Blg./Bp./Hal.) are handled by the base classifier's prepositive and number branches; it is NOT one of the CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE languages, so that flag stays off and capital followers flow through the split-mode ambiguity dial, matching the legacy per-line protection on Tagalog text. Self-gate green: tests/lang/test_tagalog.py (26 passed), full suite (2055 passed, 9 pre-existing xfail), ruff check + format, zero-dep + span round-trip (332 passed). The differential oracle reports ZERO position-level divergences and ZERO segment() diffs vs the legacy path across a 39-input Tagalog corpus (all 26 shipping cases plus extras covering prepositive, number-abbr with lowercase/paren/Roman/?? followers, initialism chains, multi-period, embedded English U.S.A./vol./pp., and capital-follower shapes) across all three split modes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Armenian (hy) overrides zero scan methods and uses no elision, so it rides the base PeriodClassifier (BASE_POLICY) directly — identical shape to Hindi/ Marathi/Tagalog. The only hook needed is flipping USE_PERIOD_CLASSIFIER = True on a nested AbbreviationReplacer. It inherits Standard.Abbreviation (the English-derived lists), and is NOT one of the CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE languages, so that flag stays off and capital followers flow through the split-mode ambiguity dial, matching the legacy per-line protection on Armenian text. Armenian terminates sentences with native punctuation (։ verjaket, ՜ batsaganchakan, : as full stop), so the Latin "." is never a terminator anyway; the classifier just protects abbreviation periods exactly as the legacy path did. Self-gate green: tests/lang/test_armenian.py (26 passed), full suite (2055 passed, 9 pre-existing xfail), ruff check + format, zero-dependency import + span round-trip (332 passed). The differential oracle reports ZERO position-level divergences and ZERO segment() diffs vs the legacy path across a 19-input Armenian corpus (all shipping golden/more cases plus embedded Latin prepositive titles Mr./Dr./Prof./St., number abbreviations Vol./No./fig./pp. with lowercase/paren/Roman/?? followers, the U.S.A. initialism chain, p. No. chaining, Inc., decimals, and mixed Armenian/English text) across all three split modes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Amharic inherits Common, Standard directly and overrides zero scan methods with no elision, so it rides the base PeriodClassifier (BASE_POLICY) exactly like Armenian/Hindi/Marathi/Tagalog. Flip USE_PERIOD_CLASSIFIER=True on a thin AbbreviationReplacer subclass; no other hook is needed. Amharic terminates sentences with native punctuation (። ፧ ! ?), so the Latin '.' only ever marks embedded abbreviations (Dr., U.S., Vol. IV); the classifier protects those periods identically to the legacy path. Gates: amharic tests pass; full suite 2055 passed / 9 xfailed (no new failures, no xpass); ruff check + format clean; zero-dep + span round-trip green. Oracle differential on a 16-line Amharic corpus shows 0 divergences (legacy == V2 at every protected position) and 0 end-to-end segmentation differences. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Burmese overrides zero scan methods and uses no elision, so it rides the base PeriodClassifier (BASE_POLICY) directly via the minimal nested AbbreviationReplacer hook — identical shape to Amharic. It inherits Standard.Abbreviation and the unicameral Burmese script has no letter case, so CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE stays off. Gates: tests/lang/test_burmese.py + full suite (2055 passed, 9 pre-existing xfail) + ruff + zero-dep + span round-trip all green. Differential oracle shows zero legacy-vs-V2 divergences across a 15-line Burmese corpus. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Urdu overrides zero scan methods and uses no elision (it inherits Standard.Abbreviation with ELISION_CHARACTERS == ""), so it rides the base PeriodClassifier (BASE_POLICY) directly via the minimal nested AbbreviationReplacer hook — identical shape to Burmese/Amharic. It inherits Standard.Abbreviation (the English-derived lists) and the Arabic script Urdu uses is unicameral (no letter case), so it is NOT one of the CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE languages; that flag stays off and capital (Latin) followers flow through the split-mode ambiguity dial, matching the legacy per-line protection on Urdu text. Urdu terminates sentences with the danda "۔", "؟", plus "!" and "?", so the Latin "." is never a terminator; the classifier just protects embedded abbreviation periods (e.g. "Dr.", "U.S.") exactly as the legacy path did. Gates: tests/lang/test_urdu.py + full suite (2055 passed, 9 pre-existing xfail) + ruff + zero-dep + span round-trip all green. Differential oracle shows zero legacy-vs-V2 divergences across a 17-line Urdu corpus. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reimplement the combined English/Spanish/Chinese profile's abbreviation
protection as an AbbrPolicy on top of the V2 PeriodClassifier and flip
USE_PERIOD_CLASSIFIER on for it, deleting the two engine-method overrides
(replace_period_of_abbr + scan_for_replacements, ~70 lines) it carried.
EN_ES_ZH_POLICY re-encodes the override's behavior as data:
- follower_class [^\W\d_]: any Unicode letter may follow an abbreviation.
- cjk_follower_class [㐀-鿿]: a CJK ideograph immediately after the
period protects WITHOUT an intervening space ("U.S.标准", "etc.标准");
with a space the [^\W\d_] class already covers it. Woven into the regular,
prepositive and number-lower suffix patterns (NOT the number-upper arms,
matching legacy: a CJK follower is always a separate no-space candidate).
- ascii_only_upper_heuristic: the capital-follower-is-boundary cue fires only
for an ASCII capital. A non-ASCII capital ("Sr. Élena") is not a cue, so it
flows through the regular/prepositive branches and stays joined; a number
abbreviation before a non-ASCII capital ("Fig. Él") still starts a sentence
in balanced/aggressive (multi-char fallthrough guard) but JOINS in
conservative (RE_NUM_LOW_JOIN widens the letter slot to [^\W\d_], mirroring
the legacy _leans_join branch). The legacy _HEURISTIC_ABBREVIATIONS gate was
a no-op (that set equals the full abbreviation set) and is dropped.
All three additions are policy-gated and inert for BASE_POLICY languages.
Adjudication: end-to-end segment() is byte-identical to the legacy override
across a 5,484-pair (text x split_mode) adversarial corpus (multi-abbr lines,
mixed scripts, no-space CJK, accented/umlaut capitals, ?? placeholders). No
reviewed output diffs. Gates green: full suite, English Golden Rules, en_es_zh
tests, split_mode dial, zero-dep, span round-trip, ruff.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reimplement the German profile's abbreviation protection as an AbbrPolicy
on top of the V2 PeriodClassifier and flip USE_PERIOD_CLASSIFIER on for it,
deleting the bespoke scan_for_replacements override it carried.
The legacy Deutsch.AbbreviationReplacer overrode scan_for_replacements to a
SINGLE rule, re.sub(r"(?<={am})\.(?=\s)", "∯"), bypassing the base
prepositive/number/regular trichotomy entirely: PROTECT a known abbreviation's
period whenever it is followed by whitespace, regardless of the follower's
case (German capitalizes all nouns, so a capital follower is not a
sentence-start cue, e.g. "Dr. med. Meyer" keeps both periods).
DE_POLICY re-encodes that as data:
- classify_special: PROTECT before whitespace, else BOUNDARY — one decision
for every candidate, no branch dispatch.
- realize_suffix (new AbbrPolicy seam): pin the global realization pass to
the same \.(?=\s) suffix so PROTECT is realized over every occurrence with
the rule that decided it (the branch-derived _suffix_for no longer
describes a collapsed-branch decision). Base None == branch-derived suffix,
inert for every other policy.
German's reordered replace() is preserved: whole-text (not per-line)
protection; no Kommanditgesellschaft / compact-ampm / uppercase-initialism /
allcaps-imprint / standalone-I passes. Only the protection step now delegates
to the classifier (search_for_abbreviations_in_string routes through rewrite()
when USE_PERIOD_CLASSIFIER is True), so the whole-text semantics are kept.
Quirk FIXED (BC not required, plan §3): the legacy interpolated {am}
(== m.group(), boundary char + abbreviation) UNescaped into the lookbehind,
working only by accident of the German list containing no regex
metacharacters. The V2 _full_pattern re.escapes the abbreviation, so dotted
abbreviations (z.b, d.h, u.a) are matched literally — escape-everything-correct.
Adjudication: end-to-end segment() is byte-identical to the legacy override
across a 120-case corpus (40 adversarial texts x 3 split modes): multi-abbr
lines, number abbreviations (art/ca/no/nos/nr/pp), capital followers,
standalone-I, ordinal ranges, multi-line whole-text inputs, and the
unescaped-am quirk path. No reviewed output diffs. The differential oracle's
new_only positions (med./kath./Dr. before a capital) are the intended
"protect before whitespace regardless of case" behavior, confirmed by the
German Golden Rules; the base-engine oracle does not model German's
whole-text override, so those are oracle-modeling artifacts, not regressions.
Gates green: full suite (2055 passed, 9 xfailed), English Golden Rules,
deutsch tests + German regressions, zero-dep, span round-trip, ruff
check + format.
tests/v2/test_oracle.py: the non-opted-language assertion used "de" as its
example; switched to "ru" (still on the legacy path) since German opted in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reimplement Russian abbreviation protection as an AbbrPolicy hook on top
of the V2 single-pass period classifier and flip USE_PERIOD_CLASSIFIER on.
The legacy Russian AbbreviationReplacer overrode ONLY the regular branch
(replace_period_of_abbr); PREPOSITIVE/NUMBER lists are empty so every
abbreviation flowed through it. RU_POLICY.classify_special re-encodes that
override as data:
- protect a known abbreviation's period unconditionally (the legacy regex
had no follower-class lookahead, so "5 куб.м." keeps "куб." even with no
space before the Cyrillic "м");
- keep a BOUNDARY for a SENTENCE_FINAL language-tag abbreviation (рус.,
англ., др., …) directly before a Cyrillic capital ("…и др. Она" splits),
while a Latin-capital foreign gloss ("англ. Moscow") stays joined;
- apply the "ср." (cf.) compare-phrase heuristic verbatim, including its
split-mode lean.
Because those decisions read downstream context per occurrence, add a
realize_per_occurrence policy capability: enumeration keeps every occurrence
(deduped only by exact period index) and each candidate is classified from
its own ORIGINAL context and anchored to its own period, never realized via
a single global re-anchored suffix. This mirrors the legacy per-match re.sub
callback exactly, so two "ср." on one line can decide differently.
SENTENCE_FINAL_ABBREVIATIONS stays on the language class as the data table;
the policy reads it off the replacer back-reference. The differential oracle
reports zero protected-position divergences vs the legacy path across the
full Russian corpus (Golden Rules + regression + adversarial multi-"ср."
lines), so this is a faithful re-encoding with no output change.
Update the oracle "non-opted language" test to use Slovak (still legacy)
now that Russian opted in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reimplement Slovak's AbbreviationReplacer override as the SK_POLICY hook on
the V2 PeriodClassifier and flip USE_PERIOD_CLASSIFIER on for it.
The legacy override replaced ONLY the regular branch (replace_period_of_abbr)
with a literal whole-span txt.replace(abbr + ".", abbr.replace(".", "∯") + "∯").
That has two effects the base regular branch lacks: it protects UNCONDITIONALLY
(no follower-class lookahead, since Slovak abbreviations routinely precede a
capitalized company/proper name -- "napr. XYZCorp", "apod. Niečo"), and it
sentinelizes EVERY interior period of a spaced/compact abbreviation
("s. r. o." -> "s∯ r∯ o∯", "a.s.a.p." -> "a∯s∯a∯p∯"). The PREPOSITIVE
(dr/ing/mgr/prof …) and NUMBER (č/no/nr) branches were never overridden and
inherit the base classifier unchanged.
SK_POLICY models this as classify_special (regular branch -> PROTECT
unconditionally; NOT_HANDLED for prepositive/number) + a new protect_edit hook
that splices the whole span, with realize_per_occurrence anchoring each
word-boundary occurrence to its own span. Overlapping whole-span edits (e.g.
a.s.a.p enumerating both "a.s.a.p" and "a.s") are resolved longest-first in
_dedup_sorted, mirroring the legacy length-descending mutating str.replace where
a shorter embedded span becomes a no-op. protect_positions now reports protected
offsets by diffing the rebuilt line, so whole-span PROTECT surfaces every
interior+trailing period to the oracle.
Quirk FIXED (BC not required, plan §3, reviewed Golden-Rule-anchored): the
legacy global literal str.replace also mutated an unrelated EMBEDDED occurrence
on the same line ("good s.r.o. then Xs.r.o." protected the "Xs.r.o." periods
too). The V2 per-occurrence path only edits candidates the word-boundary
reachability gate enumerates, dropping that cross-contamination. No Golden Rule
exercises it and segment() output is unchanged across the Slovak corpus.
Self-gate: tests/lang/test_slovak.py green; full suite 2055 passed / 9 xfailed;
ruff check + format clean; zero-dep + span round-trip green; oracle test updated
(sk now opted in, bg remains the legacy-path probe); 0 segment-level diffs vs the
original override across the Golden Rules + extra probes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reimplement the legacy ``Bulgarian.AbbreviationReplacer.replace_period_of_abbr``
override as ``BG_POLICY`` on the V2 PeriodClassifier and flip
``USE_PERIOD_CLASSIFIER`` on for Bulgarian.
Bulgarian's legacy override was structurally identical to Slovak's: a
regular-branch-only override (PREPOSITIVE/NUMBER abbreviation lists are empty,
so every abbreviation is regular) doing an UNCONDITIONAL trailing-period protect
plus a WHOLE-SPAN interior-period protect for Cyrillic multi-period
abbreviations ("б.р." -> "б∯р∯") so the boundary regex does not shatter the
token. ``BG_POLICY`` therefore rides the shared ``_sk_classify_special`` +
``_sk_protect_edit`` (whole-span splice), overriding only the regular branch.
Quirk FIXED (BC not a constraint, plan §3, reviewed Golden-Rule-anchored): the
legacy trailing-period regex interpolated the abbreviation UNescaped into a
lookbehind, so each interior '.' became a regex wildcard — when a genuine "б.р."
fired the automaton, the global re.sub also protected an unrelated decoy
("…б.р. … бхр. …" -> spurious "бхр∯"). The V2 path enumerates only the
re.escape-d word-boundary candidates, so the decoy keeps its boundary period.
No Golden Rule exercises this case, and it is invisible at the segment() level.
Gates green: full suite (2055 passed, 9 pre-existing xfail), Bulgarian + Cyrillic
regression tests, ruff check + format, zero-dependency import, span round-trip.
Oracle adjudication: 220 search-level + 146 segment()-level differential inputs
across all 70 Bulgarian abbreviations show zero observable-output diffs vs legacy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reimplement ArabicScriptProfile's abbreviation protection as an AbbrPolicy
hook (AR_POLICY) on top of the V2 period classifier and flip
USE_PERIOD_CLASSIFIER on for both Arabic (ar) and Persian (fa), which share
the profile.
The legacy override collapsed every branch into a single rule —
re.sub(r"(?<={re.escape(am)})\.", "∯") — protecting a matched abbreviation's
period regardless of follower (a bare \. suffix; Arabic script has no letter
case, so there is no capital-follower boundary cue). AR_POLICY reproduces
this exactly: classify_special unconditionally PROTECTs every enumerated
candidate and realize_suffix pins the global realization pass to bare \..
No quirk fix was required: the legacy rule already escaped am, so a dotted
abbreviation like "e.g" never wildcard-matched an unrelated "egg." The V2
lookbehind uses the pre-built re.escape, preserving that behavior
(tests/regression/test_arabic_script_abbreviation_metachar.py).
Faithful migration with zero output change: the differential oracle reports
no protected-position divergence between the legacy bare-protect and the V2
path across all ar/fa Golden Rules, both regression inputs, and adversarial
cases (dotted abbr, end-of-line period, non-space/capital followers,
English-through-Arabic-profile); ar/fa segmentation is byte-identical.
Update the stale oracle availability test to assert zh (still legacy)
remains unavailable, since ar/fa now opt in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reimplement standalone Chinese (zh) abbreviation protection as an AbbrPolicy hook on the V2 period classifier and enable USE_PERIOD_CLASSIFIER for it, replacing its ``replace_period_of_abbr`` override (the only engine method zh overrode) with data. The legacy zh override wove a CJK-ideograph follower ``[一-鿿]`` (BMP CJK only, no Ext-A) into the REGULAR branch suffix alone, with no leading ``\s`` so "U.S.标准" / "etc.标准" protect without an intervening space; the prepositive / number branches inherited the base (no-CJK) suffixes and CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE stayed off. Reproduce that exactly: - New ``ZH_POLICY`` (follower_class [a-z], cjk_follower_class [一-鿿], cjk_follower_regular_only=True). Distinct from EN_ES_ZH_POLICY, whose whole-method override wove CJK into every branch and widened the follower class to any Unicode letter. - New ``AbbrPolicy.cjk_follower_regular_only`` flag: when True the CJK follower alternative is woven only into RE_REGULAR (and the number-branch multi-char REGULAR fallthrough), not into the prepositive / number-lower suffixes. Inert for every existing policy (base/en_es_zh leave it False), so no other language changes. Verified byte-identical to the reconstructed legacy zh override at both the per-line protection step and full segment() output across the zh Golden + challenging corpus and an adversarial prepositive/number-before-CJK set, in all three split modes. LATIN_UPPERCASE_RESPLIT stays False (via CJKBoundaryProfile). The differential oracle's self-test moves its "not opted in" negative case from zh to kk (Kazakh), still on the legacy path. Gates: full suite (2055 passed, 9 pre-existing xfail), English gate, ruff check+format, zero-dep, span round-trip all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reimplement Japanese's ``AbbreviationReplacer.replace_period_of_abbr`` override as a data-only ``JA_POLICY`` on the V2 PeriodClassifier and flip ``USE_PERIOD_CLASSIFIER = True``. Japanese's legacy override touched ONLY the regular branch, keeping the base regular suffix and appending a kana+CJK-ideograph follower ``[-ヿ一-鿿]`` with no leading ``\s`` so "U.S.標準" / "ver.あいうえお" protect even without an intervening space. This is structurally identical to standalone Chinese (ZH_POLICY): regular branch only via ``cjk_follower_regular_only``, base prepositive/number inherited, capital-follower cue off. The only difference is the follower range — JA widens ``[一-鿿]`` to also include the kana blocks (U+3040..U+30FF), since Japanese prose continues a sentence in hiragana/katakana directly after an abbreviation period. Verified byte-identical to the legacy Japanese protection step across all three split modes over the ja Golden/clean cases plus an adversarial regular(CJK/kana)/prepositive/number/placeholder corpus (0 mismatches). Gates: tests/lang/test_japanese.py (25 passed) + full suite (2055 passed, 9 pre-existing xfail) + ruff check/format + zero-dep + span round-trip all green. Oracle ``new_only`` diffs are all kana/CJK-follower protections the oracle's flag-disabled base fallback can't see (the override method is now gone); each matches the real legacy override byte-for-byte. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Kazakh overrode ZERO scan methods (scan_for_replacements / replace_period_of_abbr inherited; PREPOSITIVE/NUMBER sets empty; CAPITALIZED_FOLLOWER_IS_BOUNDARY_CUE off), so its per-line abbreviation protection step is the BASE REGULAR branch verbatim. Route it through the PeriodClassifier (USE_PERIOD_CLASSIFIER=True, ABBR_POLICY=BASE_POLICY); the differential oracle confirms byte-identical protected positions and an end-to-end segment diff of zero over every Golden Rule + regression. All Kazakh-specific behavior lives in the THREE whole-text passes that wrap the base replace() and cannot collapse into the per-line classifier: the upstream Cyrillic single-uppercase-letter initials, the dotted single-period abbreviation protection keyed on _LOWERCASE_CONTINUATION_CHARS (the dotted forms are stored with a trailing dot, so the automaton never enumerates them and this pre-pass must sentinelize them first), and the post-pass protect-before-parenthesis (which runs after replace_multi_period_abbreviations and reads its interior sentinels). These stay in a thin replace() override, mirroring the Deutsch V2 conversion. Kazakh was the last language on the legacy path; update the oracle self-tests to assert kazakh parity and exercise the ClassifierUnavailable guard by temporarily forcing the flag off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contract-close for the PeriodClassifier cutover: 26/26 language codes on V2, 0 English oracle diffs, all gates green, ~+3-5% short-string perf. Documents the 3 unfixed correctness targets and the legacy-path retirement backlog where the maintainability win is banked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… sole path All 26 language codes route the per-line abbreviation-protection step through the V2 PeriodClassifier, so the legacy per-occurrence re.sub engine was dead code. Per the V2 plan §4 Phase-6 cutover, make the classifier the SOLE path and delete the now-unreachable legacy machinery. - abbreviation_replacer.py: drop the USE_PERIOD_CLASSIFIER flag/branch so search_for_abbreviations_in_string always delegates to the classifier; delete the legacy per-occurrence loop body, scan_for_replacements, replace_period_of_abbr, _replace_number_abbr, _replace_with_escape, _protect_number_abbr_unknown_placeholder, and _replace_starter_aware_prepositive. The classifier/policies never depended on them (verified by grep). - lang/: remove every now-redundant USE_PERIOD_CLASSIFIER = True line; drop the 11 AbbreviationReplacer subclasses that existed ONLY to set it (armenian, amharic, burmese, marathi, hindi, urdu, spanish, french, italian, tagalog, polish) so they inherit Standard.AbbreviationReplacer. - tests/v2/oracle.py: the legacy engine no longer exists, so legacy_protect_positions now reads from a FROZEN snapshot (captured while it was live) instead of replaying deleted code; classifier_protect_positions no longer gates on the removed flag. test_oracle.py asserts the frozen-snapshot + classifier mechanics and keeps the English/Kazakh parity targets (non-equality elsewhere). - Refresh stale legacy-method references in test comments/docstrings. Net LOC: -182. Full suite green (2057 passed, 9 xfailed); ruff + zero-dep green; phase_profile --size short unchanged (~0.88 ms/call). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ass classify+suffix The V2 PeriodClassifier was rebuilt for every per-call AbbreviationReplacer instance, recompiling ~9 RE_* suffix patterns and starting with a cold _full_cache on each segment() call. Those patterns and the full-pattern cache are line-independent and depend only on (policy, split_mode, data) — all immutable for a given _AbbreviationData — so cache the classifier on the shared _AbbreviationData keyed by (id(policy), split_mode), rebinding the live-instance back-ref on retrieval. This eliminates the per-call regex compiles and lets the full-pattern cache persist across calls (free-threaded-safe: published under the existing cache lock, classifier holds no per-line state). Also fold suffix selection into classification: _classify_with_suffix returns (decision, realization-suffix) in one pass so the global-realize hot path no longer re-derives am_lower/upper/the branch (and re-runs num_low.match) in a second _suffix_for pass. classify()/_classify_number() are thin wrappers, so the oracle / per-occurrence callers are unchanged; classify_special decisions return suffix=None and still fall back to _suffix_for (honoring policy.realize_suffix). Behavior identical (protected-period positions unchanged): full suite 2057 passed / 9 xfailed, English+V2 362 passed / 7 xfailed, ruff + zero-dep green. Interleaved A/B on the same box: abbr search path 169.6 -> 146.5 us/call (-13.6%, best-of-12); full pipeline 0.8141 -> 0.7760 ms/call (-4.7%); phase_profile --size short total 0.880 -> 0.856 median, --size medium 1.663 -> 1.613. Reclaims the +9% classifier regression. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ne units
The three Phase-2 correctness targets in tests/v2/corpus_en.py are owned by the
abbreviation passes that run AFTER the V2 PeriodClassifier; fix them there and
promote them from strict-xfail to GREEN.
- replace_multi_period_abbreviations: a degree/title abbreviation (e.g. "Ph.D.")
in name-prefix position — opening the line, or itself preceded only by another
protected title like "Dr." — prefixes a capitalized surname, so its final
period is a name-internal separator, not a boundary. Restricted to mixed
abbreviations with a multi-letter lowercase-bearing part ("Ph"), so a trailing
degree ("She earned a Ph.D. Smith advised her.") still splits and a pure
all-caps initialism ("A.S.E. Ackermann") still follows the split-mode dial.
Only in conservative/balanced mode; aggressive still splits.
- AmPmRules._TZ: recognize spelled-out timezone names ("Eastern Standard Time",
"Pacific Time", "Coordinated Universal Time") after "<num> a.m./p.m." as part
of the time unit, anchored on the trailing "Time" keyword so an ordinary
capitalized sentence start ("9 a.m. The meeting started.") still splits.
Adds tests/regression/test_titled_name_and_timezone.py with the fixes and their
no-regression guardrails. Full suite green (2069 passed, 6 xfailed; the 3
promoted targets dropped out of the xfail set).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Promote InvalidConfigurationError, UnknownLanguageError, register_language, and unregister_language into the top-level sentencesplit namespace and __all__, alongside the existing SentenceSplitError / list_languages. These are the exceptions callers catch and the registry functions the README documents, so they belong on the public surface. Extend test_public_surface_matches_all to assert the expanded __all__ set. Behavior-neutral: no segmentation change (segment snapshot byte-identical). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add discoverable reason= strings to the six standing xfails so they form a greppable backlog index (tag: BACKLOG[xfail-index]): arabic bidi-mark abbr, a.m./P.M.-vs-title boundary, the two no-space-after-period OCR cases, the Pt. medical-note abbreviation, and the issue-#83 four-dot ellipsis. Per QW6: the #83 xfail is kept (and annotated DO NOT DELETE) because dropping it would leave the suite asserting a model inconsistent with its passing 2-/3-dot siblings. Test-only; behavior-neutral (segment snapshot byte-identical). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The differential oracle (tests/v2/oracle.py, 174 LOC) and its self-tests (tests/v2/test_oracle.py, 148 LOC) diffed the PeriodClassifier against an 18-entry hand-frozen _LEGACY_SNAPSHOT of the legacy engine that was deleted at the Phase-6 cutover. The oracle was documented as a debugging aid, not a gate, and freezing a snapshot of deleted code carries no ongoing value. Re-home the genuinely load-bearing parity assertions as direct segment() cases before deleting: - English/en_legal (Dr./Sen./No./Vol. stay joined; Bankr. joins only in the legal profile, Cir. stays joined in both) -> three green CorpusCase records in tests/v2/corpus_en.py. CorpusCase gains a `lang` field (default "en") and the corpus driver caches one Segmenter per code so the en_legal-only arm runs in the same corpus. - Kazakh (См./рис. unprotected and split; обл. WIDE-follower keeps the period joined) -> two new segment() tests in tests/lang/test_kazakh.py. Net test reorganization; no production code touched. The 26-language segment() snapshot stays byte-identical (behavior-neutral). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…uageProfile Move every per-language rule the Processor consumed via self.lang.* onto LanguageProfile as resolved fields built once in _build, so the Processor (and the deutsch/slovak Processor subclasses) read configuration through exactly one channel (self.profile.*). The language class is no longer threaded into the Processor as self.lang; the AbbreviationReplacer is constructed from self.profile.language instead. Resolved fields added: language, punctuations, the four special-token rules, sub_single_quote_rule, single_newline_rule, question_mark_in_quotation_rule, sub_symbols_table, number_rules, ellipsis_rules (+ three_consecutive), reinsert_ellipsis_rules, double_punct_rules, exclamation_rules (+ mid_sentence / before_comma). Internal-only and behavior-neutral: the 26-language segment() snapshot is byte-identical. Extends tests/test_language_profile.py to assert the resolved static-hook set (incl. per-language Punctuations/Numbers overrides). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…on lists
Two coordinated pieces of roadmap S5/T6.
DATA-LINT (quarantined): tests/test_abbreviation_data_lint.py renders every
declared ABBREVIATIONS entry in a neutral lowercase-follower carrier
("foo <abbr>. bar baz") and asserts segment() keeps the period non-terminal —
the "if it's in the list, it works" contract that the existing storage-shape
data tests never checked. ~95 declared entries fail today (the pre-existing,
now-measured engine gap: ~80 mid-token breaks — non-ASCII multi-period,
hyphenated, &/(/!// , and 3+ token entries — plus ~15 single-letter false
positives like 'p'/'s'/'č'). They are seeded into a QUARANTINE allowlist and
converted to runtime pytest.xfail (unaffected by xfail_strict), so the suite
lands green-with-xfails; a non-allowlisted failure reds immediately. The
allowlist is the discoverable S6 backlog; a stale-entry lint keeps it honest.
NORMALIZE: adopt sorted(set(...)) over lowercased entries as the canonical
stored form for every ABBREVIATIONS list, via a shared
lang/common/canonical_abbreviations helper (matching the existing en_legal /
en_es_zh pattern). Behavior-neutral: the automaton keys on stripped.lower(),
match_re is IGNORECASE, and the abbr/prepositive/number sets are all lowercased,
so stored case/order never reaches a decision — the 26-language segment()
snapshot is byte-identical (diff()==[], segment_snapshot.json unchanged). No
entries dropped: Italian s.a/s.n.c/s.p.a/s.r.l (PREPOSITIVE) and load-bearing
multi-char-token entries (e.g. nl aanbev.comm) are preserved, so the specialized
subset relation still holds. A new test_abbreviations_are_canonical_form lint
asserts each list equals its canonical form to catch future non-canonical edits.
The roadmap's optional "drop internal-dot entries shadowed by
MULTI_PERIOD_ABBREVIATION_REGEX" sub-piece is DEFERRED: the obvious
regex-fullmatch heuristic is provably NOT behavior-neutral (the abbreviation-list
path and the MULTI_PERIOD mpa_replace path make different boundary decisions for
e.g. sk 'p.a' / 'ph.d' and ru 'у.е'), and a sound per-entry neutrality proof is
out of scope here; deferred rather than risk a silent behavior change the
Golden-input snapshot would not catch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….post_stages Complete the single-pass model's ownership story (roadmap S1): the fixed sequence of downstream per-period passes that AbbreviationReplacer.replace() hard-coded after the per-line classifier (replace_multi_period_abbreviations, compact-ampm, uppercase-initialism restore, allcaps-imprint, a.m./p.m. rules, standalone-I) now flow through the previously-unused AbbrPolicy.post_stages tuple. Each pass is a (replacer) -> None primitive; the active policy owns the ordered list, so a language declares its post-classifier pipeline as data: - DEFAULT_POST_STAGES is the historical full sequence; a policy that leaves post_stages empty inherits it (english/en_legal/greek/zh/ja/ru/sk/... unchanged). - DE_POLICY.post_stages is German's reduced pipeline (drops the Kommanditgesellschaft/compact-ampm/uppercase-initialism/allcaps-imprint/ standalone-I passes; a.m./p.m. without the non-ASCII restore), so the German replace() override only customizes the upstream rules and runs the driver. - KK_POLICY.post_stages is DEFAULT_POST_STAGES plus the Kazakh paren pass, so the Kazakh replace() override drops its hand-call after super().replace(). Behavior-preserving: stages self-gate on the same class flags as before and the 26-language segment() snapshot is byte-identical (diff() == []). The stages still consume the ∯ IR; S4 moves them out-of-band and deletes the sentinel afterward. Folded passes (S4 backlog visibility): all six base passes + German/Kazakh variants now run via post_stages. None became a per-Candidate classify (each reads whole-text context across ∯ that the typed Candidate does not carry), so all still consume ∯ — the count of sentinel-consuming passes is unchanged; this item completes ownership, not out-of-band migration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/test_period_classifier.py — a cross-language companion to the English-only tests/v2/test_classifier_en.py — covering each AbbrPolicy seam through the shipping languages that use it: the base REGULAR/PREPOSITIVE/NUMBER trichotomy plus the capital-is-boundary cue (en, nl), the cjk_follower arm (zh/ja regular-only, en_es_zh woven-everywhere with ascii_only_upper), the classify_special + realize_suffix branch collapse (de), the whole-span protect_edit path (bg), the per-occurrence realization path where two same-key occurrences decide independently (ru), and the post_stages seam (default inheritance, German's reduced pipeline, Kazakh's appended stage). Add tests/test_processor.py covering the two pipeline phase lists directly: the exact ordered membership of _text_processing_phases() / _boundary_processing_phases(), the conditional CJK-abbreviation phase, each phase being a bound str->str callable, and the process()/process_text() drivers composing them in order. Additive and behavior-neutral: the 26-language segment() snapshot is byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the base MULTI_PERIOD_ABBREVIATION_REGEX (common.py) from an
ASCII-only letter class to a Unicode (non-CJK) letter class so non-ASCII
single-final-letter multi-period initialisms work through the engine:
Arabic ا.ش.ا/ص.ب, Danish d.å/d.æ/f.å/s.å/u.å/ø.f, German o.ä/u.ä, Greek
ε.ε/κ.ά/μ.χ/π.χ, and the single-final-letter Dutch chains (b.verg.r.b,
chron.d.s, regl.r.t, …).
Two coupled changes make this work:
- The letter class deliberately EXCLUDES CJK/ideographic/syllabic scripts
(_NON_CJK_LETTER). A naive Unicode class — or the (?<!\w) lookbehind the
bg/el overrides use — treats CJK as \w and would greedily eat 号是 into
the match for 项目代号是A.I.-7。, breaking test_chinese.py. The base keeps
the non-CJK-aware (?<![A-Za-z0-9_]) anchor (roadmap S6 warning).
- The separators/terminator now accept the protected sentinel "∯" as well
as a literal ".". For a *declared* abbreviation the classifier converts
its final period to "∯" before replace_multi_period_abbreviations runs
("μ.χ∯"); without the "∯" arm the pass could not re-find the token to
protect its interior dots, so the entry split mid-token.
Greek's now-redundant Unicode override is dropped; el rides the base. The
single-final-letter limit is unchanged, so domains (example.co.uk.) still
split and multi-letter-final entries (κ.λπ) stay quarantined.
Promote the 26 now-working entries out of the S5 data-lint quarantine
(test_abbreviation_data_lint.py): data-lint xfails 95 -> 69. The remaining
quarantine is the genuinely-out-of-reach backlog (hyphenated, &/(/!//,
quote, 3+-token-spaced, single-letter NUMBER entries).
BEHAVIOR-CHANGING per roadmap, but the 26-language segment() snapshot is
byte-identical (diff()==[]): none of these abbreviations appear in the
Golden-Rule corpus, so the change is confined to the measured data-lint
gap and real-world text using these abbreviations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (S7+S8) BREAKING CHANGE: drop the `char_span` constructor flag from `Segmenter`. The union return is gone: `segment(text)` always returns `list[str]` and `segment_spans(text)` always returns `list[TextSpan]`. Migrate `Segmenter(char_span=True).segment(t)` -> `Segmenter().segment_spans(t)`. Also unify the lookahead surface: `SegmentLookahead` is now `Generic[T]`, `segment_with_lookahead -> SegmentLookahead[str]`, and `segment_spans_with_lookahead -> SegmentLookahead[TextSpan]` (previously a bare `tuple[list[TextSpan], bool]`). - Remove `_CHAR_SPAN_DEPRECATION_WARNED`, `_warn_char_span_deprecated`, the `self.char_span` attribute, and the clean/char_span validation branch; the pdf-requires-clean error message no longer mentions char_span. - StreamSegmenter keeps its own `char_span` output-shape flag and no longer forwards it to the wrapped Segmenter; its single-pass detect path now reads `lookahead.segments` / `.should_wait_for_more` off the new dataclass. - Migrate all call sites (conftest span fixtures, lang/regression/lookahead/ roundtrip tests, benchmarks, the spaCy example, README) to `segment_spans()`. - Delete tests/regression/test_char_span_deprecation.py (flag is gone). Behavior-neutral for segment() output: the 26-language segment() snapshot is byte-identical (diff() == []). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the four post-split resplit regexes (_CJK_QUOTE_RESPLIT_RE, _CJK_BANG_RESPLIT_RE, _LATIN_RESPLIT_RE, _MULTI_TERMINATOR_RESPLIT_RE), the uppercase-boundary splitter (_split_on_uppercase_boundary), and the multi-sentence-quote resplitter (_resplit_multi_sentence_quote with its quote-pair/threshold/length-align cluster) out of processor.py into a new sentencesplit/boundary_resplit.py. Add a shared merge_quote_continuations() parameterized by (closer_re, reporting_clause_re, latin_lowercase_continuation, cjk_closers, cjk_follower_re) so CJKProcessor (lang/common/cjk.py) and the en_es_zh combined profile stop re-implementing the quote-continuation merge. Both processors now call the one shared implementation. processor.py keeps the thin Processor._resplit_segments delegator (examples/custom_language_with_processor_hooks.py and benchmarks/phase_profile.py reference it by name) and imports the moved symbols from boundary_resplit. Behavior-neutral: the 26-language segment() snapshot is byte-identical (diff() == []); full suite, ruff, and zero-dependency gates all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
StreamSegmenter reached into Segmenter privates (_segmenter._strip_zero_width / _terminal_punctuation), a de-facto private contract between two shipped classes. Move the pure, language-parameterized logic into a module-level helper (sentencesplit/_normalize.py) that both classes import: strip_zero_width, strip_zero_width_before_sentence_closers, and terminal_punctuation, plus the zero-width/closer constants they need. Segmenter keeps thin instance wrappers (_strip_zero_width / _terminal_punctuation) delegating to the module functions with its own language_module.Punctuations; StreamSegmenter calls the module functions directly. Behavior-neutral: segment() snapshot byte-identical, full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spacy_factories entry point is effectively public to spaCy users but is absent from sentencesplit.__all__ and the README "Public API" contract. Document its status without coupling the public surface to spaCy's factory signature: the stable contract is the registered factory name "sentencesplit" and its `language` config option, while the underlying create_sentencesplit / SentenceSplitFactory call signature tracks spaCy's factory protocol and is intentionally not in __all__. Doc-only and behavior-neutral; segment() snapshot byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the bespoke classify_special/realize_suffix pair on KK_POLICY onto the base PeriodClassifier dispatch. The only thing the Kazakh pair did was widen the REGULAR-branch follower class from ASCII [a-z] to Kazakh-Cyrillic + Latin lowercase for the frozen 39-entry _KK_WIDE_FOLLOWER_STEMS set; every other branch (prepositive/number/capital-cue) is inert for Kazakh. Add a regular_follower_overrides field to AbbrPolicy: (stems, follower_class). The classifier pre-compiles a second REGULAR regex with the widened class and selects it per-stem in the REGULAR branch, its realization suffix, and the multi-char NUMBER fallthrough. KK_POLICY now rides the base dispatch like english/en_legal, dropping _kk_classify_special, _kk_realize_suffix, _KK_WIDE_REGULAR_RE, and _KK_WIDE_REGULAR_SUFFIX. Behavior-neutral: the widened suffix is byte-identical to the base REGULAR suffix with the lowercase slot swapped, so "обл. қала" still joins and "См. рис." still splits. segment_snapshot.json is byte-identical; diff()==[]. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ence/monotonicity) — quarantined (T3) Add tests/test_properties.py with three Hypothesis-driven structural invariants across all 26 registered language codes: * no-crash: segment()/segment_spans()/clean=True never raise on arbitrary + dirty Unicode (including the engine's in-band reserved sentinel codepoints to exercise the escape/restore collision path). Hard gate, holds for all 26 codes. * idempotence: re-segmenting an emitted segment reproduces it (mod trailing ws). * split_mode monotonicity: segment count is non-decreasing in split bias. Idempotence and monotonicity are real, pre-existing v2-engine gaps. Per the T3 roadmap they land QUARANTINED: each known-failing code carries a deterministic counterexample and is rendered as a runtime pytest.xfail() (immune to the global xfail_strict=true, so a later engine fix turns it GREEN, never XPASS-reds). A code NOT in an allowlist runs the full Hypothesis property search and reds CI on any violation, catching new regressions on currently-clean languages. Empirically (high-budget Hypothesis search): idempotence fails in all 26 codes (the whole registry is the backlog); monotonicity fails in the 14 Latin/Cyrillic period languages on ". ! e." while the other 12 hold and are a live gate. Stale-allowlist guards fail if a quarantined counterexample stops reproducing, so the backlog cannot silently rot. Promote the reusable per-script Hypothesis strategies (dirty-char pool, per-language alphabets/terminals, ALL_CODES, text_strategy) from test_span_roundtrip.py into tests/helpers.py; text_strategy imports hypothesis lazily so helpers.py stays importable without it. test_span_roundtrip.py now imports the shared strategies (collection unchanged at 329 tests). Behavior-neutral: tests-only; the 26-language segment() snapshot is byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… helper T5 down-payment (the XL data-driven rewrite stays deferred per the roadmap: it would break ~30 non-Golden files requesting named fixtures). Instead, collapse the ad-hoc "segments = seg.segment(text); segments = [s.strip() for s in segments]; assert segments == expected" idiom (and its inline-assert variants) onto tests/helpers.assert_segments across all language modules, so 24/28 modules now share one assertion style instead of 55 hand-rolled strip calls. Behavior-neutral test reorg: no source changes, the 26-language segment() snapshot stays byte-identical, and the xfail allowlist is preserved. The Kazakh raw-equality assertions that deliberately compare unstripped output (e.g. a preserved trailing space) and the span round-trip tests are left as-is, since assert_segments would change what they check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summarize the post-abbreviation-engine tech-debt paydown executed per analysis/V2_REFACTOR_ROADMAP.md: what landed (QW1-QW6, T2, S2, S5/T6, S1, T4, S6, S7+S8, S3, S9, S-decide, S10, T3, T5 down-payment), what was deferred (S4 sentinel deletion — out-of-band IR precondition unmet) and the remaining quarantined backlog, the S7+S8 BC-major migration notes, the MAJOR version implication, and the final all-green gate state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
yisding
changed the title
…gment_spans) The benchmarks only exercised general English prose, where the abbreviation phase the V2 PeriodClassifier reworked is a small slice. Add an abbreviation-dense sample: a LEGAL input through the en_legal profile (latency suite) and a DENSE size shared across ours/pysbd/punkt (competitive suite). Also benchmark segment_spans(), now the canonical span API after char_span was removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The '(next release must be 0.1.0 minor)' note is superseded — the v2 branch introduces breaking API changes, so the next release is a major. Keep the general manual-version-selection guidance; drop the specific stale hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The V2 PeriodClassifier realizes a PROTECT decision globally per
(abbr, follower) unit by re-anchoring a follower-independent prepositive
suffix. For en_legal's starter-aware court abbreviations (Cir., Bankr., ...)
the boundary decision is position-dependent (it reads the per-occurrence
follower via _follower_is_likely_sentence_start), so global realization
re-protected every "<abbr> <whitespace>" on a line once any single
occurrence joined — wrongly merging a sibling that should end a sentence
("The 9th Cir. held the 2nd Cir. The panel reversed." collapsed to one
segment in aggressive mode).
en_legal now uses AbbrPolicy(realize_per_occurrence=True) (the russian
precedent), anchoring each occurrence's edit to its own period from its own
context. This is byte-identical to the global model for every
position-independent branch, so other modes/abbreviations are unchanged.
Also fold in review cleanups: drop the dead Candidate.occ_start field and
the en_es_zh _split_on_combined_sentence_boundary copy (parameterize the
shared _split_on_uppercase_boundary), correct the _classifier_cache type
annotation, and remove the stale char_span filterwarnings/coverage notes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…riod regex sentinel-aware
Two robustness/cleanliness fixes for latent edge cases (no behavior change on
shipping data; full suite identical):
* Add a real ``Candidate.abbr_start`` (the abbreviation's start index on the
line, == ``period_idx - len(am_stripped)`` since ``re.IGNORECASE`` folds 1:1).
The per-occurrence policies — russian ``ср.`` and the slovak/bulgarian
whole-span splice — were each re-deriving that offset with subtly different
``.strip()`` / ``_elision_strip`` dances; they now read the single computed
field. Drops the whole-span ``_elision_strip`` no-op (stored forms never carry
a leading elision char, so stripping it could only mis-anchor a hypothetical
elision whole-span language).
* Make Kazakh ``MULTI_PERIOD_ABBREVIATION_REGEX`` accept the ``∯`` sentinel in
its separators/terminator (``[.∯]``), mirroring the base
``Common.MULTI_PERIOD_ABBREVIATION_REGEX`` and Kazakh's own
``protect_multi_period_abbreviations_before_parenthesis``. A declared dotless
multi-period abbreviation ("т.с.с") has its trailing period protected to "∯"
by the classifier before this pass runs; the old ``[.]``-only form could only
re-find the token via a lucky shorter-prefix match. Adds a guard test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…engine # Conflicts: # benchmarks/test_latency_codspeed.py
Relocate the tests/v2/ suite into functional homes and strip the transitional
"v2" engine label plus references to the removed analysis/ folder.
- Move segment_snapshot.{py,json} + test_segment_snapshot.py to tests/regression/;
test_classifier_en.py to tests/test_period_classifier_en.py; the English
abbreviation corpus and its test to tests/abbreviation_corpus_en.py and
tests/test_abbreviation_corpus_en.py. Remove the tests/v2/ package and fix the
moved modules' imports and CLI references.
- Reword ~40 comments/docstrings across sentencesplit/, benchmarks/, and tests/
to drop the "V2" engine label (legit "v2.0" version test data is untouched),
and remove pointers to the deleted analysis/ plan/roadmap/RFC docs along with
the dead ruff and .gitignore entries for analysis/. Drop the now-dead oracle
adapter method on AbbreviationReplacer.
- Fix collection of tests/test_corpus_compare_segmenters.py under plain pytest by
setting pythonpath=["."], so the in-tree benchmarks/ shadows any stale installed
copy lingering in site-packages.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reorganize tests/ by the role each test plays, and act on a workflow-
adjudicated audit of droppable test cases (each coverage-removing decision
adversarially verified before acting).
Layout:
- tests/contract/ public-API, property, and cross-cutting invariant tests
- tests/unit/ per-module unit tests
- tests/meta/ packaging / registry / import-hygiene guards
(incl. four reclassified out of tests/regression/)
- tests/regression/ now holds only true bug-guards + the snapshot/gate infra
- tests/data/abbreviation_corpus_en.py corpus data colocated under data/
conftest.py and helpers.py stay at tests/ root so fixtures and the
`tests.helpers` imports keep working across the new subdirs.
Prune / fix:
- drop tests/test_punctuation_replacer.py (asserted only internal sentinels;
every delimiter path is covered behaviorally by segment()-level tests)
- rewrite the processor phase-list tests from exact __name__-tuple pins to
unordered membership + a CJK-phase wiring guard + a behavioral method test
- remove duplicate cases (challenging 119i, the cross-file "Eq. 5" row, four
Armenian rows, Italian/Spanish duplicates)
- drop the brittle Kazakh "unprotected" characterization (already covered by
tests/regression/test_abbr_dot_normalization.py)
- trim the dead-flag German test and correct its docstring
- drop the docstring-substring asserts in test_processor_robustness.py
- reduce test_danish.py to its Danish-divergent rows; regenerate the segment
snapshot (da-only removals, no output changes)
Move tests/test_corpus_compare_segmenters.py to benchmarks/ (it tests the
never-shipped harness) and wire it into CI explicitly since it now sits
outside testpaths=["tests"].
Full suite green (10502 passed, 14 skipped, 113 xfailed); ruff, format,
mypy, and the snapshot diff all clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test: reorganize suite into role-based dirs and prune redundant cases
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A V2 rewrite of the abbreviation engine plus a broad tech-debt paydown taken as the v2 opportunity.
The legacy "segmentation as a sequence of global
re.subrewrites, one per abbreviation, carried in-band via∯" model is replaced by a single-passPeriodClassifier(sentencesplit/period_classifier.py): each candidate period is classified once, from the original text, then all decisions are applied in one rebuild. Per-language behaviour is anAbbrPolicyco-located in eachlang/*.py(engine holds only shared machinery). The motivation is correctness + maintainability, not speed — the perf delta is small by design (seeanalysis/V2_RFC_EVALUATION.md).Segmenter(char_span=True).segment(t)→list[TextSpan]Segmenter().segment_spans(t)→list[TextSpan]segment()returnedlist[str] | list[TextSpan](depended onchar_span)segment(t)→ alwayslist[str]char_spanconstructor argsegment_spans_with_lookahead()→ baretuple[list[TextSpan], bool]SegmentLookahead[TextSpan](nowGeneric[T])Newly exported at top level:
InvalidConfigurationError,UnknownLanguageError,register_language,unregister_language.What landed
PeriodClassifieracross all 26 language codes; legacy per-occurrence engine deleted; English parity-exact; 3 prior correctness quirks fixed.analysis/V2_REFACTOR_ROADMAP.md):LanguageProfile; extractedboundary_resplit+ a shared normalization helper; promoted shared whole-span policy tolang/common/(killed the only lang→lang private import); non-ASCII multi-period abbreviations now recognised.sorted(set())abbreviation lists + a behavioural data-lint (quarantined backlog of known gaps).segment()snapshot into CI as a regression gate; retired the now-stale legacy oracle; addedprocessor/period_classifierunit suites and quarantined property tests; standardized onassert_segments.Deferred (remaining backlog)
&X&punctuation/ellipsis sentinel family being moved out-of-band first (premature deletion corruptsclean=output — the trap the roadmap flagged). Lead item of a follow-up.Verification
segment()snapshot: byte-identical (behaviour-neutral except the explicitly-adjudicated non-ASCII-abbreviation improvements). Ruff clean.Version
Because of the breaking API changes, this must be released as a major (the prior "next release = 0.1.0 minor" note in
CLAUDE.mdis superseded). Full migration notes inanalysis/V2_TECHDEBT_PAYDOWN_REPORT.md.🤖 Generated with Claude Code