From beabe9c41d438b64a1e201eb478e0caa3f374f95 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 02:07:29 +0000 Subject: [PATCH 01/11] docs: add short/medium latency investigation + action plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add benchmarks/phase_profile.py (a reusable per-phase wall-time attribution harness) and analysis/SHORT_STRING_LATENCY_PLAN.md, which uses it plus a pySBD structural comparison to localize the ~22% short/medium gap vs pySBD. Findings: two fixed-cost phases dominate every call — abbreviation replacement (~40%, mostly the pure-Python Aho-Corasick scan) and list-item detection (~18%) — versus pySBD's C-level 'in' short-circuits, lock-free reads, and fewer always-on passes. The plan lays out a prioritized, behavior-preserving (byte-identical) optimization sequence (P1 abbreviation scan fast-path, P2 gate always-on regexes, P3 drop the per-call lock, P4 list-item guards) with guardrails. --- analysis/SHORT_STRING_LATENCY_PLAN.md | 185 ++++++++++++++++++++++++++ benchmarks/phase_profile.py | 127 ++++++++++++++++++ 2 files changed, 312 insertions(+) create mode 100644 analysis/SHORT_STRING_LATENCY_PLAN.md create mode 100644 benchmarks/phase_profile.py diff --git a/analysis/SHORT_STRING_LATENCY_PLAN.md b/analysis/SHORT_STRING_LATENCY_PLAN.md new file mode 100644 index 0000000..f214422 --- /dev/null +++ b/analysis/SHORT_STRING_LATENCY_PLAN.md @@ -0,0 +1,185 @@ +# Short/Medium `segment()` Latency — Investigation & Action Plan + +**Goal:** close the ~22% per-call latency gap vs pySBD on short/medium English +text (surfaced by the competitive CodSpeed benchmarks in #71), **without** +changing segmentation output (byte-identical) or losing the Aho-Corasick +automaton's advantage on very large abbreviation lists. + +This is a living plan. It pairs with the reusable exploration harness +`benchmarks/phase_profile.py` (the "workflow"): run it to re-attribute per-call +cost to pipeline phases after each change. + +--- + +## 1. How to reproduce the measurements (the workflow) + +```bash +# Per-phase wall-time attribution for one input size: +uv run python benchmarks/phase_profile.py --size short --iters 20000 +uv run python benchmarks/phase_profile.py --size medium --iters 12000 + +# End-to-end latency (and cProfile hot functions): +uv run python benchmarks/latency_baseline.py --iters 4000 [--profile] + +# Deterministic CI verdict (instruction count) vs pySBD/punkt: +# the competitive benchmarks run on every PR via .github/workflows/codspeed.yml +# (benchmarks/test_competitive_codspeed.py): watch test_segment[short-ours] etc. +``` + +`phase_profile.py` wraps each Processor / AbbreviationReplacer / Segmenter stage +with a timer. The `*wrapper` rows (e.g. `abbr: replace (whole)`, +`split_into_segments`) **contain** the rows below them — do not sum across them. + +--- + +## 2. Where the time goes (measured, Python 3.13, warm) + +`segment("Dr. Smith went to Washington. … Sen. Jones.")` — **short, 87 chars, +~0.40 ms/call:** + +| phase | ms/call | % of call | +|-------|--------:|----------:| +| `replace_abbreviations` (text phase) | 0.156 | **39%** | +| ↳ `search_for_abbreviations_in_string` (automaton scan + per-abbr sub) | 0.091 | 23% | +| ↳ `apply_ampm_boundary_rules` | 0.026 | 7% | +| `_mark_list_item_boundaries` (ListItemReplacer.add_line_break) | 0.078 | **20%** | +| `split_into_segments` (wrapper: boundary phases + postprocess) | 0.090 | 22% | +| ↳ `between_punctuation` | 0.013 | 3% | +| `replace_numbers` / `_protect_special_tokens` | ~0.010 ea | 2.5% ea | + +**Medium (198 chars, ~0.88 ms/call)** is the same shape: `replace_abbreviations` +**43%** (of which `search_for_abbreviations_in_string` **31%**), +`_mark_list_item_boundaries` **18%**. + +**Conclusion:** two fixed-cost phases dominate and run on *every* call regardless +of whether the input needs them: +1. **Abbreviation replacement (~40%)** — dominated by the abbreviation scan. +2. **List-item detection (~18%)** — runs 4 sub-formatters even on non-list text. + +These ~58% are the structural gap. The `between_punctuation` / zero-width work +already addressed in #72 was <4% on short input (it helped large text instead). + +--- + +## 3. Why pySBD is cheaper per call (structural delta) + +pySBD does **not** have fewer phases (~20 full-text passes, similar to ours). +The difference is per-pass constant cost and cheap short-circuits: + +1. **Abbreviation loop short-circuits in C before any regex.** pySBD iterates its + 188 English abbreviations and does `if stripped not in lowered: continue` + (`pysbd/abbreviation_replacer.py:82`). For short text ~186/188 fail the + C-level `str.__contains__` instantly; only 0–2 reach a regex. + **Ours** replaced this with an Aho-Corasick automaton + (`sentencesplit/abbreviation_replacer.py:535-564`) whose `search()` + (`:63-76`) runs a **pure-Python per-character state-machine loop over the + whole lowered text on every call** — slower than a handful of C `in` checks + on short strings. The automaton wins only when the abbreviation list is huge; + for ~200 entries on short text it is a net loss. +2. **Always-on extra passes ours added** in `replace()` that pySBD lacks: + `_COMPACT_AMPM_RE`, `_UPPERCASE_INITIALISM_BOUNDARY_RE`, allcaps-imprint + protection, non-ASCII a.m./p.m. restores (`:311-357`). Each is an + unconditional full-text regex pass. +3. **Per-call `RLock` acquisition.** Every `AbbreviationReplacer.__init__` + (`:55-58`) takes `_cache_lock` to fetch the cached `_data`, on every call. + pySBD has no such lock. + +What we already match: a no-punctuation boundary guard +(`processor.py:691` `check_for_punctuation`) skips the boundary pipeline for +segments lacking sentence-ending punctuation — same idea as pySBD. + +--- + +## 4. Action plan (prioritized by leverage × safety) + +Every item is **behavior-preserving**: the validation bar is byte-identical +`segment()`/`segment_spans()` output (full suite incl. Golden Rules stays green, +and a cross-corpus equivalence check). Implement **test-first**, on its own +branch, auto-revert if Golden Rules or the suite regress (mirror the +`improve-sentencesplit` discipline). Measure each with `phase_profile.py` and the +competitive CodSpeed benchmark. + +### P1 — Abbreviation scan fast-path (target: the ~25–30% `search_in_string` cost) + +The automaton's job is to find the **set** of abbreviations present, then run a +per-occurrence `re.sub`. Three candidate mechanisms, to prototype in order: + +- **P1a — Single compiled alternation regex (preferred to prototype first).** + Replace the pure-Python `search()` char loop with one cached + `re.compile("|".join(escaped_abbrevs))` (anchored as the automaton expects) and + a single C-level `.findall`/`.finditer` to discover the present set. One C pass + vs a Python per-char loop should beat both the automaton and pySBD's 188 `in`s. + *Risk:* must reproduce the automaton's matched-set **exactly** (case-folding, + overlap, boundary semantics). *DoD:* for a large corpus, the present-set is + identical to the automaton's for every line. +- **P1b — Length-gated fallback.** Below a small text-length threshold, use a + pySBD-style `in`-guarded loop (cheap on short text); above it, keep the + automaton (wins on long text / big lists). *Risk:* two code paths must produce + identical sets — property-test them against each other. +- **P1c — Cheap pre-filter.** Skip the scan entirely when a one-pass check proves + no abbreviation can match (e.g. no `.` in the line). Cheapest, smallest gain; + stack on top of P1a/P1b. + +### P2 — Gate the always-on `replace()` regexes (low risk, modest gain) + +Guard each unconditional pass in `AbbreviationReplacer.replace()` behind a cheap +`in`/structural check so it only runs when it can match: +- `_COMPACT_AMPM_RE` — only if a digit is adjacent to `a/p` + `.` + `m`. +- `_UPPERCASE_INITIALISM_BOUNDARY_RE`, non-ASCII a.m./p.m. restores — only if the + `∯` sentinel is present (it only exists after an abbreviation matched). +- allcaps-imprint — only if the text has an all-caps run. +*Risk:* low (each regex already requires the guarded condition). *DoD:* suite +green + byte-identical corpus output. + +### P3 — Remove the per-call lock on the abbreviation-data read (low risk, small gain) + +`AbbreviationReplacer.__init__` takes an `RLock` every call just to read the +per-class `_data`. Use a lock-free fast read (double-checked: read the dict +first, only lock to build on miss) or resolve `_data` once and stash it on the +Abbreviation class / LanguageProfile. *Risk:* low; preserve thread-safety of the +first build. *DoD:* suite green + a concurrency smoke test. + +### P4 — List-item detection guards (target: the ~18% list phase) + +`_mark_list_item_boundaries` constructs a fresh `ListItemReplacer` and runs 4 +sub-formatters every call. Safe reductions: +- **P4a — Trigger-char guards per sub-formatter.** Skip the numbered-list passes + when the text has no digit; skip the parens passes when there is no `(`/`)`; + skip alphabetical/roman passes when no single letter precedes `.`/`)`. Each + formatter's regex already requires those chars, so skipping is a no-op. +- **P4b — Early-out in `iterate_alphabet_array`** when `re.findall` returns empty + (avoid building the 26-entry `alphabet_index` dict and the lower/filter work). +*Risk:* low-medium (list logic is subtle; guards must be on chars the regexes +require). *DoD:* suite green; the list-heavy Golden Rules unchanged. + +### P5 (stretch) — a.m./p.m. rule gating + +`apply_ampm_boundary_rules` is ~5–7%. Guard the rule set on the presence of an +`m`/`.`-adjacent pattern. Lower priority; do after P1–P4 re-measure. + +--- + +## 5. Sequencing, guardrails, and non-goals + +**Sequence:** P2 + P3 first (cheap, low-risk warm-up + de-risks the harness), +then P4 (clear ~18% target), then P1 (biggest but riskiest — prototype P1a, +fall back to P1b). Re-run `phase_profile.py` + competitive CodSpeed after each. + +**Guardrails (non-negotiable):** +- Byte-identical `segment()`/`segment_spans()` output. Full suite + Golden Rules + green is the gate; add a corpus-wide equivalence diff (old vs new) per change. +- Each change is reverted automatically if the suite or Golden Rules regress. +- Keep changes isolated per branch/PR so CodSpeed attributes each delta. + +**Non-goals / risks to avoid:** +- Do **not** regress the automaton's advantage on very large abbreviation lists + or the combined `en_es_zh` profile — P1 must keep (or length-gate to) the + automaton for the large-list case. +- Do not change boundary semantics to chase speed. This is a pure + constant-factor effort; if a change can't be made byte-identical, it is out of + scope for this plan. + +**Expected envelope:** P2+P3+P4 are low-risk and should recover a meaningful +slice of the list phase + always-on passes (rough order ~10–20% of the call). +P1 is where the abbreviation ~30% lives and is the swing item; treat its gain as +unproven until P1a is prototyped and shown byte-identical. diff --git a/benchmarks/phase_profile.py b/benchmarks/phase_profile.py new file mode 100644 index 0000000..42d80b6 --- /dev/null +++ b/benchmarks/phase_profile.py @@ -0,0 +1,127 @@ +"""Phase-level latency profiler for the segmentation pipeline. + +Where ``latency_baseline.py`` answers "how fast is a call?", this answers "which +*phase* of the call is the time in?" — it wraps each Processor / AbbreviationReplacer +/ Segmenter stage with a timer and reports per-phase wall time, call count, and +share of the total, for short / medium / large input. + +This is the exploration harness for the short/medium-vs-pySBD latency gap: the +fixed per-call cost (the phases that run on every call regardless of input) is +what dominates short strings, so this attributes that cost to concrete phases. + +Run with: + uv run python benchmarks/phase_profile.py + uv run python benchmarks/phase_profile.py --size short --iters 20000 +""" + +from __future__ import annotations + +import argparse +import time +from collections import defaultdict + +from sentencesplit import Segmenter +from sentencesplit.abbreviation_replacer import AbbreviationReplacer +from sentencesplit.processor import Processor +from sentencesplit.segmenter import Segmenter as _Seg + +SHORT = "Dr. Smith went to Washington. He arrived on Jan. 5th at 3 p.m. and met with Sen. Jones." +MEDIUM = ( + "Dr. Smith went to Washington. He arrived on Jan. 5th at 3 p.m. " + "The model is GPT 3.1 and it is fast. That is all for now. Goodbye. " + "She paid $4.50 for the U.S. edition (vol. 2, p. 17). Mr. Lee agreed." +) +LARGE = " ".join([MEDIUM] * 20) +_SAMPLES = {"short": SHORT, "medium": MEDIUM, "large": LARGE} + +# (class, method) pairs to attribute. Grouped by pipeline stage; the label is what +# the report prints. Only methods that exist are wrapped. +_TARGETS = [ + # --- text-processing phases (run once per call over the whole text) --- + (Processor, "_normalize_newlines", "text: normalize_newlines"), + (Processor, "_mark_list_item_boundaries", "text: list_item_boundaries"), + (Processor, "replace_abbreviations", "text: replace_abbreviations"), + (Processor, "replace_numbers", "text: replace_numbers"), + (Processor, "replace_continuous_punctuation", "text: continuous_punct"), + (Processor, "replace_periods_before_numeric_references", "text: numeric_refs"), + (Processor, "_protect_special_tokens", "text: special_tokens"), + # --- boundary-processing phases (per segment) --- + (Processor, "_ensure_terminal_marker", "bound: terminal_marker"), + (Processor, "_apply_exclamation_word_rules", "bound: exclamation_words"), + (Processor, "between_punctuation", "bound: between_punctuation"), + (Processor, "_apply_double_punctuation_rules", "bound: double_punct"), + (Processor, "_apply_quotation_punctuation_rules", "bound: quotation_punct"), + (Processor, "_replace_list_parens", "bound: list_parens"), + (Processor, "sentence_boundary_punctuation", "bound: sentence_boundary"), + # --- post-split passes --- + (Processor, "split_into_segments", "post: split_into_segments (incl. boundary)"), + (Processor, "_resplit_segments", "post: resplit_segments"), + (Processor, "_merge_orphan_fragments", "post: merge_orphans"), + # --- abbreviation internals (the suspected fixed cost) --- + (AbbreviationReplacer, "replace", "abbr: replace (whole)"), + (AbbreviationReplacer, "search_for_abbreviations_in_string", "abbr: search_in_string"), + (AbbreviationReplacer, "apply_ampm_boundary_rules", "abbr: ampm_rules"), + # --- span mapping (segmenter) --- + (_Seg, "_match_spans", "span: match_spans"), +] + +_stats: dict[str, list[float | int]] = defaultdict(lambda: [0.0, 0]) # label -> [total_s, calls] + + +def _wrap(cls, method_name: str, label: str) -> None: + original = getattr(cls, method_name, None) + if original is None: + return + + def timed(self, *args, _orig=original, _label=label, **kwargs): + t0 = time.perf_counter() + try: + return _orig(self, *args, **kwargs) + finally: + rec = _stats[_label] + rec[0] += time.perf_counter() - t0 + rec[1] += 1 + + setattr(cls, method_name, timed) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--size", choices=["short", "medium", "large"], default="short") + ap.add_argument("--iters", type=int, default=20000) + args = ap.parse_args() + + for cls, name, label in _TARGETS: + _wrap(cls, name, label) + + seg = Segmenter(language="en", clean=False, char_span=False) + text = _SAMPLES[args.size] + for _ in range(5): + seg.segment(text) + _stats.clear() + + t0 = time.perf_counter() + for _ in range(args.iters): + seg.segment(text) + total = time.perf_counter() - t0 + + print(f"phase profile size={args.size} iters={args.iters} ({len(text)} chars)") + print(f"total: {total * 1000 / args.iters:.4f} ms/call ({total:.2f}s)") + print("=" * 78) + print(f"{'phase':<46}{'ms/call':>10}{'calls/seg':>10}{'% total':>10}") + print("-" * 78) + rows = sorted(_stats.items(), key=lambda kv: kv[1][0], reverse=True) + for label, (total_s, calls) in rows: + ms_per_call = total_s * 1000 / args.iters + calls_per_seg = calls / args.iters + pct = 100 * total_s / total + # split_into_segments wraps the boundary phases, so its % overlaps them; + # mark it so the breakdown is not misread as additive. + note = " *wrapper" if "split_into_segments" in label or label == "abbr: replace (whole)" else "" + print(f"{label:<46}{ms_per_call:>10.4f}{calls_per_seg:>10.1f}{pct:>9.1f}%{note}") + print("-" * 78) + print("* wrapper rows contain the rows below them; do not sum across them.") + + +if __name__ == "__main__": + main() From 19fc9784fd15e78f510cab40388f98c7b9173dfa Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 02:17:44 +0000 Subject: [PATCH 02/11] docs: correct the abbreviation-scan premise in the latency plan Measurement (benchmarks/abbr_scan_compare.py) disproves the earlier assumption that our Aho-Corasick scan is the short-string bottleneck: input AC in winner tiny (15c) 2.4us 8.2us AC 3.4x short (87c) 12.2us 13.5us AC ~tied medium(198) 30.5us 21.0us in 1.4x large (4k) 619us 286us in 2.2x Our pure-Python AC is faster than a pySBD-style loop on short input; only overtakes around ~100-150 chars. The scan is ~3% of the call, so the dropped P1 (replace the scan) would have slowed short text down. Re-prioritize: P1 = gate the always-on abbreviation rule passes (the real ~25 re.sub/call cost), P2 = list-item guards, P3 = drop the per-call lock. Discovery scan is explicitly left alone. --- analysis/SHORT_STRING_LATENCY_PLAN.md | 152 +++++++++++++++----------- benchmarks/abbr_scan_compare.py | 72 ++++++++++++ 2 files changed, 159 insertions(+), 65 deletions(-) create mode 100644 benchmarks/abbr_scan_compare.py diff --git a/analysis/SHORT_STRING_LATENCY_PLAN.md b/analysis/SHORT_STRING_LATENCY_PLAN.md index f214422..e065267 100644 --- a/analysis/SHORT_STRING_LATENCY_PLAN.md +++ b/analysis/SHORT_STRING_LATENCY_PLAN.md @@ -53,12 +53,42 @@ with a timer. The `*wrapper` rows (e.g. `abbr: replace (whole)`, **Conclusion:** two fixed-cost phases dominate and run on *every* call regardless of whether the input needs them: -1. **Abbreviation replacement (~40%)** — dominated by the abbreviation scan. +1. **Abbreviation replacement (~40%)** — but NOT the scan (see §2a); it is the + ~25 `re.sub` per call: per-occurrence replacements + always-on rule passes. 2. **List-item detection (~18%)** — runs 4 sub-formatters even on non-list text. These ~58% are the structural gap. The `between_punctuation` / zero-width work already addressed in #72 was <4% on short input (it helped large text instead). +### 2a. The Aho-Corasick scan is NOT the bottleneck (measured) + +An earlier draft of this plan assumed the abbreviation *scan* (Aho-Corasick) was +the cost and proposed replacing it with pySBD's `in`-loop. **Direct measurement +disproves that.** Our pure-Python automaton `search()` vs a pySBD-style +`in`-filter loop over the same 199 abbreviations (identical match sets): + +| input | AC | `in`-loop | winner | +|-------|---:|----------:|--------| +| tiny (15c) | 2.4 µs | 7.2 µs | **AC 3.0×** | +| short (87c) | 11.3 µs | 12.0 µs | **AC ~tied** | +| medium (198c)| 25.3 µs | 18.0 µs | `in` 1.4× | +| large (4k) | 497 µs | 229 µs | `in` 2.2× | + +Our AC is faster on short input; `in` only overtakes around ~100–150 chars. +(The crossover is inverted from textbook AC-wins-on-long because the automaton is +**pure Python**: its per-char loop has a large constant, so on long text 199 +C-level `in` scans beat thousands of Python iterations.) + +Crucially, the AC scan is only **~11 µs ≈ 3% of the 400 µs short call**. Swapping +it for `in` would *slow short down* and save nothing meaningful. The 39% +abbreviation cost is the **~25 `re.sub`/call** that follow discovery: +`scan_for_replacements` (a global `re.sub` per matched abbreviation), +`replace_multi_period_abbreviations`, and the always-on rule passes +(`PossessiveAbbreviationRule`, `SingleLetterAbbreviationRules`, `_COMPACT_AMPM_RE`, +`_UPPERCASE_INITIALISM_BOUNDARY_RE`, allcaps-imprint, the a.m./p.m. rules). +pySBD runs the per-occurrence regex too — so the gap is our **extra always-on +passes**, not the discovery mechanism. + --- ## 3. Why pySBD is cheaper per call (structural delta) @@ -66,17 +96,14 @@ already addressed in #72 was <4% on short input (it helped large text instead). pySBD does **not** have fewer phases (~20 full-text passes, similar to ours). The difference is per-pass constant cost and cheap short-circuits: -1. **Abbreviation loop short-circuits in C before any regex.** pySBD iterates its - 188 English abbreviations and does `if stripped not in lowered: continue` - (`pysbd/abbreviation_replacer.py:82`). For short text ~186/188 fail the - C-level `str.__contains__` instantly; only 0–2 reach a regex. - **Ours** replaced this with an Aho-Corasick automaton - (`sentencesplit/abbreviation_replacer.py:535-564`) whose `search()` - (`:63-76`) runs a **pure-Python per-character state-machine loop over the - whole lowered text on every call** — slower than a handful of C `in` checks - on short strings. The automaton wins only when the abbreviation list is huge; - for ~200 entries on short text it is a net loss. -2. **Always-on extra passes ours added** in `replace()` that pySBD lacks: +1. **Abbreviation discovery is a wash on short text.** pySBD short-circuits with + C-level `in` (`pysbd/abbreviation_replacer.py:82`); ours uses a pure-Python + Aho-Corasick automaton (`sentencesplit/abbreviation_replacer.py:535-564`). Per + §2a these cost ~the same on short input (AC is actually slightly faster), so + this is **not** where the gap is — both then run the same per-occurrence regex + work. Discovery is ~3% of the call either way. +2. **Always-on extra passes ours added** in `replace()` that pySBD lacks (this, + not the scan, is the abbreviation-phase cost): `_COMPACT_AMPM_RE`, `_UPPERCASE_INITIALISM_BOUNDARY_RE`, allcaps-imprint protection, non-ASCII a.m./p.m. restores (`:311-357`). Each is an unconditional full-text regex pass. @@ -99,71 +126,63 @@ branch, auto-revert if Golden Rules or the suite regress (mirror the `improve-sentencesplit` discipline). Measure each with `phase_profile.py` and the competitive CodSpeed benchmark. -### P1 — Abbreviation scan fast-path (target: the ~25–30% `search_in_string` cost) - -The automaton's job is to find the **set** of abbreviations present, then run a -per-occurrence `re.sub`. Three candidate mechanisms, to prototype in order: - -- **P1a — Single compiled alternation regex (preferred to prototype first).** - Replace the pure-Python `search()` char loop with one cached - `re.compile("|".join(escaped_abbrevs))` (anchored as the automaton expects) and - a single C-level `.findall`/`.finditer` to discover the present set. One C pass - vs a Python per-char loop should beat both the automaton and pySBD's 188 `in`s. - *Risk:* must reproduce the automaton's matched-set **exactly** (case-folding, - overlap, boundary semantics). *DoD:* for a large corpus, the present-set is - identical to the automaton's for every line. -- **P1b — Length-gated fallback.** Below a small text-length threshold, use a - pySBD-style `in`-guarded loop (cheap on short text); above it, keep the - automaton (wins on long text / big lists). *Risk:* two code paths must produce - identical sets — property-test them against each other. -- **P1c — Cheap pre-filter.** Skip the scan entirely when a one-pass check proves - no abbreviation can match (e.g. no `.` in the line). Cheapest, smallest gain; - stack on top of P1a/P1b. - -### P2 — Gate the always-on `replace()` regexes (low risk, modest gain) - -Guard each unconditional pass in `AbbreviationReplacer.replace()` behind a cheap -`in`/structural check so it only runs when it can match: -- `_COMPACT_AMPM_RE` — only if a digit is adjacent to `a/p` + `.` + `m`. -- `_UPPERCASE_INITIALISM_BOUNDARY_RE`, non-ASCII a.m./p.m. restores — only if the - `∯` sentinel is present (it only exists after an abbreviation matched). -- allcaps-imprint — only if the text has an all-caps run. -*Risk:* low (each regex already requires the guarded condition). *DoD:* suite -green + byte-identical corpus output. +### ~~P1 — Abbreviation scan fast-path~~ — DROPPED (premise disproved, see §2a) -### P3 — Remove the per-call lock on the abbreviation-data read (low risk, small gain) +The original P1 (replace the Aho-Corasick scan with pySBD's `in`-loop or a +compiled alternation) is **invalid**: measurement shows the AC scan is ~3% of the +call and is already faster than `in` on short input. Switching would slow short +text down. Do not pursue. The abbreviation-phase cost lives in the always-on +passes (now P1, below), not discovery. -`AbbreviationReplacer.__init__` takes an `RLock` every call just to read the -per-class `_data`. Use a lock-free fast read (double-checked: read the dict -first, only lock to build on miss) or resolve `_data` once and stash it on the -Abbreviation class / LanguageProfile. *Risk:* low; preserve thread-safety of the -first build. *DoD:* suite green + a concurrency smoke test. +*Optional, low priority:* a length-gated `in` fallback would help medium/large +discovery (~7 µs on medium, ~270 µs on 4k) but hurts short, so only worth it as a +threshold switch if medium/large latency becomes a target — not for this goal. -### P4 — List-item detection guards (target: the ~18% list phase) +### P1 — Gate the always-on abbreviation `replace()` passes (was P2; now top) + +`AbbreviationReplacer.replace()` runs ~25 `re.sub`/call on short text, several of +them unconditional rule passes that almost never match. Guard each behind a cheap +`in`/structural check so it only runs when it can fire: +- `_COMPACT_AMPM_RE` — only if a digit is adjacent to `a/p`+`.`+`m`. +- `_UPPERCASE_INITIALISM_BOUNDARY_RE`, non-ASCII a.m./p.m. restores — only if the + `∯` sentinel is present (it only exists after an abbreviation matched). +- allcaps-imprint — only if the text has a 2+ all-caps run. +- the a.m./p.m. rule set (`apply_ampm_boundary_rules`, ~5–7%) — gate on an + `[ap]·m` pattern being present. +This is the largest *safe* slice of the abbreviation phase. *Risk:* low (each +regex already requires the guarded condition). *DoD:* suite green + byte-identical +corpus output. + +### P2 — List-item detection guards (target: the ~18% list phase) `_mark_list_item_boundaries` constructs a fresh `ListItemReplacer` and runs 4 sub-formatters every call. Safe reductions: -- **P4a — Trigger-char guards per sub-formatter.** Skip the numbered-list passes +- **P2a — Trigger-char guards per sub-formatter.** Skip the numbered-list passes when the text has no digit; skip the parens passes when there is no `(`/`)`; skip alphabetical/roman passes when no single letter precedes `.`/`)`. Each formatter's regex already requires those chars, so skipping is a no-op. -- **P4b — Early-out in `iterate_alphabet_array`** when `re.findall` returns empty +- **P2b — Early-out in `iterate_alphabet_array`** when `re.findall` returns empty (avoid building the 26-entry `alphabet_index` dict and the lower/filter work). *Risk:* low-medium (list logic is subtle; guards must be on chars the regexes require). *DoD:* suite green; the list-heavy Golden Rules unchanged. -### P5 (stretch) — a.m./p.m. rule gating +### P3 — Remove the per-call lock on the abbreviation-data read (low risk, small gain) -`apply_ampm_boundary_rules` is ~5–7%. Guard the rule set on the presence of an -`m`/`.`-adjacent pattern. Lower priority; do after P1–P4 re-measure. +`AbbreviationReplacer.__init__` takes an `RLock` every call just to read the +per-class `_data`. Use a lock-free fast read (double-checked: read the dict +first, only lock to build on miss) or resolve `_data` once and stash it on the +Abbreviation class / LanguageProfile. *Risk:* low; preserve thread-safety of the +first build. *DoD:* suite green + a concurrency smoke test. --- ## 5. Sequencing, guardrails, and non-goals -**Sequence:** P2 + P3 first (cheap, low-risk warm-up + de-risks the harness), -then P4 (clear ~18% target), then P1 (biggest but riskiest — prototype P1a, -fall back to P1b). Re-run `phase_profile.py` + competitive CodSpeed after each. +**Sequence:** P3 first (mechanical, lowest risk, de-risks the harness loop), then +P1 (gate the always-on abbreviation passes — the biggest *safe* slice of the 40% +abbreviation phase), then P2 (the ~18% list phase). Re-run `phase_profile.py` + +competitive CodSpeed after each. Note the discovery scan is deliberately left +alone (see §2a). **Guardrails (non-negotiable):** - Byte-identical `segment()`/`segment_spans()` output. Full suite + Golden Rules @@ -172,14 +191,17 @@ fall back to P1b). Re-run `phase_profile.py` + competitive CodSpeed after each. - Keep changes isolated per branch/PR so CodSpeed attributes each delta. **Non-goals / risks to avoid:** -- Do **not** regress the automaton's advantage on very large abbreviation lists - or the combined `en_es_zh` profile — P1 must keep (or length-gate to) the - automaton for the large-list case. +- Do **not** touch the abbreviation *discovery* scan — it is faster than `in` on + short input and only ~3% of the call (§2a). The Aho-Corasick automaton stays. - Do not change boundary semantics to chase speed. This is a pure constant-factor effort; if a change can't be made byte-identical, it is out of scope for this plan. -**Expected envelope:** P2+P3+P4 are low-risk and should recover a meaningful -slice of the list phase + always-on passes (rough order ~10–20% of the call). -P1 is where the abbreviation ~30% lives and is the swing item; treat its gain as -unproven until P1a is prototyped and shown byte-identical. +**Expected envelope:** all three items (P1 gate always-on passes, P2 list guards, +P3 lock) are low-risk and target the always-on `re.sub` count + the list phase. +Together they address roughly the abbreviation always-on slice (a chunk of the +40%) plus the ~18% list phase. Treat individual gains as unproven until measured +per-change with `phase_profile.py` and the competitive CodSpeed run; the honest +expectation is incremental (single-digit to low-double-digit %), not a dramatic +close, because much of the abbreviation phase is genuine per-occurrence regex +work that pySBD also pays. diff --git a/benchmarks/abbr_scan_compare.py b/benchmarks/abbr_scan_compare.py new file mode 100644 index 0000000..4783058 --- /dev/null +++ b/benchmarks/abbr_scan_compare.py @@ -0,0 +1,72 @@ +"""Head-to-head: our Aho-Corasick abbreviation scan vs a pySBD-style `in` loop. + +Settles which abbreviation-*discovery* mechanism is faster at each input size. +Both find the identical set of present abbreviations; this times only that +discovery step (not the per-occurrence regex replacement that follows it). + +Finding (see analysis/SHORT_STRING_LATENCY_PLAN.md §2a): our pure-Python +automaton is faster on short input; the `in` loop only overtakes around +~100-150 chars. So the scan is *not* the short-string bottleneck — it is ~3% of +the call — and must not be replaced. + +Run with: + uv run python benchmarks/abbr_scan_compare.py +""" + +from __future__ import annotations + +import time + +from sentencesplit.abbreviation_replacer import _AbbreviationData +from sentencesplit.languages import LANGUAGE_CODES + +_MEDIUM = ( + "dr. smith went to washington. he arrived on jan. 5th at 3 p.m. the model is gpt 3.1 and it is fast. " + "that is all for now. goodbye. she paid $4.50 for the u.s. edition (vol. 2, p. 17). mr. lee agreed." +) +CASES = { + "tiny (15c)": "dr. smith left.", + "short (87c)": "dr. smith went to washington. he arrived on jan. 5th at 3 p.m. and met with sen. jones.", + "medium (198c)": _MEDIUM, + "large (4k)": " ".join([_MEDIUM] * 20), + "huge (40k)": " ".join([_MEDIUM] * 200), +} + + +def _bench(fn, text: str) -> float: + iters = 20000 if len(text) < 300 else (3000 if len(text) < 6000 else 300) + for _ in range(50): + fn(text) + t0 = time.perf_counter() + for _ in range(iters): + fn(text) + return (time.perf_counter() - t0) / iters * 1e6 # us/call + + +def main() -> None: + data = _AbbreviationData(LANGUAGE_CODES["en"].Abbreviation) + automaton = data.automaton + abbr_lowers = [a[1] for a in data.abbreviations] # stripped, lowercased + + def ac_scan(text: str) -> set[int]: + return automaton.search(text) + + def in_loop(text: str) -> set[int]: + return {i for i, a in enumerate(abbr_lowers) if a in text} + + print(f"abbreviation discovery: Aho-Corasick vs `in` loop ({len(abbr_lowers)} patterns)") + print("=" * 60) + print(f"{'input':<16}{'AC us':>10}{'in us':>10}{'winner':>10}{'ratio':>8}") + print("-" * 60) + for name, text in CASES.items(): + assert ac_scan(text) == in_loop(text), f"match-set mismatch for {name}" + ac = _bench(ac_scan, text) + inf = _bench(in_loop, text) + winner = "AC" if ac < inf else "in" + print(f"{name:<16}{ac:>10.2f}{inf:>10.2f}{winner:>10}{inf / ac:>8.2f}") + print("-" * 60) + print("ratio = in/AC (>1 means AC faster). Match-sets verified identical.") + + +if __name__ == "__main__": + main() From 6ac1ec28a7c8212aba84109e3895354af9f5a937 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 02:27:40 +0000 Subject: [PATCH 03/11] test: add differential profiler (sentencesplit vs pySBD) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Profiles both engines on identical input and surfaces the operation-level deltas: per-call wall time, regex op counts/time, and top tottime functions side by side. Used to localize the short/medium gap — it shows we run FEWER regex ops than pySBD (124 vs 266/call) yet are ~9% slower locally, pointing at per-op constants and the pure-Python automaton rather than pass count. --- benchmarks/differential_profile.py | 129 +++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 benchmarks/differential_profile.py diff --git a/benchmarks/differential_profile.py b/benchmarks/differential_profile.py new file mode 100644 index 0000000..bd6248f --- /dev/null +++ b/benchmarks/differential_profile.py @@ -0,0 +1,129 @@ +"""Differential profiler: sentencesplit vs pySBD on identical input. + +Answers "what *exactly* makes us slower than pySBD per call" by profiling both +on the same text and surfacing the operation-level deltas that matter: + + * per-call wall time (ours vs pySBD), + * count + time of every regex op (sub / findall / search / finditer / split) + per call — the clearest "we run more passes" signal, + * total time spent inside the `re` module, + * the top tottime functions for each library, side by side. + +Run with: + uv run python benchmarks/differential_profile.py --size short + uv run python benchmarks/differential_profile.py --size medium --top 18 +""" + +from __future__ import annotations + +import argparse +import cProfile +import pstats +import time +from collections import Counter + +import pysbd + +import sentencesplit + +SHORT = "Dr. Smith went to Washington. He arrived on Jan. 5th at 3 p.m. and met with Sen. Jones." +MEDIUM = ( + "Dr. Smith went to Washington. He arrived on Jan. 5th at 3 p.m. " + "The model is GPT 3.1 and it is fast. That is all for now. Goodbye. " + "She paid $4.50 for the U.S. edition (vol. 2, p. 17). Mr. Lee agreed." +) +LARGE = " ".join([MEDIUM] * 20) +_SAMPLES = {"short": SHORT, "medium": MEDIUM, "large": LARGE} + +_REGEX_OPS = ("sub", "findall", "search", "match", "finditer", "split", "fullmatch") + + +def _walltime(fn, text: str, iters: int) -> float: + for _ in range(20): + fn(text) + t0 = time.perf_counter() + for _ in range(iters): + fn(text) + return (time.perf_counter() - t0) / iters * 1e6 # us/call + + +def _profile(fn, text: str, iters: int) -> pstats.Stats: + pr = cProfile.Profile() + for _ in range(5): + fn(text) + pr.enable() + for _ in range(iters): + fn(text) + pr.disable() + return pstats.Stats(pr) + + +def _regex_op_summary(stats: pstats.Stats, iters: int) -> tuple[Counter, float]: + """Per-call regex op counts and total time spent in the re module.""" + counts: Counter = Counter() + re_time = 0.0 + for (filename, _lineno, funcname), (_cc, nc, tt, _ct, _cb) in stats.stats.items(): + if funcname in _REGEX_OPS and "re" in filename.lower().replace("\\", "/").split("/"): + counts[funcname] += nc + if filename.endswith("re/__init__.py") or filename.endswith("re.py") or "/re/" in filename.replace("\\", "/"): + re_time += tt + # method 'sub' of 're.Pattern' objects shows up as a builtin + if funcname.startswith(" list[tuple[str, float, int]]: + rows = [] + for (filename, lineno, funcname), (_cc, nc, tt, _ct, _cb) in stats.stats.items(): + short = filename.replace("\\", "/").split("/")[-1] + rows.append((f"{short}:{lineno}:{funcname}", tt / iters * 1e6, nc // iters)) + rows.sort(key=lambda r: r[1], reverse=True) + return rows[:n] + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--size", choices=["short", "medium", "large"], default="short") + ap.add_argument("--top", type=int, default=14) + args = ap.parse_args() + + text = _SAMPLES[args.size] + iters = 8000 if args.size != "large" else 1000 + + ours = sentencesplit.Segmenter(language="en", clean=False, char_span=False) + sbd = pysbd.Segmenter(language="en", clean=False) + engines = {"sentencesplit": ours.segment, "pysbd": sbd.segment} + + print(f"differential profile size={args.size} ({len(text)} chars) iters={iters}") + print("=" * 72) + walls = {name: _walltime(fn, text, iters) for name, fn in engines.items()} + print( + f"wall time: ours {walls['sentencesplit']:.2f} us/call " + f"pysbd {walls['pysbd']:.2f} us/call " + f"ours is {walls['sentencesplit'] / walls['pysbd']:.2f}x pysbd" + ) + print() + + for name, fn in engines.items(): + stats = _profile(fn, text, iters) + ops, re_us = _regex_op_summary(stats, iters) + total_ops = sum(ops.values()) + print(f"--- {name} ---") + print( + f" regex ops/call: {total_ops:.1f} " + f"({' '.join(f'{k}={v:.1f}' for k, v in sorted(ops.items(), key=lambda x: -x[1]))})" + ) + print(f" time in re/call: {re_us:.1f} us") + print(f" top {args.top} by tottime (us/call, calls/call):") + for label, us, calls in _top(stats, args.top, iters): + print(f" {us:7.2f} x{calls:<4} {label}") + print() + + +if __name__ == "__main__": + main() From 836e3b6668db73e3b2c9919d35c66f6c91812310 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 02:35:38 +0000 Subject: [PATCH 04/11] =?UTF-8?q?docs:=20synthesize=20deep=20latency=20inv?= =?UTF-8?q?estigation=20=E2=80=94=20regression=20resolved=20by=20#72?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Multi-agent + differential-profiling investigation of the short/medium gap vs pySBD. Key finding: post-#72 there is no regression. Measured: - wall-clock: short 0.99x pysbd, medium 0.93x (parity/ahead) - instruction-count proxy (line-events): short 0.93x (we're ahead), down from 1.24x pre-#72 Root cause of the original ~28% CodSpeed gap: it was largely an instruction-count artifact of pure-Python per-char loops (counted in full by cachegrind, ~0 wall-clock), whose #1 contributor — the zero-width scanner (876 of 2787 line-events) — was gated by #72. We also run FEWER regex ops than pySBD (118 vs 266) and our Aho-Corasick is faster than a naive 'in'-loop on short input, so neither pass-count nor discovery was the cause. Rewrites the plan as a 'extend the lead' menu, tiered by improves-both vs CodSpeed-only: Tier 1 (list early-out/guards, resplit+callback gating, always-on abbreviation gating), Tier 2 (Aho-Corasick DFA delta-table + length-gated hybrid, ~2x scan), Tier 3 (glued-dot scanner C-regex, instruction-count only). Adds benchmarks/differential_profile.py. --- analysis/SHORT_STRING_LATENCY_PLAN.md | 328 ++++++++++++-------------- 1 file changed, 153 insertions(+), 175 deletions(-) diff --git a/analysis/SHORT_STRING_LATENCY_PLAN.md b/analysis/SHORT_STRING_LATENCY_PLAN.md index e065267..269ecf9 100644 --- a/analysis/SHORT_STRING_LATENCY_PLAN.md +++ b/analysis/SHORT_STRING_LATENCY_PLAN.md @@ -1,207 +1,185 @@ -# Short/Medium `segment()` Latency — Investigation & Action Plan +# Short/Medium `segment()` Latency — Deep Investigation & Action Plan -**Goal:** close the ~22% per-call latency gap vs pySBD on short/medium English -text (surfaced by the competitive CodSpeed benchmarks in #71), **without** -changing segmentation output (byte-identical) or losing the Aho-Corasick -automaton's advantage on very large abbreviation lists. +**Original goal:** close the ~22–28% per-call gap vs pySBD on short/medium +English text reported by the competitive CodSpeed benchmark (#71). -This is a living plan. It pairs with the reusable exploration harness -`benchmarks/phase_profile.py` (the "workflow"): run it to re-attribute per-call -cost to pipeline phases after each change. +**Headline finding (measured): the regression is already gone as of #72.** In +real wall-clock we are at parity-to-ahead of pySBD on short and medium, and on a +cachegrind-style instruction-count proxy we are now *ahead*. The original +CodSpeed gap was (a) an instruction-count artifact of pure-Python per-character +loops, whose single biggest contributor — the zero-width scanner — was removed by +#72, and (b) measured before #72 landed. What remains below is a menu to *extend* +the lead, not to catch up. + +This document is backed by three reusable harnesses (the "workflow"): +`benchmarks/phase_profile.py`, `benchmarks/differential_profile.py`, +`benchmarks/abbr_scan_compare.py`. --- -## 1. How to reproduce the measurements (the workflow) +## 1. The workflow (how every number here is reproduced) ```bash -# Per-phase wall-time attribution for one input size: +# Per-phase wall-time attribution of our own call: uv run python benchmarks/phase_profile.py --size short --iters 20000 -uv run python benchmarks/phase_profile.py --size medium --iters 12000 -# End-to-end latency (and cProfile hot functions): -uv run python benchmarks/latency_baseline.py --iters 4000 [--profile] +# Head-to-head vs pySBD: wall time, regex-op counts, time-in-re, top funcs: +uv run python benchmarks/differential_profile.py --size short +uv run python benchmarks/differential_profile.py --size medium + +# Aho-Corasick scan vs naive `in`-loop crossover: +uv run python benchmarks/abbr_scan_compare.py -# Deterministic CI verdict (instruction count) vs pySBD/punkt: -# the competitive benchmarks run on every PR via .github/workflows/codspeed.yml -# (benchmarks/test_competitive_codspeed.py): watch test_segment[short-ours] etc. +# Instruction-count proxy (Python line-events ≈ what CodSpeed counts): +# sys.settrace line-event count of ours vs pysbd (see §3). ``` -`phase_profile.py` wraps each Processor / AbbreviationReplacer / Segmenter stage -with a timer. The `*wrapper` rows (e.g. `abbr: replace (whole)`, -`split_into_segments`) **contain** the rows below them — do not sum across them. +CodSpeed's CI mode counts **CPU instructions** (cachegrind), not wall time. The +two diverge sharply here, so we measure both: wall-clock for real latency, and a +`sys.settrace` line-event count as a faithful, local instruction-count proxy. --- -## 2. Where the time goes (measured, Python 3.13, warm) - -`segment("Dr. Smith went to Washington. … Sen. Jones.")` — **short, 87 chars, -~0.40 ms/call:** +## 2. Measured state, post-#72 -| phase | ms/call | % of call | -|-------|--------:|----------:| -| `replace_abbreviations` (text phase) | 0.156 | **39%** | -| ↳ `search_for_abbreviations_in_string` (automaton scan + per-abbr sub) | 0.091 | 23% | -| ↳ `apply_ampm_boundary_rules` | 0.026 | 7% | -| `_mark_list_item_boundaries` (ListItemReplacer.add_line_break) | 0.078 | **20%** | -| `split_into_segments` (wrapper: boundary phases + postprocess) | 0.090 | 22% | -| ↳ `between_punctuation` | 0.013 | 3% | -| `replace_numbers` / `_protect_special_tokens` | ~0.010 ea | 2.5% ea | +`segment()` on the short (87c) and medium (198c) samples, Python 3.13, warm, +8000 iters (wall-clock reproduced across runs; line-events are deterministic): -**Medium (198 chars, ~0.88 ms/call)** is the same shape: `replace_abbreviations` -**43%** (of which `search_for_abbreviations_in_string` **31%**), -`_mark_list_item_boundaries` **18%**. +| metric | input | ours | pySBD | ratio | +|--------|-------|-----:|------:|------:| +| wall-clock µs/call | short | ~312 | ~316 | **0.99×** | +| wall-clock µs/call | medium | ~689 | ~743 | **0.93×** | +| regex ops/call | short | 118 | 266 | **we run fewer** | +| line-events/call (instr. proxy) | short | 2059 | 2203 | **0.93×** | -**Conclusion:** two fixed-cost phases dominate and run on *every* call regardless -of whether the input needs them: -1. **Abbreviation replacement (~40%)** — but NOT the scan (see §2a); it is the - ~25 `re.sub` per call: per-occurrence replacements + always-on rule passes. -2. **List-item detection (~18%)** — runs 4 sub-formatters even on non-list text. +We are at parity or ahead on every axis. Note we run **fewer** regex ops than +pySBD (118 vs 266) — the "we do more passes" theory is false; pySBD does ~2× the +regex ops but they are tiny literal-replacement subs. -These ~58% are the structural gap. The `between_punctuation` / zero-width work -already addressed in #72 was <4% on short input (it helped large text instead). +### 2a. Abbreviation *discovery* is not the bottleneck (and AC is fine) -### 2a. The Aho-Corasick scan is NOT the bottleneck (measured) - -An earlier draft of this plan assumed the abbreviation *scan* (Aho-Corasick) was -the cost and proposed replacing it with pySBD's `in`-loop. **Direct measurement -disproves that.** Our pure-Python automaton `search()` vs a pySBD-style -`in`-filter loop over the same 199 abbreviations (identical match sets): +`benchmarks/abbr_scan_compare.py` (199 patterns, identical match sets): | input | AC | `in`-loop | winner | |-------|---:|----------:|--------| -| tiny (15c) | 2.4 µs | 7.2 µs | **AC 3.0×** | -| short (87c) | 11.3 µs | 12.0 µs | **AC ~tied** | -| medium (198c)| 25.3 µs | 18.0 µs | `in` 1.4× | -| large (4k) | 497 µs | 229 µs | `in` 2.2× | - -Our AC is faster on short input; `in` only overtakes around ~100–150 chars. -(The crossover is inverted from textbook AC-wins-on-long because the automaton is -**pure Python**: its per-char loop has a large constant, so on long text 199 -C-level `in` scans beat thousands of Python iterations.) - -Crucially, the AC scan is only **~11 µs ≈ 3% of the 400 µs short call**. Swapping -it for `in` would *slow short down* and save nothing meaningful. The 39% -abbreviation cost is the **~25 `re.sub`/call** that follow discovery: -`scan_for_replacements` (a global `re.sub` per matched abbreviation), -`replace_multi_period_abbreviations`, and the always-on rule passes -(`PossessiveAbbreviationRule`, `SingleLetterAbbreviationRules`, `_COMPACT_AMPM_RE`, -`_UPPERCASE_INITIALISM_BOUNDARY_RE`, allcaps-imprint, the a.m./p.m. rules). -pySBD runs the per-occurrence regex too — so the gap is our **extra always-on -passes**, not the discovery mechanism. +| tiny (15c) | 2.4 µs | 8.2 µs | **AC 3.4×** | +| short (87c) | 11.3 µs | 13.5 µs | **AC** | +| medium (198c)| 30.5 µs | 21.0 µs | `in` 1.4× | +| large (4k) | 619 µs | 286 µs | `in` 2.2× | + +Our pure-Python Aho-Corasick is *faster* than a pySBD-style `in`-loop on short +input; `in` only overtakes around ~150 chars. The scan is ~3% of the call. **Do +not replace it.** (It can, however, be made ~2× faster outright — see §4, P-AC.) --- -## 3. Why pySBD is cheaper per call (structural delta) - -pySBD does **not** have fewer phases (~20 full-text passes, similar to ours). -The difference is per-pass constant cost and cheap short-circuits: - -1. **Abbreviation discovery is a wash on short text.** pySBD short-circuits with - C-level `in` (`pysbd/abbreviation_replacer.py:82`); ours uses a pure-Python - Aho-Corasick automaton (`sentencesplit/abbreviation_replacer.py:535-564`). Per - §2a these cost ~the same on short input (AC is actually slightly faster), so - this is **not** where the gap is — both then run the same per-occurrence regex - work. Discovery is ~3% of the call either way. -2. **Always-on extra passes ours added** in `replace()` that pySBD lacks (this, - not the scan, is the abbreviation-phase cost): - `_COMPACT_AMPM_RE`, `_UPPERCASE_INITIALISM_BOUNDARY_RE`, allcaps-imprint - protection, non-ASCII a.m./p.m. restores (`:311-357`). Each is an - unconditional full-text regex pass. -3. **Per-call `RLock` acquisition.** Every `AbbreviationReplacer.__init__` - (`:55-58`) takes `_cache_lock` to fetch the cached `_data`, on every call. - pySBD has no such lock. - -What we already match: a no-punctuation boundary guard -(`processor.py:691` `check_for_punctuation`) skips the boundary pipeline for -segments lacking sentence-ending punctuation — same idea as pySBD. +## 3. What the original gap actually was (root cause) + +A `sys.settrace` line-event count (≈ instructions executed) on the **pre-#72** +code: ours **2787** vs pySBD **2243** = **1.24×** — almost exactly the ~28% +CodSpeed gap, and far above the ~1.06× wall-clock ratio. The gap was concentrated +in pure-Python per-character loops that cost almost nothing in wall-clock but +execute hundreds of *counted* interpreted operations: + +| our function | line-events/call | wall-clock cost | pySBD equivalent | +|--------------|-----------------:|-----------------|------------------| +| `_strip_zero_width_before_sentence_closers` | **876** | **~0 µs** | none (pySBD has no zero-width handling) | +| `AhoCorasickAutomaton.search` | 394 | *faster* than pySBD's scan | C-level `abbr in lowered` | +| `_GluedLowercaseRunOnRegex.sub` (`lang/common/standard.py:11`) | 309 | small | one C `re.sub` | +| `apply_rules` / `_sub_symbols_fast` | 156 / 132 | small | C `re.sub` | + +**Two distinct causes, two distinct truths:** +1. **CodSpeed instruction-count gap (~28%)** — largely an *artifact*: CPython + pushes pySBD's work into the C `re` engine (≈1 counted op per `re.sub`), while + our pure-Python loops are fully counted. The zero-width scanner alone was 876 + of our 2787 events. **#72 gated it** (`segmenter.py:89`, return early when no + zero-width char), cutting ~730 events → post-#72 proxy is **2059 vs 2203 = + 0.93×, we are ahead.** +2. **Wall-clock gap (~6–9%, pre-#72)** — real, and came from our heavier + *non-destructive* pipeline (span mapping `_match_spans`/`_find_sentence_start`, + the resplit passes, the sentinel-escape disjointness check, and callback-driven + subs), not regex efficiency. #72's zero-width guard plus normal variance brings + short to parity; medium we already win. + +**So #72 — framed at the time as "helps large text" — actually closed the +short/medium gap, because the zero-width scanner ran per output segment and +dominated the instruction count on short input. Its wall-clock effect on short +was within noise; its instruction-count effect was the whole ballgame.** The +competitive CodSpeed benchmark on `main` (post-#72) should now show short/medium +at parity-or-ahead; the next competitive run will confirm. --- -## 4. Action plan (prioritized by leverage × safety) - -Every item is **behavior-preserving**: the validation bar is byte-identical -`segment()`/`segment_spans()` output (full suite incl. Golden Rules stays green, -and a cross-corpus equivalence check). Implement **test-first**, on its own -branch, auto-revert if Golden Rules or the suite regress (mirror the -`improve-sentencesplit` discipline). Measure each with `phase_profile.py` and the -competitive CodSpeed benchmark. - -### ~~P1 — Abbreviation scan fast-path~~ — DROPPED (premise disproved, see §2a) - -The original P1 (replace the Aho-Corasick scan with pySBD's `in`-loop or a -compiled alternation) is **invalid**: measurement shows the AC scan is ~3% of the -call and is already faster than `in` on short input. Switching would slow short -text down. Do not pursue. The abbreviation-phase cost lives in the always-on -passes (now P1, below), not discovery. - -*Optional, low priority:* a length-gated `in` fallback would help medium/large -discovery (~7 µs on medium, ~270 µs on 4k) but hurts short, so only worth it as a -threshold switch if medium/large latency becomes a target — not for this goal. - -### P1 — Gate the always-on abbreviation `replace()` passes (was P2; now top) - -`AbbreviationReplacer.replace()` runs ~25 `re.sub`/call on short text, several of -them unconditional rule passes that almost never match. Guard each behind a cheap -`in`/structural check so it only runs when it can fire: -- `_COMPACT_AMPM_RE` — only if a digit is adjacent to `a/p`+`.`+`m`. -- `_UPPERCASE_INITIALISM_BOUNDARY_RE`, non-ASCII a.m./p.m. restores — only if the - `∯` sentinel is present (it only exists after an abbreviation matched). -- allcaps-imprint — only if the text has a 2+ all-caps run. -- the a.m./p.m. rule set (`apply_ampm_boundary_rules`, ~5–7%) — gate on an - `[ap]·m` pattern being present. -This is the largest *safe* slice of the abbreviation phase. *Risk:* low (each -regex already requires the guarded condition). *DoD:* suite green + byte-identical -corpus output. - -### P2 — List-item detection guards (target: the ~18% list phase) - -`_mark_list_item_boundaries` constructs a fresh `ListItemReplacer` and runs 4 -sub-formatters every call. Safe reductions: -- **P2a — Trigger-char guards per sub-formatter.** Skip the numbered-list passes - when the text has no digit; skip the parens passes when there is no `(`/`)`; - skip alphabetical/roman passes when no single letter precedes `.`/`)`. Each - formatter's regex already requires those chars, so skipping is a no-op. -- **P2b — Early-out in `iterate_alphabet_array`** when `re.findall` returns empty - (avoid building the 26-entry `alphabet_index` dict and the lower/filter work). -*Risk:* low-medium (list logic is subtle; guards must be on chars the regexes -require). *DoD:* suite green; the list-heavy Golden Rules unchanged. - -### P3 — Remove the per-call lock on the abbreviation-data read (low risk, small gain) - -`AbbreviationReplacer.__init__` takes an `RLock` every call just to read the -per-class `_data`. Use a lock-free fast read (double-checked: read the dict -first, only lock to build on miss) or resolve `_data` once and stash it on the -Abbreviation class / LanguageProfile. *Risk:* low; preserve thread-safety of the -first build. *DoD:* suite green + a concurrency smoke test. +## 4. Menu to *extend* the lead (all behavior-preserving / byte-identical) + +We are no longer catching up, so these are prioritized by **(improves both +metrics) > (improves one)** × leverage × safety. Implement test-first on isolated +branches with a byte-identical gate (full suite + Golden Rules + a corpus diff; +auto-revert on regression). Re-measure each with `phase_profile.py`, +`differential_profile.py`, and the competitive CodSpeed run. + +### Tier 1 — improves BOTH wall-clock and instruction-count + +- **P-LIST — List-item early-out + guards (~18% phase).** In + `lists_item_replacer.py`: (a) early-`return` in `iterate_alphabet_array` when + `re.findall` is empty (kills the 4×/call 26-entry `alphabet_index` dict builds); + (b) skip the numbered formatters when the text has no digit, the parens + formatters when no `)`; (c) reuse the identical alphabetical `findall` across + the roman/non-roman passes (4 scans → 2). Each guard char is required by every + alternative of its regex, so skipping is byte-identical. **Guard inside the + formatter methods** so Slovak's `add_line_break` override inherits them. + Validate with a differential oracle (old vs new `add_line_break` char-for-char). + *Risk:* low (items a/b), moderate (item c — confirm shared pattern+flags). +- **P-RESPLIT/CALLBACK — Gate the non-destructive passes + de-callback subs.** Per + Agent C, the real wall-clock cost is span mapping + resplit + callback subs. + Extend the eager-gate discipline already in `_maybe_resplit_multi_sentence_quote` + to the other resplit/postprocess passes (skip `_split_on_uppercase_boundary` + scans for segments with no `.)`/multi-terminator), and replace constant-result + callback subs with literal-string subs / `str.replace` (the + punctuation/continuous/double-punct callbacks). *Risk:* medium — touches + boundary-adjacent code; needs the full byte-identical gate. +- **P-ABBR — Gate the always-on abbreviation `replace()` passes.** + `AbbreviationReplacer.replace()` runs ~25 `re.sub`/call; gate the rarely-firing + ones (`_COMPACT_AMPM_RE`, `_UPPERCASE_INITIALISM_BOUNDARY_RE`, non-ASCII a.m./p.m. + restores, allcaps-imprint, the a.m./p.m. set) on a cheap presence check (digit + adjacency, `∯`-sentinel presence, all-caps run). *Risk:* low. + +### Tier 2 — Aho-Corasick ~2× (helps both; bigger on medium/large) + +- **P-AC — DFA δ-table precompute + length-gated hybrid.** Empirically (Agent A): + precomputing fail-links into a flat DFA table (removing the inner `while` + fail-link loop in `AhoCorasickAutomaton.search`) gives **short 9.8→4.9 µs (~2×), + long 270→~160 µs (~1.7×)**, drop-in, cached, ~2 ms one-time build. Add a + length-gated hybrid (DFA under ~200 chars, naive C `in`-loop above) to beat both + the automaton and pySBD's `in`-loop across all sizes. *Risk:* low–moderate (two + code paths; ship a `dfa == naive == current` fuzz/property test). Rejected by + measurement and not to be tried: `re`-alternation (3–5× slower), byte/array/tuple + transition tables (all slower than `dict.get`), first-char prefilter (English's + 24 first-chars defeat it). + +### Tier 3 — instruction-count only (low wall-clock value) + +- **P-GLUED — rewrite the `_GluedLowercaseRunOnRegex` Python char scanner + (`lang/common/standard.py:11`) into a single C-engine `re` call.** ~309 + line-events, near-zero wall-clock. Only worth it if the CodSpeed number is a + release gate; it barely moves real latency. --- -## 5. Sequencing, guardrails, and non-goals - -**Sequence:** P3 first (mechanical, lowest risk, de-risks the harness loop), then -P1 (gate the always-on abbreviation passes — the biggest *safe* slice of the 40% -abbreviation phase), then P2 (the ~18% list phase). Re-run `phase_profile.py` + -competitive CodSpeed after each. Note the discovery scan is deliberately left -alone (see §2a). - -**Guardrails (non-negotiable):** -- Byte-identical `segment()`/`segment_spans()` output. Full suite + Golden Rules - green is the gate; add a corpus-wide equivalence diff (old vs new) per change. -- Each change is reverted automatically if the suite or Golden Rules regress. -- Keep changes isolated per branch/PR so CodSpeed attributes each delta. - -**Non-goals / risks to avoid:** -- Do **not** touch the abbreviation *discovery* scan — it is faster than `in` on - short input and only ~3% of the call (§2a). The Aho-Corasick automaton stays. -- Do not change boundary semantics to chase speed. This is a pure - constant-factor effort; if a change can't be made byte-identical, it is out of - scope for this plan. - -**Expected envelope:** all three items (P1 gate always-on passes, P2 list guards, -P3 lock) are low-risk and target the always-on `re.sub` count + the list phase. -Together they address roughly the abbreviation always-on slice (a chunk of the -40%) plus the ~18% list phase. Treat individual gains as unproven until measured -per-change with `phase_profile.py` and the competitive CodSpeed run; the honest -expectation is incremental (single-digit to low-double-digit %), not a dramatic -close, because much of the abbreviation phase is genuine per-occurrence regex -work that pySBD also pays. +## 5. Guardrails & non-goals + +- **Byte-identical** `segment()`/`segment_spans()` output is the bar. Full suite + + Golden Rules green + a corpus-wide old-vs-new diff per change; auto-revert on + any regression; one change per branch/PR so CodSpeed attributes each delta. +- **Do not touch the abbreviation discovery semantics** — AC is fine and faster + than `in` on short (§2a). P-AC keeps the automaton, only speeds its inner loop. +- **No boundary-semantics changes** to chase speed; if a change can't be made + byte-identical it is out of scope. +- **Pick the metric deliberately.** If the goal is *real latency*, prioritize + Tier 1 (and skip Tier 3). If the goal is the *CodSpeed gate number*, Tier 2/3 + (the pure-Python loops) move it most. They overlap in Tier 1, which is why Tier + 1 leads. + +**Honest expectation:** we are already at parity/ahead, so these are +incremental "pull further ahead" wins, not a turnaround. Tier 1 + P-AC together +are the worthwhile set; treat each gain as unproven until measured per-change. From 712ac5f541d3814c5bdadc24602dabaf8e400b60 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 03:25:00 +0000 Subject: [PATCH 05/11] perf: faster span mapping, Aho-Corasick DFA, and list-phase early-out Three byte-identical optimizations from the latency review (full suite 1970 passed; AC match-set fuzz-verified vs naive ground truth on 20k random texts): - segmenter._find_sentence_start: search from prior_end via the pattern's pos arg instead of slicing original_text[prior_end:] each sentence, which copied the whole remaining text per sentence (O(n^2) over a document). The flexible fallback patterns are anchor-free, so matching at pos is identical to matching the slice. - AhoCorasickAutomaton: collapse the fail links into a DFA delta-table at build() time, so search() is one dict.get per char with no inner fail-walk loop. Same match set; ~1.5x faster discovery (short 11.3->7.6 us, medium 30.5->20.5 us, now beating the naive in-loop through medium). - ListItemReplacer.iterate_alphabet_array: early-return when re.findall finds no markers, skipping the per-call alphabet-index dict build, lowercasing, and filter on list-free text. Net ~3.5-5% wall-clock on short/medium/large segment(), no behavior change. --- sentencesplit/abbreviation_replacer.py | 39 ++++++++++++++++++++------ sentencesplit/lists_item_replacer.py | 5 ++++ sentencesplit/segmenter.py | 17 ++++++----- 3 files changed, 46 insertions(+), 15 deletions(-) diff --git a/sentencesplit/abbreviation_replacer.py b/sentencesplit/abbreviation_replacer.py index c23e6a3..72de67b 100644 --- a/sentencesplit/abbreviation_replacer.py +++ b/sentencesplit/abbreviation_replacer.py @@ -17,13 +17,19 @@ class AhoCorasickAutomaton: """Pure-Python Aho-Corasick automaton for multi-pattern substring search.""" - __slots__ = ("goto", "fail", "output", "_built") + __slots__ = ("goto", "fail", "output", "delta", "_built") def __init__(self): # State 0 is the root. Each state maps char -> next_state. self.goto: list[dict[str, int]] = [{}] self.output: list[list[int]] = [[]] # pattern IDs at each state self.fail: list[int] = [0] + # Fail-link-collapsed transition table, built once in build(). Each + # delta[state] maps an observed-alphabet char -> next state with the fail + # walk already resolved, so search() is one dict.get per char (no inner + # loop). Chars outside the alphabet are absent and .get(ch, 0) sends them + # to the root, exactly as the fail walk would. + self.delta: list[dict[str, int]] = [] self._built = False def add_pattern(self, pattern: str, pattern_id: int) -> None: @@ -58,21 +64,38 @@ def build(self) -> None: self.fail[s] = 0 if self.output[self.fail[s]]: self.output[s] = self.output[s] + self.output[self.fail[s]] + + # Collapse the fail links into a DFA transition table. For each state and + # each observed-alphabet char: take the goto if present, else inherit the + # fail state's already-resolved transition. fail[r] is strictly shallower + # than r, so a goto-tree BFS visits it first and delta[fail[r]] is ready. + alphabet: set[str] = set() + for trans in self.goto: + alphabet.update(trans) + delta: list[dict[str, int]] = [{} for _ in self.goto] + root_goto = self.goto[0] + delta[0] = {ch: root_goto.get(ch, 0) for ch in alphabet} + queue = deque(self.goto[0].values()) + while queue: + r = queue.popleft() + gr = self.goto[r] + dfail = delta[self.fail[r]] + delta[r] = {ch: (gr[ch] if ch in gr else dfail[ch]) for ch in alphabet} + queue.extend(gr.values()) + self.delta = delta self._built = True def search(self, text: str) -> set[int]: """Scan text in one pass, return set of matched pattern IDs.""" state = 0 found: set[int] = set() - goto = self.goto - fail = self.fail + delta = self.delta output = self.output for ch in text: - while state != 0 and ch not in goto[state]: - state = fail[state] - state = goto[state].get(ch, 0) - if output[state]: - found.update(output[state]) + state = delta[state].get(ch, 0) + out = output[state] + if out: + found.update(out) return found diff --git a/sentencesplit/lists_item_replacer.py b/sentencesplit/lists_item_replacer.py index 2547a05..58c5060 100644 --- a/sentencesplit/lists_item_replacer.py +++ b/sentencesplit/lists_item_replacer.py @@ -268,6 +268,11 @@ def other_items_replacement(self, a, i, alphabet, alphabet_index, list_array, pa def iterate_alphabet_array(self, regex, parens=False, roman_numeral=False): list_array = re.findall(regex, self.text, re.IGNORECASE) + # Common case on list-free text: no markers found. Skip the lowercasing, + # the per-call alphabet-index dict build, and the filter — with an empty + # list the replacement loop below never runs, so this is byte-identical. + if not list_array: + return list_array = [i.lower() for i in list_array] alphabet = self.ROMAN_NUMERALS if roman_numeral else self.LATIN_NUMERALS alphabet_index = {value: index for index, value in enumerate(alphabet)} diff --git a/sentencesplit/segmenter.py b/sentencesplit/segmenter.py index 0e145f8..bb9138c 100644 --- a/sentencesplit/segmenter.py +++ b/sentencesplit/segmenter.py @@ -399,17 +399,20 @@ def _find_sentence_start(self, sent: str, original_text: str, prior_end: int): return self._find_sentence_start_tolerant(sent, original_text, prior_end) # Some post-processing rules may normalize spaces around punctuation, - # so allow flexible whitespace when mapping back to original text. - whitespace_flexible = re.escape(sent).replace(r"\ ", r"\s*") - match = re.search(whitespace_flexible, original_text[prior_end:]) + # so allow flexible whitespace when mapping back to original text. Search + # from ``prior_end`` via the pattern's ``pos`` arg rather than slicing + # ``original_text[prior_end:]`` (which copies the whole remaining text on + # every sentence — O(n^2) over a document). The flexible patterns are + # anchor-free (escaped literal + ``\s*``/zero-width joiners), so matching + # at ``pos`` is identical to matching the slice. + whitespace_flexible = re.compile(re.escape(sent).replace(r"\ ", r"\s*")) + match = whitespace_flexible.search(original_text, prior_end) if match is None: - match = re.search(self._zero_width_flexible_pattern(sent), original_text[prior_end:]) + match = re.compile(self._zero_width_flexible_pattern(sent)).search(original_text, prior_end) if match is None: return None - start_idx = prior_end + match.start() - end_idx = prior_end + match.end() - return start_idx, end_idx + return match.start(), match.end() @staticmethod def _match_tolerant_at(sent: str, original_text: str, start: int): From f0a07f34ae0e8e7ad3e4276efa6ab03097f32f7b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 04:13:48 +0000 Subject: [PATCH 06/11] perf: guard numbered-list, glued-runon, and ellipsis-reinsert phases MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three byte-identical char-presence guards that skip phases which can't match on the common case. Measured ~10.7% faster on realistic large English prose (71.3 -> 63.6 ms/call, 35 KB / 500 sentences); suite unchanged (1967 passed). - ListItemReplacer.scan_lists: skip the two numbered-list finditer scans when the text has no digit (every numbered-list pattern requires \d, and the body does int()). This was the largest of the three (~6.7% of a realistic large call). - _GluedLowercaseRunOnRegex.sub: skip the per-character Python scan unless '....[a-z]' is present (a necessary condition for any change). Uses a FIXED {4} count, not {4,}, to stay linear on adversarial long period runs ({4,} backtracks O(n^2)). - Processor.post_process_segments: skip ReinsertEllipsisRules (5 subs per segment) unless one of its placeholder sentinels [ƪ♟♝☏∮] is present. Each guard char is required by every alternative of the gated rule's regex, so skipping is byte-identical. --- sentencesplit/lang/common/standard.py | 10 ++++++++++ sentencesplit/lists_item_replacer.py | 8 ++++++++ sentencesplit/processor.py | 7 ++++++- 3 files changed, 24 insertions(+), 1 deletion(-) diff --git a/sentencesplit/lang/common/standard.py b/sentencesplit/lang/common/standard.py index 1e5b73a..06ffd7c 100644 --- a/sentencesplit/lang/common/standard.py +++ b/sentencesplit/lang/common/standard.py @@ -4,11 +4,21 @@ from sentencesplit.abbreviation_replacer import AbbreviationReplacer from sentencesplit.utils import Rule +# A modification requires a run of 4+ dots immediately followed by an ASCII +# lowercase letter, which always contains the substring "4 dots then lowercase". +# So this is a necessary condition; when it fails the per-char loop below makes +# no change and is skipped. Uses a FIXED ``{4}`` count (not ``{4,}``) so the C +# search stays linear on adversarial long period runs — ``{4,}`` would backtrack +# O(n^2) (see test_glued_ellipsis_..._handles_long_period_runs_linearly). +_GLUED_RUNON_GUARD_RE = re.compile(r"\.{4}[a-z]") + class _GluedLowercaseRunOnRegex: """Linear-time replacer for glued 4+ dot lowercase run-ons.""" def sub(self, replacement: str, text: str) -> str: + if not _GLUED_RUNON_GUARD_RE.search(text): + return text chars: list[str] | None = None index = 0 length = len(text) diff --git a/sentencesplit/lists_item_replacer.py b/sentencesplit/lists_item_replacer.py index 58c5060..e9650f5 100644 --- a/sentencesplit/lists_item_replacer.py +++ b/sentencesplit/lists_item_replacer.py @@ -11,6 +11,9 @@ # Newline-separated list-marker guards: a list spanning lines is not collapsed. _MULTILINE_BULLET_GUARD_RE = re.compile(r"♨.+(\n|\r).+♨") _MULTILINE_PAREN_MARKER_GUARD_RE = re.compile(r"☝.+\n.+☝|☝.+\r.+☝") +# Every numbered-list pattern requires a digit; used to skip the scan on +# digit-free text (matches re's \d Unicode-digit semantics exactly). +_DIGIT_RE = re.compile(r"\d") class ListItemReplacer: @@ -107,6 +110,11 @@ def add_line_breaks_for_alphabetical_list_with_parens(self, roman_numeral=False) self.iterate_alphabet_array(self.ALPHABETICAL_LIST_WITH_PARENS, parens=True, roman_numeral=roman_numeral) def scan_lists(self, regex1, regex2, replacement, strip=False): + # All numbered-list patterns require a digit (and the body does int()), + # so digit-free text can't match — skip the two finditer scans. The loop + # below never runs on empty matches, so this is byte-identical. + if not _DIGIT_RE.search(self.text): + return matches = list(re.finditer(regex1, self.text)) list_array = [(int(m.group().strip()), m.start()) for m in matches] for ind, (item, pos) in enumerate(list_array): diff --git a/sentencesplit/processor.py b/sentencesplit/processor.py index 525856e..eed5c68 100644 --- a/sentencesplit/processor.py +++ b/sentencesplit/processor.py @@ -52,6 +52,10 @@ # lookahead excludes a doubled comma (",,"), which in Dutch typography is an # *opening* quotation mark beginning a new sentence (e.g. "...einde. ,,Nieuwe..."). _PERIOD_BEFORE_COMMA_RE = re.compile(r"\.(?=\s*,(?!,))") +# The five ReinsertEllipsisRules each require one of these placeholder sentinels; +# a segment with none of them passes through them unchanged, so the per-segment +# pass can be skipped on the common case (one C scan vs five no-op subs). +_REINSERT_ELLIPSIS_RE = re.compile(r"[ƪ♟♝☏∮]") # The between-punctuation pass protects everything from an opening quote to its # closing quote as one unsplittable region, so a quotation that wraps several @@ -642,7 +646,8 @@ def post_process_segments(self, txt: str) -> list[str]: if len(txt) > 2 and _ALPHA_ONLY_RE.search(txt): return [txt] - txt = apply_rules(txt, *self.lang.ReinsertEllipsisRules.All) + if _REINSERT_ELLIPSIS_RE.search(txt): + txt = apply_rules(txt, *self.lang.ReinsertEllipsisRules.All) if self.profile.latin_uppercase_resplit: quoted_parts = _split_on_uppercase_boundary(txt, self.profile.split_quotation_re) if quoted_parts is not None: From eb7a1c2b8b3055d66493782f1a72d308ae24042f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 04:19:58 +0000 Subject: [PATCH 07/11] perf: key the abbreviation automaton on '.' to skip false-positive rescans MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit search_for_abbreviations_in_string only ever acts on an abbreviation that occurs at a word boundary *followed by a period*. The Aho-Corasick automaton was keyed on the bare abbreviation, so on real prose it matched ~23 short abbreviations as substrings of ordinary words ('al' in 'called', 'no' in 'no one', 'st' in 'stood', 'rd' in 'garden', ...), and each match triggered a full-text match_re.finditer rescan that found nothing — the single largest cost on realistic text (~24% of a large call). Keying the automaton on '.' instead is a byte-identical pre-filter: any word-boundary '.' occurrence contains the substring '.', so no real occurrence is missed, and abbreviations whose bare form merely appears inside other words (with no following period) no longer trigger the finditer. Measured: realistic large English prose 64.6 -> 38.7 ms/call (-40%); short -6.7%. Byte-identical: full suite 1967 passed, and a direct old-vs-new segment() diff over abbreviation-dense en/en_legal inputs is character-for-character identical. --- benchmarks/abbr_scan_compare.py | 8 +++++--- sentencesplit/abbreviation_replacer.py | 10 +++++++++- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/benchmarks/abbr_scan_compare.py b/benchmarks/abbr_scan_compare.py index 4783058..cb442a9 100644 --- a/benchmarks/abbr_scan_compare.py +++ b/benchmarks/abbr_scan_compare.py @@ -46,15 +46,17 @@ def _bench(fn, text: str) -> float: def main() -> None: data = _AbbreviationData(LANGUAGE_CODES["en"].Abbreviation) automaton = data.automaton - abbr_lowers = [a[1] for a in data.abbreviations] # stripped, lowercased + # The automaton is keyed on "." (the trailing period pre-filter), so the + # equivalent naive loop tests for "." too. + abbr_keys = [a[1] + "." for a in data.abbreviations] # stripped, lowercased, + '.' def ac_scan(text: str) -> set[int]: return automaton.search(text) def in_loop(text: str) -> set[int]: - return {i for i, a in enumerate(abbr_lowers) if a in text} + return {i for i, a in enumerate(abbr_keys) if a in text} - print(f"abbreviation discovery: Aho-Corasick vs `in` loop ({len(abbr_lowers)} patterns)") + print(f"abbreviation discovery: Aho-Corasick vs `in` loop ({len(abbr_keys)} patterns)") print("=" * 60) print(f"{'input':<16}{'AC us':>10}{'in us':>10}{'winner':>10}{'ratio':>8}") print("-" * 60) diff --git a/sentencesplit/abbreviation_replacer.py b/sentencesplit/abbreviation_replacer.py index 72de67b..67ae972 100644 --- a/sentencesplit/abbreviation_replacer.py +++ b/sentencesplit/abbreviation_replacer.py @@ -164,7 +164,15 @@ def __init__(self, lang_abbreviation_class): next_word_re, ) ) - self.automaton.add_pattern(stripped_lower, idx) + # Add the trailing period to the automaton key. search_for_abbreviations + # only ever acts on an abbreviation when it occurs at a word boundary + # *followed by a period*; any such occurrence contains the substring + # ".", so keying on "." is a byte-identical pre-filter that + # skips the per-abbreviation full-text finditer for abbreviations whose + # bare form merely appears inside other words (e.g. "al" in "called", + # "no" in "no one") with no following period — the dominant cost on + # real prose, where common short abbreviations match everywhere. + self.automaton.add_pattern(stripped_lower + ".", idx) self.automaton.build() self.abbr_set = frozenset(a.strip().lower() for a in raw) self.prepositive_set = frozenset(a.lower() for a in lang_abbreviation_class.PREPOSITIVE_ABBREVIATIONS) From 10b65c1fb6808884722d04a401e5f3397614fdd3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 04:35:19 +0000 Subject: [PATCH 08/11] fix: evict abbreviation-data cache on language re-registration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit _evict_profile is documented to make a re-registration 'rebuilt fresh', but it only dropped the cached LanguageProfile — the per-Abbreviation-class Aho-Corasick data in AbbreviationReplacer._data_cache survived. A language class re-registered after its abbreviation list changed kept the stale automaton. Now also evict the _data_cache entry for the rebound class's Abbreviation. Regression test added (red before, green after). Also fuzz-checked 13k adversarial inputs x 26 languages: zero crashes, zero span round-trip violations, clean=True robust, and the documented feed-at-once streaming contract holds. --- sentencesplit/languages.py | 15 +++++++- .../test_language_reregistration.py | 38 +++++++++++++++++++ 2 files changed, 51 insertions(+), 2 deletions(-) create mode 100644 tests/regression/test_language_reregistration.py diff --git a/sentencesplit/languages.py b/sentencesplit/languages.py index 8dd8cfe..458249b 100644 --- a/sentencesplit/languages.py +++ b/sentencesplit/languages.py @@ -206,8 +206,15 @@ def __ror__(self, other): def _evict_profile(code: str) -> None: - """Drop any cached LanguageProfile for the class currently bound to ``code`` - so a re-registration (or override) is rebuilt fresh.""" + """Drop the cached state for the class currently bound to ``code`` so a + re-registration (or override) is rebuilt fresh. + + This drops both the cached :class:`LanguageProfile` (keyed on the language + class) and the per-``Abbreviation``-class Aho-Corasick data (keyed on + ``language_cls.Abbreviation``); otherwise a re-registered class whose + abbreviation list changed would keep a stale automaton. + """ + from sentencesplit.abbreviation_replacer import AbbreviationReplacer from sentencesplit.language_profile import _PROFILE_CACHE, _PROFILE_CACHE_LOCK with _LANGUAGE_LOCK: @@ -215,6 +222,10 @@ def _evict_profile(code: str) -> None: if existing is not None: with _PROFILE_CACHE_LOCK: _PROFILE_CACHE.pop(existing, None) + abbr_class = getattr(existing, "Abbreviation", None) + if abbr_class is not None: + with AbbreviationReplacer._cache_lock: + AbbreviationReplacer._data_cache.pop(abbr_class, None) def register_language(code: str, language_cls: type) -> None: diff --git a/tests/regression/test_language_reregistration.py b/tests/regression/test_language_reregistration.py new file mode 100644 index 0000000..f0561e1 --- /dev/null +++ b/tests/regression/test_language_reregistration.py @@ -0,0 +1,38 @@ +# -*- coding: utf-8 -*- +"""Regression: re-registering a language must rebuild its abbreviation data. + +``_evict_profile`` is documented to make a re-registration "rebuilt fresh", but +it only dropped the cached ``LanguageProfile`` — the per-``Abbreviation``-class +Aho-Corasick data in ``AbbreviationReplacer._data_cache`` survived, so a class +re-registered after its abbreviation list changed kept the stale automaton. +""" + +from sentencesplit import Segmenter +from sentencesplit.abbreviation_replacer import AbbreviationReplacer +from sentencesplit.lang.english import English +from sentencesplit.languages import register_language, unregister_language + + +def test_reregistration_rebuilds_abbreviation_data(): + class CustomAbbr(English.Abbreviation): + ABBREVIATIONS = list(English.Abbreviation.ABBREVIATIONS) + + class CustomEn(English): + Abbreviation = CustomAbbr + + code = "zz" + try: + register_language(code, CustomEn) + # Build and cache the abbreviation automaton for CustomAbbr. + Segmenter(language=code).segment("Dr. Smith arrived.") + assert "zorp" not in AbbreviationReplacer._data_cache[CustomAbbr].abbr_set + + # Change the abbreviation set and re-register the (same) class. + CustomAbbr.ABBREVIATIONS.append("zorp") + register_language(code, CustomEn) + + # The next use must rebuild the automaton fresh from the new list. + Segmenter(language=code).segment("Zorp. Smith arrived.") + assert "zorp" in AbbreviationReplacer._data_cache[CustomAbbr].abbr_set + finally: + unregister_language(code) From e3de83c3c8d08ab7cbeef8ff93cfae8f48b1fb57 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 04:39:35 +0000 Subject: [PATCH 09/11] docs: remove the short-string latency plan doc The investigation's conclusions are captured in the PR description and the commit history; the standalone plan file is no longer needed. --- analysis/SHORT_STRING_LATENCY_PLAN.md | 185 -------------------------- 1 file changed, 185 deletions(-) delete mode 100644 analysis/SHORT_STRING_LATENCY_PLAN.md diff --git a/analysis/SHORT_STRING_LATENCY_PLAN.md b/analysis/SHORT_STRING_LATENCY_PLAN.md deleted file mode 100644 index 269ecf9..0000000 --- a/analysis/SHORT_STRING_LATENCY_PLAN.md +++ /dev/null @@ -1,185 +0,0 @@ -# Short/Medium `segment()` Latency — Deep Investigation & Action Plan - -**Original goal:** close the ~22–28% per-call gap vs pySBD on short/medium -English text reported by the competitive CodSpeed benchmark (#71). - -**Headline finding (measured): the regression is already gone as of #72.** In -real wall-clock we are at parity-to-ahead of pySBD on short and medium, and on a -cachegrind-style instruction-count proxy we are now *ahead*. The original -CodSpeed gap was (a) an instruction-count artifact of pure-Python per-character -loops, whose single biggest contributor — the zero-width scanner — was removed by -#72, and (b) measured before #72 landed. What remains below is a menu to *extend* -the lead, not to catch up. - -This document is backed by three reusable harnesses (the "workflow"): -`benchmarks/phase_profile.py`, `benchmarks/differential_profile.py`, -`benchmarks/abbr_scan_compare.py`. - ---- - -## 1. The workflow (how every number here is reproduced) - -```bash -# Per-phase wall-time attribution of our own call: -uv run python benchmarks/phase_profile.py --size short --iters 20000 - -# Head-to-head vs pySBD: wall time, regex-op counts, time-in-re, top funcs: -uv run python benchmarks/differential_profile.py --size short -uv run python benchmarks/differential_profile.py --size medium - -# Aho-Corasick scan vs naive `in`-loop crossover: -uv run python benchmarks/abbr_scan_compare.py - -# Instruction-count proxy (Python line-events ≈ what CodSpeed counts): -# sys.settrace line-event count of ours vs pysbd (see §3). -``` - -CodSpeed's CI mode counts **CPU instructions** (cachegrind), not wall time. The -two diverge sharply here, so we measure both: wall-clock for real latency, and a -`sys.settrace` line-event count as a faithful, local instruction-count proxy. - ---- - -## 2. Measured state, post-#72 - -`segment()` on the short (87c) and medium (198c) samples, Python 3.13, warm, -8000 iters (wall-clock reproduced across runs; line-events are deterministic): - -| metric | input | ours | pySBD | ratio | -|--------|-------|-----:|------:|------:| -| wall-clock µs/call | short | ~312 | ~316 | **0.99×** | -| wall-clock µs/call | medium | ~689 | ~743 | **0.93×** | -| regex ops/call | short | 118 | 266 | **we run fewer** | -| line-events/call (instr. proxy) | short | 2059 | 2203 | **0.93×** | - -We are at parity or ahead on every axis. Note we run **fewer** regex ops than -pySBD (118 vs 266) — the "we do more passes" theory is false; pySBD does ~2× the -regex ops but they are tiny literal-replacement subs. - -### 2a. Abbreviation *discovery* is not the bottleneck (and AC is fine) - -`benchmarks/abbr_scan_compare.py` (199 patterns, identical match sets): - -| input | AC | `in`-loop | winner | -|-------|---:|----------:|--------| -| tiny (15c) | 2.4 µs | 8.2 µs | **AC 3.4×** | -| short (87c) | 11.3 µs | 13.5 µs | **AC** | -| medium (198c)| 30.5 µs | 21.0 µs | `in` 1.4× | -| large (4k) | 619 µs | 286 µs | `in` 2.2× | - -Our pure-Python Aho-Corasick is *faster* than a pySBD-style `in`-loop on short -input; `in` only overtakes around ~150 chars. The scan is ~3% of the call. **Do -not replace it.** (It can, however, be made ~2× faster outright — see §4, P-AC.) - ---- - -## 3. What the original gap actually was (root cause) - -A `sys.settrace` line-event count (≈ instructions executed) on the **pre-#72** -code: ours **2787** vs pySBD **2243** = **1.24×** — almost exactly the ~28% -CodSpeed gap, and far above the ~1.06× wall-clock ratio. The gap was concentrated -in pure-Python per-character loops that cost almost nothing in wall-clock but -execute hundreds of *counted* interpreted operations: - -| our function | line-events/call | wall-clock cost | pySBD equivalent | -|--------------|-----------------:|-----------------|------------------| -| `_strip_zero_width_before_sentence_closers` | **876** | **~0 µs** | none (pySBD has no zero-width handling) | -| `AhoCorasickAutomaton.search` | 394 | *faster* than pySBD's scan | C-level `abbr in lowered` | -| `_GluedLowercaseRunOnRegex.sub` (`lang/common/standard.py:11`) | 309 | small | one C `re.sub` | -| `apply_rules` / `_sub_symbols_fast` | 156 / 132 | small | C `re.sub` | - -**Two distinct causes, two distinct truths:** -1. **CodSpeed instruction-count gap (~28%)** — largely an *artifact*: CPython - pushes pySBD's work into the C `re` engine (≈1 counted op per `re.sub`), while - our pure-Python loops are fully counted. The zero-width scanner alone was 876 - of our 2787 events. **#72 gated it** (`segmenter.py:89`, return early when no - zero-width char), cutting ~730 events → post-#72 proxy is **2059 vs 2203 = - 0.93×, we are ahead.** -2. **Wall-clock gap (~6–9%, pre-#72)** — real, and came from our heavier - *non-destructive* pipeline (span mapping `_match_spans`/`_find_sentence_start`, - the resplit passes, the sentinel-escape disjointness check, and callback-driven - subs), not regex efficiency. #72's zero-width guard plus normal variance brings - short to parity; medium we already win. - -**So #72 — framed at the time as "helps large text" — actually closed the -short/medium gap, because the zero-width scanner ran per output segment and -dominated the instruction count on short input. Its wall-clock effect on short -was within noise; its instruction-count effect was the whole ballgame.** The -competitive CodSpeed benchmark on `main` (post-#72) should now show short/medium -at parity-or-ahead; the next competitive run will confirm. - ---- - -## 4. Menu to *extend* the lead (all behavior-preserving / byte-identical) - -We are no longer catching up, so these are prioritized by **(improves both -metrics) > (improves one)** × leverage × safety. Implement test-first on isolated -branches with a byte-identical gate (full suite + Golden Rules + a corpus diff; -auto-revert on regression). Re-measure each with `phase_profile.py`, -`differential_profile.py`, and the competitive CodSpeed run. - -### Tier 1 — improves BOTH wall-clock and instruction-count - -- **P-LIST — List-item early-out + guards (~18% phase).** In - `lists_item_replacer.py`: (a) early-`return` in `iterate_alphabet_array` when - `re.findall` is empty (kills the 4×/call 26-entry `alphabet_index` dict builds); - (b) skip the numbered formatters when the text has no digit, the parens - formatters when no `)`; (c) reuse the identical alphabetical `findall` across - the roman/non-roman passes (4 scans → 2). Each guard char is required by every - alternative of its regex, so skipping is byte-identical. **Guard inside the - formatter methods** so Slovak's `add_line_break` override inherits them. - Validate with a differential oracle (old vs new `add_line_break` char-for-char). - *Risk:* low (items a/b), moderate (item c — confirm shared pattern+flags). -- **P-RESPLIT/CALLBACK — Gate the non-destructive passes + de-callback subs.** Per - Agent C, the real wall-clock cost is span mapping + resplit + callback subs. - Extend the eager-gate discipline already in `_maybe_resplit_multi_sentence_quote` - to the other resplit/postprocess passes (skip `_split_on_uppercase_boundary` - scans for segments with no `.)`/multi-terminator), and replace constant-result - callback subs with literal-string subs / `str.replace` (the - punctuation/continuous/double-punct callbacks). *Risk:* medium — touches - boundary-adjacent code; needs the full byte-identical gate. -- **P-ABBR — Gate the always-on abbreviation `replace()` passes.** - `AbbreviationReplacer.replace()` runs ~25 `re.sub`/call; gate the rarely-firing - ones (`_COMPACT_AMPM_RE`, `_UPPERCASE_INITIALISM_BOUNDARY_RE`, non-ASCII a.m./p.m. - restores, allcaps-imprint, the a.m./p.m. set) on a cheap presence check (digit - adjacency, `∯`-sentinel presence, all-caps run). *Risk:* low. - -### Tier 2 — Aho-Corasick ~2× (helps both; bigger on medium/large) - -- **P-AC — DFA δ-table precompute + length-gated hybrid.** Empirically (Agent A): - precomputing fail-links into a flat DFA table (removing the inner `while` - fail-link loop in `AhoCorasickAutomaton.search`) gives **short 9.8→4.9 µs (~2×), - long 270→~160 µs (~1.7×)**, drop-in, cached, ~2 ms one-time build. Add a - length-gated hybrid (DFA under ~200 chars, naive C `in`-loop above) to beat both - the automaton and pySBD's `in`-loop across all sizes. *Risk:* low–moderate (two - code paths; ship a `dfa == naive == current` fuzz/property test). Rejected by - measurement and not to be tried: `re`-alternation (3–5× slower), byte/array/tuple - transition tables (all slower than `dict.get`), first-char prefilter (English's - 24 first-chars defeat it). - -### Tier 3 — instruction-count only (low wall-clock value) - -- **P-GLUED — rewrite the `_GluedLowercaseRunOnRegex` Python char scanner - (`lang/common/standard.py:11`) into a single C-engine `re` call.** ~309 - line-events, near-zero wall-clock. Only worth it if the CodSpeed number is a - release gate; it barely moves real latency. - ---- - -## 5. Guardrails & non-goals - -- **Byte-identical** `segment()`/`segment_spans()` output is the bar. Full suite + - Golden Rules green + a corpus-wide old-vs-new diff per change; auto-revert on - any regression; one change per branch/PR so CodSpeed attributes each delta. -- **Do not touch the abbreviation discovery semantics** — AC is fine and faster - than `in` on short (§2a). P-AC keeps the automaton, only speeds its inner loop. -- **No boundary-semantics changes** to chase speed; if a change can't be made - byte-identical it is out of scope. -- **Pick the metric deliberately.** If the goal is *real latency*, prioritize - Tier 1 (and skip Tier 3). If the goal is the *CodSpeed gate number*, Tier 2/3 - (the pure-Python loops) move it most. They overlap in Tier 1, which is why Tier - 1 leads. - -**Honest expectation:** we are already at parity/ahead, so these are -incremental "pull further ahead" wins, not a turnaround. Tier 1 + P-AC together -are the worthwhile set; treat each gain as unproven until measured per-change. From 24ae8a136226096048f1d611456eba4c1957defa Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 05:14:03 +0000 Subject: [PATCH 10/11] =?UTF-8?q?fix:=20keep=20bare=20automaton=20key=20fo?= =?UTF-8?q?r=20'i'-ending=20abbreviations=20(U+0130=20=C4=B0)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The '.' period pre-filter (eb7a1c2) is searched on text.lower(), but U+0130 'İ' is the only Unicode char whose .lower() changes length ('İ' -> 'i' + U+0307). An abbreviation occurrence ending in 'İ' followed by a period lowers to '...i̇.', so the 'vi.'-style key missed it and the period over-split (e.g. de 'Band vİ. Der Rest folgt.' wrongly split in 2). Keep the bare key for 'i'-ending abbreviations — the original, always byte-identical behavior — and the period pre-filter for all others. Found by adversarial review. Regression test added (red before, green after) plus a guard asserting 'İ' remains the only length-changing lowercase char. --- benchmarks/abbr_scan_compare.py | 7 ++-- sentencesplit/abbreviation_replacer.py | 10 +++++- .../regression/test_abbreviation_dotted_i.py | 33 +++++++++++++++++++ 3 files changed, 46 insertions(+), 4 deletions(-) create mode 100644 tests/regression/test_abbreviation_dotted_i.py diff --git a/benchmarks/abbr_scan_compare.py b/benchmarks/abbr_scan_compare.py index cb442a9..4b575ac 100644 --- a/benchmarks/abbr_scan_compare.py +++ b/benchmarks/abbr_scan_compare.py @@ -46,9 +46,10 @@ def _bench(fn, text: str) -> float: def main() -> None: data = _AbbreviationData(LANGUAGE_CODES["en"].Abbreviation) automaton = data.automaton - # The automaton is keyed on "." (the trailing period pre-filter), so the - # equivalent naive loop tests for "." too. - abbr_keys = [a[1] + "." for a in data.abbreviations] # stripped, lowercased, + '.' + # The automaton is keyed on "." (the trailing-period pre-filter), except + # 'i'-ending abbreviations which keep the bare key (U+0130 'İ' caveat). Mirror + # that here so the naive loop matches the automaton's match set. + abbr_keys = [a[1] if a[1].endswith("i") else a[1] + "." for a in data.abbreviations] def ac_scan(text: str) -> set[int]: return automaton.search(text) diff --git a/sentencesplit/abbreviation_replacer.py b/sentencesplit/abbreviation_replacer.py index 67ae972..98f4117 100644 --- a/sentencesplit/abbreviation_replacer.py +++ b/sentencesplit/abbreviation_replacer.py @@ -172,7 +172,15 @@ def __init__(self, lang_abbreviation_class): # bare form merely appears inside other words (e.g. "al" in "called", # "no" in "no one") with no following period — the dominant cost on # real prose, where common short abbreviations match everywhere. - self.automaton.add_pattern(stripped_lower + ".", idx) + # + # Exception: the automaton is searched on ``text.lower()`` and U+0130 + # 'İ' is the only Unicode char whose .lower() changes length ('İ' -> + # 'i' + U+0307 combining dot). An occurrence ending in 'İ' followed by + # a period lowers to '...i̇.', so the "." key (e.g. "vi.") would + # not match. Abbreviations ending in 'i' therefore keep the bare key + # (the original, always-correct behavior). + key = stripped_lower if stripped_lower.endswith("i") else stripped_lower + "." + self.automaton.add_pattern(key, idx) self.automaton.build() self.abbr_set = frozenset(a.strip().lower() for a in raw) self.prepositive_set = frozenset(a.lower() for a in lang_abbreviation_class.PREPOSITIVE_ABBREVIATIONS) diff --git a/tests/regression/test_abbreviation_dotted_i.py b/tests/regression/test_abbreviation_dotted_i.py new file mode 100644 index 0000000..343f733 --- /dev/null +++ b/tests/regression/test_abbreviation_dotted_i.py @@ -0,0 +1,33 @@ +# -*- coding: utf-8 -*- +"""Regression: the abbreviation period-prefilter must handle U+0130 'İ'. + +The automaton is keyed on '.' as a pre-filter, but it is searched on +``text.lower()``. U+0130 'İ' (LATIN CAPITAL LETTER I WITH DOT ABOVE) is the only +Unicode character whose ``.lower()`` changes length — it expands to 'i' + a +combining dot above (U+0307). So an abbreviation occurrence ending in 'İ' and +followed by a period becomes '...i̇.' when lowered, and the '.' key (e.g. +'vi.') no longer matches: the abbreviation is missed and the period over-splits. +Abbreviations ending in 'i' therefore keep the bare key (the original behavior). +""" + +import unicodedata + +from sentencesplit import Segmenter + + +def test_dotted_capital_i_does_not_break_abbreviation_prefilter(): + # 'vi' is a German NUMBER_ABBREVIATION (Roman numeral). The 'İ' spelling must + # segment identically to the plain-ASCII 'vi' control: one joined sentence. + ascii_control = Segmenter(language="de").segment("Band vi. Der Rest folgt.") + dotted_i = Segmenter(language="de").segment("Band vİ. Der Rest folgt.") + assert len(dotted_i) == len(ascii_control) == 1, dotted_i + assert dotted_i == ["Band vİ. Der Rest folgt."], dotted_i + + +def test_dotted_capital_i_is_still_the_only_length_changing_lowercase(): + # The fix assumes 'İ' is the sole char whose .lower() is multi-char (so only + # 'i'-ending abbreviations are affected). If a future Unicode update adds + # another, this fails loudly so the prefilter logic can be revisited. + expanding = [chr(c) for c in range(0x110000) if len(chr(c).lower()) != 1] + assert expanding == ["İ"], expanding + assert "İ".lower() == "i" + unicodedata.lookup("COMBINING DOT ABOVE") From 3ff22345766e248b4a807aed1e135812a710ea62 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 14 Jun 2026 05:16:17 +0000 Subject: [PATCH 11/11] fix(bench): stop double-counting regex ops in differential_profile; doc thread-safety invariants - differential_profile._regex_op_summary counted both the module-level re.sub/re.findall wrapper frame and the builtin it delegates to, double-counting uncompiled calls and biasing the op-count comparison against pre-compiled patterns. Count only the Pattern-method calls. Corrected short-input figure: ours 97 vs pysbd 133 ops/call (was reported 118 vs 266); wall-clock unaffected. - Document the load-bearing lock-ordering invariant in _evict_profile and the AhoCorasickAutomaton publish-after-build thread-safety contract (both surfaced by the concurrency review; no behavior change). --- benchmarks/differential_profile.py | 21 ++++++++++++--------- sentencesplit/abbreviation_replacer.py | 10 +++++++++- sentencesplit/languages.py | 5 +++++ 3 files changed, 26 insertions(+), 10 deletions(-) diff --git a/benchmarks/differential_profile.py b/benchmarks/differential_profile.py index bd6248f..656ab65 100644 --- a/benchmarks/differential_profile.py +++ b/benchmarks/differential_profile.py @@ -59,20 +59,23 @@ def _profile(fn, text: str, iters: int) -> pstats.Stats: def _regex_op_summary(stats: pstats.Stats, iters: int) -> tuple[Counter, float]: - """Per-call regex op counts and total time spent in the re module.""" + """Per-call regex op counts and time spent in the regex engine. + + Counts ONLY the compiled-Pattern method calls (e.g. ````), not the module-level ``re.sub``/``re.findall`` + wrapper frames that delegate to them. Counting both double-counts an + uncompiled ``re.sub(str, ...)`` — which would bias a library that does not + pre-compile (it routes through the wrapper) against one that does. + """ counts: Counter = Counter() re_time = 0.0 - for (filename, _lineno, funcname), (_cc, nc, tt, _ct, _cb) in stats.stats.items(): - if funcname in _REGEX_OPS and "re" in filename.lower().replace("\\", "/").split("/"): - counts[funcname] += nc - if filename.endswith("re/__init__.py") or filename.endswith("re.py") or "/re/" in filename.replace("\\", "/"): - re_time += tt - # method 'sub' of 're.Pattern' objects shows up as a builtin - if funcname.startswith(" None: class) and the per-``Abbreviation``-class Aho-Corasick data (keyed on ``language_cls.Abbreviation``); otherwise a re-registered class whose abbreviation list changed would keep a stale automaton. + + Lock ordering (load-bearing): the two cache locks are acquired *sequentially* + (never co-held) while the caller holds ``_LANGUAGE_LOCK``. No reader path ever + acquires ``_LANGUAGE_LOCK`` while holding ``_PROFILE_CACHE_LOCK`` or + ``_cache_lock``, so there is no lock-ordering cycle. Preserve that invariant. """ from sentencesplit.abbreviation_replacer import AbbreviationReplacer from sentencesplit.language_profile import _PROFILE_CACHE, _PROFILE_CACHE_LOCK