perf: cut segmentation + streaming latency (quote-resplit + single-pass streaming) by yisding · Pull Request #69 · yisding/sentencesplit

yisding · 2026-06-14T00:01:58Z

What

Latency-profiling harness plus two safe, behavior-preserving performance fixes it surfaced. No boundary/punctuation/language logic changes — all paths verified byte-identical to before.

Note: the CodSpeed CI infrastructure was split out into #70 so it can land on main first and capture a pre-optimization baseline. Once #70 merges and this branch rebases on main, CodSpeed will report these speedups as deltas automatically.

1. `benchmarks/latency_baseline.py`

Local wall-clock latency harness for the three latency-sensitive paths (segment(), should_wait_for_more(), StreamSegmenter.feed()) + a cProfile hot-function breakdown.

2. `perf:` skip abbreviation re-scan for quote-free segments

_resplit_segments evaluated protected_text=self.replace_abbreviations(...) eagerly for every sentence, though _resplit_multi_sentence_quote discards it unless the segment begins with a leading quote — so a 7-sentence quote-free string ran 8 abbreviation passes (1 needed + 7 wasted; ~58% of segment() time). Gated behind the same leading-quote check via _maybe_resplit_multi_sentence_quote.

3. `perf:` segment the streaming buffer once per delta

StreamSegmenter._detect ran two full segmentations of the buffer on every feed(). New Segmenter.segment_spans_with_lookahead() derives both the spans and the trailing-boundary verdict from one pass; _detect uses it.

4. `refactor:` precompile constant regexes

Hoisted the remaining constant-pattern raw re.sub/re.search calls in the hot path to compiled module/class constants. Within-noise on the warm benchmark (CPython's re cache already memoizes them) — kept for cache-eviction robustness + convention consistency.

Measured impact (Python 3.13, warm caches, local)

path	original	now	Δ
`segment()` short (87c)	0.58 ms	0.41 ms	−30%
`segment()` medium (198c)	1.29 ms	0.85 ms	−34%
`segment()` large (3979c)	20.6 ms	12.4 ms	−40%
`should_wait_for_more()` medium	2.00 ms	1.37 ms	−31%
`StreamSegmenter.feed()` conservative	0.56 ms/tok	0.29 ms/tok	−48%
`StreamSegmenter.feed()` aggressive	0.48 ms/tok	0.25 ms/tok	−48%

Correctness / tests

Full suite green: 1938 → 1967 passed, same 6 xfails.
Streaming output byte-identical to the prior implementation across 9 languages × 3 buffering modes × 3 feed granularities; char_span round-trips exact.
Per-language equivalence test: segment_spans_with_lookahead == segment_spans + should_wait_for_more for every LANGUAGE_CODE.
Both perf fixes have regression guards proven to fail on pre-fix code (8 == 1; 2 == 1).

https://claude.ai/code/session_01RBqMKEGgvcZEAXLTPP5FiV

codspeed · 2026-06-14T00:50:21Z

Merging this PR will improve performance by 54.38%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 9 improved benchmarks
✅ 1 untouched benchmark

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	`test_stream_feed_document[conservative]`	57.2 ms	26.9 ms	×2.1
⚡	`test_stream_feed_document[aggressive]`	48.8 ms	23.3 ms	×2.1
⚡	`test_segment_multilingual[ru]`	2.3 ms	1.5 ms	+47.16%
⚡	`test_should_wait_for_more[large]`	73.4 ms	50.6 ms	+44.93%
⚡	`test_segment[large]`	71.6 ms	49.5 ms	+44.66%
⚡	`test_should_wait_for_more[short]`	5.3 ms	3.7 ms	+41.94%
⚡	`test_should_wait_for_more[medium]`	5.6 ms	4 ms	+39.76%
⚡	`test_segment[medium]`	4 ms	2.9 ms	+38.92%
⚡	`test_segment[short]`	1.8 ms	1.4 ms	+31.88%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing claude/library-performance-avenues-pnykwq (c801546) with main (772fda5)}

Adds a focused wall-clock latency harness covering the three latency-sensitive paths (one-shot segment(), should_wait_for_more() lookahead, and StreamSegmenter.feed()) plus a cProfile hot-function breakdown. Used to establish a performance baseline.

_resplit_segments computed the abbreviation-protected scan (replace_abbreviations) eagerly for every postprocessed sentence, even though _resplit_multi_sentence_quote discards it immediately unless the segment begins with a leading quote. For a 7-sentence quote-free string that meant 8 full abbreviation passes (1 legitimate main-pipeline pass + 7 wasted per-sentence re-scans), the single largest contributor to segment() latency. Gate the protected-scan computation behind the same leading-quote check the resplit uses, via a new _maybe_resplit_multi_sentence_quote helper. Behavior is identical (the scan is only consulted after that gate), but quote-free segments now skip it entirely. Measured: ~30-40% lower segment()/should_wait_for_more() latency and ~23% lower per-token streaming cost; full suite unchanged (1938 -> 1939 with the added regression test guarding the call count).

…pass StreamSegmenter._detect ran two full segmentations of the unemitted buffer on every feed(): segment_spans() for the tail spans, then should_wait_for_more() — whose internal _segment_result re-ran the same process() + _match_spans mapping over the identical buffer purely to recover the trailing-boundary verdict. Add Segmenter.segment_spans_with_lookahead(), which derives both the spans and the should_wait verdict from a single process()/_match_spans pass (the spans' .sent values are exactly the comparison segments _wait_for_last_segment consumes), and have _detect use it. The boundary, punctuation, and lookahead logic are byte-for-byte unchanged across all supported languages — verified identical StreamSegmenter output vs. the prior implementation over 9 languages x 3 buffering modes x 3 feed granularities, with span round-trips still exact. Measured: ~35% lower per-token streaming latency (conservative 0.44 -> 0.29 ms/token; aggressive 0.39 -> 0.25 ms/token). Tail-probe cost for period-terminated boundaries is unchanged and still only runs for '.'. Adds per-language equivalence tests for the combined API and a guard that _detect maps the buffer exactly once per delta.

Hoist the remaining constant-pattern raw re.sub/re.search calls in the hot path (AbbreviationReplacer.replace and its a.m./p.m. and initialism rules, ListItemReplacer line-break guards and roman-numeral parens, ExclamationWords.apply_rules) and a few per-language constants (Slovak ordinal/roman/quote rules, the Russian conjunction-continuation check) to module/class-level compiled patterns, matching the existing _ALLCAPS_IMPRINT_RE convention. These patterns were recompiled via a string-literal each call. In practice CPython's internal re cache already memoizes them, so this is not a measurable speedup on the warm benchmark; the value is removing the latent re-cache-eviction cliff under many-pattern / multi-language workloads and matching the precompiled style used elsewhere. Behavior is unchanged: full suite still 1967 passed. Patterns that interpolate per-abbreviation variables (escaped/am_escaped/month) are genuinely dynamic and intentionally left as-is.

yisding changed the title ~~test: add latency baseline profiler~~ perf: skip abbreviation re-scan for quote-free segments + latency baseline Jun 14, 2026

yisding changed the title ~~perf: skip abbreviation re-scan for quote-free segments + latency baseline~~ perf: cut segmentation + streaming latency (quote-resplit + single-pass streaming) Jun 14, 2026

yisding marked this pull request as ready for review June 14, 2026 00:41

yisding mentioned this pull request Jun 14, 2026

ci: add CodSpeed latency benchmarks (baseline infra) #70

Merged

yisding force-pushed the claude/library-performance-avenues-pnykwq branch from 0ea4efa to 75fcebe Compare June 14, 2026 00:54

claude added 5 commits June 14, 2026 01:07

style: apply ruff format to latency baseline

1975bbb

yisding force-pushed the claude/library-performance-avenues-pnykwq branch from 75fcebe to c801546 Compare June 14, 2026 01:08

yisding merged commit 6b59901 into main Jun 14, 2026
9 checks passed

yisding deleted the claude/library-performance-avenues-pnykwq branch June 14, 2026 01:37

Navigation Menu

perf: cut segmentation + streaming latency (quote-resplit + single-pass streaming) - #69

perf: cut segmentation + streaming latency (quote-resplit + single-pass streaming)#69
yisding merged 5 commits into
mainfrom
claude/library-performance-avenues-pnykwq

yisding commented Jun 14, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Jun 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

yisding commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

1. benchmarks/latency_baseline.py

2. perf: skip abbreviation re-scan for quote-free segments

3. perf: segment the streaming buffer once per delta

4. refactor: precompile constant regexes

Measured impact (Python 3.13, warm caches, local)

Correctness / tests

Uh oh!

codspeed Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 54.38%

Performance Changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yisding commented Jun 14, 2026 •

edited

Loading

1. `benchmarks/latency_baseline.py`

2. `perf:` skip abbreviation re-scan for quote-free segments

3. `perf:` segment the streaming buffer once per delta

4. `refactor:` precompile constant regexes

codspeed Bot commented Jun 14, 2026 •

edited

Loading