RFC: single-pass period-classifier abbreviation engine - #75
Merged
Merged
Conversation
Captures the architectural analysis from the latency investigation: the abbreviation engine inherits pySBD's global-re.sub-rewrite model, which makes the per-period boundary decision O(distinct-abbreviations x text-length), order-dependent, and re-implemented six ways across languages. Argues that the linguistically-essential task is a local per-period classification that a single-pass design would make O(text), order-independent, and far simpler. Includes: essential-vs-accidental complexity analysis, the proposed classifier design and per-language policy hooks, the full preservation spec distilled from surveying every lang/ override (suffix-decision patterns, flags, and the genuinely hard parts), a phased English-first implementation plan, the guardrails (differential oracle, Golden-Rule anchor, all-26-language diff, fuzz, CodSpeed), acceptance criteria, and an honest risk/reward recommendation (default: leave it; v2 only as a deliberate major-version effort). No code changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A design RFC (
analysis/ABBREVIATION_ENGINE_V2_RFC.md) — no code changes — capturing the architectural analysis that came out of the latency investigation (which landed +21.7% separately, in #73).The core argument
The abbreviation engine inherits pySBD's model — segmentation as a sequence of global
re.subrewrites, decisions carried in-band as sentinels (∯). But deciding whether a period is a sentence boundary is an inherently local, per-period classification. Modeling it as repeated global rewrites turns that into:re.sub/call on dense legal text);∯) — bug-prone, and the main obstacle to optimizing the current code;A single-pass period classifier would be O(text), order-independent, and collapse the six per-language rewrites into one classifier + small policy hooks.
The honest catch
The current quirks (down to German's unescaped lookbehind and the order-dependent tie-breaks) are now the product spec — the historically-tuned golden output. So this isn't "refactor the engine," it's "re-derive every golden behavior in a new paradigm" — a multi-week, high-risk project.
What the RFC contains
AbbrPolicyhooks.lang/module: the suffix-decision patterns, the steering flags, and the genuinely hard parts (context-reading callbacks, thereplace_multi_periodinteraction, the&ᓷ&&ᓷ&placeholder injection, unescaped-amlanguages, whole-pipeline overrides).İbug), fuzz, CodSpeed, feature-flag parallel paths.Filed as a draft for discussion, not for merge-as-work.
https://claude.ai/code/session_01RBqMKEGgvcZEAXLTPP5FiV
Generated by Claude Code