Skip to content

fix(processor): delimit sentinel escape fallback - #49

Merged
yisding merged 2 commits into
mainfrom
codex/fix-sentinel-escape-corruption-issue
Jun 12, 2026
Merged

yisding merged 2 commits into
mainfrom
codex/fix-sentinel-escape-corruption-issue

Conversation

@yisding

@yisding yisding commented Jun 12, 2026

Copy link
Copy Markdown
Owner

Motivation

  • Prevent corruption of user input when multi-character private-use escape tokens are chosen and a restore operation can match across an original private-use character / escaped-token boundary.

Description

  • Wrap multi-character private-use escape token bodies in an absent noncharacter delimiter so restore matches are always aligned to whole escape tokens and cannot begin inside adjacent original private-use characters.
  • Add _NONCHARACTER_DELIMITER_RANGES, _iter_delimited_private_use_tokens, _iter_noncharacter_delimiters, and _absent_noncharacter_delimiter helpers, and update _build_sentinel_escape_tables to prefer single-codepoint escapes and fall back to delimited multi-codepoint tokens when necessary.
  • Make the compiled restore_re alternation length-sorted for safe left-to-right atomic restoration and keep str.translate/regex-based round-trip semantics.
  • Add a regression test test_sentinel_restore_does_not_match_across_original_private_use_boundary to tests/regression/test_library_review_fixes.py that exercises the cross-boundary case via both Processor.process() and the public segment_clean() path.

Testing

  • Ran the targeted regression file: uv run pytest tests/regression/test_library_review_fixes.py -q and it passed (59 passed).
  • Ran lint/format checks: uv run ruff check . and uv run ruff format --check . and both passed.
  • Ran full test suite with coverage: uv run pytest --cov=sentencesplit tests/ --color yes and the suite passed (1717 passed, 8 xfailed).

Codex Task

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 883c5a6c51

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sentencesplit/processor.py
@yisding
yisding merged commit a6ea1b6 into main Jun 12, 2026
7 checks passed
@yisding
yisding deleted the codex/fix-sentinel-escape-corruption-issue branch June 12, 2026 07:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant