CJK multi-sentence quotations suppress interior sentence boundaries (no CJK analog of _resplit_multi_sentence_quote)

[yisding]

Summary

When a CJK (zh / ja) quotation wraps multiple complete sentences and the
closing quote (」 / 』) only arrives at the end, the whole quoted span
collapses into a single segment — all interior sentence boundaries
(。 / ！ / ？) are suppressed. The identical defect for Latin/Greek/Cyrillic
text was fixed via _resplit_multi_sentence_quote (see case_0080),
but CJK is on a different code path that has no equivalent, so it was left
unaddressed.

This is the CJK analog of the case_0080 fix and tracks it as a focused follow-up
(deliberately scoped out of PR #36 / commit c3027e3, which generalized the
Latin/Greek/Cyrillic path only).

Reproduction

from sentencesplit import Segmenter
zh = Segmenter(language="zh")

# Unquoted — splits correctly:
zh.segment("第一句话在这里。第二句话在这里。第三句话在这里。")
# -> ['第一句话在这里。', '第二句话在这里。', '第三句话在这里。']   ✅

# Same three sentences wrapped in one 「…」 pair — collapses to ONE segment:
zh.segment("「第一句话在这里。第二句话在这里。第三句话在这里。」")
# -> ['「第一句话在这里。第二句话在这里。第三句话在这里。」']        ❌ BUG (expect 3)

ja is affected identically:

ja = Segmenter(language="ja")
ja.segment("「今日はいい天気です。散歩に行きます。家に帰ります。」")
# -> ['「今日はいい天気です。散歩に行きます。家に帰ります。」']      ❌ BUG (expect 3)

Expected (mirroring case_0080, opening quote rides sentence 1, closing quote
rides the last sentence):

['「第一句话在这里。', '第二句话在这里。', '第三句话在这里。」']

Root cause

Two independent facts combine:

1. The CJK languages never call _resplit_multi_sentence_quote. That helper
   runs only inside the latin_uppercase_resplit branch of
   Processor._resplit_segments (sentencesplit/processor.py:336), and the CJK
   profile sets latin_uppercase_resplit = False:

   zh: latin_uppercase_resplit=False
   ja: latin_uppercase_resplit=False
   en: latin_uppercase_resplit=True
   

   It also could not be reused as-is: it keys on a Latin/cased-letter sentence
   start (str.isupper() after a .), which is meaningless for caseless CJK.

2. The CJK resplit path only splits after a closing quote, not inside one.
   CJKProcessor (sentencesplit/lang/common/cjk.py) plus
   _CJK_QUOTE_RESPLIT_RE / _CJK_BANG_RESPLIT_RE
   (sentencesplit/processor.py:21,31) re-split at (?<=[。．!！?？][closing-quote])
   — i.e. a terminal immediately followed by a closing quote then new
   content. Interior 。 inside a still-open 「…」 are protected by the
   between-punctuation pass and never reconsidered. Hence:

   zh.segment("「天气很好。」他笑了。今天出门。")
   # -> ['「天气很好。」', '他笑了。', '今天出门。']   # splits AFTER 」, fine

   but a quote that stays open across several sentences is never re-split.

Complication: current in-quote behavior is already inconsistent

A nested inner quote already triggers a partial split, because
_CJK_QUOTE_RESPLIT_RE fires after the inner 』:

zh.segment("「他说『你好。再见。』然后走了。我看着他。」")
# -> ['「他说『你好。再见。』', '然后走了。我看着他。」']

So a fix needs to make in-quote splitting coherent, not just add a new case on
top of this.

Proposed approach

Add a CJK analog, e.g. _resplit_cjk_multi_sentence_quote, invoked from
CJKProcessor._resplit_segments (and the en_es_zh processor, which has its own
_resplit_segments). Mirror the structure and guard discipline of the Latin
helper (sentencesplit/processor.py:83):

• Fire only for a self-contained, un-nested quotation: a single matched CJK
  quote pair (「…」 or 『…』), opener near the start, matching closer at the
  end, no other quote characters in the interior.
• Split at interior CJK terminals 。．！？ (decide whether ！/？ runs count —
  see guards below).
• Reattach the opening quote to the first piece and the closing quote to the
  last piece.
• Apply minimum-size guards (analogous to _QUOTE_MIN_INTERIOR_SENTENCES /
  _QUOTE_MIN_WORDS, adapted to CJK where "word count" is not meaningful —
  likely a character-count floor per piece).
• Respect split_mode (conservative never splits; aggressive lowers thresholds),
  matching _quote_resplit_thresholds.

Guard cases a fix MUST respect

Gather real zh/ja gold before implementing; at minimum:

1. Reporting clause — keep whole. 他说：「…。…。…。」 is one reported
   sentence and must NOT split. This interacts with
   CJK_REPORTING_CLAUSE_RE / _merge_quote_continuations
   (sentencesplit/lang/common/cjk.py:18,89):

   zh.segment("他说：「第一句话在这里。第二句话在这里。第三句话在这里。」")
   # -> ['他说：「第一句话在这里。第二句话在这里。第三句话在这里。」']   # must stay 1

2. Nested quotes — keep whole (or split coherently), not the current partial
   behavior: 「他说『你好。再见。』然后走了。」
3. Emphatic ！/？ runs — keep whole, mirroring the Latin oh_dear /
   as_if_i_would guards:

   zh.segment("「快跑！危险！快！」")
   # -> ['「快跑！危险！快！」']   # one emphatic speech act

4. Single-boundary quote — keep whole (the CJK analog of the dinah /
   case_0110 guards): a two-sentence quote where the second clause may be a
   continuation.
5. Half/full-width punctuation parity (。 vs ．, ！/？).

Acceptance criteria

• zh and ja split a self-contained 「…。…。…。」 wrapping ≥3 sentences,
  opening/closing quotes attached to the first/last pieces.
• All guard cases above stay whole (new regression tests, zh + ja).
• Reporting-clause merge (他说：「…」) still produces one segment.
• en_es_zh combined profile behaves consistently (it has its own
  _resplit_segments).
• split_mode honored (conservative joins; aggressive lowers thresholds).
• Streaming (StreamSegmenter) over CJK stays byte-faithful / no text loss.
• Full suite + Golden Rules green; benchmark corpus (compare-segmenters)
  shows no regression on existing CJK cases.

Background / related work

• Latin/Greek/Cyrillic equivalent: _resplit_multi_sentence_quote
  (sentencesplit/processor.py:83), motivated by case_0080 (a Conan Doyle
  4-sentence quotation that punkt/syntok split and sentencesplit did not) —
  tests/regression/test_issues.py:908.
• Non-ASCII generalization of that helper: commit c3027e3
  (fix(processor): split multi-sentence quotations before non-ASCII capitals),
  PR feat: StreamSegmenter, byte-faithful segment_spans, hermetic regression gate, and list_languages #36. That change explicitly does not cover CJK (caseless, separate path)
  — this issue tracks closing that gap.

Relevant code

• sentencesplit/processor.py — _resplit_multi_sentence_quote (L83),
  _CJK_QUOTE_RESPLIT_RE (L21), _CJK_BANG_RESPLIT_RE (L31),
  Processor._resplit_segments (L336)
• sentencesplit/lang/common/cjk.py — CJKProcessor resplit /
  _merge_quote_continuations (L89), CJK_REPORTING_CLAUSE_RE (L18)
• sentencesplit/lang/en_es_zh.py — its own _resplit_segments