Skip to content

feat: add explicit PyPy and Pyodide support - #74

Merged
yisding merged 5 commits into
mainfrom
claude/package-pyodide-compat-yrke5l
Jun 14, 2026
Merged

yisding merged 5 commits into
mainfrom
claude/package-pyodide-compat-yrke5l

Conversation

@yisding

@yisding yisding commented Jun 14, 2026

Copy link
Copy Markdown
Owner

Summary

Makes PyPy and Pyodide support explicit and CI-verified. The library is pure Python with zero runtime dependencies and ships a universal py3-none-any wheel, so it runs unmodified on both — but getting there surfaced two real PyPy bugs (one a crash, one silent output corruption) that are now fixed and regression-guarded.

Pyodide (CPython compiled to WebAssembly) worked out of the box; verified locally against Pyodide 314.0.0 / Python 3.14.2.

Bugs found & fixed

1. Infinite recursion in LANGUAGE_CODES on PyPy (crash)

_LazyLanguageCodes read its backing store via the bound dict.keys(self) view. On PyPy that view's __len__ re-dispatches to the subclass's overridden __len__, so set(dict.keys(self)) recursed infinitely — len(LANGUAGE_CODES) / set(LANGUAGE_CODES) (e.g. pytest's parametrize(sorted(LANGUAGE_CODES))) crashed the whole suite at collection time with RecursionError. CPython reads the view length from C-level storage, so it never surfaced there.

Fix: snapshot the backing keys with a plain comprehension over the unbound dict.__iter__(self) iterator — length-hints nothing, invokes no overridden method, correct on both runtimes.

2. Possessive quantifiers ignore their minimum on PyPy (silent corruption)

PyPy 3.11 (7.3.x) does not enforce the minimum of a possessive quantifier whose minimum is ≥ 1 (++, {n}+, {n,m}+, {n,}+): it matches zero repetitions instead of failing. Minimal repro:

import re
re.search(r'a++', 'b')   # CPython: None  |  PyPy: <empty match at 0>

This corrupted clean=True output: TableOfContentsRule's \.{4,}+ matched zero dots and deleted ordinary trailing numbers ("Send it to P.O. box 6554""… box"), and the HTML rules' \w++ / [^&]++ mangled tags and escaped comparisons in prose ("x &lt; 5 and y &gt; 3").

Fix: replace each min-≥1 possessive with the equivalent atomic group (?>X+) / (?>\.{4,}) — a plain repeat (counted correctly by PyPy) made non-backtracking. Exactly equivalent to the possessive form on CPython, keeps the ReDoS-safety guarantee, correct on PyPy. *+ (minimum 0) is left as-is.

Support / tooling

  • CI: new pypy-test job (full suite on PyPy 3.11) and pyodide-test job (build wheel → micropip install into Pyodide → Node smoke test of the public API). Both gate build.
  • pyproject: add CPython + PyPy implementation classifiers.
  • README: document supported runtimes and browser/micropip usage.
  • mypy refuses to run under PyPy, so the lone mypy-subprocess test self-skips there (lint/type-check remain on the CPython jobs).

Verification

  • PyPy 3.11.13 (7.3.20), full suite: 1976 passed, 1 skipped, 6 xfailed.
  • CPython 3.13, full suite: 1977 passed, 6 xfailed, coverage 94.78% (≥ 93 gate).
  • Pyodide 314.0.0 smoke test: English/CJK/clean-TOC segmentation, 26 languages, __version__ all correct.
  • ruff check + ruff format --check clean.

New regression tests: tests/regression/test_lazy_language_codes_views.py, tests/regression/test_pypy_possessive_quantifier.py. They assert correct behaviour on every interpreter and fail loudly on PyPy if either bug returns.

Note: upstream PyPy bug

Bug #2 looks like an unreported PyPy re bug (possessive minimum not enforced). I searched the PyPy tracker and found no existing report — details and a ready-to-file write-up are in the PR thread / session for maintainer follow-up.

https://claude.ai/code/session_018tBCqG2oj6A4RSy22o1qjg


Generated by Claude Code

claude added 3 commits June 14, 2026 05:33
`_LazyLanguageCodes` reached into its real backing store through the bound
`dict.keys(self)` view (in `__len__` and `__iter__`). On PyPy that view's
`__len__` re-dispatches to this subclass's overridden `__len__`, so
`set(dict.keys(self))` recursed infinitely — `len(LANGUAGE_CODES)` and
`set(LANGUAGE_CODES)` (e.g. pytest's `parametrize(sorted(LANGUAGE_CODES))`)
crashed at import/collection time with RecursionError. CPython's view reports
length from C-level storage, so the bug never surfaced there.

Snapshot the backing keys with a plain comprehension over the *unbound*
`dict.__iter__(self)` iterator instead. A comprehension length-hints nothing and
reads the raw keys without invoking any overridden method, so it is correct on
both CPython and PyPy (`dict.copy(self)` is unusable here: on CPython it routes
through the overridden `keys()`).
PyPy 3.11 (7.3.x) does not enforce the minimum of a possessive quantifier whose
minimum is >= 1 (`++`, `{n}+`, `{n,m}+`, `{n,}+`): when fewer than the minimum
repetitions are present it matches *zero* (a spurious zero-width match) instead
of failing. Minimal repro: `re.search(r'a++', 'b')` returns None on CPython but
an empty match on PyPy.

This corrupted `clean=True` output: `TableOfContentsRule`'s `\.{4,}+` matched
zero dots and deleted ordinary trailing numbers ("Send it to P.O. box 6554" ->
"... box"), and the HTML/escaped-HTML rules' `\w++` / `[^&]++` mis-handled tags
and escaped comparisons in prose ("x &lt; 5 and y &gt; 3").

Replace each min-1 possessive with the equivalent atomic group `(?>X+)` (and
`(?>\.{4,})` for the dot leader): a plain counted/`+` repeat — which PyPy counts
correctly — wrapped so it cannot backtrack. This is exactly equivalent to the
possessive form on CPython and keeps the ReDoS-safety guarantee, while being
correct on PyPy. `*+` (minimum 0) is left as-is; it is handled correctly.
The library is pure Python with zero runtime dependencies and ships a universal
`py3-none-any` wheel, so it runs unmodified on PyPy and on Pyodide (CPython on
WebAssembly). Make that support explicit and CI-verified:

- CI: add a `pypy-test` job (full suite on PyPy 3.11) and a `pyodide-test` job
  that builds the wheel, installs it into Pyodide via micropip, and runs a Node
  smoke test (`tests/pyodide/smoke_test.mjs`) exercising the public API. Both
  gate the `build` job.
- pyproject: add the CPython and PyPy implementation classifiers.
- README: document the supported runtimes and browser/`micropip` usage.
- mypy refuses to run under PyPy, so the lone mypy-subprocess test self-skips
  there (lint/type-check stay on the CPython jobs).
@codspeed

codspeed Bot commented Jun 14, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 22 untouched benchmarks
🆕 4 new benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 WallTime test_walltime_segment_medium[pypy] N/A 273.3 µs N/A
🆕 WallTime test_walltime_throughput[pypy] N/A 2.5 ms N/A
🆕 WallTime test_walltime_segment_medium[cpython] N/A 379.7 µs N/A
🆕 WallTime test_walltime_throughput[cpython] N/A 6.1 ms N/A

Comparing claude/package-pyodide-compat-yrke5l (890a0bb) with main (04066c6)

Open in CodSpeed

@yisding
yisding marked this pull request as ready for review June 14, 2026 05:39
claude added 2 commits June 14, 2026 05:59
PyPy can't use CodSpeed's deterministic `simulation` (instruction-count) mode —
it instruments the CPython interpreter, which PyPy's JIT precludes — so add a
`benchmarks-pypy` job that measures the same hot paths in `walltime` mode. This
tracks PyPy performance over time and catches PyPy-specific regressions, while
CPython regression detection stays on the existing deterministic job.

CodSpeed can't report the same benchmark id twice for one commit, so the PyPy
benchmarks live in a dedicated file (test_pypy_walltime_codspeed.py) with ids
disjoint from the CPython simulation suite. Both pull the same corpus, now
extracted to benchmarks/_samples.py, so the runtimes exercise identical inputs
and line up case-for-case on the dashboard.
Replace the 10 PyPy-only walltime benchmarks with 4 total: two scenarios —
per-call latency (segment a medium paragraph) and batch throughput (segment a
~5 KB document) — each run on both CPython and PyPy and labelled by interpreter
(test_walltime_segment_medium[cpython|pypy], test_walltime_throughput[...]) so
the pairs sit side-by-side on the dashboard for direct comparison.

Both walltime jobs collect the shared file; each runs only the half labelled for
its interpreter and self-skips the rest, so every benchmark id is reported
exactly once per commit. The deterministic CPython `simulation` regression suite
(test_latency_codspeed.py) is unchanged.
@yisding
yisding merged commit 435c3eb into main Jun 14, 2026
13 checks passed
@yisding
yisding deleted the claude/package-pyodide-compat-yrke5l branch June 14, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants