feat: add explicit PyPy and Pyodide support - #74
Merged
Merged
Conversation
`_LazyLanguageCodes` reached into its real backing store through the bound `dict.keys(self)` view (in `__len__` and `__iter__`). On PyPy that view's `__len__` re-dispatches to this subclass's overridden `__len__`, so `set(dict.keys(self))` recursed infinitely — `len(LANGUAGE_CODES)` and `set(LANGUAGE_CODES)` (e.g. pytest's `parametrize(sorted(LANGUAGE_CODES))`) crashed at import/collection time with RecursionError. CPython's view reports length from C-level storage, so the bug never surfaced there. Snapshot the backing keys with a plain comprehension over the *unbound* `dict.__iter__(self)` iterator instead. A comprehension length-hints nothing and reads the raw keys without invoking any overridden method, so it is correct on both CPython and PyPy (`dict.copy(self)` is unusable here: on CPython it routes through the overridden `keys()`).
PyPy 3.11 (7.3.x) does not enforce the minimum of a possessive quantifier whose
minimum is >= 1 (`++`, `{n}+`, `{n,m}+`, `{n,}+`): when fewer than the minimum
repetitions are present it matches *zero* (a spurious zero-width match) instead
of failing. Minimal repro: `re.search(r'a++', 'b')` returns None on CPython but
an empty match on PyPy.
This corrupted `clean=True` output: `TableOfContentsRule`'s `\.{4,}+` matched
zero dots and deleted ordinary trailing numbers ("Send it to P.O. box 6554" ->
"... box"), and the HTML/escaped-HTML rules' `\w++` / `[^&]++` mis-handled tags
and escaped comparisons in prose ("x < 5 and y > 3").
Replace each min-1 possessive with the equivalent atomic group `(?>X+)` (and
`(?>\.{4,})` for the dot leader): a plain counted/`+` repeat — which PyPy counts
correctly — wrapped so it cannot backtrack. This is exactly equivalent to the
possessive form on CPython and keeps the ReDoS-safety guarantee, while being
correct on PyPy. `*+` (minimum 0) is left as-is; it is handled correctly.
The library is pure Python with zero runtime dependencies and ships a universal `py3-none-any` wheel, so it runs unmodified on PyPy and on Pyodide (CPython on WebAssembly). Make that support explicit and CI-verified: - CI: add a `pypy-test` job (full suite on PyPy 3.11) and a `pyodide-test` job that builds the wheel, installs it into Pyodide via micropip, and runs a Node smoke test (`tests/pyodide/smoke_test.mjs`) exercising the public API. Both gate the `build` job. - pyproject: add the CPython and PyPy implementation classifiers. - README: document the supported runtimes and browser/`micropip` usage. - mypy refuses to run under PyPy, so the lone mypy-subprocess test self-skips there (lint/type-check stay on the CPython jobs).
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| 🆕 | WallTime | test_walltime_segment_medium[pypy] |
N/A | 273.3 µs | N/A |
| 🆕 | WallTime | test_walltime_throughput[pypy] |
N/A | 2.5 ms | N/A |
| 🆕 | WallTime | test_walltime_segment_medium[cpython] |
N/A | 379.7 µs | N/A |
| 🆕 | WallTime | test_walltime_throughput[cpython] |
N/A | 6.1 ms | N/A |
Comparing claude/package-pyodide-compat-yrke5l (890a0bb) with main (04066c6)
yisding
marked this pull request as ready for review
June 14, 2026 05:39
PyPy can't use CodSpeed's deterministic `simulation` (instruction-count) mode — it instruments the CPython interpreter, which PyPy's JIT precludes — so add a `benchmarks-pypy` job that measures the same hot paths in `walltime` mode. This tracks PyPy performance over time and catches PyPy-specific regressions, while CPython regression detection stays on the existing deterministic job. CodSpeed can't report the same benchmark id twice for one commit, so the PyPy benchmarks live in a dedicated file (test_pypy_walltime_codspeed.py) with ids disjoint from the CPython simulation suite. Both pull the same corpus, now extracted to benchmarks/_samples.py, so the runtimes exercise identical inputs and line up case-for-case on the dashboard.
Replace the 10 PyPy-only walltime benchmarks with 4 total: two scenarios — per-call latency (segment a medium paragraph) and batch throughput (segment a ~5 KB document) — each run on both CPython and PyPy and labelled by interpreter (test_walltime_segment_medium[cpython|pypy], test_walltime_throughput[...]) so the pairs sit side-by-side on the dashboard for direct comparison. Both walltime jobs collect the shared file; each runs only the half labelled for its interpreter and self-skips the rest, so every benchmark id is reported exactly once per commit. The deterministic CPython `simulation` regression suite (test_latency_codspeed.py) is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes PyPy and Pyodide support explicit and CI-verified. The library is pure Python with zero runtime dependencies and ships a universal
py3-none-anywheel, so it runs unmodified on both — but getting there surfaced two real PyPy bugs (one a crash, one silent output corruption) that are now fixed and regression-guarded.Pyodide (CPython compiled to WebAssembly) worked out of the box; verified locally against Pyodide 314.0.0 / Python 3.14.2.
Bugs found & fixed
1. Infinite recursion in
LANGUAGE_CODESon PyPy (crash)_LazyLanguageCodesread its backing store via the bounddict.keys(self)view. On PyPy that view's__len__re-dispatches to the subclass's overridden__len__, soset(dict.keys(self))recursed infinitely —len(LANGUAGE_CODES)/set(LANGUAGE_CODES)(e.g. pytest'sparametrize(sorted(LANGUAGE_CODES))) crashed the whole suite at collection time withRecursionError. CPython reads the view length from C-level storage, so it never surfaced there.Fix: snapshot the backing keys with a plain comprehension over the unbound
dict.__iter__(self)iterator — length-hints nothing, invokes no overridden method, correct on both runtimes.2. Possessive quantifiers ignore their minimum on PyPy (silent corruption)
PyPy 3.11 (7.3.x) does not enforce the minimum of a possessive quantifier whose minimum is ≥ 1 (
++,{n}+,{n,m}+,{n,}+): it matches zero repetitions instead of failing. Minimal repro:This corrupted
clean=Trueoutput:TableOfContentsRule's\.{4,}+matched zero dots and deleted ordinary trailing numbers ("Send it to P.O. box 6554"→"… box"), and the HTML rules'\w++/[^&]++mangled tags and escaped comparisons in prose ("x < 5 and y > 3").Fix: replace each min-≥1 possessive with the equivalent atomic group
(?>X+)/(?>\.{4,})— a plain repeat (counted correctly by PyPy) made non-backtracking. Exactly equivalent to the possessive form on CPython, keeps the ReDoS-safety guarantee, correct on PyPy.*+(minimum 0) is left as-is.Support / tooling
pypy-testjob (full suite on PyPy 3.11) andpyodide-testjob (build wheel →micropipinstall into Pyodide → Node smoke test of the public API). Both gatebuild.micropipusage.Verification
__version__all correct.ruff check+ruff format --checkclean.New regression tests:
tests/regression/test_lazy_language_codes_views.py,tests/regression/test_pypy_possessive_quantifier.py. They assert correct behaviour on every interpreter and fail loudly on PyPy if either bug returns.Note: upstream PyPy bug
Bug #2 looks like an unreported PyPy
rebug (possessive minimum not enforced). I searched the PyPy tracker and found no existing report — details and a ready-to-file write-up are in the PR thread / session for maintainer follow-up.https://claude.ai/code/session_018tBCqG2oj6A4RSy22o1qjg
Generated by Claude Code