Conversation
…ce per operation test_schema_generates called build_schema() and test_columns_resolve called lint() inside each of their 678 parametrized cases. Both walk the whole registry and evaluate every request's string annotations as they go, so the file was quadratic: about nine and a half of the suite's ten and a half minutes in a 3.14 container. On 3.14 it was also what took the Ubuntu job down. The default coverage core there is sys.monitoring, and its SysMonitor keeps every code object it sees alive (coverage/sysmon.py appends each one to `code_objects` so that an id is never reused), including the throwaway `<string>` code of each annotation `eval`. One build_schema() left about 9,200 of them behind and one lint() about 27,300: roughly 29 MB per case, 20 GB over the file. In an amd64 ubuntu:24.04 container with uv's CPython 3.14.7 and `make test`, the run grew by about 20 MB a second and was OOM killed at 86%. On a hosted runner the same growth starves the machine until the runner stops answering, which is the "hosted runner lost communication with the server" annotation on every 3.14 run since PR-12, and why those job logs are gone. PR-12 took the registry from 588 operations to 678, and the cost grows with the square of that. macOS leaks the same way (2.3 GB for the schema cases alone) and gets through on swap. The per-op tests only read the results, so both are now module fixtures. The same container passes all 13,305 tests in 2m24s with a 761 MB peak, and macOS 3.14 runs the suite, with coverage, in about a minute.
…aceback A leak does not fail a test, it starves the machine, and on a hosted runner the log goes down with it. After every test the session now compares its peak RSS with a 4096 MB budget and, past it, stops with a failure that names the test which crossed the line. A whole run peaks under 1 GB resident on Linux and on macOS. Checked against the unfixed registry tests: the run stopped after 3m42s naming test_schema_generates[message.tone.get], with the coverage report still printed. faulthandler_timeout = 60 does the same for a hang: a test still running after a minute has every thread's traceback written to stderr while the job is alive to upload it. The slowest test takes about three seconds.
…ce on Linux The Ubuntu 3.14 job ran for up to 70 minutes before GitHub gave up on it. timeout-minutes bounds a stuck job at a few times a green run instead of the six-hour default. A timeout does not help once the runner itself is starved, and the per-test budget in tests/conftest.py only sees a test after it ends. So on Linux the pytest step runs under a `ulimit -v` of 8 GiB: a test that allocates without bound gets a MemoryError instead of taking the runner, and the log, with it. The suite peaks at about 2.6 GB of address space, and the new step passed under the cap in an amd64 container with uv's CPython 3.14.7. macOS does not enforce RLIMIT_AS, so the step leaves it alone there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
ubuntu-latest · py3.14has failed on every run since PR-12 (last green: 2b84df2; first red: 5b0a5e3). It is not a hang and not a failing test. The check run carries one annotation, "The hosted runner lost communication with the server", which is GitHub's message for a runner that was killed or starved. That also explains why the pytest step never concludes and the log blob is gone: the runner died before it could upload anything.The cause is memory.
tests/test_registry_contract.pycalledbuild_schema()in everytest_schema_generatescase andlint()in everytest_columns_resolvecase, 678 cases each. Both walk the whole registry andevalevery request's string annotations. On 3.14, coverage's default core issys.monitoring, and coverage 7.16.1'sSysMonitorkeeps every code object it sees alive (coverage/sysmon.py,self.code_objects.append(code)), including each throwaway<string>code object from thoseevals:build_schema()lint()That is about 29 MB per case and roughly 20 GB over the file. PR-12 took the registry from 588 operations to 678, and the cost grows with the square of that. Python 3.13 and older use coverage's C tracer, which does not keep code objects, so they never saw this. macOS 3.14 leaks the same way (2.3 GB for the schema cases alone) but survives on swap.
Reproduced
python:3.14, no coverage: all 13,305 passed in 10m39s, flat at ~330 MB. About 9.5 of those minutes were this one file.test_registry_contract.py.ubuntu:24.04, non-root,uv python install 3.14(CPython 3.14.7),make testexactly as CI runs it: OOM killed at 86% (make: *** Error 137,OOMKilled=true) under a 5 GB cap.The fix
tests/test_registry_contract.py:lint()andbuild_schema()become module-scoped fixtures. The per-op tests only read the results.Failing fast next time
tests/conftest.py: after each test, peak RSS is checked against a 4096 MB budget. Past it, the session stops with a failure that names the test. A normal run peaks under 1 GB on both OSes.peak RSS reached 4099 MB after tests/test_registry_contract.py::TestOperationContract::test_schema_generates[message.tone.get], and the coverage report still printed.pyproject.toml:faulthandler_timeout = 60, so a hung test writes every thread's traceback to the log. The slowest test takes about 3s.ci.yml:timeout-minutes: 15on the test job.ulimit -vof 8 GiB. This backstops a single test that allocates without bound, which the per-test budget only sees after that test ends. The suite peaks at about 2.6 GB of address space. The exact new step passed in the amd64 container (1m45s). macOS does not enforceRLIMIT_AS.continue-on-errorfor 3.14 is unchanged. Whether 3.14 should now become a gate is a separate call.Not in this PR
tlgr/daemon/dispatch.pyre-evaluates a request type's annotations (_peer_ref_fields) on every/v1/opcall. Under the sysmon core that also leaks a little per dispatched test. It is bounded and harmless now, but a cache there would help the daemon too.SysMonitorretaining untraced<string>code objects is an upstream issue.