From 11b484a0489520cd574eb8a9c38476cf9a17297a Mon Sep 17 00:00:00 2001 From: Eric Moore Date: Tue, 29 Sep 2026 11:42:10 -0500 Subject: [PATCH] fix(ci): bound the iOS leg's heaps and make the watchdog's CPU sum an integer The v0.5.225 ios-xcframework leg died in linkReleaseFrameworkIosArm64 with `GC overhead limit exceeded` after 37 min on macos-14 (3-core M1, 7 GB). client/gradle.properties gives Gradle -Xmx8g, and KGP 2.0.21 runs the Kotlin/Native release link INSIDE that JVM by default (KotlinNativeToolRunner.runInProcess; kotlin.native.disableCompilerDaemon flips it to javaexec). 0.5.224 had already taken 3h11m against ~161 min measured, which is what paging looks like; two review batches later it stopped fitting at all. The bound lives on the leg's matrix entry, not in gradle.properties, which is every desktop's setting and an input to the XCFramework cache key: --max-workers=1 one link at a time -Dorg.gradle.jvmargs=-Xmx2g Gradle only configures and waits -Pkotlin.native.disableCompilerDaemon=true the link gets its own JVM per task -Pkotlin.native.jvmArgs=-Xmx4g KGP's default for it is 3g 4 GB is a budget for 7 GB (~1.3 OS/agent, <=2 Gradle, 4 link), not a measurement; the comment says so, and says why -XX:-UseGCOverheadLimit is not an answer. build_xcframework_local.sh is the desktop path and keeps the desktop heap. Building locally before the tag is still the advice. The watchdog on the same leg printed `line 108: 438.74: syntax error` on its first heartbeat and then nothing: BSD ps prints CPU time with hundredths, bash arithmetic rejects the sum, and an expansion error ends the subshell, so the leg ran its last 33 minutes with no heartbeat and no hang detection. cpu_seconds now drops the fraction per process (and handles the GNU day prefix as days, not as another base-60 field) and prints with %d. A --self-test feeds a fake BSD ps, including that exact 438.74, through the same $(( cpu - last_cpu )); testing/test_publish_ios_leg.py runs it, proves it goes red with the fraction left in, keeps the leg's two heaps summing under the runner, and is wired into build.yml's pytest step. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0155SkTGdbwnSnR6tWBqJvUM --- .github/workflows/build.yml | 3 +- .github/workflows/publish.yml | 61 +++++++++++- AGENTS.md | 1 + packaging/run_with_watchdog.sh | 75 +++++++++++++-- testing/test_publish_ios_leg.py | 164 ++++++++++++++++++++++++++++++++ 5 files changed, 292 insertions(+), 12 deletions(-) create mode 100644 testing/test_publish_ios_leg.py diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index ab085428..e6817145 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -111,7 +111,8 @@ jobs: python3 -m pip install --quiet pytest pyyaml python3 -m pytest testing/test_driver_rules.py testing/test_gate_vendoring.py \ testing/test_bringup.py testing/test_five_platform_workflow.py \ - testing/test_flows.py testing/test_csd_state_tags.py -q + testing/test_flows.py testing/test_csd_state_tags.py \ + testing/test_publish_ios_leg.py -q - name: A published version must offer a universal wheel # Only for versions already on the index — a PR's VERSION is normally diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 9c4d1f60..9e97d88f 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -472,6 +472,9 @@ jobs: # slow step: even the long Kotlin/Native links emit task lines. idle_seconds: 600 cache_read_only: true + # ubuntu-latest has 16 GB; gradle.properties' -Xmx8g fits, and the + # AAR is not a Kotlin/Native link. Nothing to bound here. + gradle_args: "" - name: ios-xcframework # Kotlin/Native cannot cross-compile Apple targets, and release # linking is the expensive part: MEASURED at >85 minutes for @@ -495,6 +498,59 @@ jobs: # minutes with thread stacks instead of at 350 with "cancelled". idle_seconds: 1500 cache_read_only: false + # THE LINK GETS ITS OWN JVM, AND BOTH JVMS FIT THE RUNNER. + # + # `client/gradle.properties` says `org.gradle.jvmargs=-Xmx8g`. That + # is right for a desktop and larger than this runner: macos-14 is + # a 3-core M1 with 7 GB. And in Kotlin Gradle plugin 2.0.21 the + # Kotlin/Native compiler runs INSIDE the Gradle JVM by default + # (KotlinNativeToolRunner.runInProcess — `kotlin.native. + # disableCompilerDaemon` is the property that flips it to + # `javaexec`), so the whole-program release link — the LTO call + # graph, which is where it died — lived in a heap the machine + # could not back. 0.5.224 passed in 3h11m against ~161 min + # measured, which is what paging looks like; 0.5.225, two review + # batches larger, hit `GC overhead limit exceeded` at 37 min. + # + # --max-workers=1 + # One task at a time, so the peak is ONE link, never two. + # org.gradle.parallel is already false; this makes the worker + # pool say the same thing instead of relying on it. + # -Dorg.gradle.jvmargs=-Xmx2g + # Gradle only configures and waits once the link is out of + # process. Configuring AGP + Compose here is ~1 GB; 2 GB is a + # cap with room, not a commitment. A `-D` on the command line + # outranks gradle.properties, so the desktop setting — and + # the XCFramework cache key, which hashes gradle.properties — + # stay untouched. The property's `-XX:+UseParallelGC` goes + # with it; Gradle's own GC choice is irrelevant to a JVM that + # is idle for 160 minutes. + # -Pkotlin.native.disableCompilerDaemon=true + # The link runs in its own JVM, one per link task, which + # exits when the task does. Two links, two JVMs, never at the + # same time — that is the "one link per invocation" without + # splitting the Gradle invocation the watchdog wraps. + # -Pkotlin.native.jvmArgs=-Xmx4g + # That JVM's heap. KGP's own default for it is + # `-Xmx3g -XX:TieredStopAtLevel=1`. The budget on 7 GB: + # ~1.3 GB macOS + runner agent, ≤2 GB Gradle, 4 GB link. The + # C1-only flag is not carried over — it trades peak + # throughput for startup, and this JVM runs for over an hour. + # + # 4 GB IS A BUDGET, NOT A MEASUREMENT. Nothing here has measured + # the link's live set; what is measured is that 8 GB on a 7 GB box + # fails. If 4 GB is short the leg fails in tens of minutes with + # the same OOM from a process whose heap is known, which is the + # right failure — and the answer is still the one above this + # matrix: build it locally and upload it before the tag. + # `-XX:-UseGCOverheadLimit` is not an option here: it only turns + # off the 98%-time-in-GC circuit breaker, and the same JVM then + # crawls to a hard OutOfMemoryError later instead. + gradle_args: >- + --max-workers=1 + -Dorg.gradle.jvmargs=-Xmx2g + -Pkotlin.native.disableCompilerDaemon=true + -Pkotlin.native.jvmArgs=-Xmx4g steps: - uses: actions/checkout@v4 - uses: actions/setup-java@v4 @@ -632,10 +688,13 @@ jobs: # it. if: matrix.name != 'ios-xcframework' || (steps.xcf-cache.outputs.cache-hit != 'true' && steps.prebuilt.outputs.hit != 'true') working-directory: client + # `gradle_args` is unquoted on purpose: it is a space-separated list + # of single tokens (see the matrix), and quoting it would hand Gradle + # one argument it does not recognise. run: | ../packaging/run_with_watchdog.sh \ "${{ matrix.idle_seconds }}" 300 "${{ matrix.name }}" \ - -- ./gradlew --no-daemon ${{ matrix.task }} + -- ./gradlew --no-daemon ${{ matrix.gradle_args }} ${{ matrix.task }} - name: Restored, not rebuilt # SAY WHICH STORE IT CAME FROM. The two have different trust stories — diff --git a/AGENTS.md b/AGENTS.md index 2597cf3b..93cd4930 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -66,5 +66,6 @@ - Every `apt-get` goes through `.github/actions/apt`: `azure.archive.ubuntu.com` dropped, `timeout 300`, `Acquire::Retries=3`. No exceptions — an unhardened `apt-get update` is a coin flip that costs a whole job when it loses. - `gradle` is REQUIRED. It shipped `continue-on-error` for one run to answer whether the vendored tree stands alone; it does, so the flag came off in the same PR. If you ever add an advisory job, write the condition for removing it next to the flag. - **Build the iOS XCFramework locally and upload it BEFORE pushing the tag** — `packaging/build_xcframework_local.sh`. The `ios-xcframework` leg is ~161 minutes on `macos-14` (a 3-core M1 with 7 GB) and it is the entire wall clock of a publish; the same two release links take well under an hour on an Apple-silicon desktop. `packaging/xcframework_key.py` computes the key on both sides and its input list MIRRORS the `hashFiles(...)` in publish.yml's `XCFramework cache` step — change one, change the other in the same commit. `VERSION` is in that key deliberately (`generateBuildFlavor` compiles `CLIENT_VERSION` into `commonMain`), which is why this is per-release and why running it after the tag is a race, not a shortcut. +- The iOS leg's heaps are BOUNDED, on the leg and not in `client/gradle.properties`: `-Dorg.gradle.jvmargs=-Xmx2g` for Gradle and `-Pkotlin.native.disableCompilerDaemon=true -Pkotlin.native.jvmArgs=-Xmx4g` for the link, because KGP 2.0.21 runs the release link INSIDE the Gradle JVM by default and the property's `-Xmx8g` is larger than the runner (0.5.225: `GC overhead limit exceeded` at 37 min). The numbers are a budget for 7 GB, not a measurement — see the matrix comment in publish.yml before changing either. This does not replace building locally; it makes the fallback fail honestly. - Do not reach for `gh run cancel` to stop a slow iOS leg: GitHub has no per-job cancel, so it takes the wheels and `release-assets` with it. The leg declines the work itself or not at all. - The wheels job stages a placeholder when Gradle produced nothing, and the placeholder RAISES on every artifact lookup. Never make it return a path instead; a wheel that installs and silently contains no client is the failure mode this whole arrangement exists to prevent. diff --git a/packaging/run_with_watchdog.sh b/packaging/run_with_watchdog.sh index 2897042b..93df75ea 100755 --- a/packaging/run_with_watchdog.sh +++ b/packaging/run_with_watchdog.sh @@ -2,6 +2,7 @@ # Run a long build so that STALLED and SLOW stop looking the same. # # packaging/run_with_watchdog.sh