From c7eb64a5d8d8343852a29f5afa961cdc30ddbd83 Mon Sep 17 00:00:00 2001 From: David Young Date: Thu, 27 Aug 2026 18:50:15 +1200 Subject: [PATCH] ci(release-schedule): trigger from the cluster, not GitHub's cron GitHub deprioritises scheduled workflows and silently drops ticks: over one 17-hour sample this workflow's own "30 * * * *" cron fired 12 times instead of 17, with gaps up to 2h18m. harbor-themes-v0.11.5 was tagged upstream at 05:19 and the matrix had last run at 04:14, so the image sat unbuilt with nothing reporting a failure. Replaced by a k8s CronJob in infra (cronjobs/dispatch-containers-release--flux-system) that fires workflow_dispatch hourly at :30, matching the cadence this file used to declare. Same pattern as elfhosted/myprecious' release-please workflow. Also adds workflow-level concurrency. A full matrix build routinely outlives the dispatch interval, and because fetch.sh compares against PUBLISHED tags an overlapping run re-derives the same not-yet-pushed changes, so both runs race to push the same tag at different digests. Queueing was previously masked by the dropped ticks; with a reliable trigger it would not be. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/release-schedule.yaml | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/.github/workflows/release-schedule.yaml b/.github/workflows/release-schedule.yaml index c875dab821..a3e6c1606c 100644 --- a/.github/workflows/release-schedule.yaml +++ b/.github/workflows/release-schedule.yaml @@ -2,10 +2,26 @@ name: "Release: Schedule" on: + # Triggered externally via workflow_dispatch by a k8s CronJob (see infra: + # cronjobs/dispatch-containers-release--flux-system) hourly at :30 UTC. + # + # There is deliberately no `schedule:` block here. GitHub deprioritises + # scheduled workflows and silently drops ticks: over one 17-hour sample this + # workflow's own "30 * * * *" cron fired 12 times instead of 17, with gaps up + # to 2h18m. Every dropped tick delays every app whose upstream cut a release + # in that window, and the delay is invisible — nothing fails, the build just + # does not happen. workflow_dispatch: - schedule: - # Run on the half-hour - - cron: "30 * * * *" + +# A full matrix build routinely outlives the hourly dispatch interval. Because +# fetch.sh compares against PUBLISHED tags, an overlapping run re-derives the +# same not-yet-pushed changes and both runs race to push the same tag at +# different digests — churning renovate PRs and tenant clusters. Queue instead: +# cancel-in-progress: false lets the running build finish, and GitHub keeps at +# most one pending run, whose matrix is generated fresh when it starts. +concurrency: + group: release-schedule + cancel-in-progress: false env: TOKEN: ${{ secrets.GITHUB_TOKEN }}