Skip to content

Cut GA over to promoting the RC digest instead of rebuilding - #160

Merged
jsokol merged 3 commits into
masterfrom
FEATURE-ga-promote-cutover
Aug 21, 2026
Merged

Cut GA over to promoting the RC digest instead of rebuilding#160
jsokol merged 3 commits into
masterfrom
FEATURE-ga-promote-cutover

Conversation

@jsokol

@jsokol jsokol commented Aug 21, 2026

Copy link
Copy Markdown
Member

Description

Cuts master over to the build-once-promote model from docs/superpowers/specs/2026-07-10-release-image-promotion-design.md (code-development). The design landed on the testing branch in July (#147, #148) but was never brought to master, so GA still ran the pre-redesign path. That is what failed the 20260820-001 release.

What went wrong today. Merging the release PR (#157) at 14:10 fired push-to-dockerhub.yml and push-to-gh-pkgs.yml, which rebuild every image from the prod S3 bundle. The GA bundle was not uploaded until 14:27:37 — 17 minutes later — so download_and_verify_bundle.sh correctly fail-closed. The error reads as a permissions problem but is not: S3 returns 403, not 404, for a missing object under an anonymous-list-denied bucket.

curl: (22) The requested URL returned error: 403
ERROR: prod bundle simplerisk-20260820-001.tgz not found and PREGA_BUNDLE_FALLBACK != true

Meanwhile promote-latest.yml succeeded on the same push and wrote SSM /image-tag/latest = 20260820-001. It got away with it only because the RC build had already published that bare tag overnight — under master's own stated model that tag was supposed to come from the rebuild that had just failed. Production was promoted onto a tag the workflow believed did not exist. Docker Hub :latest, meanwhile, still points at 20260519-001.

Changes

promote-latest.yml — GA becomes a promote, and owns both registries.

  • workflow_dispatch only. GA is a deliberate gate, not a side effect of a branch merge.
  • Retags Docker Hub :latest to the existing RC digest via buildx imagetools create (multi-arch preserved) for both simplerisk-minimal (<V>-php85) and simplerisk (<V>-noble). No rebuild, so the bytes validated in testing are the bytes that ship.
  • A currency guard per image: refuses to promote a version whose digest is not the one :testing currently points at, so a stale committed Dockerfile version cannot push an old-but-existing release to prod.
  • Mirrors the promoted digests into GHCR, cosign-signed. Previously GHCR got its own rebuild from the prod bundle, so ghcr <V> and dockerhub <V> were two different images sharing a name.
  • skip_full_image input covers the transition — see below.

push-to-dockerhub.yml / push-to-gh-pkgs.yml — no longer publish releases.
push: master trigger removed; simplerisk-minimal jobs removed. The jammy/noble jobs stay dispatchable as a transitional escape hatch, and both headers say when the workflows can be deleted.

CLAUDE.md — the CI/CD section claimed pushes publish to both registries, wrong in both halves now.

Ordering

Important

Depends on #159 (base testing). The full-stack image has no RC build today, so simplerisk/simplerisk:<V>-jammy/-noble does not exist for any version yet. #159 adds it.

Sequence:

  1. Merge Build the full-stack image once at RC time, so GA can promote it #159 into testing.
  2. Merge this PR into master.
  3. For 20260820-001 only, dispatch promote-latest with skip_full_image: true — this release predates the full-stack RC build. It promotes simplerisk-minimal (whose RC tags exist, published 03:10–04:23 today), mirrors them to GHCR, and moves :latest off the stale 20260519-001. If a full-stack image is wanted for this release, dispatch push-to-dockerhub manually — the bundle is present and its sha256 now matches the feed, so it will succeed.
  4. From the next RC cut onward, skip_full_image is never needed and the two legacy workflows can be deleted.

Release Notes

No customer-facing change. Container release plumbing only. Going forward the published latest image is byte-identical to the release candidate validated in testing, and the GHCR and Docker Hub images for a given version are the same bytes.

Manual Validation Steps

  1. Confirm no workflow publishes on a master push: push-to-dockerhub.yml and push-to-gh-pkgs.yml list workflow_dispatch only; promote-latest.yml likewise.
  2. Merge a no-op commit to master → no image workflow should start.
  3. Dispatch promote-latest with skip_full_image: true. Expect: minimal RC digest verified, currency guard passes against :testing, Docker Hub :latest retagged, GHCR mirrored + signed, SSM written. Step summary records each digest.
  4. docker buildx imagetools inspect simplerisk/simplerisk-minimal:latest --format '{{.Manifest.Digest}}' should equal the <V>-php85 digest, and the GHCR copy should match.
  5. Negative test: dispatch against a version that is not the current RC → the currency guard should refuse before anything is retagged.

Type of Change

  • Build / CI configuration
  • Bug fix (the GA release path is currently broken)

Testing Results

  • actionlint clean on all three changed workflows.
  • shellcheck -S warning clean on every embedded run block in promote-latest.yml.
  • YAML parses; triggers and job lists verified programmatically.
  • update_workflows.sh's version-pin regex still matches the remaining pins (2 per push workflow, was 4) — make update_version is unaffected.
  • Not exercised: the promote itself. It needs Docker Hub + GHCR + AWS OIDC credentials and mutates published tags, so it cannot be dry-run from a PR. Step 3 above is the first real execution, and it is reversible (retagging :latest back to the previous digest is one imagetools create).

CIA Impact Check

MAY negatively impact Confidentiality, Integrity and/or Availability.

Justification for the change

  • The GA image build is currently broken — two workflows failed on the 20260820-001 release and no release image was published.
  • The GA image was never the tested image. Rebuilding from a separate prod bundle meant the bytes shipped to production were never the bytes validated in testing.
  • ghcr <V> and dockerhub <V> were independent builds under the same version tag — a real supply-chain ambiguity about what a version identifies.
  • SSM promotion could succeed while the image it pointed at had failed to build, which is exactly what happened today.

Potential impact on Confidentiality, Integrity and/or Availability

Integrity (net positive, with a new dependency). A single digest now flows testing → prod unchanged, and GHCR is a verified mirror rather than a parallel build. The new dependency is that GA correctness now rests on the RC having been published and on the currency guard; both fail closed and loudly. The cosign signature on the GHCR mirror is generated at promote time over the mirrored digest, preserving the existing signing property.

Availability (deploy-window, one-time). GA promotion becomes a manual dispatch, so a release is not published until someone runs it. This is deliberate — the design calls it a release gate — but it is a behavior change for whoever runs releases, and a forgotten dispatch means :latest silently stays on the previous release. Mitigated by the step summary and by :latest remaining valid (just older) until promoted.

Confidentiality. No change. No new secrets; packages: write and cosign's id-token: write are the same scopes the existing GHCR publish path already used.

Additional Implementation Steps

  • No schema or customer-side action.
  • Release runbooks that assume images publish automatically on the master merge need updating to add the promote-latest dispatch step.
  • The IMAGE_PROMOTER_LATEST_ROLE_ARN repo variable is already in use by the existing promote job — no new configuration.
  • After the first post-cutover RC, delete push-to-dockerhub.yml and push-to-gh-pkgs.yml.

jsokol and others added 3 commits August 21, 2026 11:00
master still ran the pre-redesign promote: SSM-only, auto-fired on a master
push, and its own header pointed at push-to-dockerhub to build the image. That
is what let today's GA write /image-tag/latest = 20260820-001 while the image
build for that version had already failed -- production was promoted onto a tag
that the workflow believed did not exist.

Bring master onto the build-once-promote model:

- manual workflow_dispatch only, so GA is a deliberate gate rather than a side
  effect of a branch merge;
- retag DockerHub :latest to the existing RC digest via buildx imagetools
  create, for simplerisk-minimal (<V>-php85) and simplerisk (<V>-noble). No
  rebuild, so the bytes validated in testing are the bytes that ship;
- a currency guard per image: refuse to promote a version whose digest is not
  the one :testing currently points at, so a stale committed Dockerfile version
  cannot push an old-but-existing release to prod;
- mirror the promoted digests into GHCR, cosign-signed, so ghcr <V> and
  dockerhub <V> are finally the same bytes rather than two builds sharing a name.

skip_full_image covers the transition: releases cut before the full-stack RC
build landed have no simplerisk/simplerisk RC digest to promote.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These are the workflows that failed the 20260820-001 GA. They rebuild from the
prod S3 bundle on a master push, but the release PR merges ~17 minutes before
the bundle propagation uploads it, so download_and_verify_bundle.sh fail-closed
on a bare 403 (S3 returns 403, not 404, for a missing object under an
anonymous-list-denied bucket -- it reads as a permissions error).

Rebuilding at GA is also what the promote model exists to remove: it shipped
bytes to production that were never the bytes validated in testing, and gave
GHCR a separate build under the same version tag.

Drop the `push: master` trigger and the simplerisk-minimal jobs from both.
promote-latest.yml now owns GA for both registries. The jammy/noble jobs stay
dispatchable as a transitional escape hatch for releases with no full-stack RC
digest; both headers say when they can be deleted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CI/CD section still said pushes publish to Docker Hub and GHCR, which is
now wrong in both halves: nothing publishes on a master push, and GHCR is a
mirror of the promoted digest rather than its own build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jsokol
jsokol marked this pull request as ready for review August 21, 2026 16:10
@jsokol
jsokol merged commit 9e6ff9a into master Aug 21, 2026
5 checks passed
@jsokol
jsokol deleted the FEATURE-ga-promote-cutover branch August 21, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant