From 6dc619c380d2225dc73a7dee7f3df00dd6320ef7 Mon Sep 17 00:00:00 2001 From: Josh Sokol Date: Fri, 21 Aug 2026 11:00:01 -0500 Subject: [PATCH 1/3] ci(promote-latest): promote both images at GA and mirror to GHCR master still ran the pre-redesign promote: SSM-only, auto-fired on a master push, and its own header pointed at push-to-dockerhub to build the image. That is what let today's GA write /image-tag/latest = 20260820-001 while the image build for that version had already failed -- production was promoted onto a tag that the workflow believed did not exist. Bring master onto the build-once-promote model: - manual workflow_dispatch only, so GA is a deliberate gate rather than a side effect of a branch merge; - retag DockerHub :latest to the existing RC digest via buildx imagetools create, for simplerisk-minimal (-php85) and simplerisk (-noble). No rebuild, so the bytes validated in testing are the bytes that ship; - a currency guard per image: refuse to promote a version whose digest is not the one :testing currently points at, so a stale committed Dockerfile version cannot push an old-but-existing release to prod; - mirror the promoted digests into GHCR, cosign-signed, so ghcr and dockerhub are finally the same bytes rather than two builds sharing a name. skip_full_image covers the transition: releases cut before the full-stack RC build landed have no simplerisk/simplerisk RC digest to promote. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/promote-latest.yml | 144 +++++++++++++++++++++++---- 1 file changed, 127 insertions(+), 17 deletions(-) diff --git a/.github/workflows/promote-latest.yml b/.github/workflows/promote-latest.yml index 938eceb..750a6e0 100644 --- a/.github/workflows/promote-latest.yml +++ b/.github/workflows/promote-latest.yml @@ -1,27 +1,36 @@ name: Promote latest image tag (release) -# On a release (push to master that bumps the version), promote the customers-cdk -# `latest` channel: write SSM /simplerisk/customers/image-tag/latest = in -# the customers account via OIDC, so the image-updater Lambda rolls tier=latest -# (production) services onto the just-published release image. +# GA promotion, run MANUALLY (workflow_dispatch) once the release has merged to +# master. Build-once model: both release images were already built by +# publish-testing.yml at the testing cut. GA does NOT rebuild anything -- it +# repoints tags and parameters at those existing digests, so the bytes validated +# in testing are byte-identical to the bytes that reach production. # -# The release IMAGE itself is built + pushed (:latest + :) by the existing -# push-to-dockerhub workflow on the same master push — this workflow ONLY does the -# cross-account SSM promote (the missing automation link). Path-filtered to the -# minimal Dockerfile so a docs-only master push does not roll production. +# 1. Docker Hub :latest -> the existing RC digest, for both +# simplerisk/simplerisk-minimal (-php85) and simplerisk/simplerisk +# (-noble), via `buildx imagetools create` (multi-arch preserved). +# 2. GHCR mirror -- copies the same digests to ghcr.io, cosign-signed, so the +# GHCR and Docker Hub images for a version are the same bytes. GHCR used to +# get its own rebuild from the prod bundle, which meant ghcr and +# dockerhub were different images sharing a name. +# 3. SSM /simplerisk/customers/image-tag/latest = -php85 in the +# customers account (OIDC), so the image-updater Lambda rolls tier=latest +# (production) services onto the promoted digest. # -# See design docs/superpowers/specs/2026-07-01-testing-image-promote (customers-cdk). +# See design code-development docs/superpowers/specs/2026-07-10-release-image-promotion-design. on: - push: - branches: [master] - paths: - - simplerisk-minimal/Dockerfile workflow_dispatch: + inputs: + skip_full_image: + description: 'Transitional: skip simplerisk/simplerisk (no RC digest for releases cut before the full-stack RC build landed)' + type: boolean + default: false permissions: contents: read - id-token: write + id-token: write # OIDC: AWS role + cosign/fulcio identity challenge + packages: write # GHCR mirror concurrency: group: promote-latest @@ -30,6 +39,10 @@ concurrency: env: AWS_REGION: us-east-1 SSM_PARAM: /simplerisk/customers/image-tag/latest + MINIMAL_IMAGE: simplerisk/simplerisk-minimal + FULL_IMAGE: simplerisk/simplerisk + GHCR_MINIMAL: ghcr.io/simplerisk/simplerisk-minimal + GHCR_FULL: ghcr.io/simplerisk/simplerisk jobs: promote: @@ -38,6 +51,27 @@ jobs: - name: Checkout uses: actions/checkout@v6 + - name: Install cosign + uses: sigstore/cosign-installer@v3.5.0 + with: + cosign-release: 'v2.4.0' + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v4 + + - name: Log in to Docker Hub + uses: docker/login-action@v4 + with: + username: ${{ secrets.DOCKER_USERNAME }} + password: ${{ secrets.DOCKER_TOKEN }} + + - name: Log in to GHCR + uses: docker/login-action@v4 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - name: Read release version from the minimal Dockerfile id: ver run: | @@ -51,18 +85,94 @@ jobs: fi echo "version=$VERSION" >> "$GITHUB_OUTPUT" + - name: Promote simplerisk-minimal — :latest → -php85 + env: + VERSION: ${{ steps.ver.outputs.version }} + run: | + set -euo pipefail + # The php85 immutable tag MUST already exist (built by publish-testing + # at the RC cut). Fail loudly rather than silently promoting nothing. + if ! docker buildx imagetools inspect "${MINIMAL_IMAGE}:${VERSION}-php85" >/dev/null 2>&1; then + echo "::error::${MINIMAL_IMAGE}:${VERSION}-php85 not found on Docker Hub — was the RC published?"; exit 1 + fi + # Currency guard: only promote the version that is CURRENTLY in testing. + # :testing floats to the current RC (publish-testing tags -php85 and + # :testing on the same build), so the digests match iff VERSION is the + # current RC. Prevents a stale committed Dockerfile version (or a stale + # dispatch ref) from promoting an old-but-existing release to prod :latest. + SRC_DIGEST=$(docker buildx imagetools inspect "${MINIMAL_IMAGE}:${VERSION}-php85" --format '{{.Manifest.Digest}}') + TESTING_DIGEST=$(docker buildx imagetools inspect "${MINIMAL_IMAGE}:testing" --format '{{.Manifest.Digest}}') + if [ "$SRC_DIGEST" != "$TESTING_DIGEST" ]; then + echo "::error::${MINIMAL_IMAGE}:${VERSION}-php85 ($SRC_DIGEST) is not the current testing RC ($TESTING_DIGEST) — refusing to promote a stale version to :latest"; exit 1 + fi + # Retag (no rebuild): create :latest from the existing multi-arch digest. + docker buildx imagetools create \ + --tag "${MINIMAL_IMAGE}:latest" \ + "${MINIMAL_IMAGE}:${VERSION}-php85" + echo "retagged ${MINIMAL_IMAGE}:latest -> ${VERSION}-php85 ($SRC_DIGEST)" >> "$GITHUB_STEP_SUMMARY" + + - name: Promote simplerisk (full-stack) — :latest → -noble + if: ${{ !inputs.skip_full_image }} + env: + VERSION: ${{ steps.ver.outputs.version }} + run: | + set -euo pipefail + if ! docker buildx imagetools inspect "${FULL_IMAGE}:${VERSION}-noble" >/dev/null 2>&1; then + echo "::error::${FULL_IMAGE}:${VERSION}-noble not found on Docker Hub — was the RC published? (re-run with skip_full_image for a release cut before the full-stack RC build landed)"; exit 1 + fi + SRC_DIGEST=$(docker buildx imagetools inspect "${FULL_IMAGE}:${VERSION}-noble" --format '{{.Manifest.Digest}}') + TESTING_DIGEST=$(docker buildx imagetools inspect "${FULL_IMAGE}:testing" --format '{{.Manifest.Digest}}') + if [ "$SRC_DIGEST" != "$TESTING_DIGEST" ]; then + echo "::error::${FULL_IMAGE}:${VERSION}-noble ($SRC_DIGEST) is not the current testing RC ($TESTING_DIGEST) — refusing to promote a stale version to :latest"; exit 1 + fi + docker buildx imagetools create \ + --tag "${FULL_IMAGE}:latest" \ + "${FULL_IMAGE}:${VERSION}-noble" + echo "retagged ${FULL_IMAGE}:latest -> ${VERSION}-noble ($SRC_DIGEST)" >> "$GITHUB_STEP_SUMMARY" + + - name: Mirror the promoted digests to GHCR (cosign-signed) + env: + VERSION: ${{ steps.ver.outputs.version }} + SKIP_FULL: ${{ inputs.skip_full_image }} + run: | + set -euo pipefail + # imagetools create copies the manifest (and blobs) across registries, + # so GHCR receives the identical digest rather than a rebuild. Each + # source digest is mirrored once, carrying every tag that points at it. + # cosign signs the digest (not the tag), so one signature per call. + mirror() { + local src="$1" dst="$2"; shift 2 + local args=() t digest + for t in "$@"; do args+=(--tag "${dst}:${t}"); done + docker buildx imagetools create "${args[@]}" "$src" + digest=$(docker buildx imagetools inspect "${dst}:${1}" --format '{{.Manifest.Digest}}') + cosign sign --yes "${dst}@${digest}" + echo "mirrored $src -> ${dst} [$*] ($digest)" >> "$GITHUB_STEP_SUMMARY" + } + + mirror "${MINIMAL_IMAGE}:${VERSION}-php83" "${GHCR_MINIMAL}" "${VERSION}-php83" + mirror "${MINIMAL_IMAGE}:${VERSION}-php84" "${GHCR_MINIMAL}" "${VERSION}-php84" + mirror "${MINIMAL_IMAGE}:${VERSION}-php85" "${GHCR_MINIMAL}" "${VERSION}-php85" "${VERSION}" "latest" + + if [ "$SKIP_FULL" != "true" ]; then + mirror "${FULL_IMAGE}:${VERSION}-jammy" "${GHCR_FULL}" "${VERSION}-jammy" + mirror "${FULL_IMAGE}:${VERSION}-noble" "${GHCR_FULL}" "${VERSION}-noble" "${VERSION}" "latest" + else + echo "skip_full_image set — ${GHCR_FULL} not mirrored for ${VERSION}" >> "$GITHUB_STEP_SUMMARY" + fi + - name: Configure AWS credentials (OIDC → customers account) uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ vars.IMAGE_PROMOTER_LATEST_ROLE_ARN }} aws-region: ${{ env.AWS_REGION }} - - name: Promote — SSM /image-tag/latest = + - name: Promote — SSM /image-tag/latest = -php85 env: VERSION: ${{ steps.ver.outputs.version }} run: | set -euo pipefail aws ssm put-parameter --name "$SSM_PARAM" \ - --value "$VERSION" --type String --overwrite \ + --value "${VERSION}-php85" --type String --overwrite \ --region "$AWS_REGION" - echo "promoted $SSM_PARAM = $VERSION" >> "$GITHUB_STEP_SUMMARY" + echo "promoted $SSM_PARAM = ${VERSION}-php85" >> "$GITHUB_STEP_SUMMARY" From 9d150a9718edb853d7fff1bb2e4b0230b4b373b1 Mon Sep 17 00:00:00 2001 From: Josh Sokol Date: Fri, 21 Aug 2026 11:00:33 -0500 Subject: [PATCH 2/3] ci(push-*): stop rebuilding release images on a master push These are the workflows that failed the 20260820-001 GA. They rebuild from the prod S3 bundle on a master push, but the release PR merges ~17 minutes before the bundle propagation uploads it, so download_and_verify_bundle.sh fail-closed on a bare 403 (S3 returns 403, not 404, for a missing object under an anonymous-list-denied bucket -- it reads as a permissions error). Rebuilding at GA is also what the promote model exists to remove: it shipped bytes to production that were never the bytes validated in testing, and gave GHCR a separate build under the same version tag. Drop the `push: master` trigger and the simplerisk-minimal jobs from both. promote-latest.yml now owns GA for both registries. The jammy/noble jobs stay dispatchable as a transitional escape hatch for releases with no full-stack RC digest; both headers say when they can be deleted. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/push-to-dockerhub.yml | 45 +++++++++---------------- .github/workflows/push-to-gh-pkgs.yml | 39 +++++++-------------- 2 files changed, 28 insertions(+), 56 deletions(-) diff --git a/.github/workflows/push-to-dockerhub.yml b/.github/workflows/push-to-dockerhub.yml index 32cf5f1..a0dd924 100644 --- a/.github/workflows/push-to-dockerhub.yml +++ b/.github/workflows/push-to-dockerhub.yml @@ -1,11 +1,23 @@ name: Push images to DockerHub +# LEGACY REBUILD -- manual dispatch only, retained as a transitional escape hatch. +# +# Release images are no longer built here. Under the build-once-promote model +# (code-development docs/superpowers/specs/2026-07-10-release-image-promotion-design) +# every release image is built ONCE as the RC by publish-testing.yml on the +# `testing` branch, and promote-latest.yml retags that digest at GA. Rebuilding +# from the prod bundle on a master push produced GA bytes that were never the +# tested bytes, and raced the GA bundle upload: the release PR merges here +# minutes before the bundle lands in S3, so the build failed fail-closed with a +# bare 403. +# +# The `push: master` trigger is therefore gone. These jobs remain dispatchable +# so a release cut BEFORE the full-stack RC build landed can still be produced +# by hand. Delete this workflow once the first post-cutover RC has published +# simplerisk/simplerisk -jammy/-noble digests. + on: workflow_dispatch: - push: - branches: [ "master" ] - # Publish semver tags as releases. - #tags: [ '[2022]0701-001' ] # On a job that uses a reusable workflow, it seems you cannot # use env on a with block (https://github.com/actions/runner/issues/1189#issuecomment-1741672276) @@ -36,28 +48,3 @@ jobs: main_image: true build_args: "ubuntu_version_code=noble" secrets: inherit - simplerisk-minimal-php84: - name: 'Push simplerisk/simplerisk-minimal image based on PHP 8.3 with Apache' - uses: ./.github/workflows/push-to-dockerhub_rw.yml - with: - context_path: "simplerisk-minimal" - dockerfile_path: "simplerisk-minimal/Dockerfile" - image_name: "simplerisk/simplerisk-minimal" - version: "20260820-001" - os_version: "php83" - build_args: "php_version=8.3" - platforms: linux/amd64,linux/arm64 - secrets: inherit - simplerisk-minimal-php85: - name: 'Push simplerisk/simplerisk-minimal image based on PHP 8.4 with Apache' - uses: ./.github/workflows/push-to-dockerhub_rw.yml - with: - context_path: "simplerisk-minimal" - dockerfile_path: "simplerisk-minimal/Dockerfile" - image_name: "simplerisk/simplerisk-minimal" - version: "20260820-001" - os_version: "php84" - main_image: true - build_args: "php_version=8.4" - platforms: linux/amd64,linux/arm64 - secrets: inherit diff --git a/.github/workflows/push-to-gh-pkgs.yml b/.github/workflows/push-to-gh-pkgs.yml index ceebbb0..8152167 100644 --- a/.github/workflows/push-to-gh-pkgs.yml +++ b/.github/workflows/push-to-gh-pkgs.yml @@ -1,11 +1,19 @@ name: Push images to GitHub Packages +# LEGACY REBUILD -- manual dispatch only, retained as a transitional escape hatch. +# +# GHCR release images are no longer built here. promote-latest.yml mirrors the +# promoted Docker Hub digests into ghcr.io (cosign-signed) at GA, so the GHCR +# and Docker Hub images for a version are the same bytes. Rebuilding here meant +# ghcr and dockerhub were different images sharing a name, +# and the rebuild raced the GA bundle upload the same way the Docker Hub one did. +# +# See code-development docs/superpowers/specs/2026-07-10-release-image-promotion-design. +# Delete this workflow once the first post-cutover RC has published +# simplerisk/simplerisk -jammy/-noble digests. + on: workflow_dispatch: - push: - branches: [ "master" ] - # Publish semver tags as releases. - #tags: [ '[2022]0701-001' ] # On a job that uses a reusable workflow, it seems you cannot # use env on a with block (https://github.com/actions/runner/issues/1189#issuecomment-1741672276) @@ -36,26 +44,3 @@ jobs: main_image: true build_args: "ubuntu_version_code=noble" secrets: inherit - simplerisk-minimal-php84: - name: 'Push simplerisk/simplerisk-minimal image based on PHP 8.3 with Apache' - uses: ./.github/workflows/push-to-gh-pkgs_rw.yml - with: - context_path: "simplerisk-minimal" - dockerfile_path: "simplerisk-minimal/Dockerfile" - image_name: "simplerisk-minimal" - version: "20260820-001" - os_version: "php83" - build_args: "php_version=8.3" - secrets: inherit - simplerisk-minimal-php85: - name: 'Push simplerisk/simplerisk-minimal image based on PHP 8.4 with Apache' - uses: ./.github/workflows/push-to-gh-pkgs_rw.yml - with: - context_path: "simplerisk-minimal" - dockerfile_path: "simplerisk-minimal/Dockerfile" - image_name: "simplerisk-minimal" - version: "20260820-001" - os_version: "php84" - main_image: true - build_args: "php_version=8.4" - secrets: inherit From 46399973ccb3e6d7c8edda686ca9b58411ea21a3 Mon Sep 17 00:00:00 2001 From: Josh Sokol Date: Fri, 21 Aug 2026 11:00:33 -0500 Subject: [PATCH 3/3] docs(claude): describe the build-once-promote release path The CI/CD section still said pushes publish to Docker Hub and GHCR, which is now wrong in both halves: nothing publishes on a master push, and GHCR is a mirror of the promoted digest rather than its own build. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 8b74d26..25a04c4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -117,7 +117,9 @@ The entrypoint script handles: ### CI/CD - **PRs** trigger `container-validation.yml`: builds all 4 variants (jammy, noble, php81, php83), runs Dockle (Dockerfile linter) and Grype (CVE scanner, severity cutoff: critical, only-fixed). -- **Pushes** trigger separate workflows to publish to Docker Hub and GitHub Container Registry (GHCR). GHCR images are signed with Cosign/sigstore. The `simplerisk-minimal` push builds target both `linux/amd64` and `linux/arm64`. +- **Release images are built once, then promoted — never rebuilt.** A push to `testing` runs `publish-testing.yml`, which builds both images from the current testing bundle and publishes immutable tags: `simplerisk-minimal` gets `-php83/-php84/-php85` (multi-arch `linux/amd64,linux/arm64`) and `simplerisk` gets `-jammy/-noble` (amd64). Each image's default variant also takes the bare `` and the floating `:testing`. +- **GA is a manual promote, not a build.** After the release merges to `master`, dispatch `promote-latest.yml`. It retags Docker Hub `:latest` to the existing RC digest (`buildx imagetools create`, multi-arch preserved), mirrors the same digests to GHCR cosign-signed, and writes SSM `/simplerisk/customers/image-tag/latest`. Nothing is rebuilt, so the bytes validated in testing are the bytes that ship. A currency guard refuses to promote a version whose digest is not the one `:testing` currently points at. +- `push-to-dockerhub.yml` / `push-to-gh-pkgs.yml` are **legacy rebuild workflows, manual dispatch only** — they no longer run on a `master` push. See their headers; they are deletable once the first post-cutover RC has published full-stack RC digests. - The reusable workflow files (`*_rw.yml`) are called by the entry-point workflows. ### Vulnerability ignore list