Skip to content

Fix the release publish path: GHCR for snapshots, Docker Hub for releases - #220

Merged
wu-sheng merged 1 commit into
masterfrom
fix-release-fetch-from-downloads
Aug 31, 2026
Merged

Fix the release publish path: GHCR for snapshots, Docker Hub for releases#220
wu-sheng merged 1 commit into
masterfrom
fix-release-fetch-from-downloads

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Aug 31, 2026

Copy link
Copy Markdown
Member

Publishing 0.11.0 failed on a job that should never have run, could not be resumed afterwards,
and would have failed again at the chart push even if it had got that far.

Four problems, all on the release path — which had never executed successfully before, so none
of them had ever been exercised.

1. GHCR for commits, Docker Hub for releases

The per-component images and the GHCR chart were published on both paths, so cutting a
release also built and pushed ghcr.io/apache/skywalking-swck/operator:0.11.0 — artifacts nobody
asked for, from a job whose failure then blocked the release that did not need it.

The jobs are named for the path they serve now. The old names said what a job did rather than
when it runs, which is how they came to run on the wrong one unnoticed:

job display name runs on
publish-snapshot-images Publish snapshot ⟨component⟩ image snapshot
publish-snapshot-chart Publish snapshot Helm chart to GHCR snapshot
publish-release-image Publish release image to Docker Hub release
publish-release-chart Publish release Helm chart to Docker Hub release

2. The chart push would have 401'd

It logged in to Docker Hub as registry-1.docker.io and pushed there. Both names resolve to the
same registry and either can be pulled from anonymously, which is exactly why this looked
correct:

oci://docker.io/bitnamicharts/nginx            -> nginx-18.1.0.tgz
oci://registry-1.docker.io/bitnamicharts/nginx -> nginx-18.1.0.tgz

But the credential is stored under the host you logged in as, while a push normalises the
reference back to Docker Hub's canonical host — so it would have looked up a credential that was
never written there. apache/skywalking sets DOCKER_REGISTRY=docker.io and has always
published this way.

The comment justifying the old host claimed docker.io does not serve the /v2/ API that
helm's OCI client talks to. That is untrue — helm pulls from oci://docker.io perfectly well,
as the docs' own install command does. All 11 occurrences across the workflow, docs, chart
README and NOTES.txt now use docker.io, so the hostname a user copies is the one we publish
to.

3. Both waits were pointed at something the build does not read

The image build fetched the tarball from archive.apache.org — a copy that holds every release
ever made but takes hours to receive a new one. A just-voted release could not be built until
the archive caught up, and release-passed.sh and the publish workflow each sat in a long poll
waiting for it: an hour and thirty minutes respectively.

It reads dist.apache.org/repos/dist/release, where the svn move puts the release and where it
is servable immediately. Measured against the live 0.11.0:

$ curl -sS -o /tmp/t.tgz -w '%{http_code} %{size_download}\n' \
    https://dist.apache.org/repos/dist/release/skywalking/swck/0.11.0/skywalking-swck-0.11.0-bin.tgz
200 61113165

Both waits check that and nothing else now — no mirror, no archive, no fallback chain.
downloads.apache.org survives only in the release notes, because users should be sent to the
mirror even though the build should not.

4. release-passed.sh could not be resumed

Publishing 0.11.0 moved the artifacts, removed the previous release, then died in that hour-long
wait. Re-running found dist/dev empty and refused to start, leaving every remaining step to be
done by hand.

The version now falls back to the newest entry in dist/release, the existence check accepts
either location, and move_to_release checks whether it has already happened before asking
again whether the vote passed. Tested against the exact state 0.11.0 left behind: steps 2 and 3
skip cleanly.

Also: the release notes linked to a changelog that would not be there

They linked to docs/en/changes/changes.md. That link is broken by construction: by the time
anyone follows it, the next-version PR has moved the section into changes-<version>.md and
left an empty template behind — so it lands on the next version's empty page. The notes now
carry the changelog itself, read from the tag, which is what the PMC voted on.

@wu-sheng
wu-sheng force-pushed the fix-release-fetch-from-downloads branch from 86e6fdf to 0d16599 Compare August 31, 2026 08:59
@wu-sheng wu-sheng added this to the 0.12.0 milestone Aug 31, 2026
…ases

Publishing 0.11.0 failed on a job that should never have run, could not be resumed afterwards,
and would have failed again at the chart push even if it had got that far. Four things, all on
the release path, none of which had ever executed before.

GHCR is for commits and Docker Hub is for releases. The per-component images and the GHCR chart
were published on both paths, so cutting a release also built and pushed
ghcr.io/apache/skywalking-swck/operator:0.11.0 -- artifacts nobody asked for, from a job whose
failure then blocked the release that did not need it. The jobs are named for the path they
serve now, because the old names said what a job did rather than when it runs, which is how they
came to run on the wrong one unnoticed.

The chart push logged in to Docker Hub as registry-1.docker.io and pushed there. Both names
resolve to the same registry and either can be pulled from anonymously, which is why this looked
correct, but the credential is stored under the host you logged in as while a push normalises
the reference back to Docker Hub's canonical host -- so it would have looked up a credential
that was never written there and 401'd. apache/skywalking sets DOCKER_REGISTRY=docker.io and has
always published this way. The comment justifying the old host claimed docker.io does not serve
the /v2/ API that helm's OCI client talks to, which is untrue: helm pulls from oci://docker.io
perfectly well, as the docs' own install command does.

The image build fetched the tarball from archive.apache.org, a copy that holds every release
ever made but takes hours to receive a new one. A just-voted release therefore could not be
built until the archive caught up, and both release-passed.sh and the publish workflow sat in
long polls waiting for it -- an hour and thirty minutes respectively -- neither waiting for
anything the build reads. It reads dist.apache.org/repos/dist/release, where the svn move puts
the release and where it is servable immediately. Both waits check that and nothing else now.

release-passed.sh could not be resumed. Publishing 0.11.0 moved the artifacts, removed the
previous release, then died in that hour-long wait; re-running found dist/dev empty and refused
to start, leaving the rest to be done by hand. The version falls back to the newest entry in
dist/release, the existence check accepts either location, and the move checks whether it has
already happened before asking again whether the vote passed.

And the GitHub release carried a link to the changelog rather than the changelog. That link is
broken by construction: by the time anyone follows it, the next-version PR has moved the section
into changes-<version>.md and left an empty template behind, so it lands on the next version's
empty page. The notes carry the changelog itself now, read from the tag, which is what the PMC
voted on.
@wu-sheng
wu-sheng force-pushed the fix-release-fetch-from-downloads branch from e413931 to 68d3030 Compare August 31, 2026 09:11
@wu-sheng wu-sheng changed the title Publish snapshots to GHCR and releases to Docker Hub Fix the release publish path: GHCR for snapshots, Docker Hub for releases Aug 31, 2026
@wu-sheng
wu-sheng merged commit cd1c9a0 into master Aug 31, 2026
17 checks passed
@wu-sheng
wu-sheng deleted the fix-release-fetch-from-downloads branch August 31, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants