Fix the release publish path: GHCR for snapshots, Docker Hub for releases - #220
Merged
Conversation
wu-sheng
force-pushed
the
fix-release-fetch-from-downloads
branch
from
August 31, 2026 08:59
86e6fdf to
0d16599
Compare
…ases Publishing 0.11.0 failed on a job that should never have run, could not be resumed afterwards, and would have failed again at the chart push even if it had got that far. Four things, all on the release path, none of which had ever executed before. GHCR is for commits and Docker Hub is for releases. The per-component images and the GHCR chart were published on both paths, so cutting a release also built and pushed ghcr.io/apache/skywalking-swck/operator:0.11.0 -- artifacts nobody asked for, from a job whose failure then blocked the release that did not need it. The jobs are named for the path they serve now, because the old names said what a job did rather than when it runs, which is how they came to run on the wrong one unnoticed. The chart push logged in to Docker Hub as registry-1.docker.io and pushed there. Both names resolve to the same registry and either can be pulled from anonymously, which is why this looked correct, but the credential is stored under the host you logged in as while a push normalises the reference back to Docker Hub's canonical host -- so it would have looked up a credential that was never written there and 401'd. apache/skywalking sets DOCKER_REGISTRY=docker.io and has always published this way. The comment justifying the old host claimed docker.io does not serve the /v2/ API that helm's OCI client talks to, which is untrue: helm pulls from oci://docker.io perfectly well, as the docs' own install command does. The image build fetched the tarball from archive.apache.org, a copy that holds every release ever made but takes hours to receive a new one. A just-voted release therefore could not be built until the archive caught up, and both release-passed.sh and the publish workflow sat in long polls waiting for it -- an hour and thirty minutes respectively -- neither waiting for anything the build reads. It reads dist.apache.org/repos/dist/release, where the svn move puts the release and where it is servable immediately. Both waits check that and nothing else now. release-passed.sh could not be resumed. Publishing 0.11.0 moved the artifacts, removed the previous release, then died in that hour-long wait; re-running found dist/dev empty and refused to start, leaving the rest to be done by hand. The version falls back to the newest entry in dist/release, the existence check accepts either location, and the move checks whether it has already happened before asking again whether the vote passed. And the GitHub release carried a link to the changelog rather than the changelog. That link is broken by construction: by the time anyone follows it, the next-version PR has moved the section into changes-<version>.md and left an empty template behind, so it lands on the next version's empty page. The notes carry the changelog itself now, read from the tag, which is what the PMC voted on.
wu-sheng
force-pushed
the
fix-release-fetch-from-downloads
branch
from
August 31, 2026 09:11
e413931 to
68d3030
Compare
wankai123
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publishing 0.11.0 failed on a job that should never have run, could not be resumed afterwards,
and would have failed again at the chart push even if it had got that far.
Four problems, all on the release path — which had never executed successfully before, so none
of them had ever been exercised.
1. GHCR for commits, Docker Hub for releases
The per-component images and the GHCR chart were published on both paths, so cutting a
release also built and pushed
ghcr.io/apache/skywalking-swck/operator:0.11.0— artifacts nobodyasked for, from a job whose failure then blocked the release that did not need it.
The jobs are named for the path they serve now. The old names said what a job did rather than
when it runs, which is how they came to run on the wrong one unnoticed:
publish-snapshot-imagespublish-snapshot-chartpublish-release-imagepublish-release-chart2. The chart push would have 401'd
It logged in to Docker Hub as
registry-1.docker.ioand pushed there. Both names resolve to thesame registry and either can be pulled from anonymously, which is exactly why this looked
correct:
But the credential is stored under the host you logged in as, while a push normalises the
reference back to Docker Hub's canonical host — so it would have looked up a credential that was
never written there.
apache/skywalkingsetsDOCKER_REGISTRY=docker.ioand has alwayspublished this way.
The comment justifying the old host claimed
docker.iodoes not serve the/v2/API thathelm's OCI client talks to. That is untrue — helm pulls from
oci://docker.ioperfectly well,as the docs' own install command does. All 11 occurrences across the workflow, docs, chart
README and
NOTES.txtnow usedocker.io, so the hostname a user copies is the one we publishto.
3. Both waits were pointed at something the build does not read
The image build fetched the tarball from
archive.apache.org— a copy that holds every releaseever made but takes hours to receive a new one. A just-voted release could not be built until
the archive caught up, and
release-passed.shand the publish workflow each sat in a long pollwaiting for it: an hour and thirty minutes respectively.
It reads
dist.apache.org/repos/dist/release, where the svn move puts the release and where itis servable immediately. Measured against the live 0.11.0:
Both waits check that and nothing else now — no mirror, no archive, no fallback chain.
downloads.apache.orgsurvives only in the release notes, because users should be sent to themirror even though the build should not.
4. release-passed.sh could not be resumed
Publishing 0.11.0 moved the artifacts, removed the previous release, then died in that hour-long
wait. Re-running found
dist/devempty and refused to start, leaving every remaining step to bedone by hand.
The version now falls back to the newest entry in
dist/release, the existence check acceptseither location, and
move_to_releasechecks whether it has already happened before askingagain whether the vote passed. Tested against the exact state 0.11.0 left behind: steps 2 and 3
skip cleanly.
Also: the release notes linked to a changelog that would not be there
They linked to
docs/en/changes/changes.md. That link is broken by construction: by the timeanyone follows it, the next-version PR has moved the section into
changes-<version>.mdandleft an empty template behind — so it lands on the next version's empty page. The notes now
carry the changelog itself, read from the tag, which is what the PMC voted on.