Skip to content

Restore prod bundle hashes and upgrade path clobbered by the test-branch merge - #141

Merged
jsokol merged 1 commit into
updates.simplerisk.comfrom
FIX-prod-feed-rc-contamination
Aug 22, 2026
Merged

Restore prod bundle hashes and upgrade path clobbered by the test-branch merge#141
jsokol merged 1 commit into
updates.simplerisk.comfrom
FIX-prod-feed-rc-contamination

Conversation

@jsokol

@jsokol jsokol commented Aug 21, 2026

Copy link
Copy Markdown
Member

Warning

Production feed is currently wrong. updates.simplerisk.com advertises a bundle_sha256 for the current GA release that matches no published artifact, so bundle-integrity verification fails for 20260820-001. This also blocks every simplerisk/docker master PR, because the image build verifies against this feed and fails closed.

Description

#140 merged the updates-test.simplerisk.com branch into the prod feed, carrying testing-channel truth into the production manifests.

The VM hashes in that merge are correct and are kept. This reverts only the parts that describe the prod channel.

What was wrong, and how each replacement value was confirmed

1. 20260820-001 bundle hashes — pointed at the testing bundle (61c78cca…). The testing bundle for this version was rebuilt at 15:04 today by the CHERRYPICK-ga-notes-testing merge, so it is different bytes from the GA bundle promoted at 14:41.

Restored to 3e279416…, verified by downloading both the prod S3 object and the simplerisk/code release asset — two independent artifacts that agree with each other and disagree with the feed.

2. 20260519-001 bundle hashes — same problem. Restored to 7d7fb242…, verified against the served prod S3 bundle.

3. The upgrade chain routed through releases that never shipped:

- <simplerisk-20260811-001>20260820-001</simplerisk-20260811-001>
- <simplerisk-20260709-001>20260811-001</simplerisk-20260709-001>
- <simplerisk-20260519-001>20260709-001</simplerisk-20260519-001>
+ <simplerisk-20260519-001>20260820-001</simplerisk-20260519-001>

20260709-001 and 20260811-001 are testing RCs that were never GA'dsimplerisk/code has no release for either, and 20260820-001 is the combined GA record for all three. Their bundles return 403 on the prod channel, so a customer on 20260519-001 would have been routed to a bundle that isn't there.

4. Their <release> and extra_compatibility entries are removed for the same reason.

The resulting diff against the last correct automated state (0a3cc2d7) is only:

  • the VM hashes (false → real) — your VM publish, preserved
  • <extra version="20260519-001"> and <workflows version="20260519-001" /> — legitimate additions from SimpleRisk 20260820-001 Release #140, preserved

Nothing else. That's the check that this reverts the contamination and nothing more.

Release Notes

None — internal, but customer-affecting. Restores correct bundle checksums and upgrade path on the production update feed.

Manual Validation Steps

  1. After merge, confirm the served feed agrees with the served bundle:
    curl -fsSL https://updates.simplerisk.com/releases.xml \
      | sed -n '/<release version="20260820-001">/,/<\/release>/p' | grep bundle_sha256
    curl -fsSL -o /tmp/b.tgz https://simplerisk-downloads.s3.amazonaws.com/public/bundles/simplerisk-20260820-001.tgz
    sha256sum /tmp/b.tgz     # must equal 3e27941634b3791a17e9f498ea992dc1f2c7aa566e201a8aec6e148120429ecf
  2. Re-run checks on a simplerisk/docker master PR — the image builds should pass again.
  3. Confirm the VM hashes survived (they should still be the real values, not false).
  4. Confirm upgrade_path.xml sends 20260519-001 straight to 20260820-001.

Type of Change

  • Bug fix (production data correction)

Testing Results

  • All three files parse as well-formed XML.
  • 9 assertions run against the corrected files: both bundle hashes match the bytes actually served, VM hashes preserved, testing-only releases absent, upgrade path direct with no phantom hops, extra_compatibility cleaned. All pass.
  • Both restore values were confirmed by downloading and hashing the real artifacts, not by trusting the previous feed contents.
  • Diffed against 0a3cc2d7 to prove nothing beyond the contamination was reverted.

CIA Impact Check

MAY negatively impact Confidentiality, Integrity and/or Availability.

Justification for the change

  • The prod feed advertises a checksum matching no published artifact, so any consumer that verifies — the Upgrade Extra's integrity check, installers, the docker image build — fails on the current GA release.
  • The upgrade path routes 20260519-001 customers to a bundle that 403s.

Potential impact on Confidentiality, Integrity and/or Availability

Integrity (net positive, this is the fix). Restores agreement between the published checksum and the published bytes. The risk in the change itself is writing a wrong value; mitigated by deriving both values from the served artifacts rather than from history, and by confirming 20260820-001's hash independently from two sources.

Availability (net positive). Restores one-click upgrade for 20260519-001 customers and unblocks docker CI.

Confidentiality. No change.

Additional Implementation Steps

The feed mirrors to S3 for the licensing service; confirm that mirror picks up the corrected manifests after merge.

Structural follow-up (not in this PR): the prod feed has a single writer — sync_code_repo.yml's update_feeds.sh, which writes the real GA asset hashes. Merging the testing branch into it bypasses that writer and silently replaces GA truth with RC truth, with no check that would catch it. Worth a branch rule making that merge impossible, plus a decision on pruning abandoned RCs from the test feed.



#140 merged the updates-test.simplerisk.com branch into the prod feed, which
carried testing-channel truth into the production manifests. The VM hashes in
that merge are correct and are KEPT -- this reverts only the parts that describe
the prod channel.

What was wrong, and how each value was confirmed:

- 20260820-001 bundle_md5/bundle_sha256 pointed at the TESTING bundle
  (61c78cca...). The testing bundle for this version was rebuilt at 15:04 on
  2026-08-21 by the CHERRYPICK-ga-notes-testing merge, so it is different bytes
  from the GA bundle promoted at 14:41. Restored to 3e279416..., verified by
  downloading BOTH the prod S3 object and the simplerisk/code release asset --
  two independent artifacts that agree with each other.

- 20260519-001 bundle_md5/bundle_sha256 likewise. Restored to 7d7fb242...,
  verified by downloading the served prod S3 bundle.

- 20260519-001 next_release, and upgrade_path.xml, routed upgrades through
  20260709-001 and 20260811-001. Those are testing RCs that never shipped GA --
  simplerisk/code has no release for either, and 20260820-001 is the combined
  GA record for all three. Their bundles 403 on the prod channel, so a customer
  on 20260519-001 would have been sent to a bundle that is not there. Restored
  to the direct 20260519-001 -> 20260820-001 hop.

- The 20260811-001 / 20260709-001 release and extra_compatibility entries are
  removed for the same reason: they are not prod releases.

Caught by simplerisk/docker's image build, which verifies the bundle against
this feed and fails closed. Every docker master PR has been blocked since 20:49.

The prod feed has a single writer -- sync_code_repo.yml's update_feeds.sh, which
writes the real GA asset hashes. Merging the testing branch into it bypasses
that writer and silently replaces GA truth with RC truth.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jsokol
jsokol merged commit c4e0b87 into updates.simplerisk.com Aug 22, 2026
@jsokol
jsokol deleted the FIX-prod-feed-rc-contamination branch August 22, 2026 20:56
jsokol added a commit that referenced this pull request Aug 22, 2026
The feed is a set of claims about published artifacts and nothing checked them.
On 2026-08-21 a merge of the updates-test branch into this one replaced the GA
bundle checksums with the testing channel's and added upgrade hops through
releases that never shipped. Nothing here went red. The break surfaced hours
later in simplerisk/docker, whose image build verifies the bundle against this
feed and fails closed -- blocking every PR in that repo.

Asserts the claims against reality:
  - every manifest parses, and carries no DTD (these are generated files; a
    DOCTYPE would only ever be an entity-expansion attack on the runner)
  - Current_Version names a release that exists and publishes a checksum
  - every release publishing a bundle_sha256 has a bundle actually fetchable on
    this channel, hashing to exactly that value
  - upgrade_path routes only to releases that exist, and agrees with each
    release's next_release

The checked set is deliberately "entries making a claim", not "all entries".
A historical release carries no checksum and its bundle is long gone from the
channel -- that is correct and unremarkable. Publishing a checksum is what makes
a claim checkable, so that is the trigger. Only 2 of 111 entries qualify today,
so this downloads two bundles.

Verified against both states: FAILS on a657849 with exactly the 4 real defects
and no false positives, PASSES on the #141 fix.

Worth making a required status check on both feed branches. Unlike a branch
restriction it catches a bad write however it arrives -- branch merge, hand
edit, or a broken generator run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant