security(render): prod render fetches third-party manifests unpinned, one from a moving branch

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

Measured 2026-08-17 while running the required prod authorization validator locally.

Two entries in the prod controllers render fetch third-party manifests over the network at render time:

• k8s/providers/hetzner/infrastructure/controllers/origin-ca-issuer/kustomization.yaml:8-9 — two CRDs from Cloudflare's trunk branch (a moving branch, not a pin)
• k8s/providers/hetzner/infrastructure/controllers/kubelet-serving-cert-approver/kustomization.yaml:21 — an install bundle pinned to tag v0.11.0 (a tag, no digest)

Both are active in the prod render (controllers/kustomization.yaml:43 and :45).

Observed today: raw.githubusercontent.com returned HTTP 429 on repeated attempts, and kubectl kustomize k8s/providers/hetzner/infrastructure/controllers aborted with accumulating resources ... 429: Too Many Requests. Four render-dependent tests in scripts/validate-eks-ci-role-policy failed identically on clean main and on a feature branch, which is what establishes the dependency as the cause rather than any change.

Problem

1. The render is not reproducible. trunk can move between two renders of the same commit, so one commit can produce two different expectedRenderedSurfaceSHA values with nothing in this repository having changed. That is directly relevant to the aggregate-fingerprint churn tracked in The EKS authorization surface hash goes stale on every HelmRelease dependency bump #2768.
2. Unreviewed third-party content reaches the prod authorization surface. The cert-approver install bundle contributes ServiceAccount / ClusterRole / (Cluster)RoleBinding documents to the very surface validate-eks-ci-role-policy exists to pin. A trunk move lands with no PR, no review, and no digest.
3. A required security gate is hostage to a third-party CDN. While the fetch rate-limits, the authorization measurement cannot be produced at all — which is how this was found.

Expected

Prod renders resolve entirely from reviewed, digest-pinned content in this repository.

Precedent — the pattern already exists here

scripts/update-vendored-operators.sh vendors pinned upstream bundles with SHA-256 verification and reviewed Checkov dispositions (CDI, KubeVirt), and CI re-validates the committed bytes against the pinned digest via --validate-committed. These two remotes are the outliers, not a new problem needing a new mechanism.

Acceptance criteria

• No kustomization.yaml reachable from a prod render path references a remote URL.
• Both dependencies are vendored under version control, pinned by digest, and refreshed through a reviewed updater path, with CI validating the committed bytes against that pin.
• The five authorization roots and k8s/clusters/prod render with no network access.
• A regression check fails if a new remote URL is introduced into a render path.

Suggested decomposition

1. A guard that fails when a render path references a remote URL — small, immediately valuable, and prevents a third outlier.
2. Vendor the origin-ca-issuer CRDs — most urgent, because trunk is unpinned today.
3. Vendor the kubelet-serving-cert-approver bundle — pinned to a tag, so lower risk than (2), but it is the one contributing authorization documents.

Rough size: M overall; child (1) is S.

[coderabbitai]

🔗 Related PRs

#2707 - fix(unifi): prevent pruning managed network resources [merged]
#2741 - fix(headlamp): disable unverified plugin downloads [merged]
#2742 - fix(apps): gate Actual Budget behind oauth2-proxy [merged]
#3109 - fix(deploy): avoid redundant GHCR node reboots [merged]
#3164 - fix(crossplane): restore direct provider egress [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[devantler]

🤖 Generated by the Agentic Engineer

The failure reaches CI too, and the mechanism is sharper than the filing states

The required pipeline on #3181 (head 07ec2b5f) failed at 🔍 Detect Changes → 💾 Validate two-stage PVC retirement, with 🔐 Validate EKS Authorization and every other k8s job skipped as a result. So this is not only a local-render inconvenience: while the fetch is failing, the required authorization gate does not run at all.

The mechanism. kustomize does not fail on a plain HTTP error. It reclassifies the unreachable raw URL as a git repository and then fetches that:

accumulating resources from 'https://raw.githubusercontent.com/cloudflare/origin-ca-issuer/trunk/.../clusteroriginissuers.yaml': URL is a git repository
failed to run '/usr/bin/git fetch --depth=1 https://raw.githubusercontent.com/cloudflare/origin-ca-issuer HEAD'

raw.githubusercontent.com/cloudflare/origin-ca-issuer is not a git repository, so that second fetch fails on its own terms. That is why the same root cause surfaces with two different terminal codes — 429 on my local render (kubelet-serving-cert-approver, which is accumulated first) and 404 in CI (origin-ca-issuer). The 404 is the misclassified git fetch, not evidence that Cloudflare deleted trunk.

Explicitly not claimed: that trunk is gone. I could not test it — curl returned 429 for both trunk and main from here, so the rate limit blocks the very check that would distinguish them. Whether trunk still exists is unresolved and does not need resolving, because vendoring removes the question.

Not a repo defect on its own. The immediate trigger is a GitHub-side CDN rate limit (main was green at 13:18Z, before the incident), and re-running once it clears is the sanctioned response. What is ours is the exposure this reveals: a required security gate and the prod render both fail closed on a third-party CDN, and one of the two dependencies is not pinned at all. The reproducibility argument in the filing stands independently of any incident.

[devantler]

🤖 Generated by the Agentic Engineer

New evidence — this now blocks merge-critical work, not just local renders

Filed yesterday on reproducibility grounds. Today it took a required check out on an open PR, so the
impact is larger than originally recorded.

platform#3181's CI run 32041832988 failed at 🔍 Detect Changes → 💾 Validate two-stage PVC retirement, and the cause is this dependency, not the PR:

accumulating resources from 'https://raw.githubusercontent.com/cloudflare/origin-ca-issuer/trunk/deploy/crds/cert-manager.k8s.cloudflare.com_clusteroriginissuers.yaml': URL is a git repository':
failed to run '/usr/bin/git fetch --depth=1 https://raw.githubusercontent.com/cloudflare/origin-ca-issuer HEAD': remote: 404: Not Found

The PR touches no manifests, so its diff cannot reach that step — the render pulls the CRDs live on
every validation, and an unavailable CDN fails the job regardless of the change under test. Note the
404 is still the misclassification described in the issue body (kustomize reclassifies the
unreachable raw URL as a git repo and reports its git fetch failure), not evidence that trunk
was deleted.

The "is trunk gone?" question is still open — and my probe cannot answer it

Direct probes today returned 429 for trunk, for main, and for a known-good unrelated file on
another repository. Since the control failed identically, the 429 is host-wide rate limiting on this
network rather than anything about this dependency, so it distinguishes nothing. Recording that as
unresolved rather than guessing — it does not change the fix either way.

Why this raises the priority

A third-party outage should degrade one optional job, not fail a required check on an unrelated PR.
Two independent failure modes are now observed:

1. Availability — an unreachable CDN fails required validation for changes that touch no manifests.
2. Reproducibility — trunk is a moving branch, so one commit can render two different
   authorization surfaces and therefore two different expectedRenderedSurfaceSHA values. That is a
   plausible contributor to the fingerprint churn on fix(dex): restrict GitHub auth to maintainers #2709/fix(apps): pin tenant OCI manifest tags #2737/fix(longhorn): reclaim orphaned replica directories automatically #3181.

Suggested first child, unchanged in shape but firmer in motivation

A guard that fails on any remote URL in a render path. It should ship with the two known remotes
recorded as explicit, dated exceptions rather than failing main on day one — the fix-vs-except
ladder — so the guard lands immediately and each exception is then retired by its own vendoring
change (scripts/update-vendored-operators.sh already establishes the vendored + SHA-256-pinned +
CI-revalidated pattern this should follow).

[devantler]

🤖 Generated by the Agentic Engineer

New evidence, 2026-08-18: this now evicts PRs from the merge queue

The same kubelet-serving-cert-approver fetch named above took a ready PR out of the merge queue
today — a failure mode a step beyond the local-test failures recorded when this was filed.

#3225 was enqueued at 21:04:21Z and evicted when the merge_group CI run failed in
🔍 Detect Changes → 💾 Validate two-stage PVC retirement:

error: accumulating resources from
'https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/v0.11.0/deploy/ha-install.yaml':
URL is a git repository
fatal: repository 'https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/' not found

Note the shape: kustomize's HTTP fetch failed, so it fell back to treating the URL as a git
remote, and the reported error is a git fetch 404 rather than the underlying HTTP failure. The
message therefore names neither the real cause nor this issue — it reads like a broken repository
reference, which is a misdirection cost on top of the outage.

That it was the dependency and not the change is established the same way #3196 established it
originally: the identical step succeeded on the same commit's own pull_request run
(15826dfef0, run 32181427032, 💾 Validate two-stage PVC retirement -> success) minutes earlier,
and main was green throughout. A clean re-queue then went through.

Why this raises the priority rather than just adding a datapoint. A local test failure is loud
and stops the person in front of it. A merge-queue eviction is quiet: the PR silently leaves the
queue and stays OPEN and CLEAN, so it reads exactly like one still draining. Any run that
re-queues without pulling the merge_group logs will keep paying for it without ever learning why —
and an unattended lane that does not check will simply leave finished work parked.