Prod deploy evicted from the merge queue when the autoscaler changes a node mid-transaction

[devantler]

🤖 Generated by the Agentic Engineer

Evidence

The prod deploy's node-safety transaction races the cluster autoscaler and refuses to proceed when
an autoscale node changes underneath it. Three merge-group deploys have now died this way in under
two hours, each evicting its PR from the merge queue:

When (UTC)	Run	Failure
2026-08-16 22:39	#3171	node claimed mid-transaction was replaced — identity changed … refusing image verification
2026-08-16 23:02	#3171	evicted again; orphaned Lease flux-system/ghcr-auth-refresh held past expiry
2026-08-17 00:16	#3171 (run 31980786395)	Talos node autoscale-cx43-2834820e83860317 identity changed, cordon ownership changed, or scheduling safety state changed before revision marker; refusing the mutation.

The guard itself is correct — refusing to mutate a node whose safety state moved is the right
call. The defect is that nothing reconciles the transaction with the autoscaler, so an ordinary
scale event is indistinguishable from a genuine safety violation.

Residue outlives the run. autoscale-cx43-2834820e83860317 is still cordoned=true with no
ghcr-auth-drain-owner annotation — a cordon with no owner, left behind after the refusal. A
cordoned node with no owner is exactly the state the next deploy will trip over again.

🩹 Heal Prod also failed in this run, which is the free discriminator for whose fault it is:
a base-state fault, not anything in the PR's diff.

Why it matters

Every platform merge is blocked while this recurs. The queue evicts on the failed merge_group,
so the PR bounces back to OPEN and the next re-queue hits the same race. Re-queuing without a fix is
futile by the contract's own merge-queue rule.

Prod itself was never at risk: all three deploys died before publish_platform_manifest, and prod is
currently 11/11 Kustomizations Ready — so this is a delivery-lane outage, not a production
incident.

Expected behaviour

A deploy is not evicted by a routine autoscaler scale event. Either the transaction tolerates an
autoscale node changing state while it holds no claim on that node, or it excludes autoscaler-managed
nodes from the safety marker, or it retries once against the refreshed node set.

Separately, a refused mutation leaves no cordoned node without a drain owner — the refusal path
should release its own cordon, the way the node fence self-releases elsewhere.

Acceptance criteria

• A merge-group deploy completes while an autoscale node is added or removed during the run.
• After a refused mutation, no node is left cordoned with ghcr-auth-drain-owner absent.
• The failure message distinguishes a routine autoscale event from a genuine safety violation.

Rough size: M.

Related: #3178 (stranded cx33 nodes consuming the autoscaler's budget) — a different defect on the
same autoscaler surface, not a duplicate of this one.

[coderabbitai]

🔗 Related PRs

#2985 - fix(k8s): gate deploy health on a serving CNPG cluster instead of a fully-ready one [merged]
#3034 - feat(ci): make an orphaned GHCR deploy fence diagnosable and recoverable [merged]
#3037 - fix(cilium): refuse a zero-count gate release and correct the runbook order [merged]
#3077 - fix(ci): reclaim a drain fence whose owning transaction is provably gone [merged]
#3109 - fix(deploy): avoid redundant GHCR node reboots [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[devantler]

🤖 Generated by the Agentic Engineer

Root cause located: the taint-set comparison has no exclusion for autoscaler-owned taints

The guard is node_scheduling_state_is_safe_to_reboot in scripts/refresh-flux-ghcr-auth-safety.sh.
It compares the node's entire taint set against the set captured at selection, excluding exactly
one key:

• node.kubernetes.io/unschedulable (it mirrors spec.unschedulable, so it is compared separately)

The lifecycle variant node_scheduling_state_is_safe_while_lifecycle_taints_clear excludes three
more — node.kubernetes.io/not-ready, node.kubernetes.io/unreachable,
node.cilium.io/agent-not-ready.

The cluster autoscaler's own taints are excluded nowhere, in either predicate:

• DeletionCandidateOfClusterAutoscaler (PreferNoSchedule) — added when a node merely becomes a
  candidate for removal, and removed again when it stops being one. No node deletion required.
• ToBeDeletedByClusterAutoscaler (NoSchedule) — added when removal actually starts.

So an ordinary autoscaler decision mutates the taint set, the comparison fails, and the guard reports
identity changed, cordon ownership changed, or scheduling safety state changed — which is why three
evictions with three genuinely different proximate causes all produced similar-looking messages.

Why this fires so readily here

cluster-autoscaler-hetzner-cluster-autoscaler (kube-system) re-evaluates scale-down on a ~10
second loop — its log emits Starting scale down at that cadence continuously. Its own counters at
the time of the 00:16Z eviction:

lastScaleUpTime          = 2026-08-17 00:09:41 UTC
lastScaleDownDeleteTime  = 2026-08-17 00:30:47 UTC

The deploy selected and held a node from the autoscaler-managed class and died at 00:16Z — between a
scale-up six minutes earlier and a scale-down delete fourteen minutes later. A transaction that holds
a node for minutes is racing a loop that re-decides every ten seconds, so this is expected to recur
rather than being bad luck.

One correction to the evidence above

The issue reads the leftover cordoned=true node with no ghcr-auth-drain-owner as residue from the
refusal. That is worth re-checking before it drives the fix: cordoning is also what the autoscaler
itself does to a node it is removing, so a cordon with no drain owner is the expected signature of
an in-flight scale-down, not necessarily a leaked cordon. The node in question has since been
reaped, so this cannot now be proven either way — but the acceptance criterion should be written
against a reproduced refusal rather than against that observation.

Implication for the fix

The guard is right to refuse to mutate a node that is being deleted — that is not the part to relax.
The defect is the blast radius: one node leaving the fleet aborts the whole deploy. The correct
behaviour is to deselect that node and continue with the rest, since an autoscaler-managed node
disappearing is a routine event with a specific, recognisable signature, not an unexplained safety
violation.

That also satisfies the third acceptance criterion directly: once autoscaler-owned taints are
recognised by name, the failure message can distinguish "this node is being scaled away" from
"something changed that I cannot explain", instead of collapsing both into one string.

Excluding the two autoscaler keys from the comparison without deselecting the node would be the
wrong fix — it would let the transaction proceed to reboot a node the autoscaler is deleting.

[devantler]

🤖 Generated by the Agentic Engineer

Confirmed from the autoscaler's own log — it was the advisory taint, not a deletion

The cluster-autoscaler log names the exact sequence on autoscale-cx43-2834820e83860317:

00:11:26.656  Successfully added DeletionCandidateOfClusterAutoscaler on node autoscale-cx43-2834820e83860317
00:15:43.035  Releasing taint {Key:DeletionCandidateOfClusterAutoscaler Value:1786925486 Effect:PreferNoSchedule}
00:15:43.048  Successfully released DeletionCandidateOfClusterAutoscaler on node autoscale-cx43-2834820e83860317
00:20:44.393  Successfully added DeletionCandidateOfClusterAutoscaler on node autoscale-cx43-2834820e83860317

The deploy's guard refused at 00:16 — 23 seconds after the release.

Two things this settles:

1. ToBeDeletedByClusterAutoscaler never appears anywhere in the window. The node was never
   marked for actual deletion during the transaction. The eviction was caused entirely by the
   advisory DeletionCandidateOfClusterAutoscaler marker (PreferNoSchedule) being released
   mid-transaction.
2. The marker churns by design. It is added, released and re-added on the autoscaler's ~10s
   re-evaluation loop, and its value is a regenerated timestamp — so it differs even key-for-key
   between two adds. A transaction holding a node for minutes is near-certain to observe it move.

What shipped

#3185 excludes that advisory marker from the scheduling-taint comparison at all six normalization
sites (capture and comparison, both scripts), so the report and the reference predicate cannot
disagree. ToBeDeletedByClusterAutoscaler is deliberately left in the comparison, so a node
genuinely being deleted still fails closed — that is asserted by an existing test which passes
unchanged, and by a new negative-control case.

The PR also normalizes both sides of the comparison. A recovery journal written before this
exclusion existed still carries the marker in its captured initialTaints, so a one-sided
normalization would have fixed the fresh-deploy path and still refused on the recovery path a
killed deploy leaves behind.

Correction to this issue's "residue" claim

The cordoned=true node with no drain-owner annotation is not leaked residue from the refusal:
the autoscaler cordons the nodes it removes, so that is the expected scale-down signature. It has
since cleared on its own — the node is gone and its replacement is uncordoned and untainted.

Still open (not covered by #3185)

A node that is genuinely being deleted (ToBeDeletedByClusterAutoscaler, or a set
deletionTimestamp) still aborts the entire deploy rather than being deselected from the target set.
Failing closed is correct, but aborting is heavier than necessary — the deploy could drop that node
and continue against the rest. Filed separately.