[Feature] Drain-Aware Node Upgrades

[dyu1208]

Azure Kubernetes Service (AKS) is working on a new capability that gives stateful and disruption-sensitive workloads more control over when individual nodes can be upgraded.

📋 What problem are we solving?

Some workloads need to evaluate replication health, quorum, replica roles, topology, or application state before a node can be safely drained.
Pod Disruption Budgets provide general availability protection, but they don’t allow an application to decide exactly which node can be disrupted or in what order.
Today, these workloads might experience blocked upgrades, repeated upgrade attempts, retained surge capacity, or require custom node-replacement workflows.

✨ What we’re proposing

Customers or their workload automation can mark a node as temporarily unavailable for upgrade:

kubectl label node <node> kubernetes.azure.com/upgrade-status=paused

After a normal AKS node-pool upgrade starts, AKS will:

• Skip paused nodes before cordoning or draining them.
• Continue upgrading eligible nodes.
• Keep paused nodes healthy and schedulable.
• Avoid creating or retaining surge capacity for paused nodes.
• Report paused nodes separately from unexpected upgrade failures.

To release a node:

kubectl label node <node> kubernetes.azure.com/upgrade-status-

No additional pool-level upgrade request is required after releasing the node.

🔄 Example upgrade flow

Assume a 100-node pool requires an upgrade and 80 nodes are paused:

• AKS upgrades the 20 eligible nodes.
• Active progress is measured across those 20 nodes.
• The 80 paused nodes remain schedulable and don’t retain surge replacements.
• As the workload releases nodes, AKS upgrades them to the pool’s desired version.
• The pool fully converges after all 100 nodes are upgraded.

🏷️ Paused versus quarantined nodes

State	Owner	Meaning	AKS behavior
upgrade-status=paused	Customer	The node is healthy but not currently safe to disrupt.	Skip it before execution without cordoning, draining, retaining surge capacity, or reporting a drain error.
upgrade-status=Quarantined	AKS	AKS attempted to upgrade the node, but the drain failed unexpectedly.	Apply the existing undrainableNodeBehavior and maxBlockedNodes behavior.

A separate Paused value is necessary because Quarantined represents an upgrade failure after AKS has already cordoned and attempted to drain the node. A quarantined node consumes an upgrade attempt, can retain surge capacity, and affects operation status.
A paused node instead represents customer intent before disruption and provides a release signal for later reconciliation.

🚨 Break-glass upgrades

For urgent upgrades, such as critical security updates, platform owners could use the existing AKS force-upgrade override.
For example:

az aks update \
  --resource-group <resource-group> \
  --name <cluster> \
  --enable-force-upgrade \
  --upgrade-override-until <timestamp>

az aks nodepool upgrade \
  --resource-group <resource-group> \
  --cluster-name <cluster> \
  --name <nodepool> \
  --kubernetes-version <version>

While the override is valid, AKS would ignore the pause label for that upgrade and report that the override was used.

🎯 Example scenario

Before ❌

A database operator knows that only one replica is currently safe to disrupt. AKS independently selects another node, its drain fails, and the upgrade retains replacement capacity or requires another attempt.

After ✅

The workload automation pauses nodes that aren’t ready. AKS upgrades the eligible replica first. When application health allows the next disruption, the automation removes that node’s pause label and AKS continues reconciliation.

⚠️ Initial limitations

• Customers or customer-managed automation are responsible for adding and removing the pause label.
• AKS doesn’t initially infer readiness from workload health or metrics.
• The feature changes node eligibility after an operation starts; it doesn’t change how upgrades, automatic upgrades, or Fleet upgrade runs are triggered.
• Unexpected drain failures continue to use existing quarantine behavior.
• AKS will prevent a pool from advancing when doing so would introduce more than two node versions.
• AKS doesn’t support node images older than 90 days, so long-lived pauses remain subject to AKS support and security requirements.

🔭 Future direction

We’re also exploring:

• Periodic reconciliation of released nodes using maintenance settings.
• An AKS-managed controller that automatically applies and removes pause labels for selected workloads.

Note: The label, API, CLI, status, and reconciliation examples illustrate the proposed experience and might change before release.