Kubernetes Cluster Upgrade

Upgrading a live Kubernetes cluster from one minor version to the next is one of the most critical and frequent operational tasks for a platform engineer. This page synthesizes the upgrade mechanics, strategies, and exact kubeadm commands required for the CKA exam and production environments. Source: CKA Day 34

Why Upgrades Are Frequent

Kubernetes releases a new minor version every ~3–4 months and supports only the latest three. When v1.31 ships, v1.28 goes out of support (no patches, no security fixes). For an organization running tens or hundreds of clusters, upgrades become a regular maintenance task — typically monthly or quarterly. Kubernetes Versioning & Version Skew details the release cadence and support window.

Upgrade Strategies

Strategy	How It Works	Downtime	Infrastructure Cost	Best For
All-at-once	Drain every node simultaneously, upgrade packages everywhere, then uncordon.	Total cluster downtime	None	Test/lab environments only
Rolling update (one at a time)	Drain → upgrade → uncordon one node, then repeat for the next.	Zero (if replicas > 1)	None	Standard production path
Blue/Green	Provision a new cluster on the target version, migrate workloads, retire old cluster.	Near-zero	Double (new cluster)	Large managed-cloud migrations, zero-risk cutovers

The CKA exam and most on-prem kubeadm workflows assume the rolling update strategy.

The Upgrade Order

Kubernetes enforces a strict sequence:

1. Primary control-plane node first — Run kubeadm upgrade apply here. This updates the API server, scheduler, controller-manager, and etcd Static Pod manifests in /etc/kubernetes/manifests.
2. Additional control-plane nodes (if HA) — Run kubeadm upgrade node on each secondary master. Never run apply on more than one control plane.
3. Worker nodes one at a time — Drain, upgrade kubeadm and kubelet packages, run kubeadm upgrade node, restart kubelet, then uncordon.

Why control plane first? Worker kubelets must not leap ahead of the API server. The version-skew policy allows kubelets to lag the API server by at most two minor versions, but upgrading workers before the control plane risks kubelet registration failures and API incompatibility.

kubeadm Upgrade Walkthrough (Single Control Plane)

1. Plan the Upgrade

# Update the apt repository to the target minor version (e.g., v1.30)
sudo sed -i 's/v1.29/v1.30/' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
 
# See available kubeadm versions
apt-cache madison kubeadm
 
# Preview what will change
sudo kubeadm upgrade plan

kubeadm upgrade plan prints a component-level diff: current cluster version, target version, and whether kubelet, kubectl, CNI, and etcd need manual attention.

2. Upgrade the Control Plane

# Apply the upgrade (only on the primary control-plane node)
sudo kubeadm upgrade apply v1.30.2

This rewrites the Static Pod manifests in /etc/kubernetes/manifests. The kubelet (already running) detects the changed files and restarts the control-plane containers with the new images. Verify with:

sudo grep image /etc/kubernetes/manifests/kube-apiserver.yaml

3. Evacuate and Upgrade the Control-Plane Node’s kubelet

# From any node with kubectl access
kubectl drain <master-node> --ignore-daemonsets
 
# On the master node itself
sudo apt-get install -y kubelet kubectl
sudo systemctl restart kubelet
 
# Make it schedulable again (if it runs workloads)
kubectl uncordon <master-node>

4. Upgrade Each Worker Node (One at a Time)

Repeat on every worker:

# --- On the worker node ---
sudo sed -i 's/v1.29/v1.30/' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
 
# Upgrade kubeadm binary and local node configuration
sudo apt-get install -y kubeadm
sudo kubeadm upgrade node
 
# Upgrade kubelet and kubectl
sudo apt-get install -y kubelet kubectl
sudo systemctl restart kubelet
 
# --- From the control plane (or any kubectl client) ---
kubectl drain <worker-node> --ignore-daemonsets
 
# --- After the worker packages are updated ---
kubectl uncordon <worker-node>

Worker tip: In the video demonstration, kubectl drain is executed from the control plane node against the worker by name. You can run kubectl from anywhere with a valid kubeconfig; the actual package upgrades and kubelet restart must happen on the worker itself.

5. Verify

kubectl get nodes
# All nodes should show the new version and Ready status
 
kubectl version
# Client version should match cluster version

HA Control Plane Nuance

In a stacked-etcd HA setup (3 control-plane nodes):

• Node 1: sudo kubeadm upgrade apply v1.30.2 — upgrades etcd and rewrites manifests.
• Node 2 & 3: sudo kubeadm upgrade node — only refreshes local manifests; etcd is already upgraded.

If you accidentally run kubeadm upgrade apply on a secondary master, etcd can experience split-brain or data inconsistency. Always use upgrade node for non-primary control-plane nodes.

Troubleshooting Matrix

Symptom	Root Cause	Fix
kubeadm upgrade plan shows no available versions	Apt repository still pointing to old minor version	Update /etc/apt/sources.list.d/kubernetes.list and apt-get update
kubectl drain hangs	DaemonSet Pods resisting eviction	Add --ignore-daemonsets
Node stays NotReady after upgrade	kubelet not restarted or container runtime unhealthy	systemctl status kubelet, check containerd
API server unreachable after control-plane upgrade	Manifest typo or certificate mismatch	Check /etc/kubernetes/manifests/kube-apiserver.yaml, verify certs with kubeadm certs check-expiration
Worker shows old version in kubectl get nodes	Forgot sudo kubeadm upgrade node on the worker	Run it, then restart kubelet

CKA Exam Patterns

• Version-skew trap: If the exam asks you to upgrade from v1.28 to v1.30, you must recognize the impossibility and state that only one minor version at a time is supported.
• Command recall: Know the difference between kubeadm upgrade apply (primary control plane only) and kubeadm upgrade node (all other nodes).
• Drain safety: Remember --ignore-daemonsets when the node runs CNI or CSI DaemonSets.
• Package management: Be comfortable editing apt source lists and running apt-get install for kubeadm, kubelet, and kubectl.

Practical Practice

Exam-style hands-on tasks for this topic. Complete each task before reviewing the solution. Time yourself — CKA tasks average 5–7 minutes.

Task 1: Upgrade Control Plane First Upgrade a kubeadm cluster from v1.29 to v1.30 — upgrade the control plane first, then drain and upgrade worker nodes one by one. Requirements: Update apt repo, run kubeadm upgrade plan, then apply. Verification: kubectl get nodes Solution:

# On control plane
sudo sed -i 's/v1.29/v1.30/' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.30.2
kubectl drain <master> --ignore-daemonsets
sudo apt-get install -y kubelet kubectl
sudo systemctl restart kubelet
kubectl uncordon <master>

Task 2: Upgrade a Worker Node Repeat on each worker: drain, upgrade packages, kubeadm upgrade node, restart kubelet, uncordon. Requirements: Worker kubelet must not be newer than the API server. Verification: kubectl get nodes Solution:

kubectl drain <worker> --ignore-daemonsets
# On worker:
sudo sed -i 's/v1.29/v1.30/' /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubeadm
sudo kubeadm upgrade node
sudo apt-get install -y kubelet kubectl
sudo systemctl restart kubelet
# On control plane:
kubectl uncordon <worker>
kubectl get nodes

Task 3: Fix Version Skew A node upgrade fails because the kubelet version is newer than the API server — explain the version skew policy and fix the order. Requirements: Control plane must be upgraded before workers. Verification: kubectl version and kubectl get nodes Solution:

kubectl version --short
kubectl get nodes -o wide
# Identify the out-of-order node
# If kubelet is v1.30 but API server is v1.29, downgrade kubelet or upgrade control plane first
sudo apt-get install -y kubelet=1.29.x-00
sudo systemctl restart kubelet

See Also

• Kubeadm Cluster Setup — Bootstrap procedure that must succeed before any upgrade
• Node Maintenance: Drain, Cordon, Uncordon — Evacuation mechanics that precede each node upgrade
• Kubernetes Versioning & Version Skew — Release cadence, support lifecycle, and component compatibility rules
• Kubernetes ETCD Backup and Restore — Pre-upgrade ETCD snapshot is mandatory before any upgrade; restore procedure if upgrade fails
• Kubernetes Disaster Recovery — Broader DR strategies including cluster reconstruction after failed upgrades
• Kubernetes Architecture — How Static Pods enable kubeadm to manage control-plane lifecycles
• Kubernetes Taints and Tolerations — NoExecute taint is the mechanism behind drain
• Kubernetes Static Pods — The node-local mechanism that bootstraps and upgrades control-plane containers
• Kubernetes Control Plane Troubleshooting — Day 38: upgrade-induced control plane failures and recovery
• CKA Certification — Exam weighting: cluster upgrades fall under Architecture, Installation & Configuration (~25%)
• CKA Day 34 — Step-By-Step Guide To Upgrade a Multi Node Kubernetes Cluster With Kubeadm

---

Tags: kubernetes cka kubeadm cluster-upgrade devops production version-skew