Node Maintenance: Drain, Cordon, Uncordon

Kubernetes provides three imperative commands to safely take a node out of service for maintenance, kernel patches, hardware replacement, or version upgrades. Understanding the exact behavior of each — and the edge cases around DaemonSets and standalone Pods — is essential for the CKA exam and production operations. Source: CKA Day 34

The Three Primitives

Command	What It Does to Existing Pods	What It Does to New Pods	Typical Use Case
kubectl cordon <node>	Nothing — they keep running	Prevents scheduling (unschedulable)	Temporary capacity reduction, pre-upgrade staging
kubectl drain <node>	Evicts all non-DaemonSet Pods gracefully	Prevents scheduling (unschedulable)	Node maintenance, kernel upgrades, hardware swaps
kubectl uncordon <node>	Nothing — does not move Pods back	Allows scheduling again	Node is healthy and ready to receive workloads

Cordon: The Scheduling Gate

kubectl cordon <node> adds a node.kubernetes.io/unschedulable:NoSchedule taint (internally implemented as a scheduling gate, not a true taint). The node remains in the cluster, existing Pods continue to serve traffic, but the scheduler ignores it for new Pod assignments.

kubectl cordon worker-node-1
kubectl get nodes
# worker-node-1   Ready,SchedulingDisabled

When to use cordon alone:

• You want to prevent new workloads from landing on a node that is showing elevated latency or disk pressure, but you are not yet ready to evict running Pods.
• You are staging a rolling upgrade and want to drain the node later.

Drain: Eviction + Cordon Combined

kubectl drain <node> does two things atomically:

1. Cordon the node (mark unschedulable).
2. Evict every Pod that is not managed by a DaemonSet.

For controller-managed Pods (Deployment, ReplicaSet, StatefulSet, Job), the controller notices the reduced replica count and spins up replacement Pods on other available nodes. For standalone Pods with no owner reference, drain deletes them permanently.

# Basic drain (refuses if DaemonSet Pods exist)
kubectl drain worker-node-1
 
# Production-safe drain (ignores DaemonSets)
kubectl drain worker-node-1 --ignore-daemonsets
 
# Force drain (delete standalone Pods, even if they have local storage)
kubectl drain worker-node-1 --ignore-daemonsets --force --delete-emptydir-data

The Standalone Pod Trap

A Pod created directly (e.g., kubectl run mysql --image=mysql) without a Deployment or ReplicaSet has no controller to recreate it. When drain evicts it, the Pod is gone forever. Any data stored inside that Pod’s ephemeral filesystem is lost. This is a frequent source of accidental data loss in production.

Production rule: Never run stateful or critical singletons as standalone Pods. Always wrap them in a Deployment (stateless) or StatefulSet (stateful) so drain preserves availability.

DaemonSet Resistance

DaemonSet Pods are designed to run on every node. By default, kubectl drain refuses to evict them because the DaemonSet controller would immediately recreate them anyway, causing an infinite loop. The --ignore-daemonsets flag tells drain to skip DaemonSet Pods and proceed with everything else. This is safe when the DaemonSet agent (e.g., CNI node driver, node-exporter) remains compatible after the node reboots or upgrades.

Uncordon: Restore Scheduling

kubectl uncordon <node> removes the unschedulable mark. The node can now receive new Pods. It does not reschedule already-evicted Pods back onto the node. The replacement Pods that were created on other nodes stay where they are. Only new scheduling decisions will consider this node.

kubectl uncordon worker-node-1

Complete Maintenance Workflow

The standard rolling-maintenance loop used during cluster upgrades and kernel patches:

# 1. Prevent new workloads
kubectl cordon worker-node-1
 
# 2. Evict running workloads (safe for controller-managed Pods)
kubectl drain worker-node-1 --ignore-daemonsets
 
# 3. Perform maintenance (upgrade packages, reboot, replace hardware, etc.)
# ...
 
# 4. Bring the node back
kubectl uncordon worker-node-1

Relationship to Taints and Tolerations

Drain does not use taints explicitly; it uses the internal scheduling gate. However, the effect is identical to applying a NoSchedule + NoExecute taint:

• NoSchedule prevents new Pods from landing (like cordon).
• NoExecute evicts existing Pods that lack a matching toleration (like drain).

The built-in node.kubernetes.io/unreachable:NoExecute and node.kubernetes.io/not-ready:NoExecute taints are the automatic versions of this behavior — applied by the node controller when a node stops reporting status. See Kubernetes Taints and Tolerations for the full taint catalog.

Troubleshooting

Symptom	Cause	Fix
drain fails with “cannot delete DaemonSet-managed Pods”	Missing --ignore-daemonsets	Add the flag
Standalone Pod deleted after drain	Pod had no owner reference	Use --force if acceptable, or migrate to a Deployment/StatefulSet
Node still shows SchedulingDisabled after uncordon	Forgot uncordon, or the node has a real taint	kubectl describe node <name> and look for taints
Evicted Pods stay in Pending	Cluster has no remaining capacity	Scale the cluster or uncordon other nodes

CKA Exam Patterns

• Drain is a composite: Many candidates waste time running cordon after drain. Drain already cordons. Know this to save seconds.
• Standalone Pods are destroyed: The exam may ask you to drain a node that holds a manually created Pod. If the Pod is not part of a ReplicaSet, it will be deleted. Be prepared to recreate it if the task requires it.
• DaemonSets are the exception: Always add --ignore-daemonsets in exam drain commands unless the task explicitly says otherwise.

Practical Practice

Exam-style hands-on tasks for this topic. Complete each task before reviewing the solution. Time yourself — CKA tasks average 5–7 minutes.

Task 1: Safely Drain a Node Safely evacuate node worker1 for maintenance using kubectl drain. Requirements: Ensure workload continuity. Do not delete DaemonSet Pods. Verification: kubectl get nodes and kubectl get pods -o wide Solution:

kubectl cordon worker1
kubectl drain worker1 --ignore-daemonsets
kubectl get nodes
kubectl get pods -o wide

Task 2: Restore Node to Service After maintenance, restore the node to service with kubectl uncordon. Requirements: Confirm the node is Ready before uncordoning. Verification: kubectl get nodes Solution:

kubectl get nodes
kubectl uncordon worker1
kubectl get nodes

Task 3: Handle DaemonSet Drain Failure A drain fails because of a DaemonSet Pod — use --ignore-daemonsets and explain the implications. Requirements: Explain why DaemonSet Pods resist eviction. Verification: kubectl drain worker1 --ignore-daemonsets Solution:

kubectl drain worker1
# Fails with: cannot delete DaemonSet-managed Pods
kubectl drain worker1 --ignore-daemonsets
# DaemonSet Pods (e.g., CNI, kube-proxy) remain because they are designed
# to run on every node. The DaemonSet controller would recreate them immediately,
# causing an infinite loop if drain tried to evict them.

See Also

• Kubernetes Cluster Upgrade — Rolling update strategy that uses drain/cordon/uncordon on every node
• Kubernetes ETCD Backup and Restore — Always snapshot ETCD before draining a control plane node for maintenance
• Kubernetes Disaster Recovery — DR planning for control plane node failures during maintenance windows
• Kubernetes Taints and Tolerations — Declarative scheduling constraints and automatic eviction mechanisms
• Kubernetes Manual Scheduling — Bypassing the scheduler with nodeName and nodeSelector
• Kubernetes Architecture — How kubelet reports node status and triggers the node controller’s automatic tainting
• Kubernetes DaemonSet — Workloads that resist drain and must exist on every node
• Kubernetes Control Plane Troubleshooting — Day 38: control plane node maintenance and post-drain recovery
• Kubernetes Worker Node Troubleshooting — Day 39: worker node failure diagnosis after maintenance
• CKA Day 34 — Step-By-Step Guide To Upgrade a Multi Node Kubernetes Cluster With Kubeadm

---

Tags: kubernetes cka node-maintenance drain cordon uncordon scheduling devops