Kubernetes Logging and Monitoring

How Kubernetes handles logs, metrics, and observability — from individual container stdout to cluster-wide aggregation. A core Troubleshooting topic on the CKA exam and a production necessity. Synthesized from CKA Day 36 — Kubernetes Logging and Monitoring.

The Three Levels of Kubernetes Logging

Kubernetes generates logs at three distinct levels. A CKA candidate must know how to access each one:

Level	Scope	Access Method	CKA Relevance
Container	Single container stdout/stderr	kubectl logs	High — daily debugging and exam tasks
Node	Runtime and kubelet journals	SSH + journalctl, /var/log	Medium — node NotReady diagnosis
Cluster	Aggregated from all nodes	Centralized stack (EFK, PLG)	Low on exam; high in production

---

Level 1: Container Logs

Basic Log Retrieval

Every container’s stdout and stderr are collected by the kubelet and made available via the API server:

# Default container logs
kubectl logs nginx-pod
 
# Container in a multi-container Pod
kubectl logs multi-pod -c sidecar
 
# Previous instance (after crash or restart)
kubectl logs nginx-pod --previous
 
# Live tail
kubectl logs nginx-pod -f
 
# Last N lines
kubectl logs nginx-pod --tail=100
 
# Combined: follow previous container
kubectl logs nginx-pod --previous -f

Exam Trap: kubectl logs without -c fails on multi-container Pods. Always know which container you need, or use -c explicitly. Source: CKA Day 36

The --previous Flag

When a container restarts (e.g., due to CrashLoopBackOff or liveness probe failure), the current kubectl logs output only shows the new instance. The --previous flag retrieves logs from the terminated container — this is often the only way to see the actual error that caused the crash.

# Classic troubleshooting chain
kubectl get pod nginx-pod                    # Check RESTARTS column
kubectl describe pod nginx-pod             # Check Events for the why
kubectl logs nginx-pod --previous          # Read the crash output
kubectl exec -it nginx-pod -- /bin/sh      # Debug interactively if running

This chain mirrors the “Troubleshooting Chain” taught across the CKA course: get → describe → logs → exec. Source: CKA Day 0

---

Level 2: Node Logs

When the API server is unreachable or kubectl logs returns no data, inspect the node directly.

Filesystem Paths

On clusters using containerd or CRI-O (the modern default):

Path	Contents
/var/log/containers/	Symlinks to container log files, named <pod>_<ns>_<container>-<id>.log
/var/log/pods/	Pod-level directories with container log files
/var/log/syslog or /var/log/messages	General system logs

Service Journal Logs

# kubelet logs — critical for node health
journalctl -u kubelet -n 100
 
# container runtime logs
journalctl -u containerd -n 100
 
# API server logs (on control plane node)
journalctl -u kube-apiserver -n 100

CKA Note: In the exam environment, you may be given SSH access to nodes. Knowing where to look without the API server is a differentiator.

---

Level 3: Cluster Logging Architecture

Production clusters cannot rely on kubectl logs for operational visibility. The standard pattern uses a node-level agent that forwards logs to a central store.

DaemonSet Log Collector Pattern

┌─────────────────────────────────────────────────────────────┐
│                         Node 1                               │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐     │
│  │  App Pod    │────▶│  /var/log   │────▶│ Fluent Bit  │     │
│  │  stdout     │     │  containers │     │  (DaemonSet)│     │
│  └─────────────┘     └─────────────┘     └──────┬──────┘     │
│                                                  │            │
└──────────────────────────────────────────────────┼────────────┘
                                                   │
┌──────────────────────────────────────────────────┼────────────┐
│                         Node 2                     │            │
│  ┌─────────────┐     ┌─────────────┐             │            │
│  │  App Pod    │────▶│  /var/log   │─────────────┘            │
│  │  stdout     │     │  containers │                          │
│  └─────────────┘     └─────────────┘                          │
│                                                               │
└───────────────────────────────────────────────────────────────┘
                            │
                            ▼
                   ┌─────────────────┐
                   │  Central Store  │
                   │ (Loki, EFK, S3) │
                   └─────────────────┘

• Fluentd / Fluent Bit runs as a DaemonSet so every node has a log collector Pod
• The collector mounts hostPath volumes to read /var/log/containers/
• It forwards parsed logs to Elasticsearch, Loki, CloudWatch, or another store
• This pattern connects directly to DaemonSet and hostPath concepts from CKA Day 12

Sidecar Log Shipping Pattern

For legacy applications that log to files instead of stdout, a sidecar reads the shared volume and outputs to its own stdout (so kubelet captures it):

containers:
- name: legacy-app
  image: legacy-app
  volumeMounts:
  - name: app-logs
    mountPath: /opt/app/logs
- name: log-emitter
  image: busybox
  command: ["sh", "-c", "tail -f /opt/app/logs/app.log"]
  volumeMounts:
  - name: app-logs
    mountPath: /opt/app/logs
volumes:
- name: app-logs
  emptyDir: {}

This is the same Sidecar Pattern used for monitoring and TLS termination, applied to logging. Source: CKA Day 11

---

Metrics and Monitoring

Metrics Server

The Metrics Server is an in-memory aggregator that collects CPU and memory usage from kubelets and exposes them via the Kubernetes metrics API:

# Install Metrics Server (if missing)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
 
# Verify
kubectl get pods -n kube-system | grep metrics-server

Kind Cluster Note: On Kind clusters, Metrics Server requires --kubelet-insecure-tls because Kind uses self-signed kubelet serving certificates. In production, proper certificate trust is required.

kubectl top

# Node-level metrics
kubectl top node
 
# Pod-level metrics
kubectl top pod -n default
 
# Container-level breakdown
kubectl top pod my-pod --containers
 
# Sort by CPU
kubectl top pod --all-namespaces --sort-by=cpu

Output Column	Meaning
CPU(cores)	Current CPU consumption in millicores or cores
MEMORY(bytes)	Current memory consumption in MiB/GiB

Metrics Server vs Full Monitoring

Aspect	Metrics Server	Prometheus + Grafana
Data retention	In-memory only (~1 minute)	Persistent time-series storage
Historical trends	❌ No	✅ Yes
Alerting	❌ No	✅ Yes (Alertmanager)
Custom metrics	❌ No	✅ Yes (HPA v2 with custom metrics API)
CKA exam	✅ Tested	❌ Not tested

For the exam, Metrics Server is sufficient. For production, Prometheus and Grafana (or a managed equivalent) are the standard stack.

Security-Event Monitoring

Cluster telemetry also has a security dimension beyond health metrics and app logs. Runtime threat detection with Falco turns kernel syscall violations into an alert stream that lives in the same observability fabric: Falco emits to stdout, syslog, Prometheus, or HTTP, and Falcosidekick fans alerts out to Slack, S3, Loki, and other sinks. On a cluster install, Falco alerts are read straight from the agent Pod logs — kubectl logs -n falco -l app.kubernetes.io/name=falco — with Kubernetes metadata (k8s.pod.name) attached to each event. See Runtime Security. Source: Falco CKS Scenarios

---

Troubleshooting Matrix: Logs + Metrics

Symptom	Diagnostic Chain	Root Cause
CrashLoopBackOff	kubectl describe pod → kubectl logs --previous	App crash, missing env, bad config
Pod Pending	kubectl describe pod → kubectl top node	Insufficient CPU/memory on all nodes
Node NotReady	kubectl describe node → SSH → journalctl -u kubelet	kubelet down, disk pressure, runtime failure
DNS timeouts	kubectl logs -n kube-system -l k8s-app=kube-dns	CoreDNS overloaded, network partition
Service 502 errors	kubectl logs on backend Pods	Readiness probe failing, backend crash
HPA not scaling	kubectl get hpa → kubectl top pod	Metrics Server missing, no requests defined
OOMKilled	kubectl describe pod → kubectl top pod	Memory limit exceeded
High CPU throttling	kubectl top pod --containers	CPU limit too low for workload

---

CKA Exam Speed Patterns

# Fastest log inspection sequence
kubectl get pod <name> -o wide          # IP, node, restarts
kubectl describe pod <name>             # Events and state
kubectl logs <name> --previous         # Crash reason
kubectl logs <name> -c <container>     # Multi-container target
 
# Node-level diagnosis (SSH to node)
journalctl -u kubelet -n 50
ls /var/log/containers/ | grep <pod-name>
 
# Resource pressure check
kubectl top node
kubectl top pod -n <namespace>

Time Management: The exam rewards knowing the exact log source for each component. API server issues → control plane node journals. Pod crashes → kubectl logs --previous. Network issues → CNI and kube-proxy logs in kube-system.

---

Production Best Practices

Practice	Rationale
Log to stdout	Let the runtime handle log collection; avoid file-based logging when possible
Use structured logs (JSON)	Easier parsing by Fluentd/Fluent Bit and downstream analytics
Set log rotation	Prevent node disks from filling up; configure containerLogMaxSize and containerLogMaxFiles in kubelet
Install Metrics Server before HPA	HPA cannot function without it; verify on every new cluster
Separate metrics and logs pipelines	Don’t conflate Prometheus (metrics) with Loki/EFK (logs); they have different storage and query patterns
Monitor the monitors	If Fluent Bit or Metrics Server fails, you are flying blind

---

Practical Practice

Exam-style hands-on tasks for this topic. Complete each task before reviewing the solution. Time yourself — CKA tasks average 5–7 minutes.

Task 1: Retrieve Previous Container Logs A Pod app-1 crashed — retrieve logs from the previous container instance. Requirements: Do not delete the Pod. Verification: kubectl logs app-1 --previous | tail -5 Solution:

kubectl get pod app-1
kubectl describe pod app-1 | grep -A 5 "Last State"
kubectl logs app-1 --previous | tail -20

Task 2: Stream Multi-Container Pod Logs Stream logs from all containers in a multi-container Pod api-1. Requirements: Show logs from both containers simultaneously. Verification: Visual confirmation of mixed output. Solution:

kubectl logs api-1 -c app --tail=10
kubectl logs api-1 -c sidecar --tail=10
# Or follow both in separate terminals:
kubectl logs api-1 -c app -f
kubectl logs api-1 -c sidecar -f

Task 3: Install Metrics Server Install Metrics Server and verify kubectl top node works. Requirements: Use the official release manifest. Add --kubelet-insecure-tls on Kind. Verification: kubectl top node Solution:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
# On Kind, patch the deployment:
kubectl patch deployment metrics-server -n kube-system --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}]'
kubectl top node
kubectl top pod -n kube-system

Task 4: Identify and Evict Memory Pressure A node is under memory pressure — identify the top memory-consuming Pod and evict it if necessary. Requirements: Use kubectl top to find the culprit. Verification: kubectl top pod --all-namespaces --sort-by=memory Solution:

kubectl top node
kubectl top pod --all-namespaces --sort-by=memory
kubectl get pod <highest-pod> -o yaml | grep -A 2 "qosClass"
# Evict by deleting the Pod (controller will recreate if it exists)
kubectl delete pod <highest-pod> -n <ns>
kubectl top node

---

Related Pages

• Pod Fundamentals — the unit whose logs kubectl logs reads
• Multi-Container Pods — kubectl logs -c for container selection
• Sidecar Pattern — log shipping and metrics export via auxiliary containers
• Kubernetes DaemonSet — log collectors run as node-level DaemonSets
• Kubernetes Health Probes — probe failures generate events captured by kubectl describe
• Kubernetes Resource Requests and Limits — Metrics Server and kubectl top for live resource observation
• Horizontal Pod Autoscaler (HPA) — primary consumer of Metrics Server data
• Kubernetes Architecture — control plane and node component log locations
• Kubernetes Taints and Tolerations — tolerations for log collector DaemonSets on control plane nodes
• Kubeadm Cluster Setup — node-level log paths and kubelet configuration
• CKA Certification — exam domains where logging and monitoring appear
• CKA Study Roadmap — Day 36 in the 40-day plan
• Kubernetes Application Troubleshooting — Day 37: systematic debugging of app failures using logs and events
• Kubernetes Control Plane Troubleshooting — Day 38: journalctl, crictl logs, and node-level diagnosis for control plane failures
• Kubernetes Worker Node Troubleshooting — Day 39: kubelet, container runtime, and resource pressure log diagnosis
• Observability Index — broader monitoring, logging, and tracing patterns
• Tech Tutorials with Piyush — course source

---

Tags: kubernetes logging monitoring metrics-server kubectl-top troubleshooting cka devops observability daemonset sidecar