Kubernetes Troubleshooting Guide | Step-by-Step, Enterprise Level

Disclaimer: This document provides general Kubernetes troubleshooting guidance. All resource names, examples, and commands are illustrative and use placeholders. Also, this guide uses Azure Kubernetes Cluster, but it is intended to run on other Kubernetes distributions as well.

1. Environment Overview
2. Quick Reference — Cluster Access
3. Node Problems

3.1 Node Not Ready
3.2 Kured Reboots
3.3 Azure-Initiated Node Restarts
3.4 CPU Pressure / Memory Pressure
3.5 Disk Pressure
3.6 PID Pressure
3.7 Node Scaling Issues (VMSS)

4. Pod / Container Restarts

4.1 CrashLoopBackOff
4.2 OOMKilled
4.3 Pod Stuck in Pending
4.4 Pod Stuck in Terminating
4.5 ImagePullBackOff
4.6 Pod Evicted
4.7 Init Container Failures

5. Networking Issues

5.1 Service Unreachable
5.2 Ingress / NGINX Issues
5.3 Istio Service Mesh Issues
5.4 DNS Resolution Failures
5.5 NAT Gateway / External Connectivity

6. Resource Quota & Limits
7. Helm / Deployment Failures
8. Storage Issues
9. Certificate & Secret Issues
10. Monitoring & Observability

10.1 Metrics Server Issues

11. Security — Twistlock / Pod Security
12. HPA / KEDA Autoscaling Issues
13. Redis Issues
14. Azure-Specific Troubleshooting
15. AWS EKS-Specific Troubleshooting
16. Emergency Procedures

Overview

This is a practical Kubernetes troubleshooting guide covering common cluster issues, debugging techniques, and real-world fixes. This documentation helps DevOps engineers and beginners diagnose problems related to pods, deployments, networking, storage, services, and cluster components with step-by-step explanations and commands.

1. Environment Overview

Pre-Requisites

Before starting, make sure you have info of Cluster, Region, Resource Group, and Environment.
Collect this info before starting:

Kubernetes version
Cluster name
Affected Namespace Name
Cloud provider (AKS/EKS/GKE/on-prem)
Number of nodes
Node status
Region
Resource Group

2. Quick Reference — Cluster Access

Run below commands to access your cluster:
AKS — Get Credentials Command:

# Pattern: az aks get-credentials --resource-group <RG> --name <CLUSTER> --subscription <SUB>

Verify current context in kubernetes:

kubectl config current-context
kubectl config get-contexts
kubectl cluster-info

Switch the context if needed:

kubectl config use-context <context-name>

<CONTEXT_NAME> is the name of the cluster.

3. Node Problems

Following problems can be the root cause of Node not Ready or node reboot issue:

3.1 Node Not Ready

A node in NotReady state may mean the kubelet is not communicating with the API server.

Diagnostic Checklist

Is the node visible in kubectl get nodes?
Was there a kured reboot scheduled?
Did Azure perform a maintenance event?
Is the node under resource pressure (CPU, memory, disk)?
Is the kubelet running on the node?
Is the VMSS instance healthy?

Step-by-Step Commands

Step 1: Check node status

kubectl get nodes -o wide

Step 2: Get detailed node info

kubectl describe node <node-name>

Look for:

Conditions section: check Ready, MemoryPressure, DiskPressure, PIDPressure
Events section: look for NodeNotReady, NodeHasSufficientMemory, etc.
Taints: look for node.kubernetes.io/not-ready or node.kubernetes.io/unreachable

Step 3: Check node events (last 1 hour)

kubectl get events --field-selector involvedObject.kind=Node,involvedObject.name=<NODE_NAME> --sort-by='.lastTimestamp'

Step 4: Check all cluster events for node issues

kubectl get events --all-namespaces --sort-by='.lastTimestamp' | grep -i "node\|NotReady\|reboot\|drain"

Step 5: Check if kured initiated a reboot

# Check kured logs
kubectl logs -n kube-system -l app.kubernetes.io/name=kured --tail=200

# Check for kured reboot annotation on nodes
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.weave\.works/kured-reboot-in-progress}{"\n"}{end}'

Step 6: Check Azure Activity Log for the node's VMSS

# List VMSS instances
az vmss list-instances --resource-group <RG> --name <VMSS_NAME> -o table

# Check Azure Activity Log for restarts
az monitor activity-log list --resource-group <RG> \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
  --query "[?contains(operationName.value, 'restart') || contains(operationName.value, 'deallocate') || contains(operationName.value, 'start')].{Time:eventTimestamp, Op:operationName.value, Status:status.value}" \
  -o table

Step 7: Check VMSS instance health

az vmss get-instance-view --resource-group <RG> --name <VMSS_NAME> --instance-id <INSTANCE_ID>

Step 8: If node stays NotReady — cordon and drain (Not Recommended For Production)

This step is not recommended for production environments as it may cause downtime. Use with caution.

# Cordon the node (prevent new pods)
kubectl cordon <NODE_NAME>

# Drain the node (evict existing pods)
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data --force --grace-period=60

# After node recovers, uncordon
kubectl uncordon <NODE_NAME>

3.2 Kured Reboots

Kured (KUbernetes REboot Daemon) is a Kubernetes daemonset that performs safe automatic node reboots when the need to do so is indicated by the package management system of the underlying OS. It is a common tool used in AKS clusters to automate node reboots for maintenance. If a node is rebooting, it may be due to a kured-initiated reboot. Kured (KUbernetes REboot Daemon) automatically reboots nodes when /var/run/reboot-required is present (e.g., after kernel updates).
Kured has different schedules defined.

Diagnostic Checklist

Is the node restart within the kured reboot window?
Does the node have the reboot annotation?
Is kured running on all nodes/affected node?
Was the reboot successful?

Step-by-Step Commands

Step 1: Check kured daemonset status

kubectl get ds kured -n kube-system
kubectl get pods -n kube-system -l app.kubernetes.io/name=kured -o wide

Step 2: Check kured logs for reboot activity

kubectl logs -n kube-system -l app.kubernetes.io/name=kured --tail=500 | grep -i "reboot\|drain\|uncordon\|lock"

Step 3: Check if any node has a reboot-in-progress annotation

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}: {.metadata.annotations.weave\.works/kured-reboot-in-progress}{"\n"}{end}'

Step 4: Check if a node is stuck with reboot annotation or If a node has the annotation but hasn't rebooted (e.g., draining took too long):

# Remove the stuck annotation to unlock kured
kubectl annotate node <NODE_NAME> weave.works/kured-reboot-in-progress-

# Verify annotation is removed
kubectl get node <NODE_NAME> -o jsonpath='{.metadata.annotations}'

Step 5: Check if nodes need reboot (from kured perspective)

# SSH into the node or exec into a privileged pod and check:
cat /var/run/reboot-required

Step 6: Manually block kured if needed (EMERGENCY)

# Lock kured by adding an annotation to ALL nodes (to block reboots during incident)
kubectl annotate nodes --all kured/reboot-blocked="true"

# Unlock when ready
kubectl annotate nodes --all kured/reboot-blocked-

3.3 Azure-Initiated Node Restarts

Azure may restart nodes for planned maintenance (host OS updates), unplanned hardware failures, or live migration events

Diagnostic Checklist

Check Azure Service Health for planned maintenance
Check VMSS instance events
Check Azure Activity Log for the resource group
Check Scheduled Events API from the node

Step-by-Step Commands

Step 1: Check Azure Service Health

# Via Azure CLI
az monitor activity-log list --resource-group  \
  --start-time $(date -u -d '48 hours ago' +%Y-%m-%dT%H:%M:%SZ) \
  --query "[].{Time:eventTimestamp, Operation:operationName.value, Status:status.value, Caller:caller}" \
  -o table

Step 2: Check VMSS events

az vmss list-instances --resource-group <RG> --name <VMSS_NAME> \
  --query "[].{Name:name, State:instanceView.statuses[0].displayStatus, ProvisioningState:provisioningState}" -o table

Step 3: Check for Azure Scheduled Events (from inside a node)

# If you have node access:
curl -H Metadata:true http://<IP-adress>/metadata/scheduledevents?api-version=2020-07-01

Step 4: Check AKS maintenance configuration

az aks maintenanceconfiguration list --resource-group <RG> --cluster-name <CLUSTER> -o table

Step 5: Check node uptime to identify recent restarts

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\t"}{.status.conditions[?(@.type=="Ready")].lastTransitionTime}{"\n"}{end}'

3.4 CPU Pressure / Memory Pressure

Diagnostic Checklist

Which node is under pressure?
What pods are consuming the most resources?
Are resource limits properly set?
Is the node overcommitted?
Are there pods without resource limits?

Step-by-Step Commands

Step 1: Check node conditions

kubectl get nodes -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,CPU_PRESSURE:.status.conditions[?(@.type=="MemoryPressure")].status,MEM_PRESSURE:.status.conditions[?(@.type=="MemoryPressure")].status,DISK_PRESSURE:.status.conditions[?(@.type=="DiskPressure")].status,PID_PRESSURE:.status.conditions[?(@.type=="PIDPressure")].status'

Step 2: Check node resource usage with metrics-server

kubectl top nodes

Step 3: Check node resource allocation (requests vs allocatable)

kubectl describe node <NODE_NAME> | Select-String "Allocated resources"

Step 4: Find top resource-consuming pods on a specific node

kubectl top pods --all-namespaces --sort-by=cpu | head -30
kubectl top pods --all-namespaces --sort-by=memory | head -30

Step 5: Find pods on a specific node

kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<NODE_NAME>

Step 6: Find pods WITHOUT resource limits (they can cause pressure)

kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.spec.containers[].resources.limits == null) | "\(.metadata.namespace)/\(.metadata.name)"'

Step 7: Check if node is overcommitted

# Total requests vs capacity
kubectl describe node <NODE_NAME> | grep -E "cpu|memory" | head -10

Step 8: Quick view, all nodes resource usage percentage

kubectl top nodes --no-headers | awk '{printf "%-50s CPU: %s/%s (%s) MEM: %s/%s (%s)\n", $1, $2, $3, $4, $5, $6, $7}'

3.5 Disk Pressure

Diagnostic Checklist

Is the node under DiskPressure condition?
Are old container images filling the disk?
Are emptyDir volumes consuming space?
Are container logs too large?

Step-by-Step Commands

Step 1: Check for DiskPressure condition

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="DiskPressure")].status}{"\n"}{end}'

Step 2: Check kubelet disk thresholds (default: 85% hard eviction)

kubectl get node <NODE_NAME> -o jsonpath='{.status.allocatable}' | jq .

Step 3: If you have node access, check disk usage

# From a privileged pod or SSH:
df -h
du -sh /var/lib/docker/* 2>/dev/null | sort -rh | head -10
du -sh /var/log/containers/* 2>/dev/null | sort -rh | head -10
crictl images | sort -k3 -rh | head -20

Step 4: Trigger image garbage collection

# On AKS, you can use the docker-disk-cleanup approach:
crictl rmi --prune    # (removes unused images)

Step 5: Find pods using most ephemeral storage

kubectl get pods --all-namespaces -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name) \(.spec.containers[].resources.limits.["ephemeral-storage"] // "no-limit")"'

3.6 PID Pressure

Step-by-Step Commands

Step 1: Check for PIDPressure

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t PIDPressure: "}{.status.conditions[?(@.type=="PIDPressure")].status}{"\n"}{end}'

Step 2: Find pods with excessive processes

# Get pods on the affected node
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=

# Check process count from node (if accessible)
ps aux | wc -l

3.7 Node Scaling Issues (VMSS)

Step-by-Step Commands

Step 1: Check AKS cluster autoscaler status

kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

Step 2: Check autoscaler logs

kubectl logs -n kube-system -l app=cluster-autoscaler --tail=200

Step 3: Check VMSS scaling events

az vmss list --resource-group  -o table
az vmss show --resource-group  --name <VMSS_NAME> --query "sku" -o table

Step 4: Manually scale VMSS (if autoscaler is not working)

az vmss scale --resource-group <RG> --name <VMSS_NAME> --new-capacity <COUNT>

4. Pod / Container Restarts

4.1 CrashLoopBackOff

The container keeps crashing and Kubernetes keeps restarting it with an exponential backoff.

Diagnostic Checklist

What is the exit code?
Are liveness/readiness probes failing?
Is the application running out of memory?
Are required config maps/secrets missing?
Are required dependencies (DB, Redis, external APIs) reachable?
Is the container image correct and pullable?

Step-by-Step Commands

Step 1: Identify pods in CrashLoopBackOff

kubectl get pods -n <NAMESPACE> | grep -i "crash\|error\|backoff"

Step 2: Get pod details

kubectl describe pod <POD_NAME> -n <NAMESPACE>

Look for:

Last State → Terminated → Exit Code
Restart Count
Events → container start/failure messages
Liveness / Readiness probe results

Step 3: Check current and previous container logs

# Current logs
kubectl logs <POD_NAME> -n <NAMESPACE> -c <CONTAINER_NAME>

# Previous crashed container logs
kubectl logs <POD_NAME> -n <NAMESPACE> -c <CONTAINER_NAME> --previous

Step 4: Common exit codes

Exit Code	Meaning	Common Cause
0	Success	Application exited cleanly (but Kubernetes restarts it)
1	Application Error	Code exception, missing configuration
126	Permission denied	Cannot execute the entrypoint
127	Command not found	Wrong entrypoint/command in container specification
137	SIGKILL (OOMKilled)	Container exceeded memory limit
139	SIGSEGV	Segmentation fault
143	SIGTERM	Graceful termination (normal shutdown)

Step 5: Check if probes are the problem

kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].livenessProbe}' | jq .
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].readinessProbe}' | jq .

Step 6: Temporarily disable probes for debugging (do NOT do in prod without approval)

Modify the deployment to increase probe timeouts or comment out probes to isolate the issue.

Step 7: Identify pods in CrashLoopBackOff

kubectl exec -it <POD_NAME> -n <NAMESPACE> -c <CONTAINER_NAME> -- /bin/sh

4.2 OOMKilled

Container killed because it exceeded its memory limit.

Diagnostic Checklist

What is the current memory limit?
Is the application leaking memory?
Is the JVM heap configured correctly (for Java apps)?
Did the workload change (more data, more requests)?

Step-by-Step Commands

Step 1: Confirm OOMKilled

kubectl describe pod <POD_NAME> -n <NAMESPACE> | grep -A 5 "Last State"
# Look for: Reason: OOMKilled

Step 2: Check current memory limit

kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].resources}' | jq .

Step 3: Check actual memory usage

kubectl top pod <POD_NAME> -n <NAMESPACE> --containers

Step 4: Check memory usage over time (Dynatrace or any monitoring you use)

Go to Dynatrace → Infrastructure → Kubernetes Workloads → Select the workload → Check memory usage graph.

Step 5: For Java/Scala applications — check JVM settings

kubectl exec -it <POD_NAME> -n <NAMESPACE> -- env | grep -i "java\|jvm\|heap\|xmx\|xms"

Update the appropriate values file in your infrastructure repository repo and deploy through the pipeline.

Step 6: Increase memory limit if needed Your Helm values files define resource limits. For example:

resources:
  requests:
    cpu: 100m
  limits:
    memory: 6G    # Increase this
    cpu: 1

Step 7: Check namespace quota to ensure room

kubectl describe resourcequota -n <NAMESPACE>
kubectl get resourcequota -n <NAMESPACE> -o yaml

4.3 Pod Stuck in Pending

Diagnostic Checklist

Is there enough node capacity (CPU/memory)?
Is the namespace resource quota exceeded?
Are there node affinity/anti-affinity constraints?
Are there taints preventing scheduling?
Is there a missing PersistentVolume?

Step-by-Step Commands

Step 1: Find pending pods

kubectl get pods --all-namespaces --field-selector status.phase=Pending

Step 2: Describe the pending pod

kubectl describe pod <POD_NAME> -n <NAMESPACE>
# Look at Events section for scheduling errors:
#   - "Insufficient cpu"
#   - "Insufficient memory"
#   - "0/N nodes are available: N node(s) had taint..."
#   - "exceeded quota"

Step 3: Check namespace quota utilization

kubectl describe resourcequota -n <NAMESPACE>

Step 4: Check available resources on nodes

kubectl top nodes
kubectl describe nodes | grep -A 10 "Allocated resources"

Step 5: Check for taints on nodes

kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints'

Step 6: Check node affinity rules on the pod

kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.affinity}' | jq .
kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.nodeSelector}' | jq .

4.4 Pod Stuck in Terminating

Diagnostic Checklist

Is the pod stuck due to a finalizer?
Is the node unreachable?
Is there a PDB (Pod Disruption Budget) preventing termination?
Are there taints preventing scheduling?
Is Istio sidecar preventing graceful shutdown?

Step-by-Step Commands

Step 1: Check stuck terminating pods

kubectl get pods --all-namespaces | grep Terminating

Step 2: Check for finalizers

kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.metadata.finalizers}'

Step 3: Force delete the pod (last resort)

kubectl delete pod <POD_NAME> -n <NAMESPACE> --grace-period=0 --force

Step 4: If the node is unreachable and pods are stuck

# The pods will remain Terminating until the node comes back
# If the node is permanently gone, delete it from Kubernetes
kubectl delete node 
# The VMSS autoscaler or AKS will provision a replacement

4.5 ImagePullBackOff

Diagnostic Checklist

Is the image tag correct?
Is the container registry accessible?
Are the image pull secrets valid?
Is there a network issue to the registry?
Is the registry rate-limited?

Step-by-Step Commands

Step 1: Find ImagePullBackOff pods

kubectl get pods -n <NAMESPACE> | grep -i "imagepull\|errimagepull"

Step 2: Check the exact error

kubectl describe pod <POD_NAME> -n <NAMESPACE> | grep -A 10 "Events"

Step 3: Check which image is failing

kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].image}'

Step 4: Verify image pull secrets

kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.imagePullSecrets}'
kubectl get secret <SECRET_NAME> -n <NAMESPACE> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

Step 5: Test registry access manually

# For ACR:
az acr login --name <ACR_NAME>
az acr repository show-tags --name <ACR_NAME> --repository <REPO> -o table

# Check if the specific tag exists
az acr repository show --name <ACR_NAME> --image <REPO>:<TAG>

4.6 Pod Evicted

Diagnostic Checklist

Was the node under memory/disk pressure?
Is the pod using more ephemeral storage than allowed?
Are the pod's QoS class and priority appropriate?

Step-by-Step Commands

Step 1: Find evicted pods

kubectl get pods --all-namespaces --field-selector status.phase=Failed | grep Evicted

Step 2: Check eviction reason

kubectl describe pod <EVICTED_POD> -n <NAMESPACE> | grep -i "evict\|reason\|message"

Step 3: Clean up evicted pods

kubectl get pods --all-namespaces --field-selector status.phase=Failed -o json | \
  jq -r '.items[] | select(.status.reason=="Evicted") | "\(.metadata.namespace) \(.metadata.name)"' | \
  xargs -L1 bash -c 'kubectl delete pod $1 -n $0'

Step 4: Check pod QoS class

kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.status.qosClass}'
# Guaranteed > Burstable > BestEffort (eviction priority order)

4.7 Init Container Failures

Step-by-Step Commands

Step 1: Check init container status

kubectl describe pod <POD_NAME> -n <NAMESPACE> | grep -A 20 "Init Containers"

Step 2: Check init container logs

kubectl logs <POD_NAME> -n <NAMESPACE> -c <INIT_CONTAINER_NAME>

Step 3: Common init container issues for Istio (istio-init)

# Check if istio-init is failing
kubectl logs <POD_NAME> -n <NAMESPACE> -c istio-init
# Usually related to iptables rules — check pod security policies

5. Networking Issues

5.1 Service Unreachable

Diagnostic Checklist

Is the service endpoint healthy?
Is the pod behind the service running?
Is the service selector matching pod labels?
Is there a NetworkPolicy blocking traffic?

Step-by-Step Commands

Step 1: Check service and its endpoints

kubectl get svc <SERVICE_NAME> -n <NAMESPACE>
kubectl get endpoints <SERVICE_NAME> -n <NAMESPACE>

Step 2: Verify endpoints have IP addresses

# If endpoints list is empty, the service selector doesn't match any running pods
kubectl describe endpoints <SERVICE_NAME> -n <NAMESPACE>

Step 3: Check labels match

# Get service selector
kubectl get svc <SERVICE_NAME> -n <NAMESPACE> -o jsonpath='{.spec.selector}'

# Get pod labels
kubectl get pods -n <NAMESPACE> --show-labels | grep <APP_NAME>

Step 4: Test connectivity from another pod

kubectl run test-curl --image=curlimages/curl --restart=Never -n <NAMESPACE> -- \
  curl -v http://<SERVICE_NAME>.<NAMESPACE>.svc.cluster.local:<PORT>/health
kubectl logs test-curl -n <NAMESPACE>
kubectl delete pod test-curl -n <NAMESPACE>

Step 5: Check NetworkPolicies

kubectl get networkpolicies -n <NAMESPACE>
kubectl describe networkpolicy <POLICY_NAME> -n <NAMESPACE>

5.2 Ingress / NGINX Issues

Diagnostic Checklist

Is the ingress resource configured correctly?
Is the NGINX ingress controller running?
Is the TLS certificate valid?
Is the backend service healthy?

Step-by-Step Commands

Step 1: Check ingress resources

kubectl get ingress -n <NAMESPACE>
kubectl describe ingress <INGRESS_NAME> -n <NAMESPACE>

Step 2: Check NGINX ingress controller pods

kubectl get pods -n <ingress-nginx-namespace>
kubectl logs -n <ingress-nginx-namespace> -l app.kubernetes.io/component=controller --tail=200

Step 3: Check NGINX ingress controller for errors

kubectl logs -n <ingress-nginx-namespace> -l app.kubernetes.io/component=controller --tail=500 | grep -i "error\|warn\|503\|502\|504"

Step 4: Check if external ingress controller is also healthy

kubectl get pods -n <ingress-nginx-ext-namespace>
kubectl logs -n <ingress-nginx-ext-namespace> -l app.kubernetes.io/component=controller --tail=200

Step 5: Verify ingress annotations are correct Your services typically use:

annotations:
  nginx.ingress.kubernetes.io/proxy-body-size: 1024m

Step 6: Test ingress endpoint directly

curl -vk https://<INGRESS_HOST>/<PATH>

Step 7: Check NGINX configuration for specific backend

kubectl exec -it -n <ingress-nginx-namespace> <NGINX_POD> -- cat /etc/nginx/nginx.conf | grep -A 10 ""

5.3 Istio Service Mesh Issues

Diagnostic Checklist

Is istio-proxy sidecar injected in the pod?
Is istiod (control plane) running?
Are AuthorizationPolicies blocking traffic?
Is mTLS configured correctly?

Step-by-Step Commands

Step 1: Check istiod status

kubectl get pods -n <istio-system-namespace>
kubectl logs -n <istio-system-namespace> -l app=istiod --tail=200

Step 2: Check if sidecar is injected

kubectl get pod <POD_NAME> -n <NAMESPACE> -o jsonpath='{.spec.containers[*].name}'
# Should see: <app-container> istio-proxy

Step 3: Check istio-proxy logs

kubectl logs <POD_NAME> -n <NAMESPACE> -c istio-proxy --tail=200

Step 4: Check proxy sync status

# Using istioctl if available:
istioctl proxy-status
istioctl analyze -n <NAMESPACE>

Step 5: Check AuthorizationPolicies

kubectl get authorizationpolicies -n <NAMESPACE>
kubectl describe authorizationpolicy <POLICY_NAME> -n <NAMESPACE>

Step 6: Check if mTLS is enforced

kubectl get peerauthentication --all-namespaces

Step 7: Debug Envoy proxy configuration

kubectl exec -it <POD_NAME> -n <NAMESPACE> -c istio-proxy -- pilot-agent request GET clusters
kubectl exec -it <POD_NAME> -n <NAMESPACE> -c istio-proxy -- pilot-agent request GET config_dump

5.4 DNS Resolution Failures

Step-by-Step Commands

Step 1: Check CoreDNS is running

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100

Step 2: Test DNS from a pod

kubectl run dns-test --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default
kubectl logs dns-test
kubectl delete pod dns-test

Step 3: Test external DNS resolution

kubectl run dns-test --image=busybox:1.36 --restart=Never -- nslookup google.com
kubectl logs dns-test
kubectl delete pod dns-test

Step 4: Check CoreDNS configmap

kubectl get configmap coredns -n kube-system -o yaml

Step 5: Check if nodelocaldns is running (for EKS clusters)

kubectl get ds nodelocaldns -n kube-system
kubectl get pods -n kube-system -l k8s-app=node-local-dns

5.5 NAT Gateway / External Connectivity

For AKS clusters, outbound traffic flows through NAT Gateway.

Step-by-Step Commands

Step 1: Verify NAT Gateway

az network nat gateway show --resource-group <RG_NAME> --name <NAT_GW_NAME>

Step 2: Check NAT Gateway public IP

az network nat gateway show --resource-group <RG_NAME> --name <NAT_GW_NAME> \
  --query "publicIpAddresses[].id" -o table

Step 3: Test outbound connectivity from a pod

kubectl run test-curl --image=curlimages/curl --restart=Never -n <NAMESPACE> -- \
  curl -s ifconfig.me
kubectl logs test-curl -n <NAMESPACE>
kubectl delete pod test-curl -n <NAMESPACE>

6. Resource Quota & Limits

Step-by-Step Commands

Step 1: Check all resource quotas in a namespace

kubectl describe resourcequota -n <NAMESPACE>

Step 2: Check quota usage across all namespaces

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  echo "=== $ns ==="
  kubectl describe resourcequota -n $ns 2>/dev/null
done

Step 3: Check LimitRange in a namespace

kubectl get limitrange -n <NAMESPACE>
kubectl describe limitrange -n <NAMESPACE>

Step 4: Calculate total memory limits for all deployments in a namespace

NAMESPACE="<NAMESPACE>"
kubectl get pods -n $NAMESPACE -o json | jq '[.items[].spec.containers[].resources.limits.memory // "0" | rtrimstr("Gi") | rtrimstr("Mi") | tonumber] | add'

Step 5: Check if deployments count quota is exceeded

kubectl get resourcequota -n <NAMESPACE> -o jsonpath='{.items[*].status}' | jq .

7. Helm / Deployment Failures

Diagnostic Checklist

Is the Helm release in a failed state?
Did the Helm chart values change?
Is there a version mismatch?
Did the deployment timeout?

Step-by-Step Commands

Step 1: Check Helm release status

helm list -n <NAMESPACE>
helm list -n <NAMESPACE< --failed

Step 2: Check Helm release history

helm history <RELEASE_NAME> -n <NAMESPACE>

Step 3: Check what changed in the last release

helm get values <RELEASE_NAME> -n <NAMESPACE>
helm get manifest <RELEASE_NAME> -n <NAMESPACE>

Step 4: Debug a failed Helm install/upgrade

helm upgrade --install <RELEASE_NAME> <CHART> -n <NAMESPACE> --debug --dry-run -f values.yaml

Step 5: Rollback a failed release

helm rollback <RELEASE_NAME> <REVISION> -n <NAMESPACE>

Step 6: Force delete a stuck release

helm uninstall <RELEASE_NAME> -n <NAMESPACE>
# If that fails:
kubectl delete secret -n <NAMESPACE> -l owner=helm,name<RELEASE_NAME>

Step 7: Check deployment rollout status

kubectl rollout status deployment/<DEPLOYMENT_NAME> -n <NAMESPACE>
kubectl rollout history deployment <DEPLOYMENT_NAME> -n <NAMESPACE>

Step 8: Rollback a deployment

kubectl rollout undo deployment/<DEPLOYMENT_NAME> -n <NAMESPACE>
# Or to a specific revision:
kubectl rollout undo deployment/<DEPLOYMENT_NAME> -n <NAMESPACE> --to-revision<REVISION>

8. Storage Issues

Diagnostic Checklist

Is the PVC bound?
Is the StorageClass available?
Is the PV provisioner healthy?
Are Azure Disks/Files quotas exhausted?

Step-by-Step Commands

Step 1: Check PVC status

kubectl get pvc -n <NAMESPACE>
kubectl describe pvc <PVC_NAME> -n <NAMESPACE>

Step 2: Check PV status

kubectl get pv
kubectl describe pv <PV_NAME>

Step 3: Check StorageClasses

kubectl get storageclass

Step 4: Check CSI driver pods (Azure)

kubectl get pods -n kube-system -l app=csi-azuredisk-node
kubectl get pods -n kube-system -l app=csi-azurefile-node

Step 5: Check CSI secrets store (for KeyVault)

kubectl get pods -n kube-system -l app=secrets-store-csi-driver
kubectl get secretproviderclass -n <NAMESPACE>

9. Certificate & Secret Issues

Diagnostic Checklist

Is the TLS certificate expired?
Is the SPN (Service Principal) credential expired?
Are KeyVault secrets synced?
Is the CSI SecretProviderClass configured correctly?

Step-by-Step Commands

Step 1: Check TLS secret

kubectl get secret <TLS_SECRET> -n <NAMESPACE>
kubectl get secret <TLS_SECRET>> -n <NAMESPACE> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -subject

Step 2: Check SPN expiry dates

# Check SPN in Azure AD
az ad sp credential list --id <SPN_APP_ID> -o table

Step 3: Check KeyVault CSI sync status

kubectl get secretproviderclasspodstatus -n <NAMESPACE>
kubectl describe secretproviderclass <SPC_NAME> -n <NAMESPACE>

Step 4: Check if KeyVault secrets are accessible

az keyvault secret list --vault-name <KEYVAULT_NAME> -o table
# e.g.: az keyvault secret list --vault-name <KEYVAULT_NAME> -o table

10. Monitoring & Observability

10.1 Metrics Server Issues

Step-by-Step Commands

Step 1: Check metrics server

kubectl get deployment metrics-server -n kube-system
kubectl top nodes  # If this fails, metrics server is down
kubectl top pods -n <NAMESPACE>

Step 2: Check metrics server logs

kubectl logs -n kube-system -l k8s-app=metrics-server --tail=100

11. Security — Twistlock / Pod Security

Check the twistlock security version.

Step-by-Step Commands

Step 1: Check Twistlock Defender

kubectl get ds -n twistlock
kubectl get pods -n twistlock -o wide

Step 2: Check Twistlock Defender logs

kubectl logs -n twistlock <DEFENDER_POD> --tail=200

Step 3: Check Pod Security Standards (PSS) violations

# Check namespace labels for PSS enforcement
kubectl get ns <NAMESPACE> -o jsonpath='{.metadata.labels}' | jq . | grep "pod-security"

Step 4: Check for PSS-blocked pods

kubectl get events --all-namespaces | grep -i "forbidden\|violat\|security"

12. HPA / KEDA Autoscaling Issues

Diagnostic Checklist

Is the HPA configured?
Is KEDA installed and running?
Are metrics available (CPU/custom)?
Is the target deployment correct?

Step-by-Step Commands

Step 1: Check HPA status

kubectl get hpa -n <NAMESPACE>
kubectl describe hpa <HPA_NAME> -n <NAMESPACE>

Step 2: Check current vs desired replicas

kubectl get hpa -n <NAMESPACE> -o custom-columns='NAME:.metadata.name,MIN:.spec.minReplicas,MAX:.spec.maxReplicas,CURRENT:.status.currentReplicas,DESIRED:.status.desiredReplicas,CPU_UTIL:.status.currentMetrics[0].resource.current.averageUtilization'

Step 3: Check KEDA operator

kubectl get pods -n keda
kubectl logs -n keda -l app=keda-operator --tail=200

Step 4: Check KEDA ScaledObjects

kubectl get scaledobjects -n <NAMESPACE>
kubectl describe scaledobject <SO_NAME> -n <NAMESPACE>

Step 5: Check if metrics are being reported (common HPA issue)

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/<NAMESPACE>/pods" | jq .

13. Redis Issues

Diagnostic Checklist

Is the Redis pod running?
Is Redis responding to PING?
Is Redis using too much memory?
Is Redis in read-only mode (maxmemory reached)?

Step-by-Step Commands

Step 1: Find Redis pods

kubectl get pods -n <NAMESPACE> | grep redis

Step 2: Check Redis pod logs

kubectl logs <REDIS_POD> -n <NAMESPACE> --tail=200

Step 3: Test Redis connectivity

kubectl exec -it <REDIS_POD> -n <NAMESPACE> -- redis-cli ping
# Response should be: PONG

Step 4: Check Redis memory usage

kubectl exec -it <REDIS_POD> -n <NAMESPACE> -- redis-cli info memory

Step 5: Check Redis configuration

kubectl exec -it <REDIS_POD> -n <NAMESPACE> -- redis-cli config get maxmemory
kubectl exec -it <REDIS_POD> -n <NAMESPACE> -- redis-cli config get maxmemory-policy

Your LRU-configured Redis instances use:

maxmemory 400mb
maxmemory-policy allkeys-lru

Step 6: Flush Redis cache (if needed, non-prod only)

kubectl exec -it <REDIS_POD> -n <NAMESPACE> -- redis-cli FLUSHALL

14. Azure-Specific Troubleshooting

AKS Cluster Health

# Check AKS cluster provisioning state
az aks show --resource-group <RG> --name <CLUSTER> --query "provisioningState" -o tsv

# Check AKS cluster power state
az aks show --resource-group <RG> --name <CLUSTER> --query "powerState" -o tsv

# Check node pool status
az aks nodepool list --resource-group <RG> --cluster-name <CLUSTER> -o table

# Check for Azure Advisor recommendations
az advisor recommendation list --resource-group <RG> -o table

Azure Resource Quotas

# Check VM quota usage in the region
az vm list-usage --location <AZURE_REGION> -o table | grep -i "Standard E"
az vm list-usage --location <AZURE_REGION> -o table | grep -i "Standard E"

AKS Upgrade Status

# Check available upgrades
az aks get-upgrades --resource-group <RG> --name <CLUSTER> -o table

# Check current version
az aks show --resource-group <RG> --name <CLUSTER> --query "kubernetesVersion" -o tsv

Azure Network Issues

# Check NSG rules
az network nsg list --resource-group <RG> -o table
az network nsg rule list --resource-group <RG> --nsg-name <NSG_NAME> -o table

# Check route table
az network route-table list --resource-group <RG> -o table

15. AWS EKS-Specific Troubleshooting

EKS Cluster Health

# Check cluster status
aws eks describe-cluster --name <CLUSTER> --query "cluster.status" --output text

# Check node groups
aws eks list-nodegroups --cluster-name <CLUSTER>
aws eks describe-nodegroup --cluster-name <CLUSTER> --nodegroup-name <NG_NAME>

EKS Auth Issues

# Check aws-auth configmap
kubectl get configmap aws-auth -n kube-system -o yaml

VPC CNI Issues

# Check VPC CNI pods
kubectl get pods -n kube-system -l k8s-app=aws-node
kubectl logs -n kube-system -l k8s-app=aws-node --tail=100

# Check available IPs
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.pods}{"\n"}{end}'

EFS/EBS Issues

# Check EFS CSI driver
kubectl get pods -n kube-system -l app=efs-csi-node

# Check EBS CSI driver
kubectl get pods -n kube-system -l app=ebs-csi-node

16. Emergency Procedures

16.1 Emergency Pod Restart (Single Deployment)

kubectl rollout restart deployment/<DEPLOYMENT_NAME> -n <NAMESPACE>

16.2 Emergency — Restart All Pods in a Namespace

# WARNING: This will restart ALL deployments in the namespace
kubectl get deploy -n <NAMESPACE> -o name | xargs -I{} kubectl rollout restart {} -n <NAMESPACE>

16.3 Emergency — Cordon a Problematic Node

# Prevent new pods from being scheduled
kubectl cordon <NODE_NAME>

# Move existing pods off the node
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data --force --grace-period=120

# After the issue is resolved
kubectl uncordon <NODE_NAME>

16.4 Emergency — Scale Down / Scale Up (Take Care in Production Environments)

# Scale down (e.g., to stop a misbehaving deployment)
kubectl scale deployment/<DEPLOYMENT_NAME> -n <NAMESPACE> --replicas=0

# Scale back up
kubectl scale deployment/<DEPLOYMENT_NAME> -n <NAMESPACE> --replicas=<DESIRED>

16.5 Emergency — Block Kured Reboots

# Block all kured reboots cluster-wide
kubectl annotate nodes --all kured/reboot-blocked="emergency-$(date +%Y%m%d)"

# Unblock when ready
kubectl annotate nodes --all kured/reboot-blocked-

16.6 Emergency — Delete All Evicted Pods

kubectl get pods --all-namespaces -o json | \
  jq -r '.items[] | select(.status.phase=="Failed" and .status.reason=="Evicted") | "\(.metadata.namespace) \(.metadata.name)"' | \
  while read ns pod; do kubectl delete pod $pod -n $ns; done

16.7 Emergency — Force Delete Stuck Namespace

# If a namespace is stuck in Terminating:
kubectl get namespace <NAMESPACE> -o json | \
  jq '.spec.finalizers = []' | \
  kubectl replace --raw "/api/v1/namespaces/<NAMESPACE>/finalize" -f -

Kubernetes Troubleshooting Guide - Step-by-Step, Enterprise Level

Table of Contents

Overview

1. Environment Overview

Pre-Requisites

2. Quick Reference — Cluster Access

3. Node Problems

3.1 Node Not Ready

Diagnostic Checklist

Step-by-Step Commands

3.2 Kured Reboots

Diagnostic Checklist

Step-by-Step Commands

3.3 Azure-Initiated Node Restarts

Diagnostic Checklist

Step-by-Step Commands

3.4 CPU Pressure / Memory Pressure

Diagnostic Checklist

Step-by-Step Commands

3.5 Disk Pressure

Diagnostic Checklist

Step-by-Step Commands

3.6 PID Pressure

Step-by-Step Commands

3.7 Node Scaling Issues (VMSS)

Step-by-Step Commands

4. Pod / Container Restarts

4.1 CrashLoopBackOff

Diagnostic Checklist

Step-by-Step Commands

4.2 OOMKilled

Diagnostic Checklist

Step-by-Step Commands

4.3 Pod Stuck in Pending

Diagnostic Checklist

Step-by-Step Commands

4.4 Pod Stuck in Terminating

Diagnostic Checklist

Step-by-Step Commands

4.5 ImagePullBackOff

Diagnostic Checklist

Step-by-Step Commands

4.6 Pod Evicted

Diagnostic Checklist

Step-by-Step Commands

4.7 Init Container Failures

Step-by-Step Commands

5. Networking Issues

5.1 Service Unreachable

Diagnostic Checklist

Step-by-Step Commands

5.2 Ingress / NGINX Issues

Diagnostic Checklist

Step-by-Step Commands

5.3 Istio Service Mesh Issues

Diagnostic Checklist

Step-by-Step Commands

5.4 DNS Resolution Failures

Step-by-Step Commands

5.5 NAT Gateway / External Connectivity

Step-by-Step Commands

6. Resource Quota & Limits

Step-by-Step Commands

7. Helm / Deployment Failures

Diagnostic Checklist

Step-by-Step Commands

8. Storage Issues

Diagnostic Checklist

Step-by-Step Commands

9. Certificate & Secret Issues

Diagnostic Checklist

Step-by-Step Commands

10. Monitoring & Observability

10.1 Metrics Server Issues

Step-by-Step Commands

11. Security — Twistlock / Pod Security

Step-by-Step Commands

12. HPA / KEDA Autoscaling Issues

Diagnostic Checklist

Step-by-Step Commands