My first real Kubernetes incident was a service that kept restarting every few minutes with no useful error. The logs ended mid-request. Exit code 137. I spent two hours reading application code before someone pointed at the memory limit in the deployment manifest — 256Mi, set by whoever copied the template, on a service that needed about 400.
The container was being killed by the kernel and Kubernetes was dutifully restarting it, exactly as configured. Nothing was broken. The system was doing precisely what the YAML said.
That is the shift Kubernetes asks you to make, and it is the reason it feels opaque at first. You stop telling a machine what to do and start describing a desired state, then let a control loop chase it. Once that clicks, most of the confusion clears. Here is the mental model I wish someone had given me on day one.
The Core Idea: A Loop That Chases a Description
You submit a description — "I want three copies of this container running, each with these resources, reachable at this name." Kubernetes stores it, and a set of controllers continuously compare reality against that description and act on the difference.
Pod died? The loop notices the count is two, starts another. Node disappeared? The loop reschedules its pods elsewhere. You changed the image tag? The loop replaces pods one at a time to match.
Everything else is detail on top of this. Which is also why Kubernetes will confidently and repeatedly do the wrong thing if you describe the wrong thing — as my 256Mi memory limit demonstrated. It is not trying to keep your service healthy. It is trying to make reality match your YAML.
The Pieces You Actually Need
Kubernetes has a large surface area. You can run production on a small subset of it.
Pod — one or more containers that share a network namespace and are scheduled together. Almost always one container. You rarely create pods directly.
Deployment — "keep N pods of this specification running, and roll them over safely when the specification changes." This is what you actually write for stateless services, and it is where rolling updates come from.
Service — a stable name and virtual IP in front of a changing set of pods. Pods come and go with different IPs; the service is the fixed address other things talk to. It load balances across whichever pods are currently ready.
Ingress — routes outside HTTP traffic to services, handles hostnames, paths and TLS. One entry point for the cluster rather than a load balancer per service.
ConfigMap and Secret — configuration and credentials injected as environment variables or files, so the same image runs in every environment. Note that Secrets are base64-encoded, not encrypted, unless you have enabled encryption at rest and locked down access. Many people learn this the wrong way.
Namespace — a grouping boundary for organisation, quotas and access control.
That is genuinely enough to run real services. StatefulSets, DaemonSets, Jobs, CronJobs and the rest have their uses, but you can learn them when you meet the problem they solve.
What the Control Plane Is Doing
Worth knowing at a high level, because it explains failure modes.
The API server is the only thing anything talks to — kubectl, controllers, nodes, all of it. Every change goes through it. etcd is the key-value store holding cluster state; it is the source of truth and the thing you must back up. The scheduler decides which node a new pod lands on based on resource requests and constraints. Controllers run the reconciliation loops. On each worker node, the kubelet makes sure the containers it was assigned are actually running and reports back.
The practical consequence: when people say "the cluster is down," they usually mean the API server is unreachable. Running workloads generally keep running — the kubelet on each node continues doing its job. You just cannot change anything.
A Deployment That Will Not Embarrass You
The manifest most tutorials show is missing the four things that matter most in production.
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
spec:
replicas: 3
selector:
matchLabels: { app: orders-api }
template:
metadata:
labels: { app: orders-api }
spec:
containers:
- name: app
image: registry.example.com/orders-api:1.8.3 # never :latest
ports:
- containerPort: 3000
resources:
requests: { cpu: "100m", memory: "256Mi" } # scheduling
limits: { cpu: "500m", memory: "512Mi" } # hard ceiling
readinessProbe: # ready for traffic?
httpGet: { path: /health/ready, port: 3000 }
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe: # needs a restart?
httpGet: { path: /health/live, port: 3000 }
periodSeconds: 20
failureThreshold: 3
securityContext:
runAsNonRoot: true
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
Resource requests and limits. Requests are what the scheduler reserves; limits are the hard ceiling. No request means the scheduler is guessing and will overpack nodes. Too low a memory limit means the kernel kills your container — my exit code 137.
Two different probes. This distinction is the one people get wrong most. Readiness answers "should this pod receive traffic right now?" — if it fails, the pod is removed from the service but left alone. Liveness answers "is this process wedged?" — if it fails, the pod is restarted. Point liveness at a check that only fails when a restart would genuinely help, and never at a dependency. A liveness probe that checks the database will restart every pod in your cluster during a database blip, turning a small outage into a large one.
A pinned image tag. :latest means you cannot say what is running and cannot roll back to a specific thing.
A security context. Running as root inside a container is the default and it should not be yours.
The Failures You Will Meet First
CrashLoopBackOff — the container starts and exits repeatedly. Check kubectl logs pod --previous for the crashed instance rather than the new one. Usually a missing environment variable, a failed dependency at startup, or the memory limit.
ImagePullBackOff — the wrong tag, the wrong registry, or missing pull credentials.
Pending forever — no node has room for the resource requests, or a volume cannot be attached. kubectl describe pod tells you which, in the events at the bottom. Read the events; they are where the answers are.
Intermittent 502s during deploys — almost always missing graceful shutdown. Kubernetes sends SIGTERM and removes the pod from the service, but those happen concurrently, so in-flight requests can arrive at a pod that has started shutting down. Handle SIGTERM, stop accepting new work, finish what is in progress, then exit. Add a few seconds of delay before you stop accepting, so the endpoint removal has propagated.
Do You Actually Need It?
I say this every time and I will say it here. If you run three or four services with modest traffic, a managed container platform — Cloud Run, ECS Fargate, App Service — gives you containers, autoscaling, health checks and rolling deploys without a control plane to operate. The complexity you skip is real complexity.
Kubernetes starts earning its keep when you have enough services that bin-packing them onto machines is a real problem, multiple teams deploying independently, on-premise or multi-environment requirements, or an ecosystem need — operators, service mesh, GitOps — that genuinely applies to you. And when someone can own the platform, because it is not zero-maintenance.
If you do run it, use managed control planes. EKS, GKE, AKS. Nobody should be running etcd themselves in 2026 unless they have a specific reason and a specific person.
How to Learn It Properly
Run a local cluster — kind or minikube — and deploy something small you wrote yourself. Then break it on purpose. Set the memory limit too low and watch the restarts. Point the readiness probe at a path that returns 404 and see traffic stop. Delete a pod and watch it come back. Scale to five and back to one.
Kubernetes stops being mysterious at the point where you can predict what it will do before you run the command. Fifteen deliberate failures on a laptop gets you there much faster than reading the documentation cover to cover — and considerably faster than learning it during an incident at 2am, which is how I did it.
