Kubernetes skill shows up in how you explain an outage, not in how many kubectl commands you know

I sat on the interview side of the table and asked this one more times than I can count.

"What actually happens when you run kubectl delete pod?"

The answer is usually short.

"The pod gets deleted."

Not wrong. Just useless from an operations standpoint. And I've watched people who've typed that command thousands of times split right here.

What does the API Server change when it receives that request? When does deletionTimestamp land on the Pod? How does kubelet notice? When does the preStop hook fire, and which signal reaches the container process? What happens once terminationGracePeriodSeconds runs out? If the Pod belonged to a Deployment, who creates the replacement?

Start narrating that sequence out loud and it stops being about command syntax. It's about whether you understand how Kubernetes works.

That's also what the job needs. Not memorized flags — the ability to sketch what happens, why, in what order, and how it fails.

This is where "I've run production Kubernetes" separates from "I've used Kubernetes."

1. What happens when a node runs low on memory?

The common answer:

"Memory runs out, pods die."

Outcome-wise, fine. What you need in operations is who evicts, by what criteria, and which pod goes first.

When a node comes under memory pressure, kubelet evicts pods according to its eviction policy. This is where QoS enters: BestEffort, Burstable, Guaranteed.

I spent a while repeating "BestEffort dies first, then down the list." That's not accurate. Eviction candidate selection also factors in how far a pod has exceeded its own requests, and what its Priority is. QoS is closer to a result of that calculation — the causes sitting behind it are requests/limits and Priority.

Go one step further and the question becomes:

"What should you be watching before MemoryPressure fires?"

That's the point where an interview question turns into an operations question. You want node MemoryPressure, working set, available memory, per-pod usage, and OOMKilled occurrences all in one view.

One more. I keep running into the belief that a PDB also protects against resource-pressure eviction. It doesn't. A PDB guards availability during voluntary disruptions like a drain.

2. You've got a CrashLoopBackOff. Where do you start?

Beginners lead with commands. kubectl logs, kubectl describe pod, kubectl get events. All of them useful. What separates people in practice isn't how many commands they know — it's the order they investigate in.

Cut the failure domain first.

Step 1. Did the container even start?

Check image pull failures, Secret problems, volume mount failures, config errors.

Step 2. It started, but the process exits?

Look at Last State, Reason, and Exit Code in kubectl describe pod <pod>.

Step 3. Do you need the previous container's logs?

For CrashLoopBackOff this matters a lot.

kubectl logs <pod> --previous

I've stared at the current container's logs and missed the clue from the crash right before it. Burned about 30 minutes that way, at 3am.

Step 4. Is a probe killing the container?

The application is perfectly healthy, but a badly tuned livenessProbe has kubelet restarting the container over and over.

Step 5. Are the init containers OK?

If an init container never finishes, you never reach the app container stage at all.

Good incident analysis isn't about knowing a lot of tools. It's stacking possible causes into layers and crossing them off one by one.

3. Draining a live node safely

The commands are trivial.

kubectl cordon node-47
kubectl drain node-47

In production there's a question you answer before those two lines.

"If I pull the pods off this node, is the service still up?"

If your API pods sit one each on nodes A, B, and C, three replicas looks safe at a glance. Drain multiple nodes at once and that story changes.

So first: is the replica count actually sufficient, is a PDB in place, are pods piled onto one node? Also whether StatefulSets or PV-backed workloads are mixed in, how you'll handle DaemonSets, whether any pod is holding data in an emptyDir. If the node carries control plane or critical infrastructure components, you're doing an entirely different job at that point.

Distinguish cordon from drain, too. Cordon stops new regular pods from being scheduled onto the node; drain evicts the existing workload pods so you can do maintenance.

Running the drain command is the easy part. Getting the cluster into a state where draining is safe is much harder.

4. Why rolling updates fail

Ask someone to explain RollingUpdate and you usually get two values: maxSurge and maxUnavailable. Stop there and you know the settings.

The operational question is different.

"If something breaks during a RollingUpdate, where does it break?"

Readiness probe failures are the classic. The new pod is up but never goes Ready. It gets no Service traffic and the rollout stops moving forward.

Liveness probes cause trouble just as often. The app takes a while to initialize, the probe is set aggressively, and the container restarts forever. This is where you need to look at startupProbe as well.

Set maxUnavailable high and deploy, and existing pods drop out en masse — capacity falls through the floor for a moment.

The nastiest case is when the application is alive but the service is broken. Use a probe that just returns HTTP 200 and the pod looks perfectly healthy to Kubernetes even with the DB or an external dependency severed.

A successful Deployment is not a healthy service. That's exactly why rolling updates need observability attached — metrics, logs, traces.

5. DaemonSet vs. Deployment

Textbook version, easy:

Deployment → run as many pods as the desired replica count DaemonSet → run a pod on every targeted node

What matters is why the two exist separately.

Deployments usually run applications. API servers, web apps, backends, workers. DaemonSets are for things that attach to the node itself. Log collectors, monitoring agents, node exporters, CNI components, security agents.

Which means understanding DaemonSets properly drags in taints/tolerations, nodeSelector, node affinity, hostPath, hostNetwork.

The difference between them isn't one kind: line in a YAML. It's an architectural call about what unit you place a workload on.

6. How Kubernetes networking works

This question hits harder than it looks. A few sentences in, you can tell whether someone knows it at the YAML level or down into the network.

The broad path:

External user → Load Balancer / Ingress → Service → Pod

On the pod network, each pod gets an IP and the CNI owns that setup. The Kubernetes network model starts from the premise that pod-to-pod communication happens without NAT.

The service network starts from a different fact: pod IPs change whenever. You put one stable access point in front, and it runs Service → EndpointSlice → Pod. Depending on the environment, traffic forwarding goes through kube-proxy's iptables or IPVS mode; some newer CNIs replace that layer with eBPF.

Ingress is HTTP/HTTPS routing rules. I've seen plenty of people create the Ingress resource and then ask why no traffic shows up — the thing doing the actual work is the Ingress Controller.

During real incidents you go further down.

DNS → Service → EndpointSlice → CNI → Route → MTU → NIC

MTU hurts especially on overlay networks. Encapsulation like VXLAN or Geneve adds headers to the packet. When the MTU is off, the symptom looks like this:

"Communication isn't fully broken — only certain requests fail, in weird ways."

Those are the incidents that hold people hostage the longest.

7. The service times out sometimes

The most dangerous move is deciding on a cause before you've checked anything.

"Looks like a network problem."

Said while having looked at nothing. I've led with that line myself and lost half a day in the wrong place.

Split the failure domain first.

Client → Ingress/LB → Service → Pod → Application → DB/External API

Then start asking. Does it only happen on specific pods — compare per-pod latency, errors, restarts. Only when going through the Service — check the EndpointSlice and the Service path, and if needed hit the pod IP directly to isolate the route. Only on requests of a certain size — that's where you suspect MTU. Slow while CPU utilization stays low — look at CPU limit throttling. Behind the application — DB connection pool, external APIs, thread pool.

Tracing makes this far faster. Request 10ms → Ingress 5ms → Application 20ms → Database 2,300ms. The moment that line is in front of you, most of your reasons to suspect Kubernetes evaporate.

The people who last in operations weren't the ones who guessed well. They were the ones who narrowed the failure domain fast.

8. Why requests and limits are separate

Simply put:

Requests = the basis for scheduling Limits = the ceiling during execution

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "1"
    memory: "1Gi"

The scheduler looks for a node that can at least accommodate the requests. After that, CPU and memory limits behave completely differently.

Exceed the CPU limit: CPU throttling Exceed the memory limit: OOM → container terminated

Don't look at a CPU graph and conclude "it's not even at 100%, why is it slow?" You have to look at CFS throttling metrics alongside it.

Java is more sensitive on the memory limit side. A container memory limit doesn't just cover the JVM heap — metaspace, direct memory, thread stacks, and native memory live in there too. I've watched teams size the limit from heap alone and eat an OOMKilled. Including limits I set that way myself.

9. How do you design for HA?

"I'll run three replicas."

Fine as a starting point, not enough to call it an HA design. If all three replicas land on the same node, one node A failure takes out all of them.

Split the failure domains. One each in AZ-A, AZ-B, AZ-C. Wire in topologySpreadConstraints, pod anti-affinity, PodDisruptionBudget, readiness/liveness/startup probes, requests/limits, and a sane replica count.

And that's still not the end. The application has to tolerate failure: timeouts, retries, circuit breakers, connection pools, graceful degradation. So does the operational side: metrics, logs, tracing, alerts, runbooks, incident response.

HA isn't a system that never dies. It's a system where the service keeps serving when parts of it die, and where you find and revive the dead parts fast.

10. How do you upgrade a production cluster?

This is the question that exposes actual operational experience.

On paper you're moving 1.x to 1.y. In practice it isn't a version-number edit.

Start with compatibility. Any deprecated APIs? Do your operators, CNI/CSI, ingress, and monitoring components support the new version? Validate in dev or a separate environment first. Run your critical workloads, networking, and storage all the way through.

Backups aren't a yes/no checkbox. You verify that etcd and application data actually restore. I've sat in plenty of rooms where "we have backups" was said without anyone having tested that.

Move in stages. Handle the control plane and workers in order, following the platform's official upgrade procedure and version skew policy.

One thing that's easy to miss: the upgrade path differs per distribution. Don't lump upstream Kubernetes, OpenShift/OKD, EKS, AKS, and GKE into one procedure. On OpenShift-family clusters, platform operators like the Cluster Version Operator and MachineConfigOperator are deeply involved in the upgrade.

It's not about memorizing commands. It's knowing how your platform upgrades the control plane, the nodes, and the operators, each in its own way. That's what the job needs.

So what is Kubernetes skill, really?

After a few years of running it, my take is that Kubernetes itself isn't hard. What's hard is that failures cut through multiple layers at once.

Say a user hit a timeout. The cause may not be Kubernetes at all.

User timeout → Ingress? → Service? → CNI? → Pod CPU throttling? → JVM GC? → DB connection pool? → Database?

The symptom is one line: "it's slow." The causes number in the dozens.

What accumulates with operational experience isn't a longer list of memorized commands — it's a mental map of the system's cause and effect. I think that's the biggest thing separating levels of Kubernetes skill.

Beginners search for a command when something breaks. Experienced people form a hypothesis first and cut the failure domain.

Symptom → which layer? → under what conditions does it reproduce? → what's different from normal? → eliminate variables one at a time → root cause

Running Kubernetes isn't about being good at kubectl. It's about understanding distributed-system failure structurally. Good interview questions follow the same rule: not "do you know this command," but "why does this system behave this way, and where would you look first when it breaks?"

That answer is where "used Kubernetes" splits from "operated Kubernetes." Honestly, I don't always answer it cleanly either. Still don't.