20 DevOps/SRE Interview Questions That Survive Production

What Interviewers Actually Want to See

I've sat on both sides of the table for a few years now. Been the one interviewing, been the one getting grilled. One thing became obvious along the way: interviewers don't care about your definitions. Someone who rattles off a clean answer to "what is Kubernetes?" earns far less trust than someone who doesn't flinch at "your Pod is Running but you're getting 503s — where do you look first?" You can memorize the first question. Only people who've actually been woken up at 3am can answer the second.

In production, the order you think in matters more than the right answer. How do you split a problem into layers, where do you start checking, how do you narrow the cause while keeping user impact low. Knowing one correct line matters less than being able to throw out a bad assumption quickly. That's the skill that keeps people alive in ops.

So I put together 20 questions that come up over and over in real interviews. The answers aren't written for memorization — they follow the order an operator actually runs through in their head when something breaks.

Network and Traffic Troubleshooting

1. Pod is Running but you're getting 503s. How do you debug it?

A 503 usually doesn't mean your application died. It means something couldn't find a healthy backend. Start with the app and you'll waste time. Cut the request path into layers instead.

At the Pod level, kubectl get pod -o wide for status, then kubectl logs <pod> and kubectl describe pod <pod> for detail. This is where you catch failing readiness/liveness probes, or a mismatch between the port the app actually listens on and the container port. Next layer is the Service. kubectl get endpoints tells you whether the Service is picking up the Pod at all; check that the Service selector matches the Pod labels and that targetPort equals the real container port. Last is Ingress/Gateway — Ingress Controller logs, host/path/TLS config, backend Service wiring.

The most common cause by far: a readiness probe fails and the Pod quietly drops out of the Service Endpoints. The Pod still shows Running, looking perfectly fine, but it's not in the endpoints list. So when I see a 503, kubectl get endpoints is close to a reflex. The whole trick to 503 debugging is working backwards from how far the traffic actually got.

2. How does a CNI plugin work?

CNI (Container Network Interface) is the standard interface for attaching networking to a Pod. Kubernetes doesn't implement networking itself — it delegates that job to a CNI plugin. From Kubernetes' point of view, the network isn't something it handles. It's something it hands off.

The work starts with allocating a Pod IP and creating a veth pair wired into the Pod's network namespace. On top of that: routing so Pods can talk across Nodes, NetworkPolicy enforcement, and building the overlay or underlay network.

Calico is strong on BGP-based routing and NetworkPolicy. Cilium is eBPF-based, with better security and observability stories. Which one you pick comes down to what your environment actually needs.

The thing that matters in operations: when CNI wobbles, Pod-to-Pod traffic, DNS, Service traffic, and NetworkPolicy all go with it. CNI sits at the very bottom, so when it breaks, the symptoms you see upstream point somewhere else entirely. I've lost count of the tickets that came in as "DNS is broken" and turned out to be CNI.

Scheduling and Deployment Strategy

3. How Kubernetes scheduling works

The Kubernetes Scheduler runs in two phases: Filtering and Scoring. Filtering throws out Nodes the Pod can't land on — not enough CPU or memory, taint/toleration mismatch, nodeSelector or node affinity rules that don't match, PV zone constraints. Scoring then ranks whatever survived, looking at resource headroom, Pod spread policies, affinity preferences, and topology spread constraints.

If a Pod is stuck in Pending, start with the Events in kubectl describe pod. The mistake I see people make here is that they start guessing at the cause in their head. Don't guess. Read the event message — the scheduler already told you why it couldn't place the Pod. Whatever follows "0/5 nodes are available" is basically the whole answer.

4. Zero-downtime deployment for stateful applications

Swapping Pods doesn't give you zero downtime for a stateful app. You have to think about the application version, the data, the schema, and the replication topology together. Approach it with stateless instincts and the moment your data goes sideways, recovery gets a lot more painful.

The baseline: StatefulSet for ordered Pods and stable network identity, RollingUpdate to replace one at a time. Readiness probes to keep traffic off Pods that aren't ready, a replica/standby setup, and schemas designed to be backward compatible. A backup and a known rollback path before you deploy — that part isn't optional.

The key is separating application deployment from data change in your head. DB schema changes in particular need expand → deploy → contract. Add the column first (expand), ship the new code (deploy), then clean up the old column once every old-version instance is gone (contract). Trying to change schema and code in one shot and painting yourself out of a rollback path — I saw that happen plenty in financial services.

Debugging and Troubleshooting

5. Intermittent Pod restarts with no logs

No logs usually means the app died before it wrote anything, or it had no time to flush on the way down. An empty app log doesn't mean there are no clues. It means the clues are somewhere else.

Here's the order. kubectl describe pod for Events, then kubectl get pod -o yaml to check lastState. Was it OOMKilled or a liveness probe failure? What's the Node's CPU/Memory/DiskPressure situation? What's the container exit code? Then finally kubectl logs <pod> --previous to pull the previous container's logs.

Usual suspects: OOMKilled from blowing past a memory limit, a liveness probe timeout or wrong path, node resource pressure, application init failure, probe failures caused by a slow external dependency. It's also common to see a liveness probe tuned so aggressively that a perfectly healthy app gets killed while it's still initializing. When there are no logs, staring at the app alone gets you nowhere — you need Kubernetes events, container state, and node state side by side before the picture forms.

6. Debugging a latency spike every 60 seconds

Periodic latency means you should suspect a scheduled job or periodic resource usage first. Something spiking on a fixed interval means something is watching a clock.

Candidates: cron jobs and scheduled tasks, runtime GC (JVM and friends), DB checkpoint/vacuum/backup, log flush or rotate, metrics scraping, batch jobs and cache refresh, autoscaling metric collection intervals, external API rate limits. Debugging starts with overlaying the latency spikes against infrastructure metrics on the same time axis. Then find the slow span in a trace, dig through DB slow queries and GC logs, and line those up against cron/scheduler run times.

The "every 60 seconds" pattern is itself the strongest clue you have. Lay that period down as your baseline and check the application, DB, runtime, and infrastructure jobs against it one by one — the culprit tends to fall out faster than you'd expect. Back when I worked on monitoring products, a sawtooth graph like that was almost always a periodic job.

CI/CD and Build Pipelines

7. Cutting 50-image build time from 20 minutes to 5

The core idea is to stop rebuilding everything from scratch every time. Out of 50 images, usually one or two actually changed.

So: path-based change detection to build only what changed, parallel builds, Docker layer cache. Bring in BuildKit, Kaniko, or Buildx, standardize base images to raise cache hit rates, and split the dependency install step from the app copy step. Add a remote or registry cache, run tests and builds in parallel, and that's the package.

Most of the win comes from parallelization and caching. One caveat, though. Parallelize blindly and your runner costs and registry load go up with it. Builds get faster but the bill grows, or the registry becomes the bottleneck and you're slow again. So measure where the bottleneck actually is before touching anything. Parallelizing without measuring is just operating on a hunch.

8. Designing a secure CI/CD pipeline

Security isn't one scan right before deploy. It has to be spread across the whole pipeline. Put everything on one final gate and the day that gate falls, the entire defense goes with it.

Laying out the design pieces: secrets never live in code, images, or logs — they go in a dedicated store, and CI runner permissions get trimmed to the minimum. Branch protection and approval gates. Image vulnerability scanning, IaC scanning, SAST/DAST wired into the pipeline. Sign and verify artifacts, generate an SBOM, split deploy permissions from build permissions, keep audit logs.

What really matters in operations is whether you can later trace who deployed which code and which artifact, with what permissions, to which environment. Incidents happen, and the first question out of anyone's mouth is "who pushed this?" Your pipeline had better be holding the answer.

Infrastructure Management and High Availability

9. Designing a multi-region HA system

Multi-region isn't about deploying to several regions. It's about deciding, in advance, how you'll handle traffic and data consistency when something fails. Adding regions is easy. The hard part is the behavior when one of them dies.

Decide Active-Active vs Active-Passive. Decide where global load balancing or DNS-based routing lives. Pin down cross-region replication and RTO/RPO as actual numbers. Automate failure detection and failover, confirm each region can deploy independently, and check whether both regions are secretly hanging off the same single dependency. Runbooks and DR drills belong here too.

Active-Active gives you better availability but data consistency gets tricky. Active-Passive is structurally simpler, but then switchover time and recovery procedure become the question. Neither is free.

Which is why answering "I'd go multi-region" in an interview is a bit thin. The person who says "I weighed availability, cost, consistency, and operational complexity like this, so I'd pick this shape" — that person sounds like they've actually run a DR drill.

10. Managing Terraform drift

Terraform drift is when your code and the real cloud infrastructure disagree. It starts the moment somebody fixes one thing by hand in the console.

Remote backend with state locking. Changes only through PRs. terraform plan in CI. Restrict manual console changes, run drift detection on a schedule, split state per environment, and lock down who can read the state file.

The point is running Terraform as the source of truth for your infrastructure, not as a tool you dust off occasionally. Once the code stops describing reality, Terraform loses credibility and nobody trusts a plan output anymore. If a rushed console change was genuinely unavoidable, reflect it back into code or reconcile the state afterward. Put that off and the drift piles up until even running a plan feels dangerous.

Security and Observability

11. Secret management at scale

At scale, storing secrets safely isn't the end of it. Access, rotation, and auditing have to run alongside.

The principles are simple. No secrets in Git, images, or CI logs — use a central Secret Manager. Role-based access control, secrets separated per environment, rotation on a schedule. Access logs are auditable, credentials are short-lived, and an application only gets injected with the secrets it needs.

The goal of secret management isn't hiding things. It's controlling and tracking who accessed which secret and when. Once a secret is exposed, it's exposed. So rather than obsessing over driving exposure to zero, a structure that rotates and traces quickly after exposure holds up better in the real world.

12. Designing observability

Observability isn't piling up logs. It's designing signals so that when something breaks, you can narrow the cause fast. Pile up logs indiscriminately and you just grow the bill — then during an incident you have no idea what to look for in the heap.

Three axes. Metrics show system state and performance trends. Logs capture individual events and error detail. Traces stitch together request flow across services. On the operations side, add a standard log format and correlation IDs, RED metrics per service (Rate, Errors, Duration), USE metrics per infrastructure component (Utilization, Saturation, Errors), plus dashboards and alert thresholds.

Correlation IDs especially are close to a lifeline in MSA environments. A single request passes through ten services, and if the ID doesn't travel with it, tracing where things broke during an incident becomes effectively impossible. Good observability isn't a structure for digging through logs after the incident. It's a structure that narrows down the starting point while the incident is still happening.

Incident Response and SRE Practice

13. SLO-based alerting without the noise

A good alert isn't a loud alert. It's an actionable one. When alerts fire too often, people start ignoring them. That's the genuinely dangerous state.

Design starts from user-facing SLIs — success rate, latency, availability. Attach SLOs to them (say, 99.9% request success rate, p95 latency under 300ms) and set an Error Budget. Alerts fire when that Error Budget is burning fast; internal metrics get split off into secondary alerts.

CPU at 90% may not be an incident by itself. Users don't know what the CPU number is, and they don't need to. But if user request failure rate climbs, or latency starts threatening the SLO, that's an alert worth waking someone for. That's why you anchor alert thresholds to user experience instead of infrastructure metrics.

14. First five steps of a production incident

The first goal in an incident isn't a perfect root cause analysis. It's minimizing user impact. You can dig into the cause slowly afterward — but the outage users are experiencing is still happening right now.

1. Determine blast radius: everything, part of it, or one region
2. Declare the incident and set up comms: split the responder, the decision maker, and the communicator
3. Check recent changes: deploys, config, infrastructure, external dependencies
4. Mitigate immediately: rollback, traffic shift, feature flag off, scale out, circuit breaker
5. Keep a timeline: when you checked what, and what action you took

I want to underline number 3. Most incidents start wherever something just changed. Right after a deploy, right after a config change. The people who calmly ask "what changed recently?" first are the ones who recover fastest. What you actually need during an incident isn't heroic debugging — it's calm and structure. Root cause matters, but while users are still hurting, recovery and mitigation come first.

15. Designing for graceful degradation

Graceful degradation means designing so a partial failure doesn't take the whole service down with it. If one thing slips and the entire service goes dark, your design is amplifying failures.

Recommendation API dies, serve a default recommendation list. Payment add-ons die, keep core payments alive. External API unreachable, ride on cached data. Search dies, show popular items or recent data. Cut non-critical features with feature flags, stop failure propagation with circuit breakers, and set timeout and retry policies.

Watch out for retries, though — crank them up and you'll make the outage worse. Dumping retry traffic on a dying service robs it of any chance to recover. It's called a retry storm, and once you've lived through one you don't forget it. Which is why retries come as a set with timeout, backoff, circuit breaker, and bulkhead patterns.

Cost and Upgrade Management

16. Responding to a 3x AWS cost spike

Break the increase down by service, region, account, and tag first. "Costs went up" as one lump gets you nowhere. Split it and the culprit shows up.

Look at per-service increases in Cost Explorer, then per-region increases and recently created resources. Pull out Auto Scaling, NAT Gateway, data transfer, and log storage costs separately. Check for abnormal traffic or failed batch jobs, untagged resources drifting around, and whether budget alerts and Cost Anomaly Detection are even turned on. Frequent causes: Auto Scaling misbehaving, massive log ingestion, NAT Gateway and Data Transfer blowing up, wrong instance types, test resources nobody terminated, batch job retry loops.

I treat cost as a category of incident in operations. The difference from a normal incident is that without alerts, you don't find out until the bill arrives. Organizations with no cost alerts, budget policy, tagging policy, or resource TTL policy always find out a beat late. One retry loop quietly driving up NAT Gateway costs, discovered a month later.

17. Zero-downtime Kubernetes upgrades

A Kubernetes upgrade means looking at the control plane, the nodes, and application availability all at once. Upgrade while watching only one layer and another one always blows up.

1. Check current-to-target version compatibility and API deprecations
2. Check add-on compatibility (CNI, CSI, Ingress Controller, monitoring agents)
3. Upgrade the control plane
4. Cordon/drain worker nodes one at a time
5. Uncordon after upgrading each node
6. Guarantee minimum availability with PodDisruptionBudget
7. Only send traffic to ready Pods via readiness probes
8. Smoke test the critical services

Step 2 especially: miss an API deprecation and manifests that worked fine yesterday suddenly stop applying after the upgrade. And PDB in step 6 cuts both ways. Set it wrong and a drain hangs forever; set it too loose and availability breaks mid-upgrade. Anyone who's rolled nodes one by one in an OKD environment knows that balance is a more delicate job than it sounds.

The Mindset That Actually Works

A good answer in a DevOps/SRE interview isn't the one with the most commands in it. It's the one that splits the problem into layers, judges user impact first, and looks at recent changes alongside operational risk.

The mindset that survives production isn't complicated either. Evidence before guesses. Mitigation before root cause. Look at the whole request path instead of a single component. Put deploys, infrastructure, network, permissions, and data changes on one screen together. And once the incident is over, build the prevention into the design.

This interview isn't a knowledge test. It's a test of how you think in operations. Someone who's taken a few real outages square in the face reveals their ordering, their reasoning, and their risk control without trying. Two or three sentences is usually enough to tell who memorized it and who lived it.

Where do you look first when something breaks, what do you handle first, and on what basis do you decide. Writing code, drawing architecture, owning operations — the question production asks has always been the same. And the person who answers it with their whole body is the one you can trust at 3am.