Kubernetes doesn't recover from failures — it executes the failures you defined

A manifest with readinessProbe and livenessProbe pointing at the same /health. If you run Kubernetes, you've seen this. It's where most incidents start.

Almost nobody adopts Kubernetes without self-healing on the list. Broken container? Restart it. Unhealthy Pod? Pull it out of the traffic pool. New version? Swap old Pods for new ones in order. So a certain expectation forms on its own:

"Kubernetes will figure out whether my app is healthy and fix it."

It won't. Kubernetes has no idea whether your application is actually fine. It reads the result of a Probe that you configured and performs a predetermined action. Define the problem wrong and it will execute your wrong judgment very fast and very precisely.

The part that matters in self-healing isn't the automation. It's what you decided counts as a failure.

The three questions Kubernetes asks your app

Kubernetes has three Probes for checking container state.

Probe	What it checks	On failure
Startup Probe	Has the app finished starting?	Restart the container after N failures
Readiness Probe	Can it take requests right now?	Remove from Service endpoints
Liveness Probe	Is a restart the only way back?	Restart the container after N failures

All three check state, but they're asking completely different questions. Think of a hospital. Startup asks whether the doors are ready to open. Readiness asks whether it can take a new patient right now. Liveness asks whether the whole medical system has seized up and needs a power cycle. You don't cut power to the building because intake is running a little late. You don't shut the hospital down because too many patients showed up at once. Same in Kubernetes.

The most common mistake: one /health for everything

In the field, one health URL usually gets shared between Readiness and Liveness.

readinessProbe:
  httpGet:
    path: /health
    port: 8080
livenessProbe:
  httpGet:
    path: /health
    port: 8080

Simple config. The question is what /health actually checks. If that URL checks the app process plus the database, Redis, Kafka, an external payment API, and an external auth API all at once, then the moment that payment API gets slow, /health fails.

Feed that result to Liveness and Kubernetes restarts the container. Restarting a container has never fixed an outage at someone else's API, but it restarts anyway. The problem is outside the cluster, and the only thing getting bounced is a perfectly healthy container, over and over.

How a small DB hiccup becomes a full outage

Say every Pod talks to the same database. The database gets slow for a moment. And Liveness is configured to check the DB connection.

Here's the collapse, in order. DB latency rises, every Pod's Liveness Probe fails, kubelet restarts the containers. All those restarted apps hit the DB for connections at the same time. Retries pile in from clients and from other services, so load on both the DB and the apps climbs again. Pods that were mid-recovery fail again. Then CrashLoopBackOff, then cascading failure.

The original problem was a short spike in DB response time. One bad Liveness setting turned that into a service-wide outage by restarting every application that was working fine.

Kubernetes didn't decide any of this. You told it:

"When the DB gets slow, restart every application container."

And it carried out that policy faithfully.

How to split Readiness from Liveness

Readiness answers whether it's okay to send this Pod a new request. Fail it and the Pod drops out of the Service's normal traffic pool. The container itself isn't restarted. Liveness answers whether keeping this process alive is still worth anything. Fail it enough times and kubelet restarts the container. Different questions, so different failure conditions. That's the whole idea.

Readiness is for: required initialization hasn't finished, cache or config is still warming, the app can't properly handle a request right now. Shutting down and refusing new requests belongs here too. If dropping out of traffic for a moment is better for the service overall, that's Readiness territory.

Liveness is a different animal. Main thread or event loop wedged. A critical internal worker thread dead. Process is up but request handling has stopped completely. Only states where the app can't recover on its own and a container restart plausibly fixes it.

The test is one line.

Does restarting the container actually fix this?

If the answer is fuzzy, keep it out of your Liveness conditions. Honestly, I'm not a fan of the habit of adding Liveness by reflex. You're wiring a restart button to problems that restarts don't solve.

Example 1: an ordinary web API server

For a typical web API, split the three Probes into separate endpoints by purpose.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-api
  template:
    metadata:
      labels:
        app: order-api
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: order-api
          image: example.com/order-api:1.0.0
          ports:
            - name: http
              containerPort: 8080
          startupProbe:
            httpGet:
              path: /health/startup
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 24
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /health/liveness
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3

Each endpoint answers a different question. /health/startup reports whether initial config and required data loading finished. /health/readiness reports whether the app can handle a new request right now. /health/liveness reports whether the app's internal execution flow is still alive.

Startup Probe

app.get("/health/startup", (req, res) => {
  if (initializationCompleted) {
    return res.status(200).json({ status: "STARTED" });
  }
  return res.status(503).json({ status: "STARTING" });
});

The example allows up to 24 failures at 5-second intervals.

5초 × 24회 = 약 120초

The actual restart moment shifts a bit depending on Probe execution time and system conditions. Read it as roughly two minutes of startup budget. Size it against the worst-case healthy start time, including under heavy load. Use the average and it won't be enough.

Readiness Probe

app.get("/health/readiness", (req, res) => {
  if (!acceptingTraffic) {
    return res.status(503).json({
      status: "NOT_READY"
    });
  }
  return res.status(200).json({
    status: "READY"
  });
});

A Readiness failure pulls the Pod out of new traffic instead of restarting the process.

Be careful about putting DB or external-system health into Readiness too. If every Pod checks the same dependency and they all go NotReady together, the service's entire endpoint list disappears at once. Before adding a shared dependency, ask yourself:

Without this dependency, can I really not serve a single request?

If a read cache or some secondary feature being down still leaves you able to serve a reduced set of requests, per-feature error handling or a degraded mode beats failing Readiness for the whole Pod.

Liveness Probe

app.get("/health/liveness", (req, res) => {
  if (eventLoopStalled || criticalWorkerStopped) {
    return res.status(503).json({
      status: "UNHEALTHY"
    });
  }
  return res.status(200).json({
    status: "ALIVE"
  });
});

Keep Liveness to the app's own internal liveness as much as you can. This kind of thing is dangerous:

app.get("/health/liveness", async (req, res) => {
  const databaseUp = await checkDatabase();
  const redisUp = await checkRedis();
  const paymentApiUp = await checkPaymentApi();
  if (!databaseUp || !redisUp || !paymentApiUp) {
    return res.sendStatus(503);
  }
  return res.sendStatus(200);
});

Because now a stalled DB or payment API restarts your application container, repeatedly.

Example 2: a Spring Boot app with a long startup

Spring Boot apps are slow to start for a fairly predictable set of reasons. Class and Bean initialization, database connections, loading a pile of configuration, cache setup, security module initialization, connecting to external systems. Configure only a Liveness Probe on a service like this and the app gets restarted before it ever finishes starting.

Spring Boot Actuator exposes health endpoints you can wire straight into Kubernetes Probes.

management:
  endpoint:
    health:
      probes:
        enabled: true
  endpoints:
    web:
      exposure:
        include: health,info

The two you want are /actuator/health/liveness and /actuator/health/readiness. On the Kubernetes side:

containers:
  - name: payment-api
    image: example.com/payment-api:2.1.0
    ports:
      - name: http
        containerPort: 8080
    startupProbe:
      httpGet:
        path: /actuator/health/liveness
        port: http
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 36
    readinessProbe:
      httpGet:
        path: /actuator/health/readiness
        port: http
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    livenessProbe:
      httpGet:
        path: /actuator/health/liveness
        port: http
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3

This one allows about 180 seconds to start.

5초 × 36회 = 약 180초

Readiness and Liveness checks don't even begin until the Startup Probe succeeds. That ordering is what keeps a normally-initializing app from being restarted too early.

Why you shouldn't point Probes at the full health endpoint

/actuator/health

Depending on configuration, this endpoint bundles the database, Redis, Kafka, message brokers, disk space, and external storage together. Wire that aggregate into Liveness and one external system's outage becomes an application restart. Open up the Liveness and Readiness groups and look at exactly which health indicators are in them before you deploy.

Example 3: services with long initialization, like AI models

An AI inference server still has work to do after the process comes up. GPU initialization, loading model files, validating the model, allocating memory, warm-up inference. Only then can it accept requests. Send it traffic just because the process is running and the first request either fails or crawls.

containers:
  - name: inference-api
    image: example.com/inference-api:3.0.0
    ports:
      - name: http
        containerPort: 8080
    startupProbe:
      httpGet:
        path: /health/model-loaded
        port: http
      periodSeconds: 10
      timeoutSeconds: 3
      failureThreshold: 60
    readinessProbe:
      httpGet:
        path: /health/ready
        port: http
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    livenessProbe:
      httpGet:
        path: /health/live
        port: http
      periodSeconds: 15
      timeoutSeconds: 2
      failureThreshold: 4

This gives model initialization about ten minutes.

10초 × 60회 = 약 600초

The endpoints split by role too. /health/model-loaded reports whether GPU init, model loading, and warm-up are done. /health/ready reports whether the server can take a new inference request. /health/live reports whether the inference engine and required worker threads are alive.

Think twice before dropping conditions like GPU above 90%, CPU above 90%, or memory above 90% into Readiness. When one overloaded Pod pulls itself out of traffic, its requests land on the rest. The remaining Pods overload in turn, and you cascade your way to zero available Pods.

High load isn't a problem you solve with a Probe. Design it alongside request queues, concurrency limits, rate limiting, HPA, request timeouts, circuit breakers, and gradual traffic ramp-up.

Why deploys look faster with no Probes

Remove the Probes and Pods appear to go Ready much faster. Kubernetes has no signal to check, so it treats "container started" as healthy. That has nothing to do with whether the app is ready.

During a rolling update, the moment the new container process starts, Kubernetes marks that Pod Ready and begins scaling down the old ones. Traffic goes to the new Pod. The app hasn't finished initializing, so requests fail. You skipped the step that verifies readiness, so of course it looks faster.

It cuts the other way too. A Deployment won't replace the next old Pod until the new one is available. If your Probe period is too long, the app is ready but Kubernetes sits there waiting for the next Probe to run. With a lot of Pods, that wait accumulates across the entire rolling update.

One Probe setting touches when traffic arrives, how fast rolling updates go, how many Pods stay available, when containers restart, and how far a failure spreads. The word "health check" makes it sound minor. It's a control signal holding both deploy speed and incident behavior at the same time.

A production setup that accounts for zero-downtime deploys

Zero-downtime deploys don't end with good Probes. A terminating Pod also needs time to stop accepting new requests and finish the ones already in flight.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  minReadySeconds: 10
  selector:
    matchLabels:
      app: order-api
  template:
    metadata:
      labels:
        app: order-api
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: order-api
          image: example.com/order-api:1.0.0
          ports:
            - name: http
              containerPort: 8080
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - sleep 5
          startupProbe:
            httpGet:
              path: /health/startup
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 24
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health/liveness
              port: http
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3

What each setting is doing:

설정	목적
maxUnavailable: 0	Available Pod count never drops below the pre-deploy level
maxSurge: 1	Add one new Pod at a time while replacing
minReadySeconds: 10	A Pod that just went Ready isn't immediately treated as stable
preStop	Buys time for traffic changes to propagate and for cleanup before termination
terminationGracePeriodSeconds	Gives in-flight requests time to finish
Startup Probe	Waits until initialization completes
Readiness Probe	Sends traffic only to ready Pods
Liveness Probe	Restarts only containers that can't recover themselves

The sleep 5 in preStop is a basic example, chosen because it's easy to follow. In real operations, what matters more is making the app handle SIGTERM properly. On the termination signal: stop accepting new requests, finish the in-flight ones, close DB and message broker connections, and exit within the deadline.

terminationGracePeriodSeconds covers preStop execution time and application shutdown time. Budget both together.

maxUnavailable: 0 buys you Pod availability during deploys, and that's where it stops. Node failures and cluster-wide failures are a separate problem — design replica counts, PodDisruptionBudget, node spread, and resource headroom for those.

Automatic retries make outages worse too

Even with correct Probes, a bad retry policy will expand the outage anyway. When something breaks, mobile apps, web clients, the API Gateway, the service mesh, internal microservices, message consumers, and batch jobs all retry at once. A single recovered Pod takes the entire backlog and falls over again the moment it comes back. This is the thundering herd.

So retries need brakes. Max retry counts, exponential backoff, jitter, request timeouts, circuit breakers, concurrency limits, gradual traffic recovery. Self-healing and automatic retries are both good features on their own. Combine them badly and you've built a system that amplifies failures automatically.

Questions to answer before you ship

Just check whether you have answers to these.

Startup Probe

• What's the worst-case healthy startup time?
• Can you tell a stuck initialization from a merely slow one?
• If it restarts after a failed start, is recovery actually plausible?
• Does a temporary delay in an external dependency send it into an infinite restart loop?

Readiness Probe

• Does removing this Pod actually improve the overall service?
• Is there any condition where every Pod goes NotReady at once?
• Could a shared DB or external API failure wipe out all endpoints?
• Even with some features broken, could you still serve a reduced set of requests?
• Does it properly cut off new traffic during shutdown?

Liveness Probe

• Is the cause of the failure inside the container?
• Does a restart actually fix it?
• Is there any chance every Pod restarts simultaneously?
• Is network latency being misread as process failure?
• Is the Probe itself heavy, or dependent on another shared system?

The simplest way to think about Probes

Three sentences cover all three Probes. Startup says "still getting ready, please wait." Readiness says "alive, but not taking work right now." Liveness says "waiting won't help, restart me."

A Probe isn't a health check URL. It's the failure definition your application submits to the platform. Returning 200 OK and attaching an action to a failure state are two entirely different jobs.

What to design before self-healing

Kubernetes doesn't understand why something failed. A Probe fails, so it pulls the Pod from traffic or restarts the container. The person designing the criteria for that decision is you.

Good Probes isolate a failure and buy the service time to recover. Bad Probes turn a temporary delay into a restart loop and spread one external system's outage across every Pod.

The question isn't whether you have self-healing. It's what you decided counts as a failure, and what action you attached to it.

Kubernetes doesn't recover from failures. It executes the failure policy a human wrote. In an automated system, correct decisions aren't the only thing that propagates fast. Wrong ones move at exactly the same speed. As for what number failureThreshold should be, I still recount it for every service.

참고 자료

• Kubernetes 공식 문서: Liveness, Readiness and Startup Probes
• Kubernetes 공식 문서: Configure Liveness, Readiness and Startup Probes