The 3 AM Kubernetes Outage That Turned Out to Be a Certificate

The 3 A.M. Nightmare

A coworker told me this one a few days ago. Three in the morning, PagerDuty going off nonstop, and the Kubernetes API server was flat-out unreachable. He SSH'd in and found Pods failing one after another, nodes dropping to NotReady one by one.

The logs had a one-line answer.

x509: certificate has expired

One forgotten certificate took down an entire cluster. GitHub had a similar incident in 2025. Microsoft Azure in 2019. Different scale, same root cause: nobody was tracking an expiration date.

This isn't a subtle bug. It isn't an unpreventable failure. It's just unmanaged time, accumulating until it blows. Certificates live in operations, not in code — but once you issue one, everybody forgets it like it's code. What gets missed is that issuing a cert starts a timer.

Having handled certificates in financial services, this pattern is familiar. Certificate renewal always sits in "somebody must be on top of this," right up until that somebody takes vacation or quits. The responsibility isn't written into any system.

The Certificates in Kubernetes

Start with what's actually running inside the cluster.

인증서 유형	용도	만료 시 영향
API 서버 인증서	제어 평면과의 보안 통신	클러스터 전체 접근 불가
Kubelet 인증서	노드의 API 서버 인증	노드 NotReady, Pod 스케줄링 실패
Ingress 인증서	외부 HTTPS 트래픽	서비스 접근 불가
etcd 인증서	etcd 멤버 간 통신	데이터 저장소 장애

Each has an expiration date, and when it hits, that communication path dies. The structure is simpler than you'd expect. The problem is that these certs don't live in one place. Some sit in Secrets, some on node filesystems, some between etcd members. Scattered means easy to lose sight of, and what you can't see doesn't get managed.

Mistake 1: Not Knowing the Expiration Dates

Most incidents start with "we didn't know when it expired." You can't fix what you can't see. The first value of monitoring isn't the alert — it's the visibility.

Fix: Build Automated Monitoring

Install x509-certificate-exporter first. It exposes every certificate in the cluster as Prometheus metrics.

helm repo add enix https://charts.enix.io
helm repo update

helm upgrade --install x509-certificate-exporter enix/x509-certificate-exporter \
  --set prometheusServiceMonitor.enabled=true \
  --set prometheusServiceMonitor.labels.release=prometheus \
  --set service.port=9793

Then turn on cert-manager metrics.

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.19.4 \
  --namespace cert-manager \
  --create-namespace \
  --set prometheus.enabled=true \
  --set prometheus.servicemonitor.enabled=true

What's left is the Prometheus alerting rules. Warn at 30 days and again at 7. Two thresholds because they mean different things: 30 days is room to plan, 7 days is "you actually have to touch this now."

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-cert-rules
  namespace: monitoring
data:
  cert-expiry.rules: |
    groups:
    - name: certificate_expiry
      interval: 1h
      rules:
      - alert: CertManagerCertExpiring30Days
        expr: (certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 30
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "인증서 {{ $labels.name }}가 30일 내 만료됩니다"
          
      - alert: ControlPlaneCertExpiring7Days
        expr: (x509_cert_not_after - time()) / 86400 < 7
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "긴급: 제어플레인 인증서가 7일 내 만료됩니다"

You can always check by hand too. These commands are your last resort when monitoring itself is broken.

# kubelet 인증서 확인
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate

# API 서버 인증서 확인
openssl s_client -connect localhost:6443 -showcerts 2>/dev/null | \
openssl x509 -noout -enddate

Mistake 2: Managing Certificates by Hand

Manual management is slow, error-prone, and doesn't scale. But the truly fatal part is something else. You forget. Any operation that depends on human memory will fail at least once. It just happens to be at 3 a.m.

Fix: Automate with cert-manager

Start with a ClusterIssuer.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - http01:
        ingress:
          class: nginx

Attach it to your Ingress and renewal happens on its own.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    cert-manager.io/renew-before: "720h"  # 30일 전 갱신
spec:
  tls:
  - hosts:
    - example.com
    secretName: example-com-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80

cert-manager issues the certificate and renews it 30 days before expiry. Nothing for you to touch. Moving renewal from a person's job to a system's job is the point — automation is really about relocating responsibility.

Mistake 3: Not Enabling Kubelet Certificate Rotation

When the kubelet certificate expires, the node can't talk to the API server. It goes NotReady immediately. Ingress certificates are visible enough that people stay on top of them; kubelet certificates aren't visible day to day, so a surprising number of teams miss them.

Fix: Turn On Kubelet Auto-Rotation

# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
rotateCertificates: true
serverTLSBootstrap: true

For a kubeadm-built cluster, set it like this.

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  RotateKubeletServerCertificate: true
rotateCertificates: true
serverTLSBootstrap: true

Restart the kubelet after applying and rotation kicks in.

Mistake 4: Never Testing That Rotation Works

People wire up the automation and never verify it actually runs. Then rotation fails when it counts. Backups, automation, whatever — a safety mechanism you've never tested isn't a safety mechanism.

Fix: Rehearse Rotation Regularly

Issue a test certificate with a short TTL and watch it rotate with your own eyes.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-rotation-cert
spec:
  secretName: test-rotation-tls
  duration: 2h      # 2시간만 유효
  renewBefore: 1h   # 1시간 전 갱신
  issuerRef:
    name: letsencrypt-staging  # 반드시 staging 사용!
    kind: ClusterIssuer
  dnsNames:
  - test.example.com

Always use the Let's Encrypt staging environment for tests. Production has rate limits, and hammering them in a test can block you from issuing real certificates. You'd be validating your renewal automation and end up unable to renew. Not funny in the moment.

Mistake 5: Not Backing Up Certificates

When a certificate gets deleted by accident, or rotation goes sideways, recovery without a backup is grim. Automation exists, and automation also breaks — the backup is what you fall back on when it does.

Fix: Automated Backups

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cert-backup
  namespace: kube-system
spec:
  schedule: "0 2 * * *"  # 매일 새벽 2시
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - |
              kubectl get secret --all-namespaces -o yaml > /backup/certs-$(date +%Y%m%d).yaml
            volumeMounts:
            - name: backup-volume
              mountPath: /backup
          volumes:
          - name: backup-volume
            persistentVolumeClaim:
              claimName: cert-backup-pvc
          restartPolicy: OnFailure

One thing to flag: a Secret dump contains private keys in the clear. Access control and encryption on that backup volume are a separate job you need to do. A backup that becomes a new leak path is a bad trade.

Mistake 6: Self-Signed Certificates in Production

Self-signed certificates throw browser warnings, make auto-rotation awkward, and complicate troubleshooting. The self-signed cert someone used to stand up a PoC quickly follows the service all the way into production more often than you'd think.

Fix: Use a Trusted CA

For external services, use Let's Encrypt. Free, automated, trusted everywhere. For internal services, pair cert-manager with a private CA.

Here's an internal CA setup.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: internal-ca-key-pair

Even internally, a clear trust chain lets you run rotation and validation through the same pipeline you use for external services. Not splitting certificates into two different operational models pays off over time.

Mistake 7: Operating Without an Incident Plan

When an unexpected expiration hits and there's no procedure, recovery time multiplies. Recalling commands from memory at 3 a.m. is nothing like having a doc open and typing what it says.

Fix: A Certificate Incident Playbook

First five minutes: assess. Which certificate is involved, and when did it expire. That's it.

# 영향받은 인증서 식별
kubectl get certificate --all-namespaces

# 만료일 확인
openssl x509 -in /path/to/cert.pem -noout -enddate

Next ten minutes: mitigate. For Ingress, delete the Certificate and reapply. For kubelet and the API server, renew with kubeadm.

# Ingress 인증서 강제 갱신
kubectl delete certificate <cert-name> -n <namespace>
kubectl apply -f ingress-with-cert.yaml

# Kubelet 인증서 갱신
sudo kubeadm certs renew all
sudo systemctl restart kubelet

# API 서버 인증서 갱신
sudo kubeadm certs renew apiserver

The last five minutes: verify.

# 새 인증서 확인
kubectl get nodes
kubectl get pods --all-namespaces
curl -k https://your-api-endpoint/healthz

The 5-10-5 time budget isn't decoration. When your sense of time collapses in the middle of an incident, per-step targets give you a basis for deciding "do I keep digging, or do I work around it."

What I've Taken From This

Certificate management is simpler than people assume. There's no clever logic involved — just know in advance, renew automatically, and check that it works.

Three things. Get the alert 30 days out so you have room to respond. Automate with cert-manager so there's no point where a human forgets. Test regularly so you know the automation actually runs.

Certificate outages aren't a technical problem, they're an operational one. From the code seat it looks like a one-line expiration error. From the ops seat it was a timer nobody owned. The real purpose of automation is moving that timer out of human memory and into a system.

Nobody wants to be up at 3 a.m. This is the kind of thing you set up once and stop worrying about. It's also the item where procrastinating costs you the most.