How etcd Keeps a Kubernetes Cluster Honest

The first thing everyone blames when a cluster breaks

Run enough Kubernetes incidents and a pattern shows up. Something looks wrong, and someone says "check etcd first." I used to think that was just reflex. After a few more incidents, I understood why people say it.

etcd is the center of everything in Kubernetes. Not the API server, not the scheduler, not the controller manager — etcd is the single most important component, which surprised me at first. Ask most people what matters most and they'll say API server. But think it through and it's obvious. Every other component has nothing to do without etcd. Nowhere to read state from, nowhere to write it to.

So this post is me pulling etcd apart properly. How it works, and the things I learned the hard way running it.

What etcd actually is

etcd is a distributed key-value store. Think of it as a very clever dictionary that spans multiple servers while keeping every node in agreement on the same state.

The name is the first hint. On Linux, /etc holds your system config files. etcd is short for distributed /etc — a config store for the whole cluster instead of one box. The design intent is baked right into the name.

Kubernetes has used etcd as its backing store from day one. Every Pod you create, every Service you register, every Secret you generate — it all lands in etcd.

Peek inside and the layout looks like this:

/registry/deployments/default/my-app
/registry/pods/kube-system/coredns-abc123
/registry/secrets/default/my-secret
/registry/services/specs/default/my-service

When you run kubectl get pod my-pod -o yaml, that YAML on your screen comes from here. kubectl asks the API server, the API server pulls it out of etcd.

How Raft keeps everyone in sync

You can't run a single etcd instance in production. Lose that one node and the entire cluster state goes with it. So you run 3 to 5 nodes, and that's where the Raft consensus algorithm comes in.

Raft is democracy with very strict rules. Exactly one leader exists, and every write has to go through it. Followers do what the leader tells them, and if the leader stops responding, they get ready to elect a new one. A write isn't done until a majority agrees.

The write path looks like this:

1. A write request reaches the leader ("create this new deployment")
2. The leader appends it to its log and sends it to all followers
3. Followers write it to their logs and reply "OK"
4. Once a majority replies, the leader commits
5. The leader tells followers it committed
6. The client gets "success"

Worth pausing on one thing here. A single write can't finish without a network round trip and majority sync. That's exactly why etcd is so sensitive to fast disks and a stable network.

This is also why you run an odd number of nodes. With 3 nodes you need 2 to agree; with 5 you need 3. A 4-node cluster tolerates exactly one failure — same as 3 — while costing you more money and complexity. You added a member and got nothing back in availability, so you're worse off than if you hadn't.

What happens when the leader dies? Followers wait for its heartbeat, and after a timeout (usually 150–300ms) with no signal, they decide the leader is gone and start an election. Whichever node collects a majority of votes first becomes the new leader. Why that timeout matters comes back around when I talk about disks.

Only the API server talks to etcd

Here's a point in the Kubernetes architecture you can't afford to miss: only the API server talks to etcd. Not the scheduler, not the controller manager, not kubelet. None of them touch etcd directly.

The API server is the gatekeeper. It handles authentication, authorization (RBAC), and validation, and only then does it read from or write to etcd.

The payoff is clear. etcd doesn't have to care who's asking or whether they're allowed. All the security judgment sits in the API server; etcd just stores things. Responsibility lives in one place, which is a clean design. Flip it around and you get the other half: the moment you bypass the API server and poke etcd directly, you skip every one of those safeguards. That's the real reason direct access is off-limits.

The API server speaks gRPC to etcd on port 2379, and etcd nodes talk to each other on 2380. Every connection is protected with mutual TLS, so there are a lot of certificates involved. This is also where you get those incidents where an expired cert takes the whole cluster down.

The Watch API deserves a mention too. Kubernetes control loops are constantly reconciling desired state against actual state, and Watch is what lets them learn about changes immediately instead of polling. If they polled, the load etcd absorbed would be in a completely different league.

Create a new deployment and the API server writes it to etcd, which immediately fires a watch event to the controller manager. The deployment controller wakes up and starts creating ReplicaSets and Pods. The whole event-driven pipeline flows outward from etcd.

What's in etcd, and what isn't

Worth drawing a clear line here. etcd holds every Kubernetes API object — Pods, Deployments, Services, ConfigMaps, Secrets. Plus cluster configuration and metadata, and the leader election locks that components use.

Container logs, on the other hand, pile up in node-local storage, and metrics go off to your monitoring stack. Container images and application data live outside etcd too.

etcd is not a general-purpose database. It was built for a small amount of configuration data that needs strong consistency. I've seen people try to use it like a regular DB and get burned. The recommended size ceiling is 8GB, and once you start crossing that line, performance gets visibly shaky.

When the size keeps creeping up, you need to defragment:

etcdctl defrag

This reclaims the fragmented space left behind by repeated updates and deletes. A vacuum cleaner for the database.

Backups aren't optional

etcd backups are non-negotiable. If etcd is the cluster's brain, running without backups is no different from putting a production DB on a single disk with no backups.

etcd is the only stateful component in the Kubernetes control plane. Back it up and you can restore the entire cluster state. Lose it without a backup and every resource inside goes at once — Pods, Deployments, Secrets, all of it.

Taking a snapshot is simple and doesn't require downtime:

ETCDCTL_API=3 etcdctl snapshot save /opt/backup/etcd-$(date +%Y%m%d).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

For restores, use the more recently introduced etcdutl:

# etcd 3.5 이상에서 권장
etcdutl --data-dir /var/lib/etcd-restored snapshot restore /opt/backup/etcd-20260407.db

One more thing. A backup isn't real until you've confirmed you can restore from it. Clusters where snapshots pile up nicely every day but nobody has ever run the restore procedure are more common than you'd think. A backup you've never rehearsed a restore from isn't a backup. It's a file.

Things to know before you run this in production

Here's what I picked up actually running etcd in production.

Disk I/O is everything. etcd is far more sensitive to disk latency than to CPU or memory, because it uses a WAL (Write Ahead Log) and has to fsync on every commit. Put it on shared storage or a slow disk and fsyncs back up, which turns into election timeouts and an unstable cluster. This is where that leader election timeout I mentioned earlier connects. If the disk is slow, a perfectly healthy leader gets mistaken for a dead one. SSDs aren't a preference, they're a requirement.

On node count: go with 5 for anything critical. Three nodes tolerate one failure, five tolerate two simultaneous failures. If the cluster absolutely cannot go down, run five members. The tradeoff is that more nodes means more members participating in consensus, which nudges write latency up. More is not automatically better.

Defrag needs to happen on a schedule. Run etcdctl defrag periodically, but know that defragging the leader triggers a brief leadership transfer. So order matters: followers first, leader last.

Three metrics I watch:

• etcd_server_leader_changes_seen_total: frequent leader changes mean instability
• etcd_disk_wal_fsync_duration_seconds: p99 above 10ms means your disk is struggling
• etcd_mvcc_db_total_size_in_bytes: database size

I keep those three up on the etcd dashboard permanently. Looking at them after an incident and watching the trend day to day are completely different things.

Every road leads back to etcd

etcd isn't a component you touch every day. Most of the time you barely notice it's there. And it's also the pillar quietly holding everything up. Every kubectl command passes through etcd, every control loop depends on it, every leader election runs through it.

Kubernetes has that running joke: "it's always DNS." etcd sits in a spot where "it's always etcd" could just as easily be true, which is why it worries me more. The difference is that when DNS breaks you usually lose some functionality; when etcd wobbles, the whole cluster wobbles. The blast radius isn't the same.

So I've developed a habit: hand me a new cluster and the first thing I check is etcd's disk. It's not a flashy component, but understand it and run it properly and it's the most dependable thing you have. On the surface the API server looks like the star. Look at the structure and the real center is always etcd.