I Only Wrote a Few Log Lines — So Why Is My Kubernetes Node's Disk Util at 95%?

Disk Util at 95%. On exactly one node.

CPU is calm. Memory has headroom. The application isn't writing large files either. And yet the Disk Util on one particular Worker Node sits at 90%, sometimes past 95%.

The first time I got an alert like this, I suspected storage too. Is the disk slow? Is somebody piling up temp files? Are image layers not getting cleaned up?

Walk down the layers and sometimes you find it started somewhere you never expected.

It was the logs.

Logs are not strings. Logs are I/O.

The log a developer sees and the log the OS sees are different things

To a developer, a log line is this:

2026-08-14 10:21:32 INFO request completed

A string of a few dozen bytes. So it's easy to think:

"How much of a burden could writing a string this size to disk possibly be?"

I thought that too, back when I was writing code. I've also bumped the log level to debug during a deploy and forgotten to put it back.

From the OS and storage side, the story changes. For a log to get stored, data leaving the application has to pass through several layers.

Application → stdout/stderr → Container Runtime → Log File → Filesystem → Block I/O → SSD/NVMe

And a Kubernetes node isn't home to just one application. Dozens, sometimes hundreds of Pods spit out logs at the same time. On top of that, kubelet, the Container Runtime like CRI-O or containerd, network components, and assorted system services keep writing logs of their own.

That's where the problem starts.

Where do Kubernetes logs actually go?

There's a common misconception.

"Aren't Kubernetes logs all stored in journald?"

Not exactly.

Going by the official Kubernetes docs, on a typical Linux environment it's better to split this into two categories.

First, application container logs. Logs that an application inside a Pod writes to stdout and stderr are handled by the Container Runtime. By default, kubelet has the Runtime write container logs under /var/log/pods.

Pod → stdout/stderr → Container Runtime → /var/log/pods

The kubectl logs we all use is reading these logs through kubelet. The official docs also explain that kubelet manages container log rotation and the directory structure.

So where does journald fit in?

On systemd-based Linux there's one more piece. Host-level components like kubelet, containerd/CRI-O, and systemd services write their logs to journald.

So the log structure on an actual Kubernetes node isn't a single pipeline.

        Kubernetes Node
┌─────────────────────┐
│ Application Pods    │──→ stdout/stderr ──→ Container Runtime ──→ /var/log/pods
├─────────────────────┤
│ kubelet / Runtime   │
│ system services     │──→ journald ──→ Journal File
└─────────────────────┘

What matters is that both of these use the same node's disk.

So what happens when there are a lot of logs?

Let's say one Worker Node has 100 Pods on it.

Each Pod keeps emitting logs. kubelet records state changes. The Container Runtime records events like container creation, termination, and errors. And the node has other system logs too.

Roll it up into one physical view and it looks like this:

Pod A ─┐
Pod B ─┤
Pod C ─┤
...    ├──→ Node Disk
Pod N ─┤
kubelet┤
CRI-O ─┤
systemd┘

Individually, none of it is a big deal. Together, it's a different story. Because logs are closer to a workload that writes small amounts of data very, very often.

This is where you have to separate Throughput from IOPS

When people look at storage problems, most start with capacity and transfer volume.

"We're only writing a few MB per second, so why is the disk busy?"

Disk performance has at least three axes. How much data got moved (Throughput), how many I/O operations occurred (IOPS), and how long each I/O took to complete (Latency).

Swap in a delivery analogy and it gets easy. Throughput is how many tons you delivered in a day; IOPS is how many times the driver walked up to a house. Moving one 10-ton freight load and delivering ten thousand small boxes one at a time may look similar in total volume, but they're completely different jobs.

Logs are the same. Writing a few GB as one sequential file is easier on storage than continuously writing small logs at a very high frequency.

If you conclude "it's not a disk problem" purely because Disk MB/s isn't high, you'll wander around for a long while, like I did.

journald is not log.txt

It's a problem if you picture the journal as appending one line of text to the end of log.txt. journald manages a structured binary journal.

Storing a single log involves layers like the journal data structure, the filesystem, the filesystem's metadata and journal, and the block device. Which means the "100 bytes of log" you see from the application and the "100 bytes of physical write" that actually happens at the storage device are not always 1:1.

Zoom out and this is the Write Amplification view. A small logical change balloons into more work at the layers below.

Logical Write → Journal processing → Filesystem metadata → Filesystem Journal → Block Write → Storage

On a log-heavy server, looking only at capacity means you're seeing half the picture. You have to look at frequency and storage structure together.

In Kubernetes, this problem gets amplified

On a single ordinary server, the number of applications producing logs is limited. Kubernetes is different. One Worker Node is the runtime environment for many applications. Which means log producers are densely packed onto a single node.

It gets worse when a particular application starts emitting hundreds or thousands of lines per second, or when errors repeat across several Pods at once.

Take an external system failure as an example.

External API outage → Application timeout → Retry → Error Log → Retry → Error Log → Retry...

The original problem was the external API outage. But as retries explode, so do the logs. The node's Disk I/O climbs. When Disk latency climbs, containers and system components feel it too. Which then produces more timeouts and errors.

Outage → more retries → more logs → more Disk I/O → higher Latency → more failures

That's the feedback loop. It's the point where the logs recording the incident start to grow the incident.

Which is why you shouldn't keep logs only on the node in Kubernetes

The official Kubernetes docs state a principle: cluster logs should have Storage and a Lifecycle independent of Nodes, Pods, and Containers.

Kubernetes itself doesn't provide a complete centralized log storage system. So the usual structure looks like this:

Pod/Node → Logging Agent → Central Logging → Object Storage / Log Storage → Dashboard / Search / Alert

An Agent like Fluent Bit or Vector collects logs and hands them to a Backend like Loki or Elasticsearch/OpenSearch. If you need long-term retention, you attach Object Storage.

There's one rule to hold onto: don't use a Worker Node's local disk as long-term log storage.

But does bolting on central logging end it?

It doesn't. Because the Logging Agent isn't free either.

Log generated → Local Write → Logging Agent Read → Parsing → Buffer → Network Transfer → Backend Write

A single log uses CPU, Memory, Disk, and Network over its lifetime. If useless DEBUG logs pour out in bulk, no matter how well you've built the central logging system, the cost just moves somewhere else.

The first question in log architecture isn't "where do we store this?" — I think it's "why are we writing this log at all?"

Logs are a Resource too

In Kubernetes, everyone is sensitive about CPU and Memory.

resources:
  requests:
    cpu:
    memory:
  limits:
    cpu:
    memory:

Logs, oddly, get ignored. An application developer can increase DEBUG logging tenfold and nothing blocks it immediately the way a CPU Limit would.

Logs consume resources too. CPU, Memory, Disk, IOPS, Network, and Storage Cost. In a large Kubernetes environment, I'd say Logging belongs in Capacity Planning.

When looking at an incident, look one layer down

There's something interesting about operating Kubernetes.

You think it's a Pod problem and it's a Node problem. You think it's a Node problem and it's an OS problem. You think it's an OS problem and it's a Filesystem problem. You think it's a Filesystem problem and it's a Storage problem. And in the other direction, I've had cases that looked like a Storage problem where the starting point was excessive application logging.

Application → Container → Kubernetes → Operating System → Filesystem → Storage

In incident analysis, I think what matters more than deep knowledge of one technology is the ability to walk down the layers. If CPU is high, you don't just look at CPU — you look at why it went up. If Disk Util is 95%, you don't just look at the disk — you look at who is writing what, and how often.

So where I've landed

We keep logs so we can analyze incidents. The more you keep, the safer it feels. From the infrastructure side, that's not necessarily true. Logs are data, data has to be stored, and storing costs I/O.

In an environment where countless workloads share a single node, each individual log line is nothing — but the sum becomes load.

Logs are I/O, not strings. That much I'm sure of. How far you can cut them back? I still haven't settled on an answer. Every time I turn the log level down, that's the day something breaks, and the one line I didn't keep is exactly the one I want.

References

Kubernetes Documentation — Logging Architecture
Kubernetes Documentation — System Logs
Kubernetes Documentation — Observability
systemd GitHub Issue Tracker — cases involving high Disk I/O from journald

※ The figures being circulated lately about "tens of KB or more of writes per log line" on specific filesystems can vary by environment and measurement method, so I've avoided generalizing any specific number in this post.