The Journal Ate My OKD Node's Disk — Podman and Quadlet Log Management in Practice

I dropped straight into an LLM window whenever a container refused to start. Paste the error, get a plausible-sounding answer, spend two hours acting on it. Once, journalctl -u would have told me the reason on the first screen. The answer is almost always already in the log. What I was missing wasn't information — it was the habit of opening the log first.

Then the habit sticks, and the next problem shows up. The logs work so well that the node dies.

2:40 AM, and a node got evicted

Last December. A worker node in an OKD cluster dropped to NotReady, and the alert said DiskPressure. 2:40 in the morning. I opened the laptop in bed, SSH'd in, ran df -h, and the root filesystem was at 96%. On a 120GB disk, /var/log/journal alone was holding 41GB.

The culprit wasn't a deployed app. An internal collector daemon running on the node under Quadlet was spitting out dozens of health-check failure lines per second, and all of it flowed straight into journald. Nobody had looked at it for about three weeks. It started when I bumped the log level to debug during a deploy two weeks earlier and forgot to put it back. That was me. I own it.

The next morning Kim, who runs the platform with me, dropped one line in Slack: "Our cluster doesn't die from apps, it dies from logs." Not a funny situation, but I laughed.

OKD nodes are CoreOS-based. The character didn't change when the base moved from FCOS to SCOS. /usr is read-only, you can't just install packages, nearly everything running on the node is a systemd unit, and every one of those units sends its output to one place: journald. Container workload logs get written separately by CRI-O under /var/log/pods, but node-level units like kubelet, crio, and NetworkManager — plus any helper services you stack on with Podman/Quadlet — share the journal. Which means an OKD node with no journal retention policy is running with a timer strapped to it. You just don't know what the timer is set to.

How Podman handles logs

The structure is simple. Podman grabs the container's stdout and stderr streams and hands them to wherever the logging driver says. The driver decides where they pile up and how you get them back out. In practice you'll meet two.

드라이버	저장 위치	조회 방법	순환 관리 주체
journald	systemd 저널	journalctl -u <unit>	journald (전역)
k8s-file	디스크 파일	podman logs, 파일 직접 읽기	Podman (컨테이너별)

Pick journald and container logs go directly to the logging service systemd already manages. They ride on the same infrastructure as the rest of the OS logs, so kernel messages, network events, and container output all sit on one timeline. When the cause of an incident is on the node rather than inside the container, that saves you time. You don't bounce between two screens at 3 AM.

k8s-file writes to files on disk. The format is close to Kubernetes container logging, so your existing file-based tooling reads and processes it as-is. It's just text, so grep and awk work.

Setting it at runtime is one line.

podman run --log-driver journald nginx

For anything that lives on the node as a long-running service, I go journald. I use k8s-file only for one-off containers, or when I need to ship the log file itself somewhere. There's some taste mixed into that call — I've seen teams do the exact opposite.

With Quadlet, logging lives inside the service definition

Quadlet manages containers as systemd units. So the logging config goes in the unit file, not on a command line.

[Container]
Image=nginx:latest
LogDriver=journald

For long-running services this is much better. Logging behavior becomes part of the service definition, so you never have to chase down who started what with which flags. And you query it exactly like any other unit.

journalctl -u my-container.service

Because the container already is a systemd service, there's no seam in the integration. So far so good. The problem is the journal itself. Handing rotation responsibility from Podman to systemd also means that no matter how carefully you set the Podman-side options, an empty journal policy makes them pointless. I understood that after looking at 41GB.

If you use the file driver, put a leash on it first

k8s-file lets you cap file size and count per container directly.

podman run -d \
  --log-driver=k8s-file \
  --log-opt max-size=10mb \
  --log-opt max-file=3 \
  nginx

Keep three files of up to 10MB, delete the oldest when you go over. That gives you a hard ceiling: one container can't exceed 30MB. If you're running dozens of containers, multiply that ceiling out and compare it against the node's disk at least once. We did that math for the first time after the incident.

Journal retention — this is the real part

/etc/systemd/journald.conf is where you bound how big the journal gets. Four keys are enough.

키	하는 일	예시
SystemMaxUse	저널이 쓸 최대 디스크 공간	SystemMaxUse=2G
SystemKeepFree	파일시스템에 남겨둘 최소 여유 공간	SystemKeepFree=10G
MaxRetentionSec	로그 최대 보존 기간	MaxRetentionSec=1month
MaxFileSec	새 저널 파일로 넘어가는 주기	MaxFileSec=1week

The config example you'll see floating around looks like this.

[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=1G
SystemKeepFree=100G
MaxRetentionSec=2week
RateLimitIntervalSec=30s
RateLimitBurst=10000

Before you paste that verbatim, take a good look at SystemKeepFree=100G. It means keep 100GB free — and the node I broke had a 120GB disk total. Drop that value in as-is and the journal would have been able to write essentially nothing, getting truncated constantly. A node that doesn't accumulate logs won't fill its disk, but it also shows you nothing during an incident. That's its own kind of outage. Somewhere in the 10–20% range of node disk has worked fine for me.

Applying it is one restart, checking it is one command.

sudo systemctl restart systemd-journald
journalctl --disk-usage

There's one thing I like here. When journald deletes logs due to rotation or space pressure, it records that fact in the journald service log. You get a log about deleting logs. When you're chasing "why do I only have four days of logs when there should be three weeks," that record answers it. To see how far back the oldest entry goes:

journalctl | head -n 1

On CoreOS, you can't edit this file directly

SSHing into an OKD node and editing journald.conf with vi is a stopgap. It disappears the moment the node reboots or the MCO (Machine Config Operator) reconciles the config away. Ten nodes means doing it ten times, and a node that autoscales in obviously won't have it.

The right move is shipping a drop-in file via MachineConfig. In Butane, it looks like this.

variant: openshift
version: 4.16.0
metadata:
  name: 50-worker-journald-retention
  labels:
    machineconfiguration.openshift.io/role: worker
storage:
  files:
    - path: /etc/systemd/journald.conf.d/10-retention.conf
      mode: 0644
      overwrite: true
      contents:
        inline: |
          [Journal]
          Storage=persistent
          Compress=yes
          SystemMaxUse=4G
          SystemKeepFree=10G
          MaxRetentionSec=2week
          MaxFileSec=1week
          RateLimitIntervalSec=30s
          RateLimitBurst=10000

Swap version to match your cluster version. Once it lands, the MCO rolls the nodes with a reboot. On a ten-worker cluster this took about 40 minutes. The value of doing it this way is that the setting rides along automatically when a new node comes up. Config that depends on a human touching it gets skipped eventually. Mine especially.

The user journal blind spot

A lot of Quadlet setups run rootless, as user services. In that case the logs land in the user journal, not the system journal. Look at journalctl --disk-usage, decide everything's fine, and this is exactly what you miss. I missed it twice.

Cleanup is two lines.

journalctl --user --vacuum-time=7d
journalctl --user --vacuum-size=200M

The first drops entries older than 7 days, the second trims oldest-first down under 200MB. If you're running anything as a user service, put those two lines on a timer.

Skimming unit state every morning

Quadlet containers are systemd services, so health checking is nothing special. Pull the podman/quadlet-related units with systemctl list-units and count error-level log lines from the last 24 hours for each.

for unit in $(systemctl list-units --plain --no-legend '*.service' \
  | awk '/podman|quadlet/ {print $1}'); do
  state=$(systemctl is-active "$unit")
  errs=$(journalctl -u "$unit" --since -24h -p 3..0 --no-pager | wc -l)
  printf '%-40s %-10s errors:%s\n' "$unit" "$state" "$errs"
done

-p 3..0 catches emerg through err. I have this run at 8 every morning and dump into a Slack channel. Ten lines of output have flagged problems earlier, and more often, than any fancy dashboard. If a unit's error count was 3 yesterday and 900 today, that number alone decides my morning.

Shipping it out with Vector

With one node, journalctl carries you. Past ten nodes and dozens of containers, you start by wondering which node to even SSH into. That's the point where centralization earns its keep.

Vector pairs well with journald. The pipeline goes: container output lands in journald, Vector reads the journal and transforms it, then pushes into a logging platform like OpenObserve.

sources:
  journal_logs:
    type: "journald"
    include_units:
      - "my-quadlet-service.service"

transforms:
  parse_logs:
    type: "remap"
    inputs: ["journal_logs"]
    source: |
      .message = parse_json(.message) ?? .message
      .container_name = .CONTAINER_NAME

sinks:
  openobserve_out:
    type: "http"
    inputs: ["parse_logs"]
    uri: "https://your-openobserve/api/default/ingest/_json"
    method: "post"
    auth:
      strategy: "basic"
      user: "${OPENOBSERVE_USER}"
      password: "${OPENOBSERVE_PASSWORD}"
    encoding:
      codec: "json"

Narrowing which units you read with include_units is your first line of defense. Push the node's entire journal downstream and the cost on the central platform blows up before anything else does. I ran it for a day without filters, looked at the ingest volume graph, and turned it off on the spot.

Vector's real value is that you finish preprocessing up front. Pull container name and timestamp into fields, drop 2xx status codes, and never even forward noise like health-check requests. Downstream systems receive less, storage costs less, search gets faster. In a log pipeline the leak is always upstream.

Where logs turn into a defense layer

One step further. This is a bit of a tangent, but it actually paid off for us, so I'm writing it down.

Everybody has one script that scrapes logs with regex and decides something. It grows until nobody can touch it. Parse once in Vector and hand off structured JSON, and the Rust or Go or Python app behind it just reads fields. Regex hell moves once, to the front of the pipeline, and stops there.

On top of that structure it's not hard to count how many times the same IP knocked within 30 seconds and auto-block suspicious access through nftables or a firewall API. If you're running Traefik as a reverse proxy, repeated hits on endpoints like /wp-login get cut at the proxy with a middleware rule. I saw 4,000 /wp-login.php requests a day hitting a server that doesn't even run WordPress, added the rule, and the logs went quiet from that day on. A firewall rule that came out of reading logs.

So where things stand

Node journal retention lives in a MachineConfig, user journal vacuum is on a timer, and unit error counts hit Slack at 8 AM. No DiskPressure alerts since the incident.

What I still don't know is whether SystemMaxUse=4G is the right number. When a real outage hits and a node starts dumping logs, 4GB fills in a few hours. The logs from exactly the window I need to look at are the ones that get truncated. Raise it and I'm burning disk; leave it and I can picture how that goes. No answer yet.

I started this with the habit of looking at logs first, and what I'm left with is that the habit only works if the logs are still there. There's a reasonable point somewhere between the policy that deletes and the policy that keeps. I haven't found it yet.