Shipping OKD 4.7 and 4.11 Node journald Logs to a Central Syslog — and Proving It Works First

Sometimes you need to ship systemd journal logs off your OKD nodes to a central log server.

I've already written twice about node journals eating disk. You logged a few lines. So why is your Kubernetes node's disk util at 95%? covered why it happens. The journal ate my OKD node's disk covered how to contain it with retention settings on the node itself. Both ended at the same place: "so you need to get the logs off the box." This post is that next step.

The first instinct is usually to run a Podman Quadlet or a standalone Fluentd container on every node. But OKD 4.7 and 4.11 already ship the machinery to collect node logs and forward them out.

If you have to manage both versions, this is the simplest setup that works for both:

Fluentd
  + ClusterLogForwarder
  + TCP Syslog
  + RFC5424

What I'll cover:

How OKD actually reads the binary journal
One config that works on both 4.7 and 4.11
How to verify forwarding without a central Syslog server
What to check before you turn this on in production

The short version

Both OKD 4.7 and 4.11 forward node journal logs to an external Syslog server through ClusterLogForwarder.

Same API on both versions:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder

The flow looks like this:

노드의 systemd-journald
        ↓
노드별 Fluentd Collector
        ↓
OKD ViaQ 로그 레코드로 정규화
        ↓
ClusterLogForwarder
        ↓
RFC5424 텍스트 메시지
        ↓
TCP Syslog 서버

No Quadlet collector to install on each node.

The journal is binary, so how does anything read it?

The systemd journal isn't a plain text file.

/var/log/journal/.../*.journal

It's a systemd-specific binary format. cat and tail won't get you anything useful.

OKD's Fluentd collector doesn't copy those files and ship them either. It reads individual log records and their fields through the systemd journal API.

Logically, a journal record carries something like this:

{
  "MESSAGE": "Started Kubernetes Kubelet.",
  "_HOSTNAME": "worker-01",
  "_SYSTEMD_UNIT": "kubelet.service",
  "SYSLOG_IDENTIFIER": "kubelet",
  "_PID": "1234",
  "_UID": "0",
  "PRIORITY": "6",
  "__REALTIME_TIMESTAMP": "1787712345000000"
}

Fluentd normalizes that into OKD's ViaQ log data structure. The exact fields shift by version and log type, but the shape is roughly:

{
  "@timestamp": "2026-08-26T12:34:56.123456Z",
  "hostname": "worker-01",
  "message": "Started Kubernetes Kubelet.",
  "level": "info",
  "log_type": "infrastructure",
  "systemd": {
    "u": {
      "_SYSTEMD_UNIT": "kubelet.service",
      "_PID": "1234",
      "_UID": "0",
      "_HOSTNAME": "worker-01"
    },
    "t": {
      "SYSLOG_IDENTIFIER": "kubelet",
      "PRIORITY": "6"
    }
  }
}

The binary .journal file never leaves the node. Fluentd reads records through the journal API, then converts the structured log into RFC5424 text before sending.

What "infrastructure" actually includes

This input in ClusterLogForwarder pulls in node journal logs:

inputRefs:
  - infrastructure

But infrastructure isn't journal-only. You'll typically get all of this:

journald logs from OKD nodes
OS and CRI-O runtime logs
OKD infrastructure component logs
container logs from openshift-* projects
container logs from kube-* projects
infrastructure logs from the default project

There's no knob here for "just send me this one systemd unit." If you only care about a single service, filter on the central log server using fields like _SYSTEMD_UNIT, SYSLOG_IDENTIFIER, hostname, and message.

One config for both 4.7 and 4.11

This forwards node journal and infrastructure logs to a TCP Syslog server.

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - name: central-syslog
      type: syslog
      url: tcp://syslog.example.com:514
      syslog:
        rfc: RFC5424

  pipelines:
    - name: infrastructure-to-syslog
      inputRefs:
        - infrastructure
      outputRefs:
        - central-syslog
        - default
      labels:
        source: okd

Apply it:

oc apply -f cluster-log-forwarder.yaml

Both versions require:

resource name instance
namespace openshift-logging
API logging.openshift.io/v1
Fluentd as the collector (recommended)

What the default output does

This sends to both the external Syslog server and the existing internal log store:

outputRefs:
  - central-syslog
  - default

If you don't need the internal Elasticsearch copy and only want logs going out, drop default:

outputRefs:
  - central-syslog

One catch: once you create a ClusterLogForwarder, log types you didn't define in a pipeline may stop reaching the internal store entirely. If production already has a ClusterLogForwarder/instance, back up the current config first and merge into the existing pipelines. Don't overwrite.

oc -n openshift-logging get clusterlogforwarder instance -o yaml \
  > clusterlogforwarder-backup.yaml

What goes over the wire in RFC5424

The message structure:

<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID STRUCTURED-DATA MESSAGE

In practice it looks like this:

<134>1 2026-08-26T12:34:56Z worker-01 kubelet 1234 - - Started Kubernetes Kubelet.

Whether you get Fluentd's full normalized record or just the message field depends on payloadKey.

Sending the whole record

For your first round of verification, leave payloadKey out.

syslog:
  rfc: RFC5424

Then check whether the normalized record arrives serialized into the Syslog message body. Conceptually:

<134>1 2026-08-26T12:34:56Z worker-01 fluentd - - -
{"hostname":"worker-01","message":"Started Kubernetes Kubelet.",
"log_type":"infrastructure","systemd":{"u":{"_SYSTEMD_UNIT":
"kubelet.service","_PID":"1234"},"t":{"SYSLOG_IDENTIFIER":
"kubelet","PRIORITY":"6"}}}

Sending only the original message

With this, only the record's message field becomes the Syslog body:

syslog:
  rfc: RFC5424
  payloadKey: message

Much cleaner output:

<134>1 2026-08-26T12:34:56Z worker-01 fluentd - - - Started Kubernetes Kubelet.

The cost is metadata. _SYSTEMD_UNIT, _PID, _UID, SYSLOG_IDENTIFIER, the original PRIORITY, systemd cgroup details — any of it can drop out of the body.

So look at the full record first, then decide how much you actually need.

Verifying inside OKD, no central server required

Even if the real Syslog server isn't ready, you can stand up a throwaway TCP listener Pod inside OKD and prove the forwarding path works.

The temporary receiver

This accepts data on TCP 5514 and dumps it straight to the Pod log.

apiVersion: v1
kind: Namespace
metadata:
  name: syslog-test
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tcp-receiver
  namespace: syslog-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tcp-receiver
  template:
    metadata:
      labels:
        app: tcp-receiver
    spec:
      containers:
        - name: receiver
          image: registry.access.redhat.com/ubi8/python-39:latest
          command:
            - python
            - -u
            - -c
          args:
            - |
              import socket
              import sys

              server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
              server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
              server.bind(("0.0.0.0", 5514))
              server.listen(20)

              print("TCP receiver listening on port 5514", flush=True)

              while True:
                  connection, address = server.accept()
                  print("CONNECTED:", address, flush=True)

                  while True:
                      data = connection.recv(65535)
                      if not data:
                          break

                      sys.stdout.write(
                          data.decode("utf-8", errors="replace")
                      )
                      sys.stdout.flush()

                  connection.close()
---
apiVersion: v1
kind: Service
metadata:
  name: tcp-receiver
  namespace: syslog-test
spec:
  selector:
    app: tcp-receiver
  ports:
    - name: syslog
      protocol: TCP
      port: 5514
      targetPort: 5514

Apply it:

oc apply -f syslog-test.yaml
oc -n syslog-test get pods

Tail what comes in:

oc -n syslog-test logs -f deployment/tcp-receiver

Pointing ClusterLogForwarder at it

The receiver's internal service address:

tcp-receiver.syslog-test.svc:5514

Test config:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - name: syslog-test
      type: syslog
      url: tcp://tcp-receiver.syslog-test.svc:5514
      syslog:
        rfc: RFC5424

  pipelines:
    - name: infrastructure-syslog-test
      inputRefs:
        - infrastructure
      outputRefs:
        - syslog-test
        - default

oc apply -f clusterlogforwarder-test.yaml

Generating a journal log to test with

Write one journal message you can easily spot. Pick a node first:

NODE=$(oc get nodes -o jsonpath='{.items[0].metadata.name}')
oc debug node/${NODE}

Drop into the host environment from the debug shell:

chroot /host

Fire off a test message with logger:

logger -p local0.info "OKD-SYSLOG-TEST-$(date +%s)"

Get out:

exit
exit

Then grep the receiver Pod for it:

oc -n syslog-test logs deployment/tcp-receiver \
  | grep OKD-SYSLOG-TEST

Three things confirm forwarding works:

The receiver Pod logs a TCP connection.
RFC5424-shaped text arrives.
Your OKD-SYSLOG-TEST-* message shows up in the grep.

Once you're here, going to production is just swapping the URL for your real Syslog server.

url: tcp://실제-syslog-서버:514

Cleaning up after the test

If you had an existing ClusterLogForwarder, restore the backup:

oc apply -f clusterlogforwarder-backup.yaml

If there wasn't one, delete the test resource:

oc -n openshift-logging delete clusterlogforwarder instance

And tear down the receiver:

oc delete namespace syslog-test

Things to think about before production

TCP vs TLS

If the goal is one config that covers both 4.7 and 4.11, plain TCP is the simplest thing that works.

url: tcp://syslog.example.com:514

But if the logs cross a network you don't trust, use TLS.

url: tls://syslog.example.com:6514

Depending on the Logging Operator version, 4.7 and 4.11 can differ on what certificate keys the TLS Secret needs. If keeping the config identical across versions matters more, another option is TCP on an internal network or VPN segment and let the network layer handle encryption.

Preserving the original severity

This does not pass through the journal's original priority:

facility: local0
severity: informational

It pins the facility and severity on the newly created Syslog message. The original journald PRIORITY and SYSLOG_FACILITY may still be sitting in the normalized record as separate fields.

If you need to see the original values, keep facility, severity, and payloadKey as minimal as possible during early verification.

This is not a journal backup

Syslog forwarding is for searching and analyzing log events. It's the wrong tool for:

preserving the original .journal files
making a complete copy of the systemd journal binary
retaining original evidence for digital forensics

If those are actual requirements, look at backing up the .journal files or using the systemd journal export format instead.

Wrapping up

If you want one way to manage node journal logs across OKD 4.7 and 4.11, this is what I'd use:

Fluentd Collector
  + logging.openshift.io/v1 ClusterLogForwarder
  + infrastructure 입력
  + TCP Syslog
  + RFC5424

Worth keeping straight:

The binary journal file isn't what gets sent.
Fluentd reads records through the systemd journal API.
Those records get normalized into OKD's ViaQ data structure.
ClusterLogForwarder converts them into RFC5424 text.
With payloadKey: message you get the raw message and lose some metadata.
For a first pass, skip payloadKey and look at the full structure.
No central server? Verify with a temporary TCP receiver inside OKD.
If a ClusterLogForwarder/instance already exists, merge into it. Don't overwrite it.