60% Memory Use and Your Pod Still Gets OOMKilled? Here's What's Actually Happening

Pods That Look Fine and Then Just Vanish

There was a stretch where the dashboard was all green and Pods kept dying anyway. Application logs were clean. No crash loops. Memory sat around 60%, which felt like plenty of headroom.

Then I ran kubectl describe pod and there it was: OOMKilled.

It didn't add up at first. Memory is at 60%, so why is this thing dying of memory pressure? Maybe the monitoring is lying. But once you start blaming your tools, you stop finding answers. The tools are usually honest. You're just asking them the wrong question.

What Averages Hide

Once I dug in, every metric I'd been staring at was an average over time.

"60% average memory" can be the flattened version of a process that idles at 50MB, spikes to 600MB for half a second when requests pile up, then drops back to 50MB. Safe on average. Over the limit in the moment.

That's where it splits, because Kubernetes doesn't judge on averages. A memory limit isn't a guideline, it's a hard wall. The instant a container crosses the configured limit, the kernel sends SIGKILL. No negotiation, no graceful shutdown. Half a second or five minutes — the kernel doesn't care. You crossed it once, you're done.

So we were looking at a system that was safe on average, and the kernel was looking at a system that was dangerous at peak. Same container, two views, completely different verdicts.

Why Monitoring Misses This

There are structural reasons a spike like this slips through the net.

Start with the scrape interval. Metrics usually get collected every 15–30 seconds, so a spike that lasts 200–500ms lands between two samples and disappears entirely. Then visualization flattens it again. Dashboards default to avg() over long windows, so a 600MB spike shows up as a smooth 320MB line.

The measurement point is off too. You check application heap memory and feel fine, while Kubernetes is judging container-level memory (working set). And knowing all of this, most teams still compromise toward lower resolution, because high-resolution metrics cost more to store and put more load on the system. Visibility is what you pay with.

Put the four together and they converge on one thing: the graph you're looking at isn't the system, it's a summary already processed through a scrape interval and an aggregation function. Summaries are fine most days. The problem is that incidents almost always live in the detail the summary cut out.

Changing How You Look

To catch a hidden spike you have to change the query first. Shrinking the window and switching to max is enough to make peaks start poking out from under the average line.

# 기존: avg_over_time(container_memory_usage_bytes[5m])
# 개선: max_over_time(container_memory_usage_bytes[30s])

Swap the metric too. Track container_memory_working_set_bytes — the exact number the kernel uses for its OOM decision. Application-level numbers won't help you. For critical services, pulling the scrape interval down to 1–5 seconds is worth considering. It costs more, but at least you can explain an incident after the fact. Then wire Pod restarts and OOMKilled events into the same view and overlay their timing on the memory spikes, and the causality shows up on its own.

Anyone who's done this knows the first two give you the most for the least effort. One query change and one metric swap turns "no idea why it died" into "there, it jumped to 600."

Where Memory Spikes Come From

In production, sudden memory jumps usually narrow down to a handful of causes. Concurrent requests arriving all at once and blowing up usage. Work that pulls entire payloads into memory, like JSON parsing or file uploads. Garbage collection falling behind so things waiting to be freed pile up temporarily. In-memory data transformation — batch jobs, generating large responses — does it too, and concurrency bugs like memory leaks or inefficient parallel processing show up in the same bucket.

What's worth noticing is that a lot of these are not bugs, they're normal behavior under load. Parse 600MB of JSON in one shot and of course memory jumps 600MB. The code did exactly what it was designed to do. That normal peak only becomes an incident at the moment it meets a hard memory limit.

So if you see OOMKilled and immediately suspect the code, you'll go the wrong way. The first question is: is this a bug, or is this just the shape of the load?

Practical Fixes

There's no clean solution here. It's closer to picking the tradeoff that fits your situation.

The easiest lever is leaving headroom in the memory limit. Run at 40–50% typical utilization and keep space free for spikes to land in — and accept the cost that comes with it. On the other end, you fix the application. Stream data instead of loading it whole, cut down on large object allocation, tune runtime memory settings like Node.js --max-old-space-size.

You can also catch it during verification. Load tests that simulate burst traffic instead of smooth ramps, scenarios closer to real user patterns. The point is to see the spike pattern before production shows it to you. And you can change the structure itself: set Request and Limit differently so temporary overshoot is allowed but sustained overshoot isn't.

Looking at those four from where I've been sitting: the first is an ops answer, the second a dev answer, the third a testing answer, the fourth an architecture answer. The same one-line OOMKilled splits into different prescriptions depending on which seat you're in. And in most real environments, the actual answer is some mix of all four.

It Breaks at the Edges, Not in Steady State

What this whole thing reconfirmed for me: most operational problems don't happen in steady state.

Incidents break at the edges. Unexpected traffic patterns, the complexity of systems interacting, the moment a small assumption stops holding.

Average metrics describe the normal range well. The problem is that failures happen outside the normal range. Watch only averages and you never see it coming — you go digging for the cause after it's already broken, every time.

Monitoring, to me, isn't about drawing a picture of the system when it's healthy. It's about seeing how it wobbles at the edge right before it gives out. That goes double for systems that look safe day to day. Without knowing what's hiding under that 60% average line, there's no way to stop the next 3am page. I only rebuilt my dashboards after I figured that out. Should have seen it much earlier.