OOMKilled in Kubernetes: Finding the Real Root Cause (Not Just Raising the Limit)

OOMKilled is one of the most common reasons a pod restarts in production, and also one of the most commonly mistreated. The pattern is familiar: a pod dies, kubectl describe pod shows Reason: OOMKilled, exit code 137, and the fix that ships within the hour is a bump to resources.limits.memory. The pod stops restarting. The ticket gets closed. And in most cases, the actual defect — a leak, a missing cache eviction policy, an under-provisioned node — is still sitting in the code, waiting to resurface at a slightly higher traffic level.

What OOMKilled actually means

When a container’s memory usage crosses its cgroup limit, the kernel’s OOM killer terminates it — that’s the 137 exit code (128 + signal 9, SIGKILL). This is a container-level limit, not a node-level one, which matters because there are two distinct failure modes that get lumped together under the same log line:

1. Container OOM — a single container exceeded its own resources.limits.memory, and the kubelet’s cgroup enforcement killed it. This is what most people mean by “OOMKilled.”
2. Node memory pressure — the node itself is short on memory, and the kubelet evicts pods (often ones with no limits set, or with BestEffort QoS) to protect itself. This shows up differently — as Evicted, not OOMKilled — but gets root-caused the same lazy way: raise requests, move on.

Diagnosing either one starts with the same question: is the memory growth a step function or a ramp?

The instinct to raise the limit — and why it only buys time

Raising the limit is sometimes the correct short-term mitigation. If a workload has a genuinely higher steady-state memory requirement than what it was given — say, a JVM service where the initial limit didn’t account for heap plus metaspace plus thread stacks — then the limit was simply wrong, and fixing it is the root cause fix, not a band-aid.

The problem is doing this reflexively, without checking which case you’re in. A limit raise silences a leak just as effectively as it fixes a genuine under-provisioning, and a leak doesn’t stop leaking because you gave it more room — it just takes longer to hit the new ceiling. Three weeks later the same alert fires again, at a limit that’s now 50% higher, and the team that fixes it a second time has less context than the team that saw it first.

A real diagnosis: reading the memory curve

The fastest way to tell a leak from legitimate load is to look at the shape of the memory graph in the run-up to the kill, not just the fact that a limit was crossed.

Sawtooth pattern, resetting on each request or batch — this is usually healthy garbage-collected behavior. Not the root cause.

Step increases aligned with traffic — memory rises with concurrent requests and falls back when load drops. This points to per-request memory cost being too high relative to the limit, not a leak. Fix: profile per-request allocation, or right-size the limit to actual peak concurrency.

A steady, near-linear ramp with no traffic correlation, starting fresh after every restart — this is the leak signature. Something is accumulating and never getting released: an unbounded cache, a listener that’s never unsubscribed, a connection pool that grows without a ceiling.

Here’s a real shape of that third case, taken from an actual incident:

checkout-service memory climbed steadily after each deploy and hit its 512Mi limit within roughly 40 minutes — a pattern consistent with a leak rather than a load-driven spike. Correlating the timing against recent deploys pointed to a specific commit that had added response caching without an eviction policy: every unique cart total generated a new cache key, and nothing ever expired.

The fix wasn’t a bigger limit. It was a TTL on the cache. The limit raise that would have “resolved” the alert would have turned a 40-minute mean time to OOM into roughly 60 minutes, and the underlying defect would have shipped to every downstream environment untouched.

A practical checklist before you touch the limit

1. Pull the memory graph for the container, not just the pod, across the last 2-3 restart cycles. If every restart shows the same ramp shape starting from the same baseline, that’s your leak signal.
2. Check whether the ramp correlates with request volume or deploy timing. Overlay a recent-deploys marker on the same graph — correlation with a specific commit is a much faster path to root cause than log spelunking.
3. Look for unbounded collections in the diff since the last known-good state — caches without eviction, retry queues without backpressure, and per-connection buffers that never get freed are the most common causes.
4. Check node-level memory pressure separately from the container’s own limit. If a healthy container got evicted, the bug is capacity planning on the node pool, not the workload.
5. Only then, if the workload’s steady-state need is genuinely higher than the limit, raise it — and treat that as the root cause fix, not a stopgap.

Where automated root cause analysis actually helps

Step 1 and step 2 above are exactly the kind of correlation work that’s tedious to do by hand across dozens of services but mechanical enough for an AI model to do reliably once it has the memory time series, the restart history, and the deploy log in front of it. That’s the specific job VantraOps’s AI-assisted root cause analysis is built for: it detects the OOMKilled pattern, pulls the memory curve and recent commit history for the workload, and — using whichever AI provider your org already has a contract with, never a shared model — produces the kind of correlation shown above in seconds instead of the twenty minutes it takes a human to pull the same graphs manually.

It doesn’t replace the judgment call in step 5. It just gets you to that decision faster, with the actual evidence in front of you instead of a hunch.

Frequently asked questions

Does raising the memory limit ever count as the real root cause fix? Yes, when the workload’s genuine steady-state requirement was simply underestimated at deploy time — a step-function increase in legitimate need, not a ramp. The mistake is applying that fix without first checking which pattern you’re looking at.

Why does kubectl describe pod show OOMKilled but the node has plenty of free memory? Because the limit is enforced per-container via cgroups, independent of node capacity. A node can be nearly idle and still kill a container that exceeds its own configured limit.

What’s the difference between OOMKilled and Evicted? OOMKilled is the container-level cgroup limit being enforced by the kernel. Evicted is the kubelet proactively removing pods — often ones without resource limits set — because the node itself is under memory pressure. They need different fixes: right-sizing a container limit versus fixing node-level capacity or setting limits on previously limitless workloads.

How far back should I look at restart history before concluding it’s a leak? At least two to three full restart cycles. A single data point can’t distinguish a leak from an unusually large one-off batch job; a repeating ramp with the same slope each time is much stronger evidence.

See how fleet-wide health checks and AI root cause analysis work together on the platform overview, or start free with one cluster to see your own memory curves.