Kubernetes OOMKilled: See What Used the Memory

When Kubernetes reports OOMKilled, the kernel has already sent SIGKILL, and everything that could say what filled the memory went with the process. To know what was using it, something has to look before the kill. lastwords is an open-source DaemonSet that does that, and this is how we built it and where it still falls short.

The first release failed on EKS. A leaking JVM on EKS climbed past 79% of its memory limit, the capture fired, and the JVM died before it answered. The pod showed OOMKilled, exit code 137 and a log that ended on an ordinary line:

handled 3000 checkouts
handled 4000 checkouts

Those are the last two lines of kubectl logs --previous from a later run of the same service. Nothing in them says what the memory was.

What is left after an OOMKilled pod?

If the memory is a slow leak, the limit gets hit again in minutes and you can race it with exec. In the cluster we tested, that race was often lost: containers were down between restarts, and images had no shell or jcmd.

How does lastwords capture memory before the kill?

One agent runs per node. It reads each container's cgroup files (memory.current, memory.stat, memory.events) twice a second and keeps a 10-minute timeline. A capture fires at 90% of the limit, again at 97%, or earlier if the current slope hits the limit within 15 seconds. JVMs get lower tiers, for the reason below.

What it collects depends on the runtime:

When a container dies, the agent copies that instance's log file before the next restart rotates it, and saves the pod's events. With core dumps turned on, a helper on core_pattern stores the gzipped core and matches it to the pod. Each capture becomes a Kubernetes Event on the pod, so it shows up in kubectl describe.

Here is a Go capture from 2026-11-04, taken at 72% of a 256 MiB limit, trimmed to its findings:

- 99% of the working set is anonymous memory (heap, malloc, stacks), so the growth is inside the program, not page cache.
- One allocation site holds most of the Go heap: main.renderInvoice (/src/main.go:29) keeps 56.5 MiB in use.

What did not work?

A 90% threshold, for JVMs. On EKS the JVM never reached 90%. A garbage collection took it from 80% to 96% in about a tenth of a second, and the kernel killed it before the next sample. Our slope-based trigger fired at 79%, which was too late. JVM tiers are now measured against 75% of the limit (capture.jvmGCHeadroom, 0.25 by default), which puts the two captures at 67.5% and 72.75%. The measurements are in Java pod OOMKilled at 80% of its memory limit. The 80% was SerialGC on EKS, and local runs of other collectors ended elsewhere.

The default histogram command. jcmd GC.class_histogram forces a full GC first. On EKS, from 55% of the limit, it pushed the container to a 75 to 76% peak in 4 of 4 runs. With -all the peak was 61 to 62%. A diagnostic that needs 20 points of the limit can cause the kill it was sent to explain, so lastwords now runs -all, which skips the GC and can count garbage.

Reporting the biggest class. The real histogram says byte[], which tells you nothing:

   1:         13643      190534000  [B (java.base@21.0.13)
   2:           137        2514256  [Ljdk.internal.vm.FillerElement; (java.base@21.0.13)
   3:         10621         254904  java.lang.String (java.base@21.0.13)

(Top 3 rows of a capture from 2026-11-04 on EKS, at 67% of the limit.) The class that owns those arrays, CheckoutService$CartSnapshot, was seventh at 92,800 bytes. A shallow histogram counts each object's own size, so the owner looks tiny. We now list the largest application classes separately, and we attribute a dominant array to a class with a matching instance count. That second step is a heuristic with no reference graph. It helped on a test where each Indexer$Posting held one long[]. It did not fire here, where 13,643 byte[] include every String's backing array and there are 2,900 snapshots, so a person or agent still has to make the connection.

Treating every exit as a crash. A rollout's SIGTERM (exit 143) on a pod being deleted was reported as a crash. So were the containers of an evicted pod, which the eviction capture already covered. Both are now skipped unless the reason is OOMKilled.

Upgrading with --reuse-values. Helm kept the old values, the new flag rendered empty, and --jvm-gc-headroom= stopped the agent from starting: a crash loop in a tool that exists to explain crash loops. The chart now omits flags the old values do not have.

What is still unsolved?

• Crash loops run out of budget, and one message is wrong about it. Each container gets 3 captures of each kind in 30 minutes, to protect the node's disk. A container that dies every 6 minutes uses that up. On 2026-11-04 one kill's summary said memory "went from below the threshold to the limit between samples, or the agent was not watching yet". That is false for this capture: its timeline shows a 69 second climb sampled every 500 ms. The real reason is that the snapshot budget was spent on earlier cycles. We fixed the message in v0.1.5.
• Kills between captures are only counted. After the third death in a window, nothing is saved until the window moves on. In the same data the billing service has a kill captured at 19:09 (restart 246) and the next at 19:28 (restart 249), so restarts 247 and 248 left nothing.
• Fast spikes. A 500 ms sample cannot catch memory that goes from low to the limit in less time than that. The timeline and logs survive, but there is no snapshot.
• core_pattern belongs to the node. Only one tool can own it, so core dumps are off by default and need a privileged agent. If you already ship cores, leave them off.
• Go without pprof, and every other runtime. lastwords can show that memory is anonymous and climbing, but it cannot name the allocation site.
• Permissions. Without core dumps the agent is unprivileged, but it still runs with hostPID and several capabilities including SYS_PTRACE. Read the security notes before putting it in production.

How do you try it?

helm install lastwords oci://ghcr.io/arvo-ai/charts/lastwords -n lastwords --create-namespace
kubectl lastwords list -n <namespace> --kind oomkill
kubectl lastwords show <capture-id>

The CLI is a separate install, described in the README. On EKS (t3.large, Amazon Linux 2023) we measured the agent at 8m CPU and 12 MiB on a quiet node and 20m CPU and 15 MiB with 35 containers. An MCP server exposes the same captures to AI agents. We have not measured that path: our 24 agent runs used the CLI.

Verified 2026-11-04 on EKS 1.33 (Amazon Linux 2023, kernel 6.12), lastwords v0.1.1 for the captures shown. Current release is v0.1.4, and the memory capture code is the same in both. If the pod is failing right now, start with the OOMKilled runbook.

lastwords is built by the team behind Aurora SRE, an open-source AI SRE.