HeapDumpOnOutOfMemoryError Not Working in Kubernetes

-XX:+HeapDumpOnOutOfMemoryError writes nothing in Kubernetes when the kernel kills the container first. The flag runs when the JVM throws OutOfMemoryError, and a kernel OOM kill is a SIGKILL the JVM never sees. The heap limit has to sit far enough below the container limit that the JVM fails first. Even then, three more things can lose the dump.

Which limit fires first?

If the heap limit is above the container limit, a leak reaches the second column first. On EKS on 2026-11-03, the same leaking program in a 384 MiB container gave these two results with only the heap setting changed:

What percentage is safe?

We swept MaxRAMPercentage on local Docker with a similar leak (the same 64 KiB entries in a map that never evicts, a different class) and the same limit (Temurin 21.0.13, 2 CPUs, heap dump flags plus -XX:+ExitOnOutOfMemoryError):

So the line for this program sits between 85 and 95. Do not copy that number. This is a tiny single-threaded program, and an EKS capture of it showed 310K of metaspace in use. A real service with hundreds of threads, a large class count, or Netty direct buffers needs far more room outside the heap. We would start at 50 to 70 and move up only with your own measurements. Exit 3 comes from ExitOnOutOfMemoryError. Without it, the EKS run exited 1.

If you cannot tell which limit fired, look at the exit code and the last log lines: exit 137 with no Java exception means the kernel. For the case where the kernel fires well below 100%, see Java pod OOMKilled at 80% of its memory limit.

Why is the dump missing even when the JVM fails first?

The second dump fails, because the file already exists. When java is the container's entrypoint it is PID 1, so the file name never changes, and the JVM does not overwrite an existing file. On local Docker we ran the same container twice against the same bind mount, with exec java as PID 1. The first run wrote its dump. The second printed:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /dumps/java_pid1.hprof ...
Unable to create /dumps/java_pid1.hprof: File exists
Terminating due to java.lang.OutOfMemoryError: Java heap space

We have not run this on a pod. The Kubernetes docs say emptyDir data is safe across container crashes, and our EKS dump had the same java_pid1.hprof name, so we expect every crash after the first to keep the oldest file and lose the newest evidence. Move the old file aside in the entrypoint, or build -XX:HeapDumpPath with a timestamp in a wrapper script.

A memory-backed volume gets the JVM killed while it writes. The Kubernetes docs say that with emptyDir.medium: Memory, files you write count against the memory limit of the container that wrote them. We tested the same effect with a Docker tmpfs at MaxRAMPercentage=50. Both runs were killed in the middle of the dump:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /dumps/java_pid7.hprof ...
jvm-exit=137
Killed

(Trimmed. jvm-exit is printed by our wrapper script.) That left a truncated 166 MB .hprof on the volume. Docker's tmpfs is our stand-in here, and we did not run this one on a cluster. We expect a pod to show OOMKilled, which would send you back to the wrong limit. Write dumps to disk.

The volume is gone or too small. An emptyDir survives a container restart and is deleted with the pod. Dumps are about the size of the used heap, so a 75% heap on a 4 GiB limit means about 3 GiB free. Use a PVC if you need the dump after a reschedule, and check sizeLimit and the node's ephemeral storage limit.

A setup that works

env:
  - name: JAVA_TOOL_OPTIONS
    value: >-
      -XX:MaxRAMPercentage=50
      -XX:+HeapDumpOnOutOfMemoryError
      -XX:HeapDumpPath=/dumps
      -XX:+ExitOnOutOfMemoryError
volumeMounts:
  - name: dumps
    mountPath: /dumps

These are the flags from our local Docker runs at 50, and we have not run this exact YAML on a cluster. Back /dumps with a PVC or a disk emptyDir, and move any old java_pid1.hprof aside on start. ExitOnOutOfMemoryError still writes the dump first, as the logs above show, and it stops a half-broken JVM from staying up. A .hprof holds everything in memory, including tokens and personal data, so handle it as a secret.

What if you cannot change the flags?

Then the only evidence is whatever you collect while the JVM is alive. jcmd 1 GC.class_histogram -all works if the image has jcmd, and JRE images do not. lastwords does this from outside the container through the attach protocol, on a DaemonSet, with no flags or image changes. It is described in our post on OOMKilled evidence. If the pod is failing now, the OOMKilled runbook covers the rest.

Verified 2026-11-03 on EKS 1.33 (Amazon Linux 2023, kernel 6.12) and 2026-11-04 on local Docker 29.4 with Eclipse Temurin 21.0.13. lastwords is built by the team behind Aurora SRE, an open-source AI SRE.