lastwords: what a Kubernetes container was doing before it died
lastwords is an open-source DaemonSet that snapshots a container's memory before an OOM kill and saves the logs, events and core dumps of crashes. OOMKilled leaves no heap dump, kubectl logs --previous keeps one restart, and events expire after an hour by default, so that evidence has to be collected at the time.
Install
helm install lastwords oci://ghcr.io/arvo-ai/charts/lastwords -n lastwords --create-namespaceEach node runs one agent, which requests 20m CPU and 48 MiB. Captures stay on the node, or upload to S3, and you read them through the Kubernetes API server:
kubectl lastwords list -n shop --kind oomkill
kubectl lastwords show <capture-id>
kubectl lastwords get <capture-id> jvm-class-histogram.txtWhat it saves
| When | What lastwords saves |
|---|---|
| A container nears its memory limit | Memory breakdown, timeline and processes. Java: class histogram, heap info, native memory, thread dump. Go: heap profile with allocation sites. |
| A container is OOMKilled | The memory curve up to the kill, logs of the instance that died, pod events, and links to the snapshots taken before the kill. |
| A process is OOM-killed but the container survives | Which process died and the memory state. Kubernetes does not report this. |
| A container crashes | Log tail of the dead instance, the crash site parsed from Go, Java or Python stack traces, events, and what the exit code means. |
| A process dumps core | The gzipped core and memory maps, matched to the pod. Off by default (coreDumps.enabled). |
| A pod is evicted | Eviction reason, every container's logs, and events. |
Use it from an AI agent
The MCP server is read-only and works with any MCP client.
claude mcp add lastwords -- kubectl-lastwords mcpFurther reading
- Why we built lastwords, and what is still unsolved
- 24 AI agent runs on crash-looping services, with and without lastwords
- Why a Java pod gets OOMKilled at 80% of its memory limit
- Why HeapDumpOnOutOfMemoryError writes no file in Kubernetes
- Kubernetes OOMKilled: meaning and how to fix it
Questions
What is lastwords for Kubernetes?
lastwords is an open-source DaemonSet from Arvo AI. It records what is using a container's memory before an OOM kill, and saves the logs, events and core dumps of containers that crash, so the evidence still exists when you look at the pod later.
How do I get a heap dump from a pod that was OOMKilled?
You cannot get one after the kill: the kernel sends SIGKILL, so the process runs no exit code and HeapDumpOnOutOfMemoryError never fires. The evidence has to be collected while the process is alive. lastwords watches memory twice a second and captures before the kill.
Does lastwords need changes to my images or flags?
No. The agent runs on the node. For Java it uses the JVM attach API, so jcmd does not have to be in the image. Go heap profiles need the program to serve net/http/pprof; without it you still get the memory curve, the breakdown and the logs.
Can an AI agent read lastwords captures?
Yes. lastwords ships a read-only MCP server (claude mcp add lastwords -- kubectl-lastwords mcp), and it is listed in the official MCP registry as io.github.Arvo-AI/lastwords.
Is lastwords related to Aurora?
Both are open source and made by Arvo AI, but lastwords is a separate tool. It has no dependency on Aurora and works on its own.