AI Agents Debugging OOMKilled Pods: 4/12 vs 12/12

Claude agents named the class or function behind an OOMKilled pod in 4 of 12 runs with kubectl alone, and in 12 of 12 when told lastwords was installed. The sample is small, we built the tool, the fixtures are ours and the second arm got a hint, so read this as one experiment. The most useful result is a different one: 10 of the 12 kubectl-only agents tried to run lastwords by themselves, because its captures appear as Kubernetes Events on the pod.

How was the test set up?

• Cluster: EKS 1.33 in us-east-2, two t3.large nodes, lastwords v0.1.1 from the published chart, run on 2026-11-03.
• Workload: namespace storefront with two leaking services, a Java checkout and a Go billing-api, plus healthy web and cart-redis. Neither image contains source code. The Java image has compiled classes, and the Go image is a distroless binary that serves pprof on port 6060.
• Agents: Claude Code 2.1.317 headless, claude-opus-5-5 and claude-sonnet-5-5, no user settings, memory or MCP, Bash as the only tool, in an empty directory.
• Access: the same read-only oncall kubeconfig in both arms, with exec, port-forward and read access to the lastwords API token.
• Runs: 3 per cell, 24 in all, 12 in each arm, in a fixed order over about 15 minutes against the same cluster, with no reset between runs.

Every agent got this prompt, with the service name changed:

I'm on call. In our EKS cluster the checkout service (namespace storefront) keeps restarting. kubectl is set up for you (read-only, exec allowed). Find the root cause and tell me what to change in the code. Be concise.

Arm B added one line and put the plugin on the PATH:

(Someone on the team installed lastwords on the cluster a while back, kubectl lastwords --help, not sure if it's useful here.)

A script graded each final answer. A pass had to name a symbol (CartSnapshot or SessionCache for checkout, renderInvoice or responseCache or a close variant for billing) and use wording about unbounded growth. The wording test matched all 24 answers, so in practice the grade is whether the symbol appears. Diagnosing the OOM kill was not enough, and all 8 failed runs did that much.

What were the results?

Times are the agent's own reported duration, and with 3 runs one slow run moves a mean. Sonnet on checkout in arm A was 31, 34 and 261 seconds; the last run spent its time retrying exec in loops. Lastwords was not always faster: Opus on checkout and Sonnet on billing took longer with it, and Sonnet's quick arm A billing runs were quick because they stopped without naming any code. Two Sonnet billing runs in arm B took 106 and 89 seconds against 18 for the third. Those two logged 8 and 5 API retries and the third logged none, so that gap is not a clean measure of the tool. Those two runs and the 261 second run also took far longer by wall clock (407, 391 and 621 seconds), which we did not trace.

How the four arm A passes got there matters as much as the count. Opus copied the .class files out of the checkout container and disassembled them in all 3 checkout runs. In one billing run it reached the live pprof port and read main.renderInvoice directly. Those are good investigations, and they worked because that agent kept going. All 6 Sonnet runs in arm A stopped at a generic answer. Opus disassembled the classes in all 3 arm B checkout runs too, using 9 or 10 commands where Sonnet used 3 or 4, so those 3 passes did not rest on lastwords alone.

What did a run look like?

A failure first: Sonnet, arm A, checkout, 7 commands, 31 seconds. These are the commands that hit a wall, with their outputs:

$ kubectl lastwords show 20261103T174433Z-017b5e 2>&1 | head -80
error: unknown command "lastwords" for "kubectl"

$ kubectl -n storefront exec deploy/checkout -- sh -c 'jcmd 1 GC.class_histogram 2>&1 | head -15; ...'
sh: 1: jcmd: not found

$ kubectl -n storefront exec deploy/checkout -- sh -c 'ls /app /opt /srv 2>&1 | head -30; ...'
error: Internal error occurred: unable to upgrade connection: container not found ("checkout")

(Commands shortened where marked with ....) It diagnosed the kill and the bad heap flag correctly, then wrote: "A steady 6 MiB/s climb under load looks like a leak, but I haven't seen a heap histogram." Its code advice was a list of typical culprits. That is a fair answer from an agent that could not see the evidence, and our grader failed it.

A success: Sonnet, arm B, checkout, 3 commands, 17 seconds.

$ kubectl get pods -n storefront; kubectl lastwords --help
$ kubectl logs -n storefront checkout-6487d6664-2gzmg --previous --tail=40; kubectl describe pod -n storefront checkout-6487d6664-2gzmg | tail -30; kubectl lastwords list
$ kubectl lastwords show 20261103T174438Z-c01d48; kubectl lastwords show 20261103T174545Z-c04fe6

Its answer began: "checkout:2.8.0 has a memory leak in CheckoutService. A SessionCache holds CartSnapshot objects that are never evicted." It also caught the -Xmx1g above the 384 MiB limit. Same model, same cluster, same read-only access, with the hint and the plugin on the PATH. The evidence had already been saved, so there was nothing to race.

How did agents find lastwords with no hint?

In arm A nobody mentioned lastwords. Ten of 12 runs ran kubectl lastwords anyway, because of Events like this one, from the failed run above:

checkout-6487d6664-2gzmg/checkout at 67% of its 384.0 MiB memory limit, growing 6.5 MiB/s (OOM in about 19s). Details: kubectl lastwords show 20261103T174420Z-200b75

The other 2 runs never saw lastwords in any output, so every run that saw an Event tried the command. All 10 got unknown command, because the plugin was not on their PATH. Three Opus billing runs then ran the lastwords binary inside the agent pod, and that failed too: the pod's own account could not read the API token. A fourth Opus run probed the agent's API through the pod proxy and did not find the route. Several runs told the on-call person to run the command. For anyone building tools that agents will use, this is the lesson we would keep: a tool that writes a runnable command into a pod Event was found here with no MCP config and no documentation.

What are the weaknesses of this test?

• n is 3 per cell. It shows a direction, not an effect size.
• We built the tool, and wrote both fixtures. Both leaks are simple, unbounded caches, which is the failure lastwords handles best. A slow native leak or a fragmentation problem could go differently.
• The pass test rewards what lastwords prints. It wants a symbol name. An arm A answer that said to look for a package-level cache in the invoice rendering path was pointing the right way and still failed. All 7 billing passes name the allocation site renderInvoice, none names the cache responseCache, and all 7 say they could not read the source.
• The Go result depends on pprof. The billing fixture serves it. Without it lastwords has no allocation site, so we would not expect the 6 arm B billing passes on a service that does not.
• Arm A was not a clean baseline. The DaemonSet was running, so arm A agents saw its Events, including memory growth rates. The gap against a cluster without lastwords is unmeasured.
• Arm B was told about it. A hint is not the same as an agent choosing a tool unprompted.
• The billing image leaked the setup. It was copied from a demo image by digest, so the node reported lastwords-demo/leaky-go in imageID, and the name itself says "leaky". Ten of the 12 billing answers mentioned the mismatch, 4 of them in arm A. Three of those 4 still failed, so we do not think it decided a result, but the fixture was not clean.
• Source code was absent on purpose. With the repo available, an agent could find both leaks by reading. This test measures diagnosis from a running cluster only.
• A regex graded it. One person then read all 24 answers, not blind to the arm, and agreed with every grade under that definition.
• Only Claude ran. We planned Codex and did not run it, because our OpenAI key was out of quota.
• You can check it, but not rerun it as is. The prompts, grader, workloads and all 24 transcripts are public, with the account ID and local paths replaced. The cluster no longer exists, so a rerun means building the workloads and installing lastwords yourself.

For how the tool works and what it still gets wrong, see why we built lastwords. For the same failure by hand, see OOMKilled from a slow memory leak.

Run 2026-11-03 with lastwords v0.1.1. The current release is v0.1.5, with the same memory capture code. lastwords is built by the team behind Aurora SRE, an open-source AI SRE. These numbers are about lastwords and say nothing about Aurora.