The 5 whys is a root cause analysis method: you state the problem, ask why it happened, answer with evidence, then ask why of that answer, until you reach a cause you can remove. Five is a rule of thumb, not a quota. It works when one causal chain explains all of the impact, and it misleads when two things broke at once, because every answer pulls you further down the chain you started on. We ran it on two incidents from our Kubernetes lab with every answer backed by captured output: on the first it reached the real cause in five whys, on the second the same five answers looked complete while users were getting timeouts from a fault the chain never touched.
The incidents come from the public Aurora lab, where we inject one known fault (or two) into a small app called shop on a kind cluster and keep everything the cluster showed at the time. Because we caused each fault, we know the true root cause and can check every why against it. Arvo AI builds Aurora, the AI SRE agent that the lab grades; the 5 whys on this page were done by hand from the evidence files.
What is the 5 whys in root cause analysis?
The Lean Enterprise Institute defines it as "the practice of asking why repeatedly whenever a problem is encountered in order to get beyond the obvious symptoms to discover the root cause." Its example comes from Taiichi Ohno of Toyota (the Institute cites Ohno's 1988 book): a machine stopped, and the whys run through an overloaded fuse, insufficient lubrication, a pump not pumping enough and a worn pump shaft to the fifth answer, "There was no strainer attached and metal scraps got in." The Institute is explicit that "The specific number five is not the point. Rather it is to keep asking until the root cause is reached and eliminated."
Three rules decide whether a 5 whys is worth anything in software operations. We follow them below and they are what the second example tests:
- Every answer needs evidence. A log line, a field in
kubectl describe, a diff between two revisions. An answer you cannot show is a hypothesis, and the next why built on it is a guess. - Stop at a condition you can change that would have prevented the incident. "Someone made a mistake" is not one. "Nothing checks X against Y before deploy" is.
- The root has to explain every symptom. List what was broken before the first why. When the chain is done, tick off each symptom it explains. Anything left over is a second incident and gets its own chain.
How do you do a 5 whys root cause analysis, step by step?
- Write the problem as impact, with a time. "The worker pod has been restarting in a loop since about 16:25 UTC; user requests still succeed." Not the alert name.
- Ask why and answer only with something you can show. Paste the output next to the answer.
- Ask why of that answer. Repeat.
- Stop at a cause you can remove that would have prevented the incident. In our runs the cluster could answer the first three whys; the last two were about how the change was made.
- Account for every symptom from step 1. If one is not explained, start a second chain from it.
- Turn the last two or three whys into action items with an owner each. The first whys usually only produce the mitigation.
A copyable worksheet is at the end of this page, and the full postmortem layout is in the root cause analysis template.
5 whys example 1: a worker OOMKilled after a config change
This is lab scenario 01. The fault we injected was one command:
kubectl -n shop set env deploy/worker BATCH_SIZE=2500
The alert was "KubePodCrashLooping: pod worker in namespace shop is restarting repeatedly." The evidence below is from the graded run v0-run2/r1 on 2 October 2026 (Kubernetes v1.37.0 on kind v0.33.0), except the kubectl outputs under why 2, why 3 and why 4. Those we captured on 29 October 2026 by injecting the same fault into a fresh cluster, because the agent's read-only policy had blocked rollout history during the graded run. Each block says which run it comes from.
The problem statement. Before the first why, check what users see. The load generator calls the frontend every two seconds. After the first two requests (a timeout and a 502 while the freshly deployed app was starting), all 27 later requests succeeded:
2026-10-02T16:25:04Z GET /api/items -> timeout
2026-10-02T16:25:06Z GET /api/items -> 502
2026-10-02T16:25:08Z GET /api/items -> 200
...trimmed, 25 more lines, all 200...
2026-10-02T16:26:01Z GET /api/items -> 200
So the problem is: the worker pod restarts repeatedly; the user path is fine. In shop the worker runs batch jobs off the request path (loadgen to frontend to api to redis), which matches.
Why 1: why is the worker restarting? The kernel kills it for exceeding its memory limit. Exit code 137 alone would not prove that (137 is 128 plus SIGKILL, which a grace-period kill also produces); the Reason field does:
State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Fri, 02 Oct 2026 12:25:50 -0400
Finished: Fri, 02 Oct 2026 12:25:50 -0400
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
...trimmed, Started and Finished of the previous run...
Ready: False
Restart Count: 3
Limits:
memory: 192Mi
...trimmed, Requests...
Environment:
BATCH_SIZE: 2500
That is kubectl -n shop describe pod for pod worker-86f68b9bf6-hgxwz in the graded run. Started and Finished are the same second: it dies on its first batch.
Why 2: why does it need more than 192Mi? Because of what each batch holds. The worker's code is in a ConfigMap (this output is from the 29 October rerun; the same worker.py is in the graded run's saved manifests):
$ kubectl -n shop get cm api-code -o jsonpath='{.data.worker\.py}'
import os, time
batch = int(os.environ.get("BATCH_SIZE", "500"))
print(f"worker started, batch size {batch}", flush=True)
while True:
# Each queued item holds about 100 KiB while the batch is processed.
buf = [bytearray(100 * 1024) for _ in range(batch)]
print(f"processed batch of {len(buf)} items", flush=True)
del buf
time.sleep(5)
In the graded run the log of the killed container has a single line, worker started, batch size 2500, and never reaches processed batch. 2,500 items of 102,400 bytes is 256,000,000 bytes, about 244 MiB, against a 192 MiB limit, before Python's own memory. At 500 items it is about 49 MiB. Above roughly 1,966 items the buffers alone cannot fit.
Why 3: why is the batch size 2500? Deployment revision 2 changed it from 500 and left the memory limit alone. In the graded run, kubectl describe deploy worker shows deployment.kubernetes.io/revision: 2, and the kubectl.kubernetes.io/last-applied-configuration annotation still holds BATCH_SIZE 500. The ReplicaSets keep each revision's template; this table is from the 29 October rerun, so the creation times are that day's (the ReplicaSet names match the graded run's because the pod templates are identical):
$ kubectl -n shop get rs -l app=worker -o custom-columns='NAME:.metadata.name,REVISION:.metadata.annotations.deployment\.kubernetes\.io/revision,BATCH_SIZE:.spec.template.spec.containers[0].env[0].value,MEM_LIMIT:.spec.template.spec.containers[0].resources.limits.memory,CREATED:.metadata.creationTimestamp'
NAME REVISION BATCH_SIZE MEM_LIMIT CREATED
worker-6cf88d4b58 1 500 192Mi 2026-10-29T15:49:28Z
worker-86f68b9bf6 2 2500 192Mi 2026-10-29T15:49:48Z
Why 4: why did a change that needs 244 MiB ship against a 192Mi limit? Here the cluster runs out of answers. From the 29 October rerun:
$ kubectl -n shop rollout history deploy/worker
deployment.apps/worker
REVISION CHANGE-CAUSE
1 <none>
2 <none>
CHANGE-CAUSE is copied from the kubernetes.io/change-cause annotation, which someone or some tool has to set (Kubernetes docs). Nobody did, so the cluster knows that the value changed and not why. The one hint it gives is the annotation mismatch from why 3: the live spec says 2500 while the last kubectl apply said 500, so the change was made outside the normal apply path. Who made it and why comes from people or the change record. In our lab we know it: the change was a hand-typed kubectl set env with no review and nobody looking at memory.
Why 5: why could nobody see the memory cost? The batch size and the memory limit are set in different places, and nothing (a review checklist, a load test, an admission check) computes one from the other. That is a condition you can remove, so the chain stops here.
| Why | Answer | Evidence | Source |
|---|---|---|---|
| 1 | Container OOMKilled under a 192Mi limit | Reason: OOMKilled, Exit Code: 137 | kubectl describe pod |
| 2 | Each batch allocates about 244 MiB at size 2500 | worker.py, log worker started, batch size 2500 | ConfigMap, pod log |
| 3 | Revision 2 raised BATCH_SIZE 500 to 2500, limit unchanged | ReplicaSet templates, last-applied annotation | kubectl get rs, describe deploy |
| 4 | Hand-applied change, no review of memory | CHANGE-CAUSE <none>, change outside apply; the lab's change log | rollout history, people, change record |
| 5 | Nothing ties batch size to memory limit | Absence of any check in the pipeline | The deploy process |
Symptom check. The only broken thing was the worker; the load generator was fine. The chain explains everything, so this 5 whys is complete. Action items: revert BATCH_SIZE to 500 or raise the limit with the reason written down (mitigation, why 1 to 3); require a change cause on every deploy (why 4); add a check that a batch-size change comes with a load test at the new size (why 5). The cluster-side procedure for this failure is in the OOMKilled runbook.
The split is the practical finding from this example: the first three whys took a few minutes of kubectl, and the two that produce lasting fixes are not in the cluster at all. An AI agent stops at the same place. In both graded runs Aurora reached why 3 and named the BATCH_SIZE change (a person reviewed both: correct), and its own report listed under "Not Checked" that kubectl rollout history was blocked by its read-only policy (why that happens). Whys 4 and 5 are a conversation.
5 whys example 2: where a single chain misleads
Lab scenario 12 shipped two unrelated changes back to back:
kubectl -n shop set env deploy/worker BATCH_SIZE=2500
kubectl -n shop patch svc frontend -p '{"spec":{"selector":{"app":"frontend","tier":"web"}}}'
The alert covered only the first: "KubePodCrashLooping [warning] namespace=shop deployment=worker: Pod shop/worker (container worker) is in waiting state (reason: CrashLoopBackOff)." Evidence below is from run v1-run1/r1, 2 October 2026.
The 5 whys from the alert. Start where the alert points and every answer is the same as in example 1, with the same evidence:
pod/worker-86f68b9bf6-jr2kk 0/1 OOMKilled 3 (35s ago) 51s 10.244.2.46 aurora-lab-2-worker2 <none> <none>
State: Terminated
Reason: OOMKilled
Exit Code: 137
...trimmed, Started, Finished, Last State (also OOMKilled, 137), Ready...
Restart Count: 3
Limits:
memory: 192Mi
...trimmed, Requests...
Environment:
BATCH_SIZE: 2500
Why 1: OOMKilled. Why 2: 244 MiB per batch. Why 3: revision 2 raised BATCH_SIZE. Why 4 and 5: unreviewed change, no check between batch size and memory. Every step is true, every step has evidence, and the document is word for word the one from example 1. A reviewer reading it has no way to tell that users were down during this one.
What the chain never asked. The problem statement. Here is the load generator in the same capture:
2026-10-02T20:23:26Z GET /api/items -> 502
2026-10-02T20:23:28Z GET /api/items -> 502
2026-10-02T20:23:30Z GET /api/items -> 502
2026-10-02T20:23:32Z GET /api/items -> 200
2026-10-02T20:23:35Z GET /api/items -> timeout
2026-10-02T20:23:38Z GET /api/items -> timeout
...trimmed, 13 more timeouts...
2026-10-02T20:24:20Z GET /api/items -> timeout
2026-10-02T20:24:23Z GET /api/items -> timeout
The three 502s came in the first six seconds after the app was deployed at 20:23:24Z, while it was starting. The last success was at 20:23:32Z, then 17 timeouts in a row, and the incident was created from the worker alert at 20:24:25Z, about 50 seconds after users stopped getting answers. The worker is not on the request path, so the OOM cannot explain them. And a pod listing makes the frontend look fine, which is why "unaffected" is so easy to write:
pod/frontend-76d98597b9-785dp 1/1 Running 0 60s 10.244.1.35 aurora-lab-2-worker <none> <none>
pod/frontend-76d98597b9-ccq7s 1/1 Running 0 60s 10.244.2.45 aurora-lab-2-worker2 <none> <none>
The leftover symptom starts a second chain:
- Why do user requests time out? The frontend Service has no endpoints, so nothing receives the traffic:
service/frontend ClusterIP 10.96.10.4 <none> 80/TCP 60s app=frontend,tier=web
endpointslice.discovery.k8s.io/frontend-npfll IPv4 <unset> <unset> 60s
- Why are there no endpoints? The selector requires
tier=weband the frontend pods carryapp=frontendandpod-template-hash, notierlabel. In a label selector with several requirements "all must be satisfied" (Kubernetes docs), so two healthy pods match nothing. - Why does the selector say
tier=web? A patch to the Service went out in the same window as the worker change (the second line of the break script above). - Why did nothing alert on it? The only alert that fired was about pod health. The frontend pods were healthy; the path to them was not.
- Why was the selector change not caught before it shipped? Nothing compares a Service's selector with the pods it is meant to front. That is the second root cause, with its own action items: a check that every Service has endpoints after a deploy, and an alert on the user path (load generator or synthetic check error rate), not only on pods.
Two chains, two root causes, two sets of fixes. The procedure for finding the second one from a single alert is the runbook second fault when the alert covers only one, and the Service side is in Service has no endpoints.
The AI agents made the same mistake. Aurora started from the alert in both graded runs (2 October), built exactly the chain above to BATCH_SIZE, and stopped. Run r1's report says "Other workloads in the shop namespace (api, frontend) remained healthy and unaffected"; run r2's says the api and frontend pods "were running normally and unaffected." Neither ran a command against a Service, its endpoints or the load generator. A person reviewed both: partial. In our head-to-head runs on 15 and 16 October, HolmesGPT 0.43.0 also diagnosed only the OOM, and K8sGPT 0.4.40, which scans the namespace instead of following the alert, flagged the Service but never tied the OOM to the BATCH_SIZE change. All six runs were graded partial; the HolmesGPT and K8sGPT grades come from the judge model with an AI review, not a person. The tool-by-tool detail is in the root cause analysis guide for SREs and on /benchmark.
The lesson is the same for people and agents: a chain started from the alert inherits the alert's scope. The symptom check in step 5 is the one guard we found that catches it, and it costs one look at the user path.
What are the limitations of the 5 whys?
| Failure mode | What it looks like | Guard |
|---|---|---|
| Two independent faults | The chain is right and complete for one; the other never comes up (scenario 12) | List every symptom first, check each one off at the end |
| The alert as the problem statement | Scope is set by whatever paged you | Write the problem as user impact with a time |
| Answers without evidence | "Probably a memory leak" at why 2; everything after it is built on a guess | No answer without pasted output; mark guesses as hypotheses and test them |
| Stopping when the cluster runs out | The chain ends at "revision 2 changed the env var" | Whys 4 and 5 go to the people and the change record |
| Stopping at a person | "The engineer forgot to raise the limit" | Ask why the process let a correct-looking change through |
| Several conditions needed together | One chain names one condition and misses the rest | Use a fault tree with AND gates |
The last row is the 5 whys' other structural limit. In lab scenario 19 a half-failing frontend needed four things at once (a bad ConfigMap edit, one pod restarted, nginx reading config only at start, a probe that could not see the broken path), and a single chain names one of them. The RCA guide draws that fault tree.
5 whys vs fishbone vs fault tree: which root cause analysis method fits?
| Method | What it produces | Use it when | Weak when |
|---|---|---|---|
| 5 whys | One causal chain, ending at a removable cause | One change or one failure explains all the impact | Two faults at once, or several conditions needed together |
| Fishbone (Ishikawa) | Candidate causes grouped by category | The cause is unknown and the team needs a list to test | It lists candidates; it does not say which one happened |
| Fault tree | The failure decomposed with AND and OR gates | The outage needed several conditions together | Built from a wrong model of the system |
The Lean Enterprise Institute describes the fishbone diagram as "a tool used to identify the root causes of a problem," named after Kaoru Ishikawa. They combine well: a fishbone or a symptom list to see everything that is broken, one 5 whys per symptom, and a fault tree when one chain ends in "and also".
A 5 whys template you can copy
Problem (user impact + start time):
Symptoms observed (every one, with source):
-
-
Chain 1, from symptom:
Why 1: Evidence:
Why 2: Evidence:
Why 3: Evidence:
Why 4: Evidence:
Why 5: Evidence:
Root cause (removable condition):
Symptom check: which symptoms does chain 1 explain?
Any left over -> Chain 2, from symptom:
Action items (owner, due):
Mitigation (from whys 1-3):
Prevention (from whys 4-5):
For the full write-up around it (timeline, impact, what went well), use the root cause analysis template.
Can an AI agent run the 5 whys for you?
It can do the first three whys, and fast: in the two graded runs of scenario 01 Aurora gathered the OOMKilled reason, read the worker code, did the memory arithmetic and found the revision change, with a citation for each. It got as far into why 4 as the cluster allows (both reports noticed the last-applied annotation still said 500, so the change was made outside apply), but it cannot answer whys 4 and 5, because those live in people and change records, and in scenario 12 it showed the single-chain trap as clearly as a person would. Use the agent's report as the evidence for the first whys, then run the symptom check and the last whys as a team.
If you want to try that on your own alerts, Aurora is open source at github.com/Arvo-AI/aurora, or book a demo.
Sources and verification
- Lab evidence: runs/v0-run2/01-worker-oom/r1 and runs/v1-run1/12-two-faults-oom-and-selector/r1 (2 October 2026, Kubernetes v1.37.0, kind v0.33.0); the rollout history, ReplicaSet and ConfigMap outputs come from a rerun of scenario 01 on 29 October 2026 on the same versions (kubectl v1.35.3, Kubernetes v1.37.0, kind v0.33.0). That rerun cluster was short of disk (a pod event read
no space left on device), so we used only those three API reads from it, which match the graded run's saved deployment and manifests, and none of its pod output. HolmesGPT and K8sGPT runs: runs/h2h-v1, 15 and 16 October 2026. Grades from benchmark/results.json. - 5 whys definition and the Ohno example: Lean Enterprise Institute. Fishbone: Lean Enterprise Institute.
CHANGE-CAUSE: Kubernetes Deployments. Selector AND semantics: Kubernetes labels and selectors. All checked 29 October 2026.