Blogguide16 min read

Root Cause Analysis Template, Filled In From a Real Incident

A copy-paste root cause analysis template in markdown, filled in field by field from a real lab incident, plus the five fields an AI first draft got wrong.

Ben Gervais
Ben Gervais, Engineer at Arvo AILinkedInGitHub

A root cause analysis (RCA) template is a fixed set of fields that makes every incident write-up answer the same questions: who was hurt and how badly, when it started and when anyone noticed, what set it off, which conditions had to be true for it to happen, what was ruled out and on what evidence, and what will change. The template below has those fields, ready to paste into a wiki or a repo. Under it is the same template filled in for one real incident from our lab, a ConfigMap edit that broke half of a frontend's requests, followed by the five fields where the first draft of that RCA was wrong.

That first draft was written by an AI agent (Aurora, which Arvo AI builds); a judge model graded it partial, and a person reviewed that grade and agreed. Its mistakes are the ones people make too, which is why they are useful here. For RCA methods (5 Whys, fault trees, change correlation) see root cause analysis for SREs; for the 5 Whys on its own, see the 5 Whys guide.

What is root cause analysis?

Root cause analysis is the investigation after an incident that finds the conditions that caused it, so the team removes those conditions instead of only the symptom. The write-up it produces is usually called a postmortem or post-incident review; the RCA is the part that says why. The working test for a root cause, used throughout this page: if you remove that condition, the user impact does not happen. A single sentence in the root cause field almost always names a trigger or one condition, not the set of them, which is the first mistake the example below makes.

Root cause analysis template (copy and paste)

Plain markdown. The comments say what goes in each field and the test it has to pass; delete them once the field is filled.

# RCA: <what failed, in user terms, not the alert name>

| Field     | Value                                            |
|-----------|--------------------------------------------------|
| Incident  | <ID / link>                                      |
| Date      | <YYYY-MM-DD, times below in UTC>                 |
| Severity  | <SEV level>                                      |
| Status    | Draft / Reviewed / Action items open / Closed    |
| Authors   | <names>                                          |
| Reviewer  | <name, date>                                     |

## Summary
<!-- 2 to 4 sentences: who was affected, how badly, for how long,
     what caused it, what fixed it. Write it last. -->

## Impact
<!-- Measured on the user path (client logs, synthetic checks, edge error
     rate), never inferred from pod or host status. Numbers with sources. -->
- Symptom users saw:
- Scope (share of requests, routes, customers):
- Impact start / impact end:
- Not affected, and the check that proves it:

## Detection
<!-- What fired, when, and how long after impact began.
     Did it point at the component that was actually broken? -->
- Signal:
- Fired at:
- Delay from impact start:
- Pointed at the right component? (yes / no, why)

## Timeline (UTC)
<!-- One clock. Mark IMPACT BEGINS, DETECTED, MITIGATED, IMPACT ENDS.
     Label baseline noise (startup errors, known flaps) as baseline.
     Every line has a source. -->
| Time | Event | Source |
|------|-------|--------|

## Trigger
<!-- The change or event that set it off: what, when, by what means,
     and the evidence. A trigger is not a root cause. -->

## Root causes
<!-- Every condition that had to be true. Test for each one: if it had
     not been true, would users have been hurt? If they still would have,
     it is not a root cause. -->
1.
2.
3.

## Contributing factors
<!-- Things that made it worse, wider, or harder to diagnose, but
     were not needed for it to happen. Include actions during the
     response that made it worse. -->
-

## Ruled out
<!-- Each hypothesis with the evidence that excludes it.
     "Pods were Running" is a status, not evidence. -->
-

## Not checked
<!-- What you could not verify, and why. A gap here is honest;
     a gap disguised as "ruled out" is not. -->
-

## Mitigation and fix
<!-- Mitigation stops the impact; the fix removes the cause.
     Give order, risk, rollback, and the action that would make it worse. -->
- Mitigation:
- Fix:
- Do not:
- Rollback:
- Verified by (pass condition and how long you watched):

## Action items
| Action | Type (prevent / detect / mitigate / process) | Owner | Ticket | Due |
|--------|----------------------------------------------|-------|--------|-----|

## Lessons
- What went well:
- What went wrong:
- Where we got lucky:

## Evidence
<!-- Links to the logs, dashboards, commands and outputs cited above. -->

How should each field read?

FieldWhat it must containThe mistake it prevents
ImpactA number from the user path and the source of that number"Other services unaffected", based on pod status
DetectionDelay from impact start to the signal, and whether it named the right componentTreating alert time as incident start
TimelineImpact start and alert time as separate lines; baseline noise labelledBlaming startup errors that happen on every deploy
TriggerThe change, with the evidence that it happenedA root cause section that never says what changed
Root causesEvery condition that passes the removal testOne sentence that names a single condition
Contributing factorsWhat made it worse or harder to seeMixing "made it bigger" with "made it happen"
Ruled outEach hypothesis with its disproving evidenceLeaving a disproven suspect open, so the next reader chases it
Not checkedGaps and the reason for eachHiding a gap inside a confident root cause
Mitigation and fixOrder, risk, rollback, and the action that would make it worseA restart that turns half an outage into a full one
Action itemsA type on every item, at least one prevent and one detectA list of mitigations that does nothing about the cause

Root cause analysis example: a real incident, field by field

The incident is scenario 19 of the public Aurora lab, a small shop app (load generator, two nginx frontend pods, a Python api, redis, a worker) on a three-node kind cluster, Kubernetes server v1.37.0, kind v0.33.0, nginx 1.27.5. We caused it on 2026-10-02 with the scenario's break.sh: change the upstream in the frontend-nginx ConfigMap from api:8080 to api:8000 and re-apply it, then delete one of the two frontend pods. The alert that opened the incident is scripted: the lab has no alerting rule, and its runner posts this fixed text from the scenario file, so the 48% is not a measurement:

HighErrorRate [warning] namespace=shop service=frontend: 48% of requests to /api/ returned 5xx over the last 10 minutes (threshold 5%).

Every time and number below comes from run v1-run1/19-config-drift-partial-restart/r1: the cluster state captured before the investigation (evidence-before/) and the investigation's own tool outputs (steps-full.json). This is the corrected RCA, the version a reviewer would sign:

# RCA: about half of /api/ requests through the frontend returned 504

| Field     | Value                                                    |
|-----------|----------------------------------------------------------|
| Incident  | aurora-lab scenario 19, run v1-run1/r1                   |
| Date      | 2026-10-02, times in UTC                                 |
| Severity  | high (as set on the incident)                            |
| Status    | Reviewed (lab incident, fix not applied in the run)      |

## Summary
From 20:35:40, one of the two frontend nginx pods returned 504 on every
/api/ request, about half of the route's traffic. The frontend-nginx
ConfigMap had been edited to proxy to api:8000, a port the api Service does
not expose, and only the pod recreated after the edit loaded it; the other
pod still ran the old config from memory. The readiness probe never
touches the upstream, so the broken pod stayed in the Service.

## Impact
- Symptom users saw: GET /api/items returned 504 Gateway Time-out
- Scope: 35 of the 75 client requests after startup failed (47%), from
  20:35:40 to the end of the captured log at 20:39:13 (40 returned 200,
  all of them through the api Service on 8080)
- Impact start / end: 20:35:40 / not reached (the lab run ends without a fix)
- Not affected: not measured. The client calls only /api/items, so no
  other route has a user-path check. /healthz returning 200 on both pods
  shows nginx was up, nothing more. The api pods logged only 200s for
  the requests that reached them

## Detection
- Signal: HighErrorRate on service=frontend (scripted by the lab runner,
  not computed by an alert rule)
- Fired at: 20:36:20 (incident created)
- Delay from impact start: 40 s, set by the lab runner; on a real system
  this is the number to measure
- Pointed at the right component? Yes, the frontend. Pod status did not:
  both frontend pods were 1/1 Running, Ready, 0 restarts

## Timeline (UTC)
| Time     | Event                                                        | Source |
|----------|--------------------------------------------------------------|--------|
| 20:35:25 | frontend pod kt2jr starts, loads upstream api:8080           | describe pod (Start Time), nginx log |
| 20:35:26 | BASELINE: 1 timeout, 2x 502 while the api pods start          | loadgen log; kt2jr "connect() failed (111: Connection refused)" on :8080 |
| 20:35:25-34 | ConfigMap edited to api:8000; pod 2vqr4 deleted, g5zrg created | ConfigMap resourceVersion 14417 (last-applied) vs 14598 (live); Killing / SuccessfulCreate events |
| 20:35:32 | first 200                                                    | loadgen log |
| 20:35:34 | pod g5zrg starts, loads upstream api:8000                    | describe pod (Start Time) |
| 20:35:40 | IMPACT BEGINS: first 504, "upstream timed out" to :8000      | g5zrg nginx log, loadgen log |
| 20:36:20 | DETECTED: scripted HighErrorRate posted, incident opened     | incident record |
| 20:38:34 | investigation starts (first command)                         | steps-full.json |
| 20:41:00 | RCA report written                                           | incident record |
| (none)   | MITIGATED / IMPACT ENDS: not reached in the lab run          | |

## Trigger
An in-place edit of ConfigMap frontend-nginx (upstream api:8080 -> api:8000),
followed by the deletion of one of the two frontend pods. The edit left no
Deployment revision (revision stays 1) and no event; it shows as a
resourceVersion inside last-applied-configuration (14417) older than the
live object (14598). The deletion shows as Killing on 2vqr4 and
SuccessfulCreate for g5zrg in the same second.

## Root causes
1. The ConfigMap upstream was set to api:8000, and nothing checked it
   against the api Service, which exposes only 8080/TCP.
2. One frontend pod was deleted and recreated after the edit, so it
   loaded api:8000; the pod that was not restarted kept the old config.
3. The readiness probe (/healthz) is answered by nginx itself and never
   exercises the upstream, so the failing pod stayed Ready and kept
   receiving half the traffic.

## Contributing factors
- The surviving pod runs a config that no longer exists in the API. Per
  the Kubernetes docs the kubelet also updates the file mounted in it
  (not inspected here, exec was blocked), so restarting it, or reading
  the file to check it, gives the wrong answer.
- A ConfigMap edit is not a rollout: no revision, no event, nothing
  that rolls all replicas together or can be rolled back with the Deployment.
- kubectl exec was blocked for the investigator by its command policy.

## Ruled out
- api backend down: kt2jr's log shows 38 /api/items requests returning
  200 through 10.96.10.144:8080 between 20:35:34 and 20:39:03, the api
  pods logged only 200s, and both api pods were Ready.
- Resource exhaustion or crashes: 0 restarts on every pod in the namespace.
- Network partition: kt2jr reached the api Service throughout; g5zrg's
  errors name port 8000, which the api Service does not expose.

## Not checked
- The config loaded in each nginx process: exec was blocked, and once the
  kubelet synced, the file would have shown api:8000 in both pods anyway. Each pod's error lines
  name the upstream it is using, which answers the question.
- Who made the edit: no audit log was available.

## Mitigation and fix
- Mitigation and fix are the same change here: set the upstream back to
  api:8080 in the ConfigMap, then restart every frontend pod
  (kubectl -n shop rollout restart deployment/frontend). In that order.
- Do not: restart the frontend first, or restart only the failing pod
  "to clear it". Any pod started on the current ConfigMap fails; restarting
  kt2jr would take the route from ~50% to 100% failing.
- Rollback: api:8080 is the pre-incident config kt2jr ran throughout, so
  there is no better state to go back to. If pods still fail after the
  restart, the cause is elsewhere: keep api:8080 and investigate. Do not
  re-apply api:8000 and restart, which puts both pods on the broken port.
- Verified by: the client log shows only 200 for /api/items for several
  minutes, and neither frontend pod logs an upstream on :8000.
  (Not run in the lab: the investigator is read-only.)

## Action items
| Action                                                        | Type     |
|---------------------------------------------------------------|----------|
| Put a hash of the ConfigMap in the pod template so any edit   | prevent  |
| rolls all replicas together (Helm's checksum/config pattern)  |          |
| Ship ConfigMap changes through the same pipeline as the       | process  |
| Deployment; no kubectl edit on mounted ConfigMaps             |          |
| CI check that every nginx upstream host:port matches a        | prevent  |
| Service port in the same namespace                            |          |
| Per-pod synthetic check of /api/ (not the readiness probe,    | detect   |
| which would pull every pod if the api went down)              |          |

## Lessons
- What went well: the client logged every request with its status, so
  Impact could be counted instead of estimated.
- What went wrong: every pod-level signal said healthy; the only
  difference between the pods was 9 seconds of age.
- Where we got lucky: nginx writes the upstream address into each error
  line; nginx applies config only on start or reload, so the pod that was
  not restarted kept serving (a proxy that hot-reloads would have failed
  on both); and the Killing event was still there (events expire, 1 hour
  by default).

## Evidence
aurora-lab runs/v1-run1/19-config-drift-partial-restart/r1:
evidence-before/ (get-all, events, manifests, per-pod logs), steps-full.json

Owners, tickets and due dates are left out because nobody owns a lab cluster; on a real RCA they are the columns that get the work done.

The evidence behind the two fields that carry the analysis, Impact and Root causes, is short. The client's log, captured before any investigation (trimmed):

2026-10-02T20:35:26Z GET /api/items -> timeout
2026-10-02T20:35:28Z GET /api/items -> 502
2026-10-02T20:35:30Z GET /api/items -> 502
2026-10-02T20:35:32Z GET /api/items -> 200
2026-10-02T20:35:34Z GET /api/items -> 200
2026-10-02T20:35:36Z GET /api/items -> 200
2026-10-02T20:35:40Z GET /api/items -> 504
2026-10-02T20:35:44Z GET /api/items -> 504
2026-10-02T20:35:48Z GET /api/items -> 504
2026-10-02T20:35:52Z GET /api/items -> 504
2026-10-02T20:35:54Z GET /api/items -> 200
2026-10-02T20:35:56Z GET /api/items -> 200
2026-10-02T20:36:01Z GET /api/items -> 504

Each frontend pod's own log for its first /api/ request. The pod started at 20:35:34:

2026/10/02 20:35:40 [error] 26#26: *2 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.244.1.43, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.10.144:8000/items", host: "frontend"

The pod started at 20:35:25, with its one startup error and its first success (trimmed):

2026/10/02 20:35:28 [error] 27#27: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.1.43, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.10.144:8080/items", host: "frontend"
10.244.1.43 - - [02/Oct/2026:20:35:34 +0000] "GET /api/items HTTP/1.1" 200 38 "-" "Wget" "-"

Same image, same ConfigMap mounted, two upstream ports. And the events that date the restart (trimmed):

45s         Normal    SuccessfulCreate    replicaset/frontend-76d98597b9   Created pod: frontend-76d98597b9-g5zrg
45s         Normal    Started             pod/frontend-76d98597b9-g5zrg    Container started
45s         Normal    Killing             pod/frontend-76d98597b9-2vqr4    Stopping container nginx

Two facts from the docs hold the root cause together. nginx: "Changes made in the configuration file will not be applied until the command to reload configuration is sent to nginx or it is restarted" (Beginner's Guide). Kubernetes: "When a ConfigMap currently consumed in a volume is updated, projected keys are eventually updated as well" (ConfigMaps). So the file changes in every pod and the running process changes only in pods that start after the edit. The step-by-step diagnosis of this failure is in the runbook ConfigMap change not applied to all pods.

Which fields did the first draft get wrong?

The first draft is the report Aurora wrote in run r1, from the alert text alone, with read-only kubectl, in 280 seconds and 15 tool calls. Within those 15 calls it had every piece of evidence above. A judge model graded it partial, a second model agreed, and a person at Arvo reviewed it and recorded partial.

FieldFirst draft (quoted from the r1 report)What the evidence showed
Root causes"The most likely cause is a port mismatch between the frontend's configured upstream and the actual backend service endpoint."True, and only condition 1. It never says only the recreated pod loaded the bad config, or that the probe kept that pod in service.
TriggerNo statement that the ConfigMap was edited or that a pod was deletedThe Killing event, the 9 s gap in pod start times, and the ConfigMap's resourceVersion mismatch were all in its tool output
Timeline"The error pattern spanned from pod startup (~20:35:25)"The 20:35:26 to 20:35:30 errors were baseline: they came before the api was Ready and before the edited pod existed, and kt2jr logged its one 502 against the correct port, :8080. Impact began at 20:35:40.
Ruled out"intermittent backend unavailability as a contributing factor cannot be ruled out"40 requests through the same api Service succeeded. The suspect was disproven by evidence it held; it had not read the api logs and said so.
Not checked"This discrepancy was not fully reconciled", because exec was blockedListing the blocked exec was right. Stopping there was not: each pod's error lines already name its upstream, and per the ConfigMap docs the file inside the pod would have shown the new value in both.

Two fields the first draft got right are worth copying. Its impact line, "isolated to the /api/items route rather than full service outage", is correct, though it gave no number. And it had a Not Checked section that named its gaps instead of hiding them.

The fix was not in the report by design: Aurora writes fixes to a separate suggestions list. Its suggestion, patch the ConfigMap to api:8080 and roll the frontend, was graded acceptable. Its undo field, though, was the same patch back to api:8000 followed by another restart, which would have put both pods on the broken port: a rollback that is worse than the incident. That is why the template asks for rollback and "do not" lines separately.

The second run of the same scenario, r2, was graded correct (28 tool calls). It still wrote one wrong intermediate conclusion: its summary of the failing pod's log, read on its own, said the failures meant "the backend/API service is unreachable or down", although it had already read the healthy pod's log and both api pods' logs. The final report did not repeat it. HolmesGPT 0.43.0 fell into the Timeline trap outright in its first run on this scenario: it concluded "All data points to a transient startup race, now resolved" and was graded wrong (by a judge model and AI reviewers, no person). Its second run was graded correct by the judge and partial by the blinded regrade. K8sGPT 0.4.40 printed "No problems detected" in both runs. Baseline errors at startup look like the incident unless the timeline labels them.

Why does the Impact field need its own evidence?

Because the alert often covers only part of the incident, and an Impact line written from pod status inherits the alert's blind spot. Scenario 12 of the same lab injects two faults at once: a worker OOMKilled after BATCH_SIZE went from 500 to 2500, which the alert covers, and a frontend Service whose selector matches no pods, which it does not. Six runs, three tools, and none diagnosed both faults. Aurora found the OOM in both runs and wrote, in r1, "Other workloads in the shop namespace (api, frontend) remained healthy and unaffected", with pod status as its only evidence; r2 said they "were running normally and unaffected". HolmesGPT 0.43.0 diagnosed only the OOM in both runs. K8sGPT 0.4.40, which scans the namespace instead of following the alert, flagged "Service has no endpoints, expected label tier=web" in both runs but never said why the worker ran out of memory. All six runs were graded partial.

The rule in the template's Impact comment comes from this: measure impact on the user path (a client log, a synthetic check, the edge error rate), and write "not affected" only next to the check that proves it. The runbook second fault when the alert covers only one has the commands.

How do the Google, Atlassian and PagerDuty postmortem templates compare?

Section names as they appear on each page today. Google's is the example postmortem in the SRE book, Atlassian's is the template on its postmortem guide, PagerDuty's is the PDF linked from its open-source postmortem docs.

SectionGoogle SRE book exampleAtlassian templatePagerDuty templateThis template
SummarySummaryIncident summaryOverviewSummary
ImpactImpactImpactImpact: "Be very specific here, include exact numbers"Impact, measured on the user path
DetectionDetectionDetection: "How would we have cut that time by half?"No section; "Time of the page" is a timeline entryDetection, with delay and whether it named the right component
TimelineTimeline, marking OUTAGE BEGINS separately from INCIDENT BEGINSTimeline, UTC recommendedTimeline, starting with "Time the impact began"Timeline with impact start, detection and baseline marked
TriggerTrigger, separate from root causesLeadup and FaultNo separate fieldTrigger
Root causeRoot Causes (plural)"Root cause identification: The Five Whys", then a single Root causeRoot Causes: "any conditions that contributed", plus actions that "exacerbated the issue"Root causes with the removal test, plus Contributing factors
Ruled out / not checkedNoneNoneNoneRuled out (with evidence), Not checked
ResolutionResolutionResponse, RecoveryResolution, temporary fix and long-term solutionMitigation and fix, with "do not" and rollback
Action itemsTable with Type (mitigate, prevent, process, other), Owner, BugCorrective actions, with who, when and where trackedAction Items: "actionable, specific, and bounded", with ticketsTyped table: prevent, detect, mitigate, process
LessonsWhat went well, What went wrong, Where we got luckyLessons learned"How'd we do?": What went well, What didn't go so wellGoogle's three
OtherSupporting informationBacklog check, RecurrenceResponders, Messaging (internal email, external message), meeting within 5 business daysEvidence

Our template is closest to Google's: Trigger kept apart from Root causes, plural root causes, typed action items, and "Where we got lucky". It adds the two sections none of the three has, Ruled out and Not checked, because the first draft above went wrong in both. If your team already uses Atlassian's or PagerDuty's format, those are the two sections to add. PagerDuty's Messaging section and Atlassian's Recurrence check are worth borrowing in the other direction.

The strongest objection to plural root causes is that people want one answer, and Atlassian's template ends its 5 Whys in one. Keep the one-line answer; it belongs in the Summary. The Root causes field is for the people who will write the action items, and in the example above each of the three conditions produces a different action item. A single root cause would have produced one of them.

How do you fill in an RCA template?

  1. Save evidence first. Per-pod logs, events and live manifests, before restarts and expiry remove them. Events last one hour by default (kube-apiserver flag event-ttl, default 1h0m0s).
  2. Write Impact from the user path. Client logs or synthetic checks, with a count and its source.
  3. Build the timeline on one clock. Impact start and alert time on separate lines; label baseline errors.
  4. Find the trigger and its evidence. Including changes that leave no revision, like a ConfigMap edit.
  5. List every condition and apply the removal test. What fails the test goes to Contributing factors.
  6. Close or keep each hypothesis with evidence. Ruled out needs a disproving fact; everything else goes to Not checked.
  7. Separate mitigation, fix, "do not" and rollback. Then type every action item and make sure at least one prevents and one detects.
  8. Write the Summary last.

How Aurora handles this

Aurora's investigation reports already carry a cited root cause, a Ruled Out list and a Not Checked list, as in the r1 report quoted above, and fixes go to a separate suggestions list with a risk rating for a person to apply. It also drafts postmortems from the investigation and exports them to Confluence, Notion and SharePoint; we have not graded those postmortem drafts in the lab, so this page makes no claim about them. Every run above is public, including the misses, in the lab repo and on /benchmark. The code is on GitHub, or book a demo.

Related: root cause analysis for SREs, the 5 Whys guide, automated post-mortem generation, ConfigMap change not applied to all pods, second fault when the alert covers only one.

Sources and verification. Lab data: Arvo-AI/aurora-lab, scenario 19 (runs/v1-run1/19-config-drift-partial-restart/r1 and r2, 2026-10-02) and scenario 12 (runs/v1-run1/12-two-faults-oom-and-selector, 2026-10-02; HolmesGPT 0.43.0 and K8sGPT 0.4.40 in runs/h2h-v1, 2026-10-15 and 16), Kubernetes server v1.37.0, kind v0.33.0, kubectl client v1.35.3, nginx 1.27.5. Aurora commit 87d22baf, model bedrock/us.anthropic.claude-sonnet-5. Aurora verdicts reviewed by a person; HolmesGPT and K8sGPT verdicts by a judge model and AI reviewers only. The scenario 19 alert is scripted by the lab runner, not computed. Verdicts from benchmark/results.json (generated 2026-10-21). The filled-in RCA's fix was not run in the lab. Template sections checked on the Google SRE book example postmortem, Atlassian's postmortem template page and PagerDuty's postmortem template PDF; quotes from the nginx Beginner's Guide, the Kubernetes ConfigMaps page and the kube-apiserver reference. All read on 29 October 2026.

Questions

Try Aurora for free

Open source, AI-powered incident management. Deploy in minutes.