Runbooks vs Playbooks: A Lab-Tested Runbook and a Template

A runbook is a written procedure for one known failure or task: how to confirm it, the commands that diagnose and fix it, and how to prove the fix worked. A playbook covers the incident around that failure: who leads, who talks to customers, when to escalate, and which runbooks to open. A quick test: a runbook is made of commands and expected output, a playbook is made of roles and decisions. The two words are not used consistently (Google's SRE book and AWS both use "playbook" for documents this page calls runbooks), so check which meaning a source uses before you copy its template.

We publish lab-tested runbooks at /runbooks, each built on a fault we injected into a Kubernetes lab cluster, and every output block in them was captured from a real run. Below are the two definitions side by side, one of those runbooks taken apart section by section, a template you can copy, and what we saw about runbooks when an AI SRE agent investigated 38 lab incidents without any.

What does runbook mean?

AWS's Well-Architected Framework puts it in one line: "A runbook is a documented process to achieve a specific outcome." and, on the same page, "At its simplest, a runbook is a checklist to complete a task." PagerDuty describes a runbook as a detailed how-to guide for a task or procedure that comes up again and again in IT operations, with patching a server and renewing a website's SSL certificate as its examples.

There are two kinds in practice:

• Task runbooks: planned work. Rotate a certificate, restore a backup, add a node. They start from a ticket or a calendar.
• Incident runbooks: unplanned work. They start from an alert or an error string and end when a check says the service is healthy again.

Everything on our /runbooks page is the second kind, and the page states the scope in one line: "One symptom per page. Confirm it, follow the branch the output points to, and stop when the check says the incident is over." One symptom per page is a choice. A runbook that covers "Kubernetes networking" is a wiki page; a runbook that covers "Service has no endpoints" can tell you which command to run first and what its output means.

Runbook vs playbook: what is the difference?

The two connect in one direction. A playbook points at runbooks; a runbook does not need to know who the incident commander is. AWS's guidance on playbooks says the same thing: "If the root cause is identified, the playbook should point to a runbook that describes how to resolve it."

Why do Google, AWS and PagerDuty use the words differently?

Because they split the work at different places. What each source means, checked on 29 October 2026:

So "playbook" means three things depending on who wrote it: Google's alert procedure, AWS's investigation guide, and the incident process that names roles and channels. The last is the most common meaning in incident management, and it is the one this page uses.

Our position: split by scope, not by diagnosis versus fix. AWS puts the investigation in a playbook and the fix in a runbook. In a real incident the same person does both within minutes, and the diagnosis is what decides which fix is safe. In our Service runbook, the step that proves the selector is wrong is also the step that tells you the fix is a selector patch and not a pod restart. Split across two documents, that link gets lost. The strongest case for AWS's split is routing: one "5xx errors" investigation can lead to several fixes. Our runbooks handle that with an "At a glance" table at the top that lists the look-alikes (a wrong targetPort, a failing readiness probe, a NetworkPolicy) with the command that tells them apart, and a Related line that links the sibling runbooks, so the routing lives in the same page as the commands.

What does a real runbook look like?

Here is Service has no endpoints: the selector matches no pods, upstream returns 502, built from scenario 07 of our lab (Arvo-AI/aurora-lab). The fault is one command that points the api Service at a label no pod carries:

kubectl -n shop patch svc api -p '{"spec":{"selector":{"app":"api-v2"}}}'

Every pod stays 1/1 Running, and the frontend's nginx returns 502 on every /api/ request. The runbook has these parts, in this order. Output below is from run v0-run2, 2 October 2026, Kubernetes server v1.37.0 on kind v0.33.0, kubectl client v1.35.3.

1. Header table. Applies to, Verified on (exact versions and a date), Lab folder. A reader on Kubernetes 1.29 knows at once that output formats may differ.

2. The answer, first. Two to four sentences that are correct if read alone: what it means, the likeliest cause, the one command that confirms it, the fix direction. For this runbook: confirm with kubectl get endpoints <service> -n <namespace>; an ENDPOINTS column reading <none> is the whole diagnosis.

3. At a glance. A table of Signal, Likely cause, Confirm with, Fix, ordered by likelihood. It routes the reader who has a look-alike (endpoints populated but connections refused is a targetPort problem; pods not Ready is a readiness problem) to the right page before they run anything.

4. Confirm. Proof that it is this failure:

kubectl get endpoints api -n shop

NAME   ENDPOINTS   AGE
api    <none>      2m43s

5. Scope. How much is affected and since when. kubectl get pods -n shop -o wide showed all seven pods 1/1 Running with zero restarts, which rules out the application. The load generator's log dated the first 502 to 16:35:06Z.

6. Isolate. A decision sequence, cheapest and likeliest check first, each with a stop condition. Step one compares the live selector with the pod labels (kubectl get svc api -n shop -o yaml, trimmed):

metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"api","namespace":"shop"},"spec":{"ports":[{"port":8080,"targetPort":8080}],"selector":{"app":"api"}}}
spec:
  selector:
    app: api-v2

The pods carry app=api. The runbook says "Stop here if the labels differ: this is the cause." The other three checks (targetPort, readiness, NetworkPolicy) run only if this one passes.

7. Fix, with risk and rollback. The exact change, then what can go wrong, then how to undo it:

kubectl -n shop patch svc api -p '{"spec":{"selector":{"app":"api"}}}'

Risk: low; a selector change restarts nothing, but a selector that is too broad can send traffic to pods you did not mean to include. Rollback: patch the previous value back, or apply the manifest from source control. Mitigation and fix are separated: here no mitigation is needed, and the runbook says not to restart pods, which changes nothing.

8. Verify. A pass condition, not "should be fixed": get endpoints lists the pod IPs and the frontend log shows 200 with no new connect() failed lines for a minute. The lab's own check for the broken state is one line, and its negation is the pass condition:

[ -z "$(kubectl -n shop get endpointslices -l kubernetes.io/service-name=api -o jsonpath='{.items[*].endpoints[*].addresses[*]}')" ]

9. Prevent. An alert with the real query, a guardrail and a test. The alert is kube_endpoint_info unless on (namespace, endpoint) kube_endpoint_address{ready="true"}, with a note on why the simpler-looking sum of kube_endpoint_address compared with zero never fires: kube-state-metrics emits one series per address, so a Service with no addresses has nothing to sum.

10. Common mistakes, what we saw in the lab, sources. The fixes that look right and are not (restarting pods, editing pod labels to match the Service), how Aurora SRE investigated this fault in two runs including the report that under-stated what it had found, and every source with the date it was read.

What makes a runbook good? What we learned writing ours

Most of what we got wrong in early drafts was not in the commands. It was in what surrounds them.

• Write the look-alikes. An empty endpoints list has at least two causes (a selector that matches nothing, pods that are not Ready). A runbook that only covers one sends the reader into the wrong fix with confidence.
• Put a stop condition on every check. "Stop here if the labels differ" saves the three checks that would otherwise run out of habit.
• Separate mitigation from the fix. Rolling back to stop the bleeding and correcting the cause are different steps with different risks. Some failures need no mitigation at all, and the runbook should say so.
• Give every mutating command a risk and a rollback. If you cannot write the rollback, the command does not belong in a runbook yet.
• Make the verify step a command with a pass or fail result, plus how long to watch. "Monitor the service" is not a pass condition.
• Say what you did not run. In this runbook the fix was never applied inside a graded lab run, because Aurora's command policy blocks mutations; the page says so and gives the pass condition instead of a captured result.
• Test the alert query against the broken state, not just the fix. The version of this runbook we published on 8 October alerted on sum by (namespace, endpoint) (kube_endpoint_address{ready="true"}) == 0, which never fires on the exact condition it was meant to catch. We replaced it on 19 October.
• Date it and pin versions. A runbook with no "verified on" line cannot tell its reader when to stop trusting it.

How do you test a runbook?

You break the thing on purpose and run the runbook against it. Our content standard defines a folder contract for every runbook: break.sh injects the fault, diagnose.sh runs exactly the Confirm, Scope and Isolate commands and writes a transcript, fix.sh applies the fix, verify.sh exits non-zero if the fix did not work, versions.txt is written by CI, and aurora-trace.json holds Aurora's investigation. CI is meant to rerun it on every change and every quarter, and a red run flags the page for re-verification.

That is the contract, not the current state. On 29 October 2026 every scenario folder in the lab holds scenario.yaml (the alert, the change, the root cause and what a correct answer must name), break.sh and check.sh (proves the fault is visible), and the folders behind published runbooks add a README.md copy of the runbook. ./lab.sh selftest breaks each scenario and runs its check.sh, so the broken state is tested. Nothing tests the fix: no folder has diagnose.sh, fix.sh or verify.sh, no CI job writes versions.txt, the only workflow in the repo rebuilds the benchmark table, and the versions on each page come from the captured run (evidence-before/versions.yaml). Anyone can reproduce the broken state today with ./lab.sh up and ./lab.sh break 07-service-selector-mismatch; the automated rerun of the fix and verify steps is still to be built.

Runbook template

Copy this and fill one per symptom. It is the skeleton our runbooks follow, without the lab section.

# <Exact error or symptom>: <what it means, 3 to 7 words> (<product>)

| Applies to | <product, versions, configurations this covers> |
|---|---|
| Verified on | <exact versions>, <date> |
| Owner | <team or person who keeps it current> |

**<2 to 4 sentences: what it means, the likeliest cause, the one command
that confirms it, the fix direction. Correct if read alone.>**

## At a glance
| Signal | Likely cause | Confirm with | Fix |
|---|---|---|---|
| <what you see> | <cause, most likely first> | <command> | <change> |

## Confirm: is it this failure?
<command>
<expected output, pasted from a real run>
Look-alikes: <similar symptoms with a different cause, and where their runbook is>

## Scope: how much is affected, since when, after which change?
<commands for blast radius, first failure time, recent changes>

## Isolate: which cause is it?
1. <cheapest, likeliest check>. Stop here if <condition>.
2. <next check>. Only if step 1 passed.

## Fix
Mitigation (stops the impact): <command>. Risk: <...>. Rollback: <...>.
Fix (removes the cause): <command>. Risk: <...>. Rollback: <...>.

## Verify
Pass condition: <command and the exact result that means fixed>.
Watch for: <how long, which signal>.

## Prevent
Alert: <query, tested against the broken state>
Guardrail: <policy, admission check, CI test>

## Common mistakes
- <fix that looks right and is not, and why>

## Escalate
If <condition>, page <team> and hand over: <what to include>.

The last section is where the runbook meets the playbook. It names the condition for handing over and what to hand over; who gets paged and how the incident is run belong in the playbook.

What does a playbook contain?

A playbook is about people and decisions. The common parts:

• Severity levels: what makes an incident a SEV1 or a SEV3, and what each level triggers.
• Roles: who commands, who fixes, who writes things down, who talks to customers. PagerDuty's public docs list six roles, from Incident Commander to Internal Liaison; Atlassian's list adds a social media lead and a problem manager.
• Communication: the channel, the status page, how often updates go out and who writes them.
• Escalation: when to pull in another team or a manager, and how.
• Handoff: how the incident closes and moves to a root cause analysis and postmortem.
• Links to runbooks for the failures the team already knows.

We do not publish a playbook, and Aurora SRE does not cover this side: it has no on-call scheduling, paging or escalation, and no status pages. Incident management tools (paging, roles, status pages) run the playbook; an investigation agent works the runbook side. Our open-source incident management guide covers the tools for the playbook side.

How does an AI SRE agent use runbooks today?

We can show what Aurora did in our lab, from the run folders, and what it does with notes and documents, from its source code and docs.

In the lab, it looked for notes first and found none. We ran Aurora (commit 87d22baf, Claude Sonnet 5 on Bedrock, default read-only observability_only command policy) on 19 scenarios, twice each, 38 graded runs on 2 October 2026. Every run used a fresh organization so that nothing carried over. In all 38, the second tool call was a search of the organization's memory, and all 38 searches came back empty. From scenario 07, run r2:

grep_memories {"query": "shop"}
{"status": "ok", "count": 0, "matches": [], "message": "No matches for 'shop' in memory."}

So every result in our lab, including the 34 of 38 runs graded correct on the benchmark (the other 4 partial, every verdict checked by a person), was reached with no runbook and no prior notes. That is the strongest argument against runbooks for diagnosis we have: on injected faults in a small cluster, an agent with read access found the cause without one in 34 of 38 runs. The misses are instructive. In the scenario with two faults where the alert named only one (a worker OOM), both runs explained the OOM and called the frontend unaffected, never checking the frontend Service that selected no pods, so both were graded partial. A runbook keyed to that alert would have pointed the same way; the second fault needed a check nobody had asked for. It is also the limit of our evidence, because we never measured the opposite case, the same scenarios with runbooks loaded.

Then it wrote one. 25 of the 38 runs ended with Aurora saving a note to the organization's memory under the category learned. Scenario 07 r2 ended with this (excerpt from the saved content, verbatim):

**How to apply / diagnostic checklist for 5xx + all-pods-Running in this cluster:**
1. `kubectl get svc,ep -n <ns>` — look for any Service with empty ENDPOINTS despite Ready pods.
2. `kubectl get svc <name> -o yaml` — compare live `spec.selector` to the `last-applied-configuration` annotation; a mismatch indicates manual drift.
3. `kubectl get pods -n <ns> --show-labels` — confirm which label pods actually carry.
4. Check downstream caller's logs (e.g. nginx frontend) for `connect() failed (111: Connection refused)` to the ClusterIP — this confirms the symptom is live, not theoretical.
5. Fix: `kubectl patch svc <name> -n <ns> -p '{"spec":{"selector":{"app":"<correct-label>"}}}'` to realign selector with actual pod labels (verify no intentional canary deployment exists first via `kubectl get deployments -o wide` and `kubectl get pods -l app=<new-label>` across all namespaces).

Set against our published runbook for the same fault, it is a good first draft. It has Confirm and Isolate in the right order, the last-applied-configuration clue, and a check for a deliberate api-v2 cutover before patching. It is missing the parts a reviewer adds: no look-alikes (readiness, targetPort), no rollback, no pass condition for the fix, no alert, and it is scoped to "this cluster". It is the investigation written down, not yet a runbook.

Nobody approved it. No person reviewed any of the 25 notes before they were saved, and in the version we ran nothing asked for one: at commit 87d22baf (1 October 2026) the memory write tool had no approval step and no role check, the agent's instructions told it to save a learned note after an incident whose cause was not obvious, and every finished investigation queued a separate background agent whose job is to extract more notes from the conversation. Aurora's main branch on 28 October 2026 still has no approval step; what it added is a permission setting on the Access tab that decides whose sessions may write memory (admins always, editors unless the setting is turned off, viewers only if it is turned on). Each note is versioned, marked as written by the agent, and can be edited or deleted in the memory settings, and the same instructions tell the agent to treat learned notes as hypotheses to check against fresh evidence. Note that the saved checklist above includes a kubectl patch command. Under the default policy Aurora cannot run it, but a person reading the note later can, so review the notes the way you would review a runbook.

When the notes were shared, it read them. An earlier run, v0-run1, put all ten v0 scenarios in one organization (it is superseded for grading for exactly this reason; its verdicts are the judge's, not reviewed by a person). There, later investigations opened what earlier ones had saved. Scenario 04 (a missing Secret) read the notes saved by the sidecar OOM and image tag typo investigations at its steps 2 and 6. Scenario 08 (a pod too large for any node) searched memory for "Pending" and got the missing-Secret note back. Scenario 10 read the incident index Aurora keeps on its own, and its report listed under Ruled Out "Unrelated worker BATCH_SIZE/OOM crash mechanism (seen in prior incidents)". The judge still marked 9 of 10 correct and 1 partial (scenario 09, where the report lost evidence the investigation had found). What the shared memory did was put other faults in front of the agent before it had looked at the cluster. A runbook library has the same problem when titles are vague; the fix in both cases is one symptom per entry and a specific title.

With your documents, it reads them from the same memory. Since September 2026 Aurora keeps what it knows about an organization in one memory store, with categories that include runbook, infrastructure, context and learned. You add or edit entries in the UI, the agent gets an index of entries in its prompt and opens the ones it needs, and the search_runbooks tool on Aurora's MCP server searches the titles and descriptions of runbook entries along with connected SharePoint. Its GCP setup docs show the kind of entry that matters most: a note saying which log bucket to read, because "Without this, the agent will receive a permission denied error on standard log reads and will not be able to discover the bucket name on its own." That is the runbook content an agent cannot infer from the cluster: where things are, what is normal here, what not to touch.

It does not run your runbook's fix on its own. Under the default policy, mutating commands are blocked, so Aurora proposes the fix and can open a remediation pull request on GitHub that a person reviews and merges; nothing merges automatically. None of the 38 tool logs contains a write to the cluster; the fixes Aurora proposed went into a suggestions list. If your runbook's value is in its fix steps, the agent hands them back to a person; if it is in the diagnosis, the agent may already be doing that part. For how agents differ from scripted runbook automation, see agentic incident management.

Do you still need runbooks if you have an AI SRE?

Yes, but fewer of them, and different ones. Our lab says an agent can diagnose a single known Kubernetes fault without a runbook. It also says the note the agent wrote for the Service fault, the one we compared line by line with our runbook, is missing the parts that matter when something goes wrong with the fix: rollback, pass condition, look-alikes, the alert that would have caught it earlier. And it cannot discover facts that are not in the system it can read, like which log bucket is the right one.

Keep three kinds: runbooks for actions with real risk (the fix, its rollback and its check), notes the agent cannot infer (locations, conventions, known-noisy alerts), and the playbook, which no investigation agent replaces. Let the agent draft the rest from its investigations, and have a person review the draft before it becomes the runbook everyone trusts at 3 a.m.; with Aurora today that review is a habit you add, not a step the product enforces. If you want to see what that looks like on your own cluster first, our Kubernetes pod diagnoser takes pasted kubectl output and gives the likely cause.

Aurora SRE is open source (github.com/Arvo-AI/aurora); to see it run an investigation on your stack, book a demo.

Related: all lab runbooks, Service has no endpoints, a second fault the alert did not cover, how to fix OOMKilled, root cause analysis for SREs, the lab report on 19 Kubernetes failures.

Sources and verification

• Lab: Arvo-AI/aurora-lab, runs v0-run2 and v1-run1 (38 graded Aurora runs, 2 October 2026) and the superseded shared-organization run v0-run1. Memory searches and writes counted from each run's steps-full.json. Scenario 07 output from runs/v0-run2/07-service-selector-mismatch (r1 for the endpoints and Service output, r2 for the memory search and note). Versions from evidence-before/versions.yaml: Kubernetes server v1.37.0, kind v0.33.0, kubectl client v1.35.3. Aurora commit 87d22baf, model bedrock/us.anthropic.claude-sonnet-5. Correct-run count from benchmark/results.json.
• Definitions, read 29 October 2026: AWS OPS07-BP03 Use runbooks to perform procedures, AWS OPS07-BP04 Use playbooks to investigate issues, Google SRE book, Introduction, PagerDuty, What is a Runbook?, PagerDuty Incident Response, Different Roles, Atlassian, incident response roles and responsibilities.
• Aurora memory behaviour: the Aurora repository source at the lab commit 87d22baf and on the main branch as of 28 October 2026 (the memory tools, the agent's memory instructions, the post-investigation memory task, the Access-tab memory permission, the MCP search_runbooks tool), and the GCP data access docs, read 29 October 2026.