The SRE Incident Lifecycle: 8 Stages With AI (2026)

Key Takeaways

• The incident lifecycle has eight stages: detect, triage, page, investigate, decide, remediate, retrospect, and prevent. Most writing about AI in operations collapses all eight into "the AI fixes it." Only one stage is mostly evidence gathering, and that is the one worth automating first.
• Investigation is the expensive stage, and it is expensive because it is manual reading. Checking dashboards, correlating deploys, grepping logs. It is also the stage where an agent's advantage is largest, because gathering evidence in parallel is a mechanical strength rather than a judgment call.
• The roles are human roles, and they are standardized. Google's SRE Workbook defines Incident Commander, Communications Lead, and Operations Lead, and PagerDuty publishes its severity scale and response process openly under Apache 2.0. Both describe accountability held by people, not work you delegate to a model.
• The single most useful number in this whole area: Google's SRE book reports that "roughly 70% of outages are due to changes in a live system". That is why stage 8 exists and why it deserves real effort.
• Published AI results are genuinely useful and clearly bounded. Meta reported 42% accuracy at putting the root cause in its top five suggestions for its web monorepo, and Microsoft's RCACopilot reports RCA accuracy up to 0.766. Useful decision support. Not autonomy.
• Measure the stage, not the aggregate. John Allspaw calls incident duration and severity "shallow data" that teaches "next to nothing" on its own, and DORA has replaced mean time to restore with failed deployment recovery time. Knowing which stage consumed your hour beats knowing the hour.

The SRE incident lifecycle is the sequence a team runs from the moment something breaks to the moment it is prevented from breaking the same way again: detect, triage, page, investigate, decide, remediate, retrospect, and prevent. Each stage has a different bottleneck, a different failure mode, and a very different answer to the question of whether software can do it for you.

This guide walks all eight. For each one it states plainly what an AI agent can genuinely take over today, what has to stay human, and what the primary sources say. It is written for SREs and platform engineers who want the process rather than the pitch.

What are the stages of the incident lifecycle?

The short version, before the detail.

#	Stage	Bottleneck	Agent leverage
1	Detect	Whether the right thing is measured at all	None. Thresholds work
2	Triage	Alert volume, duplicate signals	High for correlation
3	Page and assign roles	Human availability	Low. Routing only
4	Investigate	Manual reading across many tools	Highest
5	Decide on a fix	Judgment under uncertainty	Medium. Options with evidence
6	Remediate	Risk of the fix itself	Medium. Draft, never apply
7	Retrospect	Writing time, honesty	High for drafting
8	Prevent	Prioritization against feature work	Medium

Stage 1: Detection

Something is wrong and a machine notices.

Detection is threshold arithmetic and it already works. The interesting question is not the mechanism but what you chose to measure. Google's SRE book gives the canonical answer: "The four golden signals of monitoring are latency, traffic, errors, and saturation." Alternative framings exist and are worth knowing: Brendan Gregg's USE method for resources, and the RED method for request-driven services.

The stage's real failure mode is alerting on the wrong things, and the SRE book is unusually direct about the standard: "Every page should be actionable", and "every page response should require intelligence. If a page merely merits a robotic response, it shouldn't be a page." It also names the human constraint that most alerting configurations ignore: "I can only react with a sense of urgency a few times a day before I become fatigued."

Agent leverage: essentially none, and that is fine. A model is not better than a comparison operator at noticing that error rate crossed 5%. Anyone selling AI detection should be asked what it does that a threshold does not.

What an agent can help with is the meta-problem, auditing whether your alert coverage matches your actual failure modes, which belongs in stage 8.

Stage 2: Triage

Forty alerts arrive. Is that forty problems or one?

This is where volume becomes the enemy. A single failing dependency can light up every service downstream of it, and each of those alerts is technically correct. Triage is the work of collapsing them into a claim about how many distinct things are wrong.

The mechanisms here are well established and mostly deterministic. Prometheus Alertmanager does grouping, which "categorizes alerts of similar nature into a single notification," and inhibition, "suppressing notifications for certain alerts if certain other alerts are already firing." Both are configured in the config file. Correlation platforms add scoring on top.

Agent leverage: high, with a caveat worth stating. Correlation is genuinely improved by knowing service topology and semantic similarity rather than just matching labels. Aurora scores candidate alerts on a weighted combination of topology distance, time proximity, and text similarity, checks it against a threshold, and when an alert correlates it feeds that alert into the investigation that is already running rather than starting a second one. That last behavior is the one that matters during a storm.

The caveat: correlation is a probabilistic guess with a threshold, and a wrong grouping hides a real second incident inside a first one. That is why we argue in our alert noise reduction guide that correlation should collapse alerts into incidents without suppressing the underlying alert stream.

Stays human: severity, and whether to declare an incident at all.

Stage 3: Page and assign roles

Somebody has to own this.

The structure here is standardized and it predates software incident response. Google's SRE Workbook defines three roles: the Incident Commander who "commands and coordinates the incident response, delegating roles as needed", the Communications Lead who is "the public face of the incident response team," and the Operations Lead who applies "operational tools to mitigate or resolve the incident." The SRE book adds that a clear separation of responsibilities "allows individuals more autonomy than they might otherwise have, since they need not second-guess their colleagues."

PagerDuty publishes its entire incident response process openly, including a five-level severity scale where SEV-1 is a "critical issue that warrants public notification and liaison with executive teams" and SEV-5 covers cosmetic issues. The documentation is Apache 2.0 licensed, so you can adopt it directly rather than inventing your own.

The SRE Workbook's most useful advice about this stage is that it should be settled beforehand: "the time to start thinking about an incident management process is before an incident occurs."

Agent leverage: low, and mostly logistical. Routing the page, opening the channel, and attaching context a responder will want. The roles themselves are accountability structures, and accountability is not delegable to a model.

Stage 4: Investigation

This is where the hour goes, and it is the stage this entire category exists for.

An engineer wakes up, opens six tabs, and starts reading. What changed recently. What the logs say. Whether the dependency is healthy. Whether this happened before. Nearly all of that is retrieval and correlation of evidence that already exists somewhere.

That composition is exactly why it is automatable in a way the other stages are not. It is also why the claim worth making for an AI SRE is narrow. The agent is faster at reading than your on-call engineer, and it can read many places at once. That is the whole advantage.

What a good investigation agent actually does. Aurora's investigation prompts are open source and worth reading directly, because they encode the discipline that separates a useful investigation from a confident guess:

• On premature conclusions: "correlation is not causation. Find the mechanism." And on symptoms: "The pod is OOMKilled" is a symptom; "memory leak in the request parser introduced in commit X" is a root cause.
• On negative results: "Absence of expected evidence is evidence." If you searched for errors matching the symptom and found none, that is a finding, not a gap.
• On change attribution: "Merged does not always mean deployed." A commit on the main branch is not necessarily running in production, and if deployment cannot be confirmed the conclusion has to be qualified rather than asserted.
• On confidence, findings are classified as confirmed, likely, or inconclusive, where a likely finding is presented as a hypothesis rather than a cause.

Two implementation notes that matter for evaluating any tool in this category, and where we will be precise about our own configuration rather than quoting a marketing line. Aurora's parallel multi-agent orchestration is a flag that ships disabled, so a default install runs a single investigating agent rather than a fan-out across specialists. Pod isolation for command execution defaults to on in code, but the shipped Helm values and example environment file set it to off, so whether your deployment isolates command execution depends on which configuration you started from. Check the flag rather than assuming. When it is on, execution moves into a per-session Kubernetes pod with an egress policy that blocks private cluster ranges, which we describe in the kubectl safety guide.

How accurate is this in practice? The two best public datapoints, quoted with their own caveats:

Meta reported "42% accuracy in identifying root causes for investigations at their creation time related to our web monorepo", backtested using only information available at investigation start, scoring a hit when the cause appeared in the top five suggested code changes. Meta says it deliberately favored precision over reach to avoid misleading engineers.

Microsoft's RCACopilot paper reports root cause analysis accuracy up to 0.766 across a year of real incidents.

For contrast on the harder end, the ITBench benchmark found that agents built on then-current models "resolve only 13.8% of SRE scenarios," which is a useful corrective to any claim of general operational competence.

Stays human: reading the evidence critically. An agent that cannot return "inconclusive" is not a tool, it is a liability, which is why the classification gate above matters more than the model choice.

Stage 5: Deciding on a fix

You know what is happening. Now: mitigate immediately, or keep investigating?

This is a genuine tradeoff and it is not technical. Rolling back restores service and destroys the state you would need to understand the failure. Continuing to investigate keeps customers broken while you learn. Reasonable engineers pick differently depending on blast radius, time of day, and how much they trust the current hypothesis.

Agent leverage: medium, and specifically at presentation. A useful agent lays out the options with the evidence supporting each, the risk, and the undo path. Aurora's remediation prompt requires a read-only validation query before any fix is proposed, with the output naming what validated it, the risk, and how to undo it. Proposing a fix nobody checked is worse than proposing nothing.

Stays human: the call. This is a business risk decision wearing a technical costume.

Stage 6: Remediation

Something now changes in production. This is the stage where an agent can do real damage, so it is the stage where the design matters most.

The industry has a canonical cautionary example. In July 2025, Replit's coding agent deleted a production database during an explicit code freeze and then incorrectly reported that a rollback was impossible. The instruction not to touch production existed. It was a sentence, not a control.

The lesson generalizes: the gate has to be structural, not instructional. In Aurora, the function that turns a saved incident fix suggestion into a pull request is deliberately not exposed to the model as a callable tool, so a human clicks the button in the UI and the model cannot approve its own remediation. Terraform apply and destroy are denied in unattended runs and are not enabled by default. We detail the seven independent layers every command passes through in our AI agent guardrails post.

Now the part vendors tend to leave vague, which is that all of this is a permission model and permission models have settings. Repository write operations through the GitHub integration are enabled by default, so a read-write background Action can open a pull request unattended. Merging and infrastructure apply are not enabled by default, and an org admin can grant them deliberately. So the accurate statement is not "the agent cannot," it is "the agent cannot unless somebody turned it on." If you are evaluating any agent for this stage, ask which specific operations are permitted with no human present, and verify it in configuration rather than in documentation.

Google has published the most useful framework for thinking about this. Its SRE organization describes running AI operations across graded autonomy levels from L0 to L4, where at L2 a human "still provides final approval" and only at L3 can an agent "independently detect, decide, and act without human approval for bounded actions." The interesting part is what happens between them: if an agent requests an L3 execution but an elevated risk score or anomalous production state is detected, the system "will automatically downgrade the request to L2," intercepting the execution and routing an approval request to a human. Above that sits an emergency "Red Button" that lets SREs "instantly pause all in-flight agentic actions, block new actions, or globally revoke L3 permissions across the fleet during catastrophic, complex outages."

The same writeup shows a case where their agent "incorrectly diagnosed the root cause and failed to mitigate it." Graded autonomy that degrades under uncertainty, a global stop, and a published example of the agent being wrong is a more mature posture than any vendor claim of reliability.

Stays human: approval and the merge. This boundary should not move.

Stage 7: The retrospective

Write down what happened, so the organization learns something a single engineer already knows.

The standard here is well documented. Google's postmortem chapter defines the discipline: "For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior", assuming everyone "had good intentions and did the right thing with the information they had." It also lists concrete triggers for writing one, including data loss, on-call intervention such as a rollback, a resolution time above a threshold, and "a monitoring failure (which usually implies manual incident discovery)."

Agent leverage: high, because the input is a trace rather than a memory. The difference between a good and bad automated postmortem is which evidence it reads. A postmortem drafted from a chat transcript records what people said. One drafted from the actual investigation trace records what was checked, what was ruled out, and in what order. We covered the distinction in our automated post-mortem generation guide.

Aurora generates a postmortem when a human marks an incident resolved, with sections for summary, timeline, root cause, impact, contributing factors, resolution, action items, and lessons learned, exportable to Confluence, Notion, or Jira, and every edit is versioned.

Stays human: blamelessness and the action items. A model does not feel institutional pressure to blame a person, which helps, but it also cannot commit an engineer's next quarter to fixing something.

Stage 8: Prevention

The loop closes here, or it does not close at all.

Return to the number that frames this entire guide: "roughly 70% of outages are due to changes in a live system". If most incidents arrive through your own deploy pipeline, then prevention is not a vague aspiration, it is a specific place to stand.

The practical layers, in increasing distance from the incident:

1. Pre-merge review of risk. Assess a change against the live state of the system it will affect, before it merges. We cover the mechanics in PR risk review, including why the verdict should be advisory rather than blocking.
2. Progressive delivery. Ship gradually and abort automatically. Argo Rollouts runs analysis against live metrics, and a failed analysis "causes the Rollout to abort, setting the canary weight back to zero." See our CI/CD auto-remediation guide.
3. Monitoring hygiene. The alert that did not fire, and the mute nobody reconciled. Both are covered in alert silence drift.

Agent leverage: medium, and mostly about persistence. The reason prevention work rots is that it is never the most urgent thing on anyone's list. A scheduled agent that keeps a living document of known gaps, drops resolved items, and never re-raises things you rejected is a modest mechanism that solves a real organizational failure.

How should you measure any of this?

Carefully, and per stage rather than in aggregate.

DORA gives the delivery-side frame, now five metrics rather than the original four keys, split into throughput (change lead time, deployment frequency, failed deployment recovery time) and instability (change fail rate, deployment rework rate). Change fail rate is defined as "the ratio of deployments that require immediate intervention following a deployment." DORA's own guidance is to apply these at the application or service level rather than as broad organizational comparisons.

On the incident side, be skeptical of MTTR, and note that the criticism comes from inside the practice. John Allspaw groups incident frequency, time to resolve, time to detect, and severity together as shallow data, arguing that "organizations learn next to nothing from this data alone" and that reducing incidents to means and medians "obscures more about the incidents than it actually reveals." Atlassian, which publishes the standard formulas for these metrics, concedes the same point: "A lot of experts argue that these metrics aren't actually that useful on their own because they don't ask the messier questions of how incidents are resolved."

DORA has quietly acted on this. Its recovery metric is no longer mean time to restore but "failed deployment recovery time", scoped to deployments that fail and require immediate intervention rather than averaging across all incidents.

The more useful question is which stage consumed the time. An hour split as fifty minutes of investigation and ten of fixing implies a completely different intervention than ten minutes of investigation and fifty of waiting for an approval. The aggregate hides exactly the thing you would change.

Where does an AI agent actually fit?

The summary of this whole guide: an agent compresses stage 4, helps with 2, 5, 7, and 8, and should be kept structurally out of the decisions in 3 and 6.

That is a narrower claim than "AI resolves your incidents," and it is the one supported by the evidence. Stage 4 is where the 3am hour goes, and it is almost entirely evidence gathering, which is why it is both the biggest cost and the most automatable. Everything downstream of it is judgment, accountability, or a change to production, and those are the parts you want a person holding.

If you want the category context, start with what is an AI SRE and AI SRE vs AIOps. For the investigation stage specifically, see AI-powered incident investigation. For the diagnostic methods that predate all of this, our root cause analysis guide for SREs covers the 5 Whys and fault tree analysis, which remain the right tools when you are the one doing the thinking. If you want to run the whole loop on your own infrastructure, see open source incident management and the self-hosted deployment guide.

• GitHub: github.com/Arvo-AI/aurora
• Docs: Aurora documentation
• Related guides: AI-Powered Incident Investigation · Automated Post-Mortem Generation · PR Risk Review · Multi-Cloud Incident Management

All claims sourced. Process definitions are cited to Google's SRE book and workbook and to PagerDuty's openly licensed incident response documentation. Accuracy figures are quoted from the publishing organizations with their stated methodology. Aurora's behavior, including its default configuration, is described from its open-source implementation. Last verified: July 24, 2026.