AI SRE Accuracy: What the Benchmarks Show (2026)

Key Takeaways

• No AI SRE vendor has submitted its own product to an independent third-party benchmark. Two projects publish scores, but for foundation models (Rootly's SRE-skills-bench) or for their own self-curated eval set (HolmesGPT), not for the commercial agent as sold.
• The best model on the only independent AI SRE leaderboard scores 56.2%. On ITBench-AA, run by Artificial Analysis on IBM's task set, GPT-5.6 Sol (max) leads at 56.2% and nothing has yet cleared 60%. The metric is strict: "average precision at full recall."
• The most useful number in this field is the gap between naming and proving. OpenRCA 2.0 (June 2026, 500 instances, 11 frontier LLMs) found agents "identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%." Exact root-cause recovery: 20.7%.
• A leaderboard win does not transfer to your stack. An audit of three RCA benchmark families across 11 subsystems found that picking the pooled winner selects the lower-scoring method on up to 5 of 11 held-out subsystems, "with regret reaching 24.8 pp."
• Waiting for a better model will not fix it. Across 1,675 agent runs on five models, the dominant failures "persist across all models regardless of capability tier, indicating that these failures originate from the shared agent architecture rather than from individual model limitations."
• A confidently wrong answer is the expensive failure mode, because automation bias "cannot be prevented by training or instructions" and affects experts as much as novices.
• Aurora has not published a benchmark score either. We say so plainly below, along with what we would have to build first.

AI SRE accuracy is the rate at which an agent identifies the actual cause of an incident, and as of July 2026 the best independently measured result on a public incident benchmark is 56.2%. Vendor marketing in the same category advertises 92% and, at Komodor, "up to 95% accuracy in incident detection and resolution". Both numbers can be technically defensible, because nobody agrees on what counts as correct. This post lays out what the published benchmarks actually measure, why the vendor numbers are not comparable, and how to test a tool on your own incidents.

We sell an AI SRE agent. Read the section on our own gap before you decide how much to trust the rest.

What is the highest independently verified AI SRE accuracy?

56.2%, on ITBench-AA.

That evaluation matters because it is the only leaderboard in this category where an independent party grades the work. Artificial Analysis runs the harness; the task set comes from IBM Research's ITBench team. The setup, in their words: "We evaluate 59 Kubernetes incident tasks (40 from IBM's public release and 19 private shared with us by the ITBench team), with 3 repeats per task."

The metric is unusually honest, and it is the reason the numbers look low:

"The headline score is the average precision at full recall. For each task, the model scores TP / (TP + FP) when there are zero false negatives, and 0 otherwise."

Miss a single ground-truth root cause and the run scores zero. Name extra causes that are not real and your precision drops. This is the only benchmark in the category that punishes over-reporting, which is precisely the behavior that destroys on-call trust.

Here is the full published record, with what each number actually measures.

Source	Result	What it measures	Independent?
ITBench-AA	56.2% top model	Precision at full recall, 59 K8s incident tasks, 3 repeats	Yes, third-party harness
OpenRCA 2.0	20.7% exact, 61.5% grounded	Exact root-cause set, and whether a causal path is verified	Yes, independent academic preprint
ITBench	13.8% of SRE scenarios	End-to-end scenario resolution, 94 scenarios total	Yes, IBM + UIUC
Meta production	42% top-5	Root cause in top five suggested code changes, one monorepo	First-party, method disclosed
Microsoft RCACopilot	up to 0.766	Root-cause category prediction, a year of real incidents	Peer-reviewed
Rootly SRE-skills-bench	90.4% (Sonnet-4.6)	Multiple-choice SRE knowledge, not live incidents	Vendor-run, open data
Vendor marketing	92% to 95%	Undisclosed	No

Why do vendors claim 90%+ when benchmarks show 20% to 56%?

Three reasons, and only one of them is dishonest.

First, the criterion moves. OpenRCA 2.0 demonstrates this mechanically. Ask whether the agent named a service involved in the incident and you get 76.0%. Ask whether it grounded that service in "a verified causal propagation path to the observed symptom" and you get 61.5%. Ask for the exact root-cause set and you get 20.7%. Same runs, same agents, three very different headlines. The authors named the failure mode ungrounded diagnosis, and their conclusion is the sentence every buyer should read: "Outcome-only evaluation hides this failure mode."

Second, some benchmarks measure knowledge rather than diagnosis. Rootly's 90.4% is a real, reproducible number on an Apache 2.0 benchmark, and it deserves credit for existing at all. But read what it tests: their own framing is that it covers "understanding infrastructure code, reasoning about cloud configurations, and mapping code diffs to real-world pull requests." Their Incident Response track, the one that would replay real postmortems as live scenarios and grade an agent across detect, localize, diagnose, mitigate and verify, is marked Planned. So 90.4% and 20.7% are not a contradiction. They measure different things, and only one of them is the product. Rootly's own README says there is "no single blended number," which is the correct design choice.

Third, and this is the real problem: most vendor numbers have no denominator. We checked five live vendor pages. Not one publishes the denominator behind its headline figure, a definition of correctness, or a footnote. Cleric alone publishes a volume figure, "200,000+ Production-Grade Investigations", but not the sample the 92% is drawn from. Traversal advertises "82% Root Cause Analysis (RCA) accuracy," attributed to a Fortune 100 financial services customer. Cleric advertises "92% Actionable Findings," and note that actionable is not correct. NeuBird advertises "92% MTTR reduction." Resolve.ai advertises "up to 5x faster MTTR." incident.io advertises "Resolve incidents 5x faster."

To be fair to them: these are not necessarily false. They are unfalsifiable, which is different. When a vendor never defines "accuracy" or "actionable," there is no claim to disprove. That is itself the finding. And one vendor does better than the rest: Traversal's benchmarking writeup describes its grading protocol, including that "Traversal SRE researchers and customer SREs manually validated a randomly sampled subset of judgments." They publish no numbers from it, but the methodological candour is real and worth acknowledging.

Will the next frontier model fix this?

Probably not on its own, and there is now direct evidence.

Researchers ran 1,675 agent runs across five LLMs on OpenRCA and classified the failures into 12 pitfall types. The finding that matters for anyone planning to wait out the problem:

"the most prevalent pitfalls, notably hallucinated data interpretation and incomplete exploration, persist across all models regardless of capability tier, indicating that these failures originate from the shared agent architecture rather than from individual model limitations."

They also tested the obvious workaround and found it insufficient: "prompt engineering alone cannot resolve the dominant pitfalls," while enriching the inter-agent communication protocol cut communication failures by up to 15 percentage points. The architecture is the lever, not the model.

Will connecting more telemetry fix it?

This is the standard pitch in the category, and a July 2026 paper tested it directly. The authors built a reverse-reasoning agent to separate failures where evidence was present but unused (a Reasoning Gap) from failures where the evidence genuinely was not there (Data Ambiguity). Their conclusion:

"the required evidence is present in the vast majority of failures: the bottleneck is not data access but the agent's ability to reason over it correctly."

If you are being sold accuracy on the basis of more integrations, that is the paper to bring to the call.

Does a top leaderboard score mean the tool will work on your stack?

No, and this is the most practically useful result in the 2026 literature.

An audit of pooled leaderboard reporting covered OpenRCA, RCAEval and PetShop across 11 subsystems and 778 matched scoring units. Leave-one-system-out selection, which simulates a buyer picking the leaderboard winner for their own unseen environment, "picks the lower-scoring method on up to 5 of 11 held-out subsystems, with regret reaching 24.8 pp on RCAEval / Sock-Shop." Every pairwise comparison showed subsystem-level effects in both directions.

Translated: choosing the top-ranked method can hand you the worse one for your estate, and be wrong by nearly 25 percentage points. The audit compares statistical RCA methods rather than commercial agents, but the selection failure mode is structural. Benchmarks tell you the category ceiling. They do not tell you what will work on your services. That is what your own test is for.

How should you test an AI SRE tool on your own incidents?

There is no vendor-neutral published methodology for this. We looked. So here is one assembled entirely from primary sources, with each step traced to who established it.

1. Build a holdout set from resolved incidents, stratified. Google's SRE organization describes using "stratified sampling to continuously surface a diverse subset of incidents for manual review, creating Gold data," where Gold means "Verified by human experts." Stratify across service, severity and failure class, or you will score the tool on forty variations of one disk-full alert.
2. Freeze information at investigation start. Meta's protocol is the citable precedent: accuracy measured "with historical investigations and the information available at their start." Skip this and you leak the answer, because the postmortem is sitting in the ticket.
3. Grade the causal path, not just the named service. Score every output twice: did it name the right thing, and did it show a chain of evidence a reviewer accepts. The delta between those two numbers is your real trust metric, and OpenRCA 2.0 shows it is roughly 15 points wide even for frontier models.
4. Require reasoning logs as an evaluation input. Google's stated requirement is that agents "must log their 'chain of thought', the signals used, the hypotheses considered, the reasons for choosing a particular action, and the confidence level." Without this you cannot perform step 3.
5. Score per service, never pooled. Per the leaderboard audit above, one blended number for your whole estate will mislead you exactly the way a public leaderboard does.
6. Count confident-wrong separately from inconclusive. Three buckets: correct, inconclusive, confidently wrong. Weight the third heavily. An agent that abstains costs you nothing; an agent that is assertively wrong costs you the incident.
7. Run at Google's L1 before granting more. In Google's graded autonomy ladder, L1 means the agent provides "insights or suggestions" while a human must "Approve any action, manually Actuate it." Promote only after the tool clears your bar on the holdout set.
8. Re-run on a schedule, with cost attached. HolmesGPT's dated eval history is the model to copy here: its September 28, 2025 full benchmark across 105 test cases published accuracy and dollar cost per model side by side, from sonnet-4 at 94% and $0.16 per case down to gpt-4o at 62% and $0.14 per case. Accuracy without cost per investigation is not a decision input.

Why is a wrong answer worse than no answer?

Because your responders are documented to be unable to resist it, and that finding is 16 years old.

Parasuraman and Manzey's review of automation bias and complacency in Human Factors (2010) is the canonical citation. Their findings, verbatim: automation bias "results in making both omission and commission errors when decision aids are imperfect," it "occurs in both naive and expert participants, cannot be prevented by training or instructions, and can affect decision making in individuals as well as in teams." Automation complacency, separately, "cannot be overcome with simple practice."

Seniority does not protect your on-call engineer from a confident wrong root cause at 3am. Training does not either. This is why the roughly 15% of cases where an agent names a correct service it cannot justify are not a rounding error, and why an agent that can return inconclusive is worth more than one that always answers. We treat that as an architectural requirement rather than a prompt instruction: Aurora's sub-agent findings are schema-validated and rejected unless they include an explicit "what I ruled out" section and a self-assessed confidence drawn from a fixed set, and a sub-agent that times out returns an honest inconclusive stub rather than silence.

One caveat we should state rather than paper over: this research comes from aviation and process control, not from SRE. We found no SRE-specific empirical study of automation bias during incident response. The transfer is reasonable but not established.

Even Meta, publishing a genuinely useful 42%, was explicit about the risk: their system "can potentially suggest wrong root causes and mislead engineers," which is why they "prioritize closed feedback loops and explainability."

What autonomy level is defensible today?

Google's graded autonomy framework is the most credible published answer, partly because Google ships an agent that gets things wrong and documents it.

The ladder runs L0 (everything human) through L4 (the agent devises and executes multi-step resolutions). L2 means a human "must explicitly Approve any plan before the system proceeds with an actuation." L3 means the agent "can independently detect, decide, and act without human approval for bounded actions."

What makes it credible is the machinery around L3, not the definition of it:

• Automatic de-escalation. "If an agent requests an L3 (High Automation) execution, but the agent detects an elevated risk score or an anomalous production state, it will automatically downgrade the request to L2 (Partial Automation), intercepting the execution and routing an approval request to a human SRE."
• A global stop. "emergency 'Red Button' endpoints that allow SREs to instantly pause all in-flight agentic actions, block new actions, or globally revoke L3 permissions across the fleet during catastrophic, complex outages."
• A published failure. In their own evaluation figure, "the agent incorrectly diagnosed the root cause and failed to mitigate it," after which an LLM-as-judge "automatically generated a critique of the agent's logic and filed a bug."

An organization that publishes its agent's failures and designs containment for them is further along than one advertising up to 95% accuracy. For the layered version of this argument, see our AI agent guardrails post, and for the evaluation framework beyond accuracy, how to evaluate an AI SRE platform.

What Aurora has not published

The obvious question after 3,000 words of this: what does Aurora score?

We have not published a benchmark score. We should be precise about why, because "we are working on it" is what every vendor says.

Aurora has no investigation-quality evaluation harness. There is no golden dataset in the repository, no scored run history, no baseline. We have a 955-test suite covering guardrails, correlation math, multi-tenant isolation and adversarial prompt injection, and we have CI-enforced architectural invariants that fail the build if a new tenant-scoped table is not row-level-security protected. None of that measures whether an investigation reaches the right conclusion. Those are different claims and we are not going to blur them.

So by the standard this post applies to everyone else, Aurora belongs in the same bucket: a vendor with no independently verified accuracy number. The difference we can point to is that the whole system is Apache 2.0, so the investigation prompts, the confidence classification, the findings schema and the guardrail behavior are all readable rather than described. You can run step 1 through step 8 above against it without asking us for permission or a trial key.

ITBench-AA's 40 public tasks are available today, and OpenRCA 2.0's 500 instances build on the public OpenRCA dataset. The first commercial AI SRE agent to publish a reproducible score on either becomes the reference point in a category where nobody has shown their work.

The honest summary

The published evidence says AI agents are useful at incident investigation and much weaker than the marketing implies. Independent ceiling 56.2%. Exact root-cause recovery 20.7%. Grounded diagnosis 61.5%. Meta's 42% top-5 in a single monorepo. Microsoft's 0.766 on the easier categorisation task.

Against that: 92%, up to 95%, 5x, all undefined.

The useful posture is neither dismissal nor faith. It is a holdout set, a two-part grade that separates naming from proving, a per-service score, and an autonomy level you raise only when the evidence earns it.

Grade it yourself on your own holdout set

Since we are not asking you to believe a number we have not published, the consistent thing is to make ours testable. Aurora is Apache 2.0 and runs locally:

git clone https://github.com/arvo-ai/aurora.git && cd aurora

make init                # generates secrets, copies .env.example to .env
nano .env                # add one LLM API key; cloud connectors are optional
make prod-prebuilt       # pulls prebuilt images and starts

Open http://localhost:3000 and register as admin. Then run the eight-step protocol from earlier in this post against your own resolved incidents, holding back a set the agent has never seen.

Decide before you start whether an inconclusive result counts as a failure or as a correct abstention, because that choice sets the denominator, and it is exactly the denominator the vendor claims in this post leave undefined. Aurora treats an honest "insufficient evidence" as a successful outcome rather than a failed run, so score it accordingly. Grade naming and proving separately, the way OpenRCA 2.0 does, or your score will flatter the agent by counting a correct service name as a solved incident.

• Start free: aurora-ai.net (hosted, no infrastructure to run)
• GitHub: github.com/Arvo-AI/aurora
• Related guides: How to Evaluate an AI SRE Platform · AI Agent Guardrails · What Is an AI SRE? · AI-Powered Incident Investigation

All claims sourced. Benchmark results link to the arXiv papers and leaderboards; vendor claims link to the live vendor page carrying them; the automation-bias finding links to the peer-reviewed review in Human Factors. Figures verified July 28, 2026. Leaderboards update, so timestamp any number you quote from them.