Can You Measure an AI SRE's Accuracy? The Ground-Truth Gap

Key Takeaways

• You cannot check an AI SRE's answer at the moment it answers, because the ground truth is not in yet. As Cleric wrote in July 2026: "When the agent finishes, that verdict is not in yet, so there is no ground truth to check the investigation against."
• Delayed outcomes barely help. Cleric reports that real outcomes "arrive days late and only for a fraction of investigations," so "learning from them is slow, weeks to months."
• Asking an LLM to score an investigation on its own performed no better than chance. Cleric's absolute scorer "reproduced the recall ordering 27% of the time, which makes it indistinguishable from guessing," against a "random guessing sits at 33%" baseline.
• Even the method that worked never proves correctness. Their pairwise judge "never claims an investigation was right, only that it beat or lost to the ones it was paired with."
• Competitors now agree the problem is real. Rootly, announcing an acquisition in July 2026, wrote: "An AI SRE you cannot measure is a black box wearing a confident UI," and "Using AI to judge AI is like asking the same student to mark their own exam."
• Independent benchmarks put a ceiling on the confidence any vendor should project. On OpenRCA 2.0, agents recover "the exact root-cause set" in "only 20.7% of cases," and on the ITBench-AA leaderboard the top model scores 56.2%.
• This is why Aurora publishes no accuracy or MTTR-reduction figure. No investigation-quality eval harness that survives scrutiny exists, so a number would be invented. The rest of this post explains what to ask for instead.

Measuring whether an AI SRE agent is actually correct is an unsolved problem, because at the moment an investigation finishes there is no ground truth to check it against, and the real outcome arrives too late and too rarely to grade on. That is not an Aurora opinion. It is the conclusion two competitors reached publicly in July 2026, one while explaining why it built new tooling and one while explaining why it bought a company to do it.

This post explains why accuracy is so hard to measure, why "our agent is X% accurate" claims should be treated skeptically, and what a buyer can verify instead of a number.

Why can't you measure an AI SRE's accuracy directly?

Because the answer key does not exist when the agent answers. A human SRE closing an incident has the resolution as a reference point. An agent finishing an investigation does not. Cleric put the constraint plainly: "When the agent finishes, that verdict is not in yet, so there is no ground truth to check the investigation against."

The obvious fix, wait for the real outcome and grade against it, fails on availability. Cleric reports that outcomes "arrive days late and only for a fraction of investigations," and that because "they arrive late and only for some investigations, so learning from them is slow, weeks to months." A grading signal that covers a fraction of cases and lags by weeks cannot drive day-to-day quality claims.

There is a second, subtler problem. The observable outcomes, such as whether the alert stopped firing and the system recovered, do not establish that the agent's reasoning was correct. An incident can resolve for reasons unrelated to the agent's stated cause. Correct outcome and correct diagnosis are different things.

What happened when a vendor tried to grade its own agent?

It scored no better than guessing. Cleric's most useful disclosure is about a method that did not work. They built an "absolute scorer," an LLM asked to rate an investigation's quality on its own, and found "almost every investigation landed in the 80-100% range." When they checked whether that score tracked reality, the scorer "reproduced the recall ordering 27% of the time, which makes it indistinguishable from guessing," where "random guessing sits at 33%."

That result is worth sitting with. An AI asked to grade an AI defaulted to high, confident scores that carried almost no information. Rootly reached the same conclusion in different words: "Using AI to judge AI is like asking the same student to mark their own exam."

What Cleric found did work was relative, not absolute. A pairwise judge that compares two investigations and picks the better one, fitted with "a Bradley-Terry model" converted "to an Elo rating," "reproduced it 67% of the time" and "closes about half the gap between chance and a full-information oracle." That is a genuine result, and Cleric is careful about its limit: the judge "never claims an investigation was right, only that it beat or lost to the ones it was paired with." Ranking is not correctness.

Why did competitors start talking about this in 2026?

Because measurement became the reliability frontier. In July 2026 Rootly announced it "acquired ThinkHive," an AI-agent-observability team, and framed the reason around a gap in visibility: "A service going down trips every alarm you own. An AI agent that quietly starts giving wrong answers trips nothing." Their conclusion, "An AI SRE you cannot measure is a black box wearing a confident UI," is the whole category's problem stated by one of its vendors.

When two independent competitors publicly agree that grading an AI SRE is hard, and one of them spends money to acquire the capability, a buyer should downgrade trust in any tidy accuracy percentage. The convergence is the signal.

What do independent benchmarks say the ceiling is?

Third-party benchmarks are the only measurements not produced by the vendor being measured, and they put a hard ceiling on plausible confidence.

Benchmark	What it measures	Best result	Source
OpenRCA 2.0	Recover the exact root-cause set (500 instances, 11 LLMs)	20.7%	arXiv:2606.27154
OpenRCA 2.0	Name at least one correct root-cause service	76.0%	arXiv:2606.27154
OpenRCA 2.0	Ground that service in a verified causal path	61.5%	arXiv:2606.27154
ITBench-AA	Avg precision at full recall, 59 Kubernetes tasks	56.2%	Artificial Analysis

Two caveats keep this honest. OpenRCA 2.0 is a preprint marked work in progress, not peer-reviewed, and it evaluates foundation models rather than any shipped vendor product. ITBench-AA likewise scores foundation models through its own harness, and no commercial AI-SRE vendor has ever submitted its own product to an independent third-party benchmark. So these numbers bound what the underlying models can do on curated tasks; they are not a leaderboard of the products buyers actually purchase. The naming-versus-proving gap on OpenRCA, 76.0% down to 61.5%, is the more instructive figure than any single score: agents point at the right service far more often than they can prove why. The full landscape is covered in AI SRE accuracy: what the benchmarks show.

Why doesn't Aurora publish an accuracy or MTTR-reduction number?

Because it would be invented. There is no investigation-quality eval harness, golden dataset, or benchmark suite that produces a defensible accuracy or mean-time-to-resolution figure for Aurora, and the problem above is exactly why: without ground truth at answer time, any single number is either a self-graded score of the kind Cleric showed is indistinguishable from guessing, or a marketing figure with no method behind it.

That is a deliberate choice, not a gap to paper over. A vendor that hands you "94% accurate" for an investigation product is claiming to have solved a problem that two competitors and a peer-reviewed-adjacent benchmark literature say is unsolved. The honest position is to publish the controls you can verify and let the buyer test the behavior.

What should you verify instead of an accuracy percentage?

Ask for properties you can inspect, not a score you have to trust.

1. Does it separate symptom from cause, and say so? A finding that stops at "the pod is OOMKilled" is a symptom. The mechanism is the claim worth checking. Aurora's investigation prompts encode this distinction in the open-source repository rather than asserting it in marketing.
2. Does it state what it ruled out and how confident it is? Aurora rejects sub-agent findings unless they include a "What I ruled out" section and a self-assessed strength of strong, moderate, weak, or inconclusive; a timeout yields an honest inconclusive stub rather than a confident guess.
3. Can you read the reasoning, not just the verdict? A confident wrong answer for a system nobody on the team wrote is worse than no answer, because there is no independent basis to doubt it. Inspectable evidence is the antidote.
4. Does it gate action on a human? Measurement uncertainty is survivable when an agent proposes and a person approves. See who is accountable when an AI agent changes prod.
5. Can you run it on your own incidents? The only accuracy assessment that means anything for your environment is the one you produce on your own systems. That is the point of the open-source install below. For the buyer's full checklist, see how to evaluate an AI SRE platform.

Grade it on your own incidents

Aurora is Apache 2.0, so instead of trusting a vendor's accuracy figure you can run it against incidents you already understand and judge the reasoning yourself. One LLM API key is the only hard requirement; cloud connectors are optional.

git clone https://github.com/arvo-ai/aurora.git && cd aurora

make init                # generates secrets, copies .env.example to .env
nano .env                # add OPENROUTER_API_KEY (or OPENAI_API_KEY / ANTHROPIC_API_KEY)
make prod-prebuilt       # pulls prebuilt images from GHCR and starts

Open http://localhost:3000. The first user to register becomes admin.

Replay an incident whose root cause you already know, and check whether the investigation separated symptom from mechanism, stated what it ruled out, and flagged its own confidence. That is a measurement you can trust, because you hold the ground truth.

• Start free: aurora-ai.net (hosted, no infrastructure to run)
• GitHub: github.com/Arvo-AI/aurora
• Related guides: AI SRE Accuracy: What the Benchmarks Show · How to Evaluate an AI SRE Platform · AI Agent Accountability and Audit Evidence · Root Cause Analysis for SREs

Sourcing note. The ground-truth argument, the absolute-scorer result (27% versus a 33% chance baseline), the pairwise Bradley-Terry and Elo method, and the "never claims an investigation was right" limit all come from Cleric's own engineering post dated 23 July 2026. The measurement-frontier framing and the two quoted lines come from Rootly's acquisition announcement dated 28 July 2026. Benchmark figures come from the OpenRCA 2.0 preprint (arXiv:2606.27154) and the Artificial Analysis ITBench-AA leaderboard. This post publishes no Aurora accuracy or MTTR figure, because no eval harness exists to produce one honestly. Verified 24 August 2026.