AI SRE benchmark

19 Kubernetes faults. 3 agents. Every trace public.

We break a small app on a kind cluster in known ways, hand each agent only the alert, and grade the root cause against the fault we injected. A judge model scores first, then a person checks every verdict. Misses are published with the hits. Arvo builds Aurora and wrote the scenarios; read the numbers with that in mind.

Auroracommit 87d22baf

89%

34 correct, 4 partial, 0 wrong of 38 graded runs

median 154 s · $0.35 a run

bedrock/us.anthropic.claude-sonnet-5

HolmesGPT0.43.0

74%

14 correct, 4 partial, 1 wrong of 19 graded runs

median 45 s

bedrock/us.anthropic.claude-sonnet-5

K8sGPT0.4.40

0 correct, 11 partial, 8 wrong of 19 graded runs

median 13 s

us.anthropic.claude-sonnet-4-5 (K8sGPT sends temperature, which Bedrock rejects for Sonnet 5)

#	Fault	Alert the agent receives	Aurora	HolmesGPT	K8sGPT
01	Worker OOMKilled after a batch size change	KubePodCrashLooping	correct correct160 s	correct39 s	partial15 s
02	Log shipper sidecar OOMKilled, app container healthy	KubePodCrashLooping	correct correct90 s	correct39 s	partial32 s
03	Frontend rollout stuck on a mistyped image tag	KubeDeploymentRolloutStuck	correct correct89 s	correct22 s	partial17 s
04	New api pods blocked on a Secret that does not exist	KubeDeploymentRolloutStuck	correct correct164 s	correct31 s	partial14 s
05	Frontend crash loop from an invalid nginx config	KubePodCrashLooping	correct correct136 s	correct49 s	partial15 s
06	api restarts because the liveness probe targets the wrong port	KubePodCrashLooping	correct correct105 s	correct43 s	no_answer
07	api Service selects no pods after a selector change	HighErrorRate	correct correct157 s	correct55 s	partial15 s
08	Worker pod Pending, CPU request larger than any node	KubePodNotReady	correct correct226 s	correct43 s	partial12 s
09	api failing because redis is Pending on an unbindable PVC	KubePodCrashLooping	correct correct281 s	correct104 s	partial45 s
10	api crash loop from a wrong REDIS_HOST	KubePodCrashLooping	correct correct245 s	partial45 s	wrong13 s
11	frontend 5xx from a wrong api Service targetPort, next to a noisy but harmless worker	HighErrorRate [critical] namespace=shop service=frontend	correct partial340 s	partial65 s	no_answer
12	Two faults at once, worker OOM and a frontend Service selecting no pods; the alert covers only the worker	KubePodCrashLooping [warning] namespace=shop deployment=worker	partial partial344 s	partial26 s	partial23 s
13	frontend 5xx because a NetworkPolicy blocks api to redis	HighErrorRate [critical] namespace=shop service=frontend	correct correct323 s	correct110 s	no_answer
14	api rollout stuck because a ResourceQuota forbids the surge pod	KubeDeploymentRolloutStuck [warning] namespace=shop deployment=api	correct correct126 s	correct41 s	partial9 s
15	One in three api requests fails because a stray Deployment's pods carry app=api	HighErrorRate [warning] namespace=shop service=frontend	correct correct147 s	correct97 s	no_answer
16	New api pods crash because their dnsConfig search domains are for another cluster domain	KubePodCrashLooping [warning] namespace=shop deployment=api	correct correct173 s	correct82 s	wrong16 s
17	Worker OOMKilled every few minutes by a slow leak in its code; logs look clean	KubePodCrashLooping [warning] namespace=shop deployment=worker	correct correct144 s	correct72 s	no_answer
18	api rollout stuck on an init container waiting for a Service that does not exist	KubeDeploymentRolloutStuck [warning] namespace=shop deployment=api	correct correct159 s	partial43 s	partial12 s
19	Half of frontend requests fail; one frontend pod loaded an edited ConfigMap, the other still runs the old config	HighErrorRate [warning] namespace=shop service=frontend	partial correct267 s	wrong49 s	no_answer

How a run works

kind v0.33.0, Kubernetes v1.37.0, 3 nodes. The shop app: nginx, a Python API, a worker, redis, a load generator. Synthetic, no customer code.
One fault is injected with the scenario's break.sh; check.sh proves it is visible.
Each agent gets the alert text and read-only cluster access. Aurora runs under its default observability_only policy.
Correct means every must_identify item in the scenario is stated and no listed wrong answer is asserted. Judge: us.anthropic.claude-opus-5-5, then a person.

Aurora's verdicts were checked by a person and come from two repeats per scenario. HolmesGPT's and K8sGPT's are the judge model's only, from one repeat each, until a person reviews them; the second repeat and the review are next.

What is not comparable yet: K8sGPT has no notion of an alert, it scans the namespace. It also ran on Sonnet 4.5 because it always sends a temperature parameter that Bedrock rejects for Sonnet 5. Costs are recorded for Aurora only.

Rerun it, or add your agent

Everything is in Arvo-AI/aurora-lab: scenarios, the runner, every transcript and grade, and this table as results.json.

git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab
./lab.sh up && ./lab.sh break 03-bad-image-tag
./runner/h2h.sh 03-bad-image-tag runs/mine

To put another agent on the board, open an issue titled "Benchmark: <agent>" with a script that takes the alert text and prints the root cause. We run it on all 19 faults and publish the result, whatever it is. Submit an agent.

Read the full write-up: lab report v1. Fix the faults yourself: runbooks, each with Aurora's real trace.

Results updated 2026-10-15. Scenario and tool versions are recorded with every run in the repo.