# Aurora SRE > Aurora SRE (Aurora by Arvo AI, not Amazon Aurora) is an open-source (Apache 2.0) AI-powered agentic incident management and root cause analysis tool for Site Reliability Engineers (SREs). It uses LangGraph-orchestrated LLM agents to autonomously investigate cloud incidents across AWS, Azure, GCP, OVH, Scaleway, and Kubernetes. ## Company - Name: Arvo A.I. Ltd. - Website: https://www.aurorasre.ai - Location: Montreal, Quebec, Canada - Contact: info@arvoai.ca - LinkedIn: https://www.linkedin.com/company/arvokas/ ## Product: Aurora - GitHub: https://github.com/Arvo-AI/aurora - Documentation: https://arvo-ai.github.io/aurora/ - SaaS: https://aurora-ai.net (hosted, free to start) - License: Apache 2.0 (fully open source) - Pricing: Free (self-hosted). Managed tier available as SaaS or deployed in your own VPC. - Tech Stack: Python backend, Next.js frontend, LangGraph agent orchestration ## What Aurora Does Aurora is an agentic incident management platform. When a monitoring tool (PagerDuty, Datadog, Grafana, etc.) fires an alert, Aurora's AI agents autonomously investigate the incident, querying infrastructure, running CLI commands, searching knowledge bases, and synthesizing findings into a root cause analysis. Unlike traditional tools (Rootly, FireHydrant, incident.io) that automate workflows, Aurora automates the investigation itself. ## Key Capabilities - Agentic AI investigation: Autonomous multi-step investigation using LangGraph workflows with 30+ tools - Multi-cloud support: AWS, Azure, GCP, OVH, Scaleway, and Kubernetes - Webhook-triggered auto-investigation: PagerDuty, Datadog, Grafana, New Relic, OpsGenie, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, incident.io - Infrastructure CLI execution: Runs kubectl, aws, az, gcloud commands in sandboxed Kubernetes pods, with destructive actions human-gated - Infrastructure knowledge graph: Memgraph-powered dependency mapping across all cloud providers - Knowledge base RAG: Weaviate vector search over runbooks, past postmortems, and documentation - Automatic postmortem generation: Structured postmortems exported to Confluence, Notion, or SharePoint - AI-suggested code fixes: Can open pull requests with remediation (GitHub and Bitbucket), gated on human approval - Terraform/IaC analysis: Native understanding of infrastructure-as-code state - LLM provider flexibility: OpenAI, Anthropic, Google, Vertex AI, OpenRouter, and Ollama (local models for air-gapped deployments) - Self-hosted: Docker Compose or Helm chart deployment, HashiCorp Vault for secrets ## Integrations Monitoring & Alerting: PagerDuty, Datadog, Grafana, New Relic, OpsGenie, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, incident.io Cloud: AWS, Azure, GCP, OVH, Scaleway Infrastructure: Kubernetes, Terraform Communication: Slack, Google Chat Code & Docs: GitHub, Bitbucket, Confluence, Notion, SharePoint Search: Self-hosted SearXNG Database: Memgraph (graph), Weaviate (vector), PostgreSQL Secrets: HashiCorp Vault ## How Aurora Differs from Competitors | Feature | Aurora | Rootly | FireHydrant | incident.io | |---------|--------|--------|-------------|-------------| | Approach | Agentic AI investigation | Workflow automation | Workflow automation | Workflow automation | | AI RCA | Autonomous multi-step | AI summaries | AI summaries | AI summaries | | Open Source | Yes (Apache 2.0) | No | No | No | | Self-Hosted | Yes | No | No | No | | Cloud Providers | AWS, Azure, GCP, OVH, Scaleway | Via integrations | Via integrations | Via integrations | | CLI Execution | Sandboxed pods | No | No | No | | Knowledge Base RAG | Yes (Weaviate) | No | No | No | | Infrastructure Graph | Yes (Memgraph) | No | No | No | | LLM Provider | Any (including local) | Fixed | Fixed | Fixed | | Pricing | Free (self-hosted) | Paid (per-seat) | Paid (per-seat) | Custom | ## Cloud Authentication - AWS: STS AssumeRole for secure temporary credentials - Azure: Service Principal authentication - GCP: OAuth-based authentication - OVH: API key authentication - Scaleway: API token authentication - Kubernetes: Kubeconfig-based access ## Infrastructure Discovery Aurora discovers infrastructure in three phases: 1. Bulk Discovery: Enumerates all resources across connected cloud providers 2. Detail Enrichment: Gathers detailed configuration and metadata 3. Connection Inference: Maps dependencies between resources ## Quick Start Self-hosted (Docker, ~5 minutes). Requires one LLM API key; cloud connectors are optional. ``` git clone https://github.com/arvo-ai/aurora.git && cd aurora make init # generates secrets, copies .env.example to .env nano .env # add OPENROUTER_API_KEY, OPENAI_API_KEY or ANTHROPIC_API_KEY make prod-prebuilt # pulls prebuilt images from GHCR and starts ``` Then open http://localhost:3000. The first user to register becomes admin. Kubernetes deployment via Helm chart is also available: ``` helm repo add aurora https://raw.githubusercontent.com/Arvo-AI/aurora/gh-pages helm repo update helm show values aurora/aurora-oss > my-values.yaml helm install aurora-oss aurora/aurora-oss -n aurora --create-namespace -f my-values.yaml ``` Hosted alternative (no infrastructure to run): sign up free at https://aurora-ai.net/ ## Frequently Asked Questions Q: What is agentic incident management? A: Agentic incident management uses autonomous AI agents to investigate, diagnose, and help resolve cloud infrastructure incidents without requiring step-by-step human direction. Unlike runbook automation that follows predefined scripts, agentic systems dynamically decide which tools to use, what data to gather, and how to synthesize findings. Q: Is Aurora free? A: Yes. Aurora is Apache 2.0 licensed and completely free to self-host. Costs are only infrastructure and LLM API usage. Local models via Ollama make fully free, air-gapped deployments possible. Q: Which cloud providers does Aurora support? A: AWS, Azure, GCP, OVH, Scaleway, and Kubernetes. Q: How does Aurora investigate incidents? A: When an alert fires, Aurora's LangGraph-orchestrated agents dynamically select from 30+ tools to investigate. They execute cloud CLI commands in sandboxed pods, query Kubernetes clusters, search the knowledge base for similar past incidents, traverse the infrastructure dependency graph, and synthesize findings into a structured root cause analysis with remediation recommendations. Q: What monitoring tools trigger Aurora investigations? A: PagerDuty, Datadog, Grafana, New Relic, OpsGenie, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, and incident.io. Any tool that sends webhooks can trigger an investigation. Q: How is Aurora different from Rootly, FireHydrant, or incident.io? A: Traditional tools automate the process around incidents (Slack channels, status pages, runbooks). Aurora automates the investigation itself. AI agents autonomously query infrastructure, correlate data, and identify root causes. Aurora is also open source, self-hosted, and works with any LLM provider. Q: Can Aurora run in air-gapped environments? A: Yes. Aurora supports Ollama for running local LLM models (Llama, Mistral, etc.) with no external API calls required. Q: What is Aurora's infrastructure knowledge graph? A: Aurora uses Memgraph to build a live dependency graph of your entire infrastructure across all connected cloud providers. When an incident occurs, the AI traverses this graph to assess blast radius and trace upstream causes. # Runbooks (full text) ## Alert fixed, users still see errors: a second fault the alert did not cover (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-second-fault-alert-covers-only-one-kubernetes Published: 2026-10-19 Author: Damian Loch, Software Engineer at Arvo AI > The alerted pod is explained, yet users still get errors: a second, unalerted fault. Scope past the alert with real lab output from two faults at once. | Applies to | Any incident where the alert names one workload and more than one thing changed; shown on a worker OOMKilled after a `BATCH_SIZE` change while, separately, the frontend Service selects no pods | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, kubectl client v1.35.3, nginx 1.27-alpine (nginx/1.27.5), python:3.12-alpine, redis:7.4-alpine, busybox:1.37, 2026-10-02 | | Lab folder | [scenarios/12-two-faults-oom-and-selector](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/12-two-faults-oom-and-selector), runs in [runs/v1-run1/12-two-faults-oom-and-selector](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/12-two-faults-oom-and-selector) | **When the alerted problem is explained or fixed and users still see errors, treat it as a second, independent fault until a request through the user path proves otherwise: an alert covers the object its rule watches, not the incident.** In the lab, `KubePodCrashLooping` fired for a worker OOMKilled after `BATCH_SIZE` went from 500 to 2500, while the frontend Service had been patched to select `app=frontend,tier=web`, a label no pod carries, so every user request failed and every other pod read `1/1 Running`. Confirm the second fault with `kubectl -n shop describe svc` (an empty `Endpoints:` line) and the client's own log. Fix each fault separately: remove the extra selector key, revert `BATCH_SIZE`, and verify both. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | Alerted pod explained or fixed, client log still failing | A second fault on a path the alert does not watch | The client's log, then `kubectl -n describe svc` | Find and fix it on its own | | `Endpoints:` empty in `describe svc`, EndpointSlice ENDPOINTS ``, pods `1/1 Running` | Service selector matches no pods | SELECTOR column of `get svc -o wide` against pod labels | Remove the extra selector key | | Frontend pods log only `/healthz` probes, no user requests | Traffic stops before the pods: Service, Ingress or DNS | `describe svc`, `get ingress` | Fix the routing object | | `Reason: OOMKilled`, `BATCH_SIZE: 2500`, limit `192Mi` | The alerted fault: working set raised above the limit | `describe pod`, ReplicaSet env diff | Revert `BATCH_SIZE`, or raise the limit with the reason stated | ## How do you know there is a second fault? Start from what a user sees, not from the alert. The lab's `shop` app has one user path: a load generator calls `http://frontend/api/items`, the `frontend` Service sends it to nginx, and nginx proxies `/api/` to the `api` Service. The worker is not on that path. The load generator's log, captured before any investigation (run r1): ```shell kubectl -n shop logs loadgen-7c6676864-7htc9 -c loadgen --tail=200 ``` ``` 2026-10-02T20:23:30Z GET /api/items -> 502 2026-10-02T20:23:32Z GET /api/items -> 200 2026-10-02T20:23:35Z GET /api/items -> timeout 2026-10-02T20:23:38Z GET /api/items -> timeout 2026-10-02T20:23:41Z GET /api/items -> timeout ``` Every one of the 17 lines after the 200, through `20:24:23Z`, reads `timeout`. Read that word carefully: the load generator prints `${code:-timeout}`, so it means "no HTTP status came back", whether the connection was refused or timed out. The failures arrive 3 s apart, against 2 s while HTTP statuses were coming back, so each took about a second, well under the `wget -T 3` limit. Next, the frontend pods' own logs. Neither sees a user request after 20:23:32; from then on there are only kubelet probes (frontend `785dp`, r1): ``` 10.244.1.36 - - [02/Oct/2026:20:23:32 +0000] "GET /api/items HTTP/1.1" 200 38 "-" "Wget" "-" 10.244.1.1 - - [02/Oct/2026:20:23:36 +0000] "GET /healthz HTTP/1.1" 200 3 "-" "kube-probe/1.37" "-" 10.244.1.1 - - [02/Oct/2026:20:23:41 +0000] "GET /healthz HTTP/1.1" 200 3 "-" "kube-probe/1.37" "-" ``` A frontend that logs nothing while its client fails is not receiving the traffic: the break is in front of the pods. Look-alikes: - **Startup 502s.** The 502s at 20:23:26 to 20:23:30 came before the api was Ready (its log opens with `redis connection failed (attempt 1/5): [Errno 111] Connection refused`). The lab's reset waits for a `-> 200` before breaking anything, which fits the lone 200 at 20:23:32. Errors that end before the first success are baseline. - **Fallout from the alerted fault.** The scenario lists "Treating the frontend failure as a consequence of the worker crash" as a wrong answer. Isolate, step 1, rules it out. ## How do you scope past the alert? Three read-only commands, the ones the lab used to capture its evidence, cover every Service, its endpoints and the client. First, selectors and EndpointSlices: ```shell kubectl -n shop get all,pvc,endpointslices,cm,secrets,networkpolicy,resourcequota -o wide ``` Trimmed to the Services and EndpointSlices (r1): ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/api ClusterIP 10.96.234.91 8080/TCP 60s app=api service/frontend ClusterIP 10.96.10.4 80/TCP 60s app=frontend,tier=web service/redis ClusterIP 10.96.143.63 6379/TCP 60s app=redis NAME ADDRESSTYPE PORTS ENDPOINTS AGE endpointslice.discovery.k8s.io/api-tpbr5 IPv4 8080 10.244.1.34,10.244.2.43 60s endpointslice.discovery.k8s.io/frontend-npfll IPv4 60s endpointslice.discovery.k8s.io/redis-t42cj IPv4 6379 10.244.2.42 60s ``` The Kubernetes [Debug Services](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/) guide says that when the ENDPOINTS column is ``, you "should check that the spec.selector field of your Service actually selects for metadata.labels values on your Pods". With kubectl v1.35.3 against server v1.37.0 the empty slice printed `` instead, so a script grepping for `` would have passed this namespace. Second, the Services in full: ```shell kubectl -n shop describe svc ``` ``` Name: frontend Namespace: shop Selector: app=frontend,tier=web Type: ClusterIP IP: 10.96.10.4 Port: 80/TCP TargetPort: 80/TCP Endpoints: Events: ``` (Trimmed; the `api` and `redis` blocks list their pod IPs.) The selector patch left no event, so `kubectl get events` cannot show this fault; both graded investigations below ran it and saw nothing about the frontend. Third, the client log, as above, for every user path in the namespace. The lab app has no Ingress (`app/shop.yaml` defines none); on your cluster add `kubectl get ingress -n ` and check each backend Service the same way (not run in the lab). **Since when.** The client's last 200 is at 20:23:32Z and its first failure at 20:23:35Z. The new worker pod started at 20:23:33 (run r1, step 6, `Start Time`). The lab's `break.sh` runs the two changes back to back, so both landed within seconds. The alert created the incident at 20:24:25: users had been failing for about 50 s, and the page was about something else. r2 matches: last 200 at 20:47:09Z, worker start 20:47:09, alert 20:48:01. **What not to infer.** Pod status at the same moment (r1, step 5, trimmed): ``` NAME READY STATUS RESTARTS AGE api-f64486c4c-cjj7k 1/1 Running 0 72s api-f64486c4c-z7r8q 1/1 Running 0 72s frontend-76d98597b9-785dp 1/1 Running 0 72s frontend-76d98597b9-ccq7s 1/1 Running 0 72s loadgen-7c6676864-7htc9 1/1 Running 0 72s redis-59458f55d8-sc7tj 1/1 Running 0 72s worker-86f68b9bf6-jr2kk 0/1 OOMKilled 3 (47s ago) 63s ``` Both frontend pods were `1/1 Running`, Ready, with zero restarts, while every request to the frontend failed. Readiness probes go from the kubelet straight to the pod IP (`kube-probe/1.37` in the log above), never through the Service. Pod status describes the pod, not whether anything routes to it. ## Isolate: one incident or two? Cheapest check first. Each step ends when its output answers the question. **1. Is the alerted component on the failing path?** Read the proxy config. The `frontend-nginx` ConfigMap (r1) has one upstream, `server api:8080;`, and `location /api/` does `proxy_pass http://api/;`. nginx calls the api and nothing else, and no Service selects `app=worker` (the three Services above). The worker cannot break `/api/items`. Two independent faults, two fixes. If the alerted component is on the path, fix it first and re-read the client before looking further. **2. The unalerted fault: selector against labels.** The frontend pods carry `app=frontend` and `pod-template-hash=76d98597b9` (`kubectl -n shop describe pods`, r1). The Service wants `tier=web` as well. The tracked version is in the Service's own annotation (`kubectl -n shop get deploy,svc,cm,pvc -o yaml`, r1): ``` kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"frontend","namespace":"shop"},"spec":{"ports":[{"port":80,"targetPort":80}],"selector":{"app":"frontend"}}} ... selector: app: frontend tier: web ``` Applied with `app: frontend`, live with `tier: web` added: an out-of-band edit, here `kubectl patch`. **3. The alerted fault: what grew the working set.** `describe pod` shows `Reason: OOMKilled`, `Exit Code: 137`, a `192Mi` limit and `BATCH_SIZE: 2500` (r1, step 6). The two ReplicaSets side by side (r1, step 9): ```shell kubectl get replicaset -n shop -l app=worker -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{.spec.template.spec.containers[0].resources}{"\n"}{.spec.template.spec.containers[0].env}{"\n\n"}{end}' ``` ``` worker-6cf88d4b58 {"limits":{"memory":"192Mi"},"requests":{"cpu":"50m","memory":"64Mi"}} [{"name":"BATCH_SIZE","value":"500"}] worker-86f68b9bf6 {"limits":{"memory":"192Mi"},"requests":{"cpu":"50m","memory":"64Mi"}} [{"name":"BATCH_SIZE","value":"2500"}] ``` The worker allocates `buf = [bytearray(100 * 1024) for _ in range(batch)]` (the `api-code` ConfigMap, r1 step 11): 2500 items of 100 KiB is about 244 MiB against 192Mi; 500 is about 49 MiB. The [OOMKilled runbook](/runbooks/fix-oomkilled-kubernetes) covers this fault in depth. ## What is the fix for each fault? Fix the user-facing fault first, then the alerted one, as two separate changes. **Frontend Service.** Save the current object, then remove the key that matches nothing: ```shell kubectl -n shop get svc frontend -o yaml > frontend-svc.yaml kubectl -n shop patch svc frontend --type=json -p='[{"op":"remove","path":"/spec/selector/tier"}]' ``` Use the JSON patch, not the obvious one. `kubectl patch svc frontend -p '{"spec":{"selector":{"app":"frontend"}}}'` is a strategic merge patch, and maps merge: `tier: web` stays. Re-applying the manifest from git keeps it too: for a map key in neither the file nor `last-applied-configuration`, the [declarative config docs](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/) give the action "Do nothing. Keep live value." `tier` is that key. Risk: low. A selector change restarts nothing; the EndpointSlice controller adds the matching pods. Rollback: the inverse patch, `kubectl -n shop patch svc frontend -p '{"spec":{"selector":{"tier":"web"}}}'`, which puts back exactly the injected fault, with `frontend-svc.yaml` as the record of what was there. If `tier=web` was deliberate, add the label to the frontend Deployment's pod template in the manifest (a rollout) and leave the Service alone. Labelling running pods with `kubectl label pods` does not survive the next rollout. **Worker.** Revert the variable with the verb that changed it: ```shell kubectl -n shop set env deploy/worker BATCH_SIZE=500 ``` Risk: a rollout of a single-replica worker that is already down. Rollback: set 2500 again. Re-applying the manifest also works here, since the file's `500` wins. If 2500 is a real requirement, raise `resources.limits.memory` above 244 MiB in the manifest instead, with the reason next to it. There is no separate mitigation step: each fix is already the smallest change that restores service, and neither needs a manual restart. ## How do you verify it worked? Both faults have to pass. The lab's own check for the broken state is two conditions joined by `&&`: ```shell kubectl -n shop get pods -l app=worker -o jsonpath='{.items[*].status.containerStatuses[0].lastState.terminated.reason}' | grep -q OOMKilled && [ -z "$(kubectl -n shop get endpointslices -l kubernetes.io/service-name=frontend -o jsonpath='{.items[*].endpoints[*].addresses[*]}')" ] ``` Run the two halves separately; each must now fail. The EndpointSlice query must print the frontend pod IPs (`10.244.1.35` and `10.244.2.45` in r1), and the worker query must print nothing. Then the client, which is the condition users care about: ```shell kubectl -n shop logs deploy/loadgen --tail=20 ``` Every line should end `-> 200` (the lab's reset uses the same command with `--tail=1` as its healthy gate); twenty lines is about 40 s. Then watch the worker through two batch cycles (5 s sleep each) at `1/1 Running` with no new restarts. These are pass conditions, not captured results. ## How do you prevent it? **Alert on the user path.** The client logged the outage from 20:23:35Z and nothing alerted on it. A synthetic request through the Service, such as a blackbox probe of `http://frontend.shop.svc/api/items`, catches what pod-level rules cannot. (Suggested; not part of the lab.) **Alert on Services with no ready endpoints.** The query below is explained in the [Service with no endpoints runbook](/runbooks/fix-service-no-endpoints-kubernetes), section "How do you prevent it?": ``` kube_endpoint_info unless on (namespace, endpoint) kube_endpoint_address{ready="true"} ``` The metric reads the v1 Endpoints API, which the API server flagged to K8sGPT during our head-to-head runs: "v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice". (Suggested; not run in the lab.) **Guardrail.** Change Services only through their pipeline, and in CI compare the SELECTOR column of `kubectl get svc -o wide` with the manifests, since a re-apply will not remove a stray key. **Triage rule.** No component goes into an incident report as "unaffected" without one request-path check: its Service's endpoints and its client's log. ## Common mistakes - **Closing the incident when the alert clears.** The alert was never about the users' outage. - **Calling a component healthy from `get pods`.** `1/1 Running` with zero restarts was true of the frontend for the whole outage. - **Re-applying manifests to undo drift.** It reverts the worker and leaves `tier: web` on the Service. - **Linking the two faults because they arrived together.** Show the path first; here the worker was not on it. ## What we saw in the lab Scenario 12 in the [Aurora lab](https://github.com/Arvo-AI/aurora-lab): the `shop` app on a three-node kind cluster, then `kubectl -n shop set env deploy/worker BATCH_SIZE=2500` and `kubectl -n shop patch svc frontend -p '{"spec":{"selector":{"app":"frontend","tier":"web"}}}'`. Aurora got the alert text, "KubePodCrashLooping [warning] namespace=shop deployment=worker: Pod shop/worker (container worker) is in waiting state (reason: CrashLoopBackOff).", plus read-only kubectl under its default `observability_only` policy. Two findings were required: the OOM from `BATCH_SIZE`, and, separately, the frontend Service with no endpoints. Two runs, each in a fresh organization. | Run | Verdict (human reviewed) | Time | Tool calls | Aurora cost | |---|---|---|---|---| | r1 | partial | 140 s | 18 | $0.36 | | r2 | partial | 548 s | 14 | $0.32 | Both runs got the alerted fault right and missed the other one. Neither ran `get svc`, `get endpoints` or the load generator's log. r1's 13 kubectl calls covered the pod list, the worker's logs, events, ReplicaSets, Deployment and ConfigMap, a blocked `rollout history`, and nodes; r2's 11 were the same kind. Both reports state the OOM well; r2: "at `BATCH_SIZE=2500` it consumes ~244MiB, exceeding the limit and triggering an OOM kill almost immediately after startup". Then both overreached. r1: "Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]." Citations 5 and 14 are its two `get pods` calls. r2: "Other workloads in the namespace (api, frontend pods) were running normally and unaffected [4]", where 4 is also `get pods`. Both scoped the impact to the worker: r1 "Impact is confined to the `worker` deployment in the `shop` namespace", r2 "Impact is scoped to the single-replica `worker` deployment in `shop`". The frontend was down to every client during both investigations. Neither "Not Checked" list mentions Services; r1's names rollout history, previous logs and past incidents. Missing a fault is one failure. Writing that the broken component is healthy is a worse one, because a reader acts on it. The judge's r1 note: "It never checked Services or endpoints, so it missed the second fault." An AI reviewer, a second instance of the judge's model, called the claim "false and based only on pod status" and caught smaller errors, such as "both cluster nodes" when there were three. A person at Arvo reviewed both and recorded `partial`. The separate fix suggestions were graded partial too: both proposed reverting `BATCH_SIZE` and neither touched the selector. On time: r1's first tool call came 5 s after the incident was created, r2's 7 min 30 s after, and r2 then concluded in about 98 s. The lab's [REPORT.md](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md) lists 12 r2 among runs that waited for a worker slot shared with prediscovery jobs, so its 548 s is mostly queue. We ran the same scenario through two other tools ([r1](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/h2h-v1/r1/12-two-faults-oom-and-selector) 2026-10-15, [r2](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/h2h-v1/r2/12-two-faults-oom-and-selector) 2026-10-16; judge and two AI reviewers, no person yet). HolmesGPT 0.43.0, on the same model as Aurora, also missed the Service in both runs (26 s and 34 s); r1: "Root cause confirmed: shop/worker is being OOMKilled." It made no claim about the frontend either way. K8sGPT 0.4.40 (on claude-sonnet-4-5) flagged it in both runs, "Service has no endpoints, expected label tier=web". It is never given the alert: `k8sgpt analyze --explain --namespace shop` checks every object in the namespace. It never explained the OOM, r1: "Container ran out of memory (OOM - Out Of Memory) and was killed by Kubernetes.", no `BATCH_SIZE`, and suggested a limit of `"2Gi"` in r1 and `"512Mi"` in r2, plus `kubectl label pods app=frontend tier=web`. All four were graded partial. The two LLM investigators reasoned from the alert and stayed on it; K8sGPT's analyzers had no alert to follow and swept everything. That is the case for making the sweep mechanical: three commands, run before anyone writes "unaffected", whoever or whatever is doing the triage. ## How Aurora handles this Aurora receives the alert, connects to the cluster read-only through an uploaded kubeconfig, chooses its own kubectl commands, records every call and output, and cites them by number in its report. Mutating commands are blocked by policy, so fixes arrive as suggestions. On this scenario it diagnosed the alerted OOM in both runs and missed the Service in both, and its reports called the frontend unaffected on pod status alone. A claim of health needs a request-path check behind it, and today that is not enforced. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) Related: [how to fix OOMKilled](/runbooks/fix-oomkilled-kubernetes), [a Service with no endpoints](/runbooks/fix-service-no-endpoints-kubernetes), [a Service whose targetPort does not match the container](/runbooks/fix-service-targetport-mismatch-kubernetes), [HolmesGPT vs K8sGPT](/blog/holmesgpt-vs-k8sgpt), [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), and the [benchmark](/benchmark) with every graded run. ## Sources and verification - Every command and output block comes from lab run `v1-run1`, scenario `12-two-faults-oom-and-selector`, 2026-10-02: Aurora's tool-call transcripts and reports (`aurora-rca.md`) for r1 and r2, and the state `lab.sh evidence` captured before each investigation (`evidence-before/`). Trimmed where noted, otherwise unchanged. Faults injected by the scenario's `break.sh`. - Versions from `evidence-before/versions.yaml` and the logs: Kubernetes server v1.37.0 on kind v0.33.0 (three nodes, containerd 2.3.4), kubectl client v1.35.3, nginx/1.27.5. - Aurora commit `87d22baf`, single agent, model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge `us.anthropic.claude-opus-5-5`, a second Claude instance as AI reviewer (`reviewer_ai`), and a person at Arvo (`human_verdict` in each `grade.json`). Durations, calls and costs from `result.json`; first-call gaps from `timeline.json`. - HolmesGPT and K8sGPT: `holmes.md`, `k8sgpt.txt`, their `grade.json` and `meta.json` files in `runs/h2h-v1`. - Documentation quoted, read 2026-10-19: Kubernetes [Debug Services](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/) and [Declarative Management of Kubernetes Objects Using Configuration Files](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/); kube-state-metrics [endpoint metrics](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/service/endpoint-metrics.md). - Not run in a graded lab run: the fix and verify commands, the Ingress check, the probe and the endpoints alert. The merge behaviour of `patch` and `apply` on the selector is from the documentation, not a lab capture. --- ## Sidecar OOMKilled, app healthy: the pod drops out of its Service (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-sidecar-container-oomkilled-kubernetes Published: 2026-10-19 Author: Damian Loch, Software Engineer at Arvo AI > A sidecar OOMKilled under its own 16Mi limit left a healthy api 1/2 Ready and its Service empty. Find the container, fix its limit, verify. Lab-tested. | Applies to | Multi-container pods where a sidecar, such as a log shipper or proxy, is OOMKilled repeatedly while the app container stays `Ready`; shown on a two-replica Python api with a log-shipper in `spec.containers` | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, python:3.12-alpine, 2026-10-02 | | Lab folder | [scenarios/02-sidecar-oom](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/02-sidecar-oom), runs in [runs/v0-run2/02-sidecar-oom](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/02-sidecar-oom) | **When a pod shows `1/2` under READY and `OOMKilled` under STATUS while the app's own logs look fine, find out which container is being killed before you touch the app: Kubernetes keeps a separate state, `lastState` and `restartCount` for every container, but readiness is decided for the whole pod, so one dying sidecar takes a healthy app out of its Service.** Confirm it with `kubectl describe pod ` and read `Last State` and `Restart Count` under each container name. The fix belongs to the sidecar: raise its own memory limit, shrink its buffer, or roll it back. The kill itself is covered in [OOMKilled: container exceeded its memory limit](/runbooks/fix-oomkilled-kubernetes). ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | READY `1/2`, STATUS `OOMKilled`, app container `Ready: True`, `Restart Count: 0`, sidecar dead within a second of `Started` | The sidecar allocates past its own limit at startup | Each container's `Last State` and `Limits` in `kubectl describe pod` | Raise the sidecar's limit or shrink its buffer; roll back first if the Service is empty | | Sidecar killed after minutes, the same length each cycle | A leak in the sidecar | `Started` to `Finished` per cycle | See [OOMKilled from a slow memory leak](/runbooks/fix-memory-leak-oomkilled-kubernetes) | | Service has no endpoints although its selector matches the pods | One container keeps every matching pod unready | `Endpoints:` in `kubectl describe svc`, pod `Ready` condition | Fix that container; selector faults are in [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes) | ## Which container in the pod was killed? Read READY before RESTARTS. Lab run r1, step 4: ```shell kubectl get pods -n shop -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-7d4b6f6f7d-5n8h7 1/2 OOMKilled 3 (57s ago) 72s 10.244.1.8 aurora-lab-2-worker api-7d4b6f6f7d-g95qf 1/2 OOMKilled 3 (55s ago) 70s 10.244.2.11 aurora-lab-2-worker2 ... ``` `1/2` says one of two containers is not ready. `OOMKilled` and `3` belong to one container, but the row does not say which. `describe` does, per container (r1, step 6, trimmed): ```shell kubectl describe pod api-7d4b6f6f7d-5n8h7 -n shop ``` ``` Status: Running Containers: api: State: Running Started: Fri, 02 Oct 2026 16:25:30 +0000 Ready: True Restart Count: 0 Limits: memory: 128Mi log-shipper: Command: ... buf=bytearray(40*1024*1024) while True: time.sleep(5) State: Terminated Reason: OOMKilled Exit Code: 137 Started: Fri, 02 Oct 2026 16:26:10 +0000 Finished: Fri, 02 Oct 2026 16:26:10 +0000 Last State: Terminated Reason: OOMKilled Exit Code: 137 Started: Fri, 02 Oct 2026 16:25:43 +0000 Finished: Fri, 02 Oct 2026 16:25:44 +0000 Ready: False Restart Count: 3 Limits: memory: 16Mi Requests: cpu: 10m memory: 8Mi Conditions: Type Status ... Ready False ContainersReady False ``` The pod is `Running`, the api is `Running`, `Ready: True`, zero restarts. The log-shipper is the one being killed, and the pod's `Ready` and `ContainersReady` are `False` because of it. The [pod lifecycle page](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) says `describe` "shows the state for each container within that Pod"; read every block, not the first. Across replicas, filter `containerStatuses` by name. The lab's `check.sh`, which matched 5 seconds after the change in both runs, is this plus a grep: ```shell kubectl -n shop get pods -l app=api -o jsonpath='{range .items[*].status.containerStatuses[?(@.name=="log-shipper")]}{.lastState.terminated.reason}{"\n"}{end}' ``` When you do not know which container to suspect, print them all (a suggested variant, not run in the lab): ```shell kubectl get pod -n -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.restartCount}{"\t"}{.lastState.terminated.reason}{"\n"}{end}' ``` A native sidecar is an entry in `initContainers`, so its status is under `.status.initContainerStatuses` instead. Two traps: - **The events do not say OOMKilled.** Neither run's event stream contained the word. They say `Back-off restarting failed container log-shipper in pod api-7d4b6f6f7d-5n8h7_shop(...)` (r1, step 7): the container, not the reason. - **The sidecar's logs are thin.** r2 asked for the killed container's output with `kubectl logs api-7d4b6f6f7d-d68bn -n shop -c log-shipper --previous` (step 11) and got `unable to retrieve container logs for containerd://78fc1be6...`. The live one, captured before the investigation, had printed only `log-shipper: buffering 40 MB before first flush`. Pass `-c`, and expect `--previous` to fail on a container that lives one second. ## What does a crash-looping sidecar do to the Deployment and the Service? Here the two runs diverged on the same patch. ReplicaSets, r1 step 10 and r2 step 9, api rows: ``` NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR # r1 api-7d4b6f6f7d 2 2 0 102s api,log-shipper python:3.12-alpine,python:3.12-alpine app=api,pod-template-hash=7d4b6f6f7d api-f64486c4c 0 0 0 109s api python:3.12-alpine app=api,pod-template-hash=f64486c4c # r2 api-7d4b6f6f7d 2 2 0 94s api,log-shipper python:3.12-alpine,python:3.12-alpine app=api,pod-template-hash=7d4b6f6f7d api-f64486c4c 1 1 1 102s api python:3.12-alpine app=api,pod-template-hash=f64486c4c ``` CONTAINERS shows what changed and AGE shows when, without `rollout history`. In r1 the old ReplicaSet was at zero; in r2 one old pod survived. The Service shows the cost (harness capture before each investigation): ```shell kubectl describe svc api -n shop ``` ``` # r1 Selector: app=api Endpoints: # r2 Selector: app=api Endpoints: 10.244.1.18:8080 ``` r1's Service had no endpoints while both api containers answered `/healthz` with 200. Its load generator got 502 on every request from 16:26:03 to the end of the capture at 16:26:19, and nginx logged `connect() failed (111: Connection refused) while connecting to upstream` against the Service IP. r2's one endpoint was the surviving old pod, and every request from 16:54:20 on got 200. Same fault: a full outage in one run, half capacity in the other. The chain is in the docs. A pod is `Ready` when it "is able to serve requests and should be added to the load balancing pools of all matching Services", and `ContainersReady` means "all containers in the Pod are ready" ([pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)). A sidecar in `spec.containers` is ready only while it runs, so every kill and every back-off wait pulls the pod out of the Service. The split is rollout timing. With two replicas, the default 25% `maxUnavailable` rounds down to 0 and `maxSurge` rounds up to 1 ([Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)), so an old pod goes only after a new one counts as available. In r1 the controller scaled the old ReplicaSet from 2 to 0 within three seconds of creating the new pods (events at 81 s, 79 s and 78 s, step 7), so both new pods briefly counted as available: the `NewReplicaSetAvailable` reason below is set only when "All of the replicas associated with the Deployment are available" ([Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)), and the two new pods' api logs show three `/items` requests served. In r2 only one did, and the rollout stalled. Now the Deployment's own claim, r1's `kubectl describe deployment api -n shop` before the investigation: ``` Replicas: 2 desired | 2 updated | 2 total | 0 available | 2 unavailable ... Conditions: Type Status Reason ---- ------ ------ Progressing True NewReplicaSetAvailable Available False MinimumReplicasUnavailable ``` `NewReplicaSetAvailable` marks a completed rollout, and the docs say the condition "holds even when availability of replicas changes" ([Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)). A pipeline gated on `kubectl rollout status` would most likely have exited 0 in the moment both new pods counted as available, and moved on before the 502s began. Run again at capture time it would have waited, because it also compares available with updated replicas (the same page's example prints "Waiting for rollout to finish: 2 of 3 updated replicas are available..."). The condition keeps reading complete after that. A trap: the harness's EndpointSlice listing in r1 showed `10.244.1.8,10.244.2.11` for the api, the two unready pods, while the Service had no endpoints. The slice keeps unready addresses and marks them; for a pod, the `serving` condition "maps to the Pod's Ready condition" ([EndpointSlices](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/)). The pod lifecycle page says a failed readiness probe makes the controller remove "the Pod's IP address from the EndpointSlices"; in both runs the unready IPs stayed listed (r2's slice showed three addresses behind its one ready endpoint). IPs in a slice do not mean traffic has somewhere to go. ## Why is the sidecar over its limit? Limits are per container: "For each container, you can specify resource limits and requests", and "memory limits are enforced by the kernel with out of memory (OOM) kills" ([Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)). The pod's limits sum to 144Mi, but the api's unused room is not lent to the sidecar. 1. **Compare the allocation with the sidecar's own limit.** `bytearray(40*1024*1024)` is 40 MiB against 16Mi, 2.5 times the limit before the interpreter is counted. 2. **Read the lifetime.** The last two cycles lasted under a second and one second (16:25:43 to 16:25:44). That is a startup allocation. A sidecar that buffers under load dies after minutes; use the [leak runbook](/runbooks/fix-memory-leak-oomkilled-kubernetes)'s cycle method on it. 3. **Find where the sidecar came from.** In r2's `kubectl get deployment api -n shop -o yaml` (step 7), the `kubectl.kubernetes.io/last-applied-configuration` annotation holds one container and the live template two, the second being `log-shipper` with `memory: 16Mi`. The sidecar arrived outside the tracked manifest, in the lab as a `kubectl patch`. A sidecar injected by an admission webhook is not in the Deployment at all, only in the pod, so compare `kubectl get pod -o yaml` with the template before blaming a rollout. Do not wait for a memory graph: a container that lives one second can fall between scrapes, and neither run sampled pod memory. The termination record is the evidence. ### Native sidecars, as far as the docs go This log-shipper is a regular container. Since v1.33 the [sidecar containers page](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) marks native sidecars stable: an `initContainers` entry with `restartPolicy: Always`. They "can be started, stopped, or restarted without affecting the main application container", and "If a readinessProbe is specified for this init container, its result will be used to determine the ready state of the Pod." The docs do not say what a crash-looping native sidecar without a readiness probe does to pod readiness, and the lab did not run one. Test it on your version before you rely on it to protect the Service. ## What is the fix? The scenario accepts a larger sidecar limit, a smaller buffer, or removing the sidecar. None of these commands were run in the lab. **Mitigate first if the Service is empty**, as in r1: ```shell kubectl -n shop rollout undo deployment/api ``` Risk: log shipping stops until the fix ships; under GitOps, change the repository instead or the next sync reverts it. Rollback: run the same command again. With one old pod still serving, as in r2, fix forward. **Raise the sidecar's limit, not the app's.** 64Mi, the value in both runs' suggested fixes, leaves 24 MiB over the buffer; measure the shipper's real peak before settling on it: ```shell kubectl -n shop set resources deployment/api -c log-shipper --limits=memory=64Mi ``` Risk: a rollout, and the request stays at 8Mi, so the scheduler reserves 8Mi for a container that needs over 40 MiB; set `--requests=memory` to the measured working set in the same command. Rollback: `kubectl -n shop rollout undo deployment/api`. Then put the values in the tracked manifest, or the next `apply` restores 16Mi. **Shrink the buffer** below the limit when the shipper lets you configure it; when one shipper runs beside many apps, this beats raising a limit in each Deployment. Risk: more frequent flushes, or drops under a burst. Change the buffer or the limit, not both in one rollout. ## How do you verify it? Run the lab's check without its grep, the jsonpath above. While broken it prints `OOMKilled` once per pod; on new pods it prints empty lines, because a container that never terminated has no `lastState.terminated`. Then: `kubectl get pods -n shop -l app=api` shows `2/2` and `0` restarts per replica, and `kubectl describe svc api -n shop` lists one address per replica under `Endpoints:`. Pass condition: both hold for 10 minutes, when the kubelet "resets the restart backoff timer" ([pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)), and frontend errors are back to baseline. A shipper that buffers under load also needs a traffic peak. ## How do you prevent the next one? **Alert per container, and on the Service.** `kube_pod_container_status_restarts_total` is STABLE and counts "container restarts per container"; put its `container` label in the alert title so the page says `log-shipper`, not just `api`. Two suggested rules, not captured in the lab, using the kube-state-metrics [pod](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md) and [endpoint](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/service/endpoint-metrics.md) metrics: ``` increase(kube_pod_container_status_restarts_total[10m]) > 2 and on (namespace, pod, container) kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1 ``` ``` count by (namespace, endpoint) (kube_endpoint_address) unless count by (namespace, endpoint) (kube_endpoint_address{ready="true"}) ``` The first names the killed container (`last_terminated_reason` is EXPERIMENTAL; native sidecars need the `kube_pod_init_container_*` series). The second fires when a Service has addresses and none ready: r1's outage. **Set `minReadySeconds` on Deployments with sidecars.** It is "the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available" ([Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)). r1 ran the default, 0. A value longer than the sidecar survives should have kept the old pods serving; that follows from the docs and was not tested in the lab. **Review rule:** a sidecar's limit sits next to its buffer size, where a 40 MB buffer over a 16Mi limit is obvious. ## Common mistakes - **Raising the app's limit, or debugging the app, because the alert names its Deployment.** The api had zero restarts. - **Trusting a finished rollout.** r1's Deployment read `Progressing True NewReplicaSetAvailable` and `Available False` at once; a `rollout status` that exited 0 during the rollout says nothing about the minute after. - **Reading EndpointSlice IPs as healthy endpoints.** In r1 the slice listed both pods; the Service routed to nothing. ## What we saw in the lab Scenario `02-sidecar-oom`: `break.sh` JSON-patches the api Deployment to append a `log-shipper` container that allocates 40 MiB at startup under a 16Mi limit. Aurora got only the alert, `KubePodCrashLooping: pods of deployment api in namespace shop are restarting.`, a kubeconfig and read-only kubectl under its default `observability_only` policy, in a fresh user and org per run, on kind clusters `aurora-lab-2` and `aurora-lab-3`. Required findings: the sidecar, not the api, is being killed, and because 16Mi is below its usage. Wrong answers: raising the api's limit, saying the api is crashing. | Run | Verdict | Time | Tool calls | Tool errors | Cost | Fix in suggestions | |---|---|---|---|---|---|---| | r1 | correct (judge, AI review, human) | 82 s | 10 | 1 | $0.19 | "Patch `log-shipper` sidecar memory limit/request" (acceptable) | | r2 | correct (judge, AI review, human) | 98 s | 12 | 1 | $0.30 | "Raise `log-shipper` memory limit above its actual allocation" (acceptable) | Both met both findings and asserted neither wrong answer, and both found the container the same way: `1/2` in the pod list, then `describe pod` as the third kubectl call. r1 also used the CONTAINERS column of the Deployment and ReplicaSets to date the sidecar, and wrote: "The `api` container itself remained healthy with 0 restarts, isolating the crash loop to the sidecar [6]." r2 read the Deployment YAML and named the readiness mechanism: "because both containers share a pod, the pod as a whole never becomes Ready, preventing the deployment from satisfying its minimum replica availability [4]." Each run's one tool error was `kubectl rollout history`, read-only but blocked by the default policy as "Mutating kubectl operations"; both listed it under Not Checked. Where the reports went past the evidence: - **Neither followed the outage to the Service.** r1's "the service had no available replicas at the time of investigation" was true but inferred from the Deployment's `0/2`; it never says requests were failing. r2 read the surviving old ReplicaSet as "no loss of the previously stable pod", with the api at one of two replicas. Neither read the Service, an EndpointSlice or the frontend logs. - **r1's age arithmetic.** It compared the Deployment's 75 s (step 5) with ReplicaSet ages read about 30 seconds later (step 10); the judge flagged it. Its conclusion, that the sidecar "was introduced immediately before the alert fired", holds. - **r2 ruled out "Readiness-probe-triggered restarts".** Readiness probes do not restart containers; liveness probes do ([pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)). - **r2 ruled out node pressure from `kubectl get nodes`**, all `Ready`, and healthy neighbours, with no memory data. r1 ran `kubectl top nodes` (0 to 3 percent). - **r2 called the alert "high-severity"**, repeating the incident record it read at step 1; the alert text has no severity. - **No fix in either report body**, as the judge noted. Both put it in the separate suggested next steps: the sidecar's limit from 16Mi to 64Mi, per the grader, and r2 also a rollback. The grader read each report, note and suggestion list; an AI reviewer and a person at Arvo agreed with both verdicts on 2026-10-02. ## How Aurora handles this Aurora receives the alert and investigates with read-only `kubectl` behind a command policy, so by default it cannot change the cluster while it looks. For a crash-looping multi-container pod it reads each container's state and limits, the Deployment and its ReplicaSets, then writes a root cause with a citation per claim, Ruled Out and Not Checked lists, and separate suggested fixes. In both lab runs it separated the sidecar from the app on its first `describe`; it did not follow the outage to the Service. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) Related: [OOMKilled: container exceeded its memory limit](/runbooks/fix-oomkilled-kubernetes), [OOMKilled from a slow memory leak](/runbooks/fix-memory-leak-oomkilled-kubernetes), [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes), [how to fix CrashLoopBackOff](/blog/fix-crashloopbackoff-kubernetes), and the [lab report on 19 Kubernetes failures](/blog/aurora-lab-report-kubernetes-19-failures). ## Sources and verification - Outputs: lab run `v0-run2`, scenario `02-sidecar-oom`, 2026-10-02, from Aurora's recorded tool calls (by run and step, trimmed with `...`, pod UIDs as `(...)`) and from `evidence-before/`, captured seconds before each investigation (Service, EndpointSlice, Deployment conditions, frontend and load generator logs). - Versions from `evidence-before/versions.yaml`: Kubernetes server v1.37.0, kind v0.33.0, kubectl client v1.35.3. Aurora commit `87d22baf`, single agent, RCA model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge `us.anthropic.claude-opus-5-5`, an AI review, then a person (`grade.json`). Time, tool calls, errors and cost from `result.json`, measured from incident creation to report; the first tool call came about 6 s (r1) and 9 s (r2) in. - Kubernetes docs quoted, read 2026-10-19: [Pod Lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/), [Sidecar Containers](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/), [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/), [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), [EndpointSlices](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/); kube-state-metrics metric lists, same date. - Not captured in the lab: the all-containers jsonpath, every fix and rollback command, both alert rules, `minReadySeconds` as a guard, native sidecar behaviour. The moment r1's new pods counted as available is inferred from the scale-down events, the `NewReplicaSetAvailable` reason and the new pods' api logs; no capture shows them Ready. The `rollout status` behaviour is from the Deployments page, not run. --- ## Name does not resolve with dnsPolicy None: the search domains are wrong (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-pod-dnsconfig-search-domain-not-resolving-kubernetes Published: 2026-10-19 Author: Damian Loch, Software Engineer at Arvo AI > New pods log Name does not resolve for a Service that exists. The template set dnsPolicy None with search domains for another cluster domain. Lab-tested. | Applies to | Any pod with `dnsPolicy: None`, a custom `dnsConfig` search list or `ndots` that cannot resolve a Service by its short name; here a Python api calling `redis` after a rollout | |---|---| | Verified on | Kubernetes server v1.37.0 (kind v0.33.0), 2 October 2026 | | Lab folder | [scenarios/16-dns-search-domain](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/16-dns-search-domain), runs in [runs/v1-run1/16-dns-search-domain](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/16-dns-search-domain) | **When a pod logs `Name does not resolve` for a Service that exists, and the hostname in its environment is spelled right, the fault is the pod's resolver configuration, not the name: `dnsPolicy: None` with a `dnsConfig` whose search domains belong to another cluster domain (here `cluster.internal` on a `cluster.local` cluster) makes every short name expand to records that do not exist, while the dependency, CoreDNS and the previous revision's `ClusterFirst` pods stay healthy.** Confirm with `kubectl get pod -n -o jsonpath='{.spec.dnsPolicy}{"\n"}{.spec.dnsConfig}'` next to the `kubernetes` zone in the CoreDNS Corefile. The fix is to drop the `dnsConfig` and return to `ClusterFirst`, correct the search domains to the real cluster domain, or roll the Deployment back. If the hostname itself is wrong, that is the [Name does not resolve](/runbooks/fix-dns-name-not-resolving-kubernetes) runbook; this page is for when the name is right and the pod cannot look it up. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | New pods of one Deployment log `Name does not resolve`, the env hostname is an existing Service, the template has `dnsPolicy: None` | `dnsConfig.searches` carries another cluster's domain | `.spec.dnsConfig.searches` against the `kubernetes` zone in the coredns ConfigMap | Remove the `dnsConfig` and restore `ClusterFirst`, fix the suffixes, or roll back | | Same symptom, but the env hostname is not in `kubectl get svc -n ` | The name is wrong | The Environment block against the Service list | See [a wrong Service hostname](/runbooks/fix-dns-name-not-resolving-kubernetes) | | Short names fail, `..svc.cluster.local` works | `dnsPolicy: Default` (the node's resolv.conf) or `None` without the cluster suffixes | `kubectl exec -- cat /etc/resolv.conf` | Set `ClusterFirst`, or add the cluster suffixes to `searches` | | External names resolve slowly, CoreDNS (with the `log` plugin on) shows several NXDOMAIN per name | `ndots:5` applied to names with fewer than five dots | Count the queries per name in CoreDNS logs, which needs the `log` plugin in the Corefile | A trailing dot in the app, or a lower `ndots` for that pod only | ## How do you confirm the name is right and the resolver is wrong? The pod list looks exactly like the wrong-name case: one new api pod in `Error` with 3 restarts, the two old api pods and redis `1/1 Running` (run r1, step 4). So does the finished container's log (r1, step 6): ``` $ kubectl logs api-779966d57-f8nsq -n shop --previous --tail=100 redis connection failed (attempt 1/5): [Errno -2] Name does not resolve redis connection failed (attempt 2/5): [Errno -2] Name does not resolve redis connection failed (attempt 3/5): [Errno -2] Name does not resolve redis connection failed (attempt 4/5): [Errno -2] Name does not resolve redis connection failed (attempt 5/5): [Errno -2] Name does not resolve giving up: redis unavailable ``` Two checks separate this page from that one. First, the hostname the app was given (r1, step 7, trimmed): ``` Last State: Terminated Reason: Error Exit Code: 1 Started: Fri, 02 Oct 2026 20:31:18 +0000 Finished: Fri, 02 Oct 2026 20:31:28 +0000 Restart Count: 3 Environment: REDIS_HOST: redis ``` Second, whether a Service by that name exists (r2, step 10): ``` $ kubectl get svc -n shop -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR api ClusterIP 10.96.167.158 8080/TCP 2m8s app=api frontend ClusterIP 10.96.246.174 80/TCP 2m8s app=frontend redis ClusterIP 10.96.198.186 6379/TCP 2m8s app=redis ``` `REDIS_HOST` is `redis` and `redis` is a Service in the same namespace. The name is right; the question is how this pod turns it into a query. The timestamps add one fact: ten seconds for five attempts with a two-second sleep between them (`time.sleep(2)` in `api.py`, r2 step 22), so each lookup failed at once. An unreachable nameserver fails slowly, on the resolver timeout; a fast `Name does not resolve` means a server answered and said no. ## How far does it reach? **Which pods.** Only the new ReplicaSet: `kubectl get rs -n shop -o wide` shows `api-779966d57` with 1 desired and 0 ready, 89 seconds old, next to `api-f64486c4c` at 2/2, 95 seconds old (r2, step 9). Users are still served: the `api` Service's endpoints hold only the two old pods, `10.244.1.48:8080,10.244.2.58:8080` (r2, step 21). **Since when.** The Deployment status shows the stuck rollout: `replicas: 3`, `updatedReplicas: 1`, `unavailableReplicas: 1`, `Progressing: True` (r1, step 5). The new ReplicaSet is six seconds younger than the old one; the lab broke the Deployment right after creating it. **After which change.** The live template has a `dnsConfig` block and `dnsPolicy: None`; the `kubectl.kubernetes.io/last-applied-configuration` annotation on the same Deployment has neither. A field present in the live spec and absent from the annotation came from a patch, an edit or a controller, not `kubectl apply`; the lab's `break.sh` was such a patch, adding the block shown in the next section. `kubectl rollout history deployment/api -n shop` lists the revisions; the lab's read-only policy denied it in both runs, but you can run it. ## How do you isolate the cause? **1. Read the pod's DNS settings and compare them with a healthy pod's.** From the crashing pod (r2, step 13, the two fields that matter): ```shell kubectl get pod api-779966d57-v8d2r -n shop -o yaml ``` ``` dnsConfig: nameservers: - 10.96.0.10 options: - name: ndots value: "5" searches: - shop.svc.cluster.internal - svc.cluster.internal - cluster.internal dnsPolicy: None ``` From a pod of the old ReplicaSet (r2, step 14): `dnsPolicy: ClusterFirst` and no `dnsConfig` block. Same image, same command, same `REDIS_HOST`, both on `aurora-lab-3-worker`. The DNS block is the whole diff. **2. Read the cluster's real domain.** It is in the `kubernetes` plugin line of the Corefile (r1, step 11, trimmed): ```shell kubectl get configmap coredns -n kube-system -o yaml ``` ``` .:53 { kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } forward . /etc/resolv.conf { max_concurrent 1000 } } ``` CoreDNS answers for `cluster.local` and forwards everything else upstream. With this search list, a lookup of `redis` goes out as `redis.shop.svc.cluster.internal`, `redis.svc.cluster.internal`, `redis.cluster.internal` and finally `redis` on its own. None is a `cluster.local` name, so each is forwarded upstream and answered with no such record. The record `redis.shop.svc.cluster.local` existed the whole time; this pod never asked for it. **3. Check the nameserver is the right one.** It was, which is why the failure was fast rather than slow: `kubectl get svc kube-dns -n kube-system` shows ClusterIP `10.96.0.10` (r2, step 16), the address in the `dnsConfig`, and both CoreDNS pods were `1/1 Running` with 0 restarts (r2, step 15). The server is right; only the suffixes are wrong. **4. Read `/etc/resolv.conf` when you can.** The kubelet writes this file for every pod, and for `dnsPolicy: "None"` the [Kubernetes DNS page](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) says "All DNS settings are supposed to be provided using the `dnsConfig` field in the Pod Spec": the file is the `dnsConfig` written out as `nameserver`, `search` and `options` lines. ```shell kubectl -n shop exec -- cat /etc/resolv.conf ``` Nobody ran it in the lab. Aurora tried it on a healthy old pod and its read-only policy blocked `exec` as a mutating operation (r1, step 10); on the crashing pod your own attempt is also a race against a container that lives ten seconds. From the spec, the broken pod's file has to read (derived, not captured): ``` nameserver 10.96.0.10 search shop.svc.cluster.internal svc.cluster.internal cluster.internal options ndots:5 ``` while a `ClusterFirst` pod gets the cluster's own list, in the docs' example `search .svc.cluster.local svc.cluster.local cluster.local` with `options ndots:5`. Neither block is captured output. `nameserver` is who gets asked, `search` is what gets appended to short names, in order, and `ndots` is the threshold, defined in the [resolv.conf manual](https://man7.org/linux/man-pages/man5/resolv.conf.5.html) as "a threshold for the number of dots which must appear in a name given to res_query(3) (see resolver(3)) before an initial absolute query will be made. The default for n is 1". At 5, even `redis.shop.svc.cluster.local` (four dots) walks the search list before it is tried as written; a trailing dot skips the walk. The four `dnsPolicy` values, from the same page: `ClusterFirst` is the default ("If `dnsPolicy` is not explicitly specified, then "ClusterFirst" is used"); `Default` means "The Pod inherits the name resolution configuration from the node"; `ClusterFirstWithHostNet` is `ClusterFirst` for `hostNetwork` pods; `None` gives the pod only what `dnsConfig` says. Under `ClusterFirst` a `dnsConfig` is additive: its searches "will be merged into the base search domain names generated from the chosen DNS policy", so the same three `cluster.internal` suffixes would have cost a few extra failed queries per lookup and nothing else. `None` turned a sloppy copy into a dead revision. ## How do you fix it? None of these commands was run inside a graded lab run. Each is the inverse of the injected patch or a standard rollout operation. **Cause: the `dnsConfig` was copied from another cluster and nothing needs it.** Remove it and let the pod go back to `ClusterFirst`: ```shell kubectl -n shop patch deploy api --type=json \ -p='[{"op":"remove","path":"/spec/template/spec/dnsConfig"},{"op":"replace","path":"/spec/template/spec/dnsPolicy","value":"ClusterFirst"}]' ``` Risk: low, but anything else intentional in the block (a stub nameserver, an `edns0` option) goes with it, so read it first. Rollback: `kubectl -n shop rollout undo deploy/api`. Fix the manifest or Helm values in git too, or the next apply brings it back. **Cause: the pod needs `dnsPolicy: None` for a reason, and only the suffixes are wrong.** Replace them with the real domain: ```shell kubectl -n shop patch deploy api --type=json \ -p='[{"op":"replace","path":"/spec/template/spec/dnsConfig/searches","value":["shop.svc.cluster.local","svc.cluster.local","cluster.local"]}]' ``` Risk: you now maintain by hand a list the kubelet would have generated, and it goes stale when the pod moves to another namespace or cluster. Rollback: `kubectl -n shop rollout undo deploy/api`. **Alternative: roll back.** ```shell kubectl -n shop rollout undo deploy/api ``` Same result in the lab, because the DNS block was the only change in revision 2. Risk: anything else in revision 2 is undone; `--to-revision=2` reverses it. Mitigation is not needed while the old ReplicaSet is healthy: `Available: True` and two live endpoints mean users are served. ## How do you verify the fix? Pass condition: the rollout completes, every api pod reports `dnsPolicy: ClusterFirst` (or your corrected `dnsConfig`), and the new pods log a connection, not a retry loop. ```shell kubectl -n shop rollout status deploy/api kubectl -n shop get pods -l app=api -o jsonpath='{range .items[*]}{.metadata.name} {.spec.dnsPolicy} {.spec.dnsConfig.searches}{"\n"}{end}' kubectl -n shop logs -l app=api --tail=3 ``` A healthy api pod logs `connected to redis at redis:6379` and then `api listening on :8080` (`evidence-before` log of `api-f64486c4c-zt6dk`, run r2). The lab's own check for the broken state is one line; when it stops matching, you are done: ```shell kubectl -n shop get pods -l app=api -o jsonpath='{.items[*].status.containerStatuses[0].lastState.terminated.exitCode}' | grep -q 1 ``` Watch for two minutes: the new ReplicaSet should reach 2/2 and the old one scale to 0. ## How do you prevent it? **Do not set `dnsPolicy: None` to add a search domain.** Keep `ClusterFirst` and put the extra entries in `dnsConfig`; the kubelet merges them. Reserve `None` for pods that must not use cluster DNS at all, and treat any `None` in review as a question about where the pod expects `redis` to live. **Never hardcode the cluster domain in an application manifest.** It belongs to the cluster; `cluster.local` is only the default. A pre-deploy check that every `dnsConfig.searches` entry on a `dnsPolicy: None` template ends in the domain from the Corefile's `kubernetes` line is a short script against two `kubectl get` calls (suggested, not run in the lab). **Alert on the rollout, not only the pod.** `KubePodCrashLooping` fires for any crash; a Deployment that cannot finish updating is the condition that describes this one. With kube-state-metrics, `kube_deployment_status_replicas_unavailable > 0` held longer than a normal rollout is a suggested expression, not captured in the lab. ## Common mistakes - **Changing `REDIS_HOST`.** It is correct. Setting it to `redis.shop.svc.cluster.local` would make this one lookup work from the broken pod, since the nameserver is CoreDNS, and leave every other short name broken. - **Blaming CoreDNS.** The old api pods are serving `/items`, which needs redis, both CoreDNS pods are healthy, and the broken pod never sent CoreDNS a `cluster.local` query. - **Testing from a fresh debug pod.** `kubectl run` gives the new pod `ClusterFirst`, so `nslookup redis` succeeds and proves nothing about the failing pod. Read that pod's spec, or `exec` into it if it lives long enough. ## What we saw in the lab Scenario `16-dns-search-domain`: `break.sh` patched the healthy api Deployment with the `dnsPolicy: None` and `dnsConfig` block shown above, within seconds of creating it (`break.log` puts the patch 15 s into the scenario; the revision 2 ReplicaSet is dated 10 s after the Deployment in r1 and 6 s in r2). Aurora received the alert text, `KubePodCrashLooping [warning] namespace=shop deployment=api: Pod shop/api (container api) is in waiting state (reason: CrashLoopBackOff) after the latest rollout.`, and read-only kubectl under its default policy. Two runs on 2 October 2026 (Aurora commit `87d22baf`, single agent, model `bedrock/us.anthropic.claude-sonnet-5`), each in a fresh account, on cluster `aurora-lab-3`. A correct run had to name the symptom (`redis` does not resolve) and the cause (the pod's `dnsPolicy: None` and `cluster.internal` search domains), and blame neither `REDIS_HOST`, the redis Service nor CoreDNS. | Run | Verdict | Time | Tool calls | Tool errors | Cost | Fix in suggestions | |---|---|---|---|---|---|---| | r1 | correct (judge, then human) | 140 s | 15 | 2 | $0.40 | "Correct `dnsConfig.searches` in `deployment/api` to use `cluster.local`" (acceptable) | | r2 | correct (judge, then human) | 205 s | 24 | 2 | $0.56 | "Remove the invalid `dnsPolicy: None`/`dnsConfig` override from `shop/api`" (acceptable) | Both separated "name wrong" from "resolver wrong" the same way: the crashing pod's log, then the Deployment or pod YAML, then the Corefile. r1 made ten kubectl calls (44 seconds of summed tool time, issued in four parallel batches) in its 140 seconds and had the cause by step 8, when it compared the two ReplicaSets' templates. Its report: "The revision 2 pod template introduced `dnsPolicy: None` with an explicit `dnsConfig.searches` list using the domain suffix `cluster.internal` [8], whereas the cluster's actual CoreDNS zone is `cluster.local`, as declared in the `coredns` ConfigMap's Corefile (`kubernetes cluster.local in-addr.arpa ip6.arpa`) [11]." r2 made nineteen kubectl calls (84 seconds of summed tool time, seven batches) in its 205 seconds, adding the old pod's log, both Services' endpoints, the CoreDNS pods and the application code. Its report named the mechanism: "Because `dnsPolicy: None` disables fallback to default DNS behavior, pods under revision 2 could not resolve the `redis` hostname, producing `[Errno -2] Name does not resolve` across all 5 connection attempts in the container logs [7]." It also noted that the override was "absent from the `kubectl.kubernetes.io/last-applied-configuration` annotation [19]". One of its two tool errors was its own: a jsonpath containing `{"\n"}` reached kubectl with the backslash intact and failed with `unrecognized character in action: U+005C`, so it fell back to the full YAML. What the reports got wrong or did not prove: - **Neither run read `/etc/resolv.conf`.** r1 tried `exec ... cat /etc/resolv.conf`, was denied, and listed it under Not Checked; r2 never tried. Both diagnoses rest on the pod spec plus the Corefile; the file itself was never seen. - **r1 inferred the old pods' health from their status.** Its Ruled Out says revision-1 pods "connect to it successfully with zero restarts"; r1 never read an old pod's log. The second automated review flagged this. r2 did read one. - **r1's first guess survived into its report.** After one read of the Deployment it wrote "the rollout trigger was likely a ConfigMap change"; no ConfigMap changed, and the guess is still in its Not Checked list as "who/what triggered the ConfigMap or template change", which the judge called stray wording. - **r2 cited the wrong pod for a same-node claim.** It wrote that "a sibling pod on the same node resolved names and served traffic", citing the log of `api-f64486c4c-7skgm`, which runs on `worker2`; the crashing pod is on `worker`. The same review flagged its "1 of 3 desired replicas unavailable": the Deployment wants 2, and 3 is the surge total during the rollout. - **Neither report states the fix in its body.** Both judge notes read "The RCA gives no explicit remediation". The fix is in each run's closing summary (r1: "Roll back deployment/api in shop to revision 1"; r2: "kubectl rollout undo deployment/api -n shop", each with a forward-fix on the `dnsConfig` as the alternative) and in the suggestions, where it was graded acceptable. Transcripts, grades and the pre-investigation evidence: [runs/v1-run1/16-dns-search-domain](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/16-dns-search-domain). ## How Aurora handles this Aurora receives the alert, lists the pods, reads the previous container's log, and compares the crashing pod's template with a running pod's from the previous ReplicaSet, which is where `dnsPolicy: None` and the `dnsConfig` block show up. It then reads the CoreDNS Corefile for the cluster's real domain and names the mismatch, each claim cited to the tool output it came from. It runs read-only kubectl under a command policy by default, so `exec` and `rollout` are denied and listed under Not Checked, and it proposes the patch or the rollback rather than applying either. See a full investigation on the [Aurora root cause analysis](/rca) page. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) Related: [a wrong Service hostname](/runbooks/fix-dns-name-not-resolving-kubernetes), [a Service with no endpoints](/runbooks/fix-service-no-endpoints-kubernetes), [an init container waiting on a name that does not resolve](/runbooks/fix-pod-stuck-init-container-kubernetes), and [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres). ## Sources and verification - Every command and output block above comes from lab run `v1-run1`, scenario `16-dns-search-domain`, on 2026-10-02: Aurora's tool-call transcripts for r1 and r2, each block attributed to its run and step, plus `evidence-before/` captured before the investigation (the healthy pod's startup log). Versions from `evidence-before/versions.yaml`: Kubernetes server v1.37.0 on kind v0.33.0, kubectl client v1.35.3. - Aurora commit `87d22baf`, single agent, RCA model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge model `us.anthropic.claude-opus-5-5` on Aurora's full output, then a person at Arvo (`human_verdict` in each run's `grade.json`). Times, tool calls, tool errors and costs from `result.json`; summed tool time from `duration_ms` per step in `timeline.json`; the fix column from `fix_from_suggestions` in `grade.json`. - Kubernetes documentation quoted, read 2026-10-19: [DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) for the `dnsPolicy` values, the `dnsConfig` merge rule and the resolv.conf example. The `ndots` definition is quoted from the [resolv.conf(5) manual page](https://man7.org/linux/man-pages/man5/resolv.conf.5.html), which the Kubernetes page links to. - Not captured, and labelled as such above: both `/etc/resolv.conf` blocks (derived from the pod spec and the documented mapping), the search-list expansion of `redis`, the fix and verify commands, the alert expression and the CI check. The fix patches invert `scenarios/16-dns-search-domain/break.sh`; the last Verify command is the lab's `check.sh`. --- ## One in three requests to a Service fails: its selector also matches pods from another Deployment (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-service-selects-wrong-pods-label-overlap-kubernetes Published: 2026-10-19 Author: Damian Loch, Software Engineer at Arvo AI > A Service selects pods by label, not by Deployment. Its endpoints list one pod more than the Deployment runs and that pod returns 500. Confirm and fix it. | Applies to | Any Service whose label selector matches pods owned by a workload other than the one it was written for; shown on the `api` Service of the lab's `shop` app after a second Deployment, `shop-api`, was applied with the pod label `app=api` | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, kubectl v1.35.3, nginx 1.27-alpine (nginx/1.27.5), python:3.12-alpine, 2026-10-02 | | Lab folder | [scenarios/15-stray-pods-share-label](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/15-stray-pods-share-label), runs in [runs/v1-run1/15-stray-pods-share-label](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/15-stray-pods-share-label) | **A Service that fails a steady fraction of requests while every pod behind it is Ready is usually sending that fraction to a pod that is not yours: the Service selects by label, not by Deployment, and another workload's pod template carries the same label.** Confirm with `kubectl get endpoints -n ` against the Deployment's ready replicas: in the lab the `api` endpoints read `10.244.1.37:8080,10.244.1.40:8080,10.244.2.48:8080` while `deploy/api` reported `2 available`. Map the extra IP to its pod with `kubectl get pods -n -o wide`, then scale that pod's Deployment to zero and relabel it. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | A steady fraction of requests gets 5xx, every pod Ready, more endpoints than the Deployment has replicas | Another workload's pods carry the Service's selector label | `kubectl get endpoints ` against `kubectl describe deploy `; `kubectl get pods -o wide` to name the extra IP | Scale the stray workload to zero, then relabel it | | Every request fails, `endpoints/` reads `` | The selector matches no pods | `kubectl get svc -o yaml` against `kubectl get pods --show-labels` | See [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes) | | Every request refused, endpoints populated | `targetPort` is not the port the container listens on | `kubectl describe svc ` against the listen line in `kubectl logs ` | See [targetPort mismatch](/runbooks/fix-service-targetport-mismatch-kubernetes) | | A fraction fails, endpoint count equals the replica count | A real replica is failing | Per-pod `kubectl logs` | Fix that pod; the Service is fine | ## How do you confirm the Service is selecting a stray pod? The [Service docs](https://kubernetes.io/docs/concepts/services-networking/service/): "The set of Pods targeted by a Service is usually determined by a selector that you define." No Deployment in that sentence. The [labels page](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) supplies the part that bites: "Unlike names and UIDs, labels do not provide uniqueness." Any pod in the namespace with `app=api`, whoever created it, is a backend of a Service selecting `app=api`. Start at the client. The frontend nginx proxies `/api/` to the `api` Service, and its access log alternates (run r1, step 8, four of 24 `/api/items` lines): ```shell kubectl logs frontend-76d98597b9-bb4z9 -n shop --tail=100 ``` ``` 10.244.1.39 - - [02/Oct/2026:20:29:01 +0000] "GET /api/items HTTP/1.1" 200 38 "-" "Wget" "-" 10.244.1.39 - - [02/Oct/2026:20:29:05 +0000] "GET /api/items HTTP/1.1" 500 34 "-" "Wget" "-" 10.244.1.39 - - [02/Oct/2026:20:29:09 +0000] "GET /api/items HTTP/1.1" 500 34 "-" "Wget" "-" 10.244.1.39 - - [02/Oct/2026:20:29:13 +0000] "GET /api/items HTTP/1.1" 200 38 "-" "Wget" "-" ``` No `[error]` line accompanies the 500s: a program answered, and the two body sizes (38 bytes for a 200, 34 for a 500) say two different programs are answering. Count the backends (r1, step 13) against what the Deployment runs (r2, step 11, two lines of 50): ```shell kubectl get endpoints api -n shop -o wide kubectl describe deploy api -n shop ``` ``` NAME ENDPOINTS AGE api 10.244.1.37:8080,10.244.1.40:8080,10.244.2.48:8080 111s ``` ``` Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable NewReplicaSet: api-f64486c4c (2/2 replicas created) ``` Three endpoints, two replicas; the two objects count different things. The pod list prints IPs with `-o wide` (r1, step 4, three of eight rows): ```shell kubectl get pods -n shop -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-f64486c4c-r95bj 1/1 Running 0 71s 10.244.1.37 aurora-lab-2-worker api-f64486c4c-vftsb 1/1 Running 0 71s 10.244.2.48 aurora-lab-2-worker2 shop-api-69f6fc5964-jhzz8 1/1 Running 0 62s 10.244.1.40 aurora-lab-2-worker ``` `10.244.1.40` is `shop-api-69f6fc5964-jhzz8`, from a Deployment called `shop-api`. With hundreds of pods, ask for the one IP: `kubectl get pods -A -o wide --field-selector status.podIP=10.244.1.40` (`status.podIP` is a [supported Pod field selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/); not run in the lab). Confirm the label (r1, step 14, two of eight rows): ```shell kubectl get pods -n shop --show-labels ``` ``` api-f64486c4c-r95bj 1/1 Running 0 114s app=api,pod-template-hash=f64486c4c shop-api-69f6fc5964-jhzz8 1/1 Running 0 105s app.kubernetes.io/name=shop-api,app=api,pod-template-hash=69f6fc5964 ``` The stray pod carries `app=api` next to its own `app.kubernetes.io/name=shop-api`. Its log says why it fails (r1, step 15, first nine of 100 lines): ```shell kubectl logs shop-api-69f6fc5964-jhzz8 -n shop --tail=100 ``` ``` 10.244.1.38 "GET /items HTTP/1.0" 500 - Traceback (most recent call last): File "/code/app.py", line 12, in do_GET settings() File "/code/app.py", line 5, in settings return {"db": os.environ["DATABASE_URL"], "redis": os.environ.get("REDIS_HOST", "redis")} ~~~~~~~~~~^^^^^^^^^^^^^^^^ File "", line 714, in __getitem__ KeyError: 'DATABASE_URL' ``` Every `/items` request raises on a missing environment variable; every `/healthz` answers 200, so the readiness probe passes and the pod stays in the endpoints. The two `api` pods logged only 200s for `/items` (r1, steps 10 and 11). Look-alikes. Endpoints `` with every request refused is a selector matching nothing ([scenario 07](/runbooks/fix-service-no-endpoints-kubernetes)); endpoints populated with every request refused is a wrong `targetPort` ([scenario 11](/runbooks/fix-service-targetport-mismatch-kubernetes)); both log nginx `connect() failed (111: Connection refused)`, absent here after startup. A real replica failing on its dependency answers differently: the `api`'s code returns 503 `backend unavailable` when redis is unreachable (r1, step 12), not 500 with a traceback. ## How much is affected, and since when? Every client of the Service, for the share of requests that land on the stray pod. Of the load generator's 27 requests after the stack came up, 12 returned 500, 44 percent (`evidence-before` loadgen log, r1); the alert said 31 percent over ten minutes. Both are right: one of three ready endpoints fails, and kube-proxy's default iptables mode installs rules which "by default, select a backend Pod at random" ([Virtual IPs](https://kubernetes.io/docs/reference/networking/virtual-ips/)), so a one-minute sample wanders around one third. The start time is the stray Deployment's creation: `creationTimestamp: "2026-10-02T20:29:03Z"` (r1, step 16), nine seconds after `deploy/api` at `20:28:54Z` (step 18); pod ages agree, 62 s against 71 s. The load generator's first 500 is at `20:29:05Z`, after 200s at `20:29:01Z` and `20:29:03Z`. The Service did not change: its `last-applied-configuration` annotation and live spec both read `selector: app: api` (r1, step 17). What changed is the set of pods matching an unchanged selector, so a change log scoped to the Service shows nothing. ## Which pods does the Service actually select? Cheapest and likeliest first. **1. Count endpoints against ready replicas.** Three against two (steps 13 and 18, `readyReplicas: 2`). If they match, this runbook does not apply. **2. Name the owner of each endpoint IP.** r2 printed labels and owner in one command (step 13, three of eight lines): ```shell kubectl get pods -n shop -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.metadata.labels}{" "}{.metadata.ownerReferences[0].name}{"\n"}{end}' ``` ``` api-f64486c4c-ffkt4 {"app":"api","pod-template-hash":"f64486c4c"} api-f64486c4c api-f64486c4c-w8vvz {"app":"api","pod-template-hash":"f64486c4c"} api-f64486c4c shop-api-69f6fc5964-twjf9 {"app":"api","app.kubernetes.io/name":"shop-api","pod-template-hash":"69f6fc5964"} shop-api-69f6fc5964 ``` Quote the template. r1 ran it unquoted (step 21) and got `Error from server (NotFound): pods ".items[*]}{.metadata.name}..." not found`: the shell split it at the first space. **3. Read the stray pod's spec, not only its log.** The `shop-api` Deployment's env holds only `REDIS_HOST` (r1, step 16) and its readiness probe is `http-get http://:8080/healthz` (r2, step 16), a path that never touches the missing variable. A passing probe and a failing handler is what keeps a broken pod in rotation. `kubectl get secrets -n shop` was empty (r1, step 19). **4. Check whether the overlap was deliberate.** The Deployment list's selector column shows it fastest (cluster state captured before the investigation): `app=api` for `deployment.apps/api`, `app=api,app.kubernetes.io/name=shop-api` for `deployment.apps/shop-api`. The stray selector is narrower, so it manages only its own pod while the Service takes both sets. The [Deployment docs](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) warn that you "should not create other Pods whose labels match this selector, either directly, by creating another Deployment, or by creating another controller such as a ReplicaSet or a ReplicationController"; the controllers did not fight here only because the `api` ReplicaSet also selects on `pod-template-hash`, which the Service does not. **5. Rule out the dependency.** Both `api` pods logged `connected to redis at redis:6379` and served 200s (r1, steps 10 and 11). Redis is on the scenario's list of wrong answers. ## What is the fix? Take the stray pod out of the Service, then make sure it cannot come back. **Mitigation: scale the stray Deployment to zero.** r1's working notes named this command, and the graders accepted it: ```shell kubectl scale deployment shop-api -n shop --replicas=0 ``` Risk: low for the Service, whose remaining endpoints are the two healthy pods; `shop-api` stops serving whatever it legitimately served, which in the lab was nothing. Rollback: `--replicas=1`. As the pod terminates, the endpoints controller drops its IP and the 500s stop. Do not restart or scale `deploy/api`: a wrong answer in the scenario, and it changes nothing about which pods the Service matches. **Fix: give the workload labels the Service does not select.** The label is in the pod template and the Deployment's selector, and the [Deployment docs](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) say "A Deployment's label selector is immutable after creation; it cannot be updated via kubectl patch, kubectl edit, kubectl apply, or tools like helm upgrade." So this is a delete and a recreate. Save the object first: ```shell kubectl -n shop get deployment shop-api -o yaml > shop-api.yaml kubectl -n shop delete deployment shop-api ``` Re-apply with `app: api` removed from both `spec.selector.matchLabels` and `spec.template.metadata.labels`, keeping `app.kubernetes.io/name: shop-api`. Risk: `shop-api` is down between the delete and the re-apply. Rollback: `kubectl apply -f shop-api.yaml`, which also restores the collision. If the stray pods should in fact serve this Service, fix their config instead and let them keep the label. From the graders' note on Aurora's own recreate suggestion: a search-and-replace on `app: api` also matches the selector line and any longer value starting with `api`, so read the diff before applying. ## How do you verify it worked? Pass condition: the endpoints are exactly the IPs of the Deployment's ready pods, and a full minute of client requests returns 200. ```shell kubectl -n shop get endpoints api kubectl -n shop get endpointslices -l kubernetes.io/service-name=api kubectl -n shop logs deploy/loadgen --tail=30 ``` The first must read `10.244.1.37:8080,10.244.2.48:8080` (r1's api pod IPs) with no third address; the EndpointSlice is the same list under the label the [EndpointSlice docs](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/) say "enables simple lookups of all EndpointSlices belonging to a Service". The load generator writes one line every two seconds, so 30 lines is a minute; every line should end `-> 200`. The lab's own check for the broken state passes only when that log holds both a `-> 500` and a `-> 200`; its failure is your pass. ## How do you prevent it? **Select on two labels, not one.** The [recommended labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/) separate `app.kubernetes.io/name`, "The name of the application", from `app.kubernetes.io/instance`, "A unique name identifying the instance of an application", and require that "Every instance of an application must have a unique name." A selector on both cannot be joined by a template copied from another app (different name) or by a second install of the same app (different instance). The lab's stray template had the recommended name label and the colliding `app` label side by side. **Refuse pod templates without an instance label at admission.** A [ValidatingAdmissionPolicy](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) (stable since v1.30) matching `apps/v1` Deployments can evaluate `'app.kubernetes.io/instance' in object.spec.template.metadata.labels` where the docs' example evaluates `object.spec.replicas <= 5`. (Composed from the documented example; not run in the lab.) **Assert endpoint count equals ready replicas.** After each deploy, compare every Service's EndpointSlice address count with the `readyReplicas` of the Deployment it fronts and fail on a mismatch either way. On a schedule it is an alert; in the lab it would have read three against two a minute before the error rate fired. **Keep per-pod access logs.** The frontend's log shows the status but not the pod: nginx dials the ClusterIP and kube-proxy picks the backend after that. The split was visible only in each backend's own log, which is how both lab runs found it. ## Common mistakes - **Restarting or scaling `deploy/api`.** New pods, same labels, same three endpoints. A wrong answer in the scenario. - **Removing the label from the running stray pod.** `kubectl label pod shop-api-... app-` takes it out of the Service and out of its own ReplicaSet's selector, so the ReplicaSet creates a replacement with the label and the orphan keeps running. Fix the Deployment. - **Adding `DATABASE_URL` to the stray Deployment.** Its code then returns 200 with `{"items": []}` (the `shop-api` ConfigMap, r1, step 20): the error rate drops to zero while a third of responses carry the wrong data from the wrong program. A truncated note in r2's timeline, written as its report completed, begins "Add DATABASE_URL to the shop-api Deployment's"; the graded suggestion that followed was scaling to zero. - **Blaming redis.** The 500 came with a traceback on an environment variable; the api's redis failure path returns 503, and the healthy pods logged `connected to redis`. ## What we saw in the lab Scenario 15 in the [Aurora lab](https://github.com/Arvo-AI/aurora-lab): the five-component `shop` app on a three-node kind cluster, then one `kubectl apply` of a ConfigMap and a one-replica Deployment named `shop-api` whose pod template carries `app: api` and `app.kubernetes.io/name: shop-api`, whose code reads `os.environ["DATABASE_URL"]` on every request except `/healthz`, and whose env sets only `REDIS_HOST`. Aurora got the alert text ("HighErrorRate [warning] namespace=shop service=frontend: 31% of requests to /api/ returned 5xx over the last 10 minutes (threshold 5%).") and read-only kubectl under the default `observability_only` policy. Two findings were required: the failing requests come from the `shop-api` pod, selected because it shares the label; and it fails on the missing `DATABASE_URL` while the real api pods are healthy. Two runs, separate organizations. | Run | Verdict (human reviewed) | Time | Tool calls | Aurora cost | |---|---|---|---|---| | r1 | correct | 175 s | 24 | $0.59 | | r2 | correct | 118 s | 17 | $0.35 | r1 read pods, events, Services, the frontend log, both api logs and the api's source in its first eight kubectl calls, ran `get endpoints api -o wide` at step 13 (its ninth), put three IPs next to two replicas, and went straight to `--show-labels`, the stray pod's log, both Deployments and the Service. Its report: "causing its single pod (`shop-api-69f6fc5964-jhzz8`, IP 10.244.1.40) to be inadvertently pulled into the `api` Service's endpoint pool alongside the two legitimate `api` Deployment pods", and "so every request routed to this pod throws `KeyError: 'DATABASE_URL'` and returns HTTP 500, confirmed directly in the pod's logs". Both findings met. Two things to hold against it. It called the frontend log "consistent with the alert's reported 31% 5xx error rate" when the sample it cited had 11 of 24 `/api/items` lines at 500, 46 percent; the AI reviewer behind the judge raised the same objection with a looser count ("about 16 of 33"). And one tool call failed, the unquoted jsonpath at step 21, recovered from with the `-o wide` output it already had. r2 found the pod from the other direction: frontend logs, both api logs, the api Deployment, then the jsonpath above at step 13, where its note reads "Root cause candidate found: a third pod shop-api-69f6fc5964-twjf9 carries label app=api (same as the legitimate api Deployment) and is crashing on every /items request with KeyError: 'DATABASE_URL' because its container is missing a DATABASE_URL env var." It confirmed with the endpoints and `describe deploy shop-api` (steps 15 and 16) and reported "The root cause was a label collision between two unrelated Deployments sharing the `app=api` label." Three over-claims. "Since kube-proxy round-robins across all three registered endpoints": the iptables mode picks at random, which the AI reviewer called loose wording that does not change the conclusion. Its Not Checked list says the stray pod's probe configuration was "not retrieved", but step 16's output contains `Readiness: http-get http://:8080/healthz delay=0s timeout=1s period=5s`; it had the answer and did not read it. And its notes said the alert "fired right as shop-api's readiness probe passed", reading event ages of 2m7s and 2m15s as cluster time; the Deployment was created 57 s before the alert. Neither report contained a fix. Both judges said so (r1: "It gives no remediation step, such as deleting or relabeling shop-api"; r2: "it recommends no remediation") and graded on the root cause, correct in both. Aurora's separate fix suggestions, generated after each report (20:33:55 and 21:05:22, outside the durations above), were graded acceptable with "Scale `shop-api` Deployment to zero replicas" as the best in both runs; r1's notes also carried the recreate step, "selector is immutable, needs delete+reapply". Neither run named redis, a restart or the network, the scenario's three wrong answers. ## How Aurora handles this Aurora receives the alert, connects to the cluster read-only through an uploaded kubeconfig, and runs the kubectl commands shown above, choosing them itself. On this failure it counted endpoints against replicas, named the owner of the extra IP, and read the stray pod's log and Deployment spec in both runs. Mutating commands are blocked by policy, so the scale-to-zero arrives as a suggestion for a person to apply. Both reports stopped at the cause; the fix came only in the separate suggestions. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) Related: [a Service with no endpoints](/runbooks/fix-service-no-endpoints-kubernetes), [a Service whose targetPort does not match the container](/runbooks/fix-service-targetport-mismatch-kubernetes), [a ConfigMap change that reached only some pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes), the [lab report covering all 19 failures](/blog/aurora-lab-report-kubernetes-19-failures), [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), and what an [Aurora root cause analysis](/rca) looks like. ## Sources and verification - Every command and output block above comes from the lab run `v1-run1`, scenario `15-stray-pods-share-label`, on 2026-10-02: Aurora's tool-call transcripts for r1 and r2 and the cluster state captured before the investigation (`evidence-before/`). The fault was injected by the scenario's `break.sh`, one `kubectl apply` of the `shop-api` ConfigMap and Deployment. - Versions, from `evidence-before/versions.yaml` and the pod logs: Kubernetes server v1.37.0 on kind v0.33.0 (three nodes, containerd 2.3.4), kubectl client v1.35.3; nginx 1.27-alpine reporting nginx/1.27.5; python:3.12-alpine for both api programs. - Aurora commit `87d22baf`, single agent, RCA model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge model `us.anthropic.claude-opus-5-5`, a second AI reviewer, and a person at Arvo (`human_verdict` in each `grade.json`); fix suggestions graded separately (`fix_from_suggestions`). Durations (incident creation to report), tool calls and costs from each run's `result.json`; suggestion timestamps from each `timeline.json`. - The 44 percent is 12 of the 27 requests after the three startup 502s in r1's captured load generator log; the 46 percent is 11 of 24 `/api/items` lines in r1's step 8 output. - Documentation quoted, read 2026-10-19: Kubernetes [Service](https://kubernetes.io/docs/concepts/services-networking/service/), [Labels and Selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/), [Field Selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/), [Virtual IPs and Service Proxies](https://kubernetes.io/docs/reference/networking/virtual-ips/), [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), [EndpointSlices](https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/), [Recommended Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/) and [Validating Admission Policy](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/). - The scale command is Aurora's own from r1's notes. The delete, save and verify commands, the field-selector lookup, the CEL expression and the endpoint-count assertion were not run inside a graded lab run, since Aurora's policy blocks mutations; Verify states the pass condition and the lab's own check instead of a captured result. --- ## OOMKilled on a steady interval: a memory leak, not a small limit (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-memory-leak-oomkilled-kubernetes Published: 2026-10-19 Author: Damian Loch, Software Engineer at Arvo AI > OOMKilled on a steady interval with clean logs and no spec change is a leak. Tell it from a small limit by cycle length and growth, then fix the code. | Applies to | Any container with a memory limit that is killed on a repeating interval while its logs show normal work and its spec, limit and environment have not changed; shown here on a single-replica Python worker whose code is mounted from a ConfigMap | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, python:3.12-alpine, 2026-10-02 | | Lab folder | [scenarios/17-slow-memory-leak](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/17-slow-memory-leak), runs in [runs/v1-run1/17-slow-memory-leak](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/17-slow-memory-leak) | **A container that is OOMKilled on a steady interval, with clean logs and no change to its spec or limit, is leaking: every cycle it starts small, grows at a constant rate, reaches `resources.limits.memory`, is killed with `Reason: OOMKilled` and `Exit Code: 137`, restarts and begins again.** Two numbers separate it from a limit that is too small: in `kubectl describe pod`, the gap between the last container's `Started` and `Finished` is minutes, not seconds, and the same every cycle (2m20s, 2m21s and 2m20s across the three cycles captured in the lab); and `kubectl top pod --containers` rises between samples. The fix is in the code that keeps memory it should release. Raising the limit changes only how long each cycle lasts. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | `OOMKilled`, `Exit Code: 137`, the previous container ran for minutes and the same length each cycle, `--previous` logs stop mid-work, live spec equals `last-applied-configuration` | Memory retained across iterations (a leak), usually in code that arrived with the last restart | `Started` to `Finished` gap in `describe pod`; two `kubectl top pod --containers` samples a minute apart | Fix or revert the code, roll out, verify for longer than one old cycle | | `OOMKilled` within seconds of `Started`, every time, after a config or image change | The working set exceeds the limit from the first iteration | Live spec against `last-applied-configuration` | See [OOMKilled: container exceeded its memory limit](/runbooks/fix-oomkilled-kubernetes) | | `OOMKilled` only during traffic peaks, usage falls back afterwards | Load-proportional working set, no leak | `kubectl top` through a quiet period | Size the limit for the peak, or bound concurrency | | Restarts with `Reason: Error` and a traceback at the end of `--previous` | An application crash, not a kill | `kubectl logs --previous` | See [how to fix CrashLoopBackOff](/blog/fix-crashloopbackoff-kubernetes) | ## How do you confirm it is a leak and not a small limit? Confirming the kill itself is the same as for any OOMKilled container; the [OOMKilled runbook](/runbooks/fix-oomkilled-kubernetes) covers the signature and its look-alikes. The difference here is in the timestamps (lab run r1, step 6, trimmed): ```shell kubectl describe pod worker-5b67b55dd5-wlxf4 -n shop ``` ``` Status: Running State: Running Started: Fri, 02 Oct 2026 20:38:06 +0000 Last State: Terminated Reason: OOMKilled Exit Code: 137 Started: Fri, 02 Oct 2026 20:35:32 +0000 Finished: Fri, 02 Oct 2026 20:37:53 +0000 Ready: True Restart Count: 2 Limits: memory: 192Mi Environment: BATCH_SIZE: 500 Events: Warning BackOff 26s kubelet spec.containers{worker}: Back-off restarting failed container worker in pod worker-5b67b55dd5-wlxf4_shop(...) ``` The killed container ran from 20:35:32 to 20:37:53: 2 minutes 21 seconds. The cycle before it, captured before the investigation, ran 2m20s; r2's first cycle on another cluster ran 2m20s (21:04:56 to 21:07:16). A working set that exceeds the limit at startup, as in the OOMKilled runbook's batch allocation, dies within a second of starting (same-second `Started` and `Finished` there). A leak needs time to reach the limit, and because it grows at the rate the code allocates, it needs the same time every cycle. Cadence alone is not proof: a process that ramps to a steady state above its limit, a cache warming to a fixed size for instance, also dies on a fixed interval with rising `top` readings. The discriminator is the arithmetic and the code path in the Isolate section, a fixed number of bytes retained per unit of work with no plateau, and the test of a higher limit: it lengthens a leak's cycle by room divided by rate and ends a sizing problem's kills outright. The [resource management page](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) says "`memory` limits are enforced by the kernel with out of memory (OOM) kills. When a container uses more than its `memory` limit, the kernel may terminate it." Between kills the pod looks healthy: `Status: Running`, `Ready: True`, and in r2 no Warning event at all. `CrashLoopBackOff` comes late because the kubelet restarts containers "with an exponential backoff delay (10s, 20s, 40s, …), that is capped at 300 seconds (5 minutes)" and resets it only "Once a container has executed for 10 minutes without any problems" ([pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)). A container that lives 140 seconds never resets it: the gap between kills stretches while the running time inside each cycle stays fixed. ## How much is affected, since when, and what changed? One workload, one replica: r1's pod list (step 4) showed the worker `1/1 Running` with `2 (19s ago)` restarts and six other pods with none. The ReplicaSets date it (r1, step 11): ```shell kubectl get replicaset -n shop -l app=worker ``` ``` NAME DESIRED CURRENT READY AGE worker-5b67b55dd5 1 1 1 5m39s worker-6cf88d4b58 0 0 0 5m47s ``` A new ReplicaSet eight seconds after the old one looks like a rollout. The Deployment says otherwise (r1, step 5, trimmed): ```shell kubectl get deployment worker -n shop -o yaml ``` ``` metadata: annotations: deployment.kubernetes.io/revision: "2" kubectl.kubernetes.io/last-applied-configuration: | {...,"env":[{"name":"BATCH_SIZE","value":"500"}],...,"limits":{"memory":"192Mi"},...} spec: template: metadata: annotations: kubectl.kubernetes.io/restartedAt: "2026-10-02T16:33:10-04:00" spec: containers: - env: - name: BATCH_SIZE value: "500" image: python:3.12-alpine resources: limits: memory: 192Mi volumes: - configMap: name: api-code name: code ``` `BATCH_SIZE` is 500 live and 500 in `last-applied-configuration`; the limit is 192Mi in both. All revision 2 added to the template is `kubectl.kubernetes.io/restartedAt`, which `kubectl rollout restart` writes. Someone restarted the worker at 16:33:10 local time without changing its spec. A restart with no spec change points at what the container loads when it starts: here the code in the `api-code` ConfigMap, elsewhere an image tag that moved. ConfigMaps have no revision history; the annotation's timestamp is the lead you take to source control. ## Where does the memory go? 1. **Read the killed container's log to the end** (r1, step 7, 29 lines, trimmed): ```shell kubectl logs worker-5b67b55dd5-wlxf4 -n shop --previous --tail=100 ``` ``` worker started, batch size 500 processed batch of 500 items processed batch of 500 items ... processed batch of 500 items ``` Exactly 28 `processed batch` lines, then nothing: no traceback, no partial line. The same count in r2 (step 7). The log clears the application of crashing and hands you a unit: 5 seconds per batch, 28 batches is 140 seconds, and 140 seconds is the `Started` to `Finished` gap. Whatever grows, grows per batch. 2. **Sample usage more than once.** Each lab run took one sample. r1, step 10, about 40 seconds after the container started at 20:38:06: ```shell kubectl top pod worker-5b67b55dd5-wlxf4 -n shop --containers ``` ``` POD NAME CPU(cores) MEMORY(bytes) worker-5b67b55dd5-wlxf4 worker 3m 34Mi ``` r2's sample (`kubectl top pods -n shop`, step 12), about 90 seconds after its container started at 21:07:17, read `79Mi` for the worker and 12Mi or less for every other pod. Together: about 1 MiB per second of uptime. On your own incident sample the same container three times a minute apart (not captured in the lab). A leak gives three rising numbers; a working set that is too large gives one number near the limit. The [`kubectl top pod` reference](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_top/kubectl_top_pod/) warns of "the metrics pipeline delay"; a reading just after a restart lags the process. 3. **Read the code that runs per iteration.** The worker's source is a ConfigMap key (r1, step 8): ```shell kubectl get configmap api-code -n shop -o jsonpath='{.data.worker\.py}' ``` ```python import os, time batch = int(os.environ.get("BATCH_SIZE", "500")) print(f"worker started, batch size {batch}", flush=True) recent = [] # tail of recent batches, so a retried batch can skip items already done while True: # Each queued item holds about 100 KiB while the batch is processed. buf = [bytearray(100 * 1024) for _ in range(batch)] recent.extend(buf[-50:]) print(f"processed batch of {len(buf)} items", flush=True) del buf time.sleep(5) ``` `buf` is 500 items of 100 KiB, about 49 MiB, and `del buf` frees it. `recent` keeps the last 50 of them, about 4.9 MiB, every 5 seconds, and nothing trims it, whatever the comment promises. After 28 batches `recent` holds about 137 MiB; the 29th batch's 49 MiB on top passes 192Mi, so the kernel kills the process mid-batch and the log stops at 28. The same arithmetic prices a bigger limit: 384Mi buys 192 MiB of room, at 4.9 MiB per 5 seconds that is 39 more batches, about three more minutes per cycle, then the same kill. ## What is the fix? The scenario's acceptable fix is to fix or revert the code so it stops retaining items across batches. Three paths, in order of preference. **Fix the code.** Bound the list where it grows. The one-line change (a suggestion, not run in the lab): ```python recent.extend(buf[-50:]) del recent[:-50] ``` Apply the ConfigMap, then restart the Deployment: the worker read `/code/worker.py` once at startup and a ConfigMap edit does not reach a running process (see [ConfigMap change not applied to all pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes)): ```shell kubectl -n shop apply -f api-code.yaml kubectl -n shop rollout restart deploy/worker ``` Risk: one more restart of a worker already restarting every 140 seconds. Rollback: apply the previous ConfigMap and restart again. Under a GitOps tool, change the repository. **Revert the code.** When the previous version is in source control and the retention was never needed, re-apply the old ConfigMap and run the same `rollout restart`. Same risk and rollback. The lab's `break.sh` is this in reverse: a new `worker.py` applied to `api-code`, then `kubectl -n shop rollout restart deploy/worker`. **Raise the limit, as a mitigation with an expiry.** Only when the fix cannot ship in time, with the arithmetic written next to the new number: 384Mi turns a 2m20s cycle into roughly 5m35s. Edit `resources.limits.memory` in the tracked manifest and apply. Risk: the request stays at 64Mi while the limit doubles, so the scheduler does not check for the room and a full node can OOM-kill a neighbour instead. Rollback: re-apply the previous manifest. Do not combine it with the code fix, or you will not know which one stopped the kills. ## How do you verify it worked? The lab's `check.sh` greps this jsonpath for `OOMKilled`; run it without the grep: ```shell kubectl -n shop get pods -l app=worker -o jsonpath='{.items[*].status.containerStatuses[0].lastState.terminated.reason}' ``` While broken it prints `OOMKilled`; after the rollout the new pod has no terminated last state and prints nothing. Not sufficient: a leak passes that check for one cycle. Watch for at least three old cycles, seven minutes here, sampling `kubectl top pod -n shop --containers` each minute. Pass condition: `RESTARTS` stays at 0 and the readings level off instead of climbing by a steady amount per sample. ## How do you prevent the next one? **Alert on slope, not on restarts.** A restart alert such as `KubePodCrashLooping` fires after the kills have started; `kube_pod_container_status_restarts_total` (Counter, STABLE in the [kube-state-metrics pod metrics list](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md)) is the series behind restart counts. In the lab the climb was visible within 90 seconds of a container start (34Mi at about 40 seconds in r1, 79Mi at about 90 seconds in r2). A suggested expression, not captured in the lab, using cAdvisor's [`container_memory_working_set_bytes`](https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md) and kube-state-metrics' `kube_pod_container_resource_limits`: ``` predict_linear(container_memory_working_set_bytes{container!=""}[15m], 1800) > on (namespace, pod, container) kube_pod_container_resource_limits{resource="memory"} ``` It fires when the last 15 minutes of growth will cross the limit within 30 minutes, before the first kill; tune the windows to your cycle. **Review rule: a collection defined outside a loop and appended inside it needs a bound on the same screen.** `recent = []` above `while True:` with `.extend` below it is the whole bug. **Soak test the hot loop.** Run it for a fixed number of iterations under a small memory limit and fail if it is killed or if resident memory at iteration N exceeds iteration N/2 by more than a few MiB. ## Common mistakes - **Raising the limit and closing the ticket.** The kills pause for exactly as long as the arithmetic says. Here each extra 100 MiB of limit is about 100 more seconds per cycle. - **Reading one `kubectl top` sample as a trend.** One number says how much; only a second says which way. - **Treating "no config change" as "no change".** The Deployment diff was one annotation and the cause was still a change: a ConfigMap key the Deployment mounts. Code that arrives through a ConfigMap or a moving image tag leaves nothing in `kubectl rollout history`. - **Reading the current container's log.** It shows a healthy worker a few batches in. The killed one, via `--previous`, shows the clean stop that rules out a crash and gives the cycle length. ## What we saw in the lab Scenario `17-slow-memory-leak`: `break.sh` replaces `worker.py` in the `api-code` ConfigMap and runs `kubectl -n shop rollout restart deploy/worker`; `BATCH_SIZE` and the 192Mi limit are untouched. Aurora received only the alert text, `KubePodCrashLooping [warning] namespace=shop deployment=worker: Pod shop/worker (container worker) has restarted repeatedly in the last 30 minutes.`, and read-only kubectl under its default policy. Two runs in fresh accounts on two kind clusters (`aurora-lab` for r1, `aurora-lab-2` for r2). Required findings: OOMKilled rather than a crash, and retention across batches in the new code rather than the batch size. Wrong answers: `BATCH_SIZE` too large, raising the limit as the only fix, node pressure. | Run | Verdict | Time | Tool calls | Tool errors | Cost | Fix in suggestions | |---|---|---|---|---|---|---| | r1 | correct (judge, then human) | 201 s | 13 | 0 | $0.31 | "Patch `api-code` ConfigMap to bound the `recent` list and roll out" (acceptable) | | r2 | correct (judge, then human) | 86 s | 12 | 1 | $0.20 | "Patch `code/worker.py` to bound the `recent` list and roll out via ConfigMap `api-code`" (acceptable) | Both runs met both findings and asserted none of the wrong answers. r1 read the Deployment YAML second, so it had the unchanged spec and the `restartedAt` annotation early. Its root cause: "the `worker` container is repeatedly OOMKilled due to an unbounded memory leak in the application code mounted from ConfigMap `api-code` (worker.py)", the clean previous log "confirming the termination was a hard resource kill rather than a code-level fault". The note it saved for next time says of raising the limit "that only delays the crash, it does not fix the leak". r2 reasoned from the count in the previous log, and its one tool error was `kubectl rollout history`, blocked by Aurora's default policy as a mutating operation. Its root cause names the distinction this page is about: the limit "is being exceeded by this unbounded growth rather than by a one-time spike or undersized baseline". It never read the Deployment YAML, so it never saw the annotation that dates the restart. Both verdicts were correct and both reports overreach in the same places. Each called a single `kubectl top` reading "climbing"; one sample shows a level, not a direction; the automated review flagged it in both runs. Both ruled out node pressure because other pods were healthy, with no node metrics; neither ran `kubectl top nodes`. Both called the alert high-severity when its text says `[warning]`; the incident record both read at step 1 already carried `severity: high`, so the reports repeated the record rather than the alert text. r2 called the 28-batch crash point "deterministic" after one observed cycle. Neither report states the fix in its body: r1 says remediation "requires `kubectl edit/patch configmap` and `rollout restart`" without the code change; r2 names it only under Not Checked as "ConfigMap patch or memory limit increase". The fixes were in both runs' Suggested Next Steps: both proposed bounding `recent`, and both also listed raising the limit as a stopgap, r2 first. r1's bound kept 500 items, about 49 MiB, which still retains items across batches; the judge noted a reviewer might not accept it against the scenario's wording. Neither report says how the code changed or ties it to the restart: r1 had the `restartedAt` timestamp in its step 5 output and did not use it; r2 dated the ReplicaSet rollover but not the ConfigMap. The grader read Aurora's full output (report, note, suggestions) and a person at Arvo checked both verdicts. Transcripts and grades: [runs/v1-run1/17-slow-memory-leak](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/17-slow-memory-leak). ## How Aurora handles this Aurora receives the alert and investigates with read-only `kubectl` behind a command policy, so by default it cannot change the cluster while it looks. For a repeating OOMKilled it reads the container's last termination and its timestamps, the previous container's log, the code or configuration the container loads at startup, the Deployment spec against its `last-applied-configuration` annotation, and pod metrics, then writes a root cause with a numbered citation for each claim and a separate list of suggested fixes. What it could not verify, here the rollout history, goes in a Not Checked section. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) Related: [OOMKilled after a config change](/runbooks/fix-oomkilled-kubernetes), [ConfigMap change not applied to all pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes), [liveness probe failures](/runbooks/fix-liveness-probe-failed-kubernetes), [how to fix CrashLoopBackOff](/blog/fix-crashloopbackoff-kubernetes), and [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), and [a sidecar container OOMKilled while the app is healthy](/runbooks/fix-sidecar-container-oomkilled-kubernetes). ## Sources and verification - Every command and output block above comes from lab run `v1-run1`, scenario `17-slow-memory-leak`, on 2026-10-02: Aurora's tool-call transcripts for r1 and r2, each block attributed to its run and step and trimmed with `...` where noted, plus `evidence-before/` captured before the investigation (the first cycle's timestamps). - Versions, from `evidence-before/versions.yaml`: Kubernetes server v1.37.0 on kind v0.33.0, kubectl client v1.35.3. Aurora commit `87d22baf`, single agent, RCA model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge model `us.anthropic.claude-opus-5-5`, a second automated review, then a person at Arvo (`human_verdict` in each run's `grade.json`). Times, tool calls, tool errors and costs from `result.json`; the fix column from `fix_from_suggestions` in `grade.json`. Time runs from incident creation to the report: r1's 201 s includes 1 min 43 s before its first tool or model call, with nothing recorded in the gap; r2's gap was 5 s. - The memory arithmetic (4.9 MiB per batch, 49 MiB transient, 28 batches to the limit) is computed from the code and the limit; it matches the 28 log lines and the 140-second cycles but was not instrumented inside the process. The "about 1 MiB per second" rate rests on one `top` sample per run, at different uptimes, on two clusters. - Kubernetes documentation quoted, read 2026-10-19: [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/), [Pod Lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/), [kubectl top pod](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_top/kubectl_top_pod/). Metric names from the [kube-state-metrics pod metrics list](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md) and the [cAdvisor Prometheus metrics list](https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md), same date. - Not captured in the lab: the load-proportional row of the At a glance table, the repeated `top` loop, the `del recent[:-50]` patch, the fix and rollback commands (Aurora's policy blocks mutations), and the `predict_linear` expression. The Verify jsonpath is the lab's `check.sh` without its `grep`. --- ## Service has endpoints but connections are refused: targetPort does not match the container port (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-service-targetport-mismatch-kubernetes Published: 2026-10-19 Author: Damian Loch, Software Engineer at Arvo AI > Endpoints list the pod IPs, yet nginx logs Connection refused and returns 502: the Service targetPort points where nothing listens. Confirm it and fix it. | Applies to | Any ClusterIP Service whose `targetPort` names a port the selected containers do not listen on; shown on the `api` Service of the lab's `shop` app behind an nginx frontend | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, nginx 1.27-alpine (nginx/1.27.5), python:3.12-alpine, 2026-10-02 | | Lab folder | [scenarios/11-noisy-worker-svc-targetport](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/11-noisy-worker-svc-targetport), runs in [runs/v1-run1/11-noisy-worker-svc-targetport](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/11-noisy-worker-svc-targetport) | **A Service that lists endpoints and still refuses every connection is sending traffic to the wrong port on the right pods: its `targetPort` names a port nothing in the container listens on, so kube-proxy forwards each connection to a pod and the pod refuses it.** Confirm with `kubectl get endpoints -n `: in the lab the `api` endpoints read `10.244.1.131:9090,10.244.2.124:9090` while the api process had logged `api listening on :8080`. The fix is one patch to the Service's `targetPort`; nothing needs restarting. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | Endpoints list the pod IPs, every request gets `connect() failed (111: Connection refused)` and a 502, pods Ready with zero restarts | `targetPort` is not the port the container listens on | `kubectl describe svc ` (TargetPort, Endpoints) against the listening port in `kubectl logs ` | Patch `targetPort` to the listening port | | `endpoints/` reads ``, same client error | The selector matches no pods | `kubectl get svc -o yaml` against `kubectl get pods --show-labels` | See [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes) | | Endpoints populated, the client times out instead | The client dials a port the Service does not publish, or a NetworkPolicy blocks it | The client's upstream port against PORT(S) in `kubectl get svc`; `kubectl get networkpolicy` | Fix the client's port, or see [NetworkPolicy blocking traffic](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes) | | A neighbouring Deployment logs warnings and errors from about the same time | Noise, unless it sits on the request path | `kubectl get svc -o wide` for a Service that selects it; the proxy config for who it calls | Nothing here; its own ticket | ## How do you confirm it is the targetPort? Three port fields are in play and the client's error names the one that is right. The [Service docs](https://kubernetes.io/docs/concepts/services-networking/service/): "A Service can map any incoming port to a targetPort. By default and for convenience, the targetPort is set to the same value as the port field." So `port` is what clients dial on the ClusterIP (8080 here), `targetPort` is what kube-proxy forwards to on each pod (9090 after the fault), and `containerPort` is a declaration that opens nothing: the [API reference](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/) says "Not specifying a port here DOES NOT prevent that port from being exposed." The only port that matters on the pod is the one the process bound. Start with the client. The frontend nginx proxies `/api/` to the `api` Service (run r1, step 14, four of 30 lines): ```shell kubectl logs frontend-76d98597b9-ktzwl -n shop --tail=30 ``` ``` 2026/10/02 20:24:21 [error] 27#27: *49 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.2.126, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.190.97:8080/items", host: "frontend" 10.244.2.126 - - [02/Oct/2026:20:24:21 +0000] "GET /api/items HTTP/1.1" 502 157 "-" "Wget" "-" 2026/10/02 20:24:25 [error] 27#27: *51 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.2.126, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.190.97:8080/items", host: "frontend" 10.244.2.126 - - [02/Oct/2026:20:24:25 +0000] "GET /api/items HTTP/1.1" 502 157 "-" "Wget" "-" ``` The upstream is the ClusterIP on 8080, the port the Service publishes. Nothing in this log is wrong; 9090 never appears on the client side. It appears on the Service (r1, step 12, trimmed): ```shell kubectl describe svc api -n shop ``` ``` Selector: app=api Type: ClusterIP IP: 10.96.190.97 Port: 8080/TCP TargetPort: 9090/TCP Endpoints: 10.244.1.131:9090,10.244.2.124:9090 ``` Both api pods are endpoints, on port 9090. Now the pod (r1, step 7, five of 27 lines): ```shell kubectl logs api-f64486c4c-fjnsl -n shop --tail=100 ``` ``` redis connection failed (attempt 1/5): [Errno 111] Connection refused connected to redis at redis:6379 api listening on :8080 10.244.1.1 "GET /healthz HTTP/1.1" 200 - 10.244.1.133 "GET /items HTTP/1.0" 200 - ``` `listening on :8080` against `Endpoints: ...:9090` is the confirmation. kube-proxy rewrites each connection to `10.96.190.97:8080` into one to a pod IP on 9090, no process owns 9090, and the pod refuses it; nginx reports the refusal as if the Service had refused. Look-alikes. A Service with no endpoints produces the identical client line: in [scenario 07](/runbooks/fix-service-no-endpoints-kubernetes) nginx logged the same `connect() failed (111: Connection refused)` against a ClusterIP whose endpoints read ``. The endpoints column, not the error text, separates the two. A client dialling a port the Service does not publish gets a timeout instead: in [scenario 19](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes) nginx pointed at `api:8000` on a Service publishing only 8080 and logged `upstream timed out`. ## How much is affected, and since when? Every client that goes through the Service, and nothing that goes around it. The load generator's last 20 lines (r1, step 18) were all `GET /api/items -> 502`, from `20:25:27Z` to `20:26:05Z`. Both api pods kept answering `/healthz` with 200 throughout, because the kubelet probes the pod IP on the probe's own port, 8080, and never touches the Service. So every pod was `1/1 Running` with zero restarts (r1, step 4) and the Deployment reported `readyReplicas: 2` (r1, step 13). Readiness says the process is up, not that the Service reaches it. The start time is in the pod log above. The api served exactly one `GET /items`, from `10.244.1.133`, the frontend pod `dgg8q`; after that, only probes. The client agrees: the load generator's log captured before the investigation has `20:23:13Z GET /api/items -> 200` between startup 502s and the 502s that never stopped. One request got through between the api becoming Ready and the Service being patched; r2 shows the same (step 12). With real traffic, the last 200 in the api's access log is the moment the port changed. The change itself left a fingerprint (r1, step 17, trimmed): ```shell kubectl get svc api -n shop -o yaml ``` ```yaml metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"api","namespace":"shop"},"spec":{"ports":[{"port":8080,"targetPort":8080}],"selector":{"app":"api"}}} creationTimestamp: "2026-10-02T20:23:05Z" resourceVersion: "32995" spec: ports: - port: 8080 protocol: TCP targetPort: 9090 ``` The last applied manifest says `targetPort: 8080`; the live object says 9090. Someone changed the Service outside the manifest, with `kubectl patch` or `kubectl edit`. The Service's events were empty (`Events: `, step 12), so the annotation and the pod log date the change, not the event list. ## Which cause is it? Cheapest and likeliest first. **1. Are there endpoints?** Yes, two pod IPs (step 12 above). The selector is fine, so this is not the [no-endpoints](/runbooks/fix-service-no-endpoints-kubernetes) case. **2. Does the endpoint port match what the container listens on?** No: 9090 against `api listening on :8080`. This is the cause. The Deployment agrees, `containerPort: 8080` with both probes on 8080 (r1, step 13), and so does the source: r2 read the api's code out of its ConfigMap (r2, step 9) and found `PORT = int(os.environ.get("PORT", "8080"))` with no `PORT` set in the Deployment. Without a listen line in the log, read the code or run `ss -ltn` in the container; `containerPort` is documentation. **3. Is the noisy neighbour on the request path?** In the lab a worker Deployment had been redeployed seconds before the Service was patched (pod age 61 s against 69 s for everything else, r1, step 4), and its log is the loudest thing in the namespace (r1, step 19, four of 30 lines): ```shell kubectl logs worker-645795c659-fqn5f -n shop --tail=30 ``` ``` WARNING py.warnings: DeprecationWarning: StrictRedis is deprecated and will be removed in redis-py 6.0, use Redis processed batch of 500 items ERROR metrics: push to pushgateway:9091 failed: [Errno -2] Name does not resolve (will retry next batch) WARNING py.warnings: DeprecationWarning: StrictRedis is deprecated and will be removed in redis-py 6.0, use Redis ``` Two commands rule it out. `kubectl get svc -n shop` lists Services for `api`, `frontend` and `redis` only (r1, step 9): no Service selects the worker, so no request can reach it. The frontend's nginx config proxies `/api/` to `http://api/` and nowhere else (r2, step 9). And the name that fails is `pushgateway`, which nothing on the request path uses; DNS itself works, since nginx resolved `api` to `10.96.190.97` and the api connected to `redis:6379` by name. Between two simultaneous changes, the request path decides, not the log volume. **4. Was the port change intended?** The annotation says the Service was declared with 8080, and no Deployment change accompanied it (`revision: "1"`, r1, step 13). ## What is the fix? Put `targetPort` back to the port the container listens on. The lab's fault was one JSON patch, so the fix is the same patch with the right value: ```shell kubectl -n shop patch svc api --type=json \ -p='[{"op":"replace","path":"/spec/ports/0/targetPort","value":8080}]' ``` Risk: low. A Service edit restarts nothing; the endpoints controller and kube-proxy pick up the new port on their own. The risk is choosing the wrong number, so take it from the process's log line or the code, not from `containerPort`. Rollback: the same command with the previous value, or `kubectl apply` the Service from source control, which also realigns the annotation. Save `kubectl -n shop get svc api -o yaml` first. No mitigation is needed and no pod should be restarted: the pods were right the whole time. If the move to 9090 was deliberate, ship pods that listen on 9090 (`PORT=9090` in the Deployment's env) before changing the Service. ## How do you verify it worked? Pass condition: the endpoints carry the listening port, and a full minute of 200s from the client with no new `connect() failed` line from any frontend pod. ```shell kubectl -n shop get endpoints api kubectl -n shop logs deploy/loadgen --tail=30 kubectl -n shop logs -l app=frontend --tail=20 --prefix | grep 'connect() failed' ``` The first must read `10.244.1.131:8080,10.244.2.124:8080` (r1's pod IPs). The load generator logs one line every two seconds, so `--tail=30` is a minute; every line should end `-> 200`. The third should print nothing. The first line of the lab's own check for the broken state is `kubectl -n shop logs deploy/loadgen --tail=5 | grep -q -- '-> 502'`; its negation is your pass. ## How do you prevent it? **Name the port and reference the name.** The Service docs: "Port definitions in Pods have names, and you can reference these names in the targetPort attribute of a Service." With `name: http` on the container port and `targetPort: http` in the Service, a port move is one change in one place, and a numeric patch against a named port stands out in review. **Probe through the Service.** The alert here came from the load generator, which calls the Service as a user does; pod probes passed throughout. A blackbox check against `..svc:` from inside the cluster, alerting on `probe_success == 0`, catches a Service-layer break that no pod-level signal will. (Suggested; not part of the lab.) **Guardrail.** Services change through the same pipeline as the Deployments they front; an annotation that disagrees with `spec.ports` is the signature of a `kubectl patch` on a live object. A CI step comparing each Service's `targetPort` with the `containerPort` list of the pods it selects turns the confirm step above into a test. ## Common mistakes - **Restarting the api or the frontend.** A wrong answer in the scenario. The pods come back listening on 8080 and the Service still forwards to 9090. - **Blaming the worker, or DNS, because of `Name does not resolve`.** The worker is not on the request path and the unresolved name is one it alone uses. - **Reading `:8080` in the nginx error and concluding the ports match.** That is the Service port. The forwarded port is only visible in `get endpoints` or `describe svc`. - **Trusting Ready.** Probes go to the pod IP on the probe port and bypass the Service. - **Changing `containerPort` to 9090 to match.** The process still listens on 8080. - **Changing the client to `api:9090`.** The Service does not publish 9090, so the refusal becomes a timeout (scenario 19). ## What we saw in the lab Scenario 11 in the [Aurora lab](https://github.com/Arvo-AI/aurora-lab): a five-component `shop` app on a three-node kind cluster. The break script rewrote `worker.py` in the shared code ConfigMap so the worker prints a `DeprecationWarning` every batch and a metrics-push ERROR every third, rolled the worker Deployment and waited for it, then patched the `api` Service's `targetPort` from 8080 to 9090. Aurora got the alert text ("HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes (threshold 5%).") and read-only kubectl under the default `observability_only` policy. Two findings were required: the targetPort mismatch, and that the worker's noise is not the cause. Two runs, each in a fresh organization. | Run | Verdict (human reviewed) | Time | Tool calls | Aurora cost | |---|---|---|---|---| | r1 | correct | 144 s | 19 | $0.44 | | r2 | partial | 535 s | 16 | $0.57 | r1 had the answer at step 9, its sixth kubectl call, when `get svc,endpoints` put `endpoints/api 10.244.1.131:9090,10.244.2.124:9090` under `service/api ... 8080/TCP`. It confirmed with `describe svc`, the Deployment YAML and the annotation, and read the worker last, at step 19. Its report: "This caused kube-proxy to route all Service traffic to pod port 9090, where nothing was listening, resulting in connection refusals that nginx surfaced as 502s". The worker went under Not Checked: "The worker pod's `pushgateway:9091` DNS errors were observed but determined to be a separate, pre-existing cosmetic issue unrelated to this alert's causal chain, and were not further investigated". Both findings met, with two over-claims the graders caught. "Pre-existing" is wrong: the worker was redeployed seconds before the fault, and the pod ages in its own step 4 showed it. The report also said the frontend's readiness probes were "failing during this window"; the only readiness failures in the events were at startup. Its separate fix suggestion, patch `targetPort` back to 8080, was graded acceptable. r2 took 535 s with fewer tool calls, and the investigation itself was not slower. The kubectl time was the same in both runs, about 68 s (summed tool durations in each timeline), and from first tool call to conclusion r2 took 137 s; r1 took 138 s by the same measure. The difference is before the first call: r1 started 5 s after the incident was created, r2 after 6 min 37 s, and the timeline records no tool call and no model call in that gap, so the table's 535 s is mostly wait before the investigation began, for a reason the run data does not show. Its longest single call was step 9, where `describe configmap` returned the full source of both programs plus a CA certificate (15 s). It read the api's code for the listening port, pulled the Endpoints object showing `port: 9090`, and wrote the sentence this page is built on: "Kubelet-direct health checks (bypassing the Service) to both frontend and api pods succeeded throughout this window, meaning the user-facing path was fully broken while pod-level health signals remained green". It was graded partial because it never looked at the worker: its Not Checked list has "Health and configuration of the redis, worker, and loadgen deployments" with the reason "no symptoms or alerts pointed to these components, so they were not examined", so the second required finding was inferred, not shown. The judge also flagged "the fix was validated but not applied" as a claim with no fix in the report. The patch command did exist, in Aurora's working notes and in the memory it wrote at step 16, and the separate fix suggestion was graded acceptable, but the report did not contain it. Both runs named the port in under twenty calls and neither was tempted by restarts, DNS or the worker. The difference is whether the report shows that the loudest log in the namespace was looked at and dismissed. An engineer who has seen the worker's ERROR lines will not take "not examined" as a reason to ignore them. ## How Aurora handles this Aurora receives the alert, connects to the cluster read-only through an uploaded kubeconfig, and runs the kubectl commands shown above, choosing them itself. On this failure it read the Service, the Endpoints and the pod log in both runs, compared the endpoint port with the listening port, and used the `last-applied-configuration` annotation to call the change an out-of-band edit. Mutating commands are blocked by policy, so the fix arrives as a suggestion for a person to apply. One run checked the noisy neighbour and said so; the other did not. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) Related: [a Service with no endpoints](/runbooks/fix-service-no-endpoints-kubernetes), [a NetworkPolicy blocking traffic](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes), [a hostname that does not resolve](/runbooks/fix-dns-name-not-resolving-kubernetes), [a ConfigMap change that reached only some pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes), [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), [a Service selecting pods from the wrong Deployment](/runbooks/fix-service-selects-wrong-pods-label-overlap-kubernetes), and what an [Aurora root cause analysis](/rca) looks like. ## Sources and verification - Every command and output block above comes from the lab run `v1-run1`, scenario `11-noisy-worker-svc-targetport`, on 2026-10-02: Aurora's tool-call transcripts for r1 and r2 and the cluster state captured before the investigation (`evidence-before/`). The fault was injected by the scenario's `break.sh` (a ConfigMap rewrite, `rollout restart deploy/worker`, then a JSON patch of `/spec/ports/0/targetPort` to 9090); the fix above is the same patch with 8080. - Versions, from `evidence-before/versions.yaml` and the pod logs: Kubernetes server v1.37.0 on kind v0.33.0 (three nodes, containerd 2.3.4), kubectl client v1.35.3; nginx 1.27-alpine reporting nginx/1.27.5; python:3.12-alpine for the api and worker. - Aurora commit `87d22baf`, single agent, RCA model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge model `us.anthropic.claude-opus-5-5`, reviewed by a person at Arvo (`human_verdict` in each `grade.json`); fix suggestions graded separately (`fix_from_suggestions`). Durations, tool calls and costs from each run's `result.json`; the 68 s kubectl totals are summed `duration_ms` from each `timeline.json`, and the first-call gaps and the 137 s and 138 s are from the same file's `started_at` and step and model-call timestamps. - Documentation quoted, read 2026-10-19: Kubernetes [Service](https://kubernetes.io/docs/concepts/services-networking/service/) (the note under "Defining a Service" and the first sentence of "Port definitions"), the [Pod API reference](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/) (the `ports` field of Container), and [Debug Services](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/), whose checklist asks "Is the targetPort correct for your Pods (some Pods use a different port than the Service)?". - The fix and verify commands were not run inside a graded lab run; Aurora's policy blocks mutations, so Verify states the pass condition and the lab's own check instead of a captured result. The blackbox probe and CI comparison under Prevent are suggestions, not captured alerts. --- ## ConfigMap change not applied to all pods: replicas serve different configs (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes Published: 2026-10-19 Author: Damian Loch, Software Engineer at Arvo AI > After a ConfigMap edit, only pods started since the edit run the new config; others keep what they loaded at startup. Compare per-pod logs, fix, restart. | Applies to | Any pod that reads a mounted ConfigMap once at startup (nginx here) and keeps running after the ConfigMap is edited; shown on the two-replica frontend of the lab's `shop` app | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, nginx 1.27-alpine (nginx/1.27.5), 2026-10-02 | | Lab folder | [scenarios/19-config-drift-partial-restart](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/19-config-drift-partial-restart), runs in [runs/v1-run1/19-config-drift-partial-restart](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/19-config-drift-partial-restart) | **When a ConfigMap change applies to only some pods, the replicas are running different configurations: after an edit to a mounted ConfigMap and a partial restart (one pod deleted, one node drained, one crash), pods started after the edit loaded the new file while pods started before it still run what they read at startup, even though the file in their volume has since been updated underneath them.** Confirm by comparing the same log line across replicas: in the lab one nginx pod proxied to `10.96.10.144:8000` and returned 504 while its sibling, with the same ConfigMap mounted, proxied to `10.96.10.144:8080` and returned 200. Make the ConfigMap right first, then restart every pod that mounts it, in that order. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | About 1/N of requests fail for N replicas, every pod Ready, zero restarts, Deployment revision unchanged | The ConfigMap was edited after the pods started; only recreated pods loaded it | `kubectl get pods -o wide` for one pod younger than its siblings, then `kubectl logs` per pod and compare the error lines | Correct the ConfigMap, then `kubectl rollout restart` the Deployment | | Every request to the route fails from every pod | The new config is wrong and every pod has loaded it | `kubectl get configmap -o yaml` against the Service it points to | Correct the ConfigMap, restart again | | New pods crash-loop while old ones serve | The new config does not parse | `kubectl logs --previous` shows `[emerg]` | See [nginx [emerg] unknown directive](/runbooks/fix-nginx-emerg-unknown-directive-kubernetes) | | Every pod logs `connect() failed (111: Connection refused)` on the right port | The upstream Service has no endpoints | `kubectl get endpoints ` | See [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes) | ## How do you confirm pods are running different configs? Start from the client, then split by pod. The lab's load generator calls `/api/items` every two seconds through the `frontend` Service (run r1, step 9, trimmed): ```shell kubectl logs loadgen-7c6676864-9vwtl -n shop --tail=100 ``` ``` 2026-10-02T20:35:52Z GET /api/items -> 504 2026-10-02T20:35:54Z GET /api/items -> 200 2026-10-02T20:35:56Z GET /api/items -> 200 2026-10-02T20:36:01Z GET /api/items -> 504 2026-10-02T20:36:03Z GET /api/items -> 200 2026-10-02T20:36:07Z GET /api/items -> 504 2026-10-02T20:36:11Z GET /api/items -> 504 2026-10-02T20:36:15Z GET /api/items -> 504 ``` Successes and failures alternate with no trend: a Service balancing across one broken and one healthy backend. The pods look fine (r1, step 4, frontend rows of seven): ```shell kubectl get pods -n shop -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES frontend-76d98597b9-g5zrg 1/1 Running 0 3m9s 10.244.1.44 aurora-lab-3-worker frontend-76d98597b9-kt2jr 1/1 Running 0 3m18s 10.244.2.53 aurora-lab-3-worker2 ``` Every pod in the namespace was `1/1 Running` with zero restarts; the frontend pods share a ReplicaSet and differ only in age, and those nine seconds are one pod being deleted and recreated. Now read each pod's log on its own. The younger pod (r1, step 7, one of the pairs the log repeats every few seconds): ```shell kubectl logs frontend-76d98597b9-g5zrg -n shop --tail=100 ``` ``` 10.244.1.43 - - [02/Oct/2026:20:35:48 +0000] "GET /api/items HTTP/1.1" 504 167 "-" "Wget" "-" 2026/10/02 20:35:48 [error] 29#29: *8 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.244.1.43, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.10.144:8000/items", host: "frontend" ``` The older pod (r1, step 8, trimmed): ```shell kubectl logs frontend-76d98597b9-kt2jr -n shop --tail=100 ``` ``` 2026/10/02 20:35:25 [notice] 1#1: nginx/1.27.5 2026/10/02 20:35:28 [error] 27#27: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.1.43, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.10.144:8080/items", host: "frontend" 10.244.1.43 - - [02/Oct/2026:20:35:28 +0000] "GET /api/items HTTP/1.1" 502 157 "-" "Wget" "-" 10.244.1.43 - - [02/Oct/2026:20:35:34 +0000] "GET /api/items HTTP/1.1" 200 38 "-" "Wget" "-" ``` Same request, same client, same image, two upstream addresses: `:8000` in the pod that fails, `:8080` in the pod that works. nginx logs the upstream it resolved from the configuration it loaded, so these lines are as close as you can get to reading each process's configuration in memory. That is the confirmation. The older pod's single `502` at 20:35:28 is a different failure, `Connection refused` on the correct port three seconds after startup, before the api pods were ready; it never recurred. Look-alikes: if every pod logs the wrong upstream, every pod has loaded the bad config and there is no drift to find; a crash-looping new pod is a [config that does not parse](/runbooks/fix-nginx-emerg-unknown-directive-kubernetes); `Connection refused` from every pod on the right port is a Service with [no endpoints](/runbooks/fix-service-no-endpoints-kubernetes). ## How much is affected, and since when? The alert said 48% of requests to `/api/` over ten minutes. The load generator's last 78 lines in r1 (step 9) agree: 40 returned `200`, 35 returned `504`, and 3 were startup `502` or `timeout`. One bad pod of two costs about half the traffic on the route, and only that route: `/healthz` returned 200 on both pods throughout (r1, steps 7 and 8), which is why the readiness probe never removed the bad pod from the Service. Since when: the Deployment did not change. `kubectl get deploy frontend -n shop -o yaml` shows `deployment.kubernetes.io/revision: "1"`, `generation: 1` and `updatedReplicas: 2` (r1, step 12); a ConfigMap edit does not touch the pod template, so `kubectl rollout history` has nothing to tell you. The events hold the restart instead (r1, step 5, three of 52 lines): ```shell kubectl get events -n shop --sort-by=.lastTimestamp ``` ``` 3m11s Normal SuccessfulCreate replicaset/frontend-76d98597b9 Created pod: frontend-76d98597b9-g5zrg 3m11s Normal Scheduled pod/frontend-76d98597b9-g5zrg Successfully assigned shop/frontend-76d98597b9-g5zrg to aurora-lab-3-worker 3m11s Normal Killing pod/frontend-76d98597b9-2vqr4 Stopping container nginx ``` One pod killed and one created in the same second: the ReplicaSet replacing a deleted pod. `kubectl describe pod` gives the exact times, `Start Time: Fri, 02 Oct 2026 20:35:34 +0000` for `g5zrg` against `20:35:25 +0000` for `kt2jr` (r1, steps 14 and 15). The ConfigMap holds the other half of the timeline (r1, step 13, trimmed): ```shell kubectl get configmap frontend-nginx -n shop -o yaml ``` ```yaml apiVersion: v1 data: default.conf: | upstream api { server api:8000; } server { listen 80; location /healthz { return 200 "ok\n"; } location /api/ { proxy_pass http://api/; proxy_connect_timeout 2s; } location / { return 200 "shop frontend\n"; } } kind: ConfigMap metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","data":{"default.conf":"upstream api {\n server api:8000;\n} ... "},"kind":"ConfigMap","metadata":{"annotations":{},"creationTimestamp":"2026-10-02T20:35:25Z","name":"frontend-nginx","namespace":"shop","resourceVersion":"14417","uid":"d0a60a78-27aa-4974-b47d-55ec7ba0351c"}} creationTimestamp: "2026-10-02T20:35:25Z" name: frontend-nginx namespace: shop resourceVersion: "14598" ``` Two things in that output. The upstream is `api:8000`, while `kubectl get svc -n shop` shows the `api` Service publishing `8080/TCP` only (r1, step 6). And the `last-applied-configuration` annotation holds a `resourceVersion` of `14417` while the live object is at `14598`: someone fetched the ConfigMap, changed it and applied it back, the fingerprint of an in-place edit (`kubectl edit`, or `get | sed | apply`, which is what the lab did). The creation time, `20:35:25Z`, is the second the old pod started; the edit happened between then and `20:35:34Z`, when the new pod started and read the new file. ## Which cause is it? Cheapest and likeliest first. Stop when the output answers the question. **1. Is the backend up?** Half the requests succeed, so something answers on the right port. Check anyway; it is one command, and it removes the wrong answer both of Aurora's runs leaned toward at some point (r2, step 21): ```shell kubectl describe svc api -n shop ``` ``` Port: 8080/TCP TargetPort: 8080/TCP Endpoints: 10.244.1.52:8080,10.244.2.62:8080 ``` Both api pods are endpoints, and their logs show `api listening on :8080` followed by nothing but `200` responses (r2, steps 8 and 10). The backend is not the problem. The failing pod's error agrees: `upstream timed out`, not `Connection refused`, because no Service publishes port 8000 on that ClusterIP, so nothing answers and nginx waits out the 2 s `proxy_connect_timeout` before returning 504. **2. Does the upstream in each pod's log match the ConfigMap?** From the confirm step: `:8000` in the young pod, `:8080` in the old one. The ConfigMap says `api:8000`. So the young pod runs the ConfigMap as it is now, and the old pod runs it as it was. This is the cause; the rest is proof and what not to trust. **3. Is the young pod young because of the edit?** The events and start times above say one pod was deleted and replaced nine seconds after the Deployment came up; whether someone restarted it to pick up the edit or it restarted on its own, the mechanism is the same. **4. Can you read the loaded config directly?** Not from the file. Kubernetes updates the mounted file in running pods too: "When a ConfigMap currently consumed in a volume is updated, projected keys are eventually updated as well." ([Kubernetes docs, ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/)). So `cat /etc/nginx/conf.d/default.conf` inside the old pod shows `api:8000` once the kubelet has synced, and proves nothing about what nginx is running, since nginx applies a changed file only on reload or restart ([nginx Beginner's Guide](https://nginx.org/en/docs/beginners_guide.html)). The process's log lines and start time are your evidence. In the lab, Aurora's `exec` was blocked by policy in both runs (r1 step 10, r2 step 20): ``` Command blocked by organization policy: Mutating kubectl operations ``` Both runs found the port mismatch without it; only r2 went on to name the cause. ## What is the fix? Put the ConfigMap back, then restart every pod that mounts it. The order matters: a restart on its own recreates every pod with the bad ConfigMap, and half the requests failing becomes all of them. The lab's fault was a `sed` on the ConfigMap followed by deleting one pod, so the fix is the inverse: ```shell kubectl -n shop get cm frontend-nginx -o json \ | sed 's/server api:8000;/server api:8080;/' \ | kubectl apply -f - kubectl -n shop rollout restart deploy/frontend ``` Risk: low for the ConfigMap edit, one token that only pods created from now on read. The restart rolls under `RollingUpdate` with `maxSurge: 25%` and `maxUnavailable: 25%` (r1, step 12): with two replicas, one extra pod at a time and no old pod removed before its replacement is Ready. One caveat: `/healthz` is served by nginx itself (`return 200 "ok\n"` in the ConfigMap) without touching the upstream, so a wrong upstream passes readiness; `g5zrg` was `Ready: True` (r1, step 14) while failing every `/api/` request. The rollout will not catch a bad fix; the client logs will. Rollback: `kubectl apply` the ConfigMap from source control and restart again. Save `kubectl -n shop get cm frontend-nginx -o yaml` before you edit so you hold the exact previous object. Mitigation, separate from the fix: if you need the error rate down before you can roll the whole Deployment, revert the ConfigMap first and then delete only the failing pod, `kubectl -n shop delete pod frontend-76d98597b9-g5zrg`; its replacement reads the reverted ConfigMap at creation. Risk: the Service drops to one endpoint until the replacement is Ready, and a delete before the revert produces a replacement with the same bad config; do the full `rollout restart` afterwards anyway. If the edit was intended and the api really is moving to port 8000, change the Service and api pods first, then restart the frontend so every replica agrees. ## How do you verify it worked? Pass condition: a full minute of `200` on `/api/items` from the client, no frontend pod logging an upstream on `:8000`, and both frontend pods started after the fix. ```shell kubectl -n shop rollout status deploy/frontend kubectl -n shop get pods -l app=frontend -o wide kubectl -n shop logs deploy/loadgen --tail=30 kubectl -n shop logs -l app=frontend --tail=50 --prefix ``` [`--prefix`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_logs/) labels each line with its pod, so a surviving `:8000/items` stands out. The lab client logs one line every two seconds, so `--tail=30` is about a minute. For these runs the lab's own check grepped the last 30 client lines for `-> 502` next to `-> 200`; it passed only because it ran right after the break, while the startup `502`s were still in the tail, since the steady state is `504`. The check was fixed on 2026-10-19 to accept either code. Do the same: ```shell kubectl -n shop logs deploy/loadgen --tail=30 | grep -E -- '-> 50[24]' && echo still broken ``` Watch two minutes after the rollout finishes: one bad pod produces a `504` every few seconds, so two clean minutes means none is behind the Service. ## How do you prevent it? **Make a ConfigMap change a rollout.** Put a hash of the ConfigMap in the pod template, for example an annotation `checksum/config: `, the pattern Helm documents as [Automatically Roll Deployments](https://helm.sh/docs/howto/charts_tips_and_tricks/). Every ConfigMap change then changes the template and rolls all replicas together, instead of waiting for whichever pod restarts first. The alternative is to mark the ConfigMap `immutable: true` and give each version a new name referenced from the Deployment, which the Kubernetes docs say "protects you from accidental (or unwanted) updates that could cause applications outages" ([Kubernetes docs, ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/)). **Alert on per-replica divergence, not on the aggregate.** A 48% error rate on the Service is one number; split by pod it is 100% on one and 0% on the other, which names the cause. If your proxy exports per-pod metrics, alert on a 5xx ratio per pod. With kube-state-metrics alone, a start-time spread between pods of one ReplicaSet is a drift signal: ``` max by (namespace, owner_name) (kube_pod_start_time * on (namespace, pod) group_left (owner_name) kube_pod_owner{owner_kind="ReplicaSet"}) - min by (namespace, owner_name) (kube_pod_start_time * on (namespace, pod) group_left (owner_name) kube_pod_owner{owner_kind="ReplicaSet"}) > 300 ``` It also fires on any single-pod restart, so treat it as a prompt to compare configs, not a page. (Suggested query, not captured in the lab.) **Guardrail.** Edit ConfigMaps from source control through the same pipeline as the Deployment, never with `kubectl edit` on a ConfigMap that pods mount; the `resourceVersion` inside `last-applied-configuration` is how you spot an in-place edit after the fact. ## Common mistakes - **Restarting the Deployment without fixing the ConfigMap.** Every replica loads `api:8000` and half the requests failing becomes all of them; the lab scenario lists this as a wrong answer. - **Blaming the api.** `upstream timed out` reads like a slow backend, but the backend logged `200` for every request it received; the failing requests never reached it. Both Aurora runs wrote an intermediate note saying the backend was down before the per-pod comparison corrected it. - **Trusting `kubectl get configmap`.** It shows the desired state; the old pod here ran `api:8080` while the ConfigMap said `api:8000`. - **Trusting `cat` inside the pod.** After the kubelet sync the file shows the new content in the pod that is still running the old one. - **Deleting the bad pod as the fix.** Its replacement reads the same ConfigMap and fails the same way. Revert first. ## What we saw in the lab Scenario 19 in the [Aurora lab](https://github.com/Arvo-AI/aurora-lab): a five-component `shop` app on a three-node kind cluster, the `frontend-nginx` ConfigMap re-applied with `server api:8000;` in place of `server api:8080;`, one of the two frontend pods deleted, then Aurora given only the alert text ("HighErrorRate [warning] namespace=shop service=frontend: 48% of requests to /api/ returned 5xx over the last 10 minutes (threshold 5%).") and read-only kubectl under the default `observability_only` policy. Two runs, each in a fresh organization. | Run | Verdict (human reviewed) | Time | Tool calls | Aurora cost | |---|---|---|---|---| | r1 | partial | 280 s | 15 | $0.39 | | r2 | correct | 253 s | 28 | $0.84 | r2 found it. Its report: "Because nginx only reads this config at container startup and does not hot-reload, the two frontend replicas diverged: `rr7xl`, which started before the bad edit, continued running a working configuration and successfully proxied to the backend, while `r4thz`, which came up after the ConfigMap was changed, mounted the broken config and returned repeated `504 Gateway Timeout` errors on every `/api/items` request due to upstream connection timeouts to `10.96.137.47:8000`". It listed what it could not check: `rollout history` and `exec` were blocked by policy, so "the 'good' replica's config could only be inferred from its logs, not confirmed by file content". The path was not direct: 28 tool calls, four of them errors, the failing pod's log read only at step 24, and a summary of that log calling the backend "unreachable or down" before the final analysis put the two pods side by side. The judge noted the report gave no remediation step; the fix it suggested separately, patch the ConfigMap to port 8080 and roll the frontend pods, was graded acceptable. r1 is the miss, and the instructive one. It had every fact by step 13: both pods' logs, the Service on 8080, the ConfigMap on 8000, the pod ages, the events with the kill-and-create pair. Its report said: "The most likely cause is a port mismatch between the frontend's configured upstream and the actual backend service endpoint." True, and not the root cause. It never said the ConfigMap had been edited, never said only the recreated pod had loaded it, and never said the healthy pod was running a configuration that no longer exists in the API. Instead: "This discrepancy was not fully reconciled", because the `exec` was blocked, and "intermittent backend unavailability as a contributing factor cannot be ruled out", although it had not read the api logs and said so under Not Checked. The judge's verdict was partial: "It never says the ConfigMap was edited, or that the healthy pod still runs the old api:8080 config from memory because nginx only reads its config at start. It recommends no fix". A second reviewer model agreed and pointed out that the evidence was in hand, the pod age gap and the ConfigMap `resourceVersion` that differed from the last-applied one. A person at Arvo reviewed the grade and recorded partial. r1's suggested fix, graded on its own, was still right: patch the ConfigMap to `api:8080` and roll the frontend. The difference between the runs is what counts as a cause. "The ConfigMap says 8000 and the Service says 8080" describes the state. "The ConfigMap was changed and one pod restarted" explains why half the fleet differs, and it is the sentence that tells the on-call engineer not to restart the other pod. r2 wrote that sentence. r1 wrote the first one and stopped. ## How Aurora handles this Aurora receives the alert, connects to the cluster read-only through an uploaded kubeconfig, and runs the kubectl commands shown above, choosing them itself. On this failure it read each frontend pod's log separately, compared the upstream in each with the Service's port and the ConfigMap, and wrote a report that cites each tool output by number and lists what it could not check. Mutating commands are blocked by policy, so the fix arrives as a suggestion for a person to apply. One of the two runs named the full cause; the other stopped at the port mismatch, and this page says so. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) Related: [nginx [emerg] unknown directive](/runbooks/fix-nginx-emerg-unknown-directive-kubernetes), [a Service with no endpoints](/runbooks/fix-service-no-endpoints-kubernetes), [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), and what an [Aurora root cause analysis](/rca) looks like. ## Sources and verification - Every command and output block above comes from the lab run `v1-run1`, scenario `19-config-drift-partial-restart`, on 2026-10-02: Aurora's tool-call transcripts for r1 and r2 and the cluster state captured before the investigation (`evidence-before/`). The fault was injected by the scenario's `break.sh` (a `sed` on the ConfigMap piped through `kubectl apply`, then `kubectl delete` of one frontend pod); the fix above is its inverse. The 78-line count and 40/35/3 split are from the r1 step 9 output. - Versions, from `evidence-before/versions.yaml` and the pod logs: Kubernetes server v1.37.0 on kind v0.33.0 (three nodes), kubectl client v1.35.3; nginx 1.27-alpine reporting nginx/1.27.5. - Aurora commit `87d22baf`, single agent, RCA model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge model `us.anthropic.claude-opus-5-5`, reviewed by a person at Arvo (`human_verdict` in each `grade.json`); fix suggestions graded separately (`fix_from_suggestions`). - Documentation quoted or linked, all read 2026-10-19: Kubernetes [ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/) (sections "Mounted ConfigMaps are updated automatically" and "Immutable ConfigMaps") and the [kubectl logs reference](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_logs/) for `--prefix`; Helm, [Automatically Roll Deployments](https://helm.sh/docs/howto/charts_tips_and_tricks/); nginx, [Beginner's Guide](https://nginx.org/en/docs/beginners_guide.html). - The fix and mitigation commands were not run inside a graded lab run; Aurora's policy blocks mutations, so Verify states the pass condition and the lab's own check instead of a captured result. That `check.sh` keyed on `502` when these runs were recorded and was changed on 2026-10-19 to accept `502` or `504`. The kube-state-metrics query is a suggestion, not a captured alert. --- ## Pod stuck in Init:0/1: an init container never finishes (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-pod-stuck-init-container-kubernetes Published: 2026-10-19 Author: Damian Loch, Software Engineer at Arvo AI > A pod stuck in Init:0/1 has an init container that never exits, often a wait loop on a missing Service. Read its logs, check the Service, fix the spec. | Applies to | Any pod whose init container polls a dependency in a loop and never exits; shown here on a Deployment rollout where the loop waits for a Service name that does not exist | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, 2026-10-02 | | Lab folder | [scenarios/18-init-waits-missing-service](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/18-init-waits-missing-service), runs in [runs/v1-run1/18-init-waits-missing-service](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/18-init-waits-missing-service) | **A pod stuck in Init:0/1 has one init container that has not finished, and because the app containers only start after every init container exits successfully, the pod stays Pending for as long as that container keeps running.** The most common cause is a wait loop (`nc`, `curl`, `pg_isready`) polling a Service that does not exist, or is not where the loop looks for it. Confirm it with `kubectl logs -c `: in the lab the loop printed `nc: bad address 'postgres'` every two seconds. The fix is to remove the init container, point it at a dependency that exists, or roll the Deployment back. Creating a database to satisfy a check the application never uses is not a fix. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | `Init:0/1`, init container `State: Running` with `Restart Count: 0`, only Normal events, its logs repeat `bad address` or `waiting for ` | A wait loop on a hostname with no Service behind it, in this namespace or at all | `kubectl logs -c `, then `kubectl get svc -n ` and `kubectl get svc -A` | Remove the init container, fix the hostname, or roll back | | `Init:0/1`, the loop is running, the Service exists | The Service has no ready endpoints or the port is wrong | `kubectl get endpoints -n ` | See [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes) | | `Init:Error` or `Init:CrashLoopBackOff` | The init container exits non-zero (script error, missing file, a bounded wait that gave up) | `kubectl logs -c --previous` | Fix the script; the kubelet restarts it until it succeeds | | `Init:0/1` and the init container is `Waiting` with `ImagePullBackOff` or `CreateContainerConfigError` | The init container's image or config, not its command | `kubectl describe pod `, the init container's State | See [ImagePullBackOff](/runbooks/fix-imagepullbackoff-kubernetes) or [CreateContainerConfigError](/runbooks/fix-createcontainerconfigerror-kubernetes) | | Pod STATUS `PodInitializing`, condition `Initialized: True` | Init is finished; the app containers are being created | Wait a few seconds, then read the app container's logs | Not an init container problem | ## How do you confirm the pod is stuck on an init container? Start with the pod list. The STATUS column says `Init:0/1` and the pod has a node and an IP, so it was scheduled and then stopped (lab run r1, step 5): ```shell kubectl get pods -n shop -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-77b4b6956f-pkcqj 0/1 Init:0/1 0 69s 10.244.2.55 aurora-lab-2-worker2 api-f64486c4c-fb7xz 1/1 Running 0 78s 10.244.1.41 aurora-lab-2-worker api-f64486c4c-t5vss 1/1 Running 0 78s 10.244.2.52 aurora-lab-2-worker2 frontend-76d98597b9-n5447 1/1 Running 0 78s 10.244.1.42 aurora-lab-2-worker frontend-76d98597b9-z58gc 1/1 Running 0 78s 10.244.2.54 aurora-lab-2-worker2 loadgen-7c6676864-nm28w 1/1 Running 0 78s 10.244.1.43 aurora-lab-2-worker redis-59458f55d8-66gxg 1/1 Running 0 78s 10.244.2.51 aurora-lab-2-worker2 worker-6cf88d4b58-gg2b8 1/1 Running 0 78s 10.244.2.53 aurora-lab-2-worker2 ``` The [Kubernetes init container debugging page](https://kubernetes.io/docs/tasks/debug/debug-application/debug-init-containers/) defines the status: `Init:N/M` means "The Pod has `M` Init Containers, and `N` have completed so far." One init container, none finished. Describe the pod. Read the init container's state, the app container's state and the events together (r1, step 7, trimmed): ```shell kubectl describe pod api-77b4b6956f-pkcqj -n shop ``` ``` Status: Pending IP: 10.244.2.55 Controlled By: ReplicaSet/api-77b4b6956f Init Containers: wait-for-db: Image: busybox:1.37 Command: sh -c until nc -z -w 2 postgres 5432; do echo waiting for postgres:5432; sleep 2; done State: Running Started: Fri, 02 Oct 2026 20:34:56 +0000 Ready: False Restart Count: 0 Containers: api: Container ID: Image: python:3.12-alpine State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Conditions: Type Status PodReadyToStartContainers True Initialized False Ready False ContainersReady False PodScheduled True Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 88s default-scheduler Successfully assigned shop/api-77b4b6956f-pkcqj to aurora-lab-2-worker2 Normal Pulled 88s kubelet spec.initContainers{wait-for-db}: Container image "busybox:1.37" already present on machine and can be accessed by the pod Normal Created 88s kubelet spec.initContainers{wait-for-db}: Container created Normal Started 88s kubelet spec.initContainers{wait-for-db}: Container started ``` Four things decide it. `PodScheduled: True` with a node and an IP rules out scheduling, whatever Pending suggests. The init container is `Running` with `Restart Count: 0`: it has not crashed, it has not exited. The app container has an empty `Container ID` and `Reason: PodInitializing`: never created, so no logs. And every event is Normal. The [init containers documentation](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) says "If a Pod's init container fails, the kubelet repeatedly restarts that init container until it succeeds." A loop that never exits never fails, so none of that runs: no restart, no `Init:Error`, no Warning event. The failure is visible only in the init container's own output. Read it with `-c` and the init container's name (r1, step 12, the last six of the fifty lines returned): ```shell kubectl logs api-77b4b6956f-pkcqj -n shop -c wait-for-db --tail=50 ``` ``` nc: bad address 'postgres' waiting for postgres:5432 nc: bad address 'postgres' waiting for postgres:5432 nc: bad address 'postgres' waiting for postgres:5432 ``` `bad address` is the name lookup failing, not a refused connection: cluster DNS has nothing called `postgres` to answer with. If the loop printed only its own `waiting for` line, the name resolved and the connect failed, which points at the Service's endpoints instead (second row of the table). Without `-c` the request goes to the app container and fails, the first wrong turn most people take (captured by the lab before Aurora ran): ``` Error from server (BadRequest): container "api" in pod "api-77b4b6956f-pkcqj" is waiting to start: PodInitializing ``` ## How much is affected, and since when? Check whether the stuck pod is a new revision or the only copy. In the lab it was a rollout: the new ReplicaSet had one pod that never became Ready while the old one kept both (r1, step 6, trimmed to the api rows): ```shell kubectl get rs -n shop -o wide ``` ``` NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR api-77b4b6956f 1 1 0 72s api python:3.12-alpine app=api,pod-template-hash=77b4b6956f api-f64486c4c 2 2 2 81s api python:3.12-alpine app=api,pod-template-hash=f64486c4c ``` The Deployment shows the same shape and names the change (r1, step 9, trimmed): ```shell kubectl describe deployment api -n shop ``` ``` Annotations: deployment.kubernetes.io/revision: 2 Replicas: 2 desired | 1 updated | 3 total | 2 available | 1 unavailable StrategyType: RollingUpdate RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Init Containers: wait-for-db: Image: busybox:1.37 Command: sh -c until nc -z -w 2 postgres 5432; do echo waiting for postgres:5432; sleep 2; done Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True ReplicaSetUpdated OldReplicaSets: api-f64486c4c (2/2 replicas created) NewReplicaSet: api-77b4b6956f (1/1 replicas created) ``` `Available: True` with the old ReplicaSet at 2/2 means users are served and you have time: the stuck pod is the one surge pod the strategy allows, and no old pod is removed until it is Ready. The ReplicaSet ages date the change; here the new one is 9 seconds younger than the old because the lab injected the fault right after the deploy. `progressDeadlineSeconds` was `600` (r2, step 10); after ten minutes the Progressing condition flips to `ProgressDeadlineExceeded` and nothing else changes. If the Deployment has no old ReplicaSet, or the pod is a bare pod, a Job or a StatefulSet's first replica, every copy is down and the fix is urgent. ## Which dependency is the loop waiting for, and does it exist? Work from the command outward, cheapest check first. 1. **Read the loop.** The command is in `describe pod`, or straight from the template (r1, step 14): ```shell kubectl get deployment api -n shop -o jsonpath={.spec.template.spec.initContainers} ``` ``` [{"command":["sh","-c","until nc -z -w 2 postgres 5432; do echo waiting for postgres:5432; sleep 2; done"],"image":"busybox:1.37","imagePullPolicy":"IfNotPresent","name":"wait-for-db","resources":{"limits":{"memory":"16Mi"},"requests":{"cpu":"10m","memory":"8Mi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}] ``` Three facts from one line: the host is `postgres`, the port is `5432`, and the `until` loop has no bound. `-w 2` only times out a single connection attempt; the loop itself runs until the pod is deleted. 2. **Does a Service by that name exist in the pod's namespace?** The [Kubernetes DNS documentation](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) says "DNS queries that don't specify a namespace are limited to the Pod's namespace." (r1, step 10): ```shell kubectl get svc -n shop ``` ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE api ClusterIP 10.96.63.231 8080/TCP 109s frontend ClusterIP 10.96.40.77 80/TCP 109s redis ClusterIP 10.96.152.67 6379/TCP 109s ``` No `postgres`. If it had existed with no endpoints, the loop would still spin, with a different error; that case is the [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes) runbook. 3. **Does it exist anywhere?** A Service in another namespace is reachable as `..svc`, so a loop written for one namespace and deployed to another fails exactly like this (r1, step 13): ```shell kubectl get svc -A ``` ``` NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.96.0.1 443/TCP 4h18m kube-system kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 4h18m kube-system metrics-server ClusterIP 10.96.215.19 443/TCP 4h15m shop api ClusterIP 10.96.63.231 8080/TCP 2m shop frontend ClusterIP 10.96.40.77 80/TCP 2m shop redis ClusterIP 10.96.152.67 6379/TCP 2m ``` Nowhere in the cluster. 4. **Does the application need the dependency at all?** This decides between fixing the check and provisioning what it checks for. The api's code is in a ConfigMap (r1, step 19, the two lines that matter): ```shell kubectl get configmap api-code -n shop -o yaml ``` ``` REDIS_HOST = os.environ["REDIS_HOST"] with socket.create_connection((REDIS_HOST, 6379), timeout=2) as s: ``` The api talks to Redis on 6379 and to nothing else. A Postgres check on a service that never opens a Postgres connection is a mistake in the manifest, not a missing database. 5. **What changed?** Compare the templates of the two ReplicaSets. The old one has no init containers at all (r1, step 16): ```shell kubectl get rs api-f64486c4c -n shop -o jsonpath={.spec.template.spec.initContainers} ``` ``` (no output) ``` Both revisions carry the same single env var, `[{"name":"REDIS_HOST","value":"redis"}]` (r1, steps 17 and 18), so the init container is the whole diff. The Deployment's `kubectl.kubernetes.io/last-applied-configuration` annotation still described a template with no `initContainers` while the live template had one (r2, step 10): the change came from a patch or an edit, not `kubectl apply`, which is where to look for who did it. `kubectl rollout history deployment/api -n shop` lists the revisions; Aurora's default policy blocked it as a mutating operation (r1 step 15, r2 step 11), but you can run it. ## What is the fix? Three fixes, chosen by whether the dependency is real. **Roll back.** The right call when the init container is the only change in the revision, as in the lab, and the previous ReplicaSet is still healthy. Both Aurora runs proposed this. ```shell kubectl rollout undo deployment/api -n shop ``` Risk: every other change in revision 2 is undone with it, and a new rollout starts; the old pods keep serving throughout. Rollback of the rollback: `kubectl rollout undo deployment/api -n shop --to-revision=2`. **Remove the init container and keep the rest of the revision.** The lab injected the fault with a JSON patch, so its inverse is the exact change: ```shell kubectl -n shop patch deploy api --type=json \ -p='[{"op":"remove","path":"/spec/template/spec/initContainers"}]' ``` Risk: this removes every init container in the template; in the lab there was one. A new rollout starts and the stuck pod is replaced. Rollback: `kubectl rollout undo deployment/api -n shop`. If the init container is also in your source manifests, remove it there, or the next apply brings it back. **The dependency is real.** Then the check is right and its target is wrong: fix the hostname (`postgres..svc` if the Service lives elsewhere) or create the Service in the same change set as the pod that waits for it, and give the loop a bound so the next miss fails loudly. Take this path only when the application opens a connection to that dependency. In the lab it did not; a Postgres Service would have satisfied the check while adding a database nobody reads. Mitigation is not needed here because the old pods kept serving. If the stuck pod is the only copy, the rollback is also the mitigation. Do not delete the stuck pod: the ReplicaSet recreates it from the same template and the new pod loops again. ## How do you verify it worked? ```shell kubectl rollout status deployment/api -n shop kubectl get pods -n shop -l app=api -o wide kubectl -n shop get pods -l app=api -o jsonpath='{range .items[*]}{.metadata.name} {.status.initContainerStatuses[0].state.running.startedAt}{"\n"}{end}' ``` Pass condition: `rollout status` returns "successfully rolled out", two api pods `1/1 Running` with no `Init:` in STATUS, and the third command prints each pod name followed by nothing. That command is the first half of the lab's own check for the broken state, `scenarios/18-init-waits-missing-service/check.sh`, which looks for a pod with a running init container and greps its `wait-for-db` logs for `waiting for postgres`. Watch for two or three minutes: a pod that passes init and then sits at `0/1 Running` has a readiness problem, which is a different page. ## How do you prevent it? **Bound every wait loop.** An unbounded `until` makes a missing dependency look like a slow one, forever. A loop that gives up after a deadline exits non-zero, the kubelet restarts it, and the pod shows `Init:Error` and then `Init:CrashLoopBackOff` with Warning events your alerting already watches. A suggested shape, not run in the lab: ```shell timeout 120 sh -c 'until nc -z -w 2 postgres 5432; do echo waiting for postgres:5432; sleep 2; done' ``` **Alert on init containers that run too long.** kube-state-metrics exposes [`kube_pod_init_container_status_running`](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md) per container. A suggested expression, not captured in the lab: `sum by (namespace, pod, container) (kube_pod_init_container_status_running) > 0` held for ten minutes. Alerts on restarts or readiness probe failures never fire here, because the init container never restarts and the app container's probe never runs; the lab's alert, `KubeDeploymentRolloutStuck`, is the other signal that works. **Check init dependencies in CI.** Every hostname an init container waits for should match a Service in the same change set and namespace. The lab's fault is a one-line patch that adds a check for a dependency nothing in the code opens; a manifest reference check catches it before the rollout does. ## Common mistakes - **Reading the app container's logs.** `kubectl logs ` without `-c` returns `is waiting to start: PodInitializing`. The evidence is in the init container's logs. - **Treating Pending as a scheduling problem.** `PodScheduled: True`, a node and an IP are all present. [FailedScheduling](/runbooks/fix-pod-pending-failedscheduling-kubernetes) is a different page. - **Creating the dependency to make the check pass.** If the application never connects to it, you have added a database to satisfy a typo. Read the code or the env vars first. - **Trusting the absence of Warning events.** A running loop raises none. `Restart Count: 0` here means stuck, not healthy. - **Deleting the pod.** The ReplicaSet recreates it with the same init container. ## What we saw in the lab Scenario `18-init-waits-missing-service`: `break.sh` patches the api Deployment to add the `wait-for-db` init container shown above, with no `postgres` Service anywhere. Aurora received only the alert text, `KubeDeploymentRolloutStuck [warning] namespace=shop deployment=api: Rollout of deployment shop/api is not progressing for longer than 15 minutes.`, and read-only cluster access. The alert was fed to Aurora directly rather than fired by a rule, so both investigations began about a minute after the break (the stuck pod was 69 and 70 seconds old at the first `get pods`). Two runs, each in a fresh account, on two kind clusters (`aurora-lab-2` for r1, `aurora-lab` for r2). | Run | Verdict | Time | Tool calls | Tool errors | Cost | Fix in suggestions | |---|---|---|---|---|---|---| | r1 | correct (judge, then human) | 158 s | 20 | 2 | $0.40 | "Rollback `deployment/api` to revision 1" (acceptable) | | r2 | correct (judge, then human) | 160 s | 19 | 3 | $0.48 | "Roll back `deployment/api` to revision 1" (acceptable) | r1 made sixteen kubectl calls, which took 83 of its 158 seconds; the two that settled it were the init container's logs and the old ReplicaSet's empty `initContainers`. Its report: "Since the init container never completes, the main `api` container is permanently blocked from starting, which prevents the new ReplicaSet from reaching Ready and stalls the RollingUpdate." It also read the code and concluded that "the application source code (pulled from `configmap/api-code`) only ever connects to `REDIS_HOST` on port 6379, with no Postgres client logic whatsoever". r2 took a similar path with one difference that matters: it never read the init container's logs. It read the loop's command from `describe pod` and the Deployment YAML, confirmed no Service existed, and inferred the rest. It did read the `last-applied-configuration` annotation and wrote that the init container was "present in the running spec but absent from the `kubectl.kubernetes.io/last-applied-configuration` annotation, indicating an out-of-band edit rather than a normal declarative update", which a second automated review confirmed against the YAML. Its impact line: the rollout "will remain stuck indefinitely until the bad init container is removed, since no Postgres endpoint will ever become available". Fourteen kubectl calls took 53 of its 160 seconds; three failed: the denied `rollout history` and two `grep` pipes (Aurora's kubectl tool passes arguments to kubectl, not a shell, so `| grep -i postgres` returned `error: unknown shorthand flag: 'i' in -i` in both runs). Both verdicts were correct, and both reports have errors worth knowing about. r1's Ruled Out section says the image `python:3.12-alpine` "was already present on the node and pulled successfully"; the only `Pulled` event on the stuck pod is for `busybox:1.37`, because the api container was never created. r2 wrote that the alert fired "once the progress deadline was exceeded"; the Deployment was 71 seconds old with `progressDeadlineSeconds: 600` and `Progressing: True`. r2 also stated that no Postgres Pod exists anywhere in the cluster after its `get pods -A | grep` had failed and was not retried. And neither report states the fix in its body: r1's judge note reads "The RCA gives no explicit remediation" and r2's "The fix is only implied". Both put the rollback in their Suggested Next Steps, where it was graded acceptable. The grader read Aurora's full output (report, notes, suggestions); a person at Arvo then checked both verdicts. Transcripts and grades: [runs/v1-run1/18-init-waits-missing-service](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/18-init-waits-missing-service). ## How Aurora handles this Aurora receives the alert, lists the Deployment and its pods, describes the pod stuck in `Init:0/1`, reads the init container's command, checks whether the Service it waits for exists in the namespace and the cluster, reads the application's code from its ConfigMap to decide whether the dependency is real, and compares the old ReplicaSet's template with the new one to name the change. Every claim in its report cites the tool output it came from; what it could not check, here the rollout history, is listed as such. It runs read-only kubectl under a command policy by default and proposes the fix rather than applying it. See a full investigation on the [Aurora root cause analysis](/rca) page. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) Related: [a hostname that does not resolve](/runbooks/fix-dns-name-not-resolving-kubernetes), [a Service with no endpoints](/runbooks/fix-service-no-endpoints-kubernetes), [CreateContainerConfigError](/runbooks/fix-createcontainerconfigerror-kubernetes), and [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), and [a pod whose dnsConfig search domains are wrong](/runbooks/fix-pod-dnsconfig-search-domain-not-resolving-kubernetes). ## Sources and verification - Every command and output block above comes from the lab run `v1-run1`, scenario `18-init-waits-missing-service`, on 2026-10-02: Aurora's tool-call transcripts for r1 and r2, each block attributed to its run and step, plus `evidence-before/` captured before the investigation (the app container log error). - Versions, from `evidence-before/versions.yaml`: Kubernetes server v1.37.0 on kind v0.33.0, kubectl client v1.35.3. Pod and node names differ between the two runs because they ran on two clusters. - Aurora commit `87d22baf`, single agent, RCA model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge model `us.anthropic.claude-opus-5-5`, reviewed by a person at Arvo (`human_verdict` in each run's `grade.json`). Times, tool calls, tool errors and costs from `result.json`; the fix column from `fix_from_suggestions` in `grade.json`. - Kubernetes documentation quoted, read 2026-10-19: [Debug Init Containers](https://kubernetes.io/docs/tasks/debug/debug-application/debug-init-containers/), [Init Containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/), [DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/). The metric name is from the [kube-state-metrics pod metrics list](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md). - The statuses in the At a glance table other than `Init:0/1` (`Init:Error`, `Init:CrashLoopBackOff`, `PodInitializing`) are defined on the debugging page above and were not captured in this run. The `--previous` and `timeout` commands and the alert expression are suggestions, not captured output. - The fix commands were not run inside a graded lab run; Aurora's policy blocks mutations. The patch is the inverse of the injected fault (`scenarios/18-init-waits-missing-service/break.sh`), and the Verify section's jsonpath is the first half of the lab's `check.sh`. --- ## Pod Pending with FailedScheduling: Insufficient cpu, the request fits no node (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-pod-pending-failedscheduling-kubernetes Published: 2026-10-16 Author: Damian Loch, Software Engineer at Arvo AI > A pod Pending with FailedScheduling Insufficient cpu asks for more CPU than any node has. Read the event, compare request and nodes, fix the request. | Applies to | Any pod whose CPU or memory request is larger than the allocatable resources of every node; shown here on a Deployment rollout | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0 (three nodes, 10 allocatable CPU each), 2026-10-02 | | Lab folder | [scenarios/08-unschedulable-cpu](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/08-unschedulable-cpu), runs in [runs/v0-run2/08-unschedulable-cpu](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/08-unschedulable-cpu) | **A pod that stays Pending with a FailedScheduling event saying Insufficient cpu is asking for more CPU than any node has left to give, and the scheduler will keep refusing it until the request or the nodes change.** The most common cause is a request typed in whole cores where millicores were meant. Confirm it with `kubectl describe pod `: the Events section carries the exact count of nodes that failed on CPU. The fix is to set the request back to a value that fits, not to add nodes. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | `0/N nodes are available: ... Insufficient cpu` on every worker node, request far above a node's allocatable CPU | The request is wrong (cores instead of millicores, or a copied value) | `kubectl describe pod ` against `kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu` | Set the request back to what the container needs | | `Insufficient cpu` only on some nodes, request is reasonable, nodes show high Allocated resources | The cluster is full | `kubectl describe nodes`, the Allocated resources block per node | Scale down something, or add a node | | `node(s) had untolerated taint(s)` is the only reason given | A taint or missing toleration | `kubectl describe nodes` Taints line against the pod's Tolerations | Add the toleration or remove the taint | | `node(s) didn't match Pod's node affinity/selector` | Node selector or affinity excludes every node | Pod spec `nodeSelector` and `affinity` against node labels | Fix the selector or label a node | | `node(s) didn't have free ports for the requested pod ports` | A `hostPort` is already taken on every node | Pod spec `hostPort` against other pods on the nodes | Drop or change the hostPort | ## How do you confirm the pod is unschedulable because of CPU? Start with the pod list. A Pending pod with no node and no IP next to a Running pod from the same Deployment is the shape of a stuck rollout (lab run r1, step 4): ```shell kubectl get pods -n shop -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-f64486c4c-lnrzg 1/1 Running 0 5m5s 10.244.2.109 aurora-lab-worker2 api-f64486c4c-wb44h 1/1 Running 0 5m5s 10.244.1.109 aurora-lab-worker frontend-76d98597b9-6dnxv 1/1 Running 0 5m5s 10.244.2.110 aurora-lab-worker2 frontend-76d98597b9-krf8c 1/1 Running 0 5m5s 10.244.1.110 aurora-lab-worker loadgen-7c6676864-278sd 1/1 Running 0 5m5s 10.244.1.112 aurora-lab-worker redis-59458f55d8-ndfwm 1/1 Running 0 5m5s 10.244.1.108 aurora-lab-worker worker-6cf88d4b58-wcqmk 1/1 Running 0 5m5s 10.244.1.111 aurora-lab-worker worker-7fdf4ffb67-h89dz 0/1 Pending 0 4m56s ``` Then read the pending pod. The scheduler writes one event that names every reason a node was rejected (r1, step 6, trimmed to the request and the event): ```shell kubectl describe pod worker-7fdf4ffb67-h89dz -n shop ``` ``` Status: Pending IP: IPs: Controlled By: ReplicaSet/worker-7fdf4ffb67 Containers: worker: Image: python:3.12-alpine Limits: memory: 192Mi Requests: cpu: 64 memory: 64Mi Conditions: Type Status PodScheduled False Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 5m8s default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. ``` Read the message as a tally: three nodes, one excluded by a taint, two by CPU. Only the two matter here, because the tainted node is the control plane and never runs these pods. "Preemption is not helpful" says evicting lower-priority pods would not free enough either: nothing on the node adds up to the request. `cpu: 64` with no unit is 64 whole cores. The healthy pod from the previous revision asks for `50m` (r1, step 13): ```shell kubectl get pod worker-6cf88d4b58-wcqmk -n shop -o jsonpath={.spec.containers[0].resources} ``` ``` {"limits":{"memory":"192Mi"},"requests":{"cpu":"50m","memory":"64Mi"}} ``` That is a factor of 1,280. The [Kubernetes CPU resource docs](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/) state the rule the scheduler applied: "Pod scheduling is based on requests. A Pod is scheduled to run on a Node only if the Node has enough CPU resources available to satisfy the Pod CPU request." ## How much is affected, and since when? Check whether the old pods are still serving. In a rolling update the new pod has to schedule before an old one is removed, so an unschedulable new pod stalls the rollout without taking the service down (r1, step 11): ```shell kubectl get rs -n shop -o wide ``` ``` NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR worker-6cf88d4b58 1 1 1 5m43s worker python:3.12-alpine app=worker,pod-template-hash=6cf88d4b58 worker-7fdf4ffb67 1 1 0 5m34s worker python:3.12-alpine app=worker,pod-template-hash=7fdf4ffb67 ``` The Deployment status says the same from the other side (r1, step 9, status block): ```yaml status: availableReplicas: 1 conditions: - lastTransitionTime: "2026-10-02T16:36:48Z" message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available - lastTransitionTime: "2026-10-02T16:36:47Z" lastUpdateTime: "2026-10-02T16:36:56Z" message: ReplicaSet "worker-7fdf4ffb67" is progressing. reason: ReplicaSetUpdated status: "True" type: Progressing observedGeneration: 2 readyReplicas: 1 replicas: 2 terminatingReplicas: 0 unavailableReplicas: 1 updatedReplicas: 1 ``` `Available: True` with `unavailableReplicas: 1` means one old pod is doing all the work and the new revision has not landed. With `progressDeadlineSeconds: 600` (same output), the Progressing condition flips to `ProgressDeadlineExceeded` after ten minutes; it does not fix anything, it only changes the message. The revision annotation reads `"2"`, so the change is one rollout old. ## Which of the five causes is it? The event names the cause, but the numbers decide it. Compare the request with what each node can allocate (r1, step 8, `kubectl describe nodes`, trimmed to the capacity lines): ``` Name: aurora-lab-control-plane Taints: node-role.kubernetes.io/control-plane:NoSchedule Capacity: cpu: 10 Allocatable: cpu: 10 Allocated resources: Resource Requests Limits cpu 950m (9%) 0 (0%) Name: aurora-lab-worker Taints: Capacity: cpu: 10 Allocatable: cpu: 10 ``` 1. **Request larger than any node.** Allocatable CPU is 10 on every node; the pod wants 64. No amount of freeing capacity helps, and the message lists every schedulable node as `Insufficient cpu`. This is the lab's case. 2. **Cluster genuinely full.** Allocatable minus Allocated resources is smaller than the request on each node, but the request itself is sane. Here the control plane shows 950m of 10 cores used; the workers are nowhere near full. 3. **Taint.** If `untolerated taint(s)` were the only reason, every node would carry a taint the pod does not tolerate. In the lab only the control plane is tainted, and `node-role.kubernetes.io/control-plane:NoSchedule` is the default for kind; it is not the cause and both Aurora runs said so. 4. **Node selector or affinity.** The pod spec above shows `Node-Selectors: `, and the message does not mention affinity. 5. **hostPort conflict.** `Host Port: ` in the pod spec, and the message does not mention ports. When the request looks plausible and you still see `Insufficient cpu`, read the Allocated resources block of every node: the scheduler sums requests, not usage, so a node with 5% CPU in use can still be full by requests. ## What is the fix? Set the request back to a value that fits. The lab injected the fault with a JSON patch, so the inverse is the exact fix: ```shell kubectl -n shop patch deploy worker --type=json \ -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"50m"}]' ``` Or roll back to the previous revision, which carries the same `50m`: ```shell kubectl rollout undo deployment/worker -n shop ``` Risk: both commands start a new rollout. The pending pod is replaced by one that schedules; the old pod is removed only after the new one is Ready, so the worker keeps running throughout. Rollback: if the new request is still wrong, `kubectl rollout undo` again returns to the last working revision. Do not delete the Pending pod by hand: its ReplicaSet recreates it with the same request within seconds. Mitigation is not needed here because the old pod kept serving. If the failing rollout had been a scale-up with no old pods, the same patch is also the mitigation. ## How do you verify it worked? The new ReplicaSet must reach Ready and the old one scale to zero: ```shell kubectl rollout status deployment/worker -n shop kubectl get rs -n shop -l app=worker -o wide kubectl get pods -n shop -l app=worker -o wide ``` Pass condition: `rollout status` returns "successfully rolled out", exactly one worker pod, Running, with a node assigned, and no new `FailedScheduling` event for it. The lab's own check for the broken state is one line; its negation is the pass: ```shell kubectl -n shop get events --field-selector reason=FailedScheduling -o jsonpath='{.items[*].message}' | grep -q 'Insufficient cpu' ``` Watch for ten minutes: a request that fits but leaves the node tight can still fail later when another rollout lands on the same node. ## How do you prevent it? **Cap requests at the namespace.** A LimitRange rejects the pod at admission with a clear error instead of leaving it Pending (a suggested guard, not captured in the lab): ```yaml apiVersion: v1 kind: LimitRange metadata: name: container-max namespace: shop spec: limits: - type: Container max: cpu: "4" ``` **Alert on Pending, not on restarts.** This failure never restarts anything. With kube-state-metrics, a suggested expression: `sum by (namespace, pod) (kube_pod_status_phase{phase="Pending"}) > 0` held for five minutes, which is what the lab's alert text describes. **Check the arithmetic in CI.** A request above the largest node's allocatable CPU can never schedule; compare `resources.requests.cpu` in manifests against `kubectl get nodes -o jsonpath='{.items[*].status.allocatable.cpu}'` before applying. Always write CPU with a unit when you mean millicores: `50m`, never `50`. ## Common mistakes - **Adding nodes.** A 64-core request does not fit a 10-core node no matter how many you add. Read the request before touching capacity. - **Blaming the taint.** `1 node(s) had untolerated taint(s)` appears in almost every FailedScheduling message on a cluster with a control plane; it is noise unless it accounts for all nodes. - **Deleting the Pending pod.** The ReplicaSet recreates it with the same spec. - **Raising limits.** Scheduling reads requests. Changing `limits.cpu` does nothing for a Pending pod. - **Waiting for the deadline.** `ProgressDeadlineExceeded` after 600 seconds is a label, not a rollback. ## What we saw in the lab Both runs found the cause and the trigger (lab run v0-run2, 2026-10-02, judged by `us.anthropic.claude-opus-5-5` and reviewed by a person): | Run | Verdict | Time | Tool calls | Cost | Fix proposed | |---|---|---|---|---|---| | r1 | correct | 362 s | 14 | $0.35 | Correct the oversized `cpu` request on the worker Deployment (acceptable) | | r2 | correct | 89 s | 10 | $0.21 | Patch the request back to `50m`, add a LimitRange as a guard (acceptable) | r1 ran `get pods`, `get events`, `describe pod`, `get nodes`, `describe nodes`, the Deployment YAML, then `rollout history` (refused: "Command blocked by organization policy: Mutating kubectl operations", step 10), `get rs`, a postmortem lookup, and finally the old pod's resources to confirm `50m`. Its report put it this way: "Every node in the cluster has only 10 allocatable CPUs [7, 8], so no node could ever satisfy this request". The 362 seconds were mostly model time; the cluster calls together took about 40 seconds. r2 went `get pods`, `get nodes`, `describe pod`, `get events`, `describe nodes`, the Deployment YAML, `get replicaset`, and stopped. It ruled the taint out explicitly: "it accounts for only one of three nodes being excluded, not the "Insufficient cpu" failures on the other two". Two things a reader should know. r1's report only implied the fix ("manual intervention" to correct the request or roll back) and r2's stated none; the fixes above come from each run's Suggested Next Steps list, which is where Aurora puts them. And both reports inferred that the change was made outside `kubectl apply`, because the `last-applied-configuration` annotation still shows the old request; that is consistent with how the fault was injected (a `kubectl patch`), but the reports could not prove it, since `rollout history` was denied by the default policy. ## How Aurora handles this Aurora receives the alert, lists the pods, reads the pending pod's scheduling event, compares the request with every node's allocatable CPU, and reports the cause with each claim cited to a command output, as in the two runs above. It runs read-only under its default command policy and proposes the fix rather than applying it. It flags what it could not verify, here the rollout history, instead of guessing. Every investigation is a page like the ones linked in the table. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) Related: [how to fix OOMKilled](/runbooks/fix-oomkilled-kubernetes), [how to fix CreateContainerConfigError](/runbooks/fix-createcontainerconfigerror-kubernetes), [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), and [how Aurora investigates an incident](/rca). ## Sources and verification - Every command and output block above comes from the lab run `v0-run2`, scenario `08-unschedulable-cpu`, on 2026-10-02: Aurora's tool-call transcripts (`steps-full.json`) for r1 and r2, each block attributed to its run and step, plus `evidence-before/` captured before the investigation. - Versions, from `evidence-before/versions.yaml`: Kubernetes server v1.37.0 on kind v0.33.0 (three nodes, 10 allocatable CPU each, containerd 2.3.4), kubectl client v1.35.3. - Aurora commit `87d22baf`, single agent, RCA model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge model `us.anthropic.claude-opus-5-5`, reviewed by a person at Arvo (`human_verdict` in each run's `grade.json`). Times and costs from `result.json`. - Kubernetes documentation quoted: [Assign CPU Resources to Containers and Pods](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/). The messages in the At a glance table other than the lab's own are the scheduler's standard reasons and were not captured in this run. - The fix commands were not run inside a graded lab run; Aurora's policy blocks mutations. The patch is the inverse of the injected fault (`scenarios/08-unschedulable-cpu/break.sh`), and the Verify section's one-liner is the lab's `check.sh`, negated. --- ## PVC Pending: the StorageClass does not exist and the dependent app fails (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-pvc-pending-storageclass-not-found-kubernetes Published: 2026-10-16 Author: Noah Casarotto-Dinning, CEO at Arvo AI > A PVC stays Pending when its storageClassName names a class the cluster lacks. The pod never schedules and whatever depends on it fails. Lab-tested. | Applies to | Pods that mount a PersistentVolumeClaim, and the services that depend on them | |---|---| | Verified on | Kubernetes server v1.37.0 (kind v0.33.0, provisioner rancher.io/local-path), 2 October 2026 | | Lab folder | [scenarios/09-redis-pvc-pending](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/09-redis-pvc-pending), runs in [runs/v0-run2/09-redis-pvc-pending](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/09-redis-pvc-pending) | **A PersistentVolumeClaim stays Pending when its `storageClassName` names a StorageClass that does not exist in the cluster, so no volume is ever provisioned and the pod that mounts it is never scheduled.** The alert usually fires somewhere else: the service that depends on that pod starts failing, and that is what pages you. Confirm with `kubectl describe pvc -n `: a `ProvisioningFailed` event reading `storageclass.storage.k8s.io "" not found` is the whole diagnosis. The fix is to point the claim at a class that exists, or to undo the change that introduced it. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | PVC `Pending`, event `storageclass.storage.k8s.io "" not found` | The claim names a StorageClass the cluster does not have | `kubectl describe pvc ` against `kubectl get storageclass` | Recreate the claim with an existing class, or revert the change | | PVC `Pending`, no events, class exists | `volumeBindingMode: WaitForFirstConsumer` and no pod has claimed it yet | `kubectl get storageclass -o yaml` | Nothing: binding happens when the pod is scheduled | | PVC `Pending`, event from the provisioner about quota, zone or size | The class exists but the provisioner cannot satisfy the request | `kubectl describe pvc `, the provisioner's logs | Fix the request or the provisioner | | Pod `Pending` with `pod has unbound immediate PersistentVolumeClaims` | Any of the above, seen from the pod | `kubectl describe pod ` then the PVC | Fix the claim; the pod schedules on its own | ## How do you confirm the claim is the problem? Start from the alert and work back to the volume. In the lab the alert was a crash-looping api pod and 5xx on the frontend. The pod list already shows two different problems in the same namespace (lab run r1, step 4): ``` NAME READY STATUS RESTARTS AGE IP NODE api-5cf6469845-4khjs 0/1 CrashLoopBackOff 5 (119s ago) 5m59s 10.244.1.17 aurora-lab-3-worker api-f64486c4c-67cfn 1/1 Running 0 6m9s 10.244.1.14 aurora-lab-3-worker api-f64486c4c-dn9wx 1/1 Running 0 6m9s 10.244.2.16 aurora-lab-3-worker2 redis-5fff65475c-t2qf5 0/1 Pending 0 5m58s ``` A pod that is `Pending` with no node and no IP for six minutes has not been scheduled. Read the claim it mounts: ```shell kubectl describe pvc redis-data -n shop ``` Run r2, step 13: ``` Name: redis-data Namespace: shop StorageClass: fast-ssd Status: Pending Volume: Used By: redis-5fff65475c-v7qqm Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 6s (x9 over 2m) persistentvolume-controller storageclass.storage.k8s.io "fast-ssd" not found ``` `Volume` is empty and the persistentvolume-controller says why, nine times in two minutes. Compare with what the cluster actually has (r1, step 12): ``` NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE standard (default) rancher.io/local-path Delete WaitForFirstConsumer false 25m ``` One class, named `standard`. The claim asks for `fast-ssd`. That is the fault. ## How far does it reach? The claim blocks one pod, but the pod is a dependency, so count what depends on it. The redis Deployment is at zero of one, and the api Deployment is mid-rollout (captured before the investigation, r1 `evidence-before/get-all.txt`): ``` NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/api 2/2 1 2 71s deployment.apps/redis 0/1 1 0 71s NAME ADDRESSTYPE PORTS ENDPOINTS endpointslice.discovery.k8s.io/redis-69z5b IPv4 ``` The redis Service has no endpoints. Every api pod talks to redis, so every api pod is affected, in two different ways. The new api pod cannot start (r1, step 5, `kubectl logs api-5cf6469845-4khjs -n shop --tail=100 --previous`): ``` redis connection failed (attempt 1/5): [Errno 111] Connection refused redis connection failed (attempt 2/5): [Errno 111] Connection refused redis connection failed (attempt 3/5): [Errno 111] Connection refused redis connection failed (attempt 4/5): [Errno 111] Connection refused redis connection failed (attempt 5/5): [Errno 111] Connection refused giving up: redis unavailable ``` The two old api pods stay `Running` and `Ready`, because their health check does not touch redis, and fail every real request (r1, step 10, `kubectl logs api-f64486c4c-67cfn -n shop --tail=50`, trimmed): ``` 10.244.1.1 "GET /healthz HTTP/1.1" 200 - request failed: redis error: [Errno 111] Connection refused 10.244.1.15 "GET /items HTTP/1.0" 503 - 10.244.1.1 "GET /healthz HTTP/1.1" 200 - ``` So the blast radius is the whole `/api/` path, while the pod list shows two healthy api pods. Since when: the redis Deployment is at revision 2 and the claim was created at the same minute the api got its `restartedAt` annotation (r1, step 7 for the api annotation, steps 13 and 14 for the redis revision and the claim's `creationTimestamp: "2026-10-02T16:37:19Z"`). The change and the outage share a timestamp. ## Which of the four causes is it? Cheapest check first. Each one takes a single command and the output decides it. 1. **Does the class exist?** `kubectl get storageclass`. If the name on the claim is not in the list, you are done: this is the cause. The event text is explicit, `storageclass.storage.k8s.io "fast-ssd" not found`. 2. **Is the class waiting for a consumer?** If the class exists with `VOLUMEBINDINGMODE WaitForFirstConsumer`, a claim with no pod stays `Pending` by design and shows no `ProvisioningFailed` event. Schedule the pod; the claim binds then. 3. **Is the provisioner refusing?** If the class exists and the event comes from the provisioner rather than the persistentvolume-controller, read it: out of quota, wrong zone, size not supported. The claim is fine; the backend is not. 4. **Is the pod blocked by something else?** `kubectl describe pod ` on the Pending pod. In the lab the scheduler said exactly one thing (r1, step 8): ``` Warning FailedScheduling 75s (x2 over 6m25s) default-scheduler 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. not found ``` No node is excluded for CPU, memory, taints or affinity. The only reason is the unbound claim. If the message lists nodes with `Insufficient cpu` or a taint instead, the volume is not your problem. Read the claim's own manifest to see whether the class name is a typo or a class that exists elsewhere (r1, step 14, trimmed): ```yaml spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: fast-ssd volumeMode: Filesystem status: phase: Pending ``` ## How do you fix it? Two acceptable fixes. Pick by how much you trust the change that introduced the claim. **Recreate the claim with a class that exists.** `storageClassName` is immutable on a PVC, so you cannot patch it. Delete the Pending claim and create it again. The lab's claim came from a manifest; this is that manifest with the class corrected: ```shell kubectl -n shop delete pvc redis-data kubectl -n shop apply -f - <<'YAML' apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-data namespace: shop spec: accessModes: [ReadWriteOnce] storageClassName: standard resources: requests: {storage: 1Gi} YAML ``` Risk: deleting a Pending claim deletes nothing on disk, because no volume was ever bound. Check `Volume` is empty in `describe pvc` before you delete; a bound claim with `reclaimPolicy: Delete` would take its data with it. Rollback: delete the new claim. **Or revert the Deployment to the revision without the volume.** `kubectl rollout undo deployment/redis -n shop` brings back the pod template that did not mount the claim. Risk: whatever that change was meant to add (here, persistence) is gone again. Rollback: `kubectl rollout undo` once more. Do not restart the api pods and do not raise their resources. The crash loop is a symptom; once redis has endpoints the new api pod comes up on its own. ## How do you verify it? The claim binds, the pod schedules, the dependents recover. The lab's own check for the broken state is two conditions; both must turn false: ```shell kubectl -n shop get pvc redis-data -o jsonpath='{.status.phase}' kubectl -n shop get pods -l app=api -o jsonpath='{.items[*].status.containerStatuses[0].restartCount}' ``` Pass when the first prints `Bound` and the restart counts stop climbing. Then watch the service the alert came from for two minutes: `kubectl -n shop logs deploy/loadgen --tail=20` should show `GET /api/items -> 200` instead of the `503` lines the lab recorded every two seconds. Expect the api pod to need one more restart after redis is up: it only connects at startup. ## How do you prevent it? **Alert on claims that stay Pending.** With kube-state-metrics, a suggested expression (not captured in the lab): ``` kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1 ``` Hold it for five minutes to skip claims that are legitimately waiting for a consumer. The lab's alert fired on the api crash loop, two hops from the cause; this one fires on the cause. **Guardrail: validate class names before apply.** A one-line check in CI catches the typo before the cluster does: ```shell kubectl get storageclass "$(yq '.spec.storageClassName' pvc.yaml)" >/dev/null ``` **Do not use `Recreate` on a stateful singleton without a working claim.** The redis Deployment's `strategy: type: Recreate` (r1, step 13) terminated the old pod before the new one could schedule, which is why the outage was total rather than partial. With `RollingUpdate` the old redis pod would have kept serving while the new one sat Pending. ## Common mistakes - **Treating the alerted pod as the cause.** The api crash loop is real but secondary. Restarting api, raising its memory or rolling it back changes nothing while redis has no endpoints. - **Trusting `Running` and `Ready`.** Two api pods were both, and both returned 503 on every request. A health check that does not exercise the dependency hides the dependency's outage. - **Patching `storageClassName` in place.** The field is immutable after creation; the apply fails. Delete and recreate. - **Deleting a claim that is bound.** The Pending claim has no volume, so deleting it is free. A bound claim with a `Delete` reclaim policy is not. Read `Volume:` first. - **Expecting the class to be created for you.** The persistentvolume-controller retries forever (`x9 over 2m`) and never gives up; the claim will not fail loudly, it will sit there. ## What we saw in the lab Two runs on 2 October 2026, fresh user and org each time, Aurora commit 87d22baf on bedrock/us.anthropic.claude-sonnet-5, read-only kubectl under the default `observability_only` policy. Both correct. | Run | Verdict | Time | Tool calls | Cost | |---|---|---|---|---| | r1 | correct (human reviewed) | 406 s | 14 | $0.30 | | r2 | correct (human reviewed) | 155 s | 19 | $0.47 | r1 went from the alerted pod straight to the dependency: `get pods`, the crashing pod's previous log, `describe pod` on it, the api Deployment, then `describe pod` on the Pending redis pod, events, a healthy api pod's log, `get pvc`, `get storageclass`, the redis Deployment YAML and finally the claim's YAML. Its report named the chain end to end: "The root cause was a PersistentVolumeClaim (`redis-data`) created with `storageClassName: fast-ssd` [14], a storage class that does not exist in this cluster", and it ruled out OOM and an api code defect with the exit code and the clean log. The judge's note: the report never recommends a remediation. The fix was in Aurora's separate Suggested Next Steps list, where it proposed recreating the claim with `standard`; graded on everything Aurora shows the user, the fix is acceptable. r1 took 406 seconds, the slowest run in v0-run2. The lab report's Limits section attributes the delay to Aurora's own prediscovery jobs competing for its four worker slots during that batch, not to the investigation itself. r2 was faster and wider: it read the api source from the ConfigMap to prove that the api pings redis on every request, read both Deployments, `get storageclass`, `describe pvc`, a healthy api pod's log and the load generator's log, and tried `rollout history` on both Deployments, which the policy refused (steps 16 and 17: "Command blocked by organization policy: Mutating kubectl operations"). Its report added the mechanism the alert hides: "The older, already-Ready `api` pods continued passing health checks but failed every real request due to per-request Redis calls". It also ruled out a NetworkPolicy by the error type: "connection failures returned `Connection refused` (no listener), not a timeout". r2 gave no fix in Suggested Next Steps; the judge marked the fix as implied only in the report. Graded on everything Aurora showed the user, the two fixes it did offer, switch the claim to `standard` or remove it, were both acceptable. The same scenario is also the lab's best-known miss. In the superseded first batch (v0-run1), Aurora had found the claim and the missing class by step 10, then made 25 calls to its artifact tool, many with no content argument, and spent about 11 of 15 minutes and $2 of $2.61 retrying them. The report that came out called the cause "a dependency the cluster could not satisfy" and never named the claim. The investigation notes did, so on full output it counts as correct, but the report was wrong. The lab report traces it to two Aurora bugs, a missing `max_tokens` on Bedrock calls and artifact writes counting as evidence, both patched on the fix branch; the rerun of scenario 09 on that branch was 2 of 2 correct with a median of 182 seconds against 280 before. None of it recurred in v0-run2. ## How Aurora handles this Aurora receives the alert for the crashing api pod, lists the pods, reads the crashing container's previous log and the Pending pod's events, then follows the claim to the StorageClass list and reads the Deployment and PVC manifests to date the change. Each sentence of the root cause cites the command output it came from, and the two runs above show the same chain found two different ways. The investigation runs under the default read-only command policy; the fix is proposed, not applied. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) Related: [a hostname that does not resolve](/runbooks/fix-dns-name-not-resolving-kubernetes), [a Service with no endpoints](/runbooks/fix-service-no-endpoints-kubernetes), [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), and what an [Aurora root cause analysis](/rca) looks like. ## Sources and verification Every command output on this page is from the lab's captured cluster state or Aurora's tool transcripts for scenario 09, run v0-run2 (2 October 2026), on Kubernetes v1.37.0 under kind v0.33.0 with the rancher.io/local-path provisioner: [scenario folder](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/09-redis-pvc-pending), [run folder](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/09-redis-pvc-pending), [lab report](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md). The fix is the scenario's `break.sh` with the class name corrected; the verify step is its `check.sh`. StorageClass and claim semantics: the Kubernetes [persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) and [storage classes](https://kubernetes.io/docs/concepts/storage/storage-classes/) pages. The alert expression is a suggestion, not a captured query. Verified 16 October 2026. --- ## Connections time out: a NetworkPolicy blocks the app from its dependency (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-networkpolicy-blocking-traffic-kubernetes Published: 2026-10-16 Author: Ben Gervais, Engineer at Arvo AI > Requests time out while every pod is Running: a NetworkPolicy lets only other pods reach the dependency. Read the error, find the policy, fix it. | Applies to | Pods that call another pod through a Service and get a connect timeout, in a cluster whose network plugin enforces NetworkPolicy; here api to redis under kindnet | |---|---| | Verified on | Kubernetes server v1.37.0 (kind v0.33.0, kubectl v1.35.3, containerd 2.3.4), redis 7.4.11, nginx 1.27.5, 2 October 2026 | | Lab folder | [scenarios/13-networkpolicy-blocks-redis](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/13-networkpolicy-blocks-redis), runs in [runs/v1-run1/13-networkpolicy-blocks-redis](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/13-networkpolicy-blocks-redis) | **A connection that times out, while the same dependency is Running, Ready and has endpoints, is being dropped, not refused.** Inside a cluster the usual reason is a NetworkPolicy that selects the dependency's pods and lists the wrong clients in its `from` rule. Confirm with `kubectl get networkpolicies -n ` and `kubectl describe networkpolicy `: if the client pod's labels do not match any `from` selector, every packet it sends is dropped and the client waits out its own timeout. The fix is to add the client's label to the policy, or remove the policy. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | Client logs say `timed out`; dependency is Running and has endpoints | A NetworkPolicy selects the dependency and does not allow this client | `kubectl describe networkpolicy -n ` against `kubectl get pod --show-labels` | Add the client's label to the `from` list, or delete the policy | | Client logs say `timed out`; the client's own namespace has an egress policy | An egress policy on the client blocks the destination | `kubectl get networkpolicies -n -o yaml`, look for `Egress` in `policyTypes` | Allow the destination in the egress rule | | Client logs say `Connection refused` | Nothing is listening: the dependency is starting, crashed, or its Service has no endpoints | `kubectl get endpoints ` | See [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes) | | Client logs say `Name does not resolve` | Wrong hostname or broken cluster DNS | `kubectl get svc -n ` | See [Name does not resolve](/runbooks/fix-dns-name-not-resolving-kubernetes) | | Client logs say `timed out` for every destination | Cluster-wide network fault or a default-deny policy | `kubectl get networkpolicies -A`, node and CNI health | Fix the plugin or the default-deny rule | ## How do you confirm the connection is timing out? Start from the error the client prints. In the lab the api pods answered `/items` with 503 and logged the reason each time (Aurora's run r1, step 8, `kubectl logs -n shop -l app=api --tail=100`, trimmed): ``` redis connection failed (attempt 1/5): [Errno 111] Connection refused connected to redis at redis:6379 api listening on :8080 10.244.1.1 "GET /healthz HTTP/1.1" 200 - request failed: redis error: timed out 10.244.2.45 "GET /items HTTP/1.0" 503 - request failed: redis error: timed out 10.244.2.45 "GET /items HTTP/1.0" 503 - ``` Two different errors sit in that log. The first line, `Connection refused`, is the startup race before redis was listening: the kernel answered at once because nothing was on the port. The later lines say `timed out`: the connection attempt got no answer at all for 2 seconds and the client gave up. Refused means a listener is missing. Timed out means packets are being dropped somewhere between the two pods. The proxy in front shows the same failure as a status code only (frontend nginx, run r1, step 7, two lines): ``` 10.244.1.36 - - [02/Oct/2026:20:23:56 +0000] "GET /api/items HTTP/1.1" 503 32 "-" "Wget" "-" 10.244.1.36 - - [02/Oct/2026:20:24:44 +0000] "GET /api/items HTTP/1.1" 503 32 "-" "Wget" "-" ``` Health checks keep passing, because `/healthz` does not touch the dependency. Nothing restarts, nothing is Pending, and every pod reports 1/1 Running (run r1, step 4): ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-f64486c4c-2nqsk 1/1 Running 0 73s 10.244.1.34 aurora-lab-3-worker api-f64486c4c-mvpnl 1/1 Running 0 73s 10.244.2.43 aurora-lab-3-worker2 frontend-76d98597b9-h4vn7 1/1 Running 0 72s 10.244.2.45 aurora-lab-3-worker2 frontend-76d98597b9-vgf28 1/1 Running 0 72s 10.244.1.35 aurora-lab-3-worker loadgen-7c6676864-6wnwp 1/1 Running 0 72s 10.244.1.36 aurora-lab-3-worker redis-59458f55d8-n6glm 1/1 Running 0 73s 10.244.2.42 aurora-lab-3-worker2 worker-6cf88d4b58-vk9gh 1/1 Running 0 72s 10.244.2.44 aurora-lab-3-worker2 ``` A 100 percent error rate with a green pod list is the shape of this failure. ## How much is affected, and since when? Read the client that drives real traffic. The lab's load generator calls `/api/items` every 4 seconds, and every call failed (run r1, captured cluster state, `logs/loadgen-7c6676864-6wnwp.loadgen.log`, last five lines): ``` 2026-10-02T20:24:28Z GET /api/items -> 503 2026-10-02T20:24:32Z GET /api/items -> 503 2026-10-02T20:24:36Z GET /api/items -> 503 2026-10-02T20:24:40Z GET /api/items -> 503 2026-10-02T20:24:44Z GET /api/items -> 503 ``` Then find what changed. Pods, Services and Deployments do not have to change for traffic to stop: a NetworkPolicy is its own object with its own creation time. Both api replicas were affected, on both worker nodes, which rules out a single bad node. The policy's `Created on` timestamp, 20:23:54 (below), falls 9 seconds after the api pods started at 20:23:45 and 52 seconds before the alert fired at 20:24:46. If the policy is older than the outage, something else changed: a label on the client pods, or a second policy. ## Which of the possible causes is it? Work from the cheapest check to the most specific. **1. Is the dependency itself healthy?** Logs, state and endpoints, in that order (run r1, step 10, `kubectl logs -n shop -l app=redis --tail=100`, last three lines): ``` 1:M 02 Oct 2026 20:23:45.678 * Running mode=standalone, port=6379. 1:M 02 Oct 2026 20:23:45.678 * Server initialized 1:M 02 Oct 2026 20:23:45.678 * Ready to accept connections tcp ``` `describe pod` showed `State: Running`, `Ready: True`, `Restart Count: 0` (run r1, step 11), and the Service has a backend (run r2, step 13): ``` NAME ENDPOINTS AGE redis 10.244.2.55:6379 7m22s ``` A healthy dependency with a populated endpoint list and a client that still times out means the path between them is blocked. If the endpoint list is empty, you have a different problem: [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes). **2. Does the name resolve to the right Service?** The client's config and the Service list have to agree (run r2, step 12, then run r1, step 15): ``` [{"name":"REDIS_HOST","value":"redis"}] ``` ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE api ClusterIP 10.96.37.132 8080/TCP 2m2s frontend ClusterIP 10.96.44.77 80/TCP 2m1s redis ClusterIP 10.96.107.182 6379/TCP 2m2s ``` A resolution failure prints a different error (`Name does not resolve`, `Name or service not known`), so if the log says `timed out`, DNS is already ruled out. **3. Is there a NetworkPolicy selecting the dependency?** This is the check most people run last. Run it third (run r1, step 20): ``` NAME POD-SELECTOR AGE redis app=redis 2m49s ``` Then read who it lets in (run r1, step 22): ``` Name: redis Namespace: shop Created on: 2026-10-02 20:23:54 +0000 UTC Labels: Annotations: Spec: PodSelector: app=redis Allowing ingress traffic: To Port: 6379/TCP From: PodSelector: app=worker Not affecting egress traffic Policy Types: Ingress ``` **4. Do the client's labels match the `from` selector?** They have to match exactly (run r1, step 23): ``` NAME READY STATUS RESTARTS AGE LABELS api-f64486c4c-2nqsk 1/1 Running 0 3m8s app=api,pod-template-hash=f64486c4c api-f64486c4c-mvpnl 1/1 Running 0 3m8s app=api,pod-template-hash=f64486c4c ``` `app=api` is not `app=worker`. The [NetworkPolicy documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/) describes the mechanism: "By default, a pod is non-isolated for ingress; all inbound connections are allowed. A pod is isolated for ingress if there is any NetworkPolicy that both selects the pod and has "Ingress" in its policyTypes". Once the redis pods are isolated, only the listed sources get through, and the api is not listed. Nothing in the api pod's own namespace, labels or events records the drop; the only evidence on the client side is the timeout. If the policy allows the client and the timeout persists, look for a second policy with `Egress` in `policyTypes` that selects the client, then for a `namespaceSelector` that excludes the client's namespace, then at the network plugin itself. The same documentation page notes that "Creating a NetworkPolicy resource without a controller that implements it will have no effect", so a cluster whose plugin does not enforce policies will show the opposite symptom: a policy that looks restrictive but blocks nothing. The lab's `check.sh` relies on kindnet enforcing the policy: without enforcement the api keeps answering 200. ## How do you fix it? Two acceptable fixes. Pick the one that matches the policy's intent. **Add the client to the policy.** Keeps the restriction for everyone else: ```shell kubectl -n shop patch networkpolicy redis --type=json -p '[{"op":"add","path":"/spec/ingress/0/from/-","value":{"podSelector":{"matchLabels":{"app":"api"}}}}]' ``` Risk: low. The patch widens one rule; no pod restarts, no connection to any other workload changes. Rollback: `kubectl -n shop patch networkpolicy redis --type=json -p '[{"op":"remove","path":"/spec/ingress/0/from/1"}]'`, which removes the entry you added. **Remove the policy.** The lab's `break.sh` applies the policy; its inverse undoes the fault: ```shell kubectl -n shop delete networkpolicy redis ``` Risk: the dependency is open to every pod in the cluster again, which is the state before the policy existed. Rollback: re-apply the policy manifest from version control. Do not restart the api or redis. Restarting changes nothing about which pods the policy admits, and it is one of the wrong answers the lab scenario lists. ## How do you verify the fix? The client must stop logging the timeout and the caller must get 200s. The lab's `check.sh` proves the broken state with one line; its two conditions, negated, are the pass condition: ```shell kubectl -n shop logs deploy/loadgen --tail=5 | grep -q -- '-> 503' && kubectl -n shop logs -l app=api --tail=20 | grep -q 'redis error' ``` Pass when neither `grep` matches: the last five load generator lines end in `-> 200` and the api's last 20 lines contain no `redis error`. Watch for two minutes; the api opens a new connection per request, so recovery is immediate once the policy admits it. ## How do you prevent it? **Alert.** The lab alert was the right one: a 5xx ratio on the service in front, not a pod status. Every pod here was Running, so a restart or readiness alert would have stayed quiet. With nginx or ingress controller metrics, alert when 5xx responses exceed 5 percent of requests for 5 minutes (suggested expression, not captured in the lab): ``` sum(rate(nginx_ingress_controller_requests{status=~"5.."}[5m])) / sum(rate(nginx_ingress_controller_requests[5m])) > 0.05 ``` **Guardrail.** Treat a NetworkPolicy change like a firewall change: the review lists every client of the selected pods and checks each one against the `from` selectors. In the lab the policy's author allowed the worker, which never talks to redis, and forgot the api, which does on every request. ## Common mistakes - **Reading "Running" as "reachable".** Pod status says the container is up. It says nothing about who is allowed to talk to it. - **Restarting the client.** A new pod gets the same labels, so the policy drops it the same way. - **Blaming the dependency.** redis was healthy on every check: logs, state, endpoints, 3m CPU and 4Mi memory (run r1, step 12). The failure lives in the policy between them. - **Treating refused and timed out as the same error.** Refused points at the listener or the endpoints; timed out points at the path. - **Looking only at the client's namespace.** The policy lives on the dependency's pods. If the dependency is in another namespace, check there too. ## What we saw in the lab Scenario 13 was part of run v1-run1 (2 October 2026), the harder set where noise sits next to the fault. The alert said `HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes`. Aurora got the alert text and read-only kubectl under its default `observability_only` policy, two runs in fresh orgs. | Run | Verdict | Time | Tool calls | Cost | Fix proposed | |---|---|---|---|---|---| | r1 | correct (human reviewed) | 202 s | 25 | $0.69 | Patch the `redis` NetworkPolicy to allow `app=api` ingress | | r2 | correct (human reviewed) | 444 s | 19 | $0.54 | Patch the `redis` NetworkPolicy to allow ingress from `app=api` pods | Both runs met both required findings: the api cannot reach redis and redis itself is healthy, and the cause is the policy that allows only `app=worker`. Neither blamed redis or DNS, and neither proposed a restart. r1 took the long way. Its 19 kubectl calls went pods, events, frontend logs, api logs, ConfigMaps, redis logs, redis describe, `top pods`, worker logs, the `api-code` ConfigMap (where it read the 2 second socket timeout in the api's `redis_ping`), Services, `top nodes`, worker describe, nodes, the loadgen pod, and only then `get networkpolicies` as its 16th kubectl call, followed by the api Deployment, `describe networkpolicy` and the api pod labels. Its report put it plainly: "The root cause was a NetworkPolicy named `redis` in the shop namespace, created at 2026-10-02T20:23:54Z, which restricts ingress to redis:6379 to pods labeled `app=worker` only". r2 was more direct with 14 kubectl calls: pods, events, frontend log, api log, loadgen log, redis log, redis describe, Services, the api's `REDIS_HOST`, the redis endpoints, the policy YAML, then the redis and api labels, then the worker logs. It also wrote something the evidence does not support: that the worker "successfully processed Redis-backed batches during the same window". The worker's code in the `api-code` ConfigMap only allocates and frees buffers and never connects to redis, a point r1 noticed and flagged under "Not Checked". The AI review recorded in the grade flagged the sentence as unsupported; it did not change the verdict because it was not part of the root cause. Across the nine v1 scenarios this one tied for the most expensive for Aurora at a median of $0.61, and was the third slowest at a median of 323 s. The reason is visible in r1's path: the policy object is not attached to any pod, Deployment or event, so an investigator that starts from the pods has to run out of other explanations before listing policies. Running `get networkpolicies` third, as this runbook does, is the shortcut the lab taught us. ## How Aurora handles this Aurora receives the alert, reads the pods, events and logs for the service in front and the one behind, confirms the dependency is healthy, and then lists the namespace's NetworkPolicies and compares their selectors to the client's labels, citing each finding to a command output. In both lab runs it named the policy, the exact label mismatch and the fix, and it refused to blame redis or propose a restart. Aurora runs read-only by default, so the patch above stays a suggestion until a person applies it. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) ## Sources and verification - Lab scenario and fault: [scenarios/13-networkpolicy-blocks-redis](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/13-networkpolicy-blocks-redis) (`scenario.yaml`, `break.sh`, `check.sh`). - Every output block above is from [runs/v1-run1/13-networkpolicy-blocks-redis](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/13-networkpolicy-blocks-redis): the run's tool transcripts (`steps-full.json`), the captured cluster state (`evidence-before/`), Aurora's reports (`aurora-rca.md`) and the grades (`grade.json`), with the run and step named next to each block. Outputs are trimmed, never edited. - Versions from `evidence-before/versions.yaml` and `kubectl get nodes -o wide` (run r1, step 18): Kubernetes server v1.37.0, kubectl v1.35.3, kind v0.33.0, containerd 2.3.4. redis 7.4.11 from its startup log, nginx 1.27.5 from the frontend log. - The two quoted sentences on isolation and on controllers are from the [Kubernetes NetworkPolicy documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/), read on 16 October 2026. - The alert expression under Prevent is a suggestion, not something the lab captured. - Method and timings: [Root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres) and the [lab report](/blog/aurora-lab-report-kubernetes-19-failures). --- ## Exceeded quota: a ResourceQuota blocks the rollout's surge pod (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-resourcequota-exceeded-rollout-stuck-kubernetes Published: 2026-10-16 Author: Damian Loch, Software Engineer at Arvo AI > A Deployment rollout that never progresses, with FailedCreate and exceeded quota on the new ReplicaSet. Confirm the quota, give it headroom, verify. | Applies to | Kubernetes Deployments with a RollingUpdate strategy in a namespace that has a ResourceQuota on requests or limits | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, kubectl v1.35.3, 2 October 2026 | | Lab folder | [scenarios/14-resourcequota-rollout](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/14-resourcequota-rollout), runs in [runs/v1-run1/14-resourcequota-rollout](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/14-resourcequota-rollout) | **A rollout stuck on "exceeded quota" means the namespace ResourceQuota has no room for the extra pod a rolling update creates, so the new ReplicaSet cannot create any pod at all.** No pod object exists, so there are no pod events and no logs; the evidence is on the ReplicaSet. The old pods keep serving, so this is a stuck rollout, not an outage. Confirm with `kubectl describe rs -n ` and read the `FailedCreate` event; the fix is to give the quota headroom for one surge pod, or to roll out without a surge. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | New ReplicaSet `0 current / 1 desired`, `FailedCreate`, message contains `exceeded quota` | A ResourceQuota is fully used; the surge pod is rejected at admission | `kubectl describe resourcequota -n ` shows Used equal to Hard | Raise the quota or set `maxSurge: 0`, `maxUnavailable: 1` | | New pod exists and sits `Pending` with `FailedScheduling` | No node fits the pod; the quota is not involved | `kubectl describe pod ` Events | Fix requests, taints or node capacity | | New pod exists in `ImagePullBackOff` | The image reference is wrong or private | `kubectl describe pod ` Events | See the [ImagePullBackOff runbook](/runbooks/fix-imagepullbackoff-kubernetes) | | `FailedCreate` with a validation error, no `exceeded quota` | The pod template is invalid | `kubectl describe rs ` Events | Fix the template field the error names | ## How do you confirm the quota is what blocks the rollout? Start with the Deployment. In the lab, the alert was `KubeDeploymentRolloutStuck` for `shop/api`. Aurora's first command (lab run r1, step 4) showed the shape of a stuck rollout: ready, available, and nothing updated. ```shell kubectl get deployment api -n shop -o wide ``` ``` NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR api 2/2 0 2 64s api python:3.12-alpine app=api ``` `UP-TO-DATE 0` with `AVAILABLE 2` means the old pods serve and the new revision has not produced a single pod. Look at the ReplicaSets (r1, step 6): ```shell kubectl get rs -n shop ``` ``` NAME DESIRED CURRENT READY AGE api-57f65dc5db 1 0 0 64s api-f64486c4c 2 2 2 66s ``` The new ReplicaSet wants 1 pod and has 0. A pod that was never created is the signature of an admission rejection, and the reason is on the ReplicaSet, not on any pod. Describe it (r1, step 7, trimmed to the condition and the first events): ```shell kubectl describe rs api-57f65dc5db -n shop ``` ``` Replicas: 0 current / 1 desired Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed ... Conditions: Type Status Reason ---- ------ ------ ReplicaFailure True FailedCreate Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 75s replicaset-controller Error creating: pods "api-57f65dc5db-k2g2k" is forbidden: exceeded quota: compute-resources, requested: requests.memory=32Mi, used: requests.memory=200Mi, limited: requests.memory=200Mi Warning FailedCreate 75s replicaset-controller Error creating: pods "api-57f65dc5db-zqwlr" is forbidden: exceeded quota: compute-resources, requested: requests.memory=32Mi, used: requests.memory=200Mi, limited: requests.memory=200Mi Warning FailedCreate 33s (x5 over 72s) replicaset-controller (combined from similar events): Error creating: pods "api-57f65dc5db-fvv46" is forbidden: exceeded quota: compute-resources, requested: requests.memory=32Mi, used: requests.memory=200Mi, limited: requests.memory=200Mi ``` That one message is the whole diagnosis: the quota is named (`compute-resources`), the dimension (`requests.memory`), what the pod asked for (32Mi), what is used (200Mi) and the limit (200Mi). The [ResourceQuota documentation](https://kubernetes.io/docs/concepts/policy/resource-quotas/) describes the mechanism: "If creating or updating a resource violates a quota constraint, the control plane rejects that request with HTTP status code 403 Forbidden. The error includes a message explaining the constraint that would have been violated." The ReplicaSet controller receives that 403 and records it as `FailedCreate`. ## How much is affected, and since when? The rollout is stuck; the service is not down. Check the Deployment conditions and the two ReplicaSets together (r1, step 9, trimmed): ```shell kubectl describe deployment api -n shop ``` ``` Replicas: 2 desired | 0 updated | 2 total | 2 available | 1 unavailable StrategyType: RollingUpdate RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Annotations: kubectl.kubernetes.io/restartedAt: 2026-10-02T16:28:16-04:00 ... Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetCreated ReplicaFailure True FailedCreate OldReplicaSets: api-f64486c4c (2/2 replicas created) NewReplicaSet: api-57f65dc5db (0/1 replicas created) ``` `Available True` with `ReplicaFailure True` is the stuck-but-serving state. The `restartedAt` annotation tells you what started the rollout and when: here a `kubectl rollout restart`. The quota's own creation time (r1, step 13, `creationTimestamp: "2026-10-02T20:28:16Z"`) tells you whether the quota is new. In the lab both happened within seconds, which is what the scenario does on purpose; in production a quota that has been there for months and a rollout that fails today means usage grew until it reached the ceiling. Every pod in the namespace was still Running (r1, step 5), so nothing else was affected: ``` NAME READY STATUS RESTARTS AGE api-f64486c4c-lr8pl 1/1 Running 0 61s api-f64486c4c-rjvtf 1/1 Running 0 61s frontend-76d98597b9-bb5qs 1/1 Running 0 61s frontend-76d98597b9-x6jst 1/1 Running 0 61s loadgen-7c6676864-gz6mq 1/1 Running 0 61s redis-59458f55d8-7nc8l 1/1 Running 0 61s worker-6cf88d4b58-5sczh 1/1 Running 0 61s ``` ## Which quota, and why is there no room? 1. **Read the quota.** Used against Hard, per dimension (r1, step 10): ```shell kubectl describe resourcequota compute-resources -n shop ``` ``` Name: compute-resources Namespace: shop Resource Used Hard -------- ---- ---- requests.memory 200Mi 200Mi ``` Used equal to Hard, on exactly the dimension the `FailedCreate` message names. Any new pod that requests memory is rejected until something frees requests or the limit moves. 2. **Add up who holds the requests.** One request per pod, from the pod specs (r1, step 11): ```shell kubectl get pods -n shop -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[0].resources.requests.memory}{"\n"}{end}' ``` ``` api-f64486c4c-lr8pl 32Mi api-f64486c4c-rjvtf 32Mi frontend-76d98597b9-bb5qs 16Mi frontend-76d98597b9-x6jst 16Mi loadgen-7c6676864-gz6mq 8Mi redis-59458f55d8-7nc8l 32Mi worker-6cf88d4b58-5sczh 64Mi ``` 32 + 32 + 16 + 16 + 8 + 32 + 64 = 200Mi. The namespace sits exactly at its limit, so there is no 32Mi for a third api pod. Quote the shell string as shown; run r2 (step 11) sent the jsonpath unquoted and got `Error from server (NotFound)` back, then used `-o custom-columns` instead (step 13). 3. **Work out how many surge pods the rollout needs.** From the strategy (r1, step 14): ```shell kubectl get deployment api -n shop -o jsonpath='{.spec.strategy}{"\n"}' ``` ``` {"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"} ``` Per the [Deployment documentation](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), for `maxSurge` "The absolute number is calculated from the percentage by rounding up." and for `maxUnavailable` "The absolute number is calculated from percentage by rounding down." With 2 replicas, 25% surge rounds up to 1 extra pod and 25% unavailable rounds down to 0, so the controller must create a third pod before it may remove an old one. That third pod is the 32Mi the quota refuses. 4. **Rule out the look-alikes.** A scheduling problem leaves a pod object in `Pending` with a `FailedScheduling` event; here no pod exists. An image problem leaves a pod in `ImagePullBackOff`; the image was unchanged (`python:3.12-alpine` on both ReplicaSets, r1 steps 4 and 6). An invalid template also produces `FailedCreate`, but the message is a validation error, not `exceeded quota`. ## How do you fix it? Pick one of three, in this order of preference. **Raise the quota so one surge pod fits.** The lab's quota needs at least 232Mi (200Mi used plus the 32Mi surge pod); round to something with margin. ```shell kubectl -n shop patch resourcequota compute-resources --type merge -p '{"spec":{"hard":{"requests.memory":"256Mi"}}}' ``` Risk: low. The quota is a ceiling on requests, not a change to any pod. Rollback: patch it back to the previous value. The rollout resumes on its own; the ReplicaSet controller keeps retrying. **Free requests in the namespace.** Scale down or delete something that holds requests you do not need. Risk: whatever that workload does stops. Rollback: scale it back up. **Roll out without a surge.** If the quota is policy and cannot move, change the strategy so the controller removes an old pod before adding a new one: ```shell kubectl -n shop patch deployment api --type merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}}}' ``` Tradeoff: with 2 replicas you serve from 1 pod while each new pod starts, so capacity drops by half during the rollout. Risk: a bad new revision takes a serving pod with it before you see it fail. Rollback: patch the strategy back to 25% and 25%. The documentation notes the pair cannot both be 0. In the lab, both of Aurora's runs proposed the first option in their suggestions list: raise `requests.memory` on `compute-resources` (to 256Mi in r1, 232Mi in r2), with `maxSurge 0 / maxUnavailable 1` as the stopgap. Neither fix was applied in a graded run. ## How do you know it is fixed? The new ReplicaSet reaches its desired count and the Deployment stops reporting `ReplicaFailure`: ```shell kubectl -n shop rollout status deploy/api --timeout=120s kubectl -n shop get rs -l app=api kubectl -n shop describe resourcequota compute-resources ``` Pass condition: `rollout status` prints `successfully rolled out`, the new ReplicaSet shows `2 2 2`, the old one `0 0 0`, and Used is below Hard again once the old pods are gone. Watch for one full rollout. The lab's own check for the broken state is one line, and the pass is its absence for new events: ```shell kubectl -n shop get events --field-selector reason=FailedCreate -o jsonpath='{.items[*].message}' | grep -q 'exceeded quota' ``` Old `FailedCreate` events stay in the list until they age out, so judge by the ReplicaSet counts, not by that grep alone. ## How do you stop it happening again? **Alert before the quota is full.** With kube-state-metrics, `kube_resourcequota` carries both `type="used"` and `type="hard"` series per resource; alert when used reaches a percentage of hard that leaves less than one surge pod of headroom (suggested query, not captured in the lab): ``` max by (namespace, resourcequota, resource) (kube_resourcequota{type="used"}) / max by (namespace, resourcequota, resource) (kube_resourcequota{type="hard"}) > 0.9 ``` **Size quotas for rollouts, not for steady state.** A namespace quota must cover steady-state requests plus one surge pod for the largest Deployment that rolls out (or `maxSurge` worth of pods, if higher). Review the sum whenever a request value or a replica count changes. **Alert on the condition, not only on progress.** The Deployment sets `ReplicaFailure True` on the first rejected create (r1, step 9), long before a 15-minute rollout-stuck alert. With kube-state-metrics, `kube_deployment_status_condition{condition="ReplicaFailure",status="true"}` is the series to alert on (suggested, not captured in the lab). ## Common mistakes - **Looking for a pod that does not exist.** `kubectl get pods` shows only the old, healthy pods, and `kubectl logs` has nothing new. The evidence is on the ReplicaSet and in the namespace events. - **Treating it as a scheduling problem.** `FailedCreate` happens at admission, before the scheduler sees anything. Adding nodes changes nothing. - **Deleting the old pods to make room.** That frees quota, but you serve from zero pods until the new one starts, and if the new revision is broken you have an outage instead of a stuck rollout. - **Raising the quota by exactly the surge pod.** The next Deployment to roll out hits the same wall. Leave headroom for the largest surge in the namespace. - **Forgetting that limits can be quotaed too.** This lab quotas `requests.memory`; a quota on `limits.memory` or `requests.cpu` fails the same way with a different dimension in the message. ## What we saw in the lab This scenario is from run v1-run1 (2 October 2026), the harder set: the quota was created with the namespace already at exactly its limit, then `kubectl rollout restart deploy/api` was issued. Aurora received only the alert text and read-only kubectl under its default `observability_only` policy. | Run | Verdict | Time | Tool calls | Cost | Fix proposed | |---|---|---|---|---|---| | r1 | correct (human reviewed) | 108 s | 14 (1 error) | $0.24 | raise the quota to 256Mi, or maxSurge 0 / maxUnavailable 1 | | r2 | correct (human reviewed) | 144 s | 19 (2 errors) | $0.39 | raise the quota to 232Mi, or maxSurge 0 / maxUnavailable 1 | Both runs met both required findings: the new ReplicaSet cannot create pods because the `compute-resources` `requests.memory` quota is exhausted, and the old pods still serve. Neither asserted the listed wrong answers (node capacity, image pull, application crash). r1's report put it this way: "Because the quota had zero headroom, every attempt by the new ReplicaSet to create its surge pod was rejected by quota admission control, producing repeated FailedCreate events on `api-57f65dc5db` [7, 8] and permanently stalling the rollout." r1's path, in order: Deployment, pods, ReplicaSets, describe the new ReplicaSet (the `FailedCreate` events above), namespace events, describe the Deployment, describe the quota, per-pod memory requests, `rollout history` (refused by policy, see below), the quota YAML, the strategy. Eleven kubectl calls to the answer; the other 3 of the 14 were the incident list, a memory search and the cluster lookup. r1 wrote no memory. r2 reached the same evidence in a different order: Deployment, pods, describe the Deployment, ReplicaSets, events, describe the quota, a per-pod request query that failed because the jsonpath was unquoted, `rollout history` (refused), the same query again with `custom-columns`, describe the new ReplicaSet, the quota YAML, nodes. Twelve kubectl calls, two of them errors; the other 7 of the 19 were the incident list, two memory searches, the cluster lookup, the incident fetch and two memory writes. The reports had two slips the judge flagged: both called the alert high-severity when it was a warning, and r1's timeline said the rollout began at 20:28:14Z, two seconds before the quota's 20:28:16Z. That earlier time is the Deployment's creation time, not the rollout's: the `restartedAt` annotation in the same output (r1, step 9) is 16:28:16-04:00, the same second as the quota, and the scenario creates the quota first. The judge also flagged r2's "1 unavailable" as a slip; kubectl prints exactly that (r1, step 9), counting the surge pod that was never created. Neither report stated a remediation; the fixes were in Aurora's separate suggestions list and were graded acceptable in both runs. `kubectl rollout history` is read-only, but Aurora's default policy denied it in both runs (r1 step 12, r2 step 12) because the deny rule matches the whole `rollout` verb. Aurora used the `restartedAt` annotation and the two ReplicaSets instead. ## How Aurora handles this Aurora receives the alert, reads the Deployment and its ReplicaSets, finds the `FailedCreate` event on the one that has no pods, reads the quota it names and adds up the requests that fill it, then writes the root cause with each claim cited to a command output, as in the two runs above. It states that the old pods still serve and separates the quota from scheduling, image and application causes. The fix goes in a suggestions list the on-call can apply or reject. See an investigation end to end on the [root cause analysis](/rca) page. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) ## Sources and verification - Lab scenario [14-resourcequota-rollout](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/14-resourcequota-rollout): `scenario.yaml`, `break.sh`, `check.sh`. Runs [v1-run1/14-resourcequota-rollout](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/14-resourcequota-rollout), r1 and r2, 2 October 2026: every output block above is pasted from those transcripts and captured cluster state, trimmed but not altered. - Versions: Kubernetes server v1.37.0, kubectl v1.35.3, kind v0.33.0, containerd 2.3.4 (lab `versions.yaml` and r2 step 16). - [Resource Quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/), [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), Kubernetes documentation, quoted 16 October 2026. - Related: [ImagePullBackOff](/runbooks/fix-imagepullbackoff-kubernetes), [Pod Pending with FailedScheduling](/runbooks/fix-pod-pending-failedscheduling-kubernetes), [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres). --- ## CreateContainerConfigError: the Secret or ConfigMap is missing (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-createcontainerconfigerror-kubernetes Published: 2026-10-15 Author: Ben Gervais, Engineer at Arvo AI > CreateContainerConfigError means a Secret or ConfigMap the pod references is missing. Read the event, create the object or roll back, verify. Lab-tested. | Applies to | Pods whose containers take an env var or volume from a Secret or ConfigMap; here a Deployment rollout in Kubernetes | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, containerd 2.3.4, 2 October 2026 | | Lab folder | [scenarios/04-missing-secret](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/04-missing-secret), runs in [runs/v0-run2/04-missing-secret](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/04-missing-secret) | **CreateContainerConfigError means the kubelet could not build the container's configuration, almost always because a Secret or ConfigMap the pod references does not exist.** The image is on the node, the pod is scheduled, and nothing has run yet, so there are no logs. Confirm it with `kubectl describe pod `: the last event reads `Error: secret "" not found` (or `configmap "" not found`). The fix is to create the missing object with the key the pod expects, or roll the change back. Nothing else helps. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | `CreateContainerConfigError`, event `Error: secret "" not found` | The Secret named in `secretKeyRef` or a `secret` volume does not exist in this namespace | `kubectl get secret -n ` | Create the Secret with the expected key, or roll back | | `CreateContainerConfigError`, event `Error: configmap "" not found` | Same for a ConfigMap (`configMapKeyRef` or a `configMap` volume) | `kubectl get configmap -n ` | Create the ConfigMap, or roll back | | `CreateContainerConfigError`, event `couldn't find key in Secret /` | The object exists, the key does not | `kubectl get secret -n -o jsonpath='{.data}'` | Add the key, or fix the key name in the spec | | `ImagePullBackOff` instead | A different failure: the image, not the config | See [ImagePullBackOff](/runbooks/fix-imagepullbackoff-kubernetes) | | | `CrashLoopBackOff` instead | The container started and exited | `kubectl logs --previous` | See [OOMKilled](/runbooks/fix-oomkilled-kubernetes) for one common cause | ## How do you confirm it is CreateContainerConfigError? List the pods. The failing pod shows the reason in the STATUS column and 0 restarts, because nothing has started (from the lab run r1, step 5): ```shell kubectl get pods -n shop -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-5f7c48ddc6-nc8f7 0/1 CreateContainerConfigError 0 71s 10.244.1.107 aurora-lab-worker api-f64486c4c-c9rqz 1/1 Running 0 81s 10.244.2.106 aurora-lab-worker2 api-f64486c4c-dr5x9 1/1 Running 0 81s 10.244.1.103 aurora-lab-worker ``` Then describe the pod. The container state and the events name the missing object (run r1, step 6, trimmed): ```shell kubectl describe pod api-5f7c48ddc6-nc8f7 -n shop ``` ``` State: Waiting Reason: CreateContainerConfigError Ready: False Restart Count: 0 Environment: REDIS_HOST: redis DB_PASSWORD: Optional: false Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 82s default-scheduler Successfully assigned shop/api-5f7c48ddc6-nc8f7 to aurora-lab-worker Normal Pulled 3s (x8 over 82s) kubelet spec.containers{api}: Container image "python:3.12-alpine" already present on machine and can be accessed by the pod Warning Failed 3s (x8 over 82s) kubelet spec.containers{api}: Error: secret "api-secrets" not found ``` Three things in that output decide the diagnosis. `Pulled` says the image is fine. `Restart Count: 0` and an empty `Container ID` say the container never ran, so `kubectl logs` has nothing to show. `Optional: false` says the pod is not allowed to start without the value. The [Secrets documentation](https://kubernetes.io/docs/concepts/configuration/secret/), written about Secret volumes, describes the same retry that applies here: "a Secret needs to be created before any Pods that depend on it. If the Secret cannot be fetched (perhaps because it does not exist, or due to a temporary lack of connection to the API server) the kubelet periodically retries running that Pod. The kubelet also reports an Event for that Pod, including details of the problem fetching the Secret." ## How much is affected, and since when? Check whether the failing pod is a new revision or the only copy. In the lab it was a rollout: the old ReplicaSet kept two Ready pods while the new one could not start (run r1, step 7): ```shell kubectl get rs -n shop -o wide ``` ``` NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR api-5f7c48ddc6 1 1 0 85s api python:3.12-alpine app=api,pod-template-hash=5f7c48ddc6 api-f64486c4c 2 2 2 95s api python:3.12-alpine app=api,pod-template-hash=f64486c4c ``` The Deployment confirms the shape: a stuck rollout, not an outage (run r1, step 15, trimmed): ``` Annotations: deployment.kubernetes.io/revision: 2 Replicas: 2 desired | 1 updated | 3 total | 2 available | 1 unavailable Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True ReplicaSetUpdated OldReplicaSets: api-f64486c4c (2/2 replicas created) NewReplicaSet: api-5f7c48ddc6 (1/1 replicas created) ``` `Available: True` with an old ReplicaSet still at 2/2 means users are served and you have time. If the Deployment has no old ReplicaSet, or the pod is a bare pod or a Job, every replica is down and the fix is urgent. The ReplicaSet age tells you when the change landed: here the new ReplicaSet was 10 seconds younger than the old one. ## Which object is missing, and why? Work from the pod spec outward, cheapest check first. 1. Read what the new pods want. Compare the running template to a running pod of the old revision (run r1, steps 13 and 10): ```shell kubectl get deployment api -n shop -o jsonpath='{.spec.template.spec.containers[0].env}' ``` ``` [{"name":"REDIS_HOST","value":"redis"},{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"key":"db-password","name":"api-secrets"}}}] ``` ```shell kubectl get pod api-f64486c4c-dr5x9 -n shop -o jsonpath='{.spec.containers[0].env}' ``` ``` [{"name":"REDIS_HOST","value":"redis"}] ``` The diff is the change: one new env var, `DB_PASSWORD`, read from Secret `api-secrets`, key `db-password`. 2. Check the object exists in the pod's namespace (run r1, step 8). Secrets are namespaced; a Secret in another namespace does not count. ```shell kubectl get secrets -n shop ``` ``` ``` Empty. No Secret of any name in the namespace. 3. Check it was not created somewhere else by mistake (run r1, step 12): ```shell kubectl get secrets -A ``` ``` NAMESPACE NAME TYPE DATA AGE kube-system bootstrap-token-abcdef bootstrap.kubernetes.io/token 6 23h ``` Nowhere in the cluster. If the object does exist, the event would instead say `couldn't find key `, and the check is the key list: `kubectl get secret -n -o jsonpath='{.data}'`. 4. If a secrets operator is supposed to create it (External Secrets, Sealed Secrets, Vault Agent), check that its pods exist and read its own status object. Aurora's run r2 checked for operator pods with `kubectl get pods -A` (step 13) and found none, so in the lab there was nothing to wait for. 5. Find out what changed. `kubectl rollout history deployment/api -n shop` shows the revisions. In the lab the `last-applied-configuration` annotation on the Deployment still held the old spec with only `REDIS_HOST`, while the live template had `DB_PASSWORD` (run r2, step 9): the change was made out of band, not through `kubectl apply`. ## How do you fix it? Two acceptable fixes. Pick by whether the new revision should exist. **Create the missing object with the key the pod expects.** The pod recovers on the kubelet's next retry, within about ten seconds in the lab; no restart is needed. ```shell kubectl create secret generic api-secrets -n shop --from-literal=db-password='' ``` Risk: you are adding a credential the running code may not use. In the lab the api code read only `REDIS_HOST` (run r1, step 16), so the variable was introduced into the manifest without a code change, which is worth a question to whoever made it. Rollback: `kubectl delete secret api-secrets -n shop` after rolling the Deployment back, never while pods depend on it. **Roll the Deployment back** when the new revision was not meant to ship yet: ```shell kubectl rollout undo deployment/api -n shop ``` Risk: you also undo any other change in that revision. Rollback of the rollback: `kubectl rollout undo deployment/api -n shop --to-revision=`. Do not mark the reference `optional: true` as a fix. The pod would start without the value, and the application would fail later, further from the cause. ## How do you verify the fix? Pass condition: no pod of the Deployment in `CreateContainerConfigError`, and the rollout complete. ```shell kubectl -n shop get pods -l app=api -o jsonpath='{.items[*].status.containerStatuses[0].state.waiting.reason}' kubectl rollout status deployment/api -n shop ``` The first command prints nothing when every container is past the waiting state. It is the negation of the lab's own check for the broken state, `scenarios/04-missing-secret/check.sh`, which greps that same field for `CreateContainerConfigError`. Watch for two minutes: a pod that leaves the waiting state and then restarts has a second problem. ## How do you prevent it? **Alert on the waiting reason.** With kube-state-metrics (a suggested query, not one captured in the lab): ``` sum by (namespace, pod) (kube_pod_container_status_waiting_reason{reason="CreateContainerConfigError"}) > 0 ``` **Validate references before the rollout.** In CI, render the manifests and check that every `secretKeyRef`, `configMapKeyRef`, `secret` volume and `configMap` volume names an object that exists in the target namespace, or is created in the same change. The lab's fault is a one-line patch that adds a reference without the object; a dry-run cannot catch it, only a reference check can. **Create the Secret in the same change set as the pod that reads it**, and in the same namespace. ## Common mistakes - **Reading logs.** `kubectl logs` returns nothing useful; the container never started. The evidence is in `describe pod` events. - **Treating it as an image problem.** The `Pulled` event is Normal. `ImagePullBackOff` is a different reason with a different fix. - **Creating the Secret in the wrong namespace.** `kubectl get secrets -A` shows where it landed. - **Creating the Secret without the key.** The error changes from `not found` to `couldn't find key`. Check `.data`. - **Deleting the stuck pod.** The ReplicaSet makes an identical one; the reference is in the template. - **Setting `optional: true` to make the pod start.** The failure moves into the application, where it is harder to find. ## What we saw in the lab Scenario `04-missing-secret`: `break.sh` patches the api Deployment to add `DB_PASSWORD` from `secretKeyRef` `api-secrets/db-password` and never creates the Secret. Aurora received only the alert text, `KubeDeploymentRolloutStuck: deployment api in namespace shop has not progressed; new pods not starting.`, and read-only cluster access. Two runs, each in a fresh account. | Run | Verdict | Time | Tool calls | Cost | Fix in suggestions | |---|---|---|---|---|---| | r1 | correct (judge, then human) | 194 s | 18 | $0.48 | none | | r2 | correct (judge, then human) | 133 s | 17 | $0.35 | create the Secret; roll back as the stopgap | Both runs reached the same evidence: list the Deployment, list the pods, describe the stuck pod, read the ReplicaSets, `get secrets -n shop` (empty), then `get secrets -A` (nothing but the bootstrap token). Both tried `kubectl rollout history` and were refused by Aurora's default `observability_only` command policy, which denies the whole `rollout` verb including the read-only subcommands (one tool error per run). Both worked around it by comparing the live template with a running pod of the old revision, and r2 also read the `last-applied-configuration` annotation. r1's report named the cause plainly: "The secret `api-secrets` was never created in the `shop` namespace, and no application secrets exist anywhere in the cluster", and added a point from reading the code in the `api-code` ConfigMap: the application never reads `DB_PASSWORD`, so the variable was added without a code dependency. r1 ruled out image pull, resource limits, node problems and probe failures with the evidence for each. The report says no fix was applied because mutating kubectl was policy-blocked; it never attempted one, so the sentence describes the policy, not an attempt. r1 left its Suggested Next Steps list empty, which is why the fix column reads none. Its closing message did name both fixes, create the Secret with the `db-password` key or roll the Deployment back, and graded on everything Aurora showed, both were acceptable. r2's suggestions were to create `api-secrets` with the `db-password` key, with rollback as the stopgap, both acceptable. The grader read Aurora's full output (report, notes, suggestions); a person at Arvo then checked both verdicts. Transcripts and grades: [runs/v0-run2/04-missing-secret](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/04-missing-secret). ## How Aurora handles this Aurora receives the alert, lists the Deployment and its pods, describes the pod in `CreateContainerConfigError`, confirms the referenced Secret is missing in the namespace and in the cluster, and diffs the new template against a running pod to name the change. Every claim in its report cites the tool output it came from, and what it could not check is listed as such. It runs with read-only kubectl under a command policy by default. See a full investigation on the [Aurora root cause analysis](/rca) page. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) ## Sources and verification Every command output above is pasted from the lab runs of 2 October 2026 (Kubernetes server v1.37.0, kind v0.33.0, containerd 2.3.4, Aurora commit 87d22baf, model bedrock/us.anthropic.claude-sonnet-5) and attributed to its run and step. The Secret retry behaviour is quoted from the [Kubernetes Secrets documentation](https://kubernetes.io/docs/concepts/configuration/secret/), read 15 October 2026. The scenario, fault injection and check are in [scenarios/04-missing-secret](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/04-missing-secret). Related: [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes), [a pod stuck in Init:0/1](/runbooks/fix-pod-stuck-init-container-kubernetes). --- ## nginx [emerg] unknown directive: a ConfigMap typo crash-loops the pod (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-nginx-emerg-unknown-directive-kubernetes Published: 2026-10-15 Author: Damian Loch, Software Engineer at Arvo AI > nginx [emerg] unknown directive means a config line nginx does not know, usually a typo in a mounted ConfigMap. Find the line, fix it, restart, check. | Applies to | nginx containers on Kubernetes whose configuration comes from a mounted ConfigMap; here the frontend of the lab's `shop` app | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, nginx 1.27-alpine (nginx/1.27.5), 2026-10-02 | | Lab folder | [scenarios/05-nginx-config-error](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/05-nginx-config-error), runs in [runs/v0-run2/05-nginx-config-error](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/05-nginx-config-error) | **`nginx: [emerg] unknown directive "..."` means nginx read a configuration line it does not recognise and refused to start, so the container exits with code 1 and the pod crash-loops.** The most likely cause is a typo in the ConfigMap that is mounted as the config file. Confirm with `kubectl logs --previous`: the `[emerg]` line names the directive, the file and the line number. The fix is to correct that line in the ConfigMap and restart the pods that mount it. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | `CrashLoopBackOff`, exit code 1, `[emerg] unknown directive` in the previous log | A directive is misspelled or unsupported in the mounted config | `kubectl logs --previous` then `kubectl get configmap -o yaml` | Correct the directive, restart the Deployment | | `[emerg]` names a directive that exists, but the module is missing | The image was built without that module | `nginx -V` in a healthy pod of the same image | Use an image with the module, or remove the directive | | Exit code 137, no `[emerg]` line | OOMKilled, not a config problem | `kubectl describe pod` shows `Reason: OOMKilled` | See [how to fix OOMKilled](/runbooks/fix-oomkilled-kubernetes) | | Pod Running, `Readiness probe failed`, no restarts | The app is up but the probe path or port is wrong | `kubectl describe pod` events, `kubectl logs` with no `[emerg]` | Fix the probe, not the config | ## How do you confirm it is a config error? Read the pods, then the log of the container that already died. In the lab the alert said a frontend pod was restarting after a config change. lab run r1, step 4: ```shell kubectl get pods -n shop -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-f64486c4c-nxp5w 1/1 Running 0 72s 10.244.2.12 aurora-lab-2-worker2 api-f64486c4c-q657d 1/1 Running 0 72s 10.244.1.9 aurora-lab-2-worker frontend-7547556c6b-svhfr 0/1 CrashLoopBackOff 3 (30s ago) 65s 10.244.2.16 aurora-lab-2-worker2 frontend-76d98597b9-7fnr2 1/1 Running 0 72s 10.244.1.10 aurora-lab-2-worker frontend-76d98597b9-crvl6 1/1 Running 0 72s 10.244.2.15 aurora-lab-2-worker2 ``` One frontend pod crash-loops while two others, with a different pod-template-hash, are fine. The crashing one belongs to a newer ReplicaSet. Its current container has nothing useful to say, so read the previous one (r1, step 6): ```shell kubectl logs frontend-7547556c6b-svhfr -n shop --previous --tail=100 ``` ``` /docker-entrypoint.sh: Configuration complete; ready for start up 2026/10/02 16:30:48 [emerg] 1#1: unknown directive "proxy_conect_timeout" in /etc/nginx/conf.d/default.conf:9 nginx: [emerg] unknown directive "proxy_conect_timeout" in /etc/nginx/conf.d/default.conf:9 ``` That line is the whole diagnosis: the directive, the file and the line number. `describe pod` confirms the shape of the failure (r1, step 7, trimmed): ``` State: Terminated Reason: Error Exit Code: 1 Last State: Terminated Reason: Error Exit Code: 1 Ready: False Restart Count: 4 Readiness: http-get http://:80/healthz delay=0s timeout=1s period=5s successThreshold=1 failureThreshold=3 Mounts: /etc/nginx/conf.d from conf (rw) ``` Exit code 1 with an `[emerg]` line is a config parse failure. The readiness probe failures in the events are a consequence: nothing is listening on port 80 because nginx never started. ## How much is affected, and since when? The Deployment view shows a rollout stopped halfway (cluster state captured before the investigation, run r1): ``` NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/frontend 2/2 1 2 57s nginx nginx:1.27-alpine app=frontend NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR replicaset.apps/frontend-7547556c6b 1 1 0 50s nginx nginx:1.27-alpine app=frontend,pod-template-hash=7547556c6b replicaset.apps/frontend-76d98597b9 2 2 2 57s nginx nginx:1.27-alpine app=frontend,pod-template-hash=76d98597b9 ``` Two old pods are Ready and still in the Service (run r2, step 14): ```shell kubectl get endpoints frontend -n shop ``` ``` NAME ENDPOINTS AGE frontend 10.244.1.23:80,10.244.2.27:80 2m27s ``` So users are not down yet. The rolling update stopped at the first failing pod, and the old pods keep serving. They carry the same bad ConfigMap mount, so they will fail the next time they restart. Since when: the Deployment's pod template carries `kubectl.kubernetes.io/restartedAt: "2026-10-02T12:30:13-04:00"` (r1, step 9), the moment someone ran a rollout restart to pick up the edited config. In the lab that restart was the second line of the injected fault. ## Which of the four causes is it? Cheapest check first. 1. **Is it OOM?** Exit code 1, not 137, and `Reason: Error`, not `OOMKilled` (r1, step 7). No. 2. **Is it the image?** Both ReplicaSets run `nginx:1.27-alpine`, and the events say `Container image "nginx:1.27-alpine" already present on machine`. The old pods run the same image fine. No. 3. **Is it the probe?** The probe fails with `connect: connection refused`, which is what happens when the process is not listening. The `[emerg]` line in the log comes first. The probe is a symptom. 4. **Is it the config?** Read the ConfigMap the pod mounts at `/etc/nginx/conf.d` (r1, step 8): ```shell kubectl get configmap frontend-nginx -n shop -o yaml ``` ```yaml apiVersion: v1 data: default.conf: | upstream api { server api:8080; } server { listen 80; location /healthz { return 200 "ok\n"; } location /api/ { proxy_pass http://api/; proxy_conect_timeout 2s; } location / { return 200 "shop frontend\n"; } } kind: ConfigMap ``` Line 9 of `default.conf` is `proxy_conect_timeout 2s;`. The directive nginx knows is [`proxy_connect_timeout`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout). One missing letter. The `last-applied-configuration` annotation on the ConfigMap carries the same typo, so the bad value was applied, not patched by hand. ## What is the fix? Correct the directive and restart the pods that mount the ConfigMap. The lab's acceptable fixes are to fix the typo and restart the frontend, or to restore the previous ConfigMap. ```shell kubectl -n shop get cm frontend-nginx -o json \ | sed 's/proxy_conect_timeout/proxy_connect_timeout/' \ | kubectl apply -f - kubectl -n shop rollout restart deploy/frontend ``` Risk: low. The edit changes one token in one file. The restart replaces pods one at a time under the Deployment's `RollingUpdate` strategy (`maxSurge: 25%`, `maxUnavailable: 25%`, r1 step 9), so a still-broken config stops the rollout again instead of taking the site down. Rollback: `kubectl apply` the ConfigMap from source control, then restart again. Do not skip the restart. A mounted ConfigMap is only read when nginx starts, and the crash-looping pod rereads it on each restart, but the healthy old pods do not. Restarting the Deployment is what gets every pod onto the corrected file. Validate the file before you apply it if you can: `nginx -t` against the config catches an unknown directive in a second. ## How do you verify it worked? Pass condition: every frontend pod is Ready, the restart count stops climbing, and the new pods' logs show nginx starting instead of `[emerg]`. ```shell kubectl -n shop rollout status deploy/frontend kubectl -n shop get pods -l app=frontend kubectl -n shop logs -l app=frontend --tail=5 ``` The healthy pods in the lab logged this at startup (frontend pod log, captured before the investigation, run r1): ``` 2026/10/02 16:30:07 [notice] 1#1: using the "epoll" event method 2026/10/02 16:30:07 [notice] 1#1: nginx/1.27.5 ``` The lab's own check for the broken state is one line, and its negation is the pass condition: ```shell kubectl -n shop get pods -l app=frontend -o jsonpath='{.items[*].status.containerStatuses[0].state.waiting.reason}' | grep -Eq 'CrashLoopBackOff|Error' ``` It exits 0 while any frontend pod is crash-looping and non-zero once none is. Watch for two minutes: a pod that was mid-backoff needs one more restart to pick up the fix. ## How do you prevent it? **Test the config in CI.** Run `nginx -t -c ` in the same image the pods use, against the rendered ConfigMap, before the manifest is applied. It fails on exactly the line that would have crashed production. **Alert on the restart loop, not only on readiness.** With kube-state-metrics, a container waiting in CrashLoopBackOff is visible as `kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} > 0`, which fires minutes before the old pods are gone. Scope it to the namespace and container you care about. (Suggested query, not captured in the lab.) **Guardrail.** Keep the ConfigMap in source control and apply it from there; review a config change like code. In the lab the fault was a one-character edit followed by a restart, which is also how most real nginx outages start. ## Common mistakes - **Reading the current container's log.** It is empty or shows only the entrypoint lines. The error is in the previous container: `kubectl logs --previous`. - **Fixing the readiness probe.** The events say `Readiness probe failed: ... connect: connection refused`. The probe is right; nothing is listening because nginx exited. - **Raising the memory limit.** The exit code is 1, not 137. The exit code is 1, not 137, so the 64Mi limit is not what killed it. - **Correcting the ConfigMap and waiting.** The old, healthy pods keep the config they started with until they restart. Restart the Deployment to roll everyone onto the fixed file, and to confirm the fix works before the next unplanned restart does it for you. - **Rolling back the image.** Both ReplicaSets run the same image. Only the mounted file changed. ## What we saw in the lab Scenario 05 in the [Aurora lab](https://github.com/Arvo-AI/aurora-lab): a five-component `shop` app on a three-node kind cluster, the `frontend-nginx` ConfigMap edited so `proxy_connect_timeout` reads `proxy_conect_timeout`, the frontend restarted to pick it up, then Aurora given only the alert text ("KubePodCrashLooping: a frontend pod in namespace shop is restarting after a config change.") and read-only kubectl. | Run | Verdict (human reviewed) | Time | Tool calls | Aurora cost | |---|---|---|---|---| | r1 | correct | 120 s | 13 | $0.27 | | r2 | correct | 152 s | 16 | $0.36 | Both runs found both required facts: nginx fails to start with `[emerg] unknown directive`, and the bad directive is `proxy_conect_timeout` in the `frontend-nginx` ConfigMap. Neither asserted the wrong answers the scenario lists (a probe misconfiguration, resource limits). r1 went pods, previous log, describe pod, ConfigMap, Deployment, a refused `rollout history`, events, nodes: eight kubectl calls, and the answer was in hand by step 8. r2 went pods, events, previous log, describe pod, ConfigMap, a refused `rollout history`, ReplicaSets, Deployment, then the Service endpoints, which is how it could state that two old pods were still serving traffic. The difference worth knowing about is in the fixes. r2 suggested correcting the typo and restarting the rollout, the lab's first acceptable fix. r1 suggested patching the ConfigMap and said no restart was needed; the judge accepted the fix but noted that a restart may still be required, which this runbook agrees with. Both runs tried `kubectl rollout history deployment/frontend` and were refused with `Command blocked by organization policy: Mutating kubectl operations`; the default policy matched the whole `rollout` verb, a false positive fixed on the lab's fix branch. Both worked around it by comparing the ReplicaSets and the `restartedAt` annotation. Both listed what they could not check instead of guessing: the rollout history and who made the edit. ## How Aurora handles this Aurora receives the alert, connects to the cluster read-only through an uploaded kubeconfig, and runs the same kubectl commands shown above, choosing them itself. It reads the previous container log, opens the mounted ConfigMap, matches the `[emerg]` line to the directive and the line number, and writes a report that cites each tool output by number and lists what it could not check. Mutating commands are blocked by policy, so the fix arrives as a suggestion for a person to apply. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) Related: [how to fix OOMKilled](/runbooks/fix-oomkilled-kubernetes), [how to fix CrashLoopBackOff](/blog/fix-crashloopbackoff-kubernetes), [a ConfigMap change that reached only some pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes), [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), and what an [Aurora root cause analysis](/rca) looks like. ## Sources and verification - Every command and output block above comes from the lab run `v0-run2`, scenario `05-nginx-config-error`, on 2026-10-02: Aurora's tool-call transcripts (`steps-full.json`) for r1 and r2, and the cluster state captured before the investigation (`evidence-before/`). The fault was injected by the scenario's `break.sh` (a `sed` on the ConfigMap followed by `kubectl rollout restart deploy/frontend`); the fix command above is its inverse. - Versions, from `evidence-before/versions.yaml` and the pod logs: Kubernetes server v1.37.0 on kind v0.33.0 (three nodes, containerd 2.3.4), kubectl client v1.35.3; nginx 1.27-alpine, reporting nginx/1.27.5. - Aurora commit `87d22baf`, single agent, RCA model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge model `us.anthropic.claude-opus-5-5`, reviewed by a person at Arvo (`human_verdict` in each `grade.json`). - nginx documentation linked: [ngx_http_proxy_module, proxy_connect_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout), read 2026-10-15. - The fix command was not run inside a graded lab run; Aurora's policy blocks mutations. The Verify section states the pass condition and the lab's own check instead of a captured result. --- ## Liveness probe failed: wrong port, healthy app restarted (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-liveness-probe-failed-kubernetes Published: 2026-10-15 Author: Noah Casarotto-Dinning, CEO at Arvo AI > Liveness probe failed with connection refused while the app serves traffic: the probe points at a port nothing listens on. Confirm, fix the port, verify. | Applies to | Kubernetes containers with an HTTP or TCP liveness probe; the lab app is a Python HTTP server behind a Deployment | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, python:3.12-alpine, 2026-10-02 | | Lab folder | [scenarios/06-liveness-wrong-port](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/06-liveness-wrong-port), runs in [runs/v0-run2/06-liveness-wrong-port](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/06-liveness-wrong-port) | **A liveness probe that fails with connection refused while the app is serving requests means the probe is checking a port, path or scheme the app does not expose, and the kubelet is killing a healthy container on a timer.** The pod shows `Running`, the restart count climbs, the last state reads `Error` with exit code 137, and the readiness probe keeps passing between kills. Confirm with `kubectl describe pod -n `: a `Liveness probe failed` warning next to a `Killing` event that says `failed liveness probe, will be restarted` is the whole diagnosis. The fix is to point the probe at the port the app listens on, not to remove the probe. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | `Liveness probe failed: ... connect: connection refused` on a port different from the container port, readiness passing | Probe port, path or scheme does not match what the app exposes | `kubectl get deploy -o yaml`: compare `livenessProbe.httpGet.port` with `containerPort` and the app's own log line | Set the probe to the port the app listens on | | `Liveness probe failed` with an HTTP 5xx, app logs show errors | The app really is unhealthy | `kubectl logs --previous` shows the failure | Fix the app, not the probe | | Probe failures only in the first seconds after start, then clean | `initialDelaySeconds` or `failureThreshold` too tight for the app's startup | Compare `Started` and the first `Unhealthy` event age in `kubectl describe pod` | Raise the delay or add a startup probe | | `Liveness probe failed: ... context deadline exceeded`, `kubectl top` shows the container at its CPU request | The app cannot answer within `timeoutSeconds` under load | `kubectl top pods`, `kubectl describe node` | Raise `timeoutSeconds` or the CPU request | ## How do you confirm the liveness probe is the thing restarting the container? Start from the pod list. A liveness kill looks like a crash loop from a distance: `Running`, `1/1`, and a restart count that keeps moving. In the lab, run r1, step 4: ``` $ kubectl get pods -n shop -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-c7b987b4d-9knph 1/1 Running 1 (62s ago) 2m12s 10.244.2.14 aurora-lab-3-worker2 api-c7b987b4d-c897z 1/1 Running 1 (63s ago) 2m14s 10.244.1.13 aurora-lab-3-worker frontend-76d98597b9-5x7wj 1/1 Running 0 2m21s 10.244.2.12 aurora-lab-3-worker2 frontend-76d98597b9-x8l47 1/1 Running 0 2m21s 10.244.1.11 aurora-lab-3-worker ``` Both api pods are `Ready` and both have restarted. That combination, ready but restarting, already says the kubelet is killing something that works. Read the events on one of them (run r1, step 10, the end of `kubectl describe pod api-c7b987b4d-c897z -n shop`): ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m58s default-scheduler Successfully assigned shop/api-c7b987b4d-c897z to aurora-lab-3-worker Normal Killing 67s (x2 over 2m17s) kubelet spec.containers{api}: Container api failed liveness probe, will be restarted Normal Pulled 37s (x3 over 2m57s) kubelet spec.containers{api}: Container image "python:3.12-alpine" already present on machine and can be accessed by the pod Normal Created 37s (x3 over 2m57s) kubelet spec.containers{api}: Container created Normal Started 37s (x3 over 2m57s) kubelet spec.containers{api}: Container started Warning Unhealthy 37s (x3 over 2m57s) kubelet spec.containers{api}: Readiness probe failed: Get "http://10.244.1.13:8080/healthz": dial tcp 10.244.1.13:8080: connect: connection refused Warning Unhealthy 7s (x8 over 2m37s) kubelet spec.containers{api}: Liveness probe failed: Get "http://10.244.1.13:8081/healthz": dial tcp 10.244.1.13:8081: connect: connection refused ``` Two lines matter. `Killing ... Container api failed liveness probe, will be restarted` names the kubelet as the one stopping the container. `Liveness probe failed: Get "http://10.244.1.13:8081/healthz"` names the port it is probing: 8081. The readiness failures above it are different: they hit port 8080 once per start, in the second before the server is up, and then stop. The liveness failures keep coming on the 10 second period, eight times so far. The same describe output shows the container state (run r1, step 10): ``` State: Running Started: Fri, 02 Oct 2026 16:32:59 +0000 Last State: Terminated Reason: Error Exit Code: 137 Started: Fri, 02 Oct 2026 16:31:49 +0000 Finished: Fri, 02 Oct 2026 16:32:59 +0000 Ready: True Restart Count: 2 Limits: memory: 128Mi Requests: cpu: 50m memory: 32Mi Liveness: http-get http://:8081/healthz delay=15s timeout=1s period=10s successThreshold=1 failureThreshold=3 Readiness: http-get http://:8080/healthz delay=0s timeout=1s period=5s successThreshold=1 failureThreshold=3 ``` Exit code 137 is a SIGKILL. With `Reason: Error` rather than `OOMKilled`, and the `Killing` event above, the signal came from the kubelet's probe handling, not from the kernel's memory killer. The two probe lines sit next to each other: liveness on 8081, readiness on 8080. The probe decision comes about 45 seconds after start: 15 seconds of initial delay, then three failed probes 10 seconds apart. The container is gone about 70 seconds after start (Started 16:31:49, Finished 16:32:59 in the output above); the difference is the 30 second `terminationGracePeriodSeconds` between the kubelet's SIGTERM and its SIGKILL. The Kubernetes [pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) page states the rule: "If a container fails its liveness probe more times than the configured tolerance, the kubelet restarts that container." ## How much is affected and since when? Check whether the restarts follow a change. The Deployment carries the answer in its annotations (run r1, step 6, `kubectl get deployment api -n shop -o yaml`, trimmed to the lines that differ): ``` metadata: annotations: deployment.kubernetes.io/revision: "2" kubectl.kubernetes.io/last-applied-configuration: | {... "livenessProbe":{"httpGet":{"path":"/healthz","port":8080},"initialDelaySeconds":15,"periodSeconds":10} ...} spec: template: spec: containers: - image: python:3.12-alpine livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 8081 scheme: HTTP initialDelaySeconds: 15 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: api ports: - containerPort: 8080 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 8080 scheme: HTTP ``` Revision 2, and the last applied configuration still says port 8080 while the live spec says 8081. Something changed the probe outside the manifest that was applied. The scope is every pod of this Deployment, from the moment the new ReplicaSet rolled out. In the lab the events (run r1, evidence captured before the investigation) show the old ReplicaSet `api-f64486c4c` scaled to 0 and `api-c7b987b4d` scaled to 2 about two minutes before the first `Killing`. `kubectl rollout history deployment/api` would show the revisions, if your policy allows it. Aurora's default read-only policy blocked it in both lab runs (run r1, step 8): ``` $ kubectl rollout history deployment/api -n shop Command blocked by organization policy: Mutating kubectl operations ``` ## Which of the four causes is it? Cheapest and likeliest check first. **1. Does the probe target match what the app exposes?** Compare three numbers: the probe port, `containerPort`, and the port in the app's own startup line. In the lab all three are visible without touching the app. The Deployment above says probe 8081 and `containerPort: 8080`. The previous container's log (run r1, step 7) says what the process actually bound: ``` $ kubectl logs api-c7b987b4d-c897z -n shop --previous --tail=100 connected to redis at redis:6379 api listening on :8080 10.244.1.1 "GET /healthz HTTP/1.1" 200 - 10.244.1.11 "GET /items HTTP/1.0" 200 - 10.244.2.12 "GET /items HTTP/1.0" 200 - 10.244.1.11 "GET /items HTTP/1.0" 200 - 10.244.1.1 "GET /healthz HTTP/1.1" 200 - ``` The killed container was listening on 8080, answering `/healthz` with 200 and serving `/items` right up to the kill. The probe asked 8081. That is the cause, and the next three checks are how you rule out the alternatives in a case that is less clear. **2. Is the app actually unhealthy?** A real liveness failure comes with an HTTP 5xx or a timeout, and the logs show the app in trouble. Here the probe error is `connection refused`, which means nothing accepted the TCP connection on that port. An app that is up on 8080 cannot refuse on 8080. Run r2 read both pods at once (step 7): ``` $ kubectl logs -n shop -l app=api --all-containers --tail=100 --prefix [pod/api-c7b987b4d-f98rz/api] connected to redis at redis:6379 [pod/api-c7b987b4d-f98rz/api] api listening on :8080 [pod/api-c7b987b4d-f98rz/api] 10.244.2.1 "GET /healthz HTTP/1.1" 200 - [pod/api-c7b987b4d-mbfs7/api] connected to redis at redis:6379 [pod/api-c7b987b4d-mbfs7/api] api listening on :8080 [pod/api-c7b987b4d-mbfs7/api] 10.244.1.1 "GET /healthz HTTP/1.1" 200 - ``` **3. Is it a startup race?** If the only probe failures are in the first seconds after `Started` and the restart count stays at 0, the probe is firing before the app is up. Here `initialDelaySeconds: 15` is generous for a server that logs `listening` within a second, the failures continue every 10 seconds for the life of the container, and the restart count climbs. Not a startup race. **4. Is it resource pressure?** A starved container can miss `timeoutSeconds: 1`. Memory and CPU for the api pods were tiny (run r1, step 11): ``` $ kubectl top pods -n shop NAME CPU(cores) MEMORY(bytes) api-c7b987b4d-9knph 2m 11Mi api-c7b987b4d-c897z 5m 14Mi frontend-76d98597b9-5x7wj 1m 7Mi frontend-76d98597b9-x8l47 1m 7Mi loadgen-7c6676864-xxv7f 4m 0Mi redis-59458f55d8-rkw9m 4m 4Mi worker-6cf88d4b58-kdmbc 3m 3Mi ``` 14Mi against a 128Mi limit and 5m of CPU. The probe failures are also `connection refused`, not `context deadline exceeded`, which is what a slow answer produces. ## How do you fix it? **Cause 1, probe target wrong.** Point the probe at the port the app listens on. In the lab that is one patch: ``` kubectl -n shop patch deploy api --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/port","value":8080}]' ``` Risk: the patch triggers a rolling update. With two replicas, `maxUnavailable: 25%` rounds down to 0 and `maxSurge: 25%` rounds up to 1, so a new pod is created and must be Ready before an old one is removed. The service stays up for that reason. Rollback: the same patch with the previous value, or `kubectl rollout undo deploy/api -n shop` to return to the prior revision. Mitigation if you cannot change the spec right now: nothing safe. Deleting the probe stops the restarts but leaves the Deployment without a liveness check, and the scenario's own rubric lists that as a wrong answer. If the manifest is managed by GitOps or a Helm chart, fix it at the source too, or the next sync puts 8081 back. The `last-applied-configuration` annotation in the lab shows the applied manifest had 8080, so the live edit was the drift. **Cause 2, app unhealthy.** Fix the app. Do not loosen the probe to hide it. **Cause 3, startup race.** Add a `startupProbe` with a long `failureThreshold`, or raise `initialDelaySeconds`. Rollback is the previous values. **Cause 4, resource pressure.** Raise `timeoutSeconds` to a value the app meets under load, or raise the CPU request. Rollback is the previous values. ## How do you verify the fix? Pass condition: restart counts stop increasing and no new `Liveness probe failed` event appears for at least three probe periods plus the initial delay, here 45 seconds, and the probe line in `describe pod` shows the corrected port. ``` kubectl -n shop get pods -l app=api -w kubectl -n shop describe pod -l app=api | grep -E 'Liveness|Restart Count' kubectl -n shop get events --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail -5 ``` The lab's own check for the broken state is two conditions, a `Liveness probe failed` event and a non-zero restart count on an api pod (`scenarios/06-liveness-wrong-port/check.sh`): ``` kubectl -n shop get events --field-selector reason=Unhealthy -o jsonpath='{.items[*].message}' | grep -q 'Liveness probe failed' && kubectl -n shop get pods -l app=api -o jsonpath='{.items[*].status.containerStatuses[0].restartCount}' | grep -Eq '[1-9]' ``` After the fix, new pods start at 0 restarts and stay there. Watch for five minutes; the restart interval before the fix was 70 seconds, so five minutes without a restart is well past it. ## How do you prevent it? **Alert on restarts, not on CrashLoopBackOff.** In the lab the pods showed `Running` with a restart count of 1 and then 2 while the kills were already happening, so an alert on the status misses the first restarts. With kube-state-metrics, alert on the restart counter moving: ``` increase(kube_pod_container_status_restarts_total[15m]) > 2 ``` This is a suggested expression, not one captured in the lab; check the metric name against your kube-state-metrics version. **Guardrail: one source of truth for the port.** Name the container port and reference it by name in both probes, so a port change is one edit: ``` ports: - name: http containerPort: 8080 livenessProbe: httpGet: path: /healthz port: http readinessProbe: httpGet: path: /healthz port: http ``` **Test:** in CI, apply the manifest to a kind cluster and fail the pipeline if any pod restarts within two minutes of becoming Ready. The two conditions in the lab's `check.sh` above, negated, are that test. ## Common mistakes - **Reading the restart as a crash.** Exit code 137 plus `Reason: Error` plus a `Killing ... failed liveness probe` event is the kubelet, not the application. Only `Reason: OOMKilled` is the memory killer. See [OOMKilled](/runbooks/fix-oomkilled-kubernetes) for that case and [CrashLoopBackOff](/blog/fix-crashloopbackoff-kubernetes) for a container that exits on its own. - **Deleting the liveness probe to stop the restarts.** The restarts stop and so does the protection the probe existed for. Fix the target. - **Trusting `kubectl logs` without `--previous`.** The current container is a fresh one that has not been killed yet. The evidence is in the one that was. - **Treating the readiness failures as the problem.** In this lab both probes show `Unhealthy` events. The readiness ones happen once per start and stop; the liveness ones repeat. Count them. - **Fixing the live object only.** If a manifest or chart still says the wrong port, the next apply reintroduces it. The `last-applied-configuration` annotation tells you which side drifted. ## What we saw in the lab Two runs on 2026-10-02, Aurora commit 87d22baf, model bedrock/us.anthropic.claude-sonnet-5, single agent, read-only kubectl under the default `observability_only` policy, a fresh user and org per run. Aurora got the alert text only: "KubePodCrashLooping: api pods in namespace shop restart every minute or so." | Run | Verdict (human reviewed) | Time | Tool calls | Cost | Fix proposed | |---|---|---|---|---|---| | r1 | correct | 123 s | 14 | $0.35 | patch the liveness probe port from 8081 to 8080 (acceptable) | | r2 | correct | 87 s | 12 | $0.28 | patch the liveness probe port to 8080 (acceptable) | Both runs named both required findings: the kubelet restarts the container because the liveness probe fails, and the probe targets 8081 while the app listens on 8080. Neither asserted the two wrong answers the scenario lists, OOM or an application crash; both ruled them out explicitly from `top pods` and the logs. Neither proposed removing the probe. r1's path: list open incidents, search memory (empty), list clusters, `get pods -o wide`, `get events --sort-by=.lastTimestamp`, `get deployment api -o yaml` (where it saw 8081 against the annotation's 8080), `logs --previous` (the app listening on 8080 and serving 200s until the kill), `rollout history` (blocked), `get pods -l app=api` (restart count now 2), `describe pod` (the `Killing` and `Liveness probe failed` events), `top pods`, the `api-code` ConfigMap (to confirm the code binds `PORT`, default 8080), `get nodes -o wide`. Its report called the kills "kubelet-initiated SIGKILL from failed liveness probe" and listed the probe port change as the root cause. r2 took a shorter route: pods, events, the Deployment YAML, then logs for both pods with `--prefix` (step 7, above), `rollout history` (blocked again), `describe pod -l app=api`, the ConfigMap, `top pods`, nodes. 12 calls, 87 seconds. Two things to note. `kubectl rollout history` is read-only; Aurora's default policy denies the whole `rollout` verb, so in both runs the "what changed" question was answered from the annotation diff instead. The policy fix is described in the [lab report](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md). And neither report stated a remediation in the report body; both fixes were in Aurora's separate suggestions list, which is where Aurora puts them by design. ## How Aurora handles this Aurora receives the alert, lists the namespace's pods and events, reads the Deployment and the previous container's logs, and separates a probe kill from a crash by the `Killing` event, the exit code and the serving logs, with each claim cited to a tool output. It runs under a read-only kubectl policy by default, so it investigates without changing anything. In both lab runs it reached this root cause in under two and a half minutes for less than 40 cents. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) ## Sources and verification Every output block above comes from the lab runs in [runs/v0-run2/06-liveness-wrong-port](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/06-liveness-wrong-port) (`r1/steps-full.json`, `r2/steps-full.json`, `r1/evidence-before/`) and the scenario definition in [scenarios/06-liveness-wrong-port](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/06-liveness-wrong-port). The fault was injected with the `break.sh` patch that sets the probe port to 8081; the fix above is the same patch with 8080. Versions: Kubernetes server v1.37.0, kind v0.33.0, python:3.12-alpine, kubectl client v1.35.3. The kubelet rule is quoted from the Kubernetes pod lifecycle page. Related: [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes). Verified 2026-10-02, written 2026-10-15. --- ## Name does not resolve: a wrong Service hostname crash-loops the app (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-dns-name-not-resolving-kubernetes Published: 2026-10-15 Author: Ben Gervais, Engineer at Arvo AI > The app exits with Name does not resolve because an env var names a Service that does not exist. Confirm it, rule out CoreDNS, fix the name. Lab-tested. | Applies to | Any pod that reaches a dependency by a Kubernetes Service name set in an env var or config; here a Python api calling redis | |---|---| | Verified on | Kubernetes server v1.37.0 (kind v0.33.0, containerd 2.3.4), 2 October 2026 | | Lab folder | [scenarios/10-wrong-redis-host](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/10-wrong-redis-host), runs in [runs/v0-run2/10-wrong-redis-host](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/10-wrong-redis-host) | **"Name does not resolve" in a crash-looping container means the hostname the app was given has no DNS record inside the cluster, almost always because an env var or config names a Service that does not exist, while the real Service has a different name.** The dependency itself is usually healthy. Confirm with `kubectl get svc -n ` next to the hostname in the pod's `Environment` block: if the name is not in the list, the fix is the name, not DNS and not the dependency. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | One Deployment's new pods log `Name does not resolve` or `Name or service not known`, old pods fine | An env var or config points at a Service name that does not exist | `kubectl describe pod ` Environment block against `kubectl get svc -n ` | Set the hostname back to the real Service name, or roll the Deployment back | | Every pod that resolves a Service name fails at once | CoreDNS is down or misconfigured | `kubectl -n kube-system get pods -l k8s-app=kube-dns` and a lookup of `kubernetes.default` from any pod | Fix CoreDNS; see the Corefile and its logs | | Name resolves, connection times out | A NetworkPolicy blocks the client, or the backend listens on another port | `kubectl get networkpolicy -n ` and `kubectl get endpoints -n ` | Allow the traffic or fix the port | | Name resolves, `Connection refused` | The dependency is down or not listening yet | `kubectl get pods -l app= -n ` and its logs | Fix the dependency | ## How do you confirm the name is the problem? Start with the pod list. The shape to recognise is one new pod crash-looping while the rest of the Deployment keeps running (run r1, step 4): ``` $ kubectl get pods -n shop -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-869d87b484-9n7bc 0/1 CrashLoopBackOff 5 (73s ago) 4m54s 10.244.1.18 aurora-lab-2-worker api-f64486c4c-hb8mm 1/1 Running 0 5m2s 10.244.1.15 aurora-lab-2-worker api-f64486c4c-n4zt7 1/1 Running 0 5m2s 10.244.2.22 aurora-lab-2-worker2 redis-59458f55d8-bj68x 1/1 Running 0 5m2s 10.244.2.21 aurora-lab-2-worker2 ``` Then read the crashing container's log. The container has already exited, so ask for the previous one (run r2, step 6): ```shell kubectl logs -n --previous --tail=100 ``` ``` $ kubectl logs api-869d87b484-chjfx -n shop --previous --tail=100 redis connection failed (attempt 1/5): [Errno -2] Name does not resolve redis connection failed (attempt 2/5): [Errno -2] Name does not resolve redis connection failed (attempt 3/5): [Errno -2] Name does not resolve redis connection failed (attempt 4/5): [Errno -2] Name does not resolve redis connection failed (attempt 5/5): [Errno -2] Name does not resolve giving up: redis unavailable ``` `[Errno -2] Name does not resolve` is the resolver saying the name has no record (glibc prints `Name or service not known` for the same condition). `Connection refused` and `Connection timed out` are different failures with different causes: see the table above. The exit is clean, code 1, so this is not an OOM kill or a probe kill (run r1, step 7): ``` State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Fri, 02 Oct 2026 16:52:53 +0000 Finished: Fri, 02 Oct 2026 16:53:04 +0000 Ready: False Restart Count: 5 Environment: REDIS_HOST: redis-master ``` The pod ran for 11 seconds: five attempts in 11 seconds, then exit. The hostname it tried is right there in `Environment`. ## How far does it reach? Three questions: which pods, since when, after which change. **Which pods.** Only pods from the new ReplicaSet `api-869d87b484` fail. The two `api-f64486c4c` pods are `1/1 Running` with 0 restarts, and so is redis (step 4 above). Traffic is still served by the old pods, so this is a stuck rollout, not an outage. Check what the old pods were given (run r1, step 9): ``` $ kubectl get pod api-f64486c4c-hb8mm -n shop -o jsonpath='{.spec.containers[0].env}' [{"name":"REDIS_HOST","value":"redis"}] ``` **Since when.** The Deployment's status shows the rollout frozen with one updated replica that never becomes ready (run r1, step 10, status block): ``` status: availableReplicas: 2 conditions: - lastTransitionTime: "2026-10-02T16:49:15Z" lastUpdateTime: "2026-10-02T16:49:23Z" message: ReplicaSet "api-869d87b484" is progressing. reason: ReplicaSetUpdated status: "True" type: Progressing ... observedGeneration: 2 readyReplicas: 2 replicas: 3 unavailableReplicas: 1 updatedReplicas: 1 ``` **After which change.** The spec says revision 2, and the `last-applied-configuration` annotation still carries the previous value, `"env":[{"name":"REDIS_HOST","value":"redis"}]`, while the live template says `redis-master` (run r1, step 10). That gap between the annotation and the live spec is the signature of an imperative change such as `kubectl set env`, which is exactly what the lab did: ```shell kubectl -n shop set env deploy/api REDIS_HOST=redis-master ``` `kubectl rollout history deployment/api -n shop` would show the revisions too. In the lab it was blocked by Aurora's read-only command policy (runs r1 and r2, steps 11 and 12); from your own terminal it is the quickest way to see when revision 2 was created. ## How do you isolate the cause? Cheapest and likeliest first. **1. Does the Service exist under that name?** (run r1, step 8) ```shell kubectl get svc -n ``` ``` $ kubectl get svc -n shop NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE api ClusterIP 10.96.180.245 8080/TCP 5m23s frontend ClusterIP 10.96.88.61 80/TCP 5m23s redis ClusterIP 10.96.239.186 6379/TCP 5m23s ``` There is `redis`, there is no `redis-master`. A normal Service gets a DNS record of the form `my-svc.my-namespace.svc.cluster-domain.example` that "resolves to the cluster IP of the Service", and queries "that don't specify a namespace are limited to the Pod's namespace", per the [DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) page. So `redis-master` from a pod in `shop` expands to `redis-master.shop.svc.cluster.local`, which does not exist. Diagnosis done; the next two steps are for when the name is present. **2. Is cluster DNS working at all?** If the Service exists and the name still fails, test resolution from a pod in the same namespace and check CoreDNS: ```shell kubectl -n run dns-check --rm -it --restart=Never --image=busybox:1.37 -- nslookup kubectl -n kube-system get pods -l k8s-app=kube-dns kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50 ``` A broken CoreDNS fails every name, including `kubernetes.default`, from every pod. Here redis, the frontend and the worker were all running and the frontend was serving requests through the `api` Service name (frontend and loadgen logs in the captured evidence), so cluster DNS was fine. **3. Is the dependency itself healthy?** Only relevant once the name resolves. The lab's redis had a live endpoint (run r1, step 12): ``` $ kubectl get endpoints redis -n shop NAME ENDPOINTS AGE redis 10.244.2.21:6379 5m41s ``` If the client logged `Connection refused` against a resolving name, this is where the problem would be. If it logged a timeout against a resolving name with live endpoints, look at NetworkPolicies next. ## How do you fix it? **Cause: the hostname names a Service that does not exist.** Set it back to the real name. In the lab the Service is `redis`: ```shell kubectl -n shop set env deploy/api REDIS_HOST=redis ``` This creates revision 3 with the old value. Risk: low; the old pods are already running with this value. Rollback: `kubectl -n shop rollout undo deploy/api` returns to the previous revision. If the value lives in a manifest in git, change it there and apply, otherwise the next `kubectl apply` reintroduces the bad name. **Alternative: roll back the deploy.** ```shell kubectl -n shop rollout undo deploy/api ``` Same effect here because the only change in revision 2 was the env var. Risk: it also undoes anything else that shipped in that revision. **Cause: the Service really should have that name.** Create it, or rename the existing Service, and then fix every client that uses the old name. Do not leave two Services pointing at the same pods as a workaround; the next person reads it as two dependencies. Mitigation is not needed: the old pods keep serving while the rollout is stuck, which is the point of `maxUnavailable: 25%`. ## How do you verify the fix? The pass condition: the new ReplicaSet's pods reach `1/1 Running` with 0 restarts, and their log shows a connection, not a retry loop. ```shell kubectl -n shop rollout status deploy/api kubectl -n shop get pods -l app=api kubectl -n shop logs -l app=api --tail=5 ``` A healthy pod logs what the old ones logged (frontend-facing api pod in the captured evidence, `api-f64486c4c-hb8mm`): ``` connected to redis at redis:6379 api listening on :8080 ``` The lab's own check for the broken state is one command; its inverse is your pass condition: ```shell kubectl -n shop get pods -l app=api -o jsonpath='{.items[*].status.containerStatuses[0].lastState.terminated.exitCode}' | grep -q 1 ``` Watch for two minutes: the rollout should complete and the old ReplicaSet scale to 0. ## How do you prevent it? **Alert on the rollout, not only the pod.** A crash-looping pod fires `KubePodCrashLooping`; a stuck rollout is the condition that matters to users of the next version. With kube-state-metrics (a suggested expression, not captured in the lab): ``` kube_deployment_status_replicas_updated != kube_deployment_spec_replicas ``` held for longer than your rollout normally takes. **Guardrail: fail at build time, not at runtime.** Put Service names in the same manifest as the Service that defines them (Kustomize or Helm values), so a rename touches both at once. For env vars that must hold a hostname, a `kubectl set env` on a live Deployment drifts from the manifest in git; the `last-applied-configuration` gap in Scope is how you catch that after the fact. A pre-deploy check that every hostname in a pod spec matches an existing Service in the namespace is a short script against `kubectl get svc -o name`. **Make the app say the name.** The lab app prints the hostname only on success (`connected to redis at redis:6379`), not on failure. Log the host and port in the failure line too, and the diagnosis is one `kubectl logs` away. ## Common mistakes - **Restarting the pod.** It will fail again in 11 seconds. Nothing about a restart changes a name. - **Blaming CoreDNS.** One Deployment failing while seven other pods resolve names fine is not a DNS outage. Check whether the name exists before touching kube-system. - **Fixing the dependency.** redis was `1/1 Running` with a live endpoint the whole time. `Name does not resolve` is a client-side message about the name, not a server-side one. - **Patching the running pod's env.** Pods are immutable; the fix is on the Deployment, and it rolls out a new ReplicaSet. - **Trusting `kubectl logs` without `--previous`.** The current container may be mid-restart with an empty log; the finished one has the evidence. ## What we saw in the lab Scenario `10-wrong-redis-host`: `kubectl -n shop set env deploy/api REDIS_HOST=redis-master` on a healthy app, then the alert "KubePodCrashLooping: an api pod in namespace shop is restarting after the latest deploy." Aurora got the alert text and read-only kubectl, nothing else. Two independent runs on 2 October 2026 (Aurora commit 87d22baf, model bedrock/us.anthropic.claude-sonnet-5), each in a fresh org. | Run | Verdict | Time | Tool calls | Cost | Fix proposed | |---|---|---|---|---|---| | r1 | correct (judge and human) | 292 s | 12 | $0.23 | revert `REDIS_HOST` to `redis` on `deployment/api` | | r2 | correct (judge and human) | 197 s | 16 | $0.34 | set `REDIS_HOST` back to `redis`, or `rollout undo` | Both runs met both required findings: the api exits because it cannot resolve `redis-master`, and `REDIS_HOST` is `redis-master` while the Service is `redis`. Neither asserted the two wrong answers the scenario lists, redis being down or cluster DNS being broken. r1's path, in order: list open incidents, search memory (empty), list clusters, `get pods -o wide`, `get events --sort-by=.lastTimestamp`, `logs --tail=100` (the five failed attempts above), `describe pod`, `get svc` (no `redis-master`), the old pod's env (`redis`), the Deployment YAML (revision 2, `redis-master` in the live template), `rollout history` (blocked), `get endpoints redis` (live). Its report: "No Kubernetes Service named `redis-master` exists in the `shop` namespace" and names the real one, `redis` at 10.96.239.186, with a live endpoint. r2 started the same way, `get pods` then `get events`, and went to `logs --previous` at step 6, then `describe pod`, the Deployment YAML, `get svc`, `get rs -o wide`, the old pod's env, `rollout history` (blocked again), `get nodes`, the api-code ConfigMap (the jsonpath returned nothing, so it read no code), and `top pods`. Its report: "pods from the new ReplicaSet fail DNS resolution for the Redis host" and "the old ReplicaSet's 2 pods, still configured with the correct REDIS_HOST=redis, remain Running and continue serving traffic". What the runs got wrong or left open: - **`rollout history` was denied in both runs.** Aurora's default policy denies the whole `rollout` verb, so the read-only `history` and `status` went with the mutating `restart` and `undo`. Both runs reconstructed the change from the ReplicaSets and the annotation instead. The lab's fix branch narrows the deny rule; on that branch scenario 10 ran 2/2 again and `rollout history` was allowed (see the [lab report](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md)). - **Neither run directly checked the redis pod or CoreDNS.** r2 inferred redis was healthy from the old api pods serving traffic; r1 checked the endpoint. The judge accepted both, but a reader who wants the ruling-out spelled out will not find a `get endpoints redis` in r2. - **Small inconsistencies in r1's report:** the Deployment timestamp appears as 16:49:15Z in one place and 16:49:23Z in another (creation vs the new ReplicaSet), and "3 desired replicas" reflects the surge pod, not `spec.replicas: 2`. ## How Aurora handles this Aurora receives the alert, lists the pods and events, reads the previous container's log, compares the new pod template with the running pods and the Service list, and writes the root cause with each claim cited to a tool output, as in the two runs above. It runs under a read-only command policy by default, so it reports the fix and leaves the `set env` or `rollout undo` to you. Where the policy blocks a read-only command, it says so under "Not checked" rather than guessing. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) Related runbooks: [a Service with no endpoints](/runbooks/fix-service-no-endpoints-kubernetes) (the name resolves, nothing answers), [OOMKilled](/runbooks/fix-oomkilled-kubernetes) (a crash loop with exit code 137 instead of 1), and the [root cause analysis guide](/blog/root-cause-analysis-complete-guide-sres), and [a name that fails only because dnsPolicy None set the wrong search domains](/runbooks/fix-pod-dnsconfig-search-domain-not-resolving-kubernetes). ## Sources and verification - Lab scenario, fault and check: [scenarios/10-wrong-redis-host](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/10-wrong-redis-host) (`scenario.yaml`, `break.sh`, `check.sh`). - Captured outputs: [runs/v0-run2/10-wrong-redis-host](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/10-wrong-redis-host), r1 and r2: `steps-full.json` (every tool call), `evidence-before/` (cluster state and pod logs before the investigation), `aurora-rca.md`, `grade.json` with the human verdict. - Versions: Kubernetes server v1.37.0, kubectl v1.35.3, kind v0.33.0, containerd 2.3.4 (`evidence-before/versions.yaml` and `get nodes -o wide` in r2 step 13). Verified 2 October 2026. - Kubernetes documentation: [DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/) for the Service record form and namespace scoping, quoted above. - The kube-state-metrics expression under Prevent is a suggested query, not a captured one. --- ## Service has no endpoints: the selector matches no pods, upstream returns 502 (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-service-no-endpoints-kubernetes Published: 2026-10-08 Author: Ben Gervais, Engineer at Arvo AI > A Service with no endpoints has a selector that matches no pods, so its ClusterIP refuses connections and the proxy returns 502. Confirm it and fix it. | Applies to | Kubernetes Services of type ClusterIP whose selector matches no pods; any upstream that proxies to the Service, here nginx | |---|---| | Verified on | Kubernetes server v1.37.0, kind v0.33.0, nginx 1.27-alpine, 2026-10-02 | | Lab folder | [scenarios/07-service-selector-mismatch](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/07-service-selector-mismatch), runs in [runs/v0-run2/07-service-selector-mismatch](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/07-service-selector-mismatch) | **A Service with no endpoints means its selector matches none of the pods, so every connection to its ClusterIP is refused and whatever proxies to it answers 502.** The pods behind it are usually healthy, which is why restarts change nothing. Confirm with `kubectl get endpoints -n `: an `ENDPOINTS` column reading `` is the whole diagnosis. The fix is to make the selector match the pod labels again, nothing else. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | `endpoints/` shows ``, pods are Running and Ready | Service selector does not match the pod labels | `kubectl get svc -o yaml` against `kubectl get pods --show-labels` | Patch the selector back to the pod label | | Endpoints list the pod IPs, upstream still refuses | `targetPort` is not the port the container listens on | `kubectl get svc -o yaml` against the container port in the pod spec | Set `targetPort` to the container port | | Endpoints empty, pods `0/1` or `Running` but not Ready | Readiness probe failing, so the pods are left out of the endpoints | `kubectl describe pod ` for `Readiness probe failed` | Fix the probe or the app it probes | | Endpoints populated, connections time out instead of being refused | A NetworkPolicy blocks the client | `kubectl get networkpolicy -n ` | Allow the traffic in the policy | ## How do you confirm the Service has no endpoints? Start from the error the client sees, then read the endpoints. In the lab the frontend nginx proxied `/api/` to the `api` Service and logged this (frontend pod log, captured before the investigation, run r1): ``` 2026/10/02 16:35:46 [error] 25#25: *30 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.1.14, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.89.40:8080/items", host: "frontend" 10.244.1.14 - - [02/Oct/2026:16:35:46 +0000] "GET /api/items HTTP/1.1" 502 157 "-" "Wget" "-" ``` `Connection refused` against a ClusterIP, with no SYN timing out, points at the Service layer rather than the network. Read the endpoints next: ```shell kubectl get endpoints api -n shop ``` Aurora's run r1, step 15: ``` NAME ENDPOINTS AGE api 2m43s ``` The newer object says the same thing (r1, step 17): ```shell kubectl get endpointslices -n shop -l kubernetes.io/service-name=api -o yaml ``` ```yaml - addressType: IPv4 apiVersion: discovery.k8s.io/v1 endpoints: null kind: EndpointSlice metadata: labels: endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io kubernetes.io/service-name: api name: api-qb4rq namespace: shop ports: null ``` The [debug services](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/) guide covers this check: an empty ENDPOINTS column means the Service selector matched no pods. Look-alikes: a Service whose endpoints are populated but whose upstream still refuses has a `targetPort` problem, not a selector problem. A Service with `0/1` pods behind it has a readiness problem. Both show endpoints that are not ``, or pods that are not Ready, and this runbook is not the one for them. ## How much is affected, and since when? Every client of the Service is affected, because the ClusterIP has nothing to forward to. Check what else is healthy so the fix stays narrow. Aurora's first command in both runs (r1, step 4): ```shell kubectl get pods -n shop -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES api-f64486c4c-4rwgd 1/1 Running 0 79s 10.244.1.12 aurora-lab-2-worker api-f64486c4c-56fw6 1/1 Running 0 79s 10.244.2.18 aurora-lab-2-worker2 frontend-76d98597b9-ml7r6 1/1 Running 0 79s 10.244.2.20 aurora-lab-2-worker2 frontend-76d98597b9-rpcht 1/1 Running 0 79s 10.244.1.13 aurora-lab-2-worker loadgen-7c6676864-xdk98 1/1 Running 0 79s 10.244.1.14 aurora-lab-2-worker redis-59458f55d8-h9p8h 1/1 Running 0 79s 10.244.2.17 aurora-lab-2-worker2 worker-6cf88d4b58-xgqwj 1/1 Running 0 79s 10.244.2.19 aurora-lab-2-worker2 ``` Every pod is `1/1 Running` with zero restarts. That rules out the application and leaves the routing between the frontend and `api`. For "since when", the client log gives the first failure (loadgen pod log, captured before the investigation, r1): requests to `/api/items` returned 502 from `16:35:06Z` onward, every two seconds. Compare that with the Service's `resourceVersion` and any recent `kubectl apply` or `patch` in your change log; the Service's own events will usually be empty, as they were here (r1, step 20 returned only the Deployment's `ScalingReplicaSet` event). ## Which of the four causes is it? Cheapest and likeliest first. Each step ends when its output answers the question. **1. Compare the selector with the pod labels.** Read the Service (r1, step 16): ```shell kubectl get svc api -n shop -o yaml ``` ```yaml metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"api","namespace":"shop"},"spec":{"ports":[{"port":8080,"targetPort":8080}],"selector":{"app":"api"}}} spec: clusterIP: 10.96.89.40 ports: - port: 8080 protocol: TCP targetPort: 8080 selector: app: api-v2 type: ClusterIP ``` Then the pods (run r2, step 8): ```shell kubectl get pods -n shop -l app=api --show-labels ``` ``` NAME READY STATUS RESTARTS AGE LABELS api-f64486c4c-7k66j 1/1 Running 0 85s app=api,pod-template-hash=f64486c4c api-f64486c4c-bhgfg 1/1 Running 0 85s app=api,pod-template-hash=f64486c4c ``` The live selector is `app: api-v2`; the pods carry `app=api`. The `last-applied-configuration` annotation still holds the intended selector, `app: api`, which tells you the change was made outside the manifest. Stop here if the labels differ: this is the cause. If you suspect the selector is deliberate, check whether anything carries the new label (r1, step 19 and r2, step 14 both returned nothing): ```shell kubectl get pods -A -l app=api-v2 ``` **2. Check `targetPort` against the container port.** Only if the selector matches. In the Service above, `targetPort: 8080`; in `kubectl describe pod api-f64486c4c-4rwgd -n shop` (r1, step 14) the container exposes `8080/TCP`. A mismatch here produces populated endpoints and a refused connection on the wrong port. **3. Check readiness.** Only if the selector matches and endpoints are still empty. `kubectl describe pod ` shows `Ready: True` for both api pods in the captured state. A pod that is not Ready is excluded from the endpoints even when its labels match. **4. Check NetworkPolicy.** Only if endpoints are populated and the symptom is a timeout rather than a refusal. `kubectl get networkpolicy -n shop` lists any policy in the namespace. ## What is the fix? Put the selector back to the label the pods actually carry. In the lab the fault was injected with one command, so the fix is its inverse: ```shell kubectl -n shop patch svc api -p '{"spec":{"selector":{"app":"api"}}}' ``` Risk: low. Changing a selector only changes which pods receive traffic; it does not restart anything. The risk is choosing a label that matches more pods than you intend, so read `--show-labels` first. Rollback: patch the previous value back, or `kubectl apply` the manifest from source control, which is what the `last-applied-configuration` annotation preserved here. Mitigation is not needed for this failure. The pods are already up; the moment the selector matches, the endpoints controller adds them. Do not restart pods, roll the Deployment, or scale it, none of which touches the Service. If the selector change was intended (a cutover to `api-v2`), the fix is on the other side: label the new pods, or deploy the workload that carries the label, and leave the Service alone. ## How do you verify it worked? Pass condition: the endpoints list the pod IPs, and the upstream stops returning 502. ```shell kubectl get endpoints api -n shop kubectl get endpointslices -n shop -l kubernetes.io/service-name=api -o jsonpath='{.items[*].endpoints[*].addresses[*]}' ``` Both must show the api pod IPs (the api pod IPs seen in run r1 were `10.244.1.12` and `10.244.2.18`; the fix was not run in a graded lab run). The EndpointSlice also lists pods that are not ready, so an IP there does not prove traffic reaches it; `get endpoints` lists only ready ones. In our [sidecar OOMKilled](/runbooks/fix-sidecar-container-oomkilled-kubernetes) run the slice still held both pod IPs while the Service had no endpoints. Then watch the client for a minute: ```shell kubectl logs -n shop -l app=frontend --tail=20 -f ``` `GET /api/items` lines should return `200`, with no new `connect() failed` entries. The lab's own check for the broken state is one line, and its negation is the pass condition: ```shell [ -z "$(kubectl -n shop get endpointslices -l kubernetes.io/service-name=api -o jsonpath='{.items[*].endpoints[*].addresses[*]}')" ] ``` It exits 0 while the Service has no endpoints and non-zero once it does. ## How do you prevent it? **Alert on Services with no endpoints.** With kube-state-metrics, a Service that has a selector but no ready endpoint addresses can be alerted on by listing every endpoints object and removing the ones that still have a ready address: ``` kube_endpoint_info unless on (namespace, endpoint) kube_endpoint_address{ready="true"} ``` A sum of `kube_endpoint_address` compared with zero looks simpler but never fires: kube-state-metrics emits one series per address, so a Service with no addresses has no series to sum. Scope it to Services that are supposed to have backends, and fire it before the error-rate alert does, because the empty endpoint list precedes the first 502. **Guardrail.** Keep Services in source control and apply them from there. The lab's fault was an out-of-band `kubectl patch`; the `last-applied-configuration` annotation showing a different selector than the live spec is the signature of that. A policy engine (Gatekeeper or Kyverno) can require that a Service selector matches at least one existing pod template label in the namespace at admission. **Test.** In CI, after deploying to a test namespace, assert that every Service with a selector has at least one endpoint address before running smoke tests. ## Common mistakes - **Restarting the pods.** Both api pods were healthy. A rollout restart recreates them with the same labels, and the Service still matches none of them. - **Fixing the app or its dependency.** Aurora's r1 investigation noticed `redis connection failed (attempt 1/5)` in the api logs; it was a startup race followed by `connected to redis at redis:6379`, and unrelated. The 502 is produced before a request reaches application code. - **Reading `kubectl get svc` and stopping.** The default columns do not show the selector or the endpoints. `-o wide` adds the selector; `get endpoints` or `get endpointslices` shows what it matched. - **Trusting the manifest in git.** The annotation and the live spec can disagree, as they did here. The live `spec.selector` is what the controller uses. - **Changing the pod labels to match the Service.** That changes the Deployment's own selector match as well, which can orphan pods. Fix the Service. ## What we saw in the lab Scenario 07 in the [Aurora lab](https://github.com/Arvo-AI/aurora-lab): a five-component `shop` app on a three-node kind cluster, the `api` Service patched to `selector: app=api-v2`, then Aurora given only the alert text ("HighErrorRate: frontend /api/ requests in namespace shop are failing with 5xx; all pods are Running") and read-only kubectl under its default `observability_only` policy. Two runs, each in a fresh organization so nothing carried over. | Run | Verdict (human reviewed) | Time | Tool calls | Aurora cost | |---|---|---|---|---| | r1 | correct | 175 s | 22 | $0.52 | | r2 | correct | 138 s | 15 | $0.40 | Both runs reached the root cause and proposed the acceptable fix (patch the selector to `app=api`). The paths differed. r2 was direct: pods, events, then `kubectl get svc,ep -n shop` at step 6, which showed `endpoints/api ` next to populated endpoints for `frontend` and `redis`, then the Service YAML with `app: api-v2`, then `--show-labels` on the pods. Eleven kubectl calls and it had everything, including a search for `app=api-v2` across all namespaces to rule out a deliberate cutover. r1 took the longer road: it read the frontend, api and loadgen logs first, found the 502s and the `Connection refused` lines, and only at step 10 ran `get endpoints api -o yaml`. The Endpoints object had no `subsets` at all, which it then confirmed with `get endpoints` and the EndpointSlice at steps 15 and 17. It also ran `kubectl top pods` and `get nodes` to rule out resource and node problems, which the captured state shows were never in question. The miss worth knowing about is in r1's written report. Aurora had read `selector: app: api-v2` at step 16, and its final chat message and suggested fix both named it. The root-cause paragraph of the report did not: it said the selector "does not currently align with the running pods" and left `api-v2` to a "Ruled Out" line about an intentional cutover. The judge marked the run correct on the full output; a second model reading only the report called it partial for that reason, and a person at Arvo reviewed both and recorded `correct`. The investigation was right. The report under-stated what it knew. Neither run was tempted by the wrong answers the scenario lists (restarting pods, an api application failure, NetworkPolicy or DNS). Neither could run `kubectl rollout history`, which the default policy blocked in 16 of 20 runs in this series as a false positive on the `rollout` verb; it did not matter here because the Service, not a Deployment, had changed. ## How Aurora handles this Aurora receives the alert, connects to the cluster read-only through an uploaded kubeconfig, and runs the same kubectl commands shown above, choosing them itself. It records every tool call and output, cites them by number in the report, and lists what it could not check. Mutating commands are blocked by the organization's command policy by default, so it proposes the fix rather than applying it. On this failure it found the empty endpoint list and the selector mismatch in both runs, in about two to three minutes, for under a dollar each. The full traces for both runs are in the lab repository. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) Related: [how to fix OOMKilled](/runbooks/fix-oomkilled-kubernetes), [how to fix ImagePullBackOff](/runbooks/fix-imagepullbackoff-kubernetes), [root cause analysis for SREs](/blog/root-cause-analysis-complete-guide-sres), [a Service whose targetPort does not match the container](/runbooks/fix-service-targetport-mismatch-kubernetes), [a ConfigMap change that reached only some pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes), [a Service selecting pods from the wrong Deployment](/runbooks/fix-service-selects-wrong-pods-label-overlap-kubernetes), [a second fault the alert did not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes), and what an [Aurora root cause analysis](/rca) looks like. ## Sources and verification - Every command and output block above comes from the lab run `v0-run2`, scenario `07-service-selector-mismatch`, on 2026-10-02: Aurora's tool-call transcripts (`steps-full.json`) for r1 and r2, and the cluster state captured before the investigation (`evidence-before/`). The fault was injected with `scenarios/07-service-selector-mismatch/break.sh`. - Versions, from `evidence-before/versions.yaml`: Kubernetes server v1.37.0 on kind v0.33.0 (three nodes, containerd 2.3.4), kubectl client v1.35.3; nginx 1.27-alpine as the frontend. - Aurora commit `87d22baf`, single agent, RCA model `bedrock/us.anthropic.claude-sonnet-5`. Verdicts: judge model `us.anthropic.claude-opus-5-5`, reviewed by a person at Arvo (`human_verdict` in each `grade.json`). - Kubernetes documentation quoted: [Debug Services](https://kubernetes.io/docs/tasks/debug/debug-application/debug-service/) and [Service](https://kubernetes.io/docs/concepts/services-networking/service/), read 2026-10-08. - The fix command was not run inside a graded lab run; Aurora's policy blocks mutations. It is the inverse of the injected fault, and the Verify section states the pass condition instead of a captured result. --- ## OOMKilled (exit code 137): container exceeded its memory limit (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-oomkilled-kubernetes Published: 2026-09-28 (updated 2026-10-08) Author: Noah Casarotto-Dinning, CEO at Arvo AI > OOMKilled means the kernel killed a container for exceeding its memory limit. Confirm it, find the change behind it, fix and verify, with real lab output. | Applies to | Kubernetes pods with a container memory limit, any runtime that reports `OOMKilled` | |---|---| | Verified on | Kubernetes v1.37.0 (server), kind v0.33.0, 2026-10-02 | | Lab folder | [scenarios/01-worker-oom](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/01-worker-oom), runs in [runs/v0-run2/01-worker-oom](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/01-worker-oom) | **OOMKilled with exit code 137 means the kernel killed the container because it used more memory than its `resources.limits.memory`.** The most likely cause is a recent change that raised the working set, not a leak and not node pressure. Confirm it with `kubectl describe pod ` and look for `Reason: OOMKilled` under `Last State`. Then either undo the change that grew the working set or raise the limit with the reason written down. ## At a glance | Signal | Likely cause | Confirm with | Fix | |---|---|---|---| | `Last State: Terminated, Reason: OOMKilled, Exit Code: 137`, restarts climbing | A recent config or code change raised the working set above the limit | `kubectl get deployment -o yaml`, compare the live spec with `last-applied-configuration` | Revert the change, or raise the limit with the reason recorded | | Same signature, memory climbs between restarts | Allocation without a bound (a leak) | Watch `kubectl top pod` over several minutes, or compare `--previous` logs | Fix the allocation; a higher limit only delays the next kill ([leak runbook](/runbooks/fix-memory-leak-oomkilled-kubernetes)) | | `OOMKilled` on a container with no limit set | The node killed it under pressure | `kubectl describe nodes`, look for `OOMKilling` | Set a limit, or expect the node to pick it again | | `Pending`, `FailedScheduling`, `Insufficient memory` | Not OOMKilled at all: the request is larger than any node | `kubectl describe pod ` | Lower the request or add capacity | ## Confirm: is it really OOMKilled? Read the container state, not the pod phase. In the lab the pod showed `Running` as its phase while the only container inside it was being killed every few seconds. From run r1, step 6 of Aurora's investigation: ```shell kubectl describe pod worker-86f68b9bf6-hgxwz -n shop ``` ``` Status: Running ... State: Terminated Reason: OOMKilled Exit Code: 137 Started: Fri, 02 Oct 2026 16:26:32 +0000 Finished: Fri, 02 Oct 2026 16:26:32 +0000 Last State: Terminated Reason: OOMKilled Exit Code: 137 Started: Fri, 02 Oct 2026 16:25:50 +0000 Finished: Fri, 02 Oct 2026 16:25:50 +0000 Ready: False Restart Count: 4 Limits: memory: 192Mi Requests: cpu: 50m memory: 64Mi Environment: BATCH_SIZE: 2500 ``` `Started` and `Finished` carry the same second: the process died within a second of starting, which is what a batch allocation that exceeds the limit looks like. The events at the bottom of the same output show the kubelet doing its job, nothing more: ``` Normal Started 3s (x5 over 84s) kubelet spec.containers{worker}: Container started Warning BackOff 2s (x4 over 82s) kubelet spec.containers{worker}: Back-off restarting failed container worker in pod worker-86f68b9bf6-hgxwz_shop(...) ``` The Kubernetes [memory resources page](https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/) describes exactly this: "If a Container allocates more memory than its limit, the Container becomes a candidate for termination. If the Container continues to consume memory beyond its limit, the Container is terminated. If a terminated Container can be restarted, the kubelet restarts it, as with any other type of runtime failure." That restart loop is why the same pod shows `CrashLoopBackOff` a minute later, as it did in run r2: ``` NAME READY STATUS RESTARTS AGE worker-86f68b9bf6-thj65 0/1 CrashLoopBackOff 4 (73s ago) 2m56s ``` Look-alikes to rule out before you touch anything: - `CrashLoopBackOff` with `Reason: Error` and a non-137 exit code is an application crash. Steps for the backoff itself are in [how to fix CrashLoopBackOff](/blog/fix-crashloopbackoff-kubernetes). - `Pending` with `FailedScheduling` and `Insufficient memory` was never started. Scheduling uses the request, not the limit. - `Exit Code: 137` with no `OOMKilled` reason can be a SIGKILL from a failed liveness probe or an eviction. Check the events. ## Scope: how much is affected, since when, after which change? Three commands answer all three questions. First, who else is hurt. Run r1, step 4: ```shell kubectl get pods -n shop -o wide ``` ``` NAME READY STATUS RESTARTS AGE api-f64486c4c-dxmf8 1/1 Running 0 75s api-f64486c4c-w5pvs 1/1 Running 0 75s frontend-76d98597b9-8xkmw 1/1 Running 0 75s frontend-76d98597b9-jnq4f 1/1 Running 0 75s loadgen-7c6676864-g44sc 1/1 Running 0 75s redis-59458f55d8-6ft5c 1/1 Running 0 75s worker-86f68b9bf6-hgxwz 0/1 OOMKilled 3 (50s ago) 67s ``` One workload, and it has a single replica, so whatever the worker does is not happening at all. Second, is the node under pressure. Run r2, step 11: ```shell kubectl top nodes ``` ``` NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%) aurora-lab-control-plane 96m 0% 727Mi 3% aurora-lab-worker 21m 0% 159Mi 0% aurora-lab-worker2 32m 0% 198Mi 0% ``` Nodes at 0 to 3 percent memory rule out node pressure and eviction. The kill came from the container's own limit. Third, since when. The ReplicaSets tell you a rollout happened, and the pod's age tells you the crashes started with it. Run r2, step 12: ```shell kubectl get replicaset -n shop -l app=worker ``` ``` NAME DESIRED CURRENT READY AGE worker-6cf88d4b58 0 0 0 3m46s worker-86f68b9bf6 1 1 0 3m39s ``` A new ReplicaSet seven seconds after the old one, with zero ready pods, is a change that broke the workload on arrival. `kubectl rollout history` would name the revision, if your policy allows it (see the lab section below). ## Isolate: which change grew the working set? Start with the cheapest check that distinguishes the two common causes: a step change in configuration, or a slow climb. The previous container's own log is the first thing to read. Run r2, step 6: ```shell kubectl logs worker-86f68b9bf6-thj65 -n shop --tail=100 --previous ``` ``` worker started, batch size 2500 ``` One line, then death. There is no climb to watch; the process allocates its batch at startup and exceeds the limit immediately. If `--previous` returns `unable to retrieve container logs` (run r1 got `unable to retrieve container logs` at step 9; the killed container was no longer available to the runtime), go straight to the spec. Compare what is running with what was last applied. Run r1, step 7: ```shell kubectl get deployment worker -n shop -o yaml ``` ``` metadata: annotations: deployment.kubernetes.io/revision: "2" kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apps/v1","kind":"Deployment",...,"env":[{"name":"BATCH_SIZE","value":"500"}],...} generation: 2 ... containers: - command: - python - -u - /code/worker.py env: - name: BATCH_SIZE value: "2500" image: python:3.12-alpine resources: limits: memory: 192Mi requests: cpu: 50m memory: 64Mi ``` The live spec says `BATCH_SIZE=2500`. The `last-applied-configuration` annotation, which `kubectl apply` writes, still says `500`. Something changed the Deployment outside the tracked manifest (in the lab, `kubectl set env deploy/worker BATCH_SIZE=2500`). That annotation diff is the single most useful artifact in an OOMKilled investigation, because it hands you both the cause and the rollback in one read. Then confirm the arithmetic against the code, so you know whether 2500 is a bug or a new requirement. Run r1, step 10, from the `api-code` ConfigMap: ```python batch = int(os.environ.get("BATCH_SIZE", "500")) print(f"worker started, batch size {batch}", flush=True) while True: # Each queued item holds about 100 KiB while the batch is processed. buf = [bytearray(100 * 1024) for _ in range(batch)] ``` 2500 items at 100 KiB each is about 244 MiB, against a 192Mi limit. 500 items is about 49 MiB. The limit was never the problem; the batch size was. When the annotation diff is empty, the change came through a normal apply and lives in version control, or there was no change and you are looking at a leak (see [OOMKilled from a slow memory leak](/runbooks/fix-memory-leak-oomkilled-kubernetes)). For a leak, `kubectl top pod` sampled over minutes shows memory climbing toward the limit between restarts, and the previous log shows the process running for a while before it dies. ## Fix: revert the change or raise the limit? Revert when the change was unintended or untested, which is the usual case. Undo the drifted variable with the same verb that caused it: ```shell kubectl -n shop set env deploy/worker BATCH_SIZE=500 ``` Risk: this is a rollout, so the pod restarts once more. For a single-replica worker that is already down, there is nothing left to lose. Rollback: set the value back to 2500, which puts you where you started. If the Deployment is managed by `kubectl apply` or a GitOps tool, re-applying the tracked manifest does the same thing and also removes the drift. Raise the limit only when the larger working set is a real requirement and the process is stable under it. For 2500 items you need headroom above 244 MiB, so 320Mi is the smallest round number that leaves a margin. Edit `resources.limits.memory` in the tracked manifest and apply it, so the annotation and the live spec agree. Risk: a higher limit on a node without the room moves the problem to the scheduler or to the node OOM killer. Rollback: re-apply the previous manifest. Do not combine both changes at once. If you revert the batch size and raise the limit together, you will not know which one stopped the kills, and the next person to raise `BATCH_SIZE` learns nothing from the limit. ## Verify: what counts as fixed? The container's last termination reason stops being `OOMKilled` and the restart count stops moving. The lab's own check for "still broken" is a one-liner you can invert: ```shell kubectl -n shop get pods -l app=worker -o jsonpath='{.items[*].status.containerStatuses[0].lastState.terminated.reason}' ``` While broken it prints `OOMKilled`. After the fix the new pod has no terminated last state, so it prints nothing. Then watch for two batch cycles, in this app ten seconds, and confirm the pod is `1/1 Running` with `RESTARTS` at 0: `containerStatuses[0]` reads only the first container. If the pod has a sidecar, check every container: see [sidecar container OOMKilled](/runbooks/fix-sidecar-container-oomkilled-kubernetes). ```shell kubectl -n shop get pods -l app=worker -w ``` If the restart count moves again at the new setting, the working set is still above the limit, and you are now looking at a second cause. ## Prevent: how do you catch the next one before the pager does? Alert on the termination reason, not on restarts. With kube-state-metrics, `kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}` is 1 for a container whose last termination was an OOM kill ([kube-state-metrics pod metrics](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md)). An alert on that series firing for five minutes names the container and the reason in one line, instead of a generic `KubePodCrashLooping`. Guardrails: - Treat a non-empty diff between the live spec and `kubectl.kubernetes.io/last-applied-configuration` as a finding. Drift is how this incident happened. GitOps tools surface it as "OutOfSync"; without one, a scheduled `kubectl diff` against the repo does the same. - Put the memory math next to the knob. A comment on `BATCH_SIZE` that says "100 KiB per item, limit 192Mi, keep under 1800" would have stopped this change in review. - Give the limit a reason in the manifest. A limit nobody can explain gets raised blindly the next time this fires. ## Common mistakes - **Raising the limit first.** It makes the kills stop and hides a leak or an unreviewed change. In this case the change was the bug; 320Mi would have "fixed" a worker that then needs 400Mi next quarter. - **Reading the pod phase.** `Status: Running` was true in run r1 while the container was dying every second. Read `containerStatuses`. - **Treating restarts as evidence.** A restart count climbs for probe failures and application crashes too. Only `Reason: OOMKilled` with `Exit Code: 137` is an OOM kill. - **Blaming the node.** `kubectl top nodes` at 0 to 3 percent memory settles it in one command. Eviction and node pressure leave `Evicted` pods and node events, not `OOMKilled` containers. - **Trusting `kubectl logs` without `--previous`.** The current container may have one line or nothing. The killed one had the evidence, and in run r1 it was already gone. ## What we saw in the lab The scenario is [01-worker-oom](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/01-worker-oom): a healthy `shop` namespace on a 3-node kind cluster, then `kubectl -n shop set env deploy/worker BATCH_SIZE=2500` against a 192Mi limit. Aurora got only the alert text, `KubePodCrashLooping: pod worker in namespace shop is restarting repeatedly`, a kubeconfig, and read-only kubectl under its default `observability_only` policy. Each run used a fresh user and org. Model: `bedrock/us.anthropic.claude-sonnet-5`, Aurora commit `87d22baf`, single agent. | Run | Verdict | Time | Tool calls | Cost | |---|---|---|---|---| | r1 | correct (judge, second model and human review agree) | 91 s | 12 | $0.25 | | r2 | correct (same) | 228 s | 14 | $0.35 | Both runs met both required findings: the container is OOMKilled rather than crashing for another reason, and the trigger is the `BATCH_SIZE` change from 500 to 2500. Neither asserted the two wrong answers the scenario lists, a memory leak or node pressure. Both suggested the acceptable fix, reverting `BATCH_SIZE` to 500, and r1 also offered raising the limit to 320Mi with the reason stated. r1's path, in order: list open incidents, search memory (empty), list clusters, `get pods -o wide`, `get events --sort-by=.lastTimestamp`, `describe pod` (the OOMKilled state above), `get deployment -o yaml` (the annotation diff), `rollout history` (blocked, see below), `logs --previous` (unavailable, garbage collected), `get configmap api-code -o yaml` (the 100 KiB per item line), `top pods`, then it wrote a memory entry titled "shop/worker OOMKilled from BATCH_SIZE env drift". Its report put the cause as "a drifted environment variable on the worker deployment: the live spec sets BATCH_SIZE=2500, while the kubectl.kubernetes.io/last-applied-configuration annotation ... shows BATCH_SIZE=500, indicating an out-of-band change". r2 took a different route to the same answer: after `get pods` and `get events` it went to `logs --previous` (step 6), which worked this time and returned `worker started, batch size 2500`, then `describe pod`, the ConfigMap, `rollout history` (blocked again), the Deployment YAML, `top nodes` and the ReplicaSets. Its root cause states the arithmetic: "BATCH_SIZE=2500 requires ~244MiB while the container's memory limit is 192Mi". Two things went wrong, and both are worth knowing about: - **`kubectl rollout history` was denied in both runs** with "Command blocked by organization policy: Mutating kubectl operations". The command is read-only; Aurora's default deny rule matched the whole `rollout` verb. Aurora worked around it by diffing the ReplicaSets and the annotation, and both reports listed the blocked call under "Not checked" rather than guessing at the history. The policy has since been narrowed on a lab branch to deny only `rollout restart|undo|pause|resume`; details are in the [lab report](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md). - **Neither report stated a remediation.** The judge flagged both. The fixes were in Aurora's separate "Suggested Next Steps" list, which the first version of the grader never read: both runs proposed reverting `BATCH_SIZE` to 500 and both offered raising the limit to 320Mi with the reason stated. Graded on everything Aurora showed the user, both runs are correct and their fixes acceptable. Both reports also carried an honest "Not checked" section: the blocked rollout history, the unavailable previous log in r1, and the absence of prior incidents in memory. Neither claimed the author or time of the change, which no command they were allowed to run could reveal. ## How Aurora handles this Aurora receives the alert and investigates on its own with read-only `kubectl` behind an allowlist, so by default it cannot change the cluster while it looks. For OOMKilled it reads the container state, the events, the Deployment spec against its `last-applied-configuration` annotation, the code or config that sets the working set, and node and pod metrics, then writes a root cause with a numbered citation for each claim and a separate list of suggested fixes. Anything it could not verify goes in a "Not checked" section instead of the root cause. The full traces for both lab runs are in the lab repository. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) Related: [ImagePullBackOff runbook](/runbooks/fix-imagepullbackoff-kubernetes), [a rollout stuck on an init container](/runbooks/fix-pod-stuck-init-container-kubernetes), [root cause analysis guide for SREs](/blog/root-cause-analysis-complete-guide-sres), and [an OOMKilled worker alongside a second, unalerted fault](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). ## Sources and verification - Every command output on this page is pasted from Aurora's recorded tool calls in [runs/v0-run2/01-worker-oom](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/01-worker-oom) (files `steps-full.json`, `aurora-rca.md`, `grade.json`, `result.json`) or from the cluster state captured before the investigation (`evidence-before/`), trimmed with `...` where noted and otherwise unchanged. Run date 2026-10-02. - Cluster: kind v0.33.0, Kubernetes server v1.37.0, from `evidence-before/versions.yaml`. Aurora commit `87d22baf`, model `bedrock/us.anthropic.claude-sonnet-5`. - The termination rule, the kubelet restart sentence and the `OOMKilled` / `137` signature are quoted from the Kubernetes page [Assign Memory Resources to Containers and Pods](https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/), checked 2026-10-08. - `kube_pod_container_status_last_terminated_reason` is documented in the [kube-state-metrics pod metrics reference](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/workload/pod-metrics.md), checked 2026-10-08. - Aurora's default command policy and the fix branch are described in the lab's [REPORT.md](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md) and [FINDINGS.md](https://github.com/Arvo-AI/aurora-lab/blob/main/FINDINGS.md). ![Diagram of an OOMKilled diagnosis. The symptom is a container terminated with reason OOMKilled and exit code 137, because a container is not allowed to use more memory than its limit and the kubelet restarts it. Confirm with kubectl describe pod, scope with kubectl top nodes, isolate by comparing the live spec with last-applied-configuration, then revert the change or raise the limit with a reason.](/runbooks/oomkilled-diagnosis.svg) --- ## ImagePullBackOff: the image could not be pulled and Kubernetes is backing off (Kubernetes) URL: https://www.aurorasre.ai/runbooks/fix-imagepullbackoff-kubernetes Published: 2026-09-28 (updated 2026-10-08) Author: Damian Loch, Software Engineer at Arvo AI > ImagePullBackOff means the kubelet could not pull the image and is retrying, up to 5 minutes apart. Read the event, fix the reference, verify. Lab-tested. | Applies to | Kubernetes Deployments and Pods on any cluster; the registry error text is from containerd | |---|---| | Verified on | Kubernetes v1.37.0 (kind v0.33.0, containerd 2.3.4), 2 October 2026 | | Lab folder | [scenarios/03-bad-image-tag](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/03-bad-image-tag), runs in [runs/v0-run2/03-bad-image-tag](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/03-bad-image-tag) | **ImagePullBackOff means the kubelet could not pull the container image and is now waiting longer between attempts, up to five minutes.** The most common cause is an image reference that does not exist: a mistyped tag, a tag that was never pushed, or the wrong registry. `kubectl describe pod ` shows the exact pull error in the events. Fix the reference (or roll the Deployment back) and the pod leaves Waiting on the next pull. ## At a glance | Signal in the events | Likely cause | Confirm with | Fix | |---|---|---|---| | `failed to resolve reference "...": not found` | The tag or repository does not exist (typo, never pushed) | Pull it yourself, or query the registry for that tag | Correct the image field, or `kubectl rollout undo` | | `pull access denied`, `unauthorized`, `authentication required` | Private registry with no or wrong `imagePullSecrets` | `kubectl get secret -n ` for the referenced secret, same namespace and type | Create the secret in the pod's namespace, reference it in the pod spec | | `dial tcp`, `i/o timeout`, `no such host` | The node cannot reach the registry | Pull from a node, check egress and DNS | Fix network egress or the registry hostname | | `ErrImageNeverPull` | `imagePullPolicy: Never` and the image is not on the node | `kubectl get pod -o jsonpath='{.spec.containers[*].imagePullPolicy}'` | Pre-load the image or change the policy | The [container images](https://kubernetes.io/docs/concepts/containers/images/#imagepullbackoff) documentation defines the status: "The status `ImagePullBackOff` means that a container could not start because Kubernetes could not pull a container image (for reasons such as invalid image name, or pulling from a private registry without `imagePullSecret`)." It also fixes the retry cap: "Kubernetes raises the delay between each attempt until it reaches a compiled-in limit, which is 300 seconds (5 minutes)." Waiting does not fix a pull. ## Confirm: is it ImagePullBackOff, and which pods? Start with the Deployment, then the pods. A Deployment that is `READY 2/2` but `UP-TO-DATE 1` is mid-rollout and stuck, which is the shape this runbook reproduces. Output from the lab, run r1: ```shell kubectl get deployment frontend -n shop -o wide ``` ``` NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR frontend 2/2 1 2 78s nginx nginx:1.27-alpne app=frontend ``` The image column already shows the problem if you know the right tag. Most people do not spot `alpne` on the first read. List the pods: ```shell kubectl get pods -n shop -o wide ``` ``` NAME READY STATUS RESTARTS AGE IP NODE frontend-76d98597b9-jkkw9 1/1 Running 0 82s 10.244.2.9 aurora-lab-3-worker2 frontend-76d98597b9-rb9z6 1/1 Running 0 82s 10.244.1.7 aurora-lab-3-worker frontend-7b6c998456-phg6r 0/1 ImagePullBackOff 0 75s 10.244.1.9 aurora-lab-3-worker ``` Two pods from the old ReplicaSet (`76d98597b9`) are serving. One pod from the new ReplicaSet (`7b6c998456`) never started. `RESTARTS 0` is a useful tell: the container has not run once, so this is not a crash and `kubectl logs` has nothing to show. Look-alikes: `CrashLoopBackOff` has a restart count climbing and logs to read, see [how to fix OOMKilled](/runbooks/fix-oomkilled-kubernetes) for the memory variant. `Pending` with `FailedScheduling` means the pod has no node yet; here the pod was scheduled (`PodScheduled True`) and only the pull failed. ## Scope: is anything down? Describe the Deployment and read the replica line and the two ReplicaSets: ```shell kubectl describe deployment frontend -n shop ``` ``` Annotations: deployment.kubernetes.io/revision: 2 Replicas: 2 desired | 1 updated | 3 total | 2 available | 1 unavailable StrategyType: RollingUpdate RollingUpdateStrategy: 25% max unavailable, 25% max surge Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True ReplicaSetUpdated OldReplicaSets: frontend-76d98597b9 (2/2 replicas created) NewReplicaSet: frontend-7b6c998456 (1/1 replicas created) ``` `2 available` of `2 desired` means users are not affected: the RollingUpdate strategy created one surge pod (25% of 2, rounded up) and will not remove an old pod until a new one is Ready. The old ReplicaSet keeps serving. This is a stuck rollout, not an outage, and that changes the urgency. If `available` were below `desired`, it would be an outage and the rollback in the Fix section comes first. The Deployment carries `progressDeadlineSeconds: 600` (the lab manifest, and the Kubernetes default). The [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#failed-deployment) documentation says that once that deadline passes the controller sets the `Progressing` condition to `False` with reason `ProgressDeadlineExceeded`. Until then the condition reads `ReplicaSetUpdated`, as above, so do not wait for it to tell you the rollout is stuck. ## Isolate: why can the kubelet not pull? Describe the waiting pod. The events hold the full error from the container runtime. Run r1, trimmed to the container state and events: ```shell kubectl describe pod frontend-7b6c998456-phg6r -n shop ``` ``` Status: Pending Controlled By: ReplicaSet/frontend-7b6c998456 Containers: nginx: Image: nginx:1.27-alpne State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Conditions: PodScheduled True Events: Type Reason Age From Message Normal Scheduled 85s default-scheduler Successfully assigned shop/frontend-7b6c998456-phg6r to aurora-lab-3-worker Normal BackOff 15s (x4 over 83s) kubelet spec.containers{nginx}: Back-off pulling image "nginx:1.27-alpne" Warning Failed 15s (x4 over 83s) kubelet spec.containers{nginx}: Error: ImagePullBackOff Normal Pulling 1s (x4 over 84s) kubelet spec.containers{nginx}: Pulling image "nginx:1.27-alpne" Warning Failed 0s (x4 over 84s) kubelet spec.containers{nginx}: Failed to pull image "nginx:1.27-alpne": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.27-alpne": failed to resolve reference "docker.io/library/nginx:1.27-alpne": docker.io/library/nginx:1.27-alpne: not found Warning Failed 0s (x4 over 84s) kubelet spec.containers{nginx}: Error: ErrImagePull ``` Read the last `Failed` line word by word. Three things are in it: 1. `docker.io/library/nginx:1.27-alpne` is the fully resolved reference. The pod spec said `nginx:1.27-alpne`; the runtime filled in the registry and the `library/` namespace, as the images documentation describes: "If you don't specify a registry hostname, Kubernetes assumes that you mean the Docker public registry." 2. `code = NotFound` and `not found` mean the registry answered and said the reference does not exist. This rules out credentials (that would be `unauthorized` or `denied`) and network (that would be a timeout or a DNS error). The decision is made here. 3. `x4 over 84s` is the backoff doing its job. The count will keep growing until the reference changes. Then find what the working pods run. Compare the ReplicaSets, cheapest check first. Run r1: ```shell kubectl get replicaset -n shop -l app=frontend -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[0].image}{" replicas="}{.spec.replicas}{" ready="}{.status.readyReplicas}{" "}{end}' ``` ``` frontend-76d98597b9 nginx:1.27-alpine replicas=2 ready=2 frontend-7b6c998456 nginx:1.27-alpne replicas=1 ready= ``` The diff is one letter. If the two tags look identical, check the ReplicaSet revisions directly; run r2 read both ReplicaSets as YAML and the relevant fields were: ```shell kubectl get rs -n shop -l app=frontend -o yaml ``` ``` deployment.kubernetes.io/revision: "1" name: frontend-76d98597b9 - image: nginx:1.27-alpine readyReplicas: 2 deployment.kubernetes.io/revision: "2" name: frontend-7b6c998456 - image: nginx:1.27-alpne replicas: 1 ``` If you can, confirm the correct tag exists before you change anything. Aurora did this in run r1 against the Docker Hub API and got a 200 for `1.27-alpine`: ```shell curl -s -o /dev/null -w "%{http_code} " https://registry.hub.docker.com/v2/repositories/library/nginx/tags/1.27-alpine/ ``` ``` 200 ``` For a private registry, `docker pull ` or `crane manifest ` from a machine with the same credentials answers the same question. If a pull from your laptop succeeds and the node's events still say `not found`, the node is pulling from a different registry (a mirror, or a different `imagePullSecrets` host) and that is the next thing to check. The other branches of the table are quick to rule out: `kubectl get pod -o jsonpath='{.spec.imagePullSecrets}'` shows whether a secret is referenced at all, and `kubectl get secret -n ` shows whether it exists in the pod's namespace, which the documentation requires: "All `imagePullSecrets` must be Secrets that exist in the same Namespace as the Pod." ## Fix: correct the reference or roll back Two acceptable fixes. Pick by how sure you are of the right tag. **Set the correct image.** Starts a new rollout (revision 3) with the right reference: ```shell kubectl -n shop set image deploy/frontend nginx=nginx:1.27-alpine ``` Risk: if the new tag is also wrong, you get the same stuck rollout one revision later and nothing else changes, because the old pods keep serving. Rollback: `kubectl -n shop rollout undo deploy/frontend`. **Roll back to the last working revision.** Use this when you do not know the intended tag, or when the Deployment is an outage rather than a stuck rollout: ```shell kubectl -n shop rollout undo deploy/frontend ``` Risk: you lose whatever else was in the new revision (env, resources, config changes bundled with the image bump). Rollback of the rollback is `kubectl rollout undo --to-revision=`. Mitigation is not needed here because the old pods are serving. If `available` had dropped below `desired`, `rollout undo` is both the mitigation and the fix. Deleting the waiting pod does nothing: the ReplicaSet recreates it with the same bad spec. Do not disable the readiness probe or raise `maxUnavailable` to make the rollout "progress"; that trades a stuck rollout for serving pods that were never pulled. ## Verify: the rollout completes ```shell kubectl -n shop rollout status deploy/frontend --timeout=120s kubectl -n shop get rs -l app=frontend ``` Pass condition: `rollout status` returns `deployment "frontend" successfully rolled out`, the Deployment shows `READY 2/2`, `UP-TO-DATE 2`, `AVAILABLE 2`, the ReplicaSet with the bad tag is scaled to 0, and no pod in the namespace has `ImagePullBackOff` or `ErrImagePull` in its waiting reason. Watch for two minutes: a `Pulling` event followed by `Pulled` and `Started` on the new pods is the proof, and the backoff counter stops growing. ## Prevent: catch a bad reference before it reaches the cluster **Alert.** With kube-state-metrics, these two expressions cover the pod and the Deployment (not tested in the lab, where the alert was a text string sent to Aurora): ``` kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull"} > 0 kube_deployment_status_replicas_updated < kube_deployment_spec_replicas ``` The second one watches the Deployment rather than the pod, so a stuck rollout with zero updated replicas still fires. **Guardrail.** Pin images by digest (`nginx@sha256:...`) so a tag typo cannot resolve at all, or have CI verify every image reference in the manifest exists before `kubectl apply`: ```shell crane manifest nginx:1.27-alpine > /dev/null ``` A non-zero exit fails the pipeline. `docker manifest inspect ` does the same without extra tooling. **Test.** A pre-merge policy check (Kyverno or OPA Gatekeeper) that rejects `:latest` and untagged images removes the other half of this class of failure. The images documentation recommends it: avoid `:latest` "as it is harder to track which version of the image is running and more difficult to roll back properly." ## Common mistakes - **Treating it as a registry auth problem.** `not found` is the registry saying the reference does not exist. Adding an `imagePullSecret` changes nothing. Read the error code before you touch secrets. - **Waiting it out.** The backoff caps at five minutes and never recovers on its own. Nothing changes until the spec changes. - **Deleting the pod.** The ReplicaSet recreates it with the same image. The restart count stays at 0 and the events start over. - **Calling it an outage.** Check `available` against `desired` first. In the lab the old pods served the whole time; the right urgency is "blocked deploy", not "site down". - **Restarting the rollout.** `kubectl rollout restart` creates yet another ReplicaSet with the same bad image. - **Fixing the tag without checking it exists.** Run r1 confirmed `1.27-alpine` against the registry before suggesting it. One more typo costs another rollout. ## What we saw in the lab We injected this fault with one command, `kubectl -n shop set image deploy/frontend nginx=nginx:1.27-alpne`, into a 3-node kind v0.33.0 cluster running a small app (`shop`: nginx frontend, a Python api, a worker, redis, a load generator). Aurora got only the alert text, "KubeDeploymentRolloutStuck: deployment frontend in namespace shop has not progressed.", read-only kubectl under its default `observability_only` policy, model `bedrock/us.anthropic.claude-sonnet-5`, single agent, Aurora commit `87d22baf`. Two runs, each with a fresh user and org. | Run | Verdict (person-checked) | Time | Tool calls | Cost | Fix suggested | |---|---|---|---|---|---| | r1 | correct | 101 s | 13 (1 blocked) | $0.24 | `kubectl set image deployment/frontend nginx=nginx:1.27-alpine -n shop` | | r2 | correct | 77 s | 11 (1 blocked) | $0.19 | set the tag to `nginx:1.27-alpine`, `rollout undo` as the alternative | Both runs took the same path this runbook takes: `get deployment` (the image column), `get pods` (one pod in `ImagePullBackOff`, two serving), `describe pod` (the `not found` event), then a ReplicaSet comparison to find the good tag. r1 used the jsonpath above; r2 read both ReplicaSets as YAML. Both named the typo, both said the old pods were still serving so there was no user-facing outage, and both ruled out registry credentials and network problems, the two wrong answers the scenario was built to tempt. r1 also checked that `1.27-alpine` exists on Docker Hub before suggesting it. Both runs tried `kubectl rollout history deployment/frontend -n shop` to see what changed and got: ``` Command blocked by organization policy: Mutating kubectl operations ``` `rollout history` is read-only; Aurora's default policy denies the whole `rollout` verb. Both runs worked around it with the ReplicaSet comparison. The policy fix (deny only `rollout restart|undo|pause|resume`) is in the lab's [fix branch](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md) and the blocked call counts as the one tool error in each run. What the reports got wrong, from the grades: r1 described "3 total pods" and "reduced redundancy" although 2 of 2 desired pods were available, and said the cluster could not schedule a replacement pod when the pod was scheduled and only the pull failed. r2 said the Deployment was created at 16:57:25 (that was the revision 1 ReplicaSet) and called `rollout history` a mutating operation because the policy message said so. None of it touched the root cause or the fix, and the person reviewing the grades kept both verdicts at correct. Full traces: [r1](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/03-bad-image-tag/r1), [r2](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/03-bad-image-tag/r2). ## How Aurora handles this Aurora receives the alert, lists the Deployment and its pods, reads the waiting pod's events and compares the ReplicaSets, then writes the root cause with each claim cited to a tool output, as in the two runs above. It runs read-only by default: kubectl goes through an allowlist, and the fix is written as a suggested next step for a person to apply, not executed. It records the investigation as memory, so the next `KubeDeploymentRolloutStuck` in the same org starts from what it learned. The [root cause analysis](/rca) page shows what a finished investigation looks like; the method behind it is in the [root cause analysis guide](/blog/root-cause-analysis-complete-guide-sres). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) ## Sources and verification - Every output block above is from the lab runs of 2 October 2026, [runs/v0-run2/03-bad-image-tag](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/03-bad-image-tag) (r1 and r2: every tool call and output in `steps-full.json`, cluster state in `evidence-before/`), trimmed to the relevant lines and otherwise unchanged. Fault injection: [scenarios/03-bad-image-tag](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/03-bad-image-tag). - Versions: Kubernetes server v1.37.0, kind v0.33.0, containerd 2.3.4, kubectl client v1.35.3 (`evidence-before/versions.yaml`). Aurora commit `87d22baf`, model `bedrock/us.anthropic.claude-sonnet-5`. - The ImagePullBackOff definition, the 300-second cap, the registry defaults, the `imagePullSecrets` namespace rule and the `:latest` advice are quoted from the Kubernetes [container images](https://kubernetes.io/docs/concepts/containers/images/) documentation. The `ProgressDeadlineExceeded` behavior is from the [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#failed-deployment) documentation. Both read 8 October 2026. - The grades, including the misstatements listed above, are in each run's `grade.json` with `human_verdict` recorded on 2 October 2026. # Blog posts (full text) ## Edwin AI vs Dynatrace Davis AI vs Datadog Bits AI (2026) URL: https://www.aurorasre.ai/blog/edwin-ai-logicmonitor-alternatives Published: 2026-10-21 Author: Arvo AI Team, Engineering > Edwin AI, Dynatrace Davis AI and Datadog Bits AI on root-cause evidence, remediation, hybrid coverage and price (checked 2026-10-21), plus tested alternatives. Edwin AI is LogicMonitor's AIOps layer: it correlates alerts from LogicMonitor and third-party tools into "insights", then adds AI agents that investigate an insight and suggest or run a fix. Compared with Dynatrace Intelligence (formerly Davis AI) and Datadog Bits Investigation (formerly Bits AI SRE), its difference is where the evidence comes from: Edwin starts from correlated alerts plus CMDB and change records, Dynatrace from its own dependency graph, and Datadog from its own telemetry. LogicMonitor publishes a starting list price for the package that includes Edwin's event correlation ($53 per Hybrid Unit a month), but Edwin's AI Agents, which run the investigations, are an add-on with no published price (and not available on the entry package). If you want the investigation layer without a monitoring platform under it, the open source agents (Aurora, HolmesGPT) are the alternatives we have measured. Arvo builds Aurora and wrote the lab scenarios cited below. We have not tested Edwin AI, Dynatrace or Datadog: their AI layers are hosted services we could only try in a vendor trial, and none publishes a test set a buyer can rerun. Every claim about them links the vendor's own pages, read on 2026-10-21. ## What is Edwin AI and what does it do today? Edwin AI is sold inside LogicMonitor's platform, in three parts listed on the [pricing page](https://www.logicmonitor.com/pricing): Event Intelligence, AI Agents and AI Automation. **Event intelligence (correlation).** Edwin "ingests events and transforms them into contextualized insights": it deduplicates repeat events into one alert, then correlates, groups and prioritizes alerts into insights, which LogicMonitor says reduces noise "with up to 90-95% efficiency" ([Introduction to Edwin AI](https://www.logicmonitor.com/support/edwin/introduction-to-edwin-ai)). Correlation is model-based and configurable: each model has a dataset filter, the alert fields to compare, a string-based or list-based method, a similarity percentage and a minimum cluster density of at least 2 alerts ([Edwin AI Models](https://www.logicmonitor.com/support/edwin/models)). ServiceNow CMDB data enriches alerts, and alerts can be reclustered when a better grouping appears. **AI Investigation (root cause).** For an insight or a single alert, AI Investigation "analyzes alerts, metrics, logs, change requests, diagnostic and remediation source results and other available data" and returns a title, summary, most probable root cause, impact, timeline and suggested remediation split into "Immediate Fix" and "Preventative Actions". By default it runs automatically on critical and major insights; lower severities are triggered by hand. The docs say root causes "can be derived from alert data and available LogicMonitor log events, and ServiceNow Change Requests" for the impacted configuration items. It needs the AI Agents SKU ([AI Investigation](https://www.logicmonitor.com/support/edwin-ai-investigation)). **AI Agents (chat and orchestration).** An orchestrator agent picks among specialized agents: IT Ops (triage steps and draft RCAs), Insights, Charts, Public Knowledge (vendor docs and known bugs), Metrics, Similar Insights, Similar Incidents, Private Knowledge Base and Change Request. The Similar, Private Knowledge Base and Change Request agents need the ServiceNow app (LMDX). The agents work on the events, alerts, insights and metadata Edwin has received ([AI Agents overview](https://www.logicmonitor.com/support/edwin-ai-agents-overview), updated 2026-07-31). [Agent Studio](https://www.logicmonitor.com/edwin-ai/agent-studio) controls which agents are enabled and which MCP tools and integrations they may use, and its Agent Observability view shows each agent's data sources, tool activity and failed tool calls. A LogicMonitor MCP server exposes Edwin investigations to clients such as Cursor, Codex and Claude; "The current release is read-only" ([MCP](https://www.logicmonitor.com/platform/edwin-ai-mcp-model-context-protocol)). **AI Automation (remediation).** Edwin searches for existing Red Hat Ansible, Terraform and Rundeck workflows first; if none fits, "IBM watsonx Code Assistant turns Edwin's diagnosis into precise Ansible YAML", executed through Ansible Automation Platform "with approvals, rollback, and audit trails" ([AI Automation](https://www.logicmonitor.com/edwin-ai/ai-automation)). Separately, the core platform runs Groovy or PowerShell DiagnosticSources and RemediationSources, by hand or from Action Rules that match alert conditions, on collector-monitored resources ([docs](https://www.logicmonitor.com/support/automated-diagnostics-and-remediation-overview)). LogicMonitor's [AI governance page](https://www.logicmonitor.com/ai-governance) says "Every output is a recommendation" and that people "decide whether to permit autonomous action and remediation." **Integrations.** The [integrations list](https://www.logicmonitor.com/support/edwin-ai-integrations-overview) (updated 2026-10-19) names monitoring sources Catchpoint, Splunk, Datadog, AppDynamics, ThousandEyes, Dynatrace, Cribl, Elastic, Grafana, New Relic, SolarWinds and Nexthink; ITSM targets ServiceNow, Jira (Data Center and Service Management), MS Dynamics, PagerDuty, ConnectWise, xMatters, BMC Helix, BigPanda, Freshservice and Zendesk; automation through Ansible and Rundeck; Microsoft Teams; and a Python SDK. Several entries say "For documentation, please contact your CSM". Third-party tools send events: the [Dynatrace integration](https://www.logicmonitor.com/support/edwin-ai-dynatrace-integration), for example, is a Dynatrace problem webhook posting to Edwin's event endpoint. So Edwin can sit on top of Dynatrace or Datadog as a correlation layer, but it receives their problems and alerts, not their traces. **Hybrid and on-prem coverage.** LogicMonitor's collector "runs on a Linux or Windows server within your infrastructure", polls devices with standard protocols, and sends the data "back to the LogicMonitor servers over an outgoing SSL connection" ([collector docs](https://www.logicmonitor.com/support/collectors/collector-overview/about-the-logicmonitor-collector)). That is LogicMonitor's strength: network gear, servers, storage and wireless on-prem, plus cloud, under one license unit. Edwin itself runs in LogicMonitor's cloud: collectors send to LogicMonitor's servers, and third-party tools post events to Edwin's endpoint. We found no self-hosted or on-prem Edwin option in LogicMonitor's docs or pricing page. **Models.** LogicMonitor announced a "strategic collaboration" with OpenAI in March 2025 that brings OpenAI's reasoning and natural-language capabilities into Edwin, for contextualizing alerts, explaining root-cause findings and generating suggested actions ([LogicMonitor blog](https://www.logicmonitor.com/blog/logicmonitor-edwin-ai-open-ai-collaboration)). Its governance page says customer data "is not used to train third-party LLMs", PII fields can be excluded, and model upgrades pass "structured offline evaluation against a benchmark suite". The Agent Studio page mentions "model flexibility" and work with OpenAI, IBM Red Hat and AWS, but we found no documentation of choosing a model or bringing your own, and the benchmark results are not published. LogicMonitor's own outcome figures (a "313% ROI" Forrester TEI study it commissioned, customer quotes on noise cut) are on the [Edwin AI page](https://www.logicmonitor.com/edwin-ai). We found no published root-cause accuracy figure. ## How is Edwin AI priced and licensed? From the [LogicMonitor pricing page](https://www.logicmonitor.com/pricing) on 2026-10-21. Prices are "starting monthly list pricing based on standard minimum quantities and terms", in USD. | Package | List price | Edwin Event Intelligence | Edwin AI Agents (includes AI Investigation) | Edwin AI Automation | |---|---|---|---|---| | Essentials | $16 per Hybrid Unit | Not available | Not available | Not available | | Advanced | $27 per Hybrid Unit | Add-on | Add-on | Add-on | | Signature + Edwin AI | $53 per Hybrid Unit | Included | Add-on | Add-on | - **The unit.** "One Hybrid Unit is equivalent to one on-prem collector monitored device, one cloud IaaS, seven cloud PaaS, five wireless access points, or 100 gigabytes of OpenTelemetry metrics." Essentials stops at 999 Hybrid Units. - **Quote-only.** No add-on has a published price, so AI Agents and AI Automation are quote-only on Advanced and Signature + Edwin AI (Essentials does not offer them), and Event Intelligence is quote-only on Advanced. The agentic investigation that this page compares with Dynatrace and Datadog is therefore not priced publicly anywhere. Discounts "Potentially, yes", through the account team. - **Trial.** The pricing page says "Try any plan free for 15 days"; the site navigation elsewhere offers "14-day access to the full LogicMonitor platform". - **Our arithmetic, not LogicMonitor's figure:** Signature + Edwin AI lists $26 a month more per Hybrid Unit than Advanced. That gap also buys other Signature features, so it is not Edwin's price. At list, 500 on-prem devices on Signature + Edwin AI is $26,500 a month before the AI Agents add-on. Edwin is not sold as a standalone product on that page: it is licensed per monitored resource as part of a LogicMonitor package, and third-party sources feed it through integrations. ## Edwin AI vs Dynatrace Davis AI vs Datadog Bits AI: root-cause evidence, remediation and hybrid coverage How they compare on root-cause evidence, remediation and hybrid coverage, from each vendor's docs and pricing pages on 2026-10-21. We have run none of them. | | Edwin AI (LogicMonitor) | Dynatrace Intelligence (formerly Davis AI) | Datadog Bits Investigation (formerly Bits AI SRE) | |---|---|---|---| | Starts from | Correlated alerts ("insights") and singleton alerts; auto-investigates critical and major ones | Problems Dynatrace detects; agentic tasks from Dynatrace Assist, workflows, MCP or the API | A Datadog monitor alert, Synthetic test, RUM or a free-text prompt | | Root-cause evidence | Alerts, metrics, LogicMonitor logs, ServiceNow change requests, diagnostic and remediation source results; CMDB data enriches the alerts ([docs](https://www.logicmonitor.com/support/edwin-ai-investigation)) | "Deterministic, causation-based analysis" over the Smartscape dependency graph and Grail data, with code, deployment and configuration changes ([docs](https://docs.dynatrace.com/docs/dynatrace-intelligence)) | Iterative hypotheses tested against Datadog metrics, traces, logs, Change Tracking, GitHub source code ([docs](https://docs.datadoghq.com/bits_ai/bits_investigation/investigate_issues/)) | | Remediation | Suggested fix and preventive steps; AI Automation finds or generates Ansible playbooks and runs them with approvals | Proposes fixes or rollbacks through agentic workflows, in Preview; runs approved runbooks | Triage actions (Slack, Teams, paging, Jira); rollbacks through Datadog Workflows ([blog](https://www.datadoghq.com/blog/bits-ai-sre-deeper-reasoning/)) | | Hybrid and on-prem | On-prem devices through collectors; Edwin itself is hosted | Dynatrace Managed runs on-prem, but its agentic and generative AI "is not available for Dynatrace Managed customers" ([FAQ](https://docs.dynatrace.com/docs/dynatrace-intelligence/agentic-and-generative-ai/agentic-and-generative-ai-faq)) | Bits runs in Datadog's SaaS; not available to FedRAMP customers ([docs](https://docs.datadoghq.com/account_management/billing/ai_credits/)) | | Third-party data | Events from Datadog, Dynatrace, Splunk, Grafana, New Relic and more, plus ITSM | Data ingested into Dynatrace | Grafana, Dynatrace, Sentry, Splunk, ServiceNow, Confluence, all "in Preview" | | Price of an investigation | Not published (AI Agents add-on) | 30 AI units average at $0.10 a unit: about $3 by our arithmetic; not billed inside Dynatrace Assist "for a limited time" ([pricing](https://www.dynatrace.com/pricing/)) | About 6.5 AI credits; $1 to $1.30 a credit, so about $6.50 to $8.45 by our arithmetic ([pricing](https://www.datadoghq.com/pricing/?product=bits-ai-sre)) | | Data you must have | LogicMonitor monitoring or third-party events sent to Edwin; ServiceNow for several agents | Dynatrace monitoring (OneAgent, cloud integrations) on Dynatrace SaaS | Datadog monitors and telemetry | | Public accuracy test | None | None | None (internal eval platform described, scores not published) | The short version: Dynatrace's root cause is the most structured, because it walks a dependency model it built itself; Datadog's is the most investigative inside its own data; Edwin's is the broadest in sources, because it is a correlation layer that takes events from other tools, but its evidence for those sources is the event, not the underlying telemetry. That makes Edwin strongest where the problem is alert volume across many tools and a ServiceNow process, and weakest where the cause hides in data that never raised an alert. ## When does each one fit? - **Edwin AI** fits a hybrid IT estate (on-prem devices and network gear, wireless, cloud) already monitored by LogicMonitor or by several tools, run through ServiceNow, where the first problem is too many alerts. - **Dynatrace Intelligence** fits teams whose applications run under OneAgent on Dynatrace SaaS and who want root cause tied to service topology. On Dynatrace Managed you keep the deterministic RCA but not the agentic layer. - **Datadog Bits Investigation** fits teams whose telemetry is all in Datadog and whose alert volume makes about 6.5 credits per investigation acceptable. See our [Datadog Bits AI alternatives](/blog/datadog-bits-ai-sre-alternative-open-source) page for the credit arithmetic. - **None of them** fits a team that needs the investigation agent itself in its own network, wants to choose the model, or does not want to buy a monitoring platform to get an investigator. That is where the open source agents come in. ## What are the alternatives to Edwin AI? Checked against each project's or vendor's site on 2026-10-21. | Tool | What it is | Reads LogicMonitor | Runs where | Price | Lab-tested by us | |---|---|---|---|---|---| | [Aurora](https://github.com/Arvo-AI/aurora) | Open source agent that investigates each incoming alert | No connector; ingests PagerDuty, Datadog, Grafana, New Relic, OpsGenie, incident.io, Dynatrace and more | Docker Compose, Helm, air-tight bundle | Free, Apache 2.0; you pay hosting and model calls | Yes | | [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) | Open source investigation agent with toolsets | No built-in toolset (list checked 2026-10-21); has Datadog, New Relic, Prometheus, ServiceNow and others, and connects to remote MCP servers, so LogicMonitor's read-only MCP server is a possible route we have not tried | CLI, server, Kubernetes | Free, Apache 2.0; you pay model calls | Yes | | [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) | Open source Kubernetes scanner with LLM explanations | No | CLI, in-cluster operator | Free, Apache 2.0 | Yes, as a scanner | | [Dynatrace Intelligence](https://www.dynatrace.com/platform/artificial-intelligence/) | Platform RCA plus agentic investigation | No LogicMonitor integration found | Dynatrace SaaS for the agentic part | AI units, see above | No | | [Datadog Bits Investigation](https://www.datadoghq.com/product/ai/bits-investigation/) | Investigation agent over Datadog data | Not in its documented sources | Datadog SaaS | AI credits, see above | No | | [BigPanda](https://www.bigpanda.io/) | Alert correlation platform with an investigation swarm | Edwin lists BigPanda as an ITSM target; we did not check the reverse | SaaS, EU instance | Credit plans, no dollar figures | No | For the correlation-first products in this category, see [BigPanda alternatives](/blog/bigpanda-alternative-open-source) and [Moogsoft (Dell APEX AIOps) alternatives](/blog/moogsoft-dell-apex-alternative-open-source); for hosted AI SRE agents (Traversal, Resolve AI, Cleric and others), [AI incident investigation tools compared](/blog/best-ai-powered-incident-investigation-tools); for pricing across the category, [AIOps platforms with root cause analysis](/blog/top-10-aiops-platforms-free-root-cause-analysis-2026). An open source agent is not a drop-in replacement for Edwin. Aurora's correlation engine links related alerts into the running investigation but never suppresses or mutes them, so it does not do Edwin's noise reduction, and it does not monitor devices. It replaces the investigation step, not the monitoring or the event pipeline. If LogicMonitor stays, the open question is how its alerts reach the agent; Aurora has no LogicMonitor connector today. ## How did the open source alternatives do in our lab? This is the part we measured. The lab is a small app (nginx frontend, Python api, worker, redis, load generator) on a 3-node kind v0.33.0 cluster with Kubernetes v1.37.0. One fault is injected per scenario (two in scenario 12), 19 scenarios, two runs each. Aurora and HolmesGPT get the alert; K8sGPT takes no alert and scans the namespace. All three got the same admin kubeconfig, each behind its own guard: Aurora's observability-only command policy, HolmesGPT's allow-list, K8sGPT's read-only `analyze`. Aurora (commit 87d22baf, on `bedrock/us.anthropic.claude-sonnet-5`) ran on 2026-10-02; HolmesGPT 0.43.0 (same model) and K8sGPT 0.4.40 ran on 2026-10-15 and 2026-10-16, on the same cluster and faults; K8sGPT used `us.anthropic.claude-sonnet-4-5` because it sends a temperature setting that Bedrock rejects for Sonnet 5. There is no LogicMonitor, Dynatrace or Datadog in the lab: the agents read the cluster directly. | | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | Correct | 34 of 38 | 31 of 38 | not scored as a peer | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 | | "No problems detected" | 0 | 0 | 12 runs (6 scenarios) | | Median time per run | 153.5 s | 45 s | 12.5 s for two analyze passes | | Model cost per run | $0.19 to $0.84, median $0.35 | not captured | not reported | | Who confirmed the grades | a person at Arvo | judge model only | judge model only | A judge model (`us.anthropic.claude-opus-5-5`) graded every answer. A person at Arvo read all 38 Aurora grades and changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model re-read all 76 and agreed, a third pass graded them from scratch and agreed on all 38 HolmesGPT verdicts while disputing 6 K8sGPT labels, and no person has checked them. A blinded regrade, with tool names removed from the judge's prompt, matched the published verdict in 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs. The prompts differ in wording, and HolmesGPT ran `holmes ask` with default toolsets. A two-sided Fisher exact test on the totals, computed on [/benchmark](/benchmark), gives p = 0.52 for Aurora against HolmesGPT, comparing human-read Aurora grades with judge-only HolmesGPT grades: treat the two as roughly even on accuracy. HolmesGPT is about 3x faster at the median. What the cost line leaves out: Aurora's background discovery job cost about $1 per cluster connection in the lab ($29.70 for 30 completed runs, about twice the $14.53 of the 38 graded investigations), plus your own hosting. HolmesGPT's log says the model "has no entry in litellm's cost map", so its cost was recorded as $0. K8sGPT is a different kind of tool. It runs analyzers over a namespace and explains what they flag, without an alert. In 12 runs it printed "No problems detected", on faults such as a NetworkPolicy block or a slow memory leak where every object looks healthy. Use it as a scanner, not an investigator. ### The miss that matters for correlation-first tools Edwin, Dynatrace, Datadog, Aurora and HolmesGPT all start from an alert or a problem. Scenario 12 shows the limit of that design: a worker OOMKilled fires the alert while, separately, the frontend Service selects no pods. Both agents found the OOM in both runs, and neither reported the frontend outage. Aurora went further and stated the opposite: > Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]. Pod status was its only evidence. Alert correlation does not fix this: it can only group alerts that fired, and a fault no monitor covers never enters the stream. Whatever you buy, test it on a second fault the alert does not mention; the checks are in the runbook [a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). ## How to trial Edwin AI, Dynatrace or Datadog against the same faults LogicMonitor's pricing page and Dynatrace offer 15-day trials, and Datadog a 14-day trial. None of them publishes a test set, so bring your own. Each scenario in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab) has the alert, the true root cause and the findings a correct answer must contain. ``` git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab ./lab.sh up ./lab.sh break 12-two-faults-oom-and-selector ``` The cluster has to report to the trial you are testing, since each product works from its own data: a LogicMonitor collector, a Dynatrace OneAgent or a Datadog Agent. Build the alert in that product yourself, let it investigate, grade the answer against `must_identify` in the scenario's `scenario.yaml`, then `./lab.sh reset`. Run every scenario twice, as we did, and keep the misses. For a single failure on your own cluster, use [/diagnose](/diagnose). ## What the lab does not tell you The lab is Kubernetes-only and synthetic, with faults Arvo wrote. Edwin's strengths are elsewhere: on-prem devices, network gear and ServiceNow process, none of which the lab touches, and it says nothing about alert-noise reduction at volume. HolmesGPT 0.44 and 0.45 were already out when we ran 0.43.0; those versions are untested. Vendor packaging changes often (Dynatrace's Davis AI and Datadog's Bits AI SRE URLs now redirect to Dynatrace Intelligence and Bits Investigation), so recheck the pricing pages before you sign. Verified 2026-10-21: LogicMonitor pricing, Edwin AI product pages, Edwin AI docs and governance page; Dynatrace docs, FAQ and pricing; Datadog pricing and docs, as linked. Lab numbers from [results.json](https://raw.githubusercontent.com/Arvo-AI/aurora-lab/main/benchmark/results.json); full method in the [Aurora lab report](/blog/aurora-lab-report-kubernetes-19-failures) and [Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt). See also [Dynatrace Davis alternatives](/blog/dynatrace-davis-alternative-open-source). Aurora is on [GitHub](https://github.com/Arvo-AI/aurora) under Apache 2.0, or [book a demo](https://cal.com/arvo-ai/demo) to run it on your own alerts. --- ## A kubectl deny rule blocked rollout history in 26 of 38 runs URL: https://www.aurorasre.ai/blog/kubectl-rollout-history-blocked-by-command-policy Published: 2026-10-19 Author: Ben Gervais, Engineer at Arvo AI > Aurora's default kubectl policy denied rollout history in 26 of 38 lab runs. The regex, the ReplicaSet workaround, the fix, and what is still open. In the lab run behind our [first lab report](/blog/aurora-lab-report-kubernetes-19-failures), scenario 03 is a mistyped image tag: `kubectl -n shop set image deploy/frontend nginx=nginx:1.27-alpne`. Aurora had the pod in ImagePullBackOff by step 6 and the registry's "not found" by step 7. Step 9 was the obvious next question, what changed, and the answer came from our own code: ```text $ kubectl rollout history deployment/frontend -n shop Command blocked by organization policy: Mutating kubectl operations ``` The tool result carried `"code": "POLICY_DENIED"`. `kubectl rollout history` writes nothing. It reads the ReplicaSets a Deployment keeps and prints one line per revision. Our default command policy, the one that exists to keep an investigating agent from changing a cluster, refused it, and the report filed the gap under "Not Checked": the "full revision diff/history was not reviewed directly". ## How often it happened Across the 38 graded runs on Aurora commit `87d22baf`, 26 hit this denial: 16 of 20 in v0-run2 and 10 of 18 in v1-run1, 27 calls in total (counted from each run's `steps-full.json`: 17 and 10 `on_prem_kubectl` calls with `rollout history` in the command and `POLICY_DENIED` in the output). Every one of the 27 was `kubectl rollout history deployment/ -n shop`. Aurora never asked for `rollout restart` or `rollout undo`. The denial also leaked into Aurora's reasoning. In the superseded v0-run1, the scenario 05 report listed "Full rollout history (`kubectl rollout history`)" as "blocked by organizational policy on mutating kubectl operations", and in its chat notes it wrote "`rollout history` is blocked by policy (mutating-kubectl denylist)". A read had been relabelled a write, and the model repeated the label. ## The workaround that did not work well Without the command, the agent fell back on listing ReplicaSets. Scenario 03, step 8, one step before the denial: ```text $ kubectl get replicaset -n shop -l app=frontend -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[0].image}{" replicas="}{.spec.replicas}{" ready="}{.status.readyReplicas}{"\n"}{end}' frontend-76d98597b9 nginx:1.27-alpine replicas=2 ready=2 frontend-7b6c998456 nginx:1.27-alpne replicas=1 ready= ``` That works, as far as it goes. The Kubernetes docs say it plainly: "A Deployment's revision history is stored in the ReplicaSets it controls" ([Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)). Listing them and printing the image gives the same fact `rollout history --revision=N` would, for that one field. Eleven of the 20 v0 runs listed ReplicaSets this way, and all 20 ended with a correct verdict. It is still the wrong tool, and the runs show the cost. Aurora had to compose a jsonpath expression for each field it wanted to compare, so every further question (an env var, a probe, a limit) is another call, where `rollout history --revision=N` prints the whole pod template at once. It had to `describe deployment` in step 10 to learn it was on revision 2. In scenario 01 the report said that because of the policy "the exact author/timestamp of the out-of-band env var change could not be retrieved". And in scenario 12, where a second fault sat unalerted in the same namespace and both runs missed it, one of r1's 18 tool calls went to a command that returned a policy string. A workaround that costs steps costs attention. ## The bug Every command Aurora runs against a cluster passes through `command_gate`, which calls `evaluate_compound_command` in `server/utils/auth/command_policy.py`. The file's docstring gives the order: the denylist is checked first and a match is denied; the allowlist is checked second, a match is allowed and no match is denied; with both off, everything is allowed. A new org is seeded with the `observability_only` template and both lists on. The deny rule that matched, [line 704 at `87d22baf`](https://github.com/Arvo-AI/aurora/blob/87d22baf/server/utils/auth/command_policy.py#L704) (line 790 carries the same bare `rollout` for `standard_ops`): ```text # before \bkubectl\s+(exec|cp|run|attach|port-forward|apply|delete|create|edit|patch|replace|scale|rollout|drain|cordon|uncordon|taint)\b # after \bkubectl\s+(exec|cp|run|attach|port-forward|apply|delete|create|edit|patch|replace|scale|rollout\s+(restart|undo|pause|resume)|drain|cordon|uncordon|taint)\b ``` `rollout` followed by `\b` matches `kubectl rollout history` exactly as well as it matches `kubectl rollout undo`. The rule's description, "Mutating kubectl operations", is the string the agent saw, and it is wrong for two of the six rollout subcommands. The rule structure also rules out the one-line version of the fix, narrowing the deny rule and stopping there. With the allowlist on, a command that clears the deny list is then matched against the allow rules, and the Kubernetes read rule at line 643 listed `get|describe|logs|top|explain|api-resources|api-versions|cluster-info|config ...` with no `rollout` entry. The engine's own code for that case returns `allowed=False`, `rule_description="No matching allow rule"` and `allowlist_exhausted=True`: a different reason string, the same refusal for the agent. We did not run that version; this is what the evaluator's code does, and the test below checks for it. So the fix touches both lists in both templates: the deny rule narrows to `rollout\s+(restart|undo|pause|resume)`, and `rollout\s+(history|status)` joins the read allow rule. Four lines changed. ## The test The useful part of the test is the path, not the assertion. The policy had tests before; none loaded the shipped template and asked the real evaluator about `rollout history`. The new `server/tests/auth/test_command_policy_rollout.py` pulls each template from `get_policy_templates()`, sets both lists on as `seed_default_command_policy` leaves a new org, and goes through `evaluate_compound_command`, the function `command_gate` calls: ```python @pytest.fixture(params=["observability_only", "standard_ops"]) def template_policy(request, monkeypatch): tpl = next(t for t in get_policy_templates() if t["id"] == request.param) allow, deny = _rules(tpl["allow"], "allow"), _rules(tpl["deny"], "deny") states = ListStates(allowlist_enabled=True, denylist_enabled=True) ... @pytest.mark.parametrize("cmd", READ_ONLY) def test_rollout_reads_are_allowed(template_policy, cmd): verdict = evaluate_compound_command("org-1", cmd) assert verdict.allowed, (template_policy, cmd, verdict) assert not verdict.allowlist_exhausted @pytest.mark.parametrize("cmd", MUTATING) def test_rollout_mutations_are_denied_by_deny_rule(template_policy, cmd): verdict = evaluate_compound_command("org-1", cmd) assert not verdict.allowed, (template_policy, cmd) assert verdict.deny_rule_id is not None assert verdict.rule_description.startswith("Mutating kubectl operations") ``` `READ_ONLY` is `rollout history` and `rollout status`; `MUTATING` is `restart`, `undo`, `pause` and `resume`. The line `assert not verdict.allowlist_exhausted` is the check for the deny-only mistake: a read that clears the deny list but matches no allow rule exhausts the allowlist and fails it. Two templates, six commands, 12 tests. On the unfixed code 4 of the 12 fail, the four read-only cases. With the existing `test_command_policy.py` and `test_allowlist_bypass.py`, the three files pass 145 tests. The change is an open [pull request on Arvo-AI/aurora](https://github.com/Arvo-AI/aurora/pull/733), opened 2026-10-19 and not merged at the time of writing. ## The rerun We rebuilt the lab images on the patched code and reran scenarios 05, 09 and 10 twice each ([runs/v0-fix-run1](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-fix-run1)). Each run got a fresh org, which matters here: the template is copied into `org_command_policies` rows when an org is created, so a new org gets the new rule and an existing org keeps its old rows until the template is re-applied from Settings > Security. | Scenario | 87d22baf | Patched | Median time, before / after | Median cost, before / after | |---|---|---|---|---| | 05 Invalid nginx directive | 2/2 | 2/2 | 136 s / 145 s | $0.32 / $0.36 | | 09 PVC with missing StorageClass | 2/2 | 2/2 | 280 s / 182 s | $0.39 / $0.44 | | 10 Wrong REDIS_HOST | 2/2 | 2/2 | 244 s / 100 s | $0.28 / $0.32 | `rollout history` ran 14 times across the 6 runs and was denied 0 times. In 10 r2 Aurora asked for `--revision=1` and `--revision=2` and got both pod templates back, `REDIS_HOST: redis` in the first and `REDIS_HOST: redis-master` in the second, which is the injected fault. All 6 runs were graded correct and confirmed by a person. The faster medians on 09 and 10 are mostly not the fix: the earlier runs shared worker slots with background prediscovery jobs and these did not. ## Two other bugs from the same lab day The fix branch carries three patches. The other two came out of one run, v0-run1 scenario 09, superseded because all ten runs shared one org. **No `max_tokens` on Bedrock.** Aurora found the PVC with the missing `fast-ssd` StorageClass by step 10, then made 25 `write_artifact` calls over about 11 of the run's 15 minutes and $2 of its $2.61. In `timeline.json`, 19 agent responses in that stretch have `output_tokens` of exactly 4096. The Bedrock provider (`chat/backend/agent/providers/bedrock_provider.py`, lines 180 to 197 at `87d22baf`) never sets `max_tokens`, so Converse used its default. The model was rewriting a whole "Incident Index" artifact in one call, the arguments JSON was cut off before `content`, pydantic rejected the call before the tool function ran, and the model saw only "content: Field required", retried, then tried "placeholder" and "test2". Patch 0002 registers the tool with the same schema as a plain JSON dict so the call reaches `write_artifact`, which now returns an error saying the call was most likely cut off by the output-token limit and asking for a shorter document. The cap itself is unchanged. **The 30-citation window.** The RCA summary is written from `citations[-30:]` (`chat/background/summarization.py`, line 211), and every tool message counted except a short `_NON_EVIDENCE_TOOLS` set. That run had 45 tool calls; by our reconstruction calls 19 to 45 were artifact and memory calls, so the PVC, StorageClass and PVC-yaml results from steps 7, 8 and 11 never reached the summary prompt. The report cited only [17] and [18], named the cause as "a dependency the cluster could not satisfy", and filed the PVC under "Not Checked". Patch 0003 adds `write_artifact`, `write_memory`, `append_to_memory` and `edit_memory` to the non-evidence set; reads stay citable. The window is unchanged. Both patches are in a second [pull request on Arvo-AI/aurora](https://github.com/Arvo-AI/aurora/pull/734), opened 2026-10-19 and not merged at the time of writing. Each has unit tests (4 and 2) and nothing else: the artifact loop did not recur in the 26 later runs (0 `write_artifact` calls), so the rerun never exercised them. ## Still open - The Ask-mode classifier. `utils/security/read_only_classifier.py` (line 47 at `87d22baf`) lists `rollout` in `WRITE_VERBS`, so on the `cloud_exec` kubectl path for EKS, GKE and AKS, `kubectl rollout history` is still refused. The lab's `on_prem_kubectl` tool does not pass through it. It is a security gate and needs a kubectl-specific `rollout ` rule, not a one-word deletion. - Existing orgs keep the old pattern until someone re-applies the template. Migrating their rows is a separate decision. - `kubectl exec ... -- cat` is also denied as mutating. In scenario 19 r1 that stopped Aurora from comparing the live nginx config of two pods. Denying `exec` is defensible and we have not changed it. - The 4096 token cap and the 30-citation window are what they were. An investigation with more than 30 real evidence calls still loses its earliest evidence. ## The lesson A deny rule should name the verb that mutates, not the family it belongs to. `rollout` is a family; `rollout undo` is a verb. And a test that checks a regex in isolation tells you about the regex. The test that would have caught this loads the shipped template, turns on both lists the way a new org gets them, and asks the same function the gate asks. We had the regexes. We did not have that test, and 26 of 38 investigations paid for it, 27 blocked calls in all. Read-only by default is the right default for an agent in a production cluster. This is what it costs when the definition of read-only is one word too wide, and why we run the lab against our own build. Aurora is open source, and the policy, the test and the reruns are all public in the links above. ## Related The runs are in the [lab report](/blog/aurora-lab-report-kubernetes-19-failures); the policy engine is described in [AI agent kubectl safety](/blog/ai-agent-kubectl-safety); the scenario behind the other two bugs has a runbook, [PVC Pending, StorageClass not found](/runbooks/fix-pvc-pending-storageclass-not-found-kubernetes). ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) > **Sourcing note.** Every number, command, output and quote above comes from the lab repo [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab) as of its 2026-10-02 runs ([FINDINGS.md](https://github.com/Arvo-AI/aurora-lab/blob/main/FINDINGS.md), [REPORT.md](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md), [patches/NOTES.md](https://github.com/Arvo-AI/aurora-lab/blob/main/patches/NOTES.md) and the `runs/` folders, each run with `steps-full.json`, `aurora-rca.md` and `grade.json`), from the Aurora source at commit 87d22baf and on the fix branch, and from the two pull requests linked above. This post shows Aurora source paths, line numbers and pull request links because our engineering-post standard requires real code linked to the file or PR. Versions: kind v0.33.0, Kubernetes v1.37.0, RCA model bedrock/us.anthropic.claude-sonnet-5, judge us.anthropic.claude-opus-5-5. Written 2026-10-19. --- ## We broke Kubernetes 19 ways and let an AI SRE investigate: lab report v1 URL: https://www.aurorasre.ai/blog/aurora-lab-report-kubernetes-19-failures Published: 2026-10-08 Author: Ben Gervais, Engineer at Arvo AI > 38 graded runs of Aurora on 19 injected Kubernetes faults: 34 correct, 4 partial, 0 wrong, median 136 to 168 s, $0.32 to $0.40 a run. Every miss included. We built a small Kubernetes app, broke it on purpose in 19 known ways, gave Aurora nothing but the alert text and read-only access to the cluster, and scored its root cause analysis against the fault we injected. Across 38 graded runs on the current code, Aurora was correct 34 times, partial 4 times and wrong 0 times, at a median of 136 to 168 seconds and $0.32 to $0.40 per investigation. Arvo builds Aurora, and Arvo wrote these scenarios. Read the numbers with that in mind. Everything here is in the public lab repo, [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab): the scenarios, every run's tool calls and report, the grades, and the human review. Nothing was left out. ## How the lab works **Cluster.** [kind](https://kind.sigs.k8s.io) v0.33.0, Kubernetes v1.37.0, 3 nodes (1 control plane, 2 workers), on OrbStack on a Mac. Three identical clusters ran scenarios in parallel, one per lane. metrics-server v0.9.0 is installed, so `kubectl top` works. **App.** `shop`: an nginx frontend, a Python api, a worker, redis and a load generator. It is synthetic and holds no customer code or data. Names and labels do not hint at the fault. **Fault injection.** Each scenario is a folder under [scenarios/](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios) with `scenario.yaml` (the alert text, the ground truth, the findings a correct answer must contain, the listed wrong answers), `break.sh` (one command that injects the fault) and `check.sh` (proof that the fault is visible in the cluster). Aurora never sees `scenario.yaml`. **Aurora.** Commit `87d22baf`, RCA model `bedrock/us.anthropic.claude-sonnet-5`, single agent (the multi-agent orchestrator off), read-only cluster access under the default `observability_only` command policy. Every run got a fresh user and org, so no memory, artifact or incident history carried over between runs. The runner checks Aurora's first `list_incidents` call for any other run's incident ID; all 38 runs were clean. **What counts.** An answer is correct when it meets every `must_identify` item in the scenario and asserts none of the listed wrong answers. Partial means some items were met. A false claim outside the listed wrong answers does not make a run wrong. Stating a fix is not required for correct: Aurora keeps fixes in a separate "Suggested Next Steps" list, graded on its own. **Who scored it.** A judge model (`us.anthropic.claude-opus-5-5`, a different model from the one Aurora ran on) marked each required finding as met, partial or missed and quoted Aurora's words. A second Claude instance read every RCA, the tool outputs and the judge's grade and wrote its own verdict. A person at Arvo then reviewed every verdict and recorded it as `human_verdict` in each `grade.json`. The verdicts below are the human ones. ## Results v0 is the first ten scenarios, one clear fault each. v1 is nine harder scenarios: noise next to the fault, two faults at once, a slow leak, a partial restart. Each scenario ran twice. | # | Fault | Alert Aurora received | Correct | Median time | Median cost | |---|---|---|---|---|---| | 01 | Worker OOMKilled after a batch size change | KubePodCrashLooping | 2/2 | 160 s | $0.30 | | 02 | Sidecar OOMKilled, app container healthy | KubePodCrashLooping | 2/2 | 90 s | $0.25 | | 03 | Mistyped image tag, rollout stuck | KubeDeploymentRolloutStuck | 2/2 | 89 s | $0.22 | | 04 | Env var references a Secret that does not exist | KubeDeploymentRolloutStuck | 2/2 | 164 s | $0.42 | | 05 | Invalid nginx directive in a ConfigMap | KubePodCrashLooping | 2/2 | 136 s | $0.32 | | 06 | Liveness probe on the wrong port | KubePodCrashLooping | 2/2 | 105 s | $0.32 | | 07 | Service selector matches no pods | HighErrorRate | 2/2 | 156 s | $0.46 | | 08 | CPU request larger than any node | KubePodNotReady | 2/2 | 226 s | $0.28 | | 09 | redis Pending on a PVC with a missing StorageClass | KubePodCrashLooping | 2/2 | 280 s | $0.39 | | 10 | Wrong REDIS_HOST, name does not resolve | KubePodCrashLooping | 2/2 | 244 s | $0.28 | | 11 | api Service targetPort wrong, worker logs noisy but harmless | HighErrorRate | 1/2 | 340 s | $0.51 | | 12 | Worker OOM plus a frontend Service selecting no pods; alert covers only the worker | KubePodCrashLooping | 0/2 | 344 s | $0.34 | | 13 | NetworkPolicy blocks api to redis | HighErrorRate | 2/2 | 323 s | $0.61 | | 14 | ResourceQuota forbids the rollout's surge pod | KubeDeploymentRolloutStuck | 2/2 | 126 s | $0.32 | | 15 | Stray Deployment's pods carry app=api, 1 in 3 requests fail | HighErrorRate | 2/2 | 146 s | $0.47 | | 16 | dnsConfig search domains for the wrong cluster domain | KubePodCrashLooping | 2/2 | 172 s | $0.48 | | 17 | Slow memory leak in worker code, logs clean | KubePodCrashLooping | 2/2 | 144 s | $0.25 | | 18 | Init container waits for a Service that does not exist | KubeDeploymentRolloutStuck | 2/2 | 159 s | $0.44 | | 19 | ConfigMap edited, only one of two frontend pods restarted | HighErrorRate | 1/2 | 266 s | $0.61 | Totals: v0, 20 of 20 correct, median 136 s and $0.32, $6.44 for the run. v1, 14 correct and 4 partial of 18, median 168 s and $0.40, $8.09 for the run. Costs are Aurora's LLM spend only; the judge is not included. The per-run tables with every verdict, time, tool-call count and cost are in [runs/v0-run2/report.md](https://github.com/Arvo-AI/aurora-lab/blob/main/runs/v0-run2/report.md) and [runs/v1-run1/report.md](https://github.com/Arvo-AI/aurora-lab/blob/main/runs/v1-run1/report.md). The two 12 runs are partial even though both said a broken Service was healthy: that claim is not on the scenario's list of wrong answers, so under the rubric as written it does not make the run wrong. It is the worst result in the set and gets its own section below. ## A run that went right: the mistyped image tag Scenario 03. The change was `kubectl set image deploy/frontend nginx=nginx:1.27-alpne`, a tag that does not exist. The new ReplicaSet's pod sits in ImagePullBackOff; the old pods keep serving. Aurora received one line: "KubeDeploymentRolloutStuck: deployment frontend in namespace shop has not progressed." Run r1 took 101 seconds, 13 tool calls and $0.24. The full transcript is in [runs/v0-run2/03-bad-image-tag/r1](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/03-bad-image-tag/r1). Steps 1 to 4 are housekeeping: list open incidents, a memory search, list connected clusters, a second memory search (no matches, this org is new). Step 5 is the first real command: ```text $ kubectl get deployment frontend -n shop -o wide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR frontend 2/2 1 2 78s nginx nginx:1.27-alpne app=frontend ``` The image column already shows the typo, but Aurora does not jump. Step 6 lists the pods and finds the one that is not running: ```text $ kubectl get pods -n shop -o wide NAME READY STATUS RESTARTS AGE frontend-76d98597b9-jkkw9 1/1 Running 0 82s frontend-76d98597b9-rb9z6 1/1 Running 0 82s frontend-7b6c998456-phg6r 0/1 ImagePullBackOff 0 75s ``` Step 7 describes that pod. The container state is `Waiting`, reason `ImagePullBackOff`, and the events carry the registry's answer, which Aurora quotes in its report: "docker.io/library/nginx:1.27-alpne: not found". Step 8 is the one a careful engineer would run next, a jsonpath over the ReplicaSets to compare old and new side by side: ```text frontend-76d98597b9 nginx:1.27-alpine replicas=2 ready=2 frontend-7b6c998456 nginx:1.27-alpne replicas=1 ready= ``` Step 9 is the one miss in this run. Aurora asked for `kubectl rollout history deployment/frontend -n shop` and the command policy refused it: ```text Command blocked by organization policy: Mutating kubectl operations ``` `rollout history` is read-only. The policy's deny rule matches the whole `rollout` verb, so it blocks `history` and `status` along with `restart` and `undo`. Aurora noted the refusal under "Not Checked" and worked around it, which is why it diffed the ReplicaSets in step 8. Steps 10 and 11 describe the Deployment (revision 2, "2 desired | 1 updated | 3 total | 2 available | 1 unavailable", RollingUpdate with 25% max unavailable) and list the nodes (all three Ready, v1.37.0). Step 12 is a check we did not expect: a `curl` to Docker Hub's API for the tag `1.27-alpine`, which returned 200, confirming the correctly spelled tag exists. Step 13 writes a memory note for the org. The report named the typo, quoted the kubelet error, said there was no user-facing outage because the two old pods stayed Available, and listed node health, resource pressure and scheduling as ruled out with the evidence for each. Both the judge and the person marked it correct. Run r2 reached the same answer in 77 seconds and 11 tool calls. ## A run that went wrong: two faults, one alert Scenario 12 injects two faults at once: the worker is OOMKilled after a batch-size change, and the frontend Service's selector is changed so it selects no pods. The alert only mentions the worker. Both runs found the alerted fault and declared everything else healthy without checking. Both runs found the OOMKilled worker and the BATCH_SIZE change. Neither ran `get svc` or `get endpoints`, and neither looked at the load generator, which was getting errors from the frontend. r1's 18 tool calls were pods, describe, logs, events, ReplicaSets, the worker Deployment, a ConfigMap, a blocked `rollout history`, `top nodes` and nodes. Then the report said: > Other workloads in the shop namespace (api, frontend) remained healthy and unaffected [5, 14]. r2 said the same: "Other workloads in the namespace (api, frontend pods) were running normally and unaffected [4]." Pod status was the only evidence for both claims. A real second outage went unreported, and the report said it was not there. The transcripts are in [runs/v1-run1](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1). This is the failure mode we care most about. "All pods Running" is not "the service works", and an agent that writes "unaffected" from pod status alone will miss every Service, endpoint and routing fault. The scenario was written to tempt exactly this, and it did. **Scenario 19, run 1: the right clue, the wrong conclusion.** A ConfigMap was edited and only one of two frontend pods was restarted, so one pod serves the new config and one the old. Aurora read the frontend-nginx ConfigMap, saw `server api:8000` against a Service on 8080, and tied it to the failing pod. It did not work out that only the recreated pod had loaded the edited ConfigMap. It tried to read the live config inside the pod and the policy blocked it: ```text $ kubectl exec frontend-76d98597b9-g5zrg -n shop -c frontend -- cat /etc/nginx/conf.d/default.conf Command blocked by organization policy: Mutating kubectl operations ``` So the report left it open: "This discrepancy was not fully reconciled, the live nginx configuration inside the running pod could not be inspected." Its own `get pods` output had the clue it needed: the failing pod was 9 seconds younger than the healthy one. Run 2 got it right, with 28 tool calls. **Scenario 11, run 2: graded partial for not checking the red herring.** Aurora found the targetPort 9090 against the container's 8080. The worker's noisy errors are in the scenario to tempt a wrong answer. Aurora never looked at the worker: "the redis, worker, and loadgen deployments ... were not examined". The judge and the second model both marked "the worker's errors are not the cause" as partial. You could argue this run deserves correct, since it was not misled. We kept the rubric as written and report the verdict. ## Patterns across the 38 graded runs - `kubectl rollout history` was blocked by the default `observability_only` policy in 26 of 38 runs (16 of 20 in v0, 10 of 18 in v1; 27 calls). "What changed" is the first question in most incidents, and Aurora had to answer it by diffing ReplicaSets instead. - Fixes: 34 of 38 runs suggested an acceptable fix, 2 were partial (both 12 runs fixed only the OOM) and 2 had no suggestions (04 r1 and 09 r2 in v0). - The second model found claims the evidence does not support in 15 of the 18 v1 reports, none of them part of the root cause: "both cluster nodes" (there are three), "pulled successfully" for an image that had no Pulled event, a quota timeline with the rollout before the quota, a worker that "processed Redis-backed batches" when the worker never connects to redis, and "34Mi and climbing" from one sample. - Aurora's "Not checked" sections were honest. When it could not verify something it said so instead of guessing. **A superseded run, kept for the record.** Before v0-run2 there was a v0-run1 in one shared org, which is why it does not count. Its scenario 09 run shows the kind of failure that does not look like a wrong answer. Aurora found the PVC with the missing StorageClass by step 10. Then it made 25 `write_artifact` calls, many with no `content` argument and some with test strings like "placeholder" and "test2". That took about 11 of 15 minutes and $2 of $2.61. The report's root cause never named the PVC and said redis was "running, not ready" (it was Pending). The investigation notes and suggested next steps did name the PVC and the missing `fast-ssd` StorageClass, so graded on everything Aurora showed, the run is correct. The causes, from the fix branch work: Aurora set no `max_tokens` for Bedrock, 19 agent responses stopped at exactly 4096 output tokens and the tool arguments were cut off mid-`content`; pydantic rejected the call before the tool ran, so the model only saw "Field required"; and artifact writes counted as evidence, which pushed the real evidence out of the summary's citation window. It did not happen again in v0-run2 (0 `write_artifact` calls in 20 runs). ## The fix branch Three one-patch commits on `87d22baf`, in [patches/](https://github.com/Arvo-AI/aurora-lab/tree/main/patches): the policy denies only `rollout (restart|undo|pause|resume)` and allows `rollout (history|status)`; a `write_artifact` call with cut-off arguments reaches the tool and returns an error that says so; artifact and memory write receipts no longer count as evidence. Scenarios 05, 09 and 10 were rerun on the patched build. | Scenario | Current code | Fix branch | Median time, before / after | Median cost, before / after | |---|---|---|---|---| | 05 Invalid nginx directive | 2/2 | 2/2 | 136 s / 145 s | $0.32 / $0.36 | | 09 PVC with missing StorageClass | 2/2 | 2/2 | 280 s / 182 s | $0.39 / $0.44 | | 10 Wrong REDIS_HOST | 2/2 | 2/2 | 244 s / 100 s | $0.28 / $0.32 | What it shows: `rollout history` ran 14 times across the 6 runs and was never denied. What it does not show: the other two patches were not exercised, because the artifact loop never came back, so only their unit tests support them; and the time differences mostly come from background contention the earlier runs had and this run did not, not from the fix. ## What this does and does not show - kind is not production. Three nodes, one namespace, one small app, no real traffic and no history. Production clusters are noisier, which is harder, and also have dashboards and runbooks, which Aurora can use. - The vendor wrote the scenarios and graded its own product. The v1 scenarios were written to be harder, but we still chose them. - n=2 per scenario. 20 of 20 on v0 does not mean the error rate is 0. - One person reviewed the verdicts, and that person works at Arvo. The judge is an LLM, and the second review is an LLM from the same model family. - Durations are inflated for some runs. Each kubeconfig upload scheduled a background "prediscovery" job that shared worker slots with the investigations; the lab's attempt to stop those jobs did nothing until late in the day. Runs that waited for a slot include 08 r1 (362 s) and 09 r1 (406 s). The jobs ran in separate orgs and finished after the investigations, so they could not have fed Aurora answers. - Money: $16.77 for all 44 investigations. The whole day, including about $30 of prediscovery and about $6.60 of judge calls, came to about $54. No customer data was used anywhere in this work, and none will be in later versions. ## Rerun it ```shell git clone https://github.com/Arvo-AI/aurora-lab cd aurora-lab ./lab.sh up # 3-node kind cluster ./lab.sh reset # deploy the shop app and wait for 200s ./lab.sh break 03-bad-image-tag python3 runner/run.py # capture state, send the alert, save the trace python3 runner/grade.py # judge the report against scenario.yaml ``` The [README](https://github.com/Arvo-AI/aurora-lab#readme) covers the Aurora side (a local Aurora build, a kubeconfig upload, the alert). If you run another agent on the same scenarios, we want the results, including the ones that beat Aurora. ## Related Two of these failures have reproduced runbooks with the lab trace on them: [OOMKilled](/runbooks/fix-oomkilled-kubernetes) and [ImagePullBackOff](/runbooks/fix-imagepullbackoff-kubernetes). The lab is also how the [Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt) comparison gets rebuilt on hands-on runs. What Aurora produces at the end of an investigation is described at [root cause analysis](/rca). ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) > **Sourcing note.** Every number, verdict and quote above comes from the lab repo as of 2026-10-02: [REPORT.md](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md) (human-reviewed verdicts), [FINDINGS.md](https://github.com/Arvo-AI/aurora-lab/blob/main/FINDINGS.md), and the run folders [runs/v0-run2](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2), [runs/v1-run1](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1) and [runs/v0-fix-run1](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-fix-run1), each with `steps-full.json`, `aurora-rca.md` and `grade.json` per run. Versions: kind v0.33.0, Kubernetes v1.37.0, Aurora commit 87d22baf, RCA model bedrock/us.anthropic.claude-sonnet-5, judge us.anthropic.claude-opus-5-5. Written 2026-10-08. --- ## Cleric vs Resolve.ai: AI SRE Compared (2026) URL: https://www.aurorasre.ai/blog/cleric-vs-resolve-ai Published: 2026-10-01 Author: Noah Casarotto-Dinning, CEO at Arvo AI > Cleric is a per-credit AI SRE that opens fix PRs for approval. Resolve.ai is an autonomous on-call AI SRE funded past $190M. How the two differ, with pricing. > ## Key Takeaways > > - **Cleric and Resolve.ai are both commercial AI SREs that go on call, but they diverge on pricing transparency, how much authority the agent holds, and company scale.** Cleric [publishes per-credit pricing](https://cleric.ai/pricing); Resolve.ai publishes none and sells through sales. > - **Cleric meters usage per credit.** Tiers run "$100 / month" for "100 credits" up to "$2,000 / month" for "2,000 credits," where "Credits cost $1 on monthly plans" and an issue investigation is "10 credits." > - **Cleric keeps write access opt-in.** Its site says "Read-only by default" and "Read access by default, write access when you're ready," and it "opens PR" for a fix once approved. > - **Resolve.ai markets autonomous resolution.** Its site says the agent "Goes on-call on your behalf and autonomously resolves incidents" and "Holds the pager. Triages and mitigates alerts, escalating only when you are needed." > - **Resolve.ai is the larger-funded company.** It [announced](https://resolve.ai/news) a "$40M series A extension" on "April 16, 2026," bringing "more than $190M in funding" at a "$1.5B" valuation, with founders described as "co-creators of OpenTelemetry." > - **The real choice is operating model, not a feature checklist.** Metered and staged-authority versus autonomous and enterprise-priced, and how much you want the agent to do without a human in the loop. **Cleric and Resolve.ai are both commercial AI SREs that go on call and investigate incidents, but Cleric meters usage per credit with opt-in write access, while Resolve.ai sells an autonomous agent and publishes no price.** The two are compared because both automate the first pass of an incident and both offer to hold the pager, yet they differ on cost model, how much the agent is allowed to do on its own, and company scale. ## What is Cleric? Cleric is a commercial, hosted AI SRE that takes alerts, investigates them, and can propose fixes through review. Its [site](https://cleric.ai/) positions it around prevention and on-call relief: "Agents that check every production change, then fix regressions before customers notice," and "Give the pager to Cleric." The access model is staged: "Read-only by default. Every action logged, every investigation auditable," and "Read access by default, write access when you're ready." When it acts, it does so through review, with the site showing steps like "Cleric opens PR #484 for your approval" and describing how it "brings you the cause, the evidence, and a fix PR to review." Cleric is operated by "Agentik, Inc. dba Cleric" and states it is "SOC 2 Type II compliant, with regular manual penetration testing." It was [named](https://cleric.ai/) a "Gartner® Cool Vendor in AI for SRE and Observability." ## What is Resolve.ai? Resolve.ai is a commercial AI SRE built to go on call and resolve incidents with less human involvement. Its [site](https://resolve.ai/) describes the agent as one that "Goes on-call on your behalf and autonomously resolves incidents," that "Holds the pager. Triages and mitigates alerts, escalating only when you are needed," and that "Identifies the root cause quickly, mitigates and restores service in minutes, not hours." It also "Fixes recurring issues by finding root problems and giving the production context to coding agents" and lets teams "Build custom SRE agents for any automation you need, fully managed for you." The company is well funded: its [news page](https://resolve.ai/news) states it was "Founded October, 2024," announced a "$40M series A extension" on "April 16, 2026" for "more than $190M in funding" at a "$1.5B" valuation, with investors including DST Global, Lightspeed, Greylock, and Unusual Ventures, and a team that "includes Co-creators of OpenTelemetry." Resolve.ai does not publish pricing; it is sold through its sales team. Note that the outcome figures on Resolve.ai's site, such as cutting downtime or on-call load by a stated percentage, are the vendor's own marketing numbers rather than independently verified results, so they are not restated as fact here. ## How do Cleric and Resolve.ai compare? The clearest axis is operating model: how the agent is priced and how much authority it holds by default. | Dimension | Cleric | Resolve.ai | |---|---|---| | Model | Commercial, hosted (Agentik, Inc. dba Cleric) | Commercial, hosted | | Pricing | Public, per credit: "$100 / month" to "$2,000 / month" plus Enterprise | Not published; sales-led | | Authority posture | "Read-only by default," write access opt-in | "autonomously resolves incidents," escalates "only when you are needed" | | Acts on systems | Investigates, then "opens PR" for approval | "Triages and mitigates alerts," hands fixes to coding agents | | On-call | "Give the pager to Cleric" | "Holds the pager" | | Compliance / scale | "SOC 2 Type II compliant" | "$1.5B" valuation, "more than $190M in funding" | The table makes the trade explicit. Cleric gives a metered agent whose default is read-only and whose fixes land as a pull request a human approves, so cost and authority are both easy to bound. Resolve.ai markets a more autonomous agent that triages and mitigates on its own and escalates only when needed, backed by a much larger funding base, with pricing you get from a sales conversation rather than a public page. ## How much does each cost? Only one of the two publishes a price. Cleric's [pricing page](https://cleric.ai/pricing) lists Starter at "$100 / month" for "100 credits / month," Team at "$600 / month" for "600 credits / month," Pro at "$2,000 / month" for "2,000 credits / month," and Enterprise at "Custom." The unit economics are stated plainly: "Credits cost $1 on monthly plans," an "Issue investigation" is "10 credits" (so "$10"), and checking "one production change for regressions" is also "10 credits." Unused credits "roll over for one month, up to twice your monthly allocation," and a trial gives "500 evaluation credits · No time limit." Resolve.ai publishes no pricing page, so its cost is a sales-led quote rather than a figure you can map to incident volume in advance. For buyers, that is itself a planning difference: Cleric lets a team estimate a monthly bill from its expected incident count, while Resolve.ai requires a conversation to size. ![Diagram comparing Cleric and Resolve.ai across two axes. The vertical axis is pricing transparency, from public per-credit pricing at the bottom to sales-led unpublished pricing at the top. The horizontal axis is agent authority, from read-only by default with opt-in write on the left to autonomous mitigation and resolution on the right. Cleric sits bottom-left: hosted and SOC 2 Type II compliant, read-only by default with write access opt-in, priced per credit from one hundred to two thousand dollars per month, opening a pull request for approval when it proposes a fix. Resolve.ai sits top-right: a 1.5 billion dollar valuation on more than 190 million dollars of funding, an agent that holds the pager and autonomously resolves incidents, escalating only when needed, with no published price.](/blog/cleric-vs-resolve-ai.svg) ## Which should an SRE team choose? It depends on how much the agent should do without a human and how predictable the bill needs to be. A team that wants a metered agent, a read-only default, and fixes that always arrive as a pull request to approve fits Cleric, and can estimate cost from incident volume up front. A team that wants an agent to hold the pager and mitigate incidents more autonomously, and is prepared to run an enterprise sales process to price it, fits Resolve.ai. A third option keeps the control of open source while still letting the agent propose a fix. [Aurora](https://github.com/Arvo-AI/aurora) is built that way: a single investigation agent by default with an opt-in multi-agent orchestrator, powerful tools behind an explicit allowlist, sandboxed Kubernetes execution when enabled, and a structural chokepoint that blocks mutating writes when no human is present, so remediation only ever lands as a pull request a human merges. For how to weigh these, see [how to evaluate an AI SRE platform](/blog/how-to-evaluate-ai-sre-platform), the [open-source Resolve.ai alternative](/blog/resolve-ai-alternative-open-source), [HolmesGPT vs Cleric](/blog/holmesgpt-vs-cleric), and [Aurora root cause analysis](/rca). ## The summary Cleric and Resolve.ai both automate incident investigation and both offer to hold the pager, but they sit at different points. Cleric is metered per credit ($100 to $2,000 a month, $10 per investigated issue), read-only by default with opt-in write, and lands fixes as an approved pull request. Resolve.ai markets a more autonomous agent that mitigates incidents on its own, is backed by more than $190M in funding at a $1.5B valuation, and publishes no price. Decide on operating model and authority, not a feature scorecard. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** Cleric's positioning, read-only and write-access wording, PR-approval flow, SOC 2 Type II statement, Gartner Cool Vendor mention, and all pricing (tiers, credits, per-credit cost, issue-investigation cost, rollover, trial) are quoted from cleric.ai and its pricing page, verified 1 October 2026. Resolve.ai's product wording, founding date, funding total, valuation, and OpenTelemetry-founder description are quoted from resolve.ai and resolve.ai/news, verified the same day. Resolve.ai's own outcome percentages are vendor marketing and are not restated as fact. Pricing and funding change; re-check each page before relying on a figure. Aurora's controls are described from its open-source repository. --- ## PagerDuty vs Jira Service Management (2026) URL: https://www.aurorasre.ai/blog/pagerduty-vs-jira-service-management Published: 2026-10-01 Author: Noah Casarotto-Dinning, CEO at Arvo AI > PagerDuty is an operations and on-call platform. Jira Service Management is an ITSM service desk with on-call built in. How the two compare for incident response, with pricing. > ## Key Takeaways > > - **PagerDuty and Jira Service Management overlap on incident response and on-call but start from different jobs: PagerDuty is an operations platform built around alerting and on-call, while Jira Service Management is an IT service desk that added on-call and incident management.** This decision now faces every team migrating off Opsgenie. > - **PagerDuty centers on on-call and alerting.** Its site describes "On-call scheduling and escalations" that "safeguard the business with 24x7 coverage" and notifications "via custom-defined methods such as SMS messages, mobile app push notifications, phone calls, or emails." > - **Jira Service Management bundles incident management into a service desk.** Atlassian describes "Incident management for high-velocity teams" that can "Aggregate incoming alerts and signals from all your monitoring, service desk and logging applications" and route them "using powerful routing rules." > - **PagerDuty's AIOps and SRE Agent are separate from the base plan.** Its [AIOps page](https://www.pagerduty.com/platform/aiops/) markets event correlation and noise reduction as an add-on, and its SRE Agent "Detects, triages, diagnoses incidents and performs approved remediation." > - **Pricing uses different units.** PagerDuty's incident-management Professional tier is "$25" monthly or "$21" per user per month annual; Jira Service Management prices per agent and is tightly coupled to the Atlassian suite. > - **This is now an Opsgenie migration decision.** Atlassian is retiring Opsgenie, so many teams are choosing between consolidating into Jira Service Management or moving to PagerDuty. **PagerDuty is an operations platform built around on-call, alerting, and incident response, while Jira Service Management is an IT service desk that added on-call and incident management to a ticketing and ITSM core.** They are compared most often now because Atlassian is retiring Opsgenie, which forces its users to pick a replacement, and the two leading paths are consolidating into Jira Service Management or adopting PagerDuty. ## Why are teams comparing PagerDuty and Jira Service Management now? Because Opsgenie is being discontinued and its users have to move. Atlassian's [announcement](https://www.atlassian.com/blog/announcements/evolution-of-it-operations) set an end of sale of "June 4th, 2025," after which "you'll no longer be able to purchase or sign up for Opsgenie," and an end of support of "April 5th, 2027," when "access to Opsgenie will be shut off, and the product will no longer accessible. Any unmigrated data in Opsgenie will also be deleted." Atlassian is steering Opsgenie customers into Jira Service Management with "a personalized in-product migration tool," which puts the PagerDuty-versus-JSM choice in front of every Opsgenie team. The migration mechanics are covered in depth in the [Opsgenie 2026 guide](/blog/opsgenie-complete-guide-2026); this comparison is about the two replacements themselves. ## What does PagerDuty do? PagerDuty is an operations platform whose core is on-call management and incident response. Its [on-call page](https://www.pagerduty.com/platform/on-call-management/) describes scheduling and escalations as "key to equitably distributing workloads while safeguarding the business with 24x7 coverage," lets teams "Set up on-call schedules to support any kind of standard or custom rotation (weekly, weekend, follow-the-sun, etc.)," and escalates automatically: "If an alert is not acknowledged, it is passed to the next responder." It notifies responders "via custom-defined methods such as SMS messages, mobile app push notifications, phone calls, or emails." Its [incident-management page](https://www.pagerduty.com/platform/incident-management/) frames the product as "End-to-end orchestration for rapid issue resolution" with "Guided remediation throughout the entire incident lifecycle." Advanced correlation is a separate [AIOps](https://www.pagerduty.com/platform/aiops/) capability that "leverages AI algorithms to automatically correlate and deduplicate large volumes of incoming events," and PagerDuty also markets an [SRE Agent](https://www.pagerduty.com/platform/ai-agents/) that "Detects, triages, diagnoses incidents and performs approved remediation." PagerDuty states it has "750+ platform integrations." ## What does Jira Service Management do? Jira Service Management is an IT service management (ITSM) desk from Atlassian that has incident management and on-call built on top of its ticketing core. Its [incident-management page](https://www.atlassian.com/software/jira/service-management/features/incident-management) describes "Incident management for high-velocity teams" that lets a team "respond, resolve, and learn from incidents." It can "Aggregate incoming alerts and signals from all your monitoring, service desk and logging applications," "Automatically reduce noise and surface the most important issues to the right team," and "Notify the right people using powerful routing rules and multiple communication channels including SMS, email, push notifications, chat and voice calls." It also supports "incident swarming and on-call capabilities" and lets teams "attach runbooks to alerts." The on-call and alerting capabilities inside Jira Service Management are the direct successor to Opsgenie's: Atlassian [documents](https://support.atlassian.com/opsgenie/docs/managing-operations-in-compass-and-jira-service-management-at-the-same-time/) that "When you migrate from Opsgenie to Jira Service Management or Compass, Operations will function across both products," and that "All alerts, on-call schedules, integrations, and policies will sync across both products." The distinguishing feature is the deep tie to the rest of Atlassian: Jira issues, Confluence, and the service desk sit in one suite. ## How do PagerDuty and Jira Service Management compare? The split is operations platform versus service desk. One is built outward from on-call; the other is built outward from ticketing and ITSM. | Dimension | PagerDuty | Jira Service Management | |---|---|---| | Starting point | Operations and on-call platform | ITSM service desk with on-call added | | On-call and escalation | Core; custom rotations, auto-escalation on no-ack | Built in; "incident swarming and on-call capabilities" | | Alerting channels | "SMS messages, mobile app push notifications, phone calls, or emails" | "SMS, email, push notifications, chat and voice calls" | | Alert correlation / AIOps | Separate AIOps add-on, "correlate and deduplicate" events | "Automatically reduce noise and surface the most important issues" | | AI agent | SRE Agent, "performs approved remediation" | Within the Atlassian AI / Rovo layer | | Suite integration | "750+ platform integrations" | Native Jira, Confluence, and service-desk suite | | Pricing unit | Per user ("$25" monthly, "$21" annual, Professional) | Per agent, coupled to the Atlassian suite | The trade is focus versus consolidation. PagerDuty is the deeper operations and on-call tool, with mature escalation, a large integration catalog, and AIOps and an SRE Agent available as added capabilities. Jira Service Management is the better fit for a team that wants on-call and incident management inside the same suite as its tickets, service desk, and documentation, which is also the lowest-friction path for an Opsgenie team already in the Atlassian ecosystem. ## How do PagerDuty and Jira Service Management price? They bill on different units, so compare the unit, not the sticker. PagerDuty's [pricing page](https://www.pagerduty.com/pricing/) lists its incident-management Professional tier at "$25" per user per month monthly, or "$21" per user per month annual, with a Free tier for "5 users" that includes "1 on-call schedule, 1 escalation policy." Its AIOps, Status Pages, and other capabilities are add-ons sold separately, so the Professional price is not an all-in figure. Jira Service Management prices per agent and is tightly coupled to the Atlassian suite; because its public pricing is rendered dynamically, confirm the current per-agent figures on [Atlassian's pricing page](https://www.atlassian.com/software/jira/service-management/pricing) before relying on them. The practical point is that PagerDuty bills per responder-style user with AIOps on top, while Jira Service Management bills per agent and derives much of its value from being inside the suite you already run. ![Diagram contrasting PagerDuty and Jira Service Management. On the left, PagerDuty is drawn as an operations platform: a core of on-call scheduling and escalation, with alerting over SMS, phone, push, and email, and AIOps event correlation plus an SRE Agent shown as separate add-on layers, priced per user at twenty-one to twenty-five dollars per month with seven hundred and fifty plus integrations. On the right, Jira Service Management is drawn as a service desk: a ticketing and ITSM core with incident management and on-call built on top, alert aggregation and noise reduction, and native ties to Jira and Confluence, priced per agent inside the Atlassian suite. A banner across the top notes that Opsgenie is being retired, with end of support on April 5, 2027, which forces this choice.](/blog/pagerduty-vs-jira-service-management.svg) ## Which should a team choose? It depends on where the team already lives and how deep its on-call needs run. A team that wants the strongest standalone on-call and incident-response platform, with mature escalation, a large integration catalog, and the option to add AIOps correlation and an SRE Agent, fits PagerDuty. A team already standardized on Jira and Confluence, especially an Opsgenie team taking the in-product migration path, gets the least friction from consolidating into Jira Service Management and keeping on-call inside the suite. Either way, both are alerting and workflow tools rather than investigators: they route the page and track the incident, but they do not find the root cause for you. An AI SRE sits alongside either one to do the investigation. [Aurora](https://github.com/Arvo-AI/aurora) is open source, runs a single investigation agent by default with an opt-in multi-agent orchestrator, and lands any fix as a pull request a human merges. For related reading, see the [Opsgenie 2026 guide](/blog/opsgenie-complete-guide-2026), the [PagerDuty alternative for root cause analysis](/blog/pagerduty-alternative-root-cause-analysis), [open-source incident management](/blog/open-source-incident-management), and [Aurora root cause analysis](/rca). ## The summary PagerDuty is an operations platform built around on-call and alerting, with AIOps and an SRE Agent as add-ons and per-user pricing (Professional "$25" monthly, "$21" annual). Jira Service Management is an ITSM service desk with on-call and incident management built in, priced per agent and most valuable inside the Atlassian suite. With Opsgenie retiring by April 5, 2027, the choice comes down to whether a team wants the deeper standalone operations tool or the lowest-friction consolidation into the suite it already runs. Neither investigates incidents; an AI SRE does that alongside either one. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** PagerDuty's on-call, incident-management, AIOps, SRE Agent, integration-count, and pricing wording are quoted from pagerduty.com pages, verified 1 October 2026. Jira Service Management's incident-management and alerting wording is quoted from atlassian.com, and the Opsgenie end-of-sale and end-of-support dates and migration wording are quoted from Atlassian's official announcement and support docs, verified the same day. Jira Service Management per-agent pricing is rendered dynamically and was not fetchable at verification time, so no figure is quoted; confirm it on Atlassian's pricing page. Vendor marketing figures such as PagerDuty's integration count are the vendor's own claims. Pricing changes; re-check each page before relying on a figure. --- ## Is AI Reducing MTTR for SRE Teams? (2026 Data) URL: https://www.aurorasre.ai/blog/is-ai-reducing-mttr-dynatrace-2026 Published: 2026-10-01 Author: Noah Casarotto-Dinning, CEO at Arvo AI > Dynatrace's State of SRE 2026 survey finds half of SREs use AI for automated incident response, yet AI is falling short on MTTR and cost. What the numbers say. > ## Key Takeaways > > - **A 2026 survey of 919 SRE leaders finds AI adoption in incident response is now widespread, but the survey reports AI is underdelivering on the two outcomes teams bought it for: lower MTTR and lower cost.** The findings are from Dynatrace's [State of SRE and Platform Engineering 2026](https://www.dynatrace.com/news/press-release/state-of-sre-platform-engineering-2026/), which it commissioned. > - **Half of SREs now use AI for incident response.** The report states "Half of SREs now use AI-powered capabilities for automated incident response." > - **Yet AI is reported as falling short on the headline outcomes.** The report says AI is "delivering less impact than expected in lowering costs and reducing mean time to resolution (MTTR)." > - **AI-model monitoring has become the top SRE use case.** The report finds "67% of SREs now naming AI model monitoring their top use case," reflecting that SREs increasingly operate AI systems, not just use AI tools. > - **SRE practice is mature on fundamentals.** "89% use service-level objectives (SLOs) across at least some teams or systems," and "92% of organizations report executive leadership support for SRE initiatives." > - **Read the survey as commissioned and fielded earlier than published.** It was "conducted and analyzed by Qualtrics partner Y2 on behalf of Dynatrace," with fieldwork from October 2025 to January 2026 and publication in August 2026. **Dynatrace's State of SRE 2026, a survey of 919 SRE leaders, reports that AI incident-response adoption reached half of SREs while delivering less impact than expected on cost and mean time to resolution.** The gap between adoption and payoff is the survey's central tension for anyone deciding whether an AI SRE tool actually moves the number that matters. ## What does the 2026 data say about AI and MTTR? It says adoption is high and the headline payoff is lagging. The [Dynatrace report](https://www.dynatrace.com/news/press-release/state-of-sre-platform-engineering-2026/) states that "Half of SREs now use AI-powered capabilities for automated incident response," which puts AI squarely inside the incident workflow for a large share of teams. In the same release, Dynatrace reports that AI is "falling short on cost reduction and MTTR," "delivering less impact than expected in lowering costs and reducing mean time to resolution (MTTR)." Adoption has outpaced measurable results on the two outcomes teams most often justify the spend with. That gap is the useful signal for a buyer. It does not say AI has no value in incident response; it says the specific promise of a lower MTTR has, in this survey of practitioners, not landed as expected yet. Treat a vendor's MTTR-reduction claim as a hypothesis to test against your own incidents, not a settled result. ## What are SREs actually using AI for? Increasingly, to operate AI systems themselves. The report finds "67% of SREs now naming AI model monitoring their top use case," and that monitoring "for model performance and accuracy" is "already the most common AI-powered capability among SREs (58%)." The job is shifting: SREs are not only adopting AI tools, they are increasingly on the hook for the reliability of the AI their companies ship. Automated incident response, at half of SREs, is a major use case but not the top one. That reframing matters for tool selection. An AI SRE is being evaluated by people who now also run AI in production and understand its failure modes firsthand, which raises the bar on transparency and on evidence that a tool's output can be trusted. ## How mature is SRE practice overall? Mature on fundamentals, by this survey. The report finds "89% use service-level objectives (SLOs) across at least some teams or systems" and "92% of organizations report executive leadership support for SRE initiatives." SLOs are near-universal and executive backing is broad, which means AI tooling is being adopted into established practice rather than filling a vacuum. The report also notes that "73% of SRE and platform engineering teams now collaborate and share responsibilities," a sign of converging platform and reliability functions. | Finding (Dynatrace State of SRE 2026) | Figure | |---|---| | SREs using AI for automated incident response | "Half" | | SREs naming AI model monitoring their top use case | "67%" | | Model performance/accuracy monitoring as AI capability | "58%" | | Teams using SLOs across at least some systems | "89%" | | Organizations with executive support for SRE | "92%" | | AI impact on cost and MTTR | "delivering less impact than expected" | ![Diagram summarizing Dynatrace's State of SRE 2026 survey of 919 SRE leaders. On the left, an adoption column shows high numbers: half of SREs use AI for automated incident response, sixty-seven percent name AI model monitoring their top use case, fifty-eight percent monitor model performance and accuracy, eighty-nine percent use SLOs, and ninety-two percent report executive support. On the right, an outcome column shows the gap: AI is delivering less impact than expected on cost reduction and on mean time to resolution. An arrow between the two columns is labeled the adoption-versus-payoff gap. A footnote notes the survey was commissioned by Dynatrace, fielded October 2025 to January 2026, and published August 2026.](/blog/is-ai-reducing-mttr-dynatrace-2026.svg) ## Why would AI adoption outrun MTTR gains? The survey reports the gap but not its mechanism, so what follows is interpretation rather than a survey finding. One plausible reason is that MTTR is dominated by stages AI touches least: coordination, change approval, and the human decision to act, not just the investigation an AI accelerates. Another is measurement itself. Without a consistent baseline, a team cannot attribute an MTTR change to a tool, which is the same reason an AI SRE vendor cannot credibly publish a single accuracy or MTTR-reduction figure that transfers across environments. This is why evaluation discipline matters more than a headline number. Measuring whether AI reduces your MTTR means instrumenting your own incident timeline and comparing like with like, a point developed in [measuring AI SRE without ground truth](/blog/measuring-ai-sre-without-ground-truth) and in [how to evaluate an AI SRE platform](/blog/how-to-evaluate-ai-sre-platform). ## What should a buyer take from this? Buy for the workflow you can verify, not the number in the pitch. AI in incident response is now mainstream, but this survey of 919 practitioners says the MTTR and cost payoff is still trailing adoption. A tool that shows its evidence, states what it ruled out, and lands actions through human approval is one you can actually measure; a tool that only reports a confident conclusion is one you cannot. [Aurora](https://github.com/Arvo-AI/aurora) is built around that measurability: it tracks per-investigation cost, produces a per-incident execution timeline, requires its sub-agent findings to include a "What I ruled out" section and a self-assessed strength, and lands any fix as a pull request a human merges. It publishes no accuracy or MTTR figure, because no benchmark exists that would make such a number defensible. For the wider picture, see [AI SRE accuracy benchmarks](/blog/ai-sre-accuracy-benchmarks) and [rising SRE toil in 2026](/blog/sre-toil-rising-2026). ## The summary Dynatrace's State of SRE and Platform Engineering 2026, surveying 919 SRE leaders, finds half of SREs now use AI for automated incident response and 67% name AI model monitoring their top use case, against a mature backdrop of 89% SLO use and 92% executive support. But the same survey reports AI is delivering less impact than expected on cost and MTTR. The lesson for buyers is to measure AI's effect on their own incidents rather than trust a vendor's MTTR claim, and to prefer tools whose output can be verified. The survey is Dynatrace-commissioned, fielded October 2025 to January 2026. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** All survey figures and quoted wording are from Dynatrace's State of SRE and Platform Engineering 2026 press release, verified 1 October 2026: half of SREs using AI for automated incident response, 67% naming AI model monitoring their top use case, 58% monitoring model performance and accuracy, 89% using SLOs, 92% with executive support, and AI "delivering less impact than expected in lowering costs and reducing mean time to resolution (MTTR)." The survey is vendor-commissioned, "conducted and analyzed by Qualtrics partner Y2 on behalf of Dynatrace," with N=919, fieldwork October 2025 to January 2026, and publication in August 2026. The reasoning about why adoption may outrun MTTR gains is interpretation, not a survey finding. Aurora's capabilities are described from its open-source repository; no Aurora accuracy or MTTR figure is claimed because no such benchmark exists. --- ## AI SRE and Incident Tool Pricing Compared (2026) URL: https://www.aurorasre.ai/blog/ai-sre-incident-tool-pricing-2026 Published: 2026-09-28 Author: Noah Casarotto-Dinning, CEO at Arvo AI > What Cleric, FireHydrant, incident.io, Rootly, and PagerDuty actually charge, from public pricing pages. Per-credit, per-responder, and per-user models compared. > ## Key Takeaways > > - **Incident and AI SRE tools price on three different units, which makes headline numbers hard to compare: per user, per responder, and per credit.** The unit matters more than the sticker price. > - **Cleric prices per credit.** Tiers run "$100 / month" for "100 credits" up to "$2,000 / month" for "2,000 credits," where "Credits cost $1 on monthly plans" and an issue investigation is "10 credits." > - **FireHydrant prices per responder.** Its Pro plan is "$25" per "/responder/month, billed annually," with a "14-day free trial" and no card required. > - **incident.io prices per user, with on-call as an add-on.** Team is "$19/user/month" monthly or "$15/user/month" annual; Pro is "$25/user/month"; on-call adds "$10" to "$20" per user. > - **Rootly publishes response and on-call prices but not its AI SRE price.** Incident Response and On-Call Essentials are each "$20 per user/mo"; the "AI SRE" product is "Contact us." > - **PagerDuty publishes incident management per user, but AIOps is a separate add-on.** Professional is "$25" monthly or "$21" annual per user; AIOps has no standalone per-user price on the pricing page. > - **Open-source tools have no license fee at all.** The cost is the LLM tokens and infrastructure you run yourself. **AI SRE and incident management tools do not share a pricing unit: some charge per user, some per on-call responder, some per credit consumed, and open-source options charge nothing for the software itself.** Comparing them means normalizing to the unit each vendor actually bills, because a per-credit and a per-user plan can look similar on the sticker and diverge sharply on the bill. ## How do AI SRE and incident tools price? They fall into four billing models, and the model determines what scales your bill. Per-user pricing scales with headcount. Per-responder pricing scales with how many people are on call. Per-credit pricing scales with usage, such as how many incidents the agent investigates. Open-source pricing scales with the LLM tokens and compute you consume, with no license fee. Knowing the unit is the first step in any evaluation, because it changes which lever raises your cost. A large team with few on-call engineers pays very differently under per-user versus per-responder pricing, and a team with heavy incident volume pays very differently under per-credit versus flat per-user pricing. ## What do AI SRE and incident tools actually charge? The figures below are quoted from each vendor's public pricing page, verified 28 September 2026. Where a product's price is not published, that is stated rather than estimated. | Tool | Unit | Published price (verbatim) | Notes | |---|---|---|---| | Cleric | Per credit | "$100 / month" (100 credits) to "$2,000 / month" (2,000 credits) | "Credits cost $1"; issue investigation "10 credits" | | FireHydrant | Per responder | Pro "$25" "/responder/month, billed annually" | "14-day free trial," no card | | incident.io | Per user | Team "$19/user/month" (mo) / "$15/user/month" (annual); Pro "$25/user/month" | On-call add-on "+ $10" to "+ $20" /user/month | | Rootly | Per user | Incident Response and On-Call Essentials each "$20 per user/mo" | "AI SRE" is "Contact us" (unpublished) | | PagerDuty | Per user | Professional "$25" (mo) / "$21" (annual) per user | AIOps is a separate add-on, no per-user price shown | Two entries deserve a caveat. Rootly [publishes](https://rootly.com/pricing) prices for its Incident Response and On-Call products but lists its "AI SRE" product only as "Contact us," so there is no public per-seat figure for the AI product specifically. PagerDuty [publishes](https://www.pagerduty.com/pricing/) a per-user Professional price for incident management, but its AIOps capability is sold as a separate add-on with no standalone per-user price on the pricing page, so quoting the $25 figure as an "AIOps" price would be wrong. ## What does Cleric's per-credit pricing mean in practice? It means you pay for work done, not seats occupied. Cleric's [pricing page](https://cleric.ai/pricing) sets the unit at "Credits cost $1 on monthly plans" and prices an "Issue investigation" at "10 credits," or $10. Its tiers bundle credits: Starter "$100 / month" for "100 credits / month," Team "$600 / month" for "600 credits / month," Pro "$2,000 / month" for "2,000 credits / month," and Enterprise "Custom." Unused credits "roll over for one month, up to twice your monthly allocation," and a trial gives "500 evaluation credits · No time limit." This model is predictable for low incident volume and scales with load rather than headcount. At $10 per investigated issue, a team can map its expected incident volume directly to a bill, which is harder to do under per-user pricing where the cost is fixed regardless of how much the tool is used. ## What about FireHydrant, incident.io, Rootly, and PagerDuty? Each publishes its core pricing, but on different units. [FireHydrant](https://firehydrant.com/pricing/) prices its Pro plan at "$25" per "/responder/month, billed annually" and offers to "Start with a 14-day free trial of our Pro plan. No credit card required," noting that its Signals alerting "is charged based on the number of alerts you send." [incident.io](https://incident.io/pricing) prices per user: Team at "$19/user/month" monthly or "$15/user/month" annual, Pro at "$25/user/month," with an on-call add-on of "+ $10/user/month" on Team and "+ $20/user/month" on Pro, and a standalone on-call product at "$20/user/month." [Rootly](https://rootly.com/pricing) lists Incident Response Essentials and On-Call Essentials each at "$20 per user/mo." [PagerDuty](https://www.pagerduty.com/pricing/) lists its Professional incident-management tier at "$25" monthly or "$21" per user per month annual. ![Diagram comparing pricing units across five incident and AI SRE tools. A vertical grouping shows four billing models. Per credit: Cleric, from one hundred dollars per month for one hundred credits to two thousand dollars per month for two thousand credits, ten credits per investigated issue. Per responder: FireHydrant Pro at twenty-five dollars per responder per month billed annually. Per user: incident.io Team at fifteen to nineteen dollars and Pro at twenty-five, Rootly Essentials at twenty dollars, PagerDuty Professional at twenty-one to twenty-five dollars, each with on-call or AIOps as separate add-ons. Open source: no license fee, cost is the LLM tokens and infrastructure you run yourself. The caption notes that Rootly does not publish its AI SRE price and PagerDuty does not publish a per-user AIOps price.](/blog/ai-sre-pricing-2026.svg) ## Where does open source fit on cost? Open-source AI SRE tools remove the license line entirely; the cost is what you spend on model tokens and the infrastructure to run them. [Aurora](https://github.com/Arvo-AI/aurora) is open source, so there is no per-user, per-responder, or per-credit fee for the software. It runs a single investigation agent by default with an opt-in multi-agent orchestrator, tracks per-investigation cost, and routes on a cost model (`RCA_OPTIMIZE_COSTS` defaults on) with prompt caching, so the token spend that is your actual cost is measured rather than hidden. That does not make open source automatically cheaper; a hosted per-credit or per-user plan can be cheaper than the engineering time to run a tool well. It makes the cost structure different: you trade a predictable subscription for token and operating costs you control directly. For choosing on more than price, see [how to evaluate an AI SRE platform](/blog/how-to-evaluate-ai-sre-platform), the per-investigation view in [the cost of AI incident investigation](/blog/ai-investigation-cost), and the field in [top AI SRE tools 2026](/blog/top-ai-sre-tools-2026). ## The summary Incident and AI SRE pricing does not compare cleanly because the unit differs: Cleric bills per credit ($100 to $2,000 a month, $10 per investigated issue), FireHydrant per responder ($25/responder/month annual), incident.io and Rootly and PagerDuty per user (roughly $15 to $25), and open source not at all beyond tokens and compute. Rootly does not publish its AI SRE price and PagerDuty does not publish a per-user AIOps price, so both are "contact sales" for the AI capability. Normalize to the billing unit before comparing. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** All prices are quoted from the vendors' public pricing pages, verified 28 September 2026: cleric.ai/pricing, firehydrant.com/pricing, incident.io/pricing, rootly.com/pricing, and pagerduty.com/pricing. Where a product's price is not published (Rootly's AI SRE, PagerDuty's per-user AIOps), that is stated rather than estimated. Pricing changes often; re-check each page before relying on a figure. Aurora's cost model is described from its open-source repository. --- ## HolmesGPT vs Cleric: AI SRE Compared (2026) URL: https://www.aurorasre.ai/blog/holmesgpt-vs-cleric Published: 2026-09-28 Author: Noah Casarotto-Dinning, CEO at Arvo AI > HolmesGPT is an open-source, read-only investigation agent. Cleric is a commercial agent that holds the pager and opens fix PRs. How the two differ, with pricing. > ## Key Takeaways > > - **HolmesGPT and Cleric solve the same problem from opposite starting points: HolmesGPT is a free, open-source investigation agent you run yourself, and Cleric is a commercial hosted agent priced per credit that also opens fix pull requests.** HolmesGPT is [Apache-2.0 and a CNCF Sandbox project](https://github.com/robusta-dev/holmesgpt); Cleric [publishes per-credit pricing](https://cleric.ai/pricing). > - **HolmesGPT is read-only by design.** Its README states it "has read-only access and respects RBAC permissions. It is safe to run in production environments." > - **Cleric starts read-only but can escalate.** Its site says "Read-only by default" and "Read access by default, write access when you're ready," and it "opens PR" for a fix once approved. > - **Both work beyond Kubernetes.** HolmesGPT "Works with any stack," naming Kubernetes, VMs, cloud providers, databases, and SaaS platforms. Cleric is a hosted product with SOC 2 Type II compliance. > - **Cleric's pricing is public and usage-based.** Tiers run from Starter at "$100 / month" for "100 credits" to Pro at "$2,000 / month" for "2,000 credits," where an issue investigation costs "10 credits" at "$1" per credit. > - **The real choice is operating model, not feature checklists.** Self-hosted and free versus hosted and metered, read-only versus opt-in write, and how much of the run you want to own. **HolmesGPT is an open-source, read-only AI agent for investigating incidents, and Cleric is a commercial hosted AI SRE that investigates alerts and can open fix pull requests once granted write access.** The two are often compared because both automate the first pass of an incident, but they differ on licensing, cost model, and how far each is allowed to act. ## What is HolmesGPT? HolmesGPT is an open-source investigation agent from Robusta that reads observability data and runs read-only checks to find the root cause of an incident. Its [GitHub README](https://github.com/robusta-dev/holmesgpt) describes it as an "SRE Agent" and a "CNCF Sandbox Project," distributed under the Apache 2.0 License. The safety posture is a headline feature: "By design, HolmesGPT has read-only access and respects RBAC permissions. It is safe to run in production environments." It is not Kubernetes-only. The README states it "Works with any stack," listing Kubernetes, VMs, cloud providers, databases, and SaaS platforms, and works "with any infrastructure," from bare metal to containers. It connects to observability backends through "Deep integrations" the README lists as "Prometheus, Grafana, Datadog, Kubernetes, and many more," plus any REST API, and supports multiple model providers including "OpenAI, Anthropic, Azure, Bedrock, Gemini." Because it is open source, you run it on your own infrastructure and pay only for the LLM tokens it consumes. ## What is Cleric? Cleric is a commercial, hosted AI SRE that takes alerts, investigates them, and can propose fixes. Its [site](https://cleric.ai/) positions it around prevention and on-call relief: "Agents that check every production change, then fix regressions before customers notice," and "Give the pager to Cleric." Its access model is staged: "Read-only by default" and "Read access by default, write access when you're ready." When it acts, it does so through review, with the site showing steps like "Cleric opens PR #484 for your approval." Cleric is operated by "Agentik, Inc. dba Cleric" and states it is "SOC 2 Type II compliant, with regular manual penetration testing," with data "encrypted everywhere and never used for training." ## How do HolmesGPT and Cleric compare? The clearest way to compare them is by operating model, not feature count. One is a library you run; the other is a service you subscribe to. | Dimension | HolmesGPT | Cleric | |---|---|---| | License / model | Open source, Apache 2.0, CNCF Sandbox | Commercial, hosted (Agentik, Inc. dba Cleric) | | Cost | Free software; you pay LLM token costs | Per-credit, "$100 / month" to "$2,000 / month" plus Enterprise | | Access posture | "read-only access and respects RBAC permissions" | "Read-only by default," write access opt-in | | Acts on systems | Investigates only | Investigates, then "opens PR" for approval | | Where it runs | Your infrastructure | Hosted; "SOC 2 Type II compliant" | | Scope | "Works with any stack" | Production monitoring, change verification, on-call | The table makes the trade explicit. HolmesGPT gives you an auditable, read-only investigator with no per-incident cost and no data leaving your control, at the price of running and maintaining it yourself. Cleric gives you a managed agent that also drafts fixes and holds the pager, at a metered cost and with your telemetry flowing through a hosted service that states it does not train on your data. ## How much does Cleric cost? Cleric publishes usage-based pricing, which is unusual in this category and useful for planning. Its [pricing page](https://cleric.ai/pricing) lists Starter at "$100 / month" for "100 credits / month," Team at "$600 / month" for "600 credits / month," Pro at "$2,000 / month" for "2,000 credits / month," and Enterprise at "Custom." The unit economics are stated plainly: "Credits cost $1 on monthly plans," an issue investigation is "10 credits" (so "$10"), and chats are "1 credit / min." Unused credits "roll over for one month, up to twice your monthly allocation," and there is a trial of "500 evaluation credits · No time limit." HolmesGPT has no license cost because it is open source; its running cost is the LLM tokens each investigation consumes plus the infrastructure you host it on. That is the fundamental difference in the cost model: Cleric prices the outcome (an investigated issue), HolmesGPT prices the inputs (tokens and compute). ![Diagram comparing HolmesGPT and Cleric across two axes. The vertical axis is operating model, from self-hosted open source at the bottom to commercial hosted service at the top. The horizontal axis is authority, from read-only investigation on the left to opt-in write access that opens fix pull requests on the right. HolmesGPT sits bottom-left: Apache-2.0 CNCF Sandbox project, read-only by design, priced only by the LLM tokens it consumes. Cleric sits top-right: hosted and SOC 2 Type II compliant, read-only by default with write access opt-in, priced per credit from one hundred to two thousand dollars per month, opening a pull request for approval when it proposes a fix.](/blog/holmesgpt-vs-cleric.svg) ## Which should an SRE team choose? It depends on how much of the run you want to own and whether you want the agent to act, not just advise. A team that needs an auditable, read-only investigator inside its own perimeter, with no per-incident cost and full control of the model and data path, fits HolmesGPT. A team that wants a managed agent to hold the pager, verify production changes, and draft fixes, and that prefers a predictable per-credit bill over operating the tooling, fits Cleric. A third option is a self-hosted agent that stays read-only for investigation but can propose a fix as a human-merged pull request, keeping both the control of open source and the option to act. [Aurora](https://github.com/Arvo-AI/aurora) is built that way: a single investigation agent by default with an opt-in multi-agent orchestrator, powerful tools behind an explicit allowlist, sandboxed Kubernetes execution when enabled, and a structural chokepoint that blocks mutating writes when no human is present so remediation only ever lands as a pull request a human merges. For the wider field, see [open-source AI SRE: Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt), the [HolmesGPT alternative for multi-cloud](/blog/holmesgpt-alternative-multi-cloud), and the [root cause analysis guide for SREs](/blog/root-cause-analysis-complete-guide-sres). ## The summary HolmesGPT and Cleric answer the same need from opposite ends. HolmesGPT is free, open source, and read-only by design, and you run it yourself and pay for tokens. Cleric is hosted, priced per credit from $100 to $2,000 a month, read-only by default but able to open a fix pull request once you grant write access. The decision is about operating model and how much authority you want the agent to have, not a feature scorecard. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** HolmesGPT's description, Apache 2.0 license, CNCF Sandbox status, read-only and RBAC wording, and supported-stack wording are quoted from its GitHub README, verified 28 September 2026. Cleric's positioning, read-only and write-access wording, PR-approval flow, SOC 2 Type II statement, and all pricing (tiers, credits, per-credit cost, issue-investigation cost, rollover, trial) are quoted from cleric.ai and its pricing page, verified the same day. Pricing changes; re-check the pricing page before relying on a figure. Aurora's controls are described from its open-source repository. --- ## How to Fix CrashLoopBackOff in Kubernetes (2026) URL: https://www.aurorasre.ai/blog/fix-crashloopbackoff-kubernetes Published: 2026-09-28 Author: Noah Casarotto-Dinning, CEO at Arvo AI > CrashLoopBackOff means a container keeps crashing and Kubernetes is waiting before it restarts again. A step-by-step diagnosis with the exact kubectl commands. > ## Key Takeaways > > - **CrashLoopBackOff is a status, not a root cause: it means a container has crashed repeatedly and the kubelet is now waiting before restarting it.** The Kubernetes docs define the state as one where "the backoff delay mechanism is currently in effect for a given container that is in a crash loop." > - **The wait grows on a fixed schedule.** The kubelet "restarts them with an exponential backoff delay (10s, 20s, 40s, …), that is capped at 300 seconds (5 minutes)," per the [pod lifecycle docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy). > - **A high Restart Count is the tell.** The debug docs note "Restart Count tells you how many times the container has been restarted; this information can be useful for detecting crash loops." > - **The crash log is the fastest path to the cause.** When a container has already died, its logs are gone from the live view; retrieve them with `kubectl logs -c --previous`. > - **Most crash loops trace to one of five causes:** a failing application process, a bad or missing config or secret, a failed liveness probe, an OOMKill, or a missing dependency at startup. > - **The loop resets itself once the container is healthy.** "Once a container has executed for 10 minutes without any problems, the kubelet resets the restart backoff timer for that container." **CrashLoopBackOff is a Kubernetes pod status that means a container started, crashed, and has been restarted enough times that the kubelet is now pausing between attempts with an increasing delay.** It is a symptom the scheduler surfaces, not a diagnosis, so fixing it means finding why the container exits and reading the crash log before the next restart hides it. ## What does CrashLoopBackOff mean in Kubernetes? It means a container in the pod keeps terminating shortly after it starts, and the kubelet has begun spacing out the restarts. The [Kubernetes pod lifecycle documentation](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy) describes the state directly: it "indicates that the backoff delay mechanism is currently in effect for a given container that is in a crash loop." The container is not stuck; it is being restarted, failing again, and being held back a little longer each time. The delay is deterministic. Per the same page, when containers keep failing "the kubelet restarts them with an exponential backoff delay (10s, 20s, 40s, …), that is capped at 300 seconds (5 minutes)." That cap is why a crash-looping pod eventually only retries once every five minutes, and why an obviously broken pod can sit in CrashLoopBackOff for a long time without a human touching it. The counter is not permanent: "Once a container has executed for 10 minutes without any problems, the kubelet resets the restart backoff timer for that container." So a fix is confirmed when the pod runs clean past that ten-minute mark. ## How do I diagnose a CrashLoopBackOff pod? Start by confirming the status and the restart count, then read the crash log, then read the pod's events. Three commands cover almost every case. First, list the pod and check how many times it has restarted: ```shell kubectl get pods ``` A climbing restart count on a pod in `CrashLoopBackOff` is the confirmation. The [debug-running-pod guide](https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/) notes that "Restart Count tells you how many times the container has been restarted; this information can be useful for detecting crash loops in containers that are configured with a restart policy of `Always`." Second, read the log from the instance that already died. The live `kubectl logs` output is from the current container, which may be too young to show the error, so ask for the previous one. The Kubernetes docs give the exact form: "If your container has previously crashed, you can access the previous container's crash log with:" ```shell kubectl logs -c --previous ``` Third, read the pod's own events and last state, which surface OOMKills, probe failures, and image or mount errors that never reach the application log: ```shell kubectl describe pod ``` ## What are the most common causes of CrashLoopBackOff? The status is one symptom with several causes. These five account for the large majority of crash loops, and each has a distinct signature in the three commands above. OOMKilled is one of those five, and it has its own page: [how to fix OOMKilled](/runbooks/fix-oomkilled-kubernetes). ImagePullBackOff is not a crash loop. The container never starts, because the image could not be pulled. See [how to fix ImagePullBackOff](/runbooks/fix-imagepullbackoff-kubernetes). | Cause | Where it shows up | Typical fix | |---|---|---| | Application error on startup | Stack trace or panic in `--previous` logs | Fix the code path or the input that triggers the exit | | Bad or missing config / secret | Log shows a missing env var, key, or file | Correct the ConfigMap, Secret, or mount | | Failing liveness probe | `describe` shows the probe killing the container | Fix the probe target or raise `initialDelaySeconds` | | Out of memory (OOMKilled) | `describe` shows `Reason: OOMKilled`, exit code 137 | Raise the memory limit or fix the leak. Steps: [how to fix OOMKilled](/runbooks/fix-oomkilled-kubernetes) | | Missing startup dependency | Log shows a connection refused to a DB or service | Fix ordering, readiness of the dependency, or retries | A liveness probe deserves special attention because it can turn a healthy container into a crash loop on its own. If the probe points at a path or port the app does not serve, or fires before the app finishes starting, the kubelet keeps killing and restarting a process that would otherwise run. The Kubernetes [pod lifecycle docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) are explicit about the mechanism: "If a container fails its liveness probe more times than the configured tolerance, the kubelet restarts that container." The documented fix for a slow-starting app is a startup probe, which the probe-configuration guide frames as a way to "Protect slow starting containers with startup probes." ## How do I fix CrashLoopBackOff step by step? Work from symptom to cause to fix, in that order, and confirm the pod stays up past the ten-minute reset. ![Diagram of a CrashLoopBackOff diagnosis workflow. A pod shows status CrashLoopBackOff with a rising restart count. Three branches read the signals: kubectl get pods confirms the restart count, kubectl logs with the previous flag reads the crash log from the container that already died, and kubectl describe pod reads events and last state. The signals sort into five causes: application startup error, bad config or secret, failing liveness probe, OOMKilled with exit code 137, and a missing startup dependency. Each cause points to its fix, and the loop is confirmed resolved once the container runs ten minutes without a restart and the kubelet resets the backoff timer.](/blog/crashloopbackoff-diagnosis.svg) The sequence is the same regardless of cause: read the restart count, read the previous container's log, read the pod events, match the signature to a cause, apply the narrowest fix, and watch the restart count stop climbing. The backoff itself is working as designed; it is buying time, not causing the failure. ## How does an AI SRE help with CrashLoopBackOff? An investigation agent runs the same read-only sequence a human would, then correlates it with what changed. The bottleneck in a crash loop is rarely running the commands; it is connecting the crash log to the deploy, config change, or resource edit that introduced it, especially when several pods fail at once. [Aurora](https://github.com/Arvo-AI/aurora) is an open-source AI SRE that reads pod status, previous-container logs, and events through Kubernetes tools, then correlates them with recent changes rather than acting blind. Aurora runs a single investigation agent by default, with an opt-in multi-agent orchestrator, and its powerful tools sit behind an explicit allowlist. It supports sandboxed Kubernetes execution when `ENABLE_POD_ISOLATION` is set. Any fix is proposed as a pull request a human merges, never an automatic change to a running cluster, which matters when the pod in question is already unhealthy. For the safety model behind reading a live cluster, see [AI agent kubectl safety](/blog/ai-agent-kubectl-safety); for the broader workflow, see [AI-powered incident investigation](/blog/ai-powered-incident-investigation) and the [root cause analysis guide for SREs](/blog/root-cause-analysis-complete-guide-sres). ## The summary CrashLoopBackOff is the kubelet telling you a container keeps dying, then pacing the restarts on a fixed backoff that caps at five minutes and resets after ten clean minutes. The fix is always the same shape: confirm the restart count with `kubectl get pods`, read the crash log with `kubectl logs --previous`, read events with `kubectl describe pod`, and match the signature to one of five common causes. The status is a pointer to the problem, not the problem itself. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) To self-host and point Aurora at your own cluster: ```shell git clone https://github.com/Arvo-AI/aurora ``` > **Sourcing note.** The CrashLoopBackOff definition, the exponential backoff delay "(10s, 20s, 40s, …)" capped at "300 seconds (5 minutes)," and the ten-minute reset are quoted from the Kubernetes pod lifecycle documentation, verified 28 September 2026. The Restart Count wording and the `kubectl logs --previous` command are quoted from the Kubernetes application-debugging docs, and the liveness-probe behavior from the probe-configuration guide, both verified the same day. Aurora's defaults are described from its open-source repository. --- ## MITRE ATLAS Maps Autonomous AI Agent Attacks (2026) URL: https://www.aurorasre.ai/blog/mitre-atlas-autonomous-ai-agent-attacks Published: 2026-09-24 Author: Noah Casarotto-Dinning, CEO at Arvo AI > In September 2026 MITRE ATLAS added autonomous-agent attack techniques and two agent-authority mitigations. What AML.T0116 to T0124, M0037, and M0038 say. > ## Key Takeaways > > - **In September 2026, MITRE ATLAS added a cluster of attack techniques for autonomous AI agents and, for the first time, two mitigations aimed squarely at controlling an agent's own authority.** Release [v2026.08](https://github.com/mitre-atlas/atlas-data/releases/tag/v2026.08) shipped on 1 September 2026 and [v2026.09](https://github.com/mitre-atlas/atlas-data/releases/tag/v2026.09) on 15 September 2026. > - **The threat model shifted from manipulating a model to weaponizing an agent's autonomy.** The v2026.08 notes even rename tactic [AML.TA0001](https://atlas.mitre.org/tactics/AML.TA0001) from "AI Attack Staging" to "AI Attack Adaptation," reflecting agents that adapt an attack at runtime. > - **New techniques name autonomous behavior directly.** They include [Autonomous Reconnaissance (AML.T0116)](https://atlas.mitre.org/techniques/AML.T0116), Autonomous Attack-Path Adaptation (AML.T0117), Autonomous AI Agent Communication (AML.T0118), AI Agent Environment Reconstruction (AML.T0121), and [Autonomous Attack Orchestration (AML.T0124)](https://atlas.mitre.org/techniques/AML.T0124). > - **The two new mitigations are the under-covered material.** [AI Agent Authority Expansion Controls (AML.M0037)](https://atlas.mitre.org/mitigations/AML.M0037) and AI Agent Scope Drift Detection (AML.M0038) describe controls that sit outside the agent, not inside its prompt. > - **A real campaign anchors the taxonomy.** ATLAS added case study AML.CS0069, the GTG-1002 Claude Code espionage campaign, which [Anthropic reported](https://www.anthropic.com/news/disrupting-AI-espionage) used AI for "80-90% of the campaign." > - **The defensive lesson is structural.** M0037 states its controls "should be implemented outside the AI agent and should not rely solely on system prompts, model alignment, or the agent recognizing that an action is out of scope." **MITRE ATLAS is a public knowledge base of attacks on AI systems, and in September 2026 it added techniques for attacks run by autonomous AI agents plus the first mitigations that bound an agent's authority.** For any team running an agent that acts against production, this reframes the defensive question from "can the model be tricked" to "what can the agent do once it decides to act." ## What did MITRE ATLAS add for autonomous AI agents in 2026? Two monthly releases in September 2026 expanded ATLAS toward autonomous and multi-agent threats. Release [v2026.08](https://github.com/mitre-atlas/atlas-data/releases/tag/v2026.08), dated 1 September 2026, states it "contains 1 matrix, 16 tactics, 114 techniques, 83 sub-techniques, 39 mitigations, and 72 case studies." Release [v2026.09](https://github.com/mitre-atlas/atlas-data/releases/tag/v2026.09), dated 15 September 2026, grew that to "120 techniques, 88 sub-techniques, 40 mitigations, and 73 case studies." The framing changed alongside the content. The v2026.08 notes rename tactic AML.TA0001 to "AI Attack Adaptation," noting it was "Previously 'AI Attack Staging'." Staging implies preparation before an attack. Adaptation implies an attacker, or an autonomous agent acting as one, adjusting the attack as it runs. That single rename captures why the release matters: ATLAS is now modeling AI that changes its own plan mid-execution. ## What are the new autonomous-agent attack techniques (AML.T0116 to AML.T0124)? They are techniques where an autonomous agent performs an attack step on its own, given only an objective. The v2026.08 release added the following, quoted by ID and name from the release notes. | Technique ID | Name | What the agent does autonomously | |---|---|---| | AML.T0116 | Autonomous Reconnaissance | Determines what to learn about a target and how | | AML.T0117 | Autonomous Attack-Path Adaptation | Adjusts its route based on what it finds | | AML.T0118 | Autonomous AI Agent Communication | Coordinates with other agents (.000 shared artifacts, .001 direct) | | AML.T0121 | AI Agent Environment Reconstruction | Rebuilds a picture of its operating environment | | AML.T0124 | Autonomous Attack Orchestration | Directs multiple sub-agents toward one goal | | AML.T0016.001 | Develop Capabilities: Autonomous Exploit Development | Builds exploit code without a human author | The technique text is specific about autonomy. [Autonomous Reconnaissance (AML.T0116)](https://atlas.mitre.org/techniques/AML.T0116) reads: "Given an objective, target, or partial lead, an agent may autonomously determine what information to obtain and how to investigate it. It may interpret observations, identify gaps in its understanding of the externally observable attack surface, and select subsequent reconnaissance actions without a human specifying each investigative step." [Autonomous Attack Orchestration (AML.T0124)](https://atlas.mitre.org/techniques/AML.T0124) describes "an operational control layer to manage multiple distinct autonomous agents or sub-agents toward a common adversary-defined objective," able to "create assignments, select executors, allocate tools or resources, establish dependencies, schedule or synchronize activities, and track progress without a human directing each" step. Release v2026.09 continued the pattern with [AI Agent Response Biasing (AML.T0130)](https://atlas.mitre.org/techniques/AML.T0130) and [Discover AI Agent Runtime Capabilities (AML.T0133)](https://atlas.mitre.org/techniques/AML.T0133), which target a deployed agent from the outside. ## What are AML.M0037 and AML.M0038, the new agent-authority mitigations? They are the first ATLAS mitigations written specifically to bound an autonomous agent's authority and watch its behavior at runtime, and they are the most useful part of the release for anyone building an agent. [AI Agent Authority Expansion Controls (AML.M0037)](https://atlas.mitre.org/mitigations/AML.M0037) begins: "Limit an AI agent's ability to autonomously acquire, assume, or otherwise obtain additional authorities that expand its effective permissions during execution. The maximum authority available to the agent should be explicitly granted prior to runtime." Anything the agent discovers while running is untrusted by default: "Additional resources, identities, services, and targets discovered during execution should be treated as outside the authorized boundary unless they are independently validated and added to scope." The load-bearing sentence is about where the control lives. M0037 states these controls "should be implemented outside the AI agent and should not rely solely on system prompts, model alignment, or the agent recognizing that an action is out of scope." The listed enforcement mechanisms are concrete: "Policy engines," "Target allowlists," "Protocol and destination restrictions," "Approval gates," and "Preventing the agent from using credentials that were not approved for the task." [AI Agent Scope Drift Detection (AML.M0038)](https://atlas.mitre.org/mitigations/AML.M0038) covers the runtime half: "Continuously evaluate whether an AI Agent's planned actions remain consistent with its current authorized objective throughout execution." It lists indicators to monitor, including "Tool usage inconsistent with the original mission" and "Repeated adaptation toward objectives requiring progressively broader authority." Its prescribed response is a set of hard stops: "When scope drift is detected, pause execution, restrict tool access, require external approval, return the agent to a known authorized plan, or terminate the task." ![Diagram of the MITRE ATLAS 2026 autonomous-agent model. An autonomous AI agent given an objective runs a loop of reconnaissance, attack-path adaptation, and orchestration of sub-agents, mapped to techniques AML.T0116, T0117, and T0124. Two mitigation gates sit outside the agent: Authority Expansion Controls (AML.M0037) enforce a pre-granted maximum authority through policy engines, allowlists, and approval gates, and Scope Drift Detection (AML.M0038) watches the agent's plan and can pause, require approval, or terminate the task when it drifts from its authorized objective.](/blog/mitre-atlas-autonomous-agent.svg) ## Which real attack does ATLAS cite for autonomous agents? The GTG-1002 Claude Code espionage campaign, added as case study AML.CS0069 in v2026.08. The ATLAS summary reads: "In September 2025, GTG-1002 used a jailbroken Claude Code agent to conduct a cyber-espionage campaign against approximately 30 organizations, succeeding against a small number. Anthropic assessed with high confidence that GTG-1002 was a Chinese state-sponsored group." ATLAS notes the operators "connected the jailbroken Claude agent to scanners, browser automation, password crackers, database tooling, and dedicated penetration-testing servers through MCP." The primary account is [Anthropic's report](https://www.anthropic.com/news/disrupting-AI-espionage), dated 13 November 2025, which states "the threat actor was able to use AI to perform 80-90% of the campaign, with human intervention required only sporadically (perhaps 4-6 critical decision points per hacking campaign)," across "infiltration into roughly thirty global targets." The same report is candid about the limits of the technique: "It occasionally hallucinated credentials or claimed to have extracted secret information that was in fact publicly-available." That caveat matters for defenders, because an autonomous attacker that fabricates its own findings is both a threat and an unreliable one, which is a defensive opening rather than a reason to relax. ## How do M0037 and M0038 map to a least-privilege AI SRE? By describing, almost line for line, the controls a defensively designed investigation agent already enforces before any tool is trusted. M0037 asks that maximum authority be granted before runtime and that discovered resources be treated as out of scope. M0038 asks that a drift from the authorized objective trigger a pause, an approval, or a stop. Both put the control outside the agent's own judgment. [Aurora](https://github.com/Arvo-AI/aurora) is built that way. Its most powerful tools sit behind an explicit allowlist rather than being callable by default, which is M0037's "target allowlists" and "approval gates" applied to tool access. It supports sandboxed Kubernetes execution for tool runs. Its structural chokepoint denies all mutating writes when no interactive human is present, so an agent cannot expand its own authority into an unattended production change, and remediation is limited to a pull request a human merges. That is M0037's principle stated as behavior: the limit is enforced outside the model, not requested of it. The layered design is described in [AI agent guardrails](/blog/ai-agent-guardrails), the tool-layer attack surface in [AI agent tool poisoning](/blog/ai-agent-tool-poisoning-security), and the accountability trail in [who is accountable when an AI agent changes prod](/blog/ai-agent-accountability-audit-evidence). None of this requires a novel defense. It requires refusing to let an agent grant itself authority. ## The summary In September 2026, MITRE ATLAS added an autonomous-agent attack cluster (AML.T0116 through T0124, extended in v2026.09) and renamed a tactic from staging to adaptation, modeling AI that adjusts an attack as it runs. More useful for builders, it added two mitigations: Authority Expansion Controls (AML.M0037) and Scope Drift Detection (AML.M0038), both of which insist the control live outside the agent rather than in its prompt. The GTG-1002 campaign, where AI ran an estimated 80 to 90 percent of the operation, is the real-world anchor. For defensive agents, the takeaway is that autonomy is safe only when its ceiling is set before runtime and enforced structurally. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** Release dates, object counts, tactic rename, technique IDs and names, mitigation IDs and names, and the GTG-1002 case study summary are quoted from MITRE ATLAS releases v2026.08 (1 September 2026) and v2026.09 (15 September 2026) in the mitre-atlas/atlas-data repository, with technique and mitigation description text verified against the ATLAS-latest.yaml release data, all checked 24 September 2026. The 80-90 percent, "4-6 critical decision points," roughly thirty targets, and hallucinated-credentials wording are quoted from Anthropic's report dated 13 November 2025. The atlas.mitre.org pages are the canonical human-readable references for each ID. Aurora's controls are described from its open-source repository. --- ## Parallel-Agent Incident Investigation: AWS vs BigPanda URL: https://www.aurorasre.ai/blog/parallel-agent-incident-investigation-2026 Published: 2026-09-24 Author: Noah Casarotto-Dinning, CEO at Arvo AI > In September 2026 AWS and BigPanda both shipped multi-agent parallel incident investigation. What each launched, how the architectures differ, and when parallel helps. > ## Key Takeaways > > - **In September 2026, two vendors shipped parallel-agent incident investigation within weeks of each other.** BigPanda announced [Swarm Investigation](https://www.bigpanda.io/blog/introducing-swarm-investigation-bigpanda/) on 4 September, and AWS introduced [Amazon CloudWatch Omni](https://aws.amazon.com/blogs/aws/introducing-amazon-cloudwatch-omni-collaborative-ai-powered-observability-for-your-applications/) with an embedded DevOps Agent on 22 September. > - **BigPanda's model runs multiple agents against competing hypotheses at once.** Swarm Investigation "coordinates specialized AI agents to investigate multiple plausible causes in parallel, evaluate evidence as it emerges." > - **AWS's model embeds one collaborating agent in an investigation session.** The Amazon DevOps Agent "participates alongside your team in investigation sessions, correlating signals and suggesting next steps." > - **Both keep the human in the loop for action.** AWS describes the agent as "suggesting next steps"; neither product is described as acting on production without human review. > - **Parallel investigation is a real capability, not a free one.** Running many agents multiplies token cost and can produce conflicting findings, which is why an [orchestrator is worth making opt-in](/blog/why-multi-agent-ai-systems-fail) rather than the default. > - **Availability differs.** BigPanda states Swarm Investigation "is generally available"; AWS states CloudWatch Omni is now available and the DevOps Agent "is enabled by default in every Omni investigation session." > - **The design question is when parallel helps.** Multiple hypotheses genuinely benefit from parallel agents; a single clear signal does not, and paying for a swarm to chase it wastes budget. **Parallel-agent incident investigation is an architecture where several AI agents work different hypotheses or data sources at the same time and pool their findings, rather than one agent reasoning step by step.** In September 2026 both BigPanda and AWS shipped a version of it. For SREs evaluating AI investigation tooling, the two launches make a useful contrast: one vendor coordinates a swarm of specialized agents, the other embeds a single collaborating agent in a shared session. ## What did BigPanda and AWS launch in September 2026? Two different takes on putting AI agents into the investigation loop, weeks apart. BigPanda announced [Swarm Investigation](https://www.bigpanda.io/blog/introducing-swarm-investigation-bigpanda/) on 4 September 2026. Per the post, it "coordinates specialized AI agents to investigate multiple plausible causes in parallel, evaluate evidence as it emerges." A Reasoning Engine "coordinates new investigation tasks" while individual investigation agents run focused tasks against connected tools and feed findings into a shared record. BigPanda states the feature "is generally available as part of AI Incident Assistant, and does not require a separate license." AWS introduced [Amazon CloudWatch Omni](https://aws.amazon.com/blogs/aws/introducing-amazon-cloudwatch-omni-collaborative-ai-powered-observability-for-your-applications/) on 22 September 2026, with an embedded Amazon DevOps Agent. The post describes the agent this way: it "participates alongside your team in investigation sessions, correlating signals and suggesting next steps," and "identifies correlated events across services, traces root cause paths through your dependency graph, and maintains investigation history for post-incident review." AWS says the DevOps Agent "is enabled by default in every Omni investigation session." AWS states CloudWatch Omni is now available and can be tried from the console. ## How do the two architectures differ? BigPanda parallelizes across hypotheses; AWS embeds a single collaborator in a session. That is the core distinction, and it shapes everything downstream. | Dimension | BigPanda Swarm Investigation | AWS CloudWatch Omni + DevOps Agent | |---|---|---| | Announced | 4 September 2026 | 22 September 2026 | | Agent model | Multiple specialized agents in parallel | One agent embedded in the session | | What it does | Investigates "multiple plausible causes in parallel" | "correlating signals and suggesting next steps" | | Root cause | Evaluates evidence as it emerges across agents | "traces root cause paths through your dependency graph" | | Human role | Investigation feeds a shared record for the team | "participates alongside your team," suggests steps | | Stated availability | "generally available," no separate license | Omni available; DevOps Agent enabled by default | | Action on prod | Not described as autonomous | Described as suggesting, not acting | Both are advisory by design. Neither vendor describes its agent taking a mutating action on production without a human. AWS's language is consistently "suggesting next steps" and providing "suggestions," with humans performing the actual actions. That matches the wider 2026 pattern in which vendors gate mutation behind human approval while letting investigation run freely, a convergence covered in [how vendors gate AI agent autonomy](/blog/ai-agent-autonomy-gated-production). ## Does running agents in parallel make investigation better? Sometimes, and it depends on whether the incident has one likely cause or several competing ones. Parallel agents shine when there are multiple plausible hypotheses that can be investigated independently: a swarm can chase "database", "network", and "recent deploy" at once and converge faster than a single agent working them in sequence. BigPanda's framing, "multiple plausible causes in parallel," is exactly this case. But parallelism is not free, and more agents is not automatically better. Running several agents multiplies token cost, and coordinating them introduces its own failure modes, including agents reaching conflicting conclusions that something must reconcile. Research on multi-agent systems has documented that coordination overhead and error propagation can erode the benefit, which is why treating a swarm as the default rather than a deliberate choice is risky. That evidence is collected in [why multi-agent AI systems fail in production](/blog/why-multi-agent-ai-systems-fail). The defensible design position is that parallel investigation is a capability to reach for when the hypothesis space is wide, not a setting to leave on for every alert. ![Diagram comparing two parallel-agent incident investigation architectures shipped in September 2026. On the left, BigPanda Swarm Investigation shows a reasoning engine coordinating several specialized agents that each investigate a different plausible cause in parallel and feed a shared record. In the middle, AWS CloudWatch Omni shows a single DevOps Agent embedded in an investigation session alongside human engineers, correlating signals and suggesting next steps. On the right, an opt-in model shows a single investigating agent by default with a multi-agent orchestrator that a team can enable when the hypothesis space is wide, with mutating actions gated behind human approval in all three.](/blog/parallel-agent-investigation.svg) ## What should an SRE weigh when choosing parallel-agent tooling? Four questions, each of which the two launches answer differently. 1. **Is the agent advisory or acting?** Both BigPanda and AWS keep the agent advisory, suggesting steps rather than changing production. Confirm where any tool draws that line before trusting it near prod, because investigation autonomy and mutation autonomy are different risks. 2. **What does parallelism cost?** A swarm multiplies LLM spend per incident. Ask whether the tool exposes per-investigation cost and whether parallelism is a default or a choice. The cost surface of a single AI investigation is broken down in [what an AI investigation actually costs](/blog/ai-investigation-cost). 3. **How are conflicting findings resolved?** Multiple agents can disagree. A design that pools findings into a shared record, as BigPanda describes, still needs a reconciliation step; a single-agent design sidesteps the conflict entirely. 4. **Is it open and portable, or tied to one stack?** CloudWatch Omni lives in the AWS console and dependency graph. A tool that is open source and multi-cloud keeps the investigation portable across providers. ## How does an opt-in orchestrator compare? By making the single-agent path the default and the swarm a deliberate opt-in, which inverts the "parallel by default" assumption. [Aurora](https://github.com/Arvo-AI/aurora) runs a single LangGraph investigating agent by default and ships a multi-agent orchestrator that is opt-in rather than on out of the box. That reflects the design view above: a single agent is the right tool for most incidents, and parallel agents are worth the cost only when the hypothesis space justifies them. Aurora is also open source and multi-cloud, so the investigation is not tied to one provider's console, and it tracks cost per investigation, which is the number that decides whether a swarm was worth running. Like both BigPanda and AWS, it keeps mutation human-gated: it denies structured mutating writes when no interactive human is present, and remediation is limited to a pull request a human merges. The point is not that a swarm is wrong, it is that whether to run one should be a choice an operator makes per incident, informed by cost, rather than a default that quietly multiplies spend. How that investigation works end to end is covered in [AI-powered incident investigation](/blog/ai-powered-incident-investigation), and the broader tool landscape in [the best AI-powered incident investigation tools](/blog/best-ai-powered-incident-investigation-tools). ## The summary September 2026 saw two parallel-agent investigation launches: BigPanda's Swarm Investigation, which runs specialized agents against competing hypotheses in parallel and is generally available, and AWS's CloudWatch Omni, which embeds a single DevOps Agent in an investigation session to correlate signals and suggest next steps. Both keep the agent advisory and the human in the loop for action. Parallel investigation genuinely helps when several hypotheses compete, but it multiplies cost and adds coordination risk, so the defensible design is to make a swarm an opt-in choice per incident rather than the default for every alert. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** BigPanda Swarm Investigation quotes, the general-availability and no-separate-license statements, and the 4 September 2026 date are from BigPanda's announcement post. AWS quotes describing the Amazon DevOps Agent, the "enabled by default" statement, CloudWatch Omni availability, and the 22 September 2026 date are from the AWS News Blog post. Both were verified 24 September 2026. Aurora's single-agent default, opt-in orchestrator, cost tracking, and human-gated remediation are described from its open-source repository, where the multi-agent orchestrator is off by default. --- ## AI Agent Observability: The OpenTelemetry Standard in 2026 URL: https://www.aurorasre.ai/blog/opentelemetry-ai-agent-observability Published: 2026-08-31 (updated 2026-09-26) Author: Noah Casarotto-Dinning, CEO at Arvo AI > OpenTelemetry now defines spans for AI agents, tools, and MCP calls. What the GenAI semantic conventions cover, their status, and how to trace agents. > ## Key Takeaways > > - **The observability standard for AI agents is being written right now, and it is not stable yet.** OpenTelemetry's [GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) define spans and metrics for agents, tools, and Model Context Protocol calls, and every one of them is marked "Development," not stable. > - **There is now a shared vocabulary for agent behavior.** The conventions define operations including `create_agent`, `invoke_agent`, `plan`, `invoke_workflow`, and `execute_tool`, so an agent's planning, delegation, and tool use can be traced with consistent names across frameworks. > - **Tool calls get first-class spans.** An `execute_tool` span carries `gen_ai.tool.name`, `gen_ai.tool.call.id`, and the tool type, making each tool invocation an inspectable unit rather than an opaque step. > - **Model Context Protocol has its own conventions.** OpenTelemetry defines `mcp.client` and `mcp.server` spans and four MCP metrics, and recommends following them "instead of RPC semantic conventions" because they carry domain-specific context. > - **Logging agent inputs is a privacy decision.** Message and tool-argument attributes are opt-in by default, and the spec warns each is "likely to contain sensitive information including user/PII data." > - **The market is consolidating around this.** In August 2026 Dynatrace [signed a definitive agreement to acquire Arize](https://www.dynatrace.com/news/blog/dynatrace-intends-to-acquire-arize/), whose OpenInference tracing is "built on OpenTelemetry." > - **An AI SRE that speaks MCP maps naturally onto these spans.** The emerging conventions describe exactly the tool-call and cost surface an incident-investigation agent already produces. **AI agent observability is the practice of tracing an autonomous agent's reasoning, tool calls, and cost as structured telemetry, which OpenTelemetry's GenAI semantic conventions now give standard names.** As of 2026 those conventions are still marked Development rather than stable. For an SRE who already lives in traces and spans, this means agent behavior is becoming just another instrumented system, with a defined vocabulary for planning, delegation, and tool use, and an MCP-native investigation tool fits that vocabulary cleanly. ## What are the OpenTelemetry GenAI semantic conventions for AI agents? They are a standard set of span, metric, and attribute names for instrumenting generative-AI and agent systems. The conventions moved in 2026 out of the main semantic-conventions repository into a dedicated one, which self-describes as "Semantic Conventions for Generative AI (GenAI), including spans, metrics, and events for GenAI clients, MCP (Model Context Protocol)." The old documentation path now carries a notice that it "has moved and is no longer maintained in this repository," so the [semantic-conventions-genai repository](https://github.com/open-telemetry/semantic-conventions-genai) is the source to cite. The practical payoff is a shared vocabulary. Before this, every framework named agent operations differently, and a trace from one agent stack meant nothing to tooling built for another. The conventions define a common `gen_ai.operation.name` enum that includes `chat`, `create_agent`, `invoke_agent`, `invoke_workflow`, `plan`, `execute_tool`, `embeddings`, and `retrieval`, among others. An SRE reading a trace can now tell a planning step from a tool call from a sub-agent invocation without knowing which framework produced it. ## Are the OpenTelemetry GenAI conventions stable or experimental in 2026? They are experimental. Every relevant document in the repository, the [agent spans](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md), the client spans, and the [MCP conventions](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md), carries the status line "Status: Development." In OpenTelemetry's maturity model, Development is below Stable, which means attribute names and semantics can still change and no stability guarantee applies yet. For anyone adopting them, the vocabulary is usable today and worth adopting, but it is a moving target, not a frozen contract. OpenTelemetry's own May 2026 writeup on GenAI observability puts it as conventions that are "already in use today and under active development." Instrument against them, but expect churn, and pin the version you build on. | Convention area | Defines | Status (2026) | |---|---|---| | Agent and framework spans | `create_agent`, `invoke_agent`, `plan`, `invoke_workflow` | Development | | GenAI client spans | `chat`, `execute_tool`, model request/response attributes | Development | | MCP conventions | `mcp.client` / `mcp.server` spans, four MCP metrics | Development | | Stable release | none published | not stable | ## What span and attribute names does OpenTelemetry define for agents and tools? The conventions split agent activity into a handful of span types with defined operation names and span kinds. - **Create agent.** The `gen_ai.operation.name` is `create_agent`, the span name "SHOULD be `create_agent {gen_ai.agent.name}`," and the span kind is `CLIENT`. It captures the construction of an agent. - **Invoke agent.** The operation is `invoke_agent`, split into a `CLIENT` span for hosted agents (the examples name the OpenAI Assistants API and AWS Bedrock Agents) and an `INTERNAL` span for in-process agents (the examples name LangChain and CrewAI). - **Plan.** The operation is `plan`, described as an "agent planning or task decomposition phase," the step where an agent "formulates a strategy before executing it." Making planning its own span is what lets you see an agent's intent separately from its actions. - **Invoke workflow.** The operation is `invoke_workflow`, for a "coordinated process composed of multiple agents," which is how multi-agent coordination shows up in a trace. - **Execute tool.** The operation is `execute_tool`, the span name "SHOULD be `execute_tool {gen_ai.tool.name}`," and the span kind is `INTERNAL`. Agents and tools carry identifying attributes. An agent span uses `gen_ai.agent.id`, `gen_ai.agent.name`, and `gen_ai.agent.description`, and conversations are tied together by `gen_ai.conversation.id`, "the unique identifier for a conversation (session, thread)." A tool span uses `gen_ai.tool.name` (required), `gen_ai.tool.call.id`, `gen_ai.tool.description`, and `gen_ai.tool.type` with values such as `function`, `extension`, and `datastore`. That is enough structure to answer "which tool did this agent call, with what call id, inside which conversation," which is the first question any incident review of an agent asks. ## How does OpenTelemetry trace Model Context Protocol tool calls? With dedicated MCP conventions, distinct from generic RPC instrumentation. The [MCP document](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/mcp.md) defines two span types, `mcp.client` and `mcp.server`, and recommends that when "instrumenting MCP calls, it's RECOMMENDED to follow MCP conventions instead of RPC semantic conventions" because the MCP spans "provide domain-specific context." Four metrics are defined: `mcp.client.operation.duration`, `mcp.server.operation.duration`, `mcp.client.session.duration`, and `mcp.server.session.duration`. Spans carry `mcp.method.name` (required, with values such as `tools/call`, `tools/list`, and `initialize`), `mcp.session.id`, and `mcp.protocol.version`. The conventions also handle trace context propagation into MCP requests and note MCP's transport independence, expecting clients to implement "at least stdio or Streamable HTTP." There is a deliberate bridge to the tool spans above. The MCP document states that "MCP tool call execution spans are compatible with GenAI `execute_tool` spans," and that if outer GenAI instrumentation is already tracing the tool execution, the MCP layer "SHOULD NOT create a separate span" but instead add MCP-specific attributes to the existing one. That avoids double-counting a single tool call as two spans. ![Diagram of an AI agent trace under the OpenTelemetry GenAI semantic conventions. A root invoke_agent span contains a plan span where the agent forms a strategy, followed by two execute_tool spans. One execute_tool span is a Model Context Protocol call carrying mcp.client attributes such as mcp.method.name and mcp.session.id. Tool argument and message attributes are marked opt-in because they may contain sensitive or PII data. Every span is labelled status Development, indicating the conventions are not yet stable.](/blog/opentelemetry-agent-trace.svg) ## What are the risks of logging agent prompts and tool arguments? The main risk is that the most useful attributes to capture are also the most sensitive, so they are opt-in by default. The conventions mark message and content attributes, including `gen_ai.input.messages`, `gen_ai.output.messages`, and `gen_ai.tool.call.arguments`, as opt-in, and warn on the message attributes that each "is likely to contain sensitive information including user/PII data." For an SRE this is a governance decision, not a default to accept blindly. Capturing full prompts and tool arguments makes a trace far more useful for debugging an agent, and far more dangerous if that trace lands in a log store with broad access. The safe posture is to treat agent content attributes the way you already treat request bodies that may contain secrets: capture deliberately, redact where possible, and control who can read the resulting spans. The same care applies to tool outputs, since a tool result folded into a span can carry data the agent was never meant to persist. ## How does an MCP-native AI SRE map onto these conventions? An investigation agent that already exposes its tools over MCP is producing exactly the activity these spans describe: agent invocations, planning steps, and a stream of tool calls, many of them MCP method calls with sessions and durations. The conventions give that activity standard names, which makes an agent's work legible to the same observability stack an SRE uses for everything else. [Aurora](https://github.com/Arvo-AI/aurora) is MCP-native. It exposes its tools over MCP with streamable HTTP transport, in tiers where powerful tools sit behind an allowlist, and it is used from clients including Cursor, Claude Desktop, Claude Code, Codex, Windsurf, and Cline. That maps directly onto the MCP conventions: `mcp.client` and `mcp.server` spans, `mcp.method.name` values like `tools/call`, and session durations. Aurora also tracks cost per investigation and builds a per-incident execution timeline, which is the same instinct these conventions formalize, making an agent's actions and their cost inspectable after the fact. The point is not that Aurora emits `gen_ai.*` spans today, but that an MCP-native, cost-aware investigation tool is already structured the way the standard is heading. How that investigation works end to end is covered in [AI-powered incident investigation](/blog/ai-powered-incident-investigation), and how it fits the incident lifecycle in [the SRE incident lifecycle with AI agents](/blog/sre-incident-lifecycle-with-ai-agents). The market is moving the same way. Dynatrace's August 2026 agreement to acquire Arize folds in OpenInference, described as "an open specification for AI tracing built on OpenTelemetry," a sign that agent observability is consolidating onto the OpenTelemetry base rather than fragmenting into per-vendor formats. ## The summary OpenTelemetry now defines a real vocabulary for AI agent observability: spans for creating, invoking, and planning agents, an `execute_tool` span for tool calls, and dedicated `mcp.client` and `mcp.server` conventions with four metrics for Model Context Protocol. The whole set is marked Development, so it is usable but not yet stable. The most valuable attributes, prompts and tool arguments, are opt-in because they carry PII. For SREs, the takeaway is that agent behavior is becoming instrumentable like any other system, and an MCP-native, cost-aware investigation agent is already shaped for where the standard is going. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** Span names, operation names, attribute names, MCP spans and metrics, and every "Status: Development" label are quoted from the OpenTelemetry semantic-conventions-genai repository on GitHub, verified 31 August 2026. The Dynatrace agreement to acquire Arize and the OpenInference description are quoted from Dynatrace's blog post dated 13 August 2026. The conventions are not stable; no stable release exists as of this writing. Aurora's MCP transport, client list, and cost-tracking are described from its open-source repository and documentation. --- ## AI Agent Tool Poisoning: What the 2026 Standards Say URL: https://www.aurorasre.ai/blog/ai-agent-tool-poisoning-security Published: 2026-08-31 (updated 2026-09-26) Author: Noah Casarotto-Dinning, CEO at Arvo AI > MITRE ATLAS added tool poisoning as a technique in July 2026. What AML.T0110, OWASP LLM06, and NCSC guidance say about securing autonomous AI agents. > ## Key Takeaways > > - **In summer 2026, two standards bodies and a national cyber agency formalized the same threat: an autonomous AI agent can be subverted through the tools it calls, not only the prompts it reads.** MITRE ATLAS minted a dedicated technique for it, [AI Agent Tool Poisoning (AML.T0110)](https://atlas.mitre.org/techniques/AML.T0110), with a modified date of 31 July 2026. > - **Tool poisoning has three distinct forms in the ATLAS taxonomy.** The technique defines sub-techniques for poisoning a tool's [Definition and Instructions](https://atlas.mitre.org/techniques/AML.T0110.000), its [Implementation](https://atlas.mitre.org/techniques/AML.T0110.001), and its [Runtime Response](https://atlas.mitre.org/techniques/AML.T0110.002), each with its own maturity rating. > - **The threat is already realized, not theoretical.** MITRE rates the implementation sub-technique "Realized" and cites a real malicious Model Context Protocol server, the Postmark backdoor, as its example. > - **OWASP names the underlying design flaw Excessive Agency.** [LLM06:2025 Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) attributes it to three root causes: "excessive functionality; excessive permissions; excessive autonomy," and prescribes human-in-the-loop approval for high-impact actions. > - **The UK's NCSC published interim guidance on the same problem.** Its [agentic AI post](https://www.ncsc.gov.uk/blogs/managing-the-cyber-risk-of-agentic-ai) (20 August 2026) recommends running agents in a sandbox, giving each agent a unique identity, restricting its credentials, and keeping a human in the loop for higher-risk actions. > - **The defenses these sources converge on are structural, not instructional.** Least-privilege tool allowlists, sandboxed execution, unique agent identity, and a human gate on mutating actions all limit what a poisoned tool can accomplish even if the model is fooled. **AI agent tool poisoning is the subversion of an autonomous agent through the tools it uses, by tampering with a tool's model-visible definition, its executable code, or the responses it returns.** As of 31 July 2026 it is a named technique in MITRE ATLAS (AML.T0110) rather than a research curiosity. For any team running an agent that calls tools against production, this reframes agent security: prompt injection is only one entry point, and the tool layer is now a documented attack surface with its own taxonomy. MITRE, OWASP, and the NCSC each describe that surface, and their recommended defenses overlap. ## What is AI agent tool poisoning (MITRE ATLAS AML.T0110)? Tool poisoning is tampering with the tools an agent trusts so that the agent's behavior diverges from what those tools claim to do. MITRE ATLAS defines the parent technique this way: "Adversaries may poison tools used by AI agents by introducing or modifying malicious content or behavior in a tool's model-visible definition, executable implementation, or runtime responses." The tools in scope are broad, explicitly including "Model Context Protocol (MCP) servers and tools, and agent skills or similar capability packages." The defining property is misrepresentation. In the ATLAS wording, "a poisoned tool is represented or trusted as performing a benign function, but its definition, implementation, or responses cause agent behavior or tool-mediated effects to differ materially from the represented function." The technique carries a created date of 30 March 2026 and a modified date of 31 July 2026, and MITRE rates its overall maturity "Realized." It is distinct from an agent faithfully retrieving malicious external data, which ATLAS files separately. This matters because most agent-security discussion still centers on prompt injection, where the attacker manipulates the text the model reads. Tool poisoning is a different surface: the attacker manipulates the capabilities the model is handed. ATLAS notes tool poisoning "frequently enables other techniques such as LLM Prompt Injection," so the two compound rather than compete. ## What are the three types of tool poisoning in MITRE ATLAS? ATLAS breaks AML.T0110 into three sub-techniques, each attacking a different layer of a tool, and each with its own maturity rating. | Sub-technique | ATLAS ID | What is poisoned | Maturity | |---|---|---|---| | Definition and Instructions | AML.T0110.000 | The tool's model-visible description, schema, parameter names, docstrings | Demonstrated | | Implementation | AML.T0110.001 | The tool's executable code, so normal invocation has hidden side effects | Realized | | Runtime Response | AML.T0110.002 | The content the tool returns, to steer the model's later reasoning | Feasible | The distinctions are precise. **Definition poisoning** works even if the tool is never called: MITRE notes "the model may process malicious instructions that are invisible or only partially visible to a human reviewer," because an agent often sees a fuller tool description than a person does in the UI. It also enables "tool shadowing," where one poisoned tool's definition alters how the agent invokes a different, trusted tool. **Implementation poisoning** needs no manipulation of the model at all. MITRE's example is concrete: "a poisoned email tool may send the requested message while silently adding an adversary-controlled blind-copy recipient." The malicious effect comes from code that runs on invocation, and MITRE notes installed copies "may continue to exhibit the poisoned behavior even after the malicious package or remote listing is removed." This sub-technique is rated "Realized," and its cited example is the Postmark MCP backdoor, a real npm package. **Runtime response poisoning** attacks the channel back into the model: a compromised tool "deliberately returning content intended to influence the model's subsequent reasoning, decisions, or actions." Because tool outputs are folded into the model's context, a poisoned response can direct the agent to call other tools or exfiltrate data, with "poisoned instructions mixed with legitimate results." ## What is Excessive Agency in the OWASP LLM Top 10? Excessive Agency is OWASP's name for giving an LLM-based system more capability, permission, or independence than the task requires. It is entry [LLM06:2025 Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) in the OWASP Top 10 for LLM Applications, and it is the design flaw that makes tool poisoning damaging. If the agent could only ever do a little, poisoning its tools would achieve little. OWASP attributes Excessive Agency to three root causes, quoted verbatim: "excessive functionality; excessive permissions; excessive autonomy." An agent wired to more tools than it needs, holding broader credentials than the task requires, or allowed to act without confirmation is exactly the agent a poisoned tool can turn into a weapon. The prescribed mitigations are a least-privilege checklist: minimize the tools an agent can reach, minimize each tool's functionality and permissions, execute tools in the user's own security context, and, for high-impact actions, "Utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken." OWASP's guidance elsewhere in the Top 10 is to implement authorization in the downstream systems the agent calls rather than trusting the model to decide what is allowed. That principle is what survives a poisoned tool: if the system enforcing the permission is not the thing that was poisoned, the attack is contained. ![Diagram of the three AI agent tool poisoning sub-techniques from MITRE ATLAS AML.T0110. A trusted tool sits between an AI agent and a downstream system. Definition and Instructions poisoning attacks the tool's model-visible description; Implementation poisoning attacks its executable code so normal calls have hidden side effects; Runtime Response poisoning attacks the content the tool returns to steer the model. A least-privilege allowlist, sandboxed execution, and a human gate on mutating actions limit what any of the three can accomplish.](/blog/ai-agent-tool-poisoning.svg) ## How do you secure autonomous AI agents in production? The three sources converge on structural controls rather than prompt-level fixes, because a poisoned tool defeats any defense that trusts the tool. The controls that hold are the ones enforced outside the agent's judgment. 1. **Least-privilege tool allowlists.** Expose the smallest set of tools that the task needs, and gate powerful tools behind an explicit allowlist rather than making everything callable by default. This is OWASP's "minimize extensions" applied directly, and it shrinks the surface a poisoned definition can exploit. 2. **Sandboxed execution.** Run tools in an isolated environment so a poisoned implementation cannot reach beyond its box. The NCSC states this directly: "Always run AI agents within a sandboxed environment that controls and manages what resources can and cannot be communicated with." 3. **Unique agent identity and scoped credentials.** The NCSC recommends that "All agents should be assigned their own unique identity in a class which differentiates them from human or individual systems," and that "you should restrict the credentials available to the agent." An agent that authenticates as itself, with narrow credentials, limits what a poisoned tool can do with its access. 4. **A human gate on mutating actions.** Both OWASP and the NCSC land here. The NCSC's model is "Human-in-the-loop: humans approve actions before they happen," and it adds that "for higher risk scenarios, we recommend having human oversight alongside technically enforced controls." A human approving the write is the control that a poisoned runtime response cannot forge. 5. **Authorization in downstream systems.** Enforce permissions in the systems the tools call, not in the model. If the poisoned component is not the component checking the permission, the check still holds. The NCSC guidance is explicitly interim, described by the agency as advice that formal guidance will "build upon, and ultimately supersede," so cite it as a current NCSC blog position rather than a finished standard. The MITRE and OWASP entries are the durable references. ## How does a least-privilege AI SRE hold up against tool poisoning? By making the structural controls the default rather than an option a user has to remember. An AI SRE that gates powerful tools behind an allowlist, runs execution in a sandbox, and denies mutating writes when no human is present is applying exactly the controls MITRE, OWASP, and the NCSC recommend, and doing so before any specific tool is trusted. [Aurora](https://github.com/Arvo-AI/aurora) is built on that footing. Its most powerful capabilities sit behind an explicit tool allowlist rather than being callable by default, it supports sandboxed Kubernetes execution for tool runs, and its structural chokepoint denies all mutating writes when no interactive human is present, so a poisoned tool cannot drive an unattended change into production. Remediation is limited to a pull request a human merges, which is OWASP's human-in-the-loop control applied to the highest-impact action. The layered design is described in [AI agent guardrails](/blog/ai-agent-guardrails), and the accountability trail that goes with it in [who is accountable when an AI agent changes prod](/blog/ai-agent-accountability-audit-evidence). None of that requires a novel defense; it requires refusing Excessive Agency by default. ## The summary As of summer 2026, tool poisoning is a named MITRE ATLAS technique (AML.T0110) with three sub-techniques, one already rated Realized with a real malicious MCP server as its example. OWASP names the enabling design flaw Excessive Agency and traces it to excessive functionality, permissions, and autonomy. The NCSC's interim guidance adds sandboxing, unique agent identity, and scoped credentials. The through-line is that prompt-level defenses do not survive a poisoned tool, and the controls that do are structural: least privilege, sandboxing, downstream authorization, and a human gate on mutating actions. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** AML.T0110 technique names, IDs, maturities, descriptions, and the created (30 March 2026) and modified (31 July 2026) dates are quoted from MITRE ATLAS release 2026.07 in the official mitre-atlas/atlas-data repository, verified 31 August 2026. OWASP wording is quoted from LLM06:2025 Excessive Agency in the OWASP Top 10 for LLM Applications repository. NCSC quotes are from "Managing the cyber risk of agentic AI," published 20 August 2026, which the NCSC labels interim advice. Aurora's controls are described from its open-source repository. --- ## Blast Radius in Automated Remediation: A 2026 Case Study URL: https://www.aurorasre.ai/blog/blast-radius-automated-remediation-azure-outage Published: 2026-08-31 (updated 2026-09-26) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Azure's July 2026 West US outage: an automated repair over-scoped its blast radius and its own rollback failed. What it teaches about gating remediation. > ## Key Takeaways > > - **A single defect in an automated blast radius calculation turned a routine repair into a regional outage, because the safety check validated each device on its own and never evaluated the aggregate effect.** Microsoft's own [post-incident review](https://azure.status.microsoft/en-us/status/history/?trackingid=ZJV6-SGG) (tracking ID ZJV6-SGG) states the defect "incorrectly expanded the scope of the repair event to include all optical devices." > - **The safety validation was structurally blind to the failure.** The PIR states the checks "validated each device individually rather than evaluating the aggregate effect of isolating all devices at once," a scenario the system "was never designed to process." > - **The automated rollback then failed for a predictable reason: it ran over the connectivity it had just severed.** Microsoft states that "because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful." > - **Impact ran from 14:44 to 19:41 UTC on 23 July 2026** in the West US region, per the PIR, and was mitigated only after engineers manually correlated the fault to the original repair and rolled back by hand. > - **The two design lessons are general, not Azure-specific.** A check that clears each unit individually can still approve an action that is catastrophic in aggregate, and a recovery path that depends on the system it is repairing will fail exactly when it is needed. > - **The containment lever is a bounded blast radius plus a human gate on mutating actions.** An AI SRE that suggests a fix and opens a pull request a human merges cannot over-scope its way into an outage the way a fully automated repair loop can. > - **This is a vendor-documented incident, reported here from Microsoft's published PIR.** No customer counts, percentages, or third-party numbers are asserted beyond what the PIR states. **Blast radius is the set of systems an action can affect, and automated remediation is only as safe as the calculation that bounds it.** Azure's West US outage on 23 July 2026 is a vendor-documented case where a defect in that calculation expanded a targeted repair to every optical device leaving a datacenter, and the automated rollback then failed because it depended on the connectivity the repair had cut. Microsoft published the full sequence in its post-incident review. The failure modes it describes are the reason [automated incident remediation](/blog/automated-incident-remediation) needs a bounded scope and a human gate rather than an unattended repair loop. ## What happened in the Azure West US outage on 23 July 2026? A break-fix repair on network hardware triggered an automated system that miscalculated which devices the repair would touch. According to Microsoft's [post-incident review](https://azure.status.microsoft/en-us/status/history/?trackingid=ZJV6-SGG), "a defect in our blast radius analysis system incorrectly expanded the scope of the repair event to include all optical devices" egressing a specific datacenter. Instead of isolating the intended hardware, the action cut network paths in and out of the region. The PIR is specific about the window. Between 14:44 and 19:41 UTC on 23 July 2026, a subset of customers in the West US region "experienced connectivity failures, increased latency, and/or difficulty accessing Azure services." Traffic staying inside the region was unaffected; the damage was to traffic entering or leaving it. The incident is worth studying precisely because it was not a model, an attacker, or a novel bug. It was an automation doing exactly what it was told, over a scope it should never have been allowed to select. ## How did an automated repair expand its own blast radius? The safety check was correct for one device and wrong for all of them at once. Microsoft states the validation "validated each device individually rather than evaluating the aggregate effect of isolating all devices at once," and that this case "was not accounted for because the system was never designed to process a full datacenter's worth of devices in a single request." This is the core lesson, and it generalizes far beyond optical hardware. A per-unit safety check answers the question "is it safe to change this one thing?" It does not answer "is it safe to change all of these things together?" The two questions have different answers whenever the units share a dependency, and network egress paths always do. An automated system that only asks the first question will approve an aggregate action that no operator would have signed off on. | The check the system ran | The check it needed | |---|---| | Is isolating this device safe on its own? | Is isolating this set of devices safe together? | | Passed for every device individually | Would have failed for the aggregate | | Scope selected by an automated calculation | Scope bounded and reviewed before execution | | Never designed for a full datacenter in one request | Must refuse or escalate an out-of-envelope request | The design fix Microsoft lists is to block simultaneous changes across all diverse paths and to add input validation so a single request cannot select an entire datacenter's devices. The behavioral principle underneath it is that a blast radius calculation has to be evaluated in aggregate and bounded by an envelope the automation cannot exceed on its own. ## Why did the automated rollback fail? Because the rollback ran over the same infrastructure the repair had broken. Microsoft states plainly that "because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful." The recovery system kept retrying against connectivity that no longer existed. The timeline shows the cost of that dependency. Automated recovery began retrying within minutes, at 14:54 UTC, but the routing anomaly was initially triaged to a different cause. Recovery was reached only after engineers "correlated the routing anomaly with the initial break-fix trigger event" at 17:19, initiated a manual rollback at 17:45, and completed it across all affected devices by 18:26. Full mitigation landed at 19:41. Microsoft's remediation list includes making the automated recovery "fail more quickly and escalate to engineers more promptly when retries" are unsuccessful, "rather than continuing to retry against disrupted connectivity." ![Diagram of the Azure West US remediation failure. An automated repair with a defective blast radius calculation isolates all optical devices egressing a datacenter, cutting regional network paths. The automated rollback attempts to recover but depends on the same connectivity that was severed, so its retries fail. Recovery only completes after engineers manually correlate the fault to the original repair and roll back by hand.](/blog/blast-radius-remediation-failure.svg) The general form of this failure is a recovery mechanism whose reachability depends on the health of the thing it repairs. It is a single point of failure hiding inside the safety system. When the repair goes wrong, the recovery path is the first thing to go dark. ## What does this teach about automated remediation in general? Two things, and both are design constraints rather than opinions. First, a per-unit safety check is not an aggregate safety check. Any automation that can select its own scope needs an explicit envelope it cannot exceed, and the aggregate effect has to be evaluated before execution, not device by device. This is the same reasoning behind bounding an investigation's [blast radius as a dependency traversal](/blog/root-cause-analysis-complete-guide-sres): the value is in knowing what a change can reach before the change runs. The same aggregate-versus-unit blindness appears in [how a retry loop took down GitHub](/blog/github-retry-storm-outage-2026), where a safeguard watched one healthy signal while the resource it was meant to protect ran to exhaustion. Second, a recovery path that depends on the system it recovers is not a recovery path. Rollback and escalation have to remain reachable when the primary system is degraded, which usually means failing fast to a human rather than retrying into a void. Both lessons point the same direction: the safe default for a mutating action against production is a bounded scope and a human gate, not an unattended loop that both applies and reverts changes on its own. The industry converged on that boundary through 2026, with investigation treated as automatable and mutation treated as human-gated, a pattern covered in [how vendors gate AI agent autonomy](/blog/ai-agent-autonomy-gated-production). ## How does a gated AI SRE avoid this failure mode? By never holding the authority that made the Azure loop dangerous. An AI SRE that diagnoses an incident and then suggests a fix, opening a remediation pull request for a human to merge, cannot expand its own blast radius into an outage, because it does not execute the change. The scope decision and the execution both stay with a person. [Aurora](https://github.com/Arvo-AI/aurora) is built on that separation. It investigates and can open a remediation pull request on GitHub, but there is no auto-merge and no unattended repair loop. Its structural chokepoint, `BACKGROUND_DENIED`, denies mutating writes when no interactive human is present, so an automated run cannot apply a change on its own regardless of what it concludes. Its blast radius work is a bounded dependency traversal used to inform a human, not a scope an automation selects and then acts on. The layered controls behind that posture are detailed in [AI agent guardrails](/blog/ai-agent-guardrails). None of this makes automation the enemy. The Azure PIR is a clear account of a specific automation failing in two specific ways, and the takeaway is not to stop automating repairs. It is to bound the scope, evaluate it in aggregate, keep the recovery path independent of the thing it recovers, and put a human on the mutating action. ## The summary The Azure West US outage on 23 July 2026 is a vendor-documented lesson in two failure modes of automated remediation. A blast radius calculation that validated each device individually approved an aggregate action that took out regional connectivity, and an automated rollback that depended on that same connectivity could not recover. Both are general design faults, and both are avoided by the same discipline: bound the scope an automation can select, evaluate the aggregate effect before executing, keep recovery independent of the repaired system, and gate the mutating action behind a human. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora multi-cloud investigation](/multi-cloud) > **Sourcing note.** All incident facts, quotes, the tracking ID (ZJV6-SGG), the UTC timeline, and the remediation items are from Microsoft's published post-incident review at azure.status.microsoft, verified 31 August 2026. No customer counts, affected-percentage figures, or third-party numbers are asserted; the PIR states only "a subset of customers" and gives no such figures. Aurora's gated-remediation posture is described from its open-source repository. --- ## Why Multi-Agent AI Systems Fail in Production URL: https://www.aurorasre.ai/blog/why-multi-agent-ai-systems-fail Published: 2026-08-27 (updated 2026-09-26) Author: Noah Casarotto-Dinning, CEO at Arvo AI > More agents is not more reliable. What 2026 research shows about authorization drift, compounding errors, and why failure attribution gets harder. > ## Key Takeaways > > - **Adding more AI agents does not make a system more reliable; past a point it makes it less reliable and much harder to debug.** The 2026 research measures three specific failure modes: authorization drift, compounding errors, and the difficulty of attributing a failure to the agent that caused it. > - **Authorization boundaries leak as agents delegate to each other.** A [600-task benchmark](https://arxiv.org/abs/2608.07556) found centralized agent hierarchies hit 93.9 to 98.6% task completion but took unauthorized actions in "2.7--19.8% of tasks versus 0.6--0.8%" for peer networks, "a gap that widens with hierarchy depth." > - **Errors compound because agents are stateful.** Anthropic's own engineering writeup states that in agent systems "minor system failures can be catastrophic for agents," which are "non-deterministic between runs, even with identical prompts," so "minor changes cascade into large behavioral changes." > - **Multi-agent systems are expensive.** Anthropic reports that "multi-agent systems use about 15x more tokens than chats," which turns coordination overhead into a real per-investigation cost. > - **Finding which agent failed is itself an unsolved problem.** A [2026 preprint](https://arxiv.org/abs/2608.10646) on failure attribution reports single-digit-to-double-digit gains over baselines, evidence that pinning a failure to a specific agent and step is still hard, not solved. > - **The design response is to default to fewer agents, not more.** Aurora runs a single investigation agent by default; multi-agent orchestration is opt-in, not the stock configuration. Rationale below. > - **None of this is an argument against AI in operations. It is an argument against unbounded fan-out.** The reliability lever is scope and authorization, not agent count. **A multi-agent AI system uses several coordinating agents to solve a task, and the 2026 research is consistent that adding agents trades away reliability and debuggability past a surprisingly low point.** Authorization boundaries leak between agents, errors compound because each agent is stateful, and attributing a failure to the agent that caused it becomes its own hard problem. For incident response, where an agent holds real credentials against production, those failure modes are not academic, and they are why an open-source AI SRE agent can reasonably default to one agent rather than a swarm. ## Does adding more AI agents make a system more reliable? No, and the assumption that it does is the trap. Intuition says more agents means more coverage and more parallelism. The measured reality is that coordination introduces failure modes a single agent does not have, and those failure modes scale with the number and depth of agents. The clearest evidence is a benchmark built specifically to test this. [MasDrift](https://arxiv.org/abs/2608.07556) (arXiv:2608.07556, a 2026 preprint) evaluated authorization preservation across single-agent, centralized-hierarchy, and decentralized peer-network architectures over 600 tasks. The result is a direct tradeoff: "centralized hierarchies achieve 93.9--98.6% task completion versus 85.7--87.0% for peer networks," but they take "unauthorized actions [in] 2.7--19.8% of tasks versus 0.6--0.8%" for the peer networks, and the authors note this is "a gap that widens with hierarchy depth." More coordination bought more task completion and far more unauthorized action. Read that as an operator. A system that completes 98% of tasks but takes an unauthorized action in up to one task in five is not a reliable system for touching production. The completion rate is the number a demo shows. The unauthorized-action rate is the number an incident review finds. | Architecture | Task completion | Unauthorized actions | Coordination cost | Failure attribution | |---|---|---|---|---| | **Single agent** | Baseline, no delegation | Lowest, no delegation chain | Lowest tokens | One linear trace | | **Peer network** | 85.7 to 87.0% | 0.6 to 0.8% of tasks | Higher | Distributed, harder | | **Centralized hierarchy** | 93.9 to 98.6% | 2.7 to 19.8% of tasks | Highest | Hardest, widens with depth | Completion and unauthorized-action figures are from [MasDrift](https://arxiv.org/abs/2608.07556); coordination cost and attribution reflect the token and tracing findings covered below. The pattern is consistent: the architecture that completes the most tasks also takes the most unauthorized actions and is the hardest to debug. ## What is authorization drift in multi-agent systems? Authorization drift is the loss of an action's original permission boundary as a goal is delegated from one agent to another. Agent A is authorized to do X. It delegates a sub-goal to agent B, which decides the sub-goal requires Y, and nobody re-checks whether the original request authorized Y. The permission attached to the top-level goal does not travel intact down the delegation chain. MasDrift is precise about why depth makes this worse: the deeper the hierarchy, the more delegation hops separate the action from the authorization that justified it, and the wider the drift. The paper tested defenses and found a real but partial fix. "Re-anchoring reduces unauthorized actions in every model configuration we evaluate, at a cost of 1.6 points" of completion, while a stricter chain-propagation defense "blocks required work instead, forfeiting up to 36.3 points." There is no free defense: tightening authorization costs task completion, and the strict version costs a lot. This maps directly onto the security guidance for agents. OWASP's [LLM06 Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) advises implementing "authorization in downstream systems rather than relying on an LLM to decide if an action is allowed." Authorization drift is what happens when that advice is not followed and each agent trusts the goal it was handed. ![Diagram of authorization drift. A top-level agent authorized for a read-only investigation delegates a sub-goal to a second agent, which delegates again to a third. At each hop the original permission boundary blurs, until the deepest agent attempts a mutating action the original request never authorized. The permission does not travel intact down the delegation chain.](/blog/multi-agent-authorization-drift.svg) ## Why do errors compound in agent systems? Because agents are stateful and sequential, so an early mistake becomes the input to every later step. This is not a fringe claim; it is Anthropic's own account of building agent systems. Anthropic's engineering writeup on [multi-agent systems](https://www.anthropic.com/engineering/multi-agent-research-system) states the mechanism plainly: agents "make dynamic decisions and are non-deterministic between runs, even with identical prompts," and because they "maintain state across many tool calls," "minor system failures can be catastrophic for agents." In its companion guidance on [building effective agents](https://www.anthropic.com/engineering/building-effective-agents), Anthropic is explicit about the cost of the autonomy: "the autonomous nature of agents means higher costs, and the potential for compounding errors," which is why it recommends "extensive testing in sandboxed environments, along with the appropriate guardrails." Two consequences follow for anyone running agents against production. First, non-determinism means the same incident can produce different investigations on different runs, so a single good demo is weak evidence. Second, compounding means the value of catching an error early is high, because an unchecked wrong step propagates. Both push toward bounded scope and toward verifying intermediate results rather than trusting a long autonomous chain. ## Multi-agent coordination is also expensive The cost is not incidental. Anthropic reports that "multi-agent systems use about 15x more tokens than chats," and that even single agents use "about 4x more tokens than chat interactions." Coordination overhead, repeated context, and inter-agent messaging all consume tokens that a single-agent investigation does not. For incident response this lands on the per-investigation bill. An architecture that fans out to many agents for a routine incident pays a large multiplier for coordination that a bounded single-agent investigation avoids. The economics of a single run are covered in [what an AI investigation actually costs](/blog/ai-investigation-cost); the point here is narrower, that agent count is itself a cost driver, not only a reliability one. ## Why is it hard to tell which agent failed? Because responsibility is distributed, and the trace of who did what, when, and why is exactly what multi-agent systems obscure. When a single agent fails, the log is a single sequence. When several agents coordinate, a failure is spread across agents and steps, and identifying the responsible one is a research problem in its own right. A [2026 preprint on failure attribution](https://arxiv.org/abs/2608.10646) (arXiv:2608.10646) builds a model specifically to answer "who caused failures, when they occurred, and why," and reports gains over prior baselines of "5.83%+" for faulty-agent detection, "10.63%+" for faulty-step detection, and "14.73%+" for failure-mode detection. Those are improvements over methods that were themselves weak. The takeaway is not the specific numbers, which are relative gains on a preprint; it is that attributing a multi-agent failure to a specific agent and step is an open problem being actively worked on, not a solved one. If you cannot reliably attribute the failure, you cannot reliably fix it, and you cannot produce the audit evidence covered in [who is accountable when an AI agent changes prod](/blog/ai-agent-accountability-audit-evidence). ## What does this mean for AI in incident response? The research does not say do not use agents. It says do not fan out without a reason, and bound authorization when you do. That is why [Aurora](https://github.com/Arvo-AI/aurora) runs a single investigation agent by default. Its multi-agent orchestrator exists but is opt-in rather than the stock configuration, so the default install is one LangGraph ReAct agent, not a swarm. This is a deliberate choice against the failure modes above: one agent has no delegation chain to leak authorization across, one linear trace to attribute a failure to, and no coordination-token multiplier. When more agents are warranted, they are turned on knowingly, with the tradeoff understood. That same choice, single agent by default versus a swarm by default, is what separates the vendor launches compared in [parallel-agent incident investigation: AWS vs BigPanda](/blog/parallel-agent-incident-investigation-2026). The authorization boundary is enforced structurally regardless of agent count. All mutating writes are denied when no interactive human is present, and remediation is limited to a pull request a human merges, so even an orchestrated run cannot drift its way into changing production. That is the OWASP prescription applied: authorization lives in the system the agents call, not in each agent's judgment. The layered controls are described in [AI agent guardrails](/blog/ai-agent-guardrails), and where the whole category drew the autonomy line in 2026 is covered in [how vendors gate AI agent autonomy](/blog/ai-agent-autonomy-gated-production). ## The summary More agents is a reliability liability, not a reliability feature, until proven otherwise for a specific task. The 2026 research measures three costs of coordination: authorization drift that widens with depth, errors that compound because agents are stateful and non-deterministic, and a failure-attribution problem that is still open. Each one is worse when an agent holds production credentials. The defensible default is fewer agents with hard authorization boundaries, adding coordination only when a task genuinely needs it, and enforcing permissions in the systems the agents call rather than in the agents themselves. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** Authorization and completion figures are from MasDrift (arXiv:2608.07556), a 2026 preprint. Statefulness, non-determinism, compounding-error, and token-multiplier claims are quoted from Anthropic's engineering writeups on multi-agent systems and building effective agents. Failure-attribution gains are from arXiv:2608.10646, a 2026 preprint reporting relative improvements over baselines. Preprints are flagged as such and are not peer-reviewed. Aurora's default single-agent configuration is described from its open-source repository. Verified August 27, 2026. --- ## AI Agent Autonomy: How Vendors Gate It in 2026 URL: https://www.aurorasre.ai/blog/ai-agent-autonomy-gated-production Published: 2026-08-27 (updated 2026-09-26) Author: Noah Casarotto-Dinning, CEO at Arvo AI > In summer 2026 four incident vendors converged on the same rule: agents investigate autonomously, but a human approves any action that changes production. > ## Key Takeaways > > - **In 2026 the autonomy line settled in the same place across the category: agents may investigate on their own, but a human approves any action that mutates production.** Four vendors published the same boundary within four weeks, and the security frameworks say the same thing. > - **NeuBird framed it as earned autonomy.** Its [August 25, 2026 post](https://neubird.ai/blog/earned-autonomy-ai-agents-in-production) argues "an AI agent needs a ladder of trust in production just as rigorous as the one we build for human engineers," and names the first rung as autonomous rollback, "a well-scoped, reversible action." > - **Dynatrace shipped autonomous agents wrapped in governance.** Its [July 27, 2026 Autonomous Operations launch](https://www.dynatrace.com/news/press-release/autonomous-operations-enterprise-ai/) says every action is "designed to be transparent, auditable, and governed," with "human approval and intervention where required." > - **PagerDuty split triage from action.** Its [August 5, 2026 SRE Agent update](https://www.pagerduty.com/blog/ai/sre-agent-enhancements-faster-triage-greater-access-controls-deeper-system-connectivity/) has the agent "begin autonomous triage immediately," then "the responder reviews the recommendation, runs the workflow, and the agent confirms the fix worked." > - **BigPanda gated the mutations, not the investigation.** Its [August 14, 2026 post](https://www.bigpanda.io/blog/improved-bigpanda-ai-incident-assistant/) lets agents "investigate multiple hypotheses simultaneously," while "mutating actions (i.e., paging on-call or creating a ticket) are governed by configurable approval policies." > - **The security guidance is explicit about the same rule.** OWASP's [LLM06 Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) says to "require a human to approve high-impact actions before they are taken" and to "implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed." > - **Aurora enforces this structurally rather than by instruction.** A single chokepoint denies every mutating write when no interactive human is present, and remediation is limited to a pull request a human merges. Details and caveats below. **AI agent autonomy in incident response is the degree to which an agent can act on production without a human in the loop.** In 2026 the category converged on a narrow answer: investigation is autonomous, but any action that changes production waits for a human to approve it. That boundary is not a marketing accident. Four incident vendors published it within four weeks of each other in summer 2026, and the security frameworks that cover autonomous agents describe the same control in near-identical language. ## Why did every AI SRE vendor gate autonomy the same way in 2026? Because the failure mode is shared, and so is the fix. An agent that reads infrastructure is low risk. An agent that changes it holds the same power as a privileged human and none of the accountability, so the industry landed on the oldest control in operations: a human approves the change. The convergence is dense enough to date. NeuBird published its earned-autonomy argument on [August 25, 2026](https://neubird.ai/blog/earned-autonomy-ai-agents-in-production). Dynatrace launched Autonomous Operations on [July 27, 2026](https://www.dynatrace.com/news/press-release/autonomous-operations-enterprise-ai/). PagerDuty shipped its SRE Agent triage and access-control update on [August 5, 2026](https://www.pagerduty.com/blog/ai/sre-agent-enhancements-faster-triage-greater-access-controls-deeper-system-connectivity/). BigPanda rearchitected its AI Incident Assistant on [August 14, 2026](https://www.bigpanda.io/blog/improved-bigpanda-ai-incident-assistant/). Each one lets the agent investigate freely and gates the actions that touch production. The frameworks arrived at the boundary independently. OWASP's [LLM06 Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) entry lists two mitigations that read like a design spec: "utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken," and "implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not." The second sentence is the important one. Asking the model to police itself is not a control. Enforcing permissions in the system the model calls is. ## Where does each vendor actually draw the line? The useful comparison is not "does it have autonomy" but "what can it do without a human, and what waits." The table below uses each vendor's own published wording from the sources above. | Vendor | Autonomous without a human | Gated behind a human | Source date | |---|---|---|---| | **NeuBird** | Investigation; proposes autonomous rollback as the first earned privilege | Broader action, until trust is "earned" up a "ladder of trust" | Aug 25, 2026 | | **Dynatrace** | Triage that "triggers autonomously on newly detected problems" | Remediation, with "human approval and intervention where required" | Jul 27, 2026 | | **PagerDuty** | "Autonomous triage immediately" when an incident triggers | Workflow execution: "the responder reviews the recommendation, runs the workflow" | Aug 5, 2026 | | **BigPanda** | Multi-hypothesis "Swarm Investigation" | "Mutating actions ... are governed by configurable approval policies" | Aug 14, 2026 | | **Aurora** | Read-only investigation across connected clouds and Kubernetes | All mutating writes when no human is present; remediation via human-merged PR | [Repository](https://github.com/Arvo-AI/aurora) | Two patterns fall out of the table. First, nobody gates investigation. Reading logs, querying a cluster, correlating alerts, and forming a hypothesis are treated as safe for an agent to do alone, and the whole category agrees. Second, the disagreement is only about *what counts as an action*. BigPanda counts paging a human and opening a ticket as mutating actions worth an approval policy. NeuBird counts rollback as safe enough to be the first thing an agent earns, on the argument that undoing a known-bad change is reversible by definition. ## Is gated autonomy the same as no autonomy? No, and the distinction matters for buyers. A gated agent still removes most of the manual work of an incident. It reads the telemetry, runs the queries a responder would run, and arrives at the approval step with a proposed action and the evidence behind it. What it does not do is press the button. PagerDuty's wording captures the split precisely: the agent begins "autonomous triage immediately," and "investigation and analysis can be well underway before a responder ever acknowledges the page," but the mutation still routes through a person who "reviews the recommendation, runs the workflow." The time saved is the investigation time. The human keeps the decision. Google's [graded autonomy framework](https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/) formalizes this as a ladder from L0 (everything human) to L4 (the agent devises and executes). Its L2 requires a human to "explicitly Approve any plan before the system proceeds with an actuation." What makes Google's version credible is the machinery around the higher rungs: an agent that "detects an elevated risk score or an anomalous production state" automatically "downgrade[s] the request" back to human approval, and there are "emergency 'Red Button' endpoints" to "instantly pause all in-flight agentic actions." Autonomy that can be revoked in one action is a different risk than autonomy that cannot. ![Diagram of the 2026 autonomy line. On the left, investigation tasks agents run without a human: reading telemetry, querying clusters, correlating alerts, forming a hypothesis. On the right, mutating actions gated behind human approval: deploying a change, opening a pull request, paging on-call, running a remediation workflow. A vertical approval gate separates the two.](/blog/ai-agent-autonomy-line.svg) ## What does earned autonomy actually require? NeuBird's phrase is the sharpest framing in the batch, so it is worth taking literally. Its [post](https://neubird.ai/blog/earned-autonomy-ai-agents-in-production) argues that "an AI agent needs a ladder of trust in production just as rigorous as the one we build for human engineers," and that the first privilege an agent should earn is autonomous rollback, because "if an agent can prove with confidence that a specific change caused an incident, undoing that change is a well-scoped, reversible action." The load-bearing word is *reversible*. A rollback to a known-good state is bounded: the worst case is a return to the state five minutes ago. An open-ended fix is not bounded, because the agent is composing a new action whose blast radius it cannot fully predict. Earned autonomy, read carefully, is not a trust score attached to a model. It is a claim about the reversibility of a specific action class. That is a more defensible line than "the agent is 95% accurate, so let it act," because reversibility is a property of the operation, not an estimate of the model. It also exposes what earned autonomy is not. It is not the same as an eval score. No commercial AI SRE agent has been submitted to an independent third-party benchmark, a gap covered in [AI SRE accuracy](/blog/ai-sre-accuracy-benchmarks). Autonomy earned on the reversibility of the action survives that gap. Autonomy justified by an unpublished accuracy figure does not. ## How does an open-source agent enforce the line? Everything above is a claim a vendor makes in a blog post. In an open-source agent the boundary is readable in source rather than asserted, which is the reason to describe [Aurora](https://github.com/Arvo-AI/aurora) here concretely rather than abstractly. Aurora enforces the boundary at a single structural chokepoint rather than through a system-prompt instruction. When no interactive human is present in a session, all structured mutating writes are denied, so a background run cannot decide to change production on its own. This is the mechanism OWASP describes: authorization enforced in the system the model calls, not a request that the model behave. A prompt telling an agent not to act is exactly the control that failed in the [Replit database deletion](https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/), where an agent deleted a production database during an explicit code freeze. Remediation follows the same rule. Aurora suggests a fix and can open a remediation pull request on GitHub or Bitbucket, gated on human approval, with no auto-merge. The alert-correlation engine links related alerts into an investigation and never suppresses or mutes them, so the agent cannot silence a signal on its own. Command execution can run in a sandboxed Kubernetes pod when `ENABLE_POD_ISOLATION` is enabled, and destructive actions are human-gated regardless. None of that requires trusting an accuracy number, because none of it lets the agent mutate production without a person. For the layered version of the runtime controls, see [AI agent guardrails](/blog/ai-agent-guardrails). For what an agent owes the human it wakes at the approval step, see [the agent-to-human handoff](/blog/ai-agent-human-handoff). For where remediation fits, see [automated incident remediation](/blog/automated-incident-remediation) and the [root cause analysis guide for SREs](/blog/root-cause-analysis-complete-guide-sres). ## What should a buyer ask in 2026? The convergence makes the buying question concrete. Every vendor now says it gates autonomy, so the differentiator is how, not whether. 1. **What exact action classes can the agent perform with no human?** Get the list, not the adjective. "Autonomous" without a list of permitted mutations is not an answer. 2. **Is the gate enforced in the executing system or in the prompt?** Prompt-level gates are the control that failed at Replit. Ask where authorization lives. 3. **Can the whole thing be stopped in one action?** Google publishes a red button. Ask for the equivalent. 4. **Is any autonomous action reversible by construction?** Rollback is. An open-ended fix is not. Grade the two differently. 5. **Is the boundary auditable, or do you take it on trust?** In an open-source agent the answer is readable in source. In a closed product it is a promise. ## The summary The 2026 story is not that AI SRE agents became autonomous. It is that the whole category agreed on where autonomy stops. Investigation runs without a human. Anything that changes production waits for one. NeuBird, Dynatrace, PagerDuty, and BigPanda published that boundary within a month of each other, and OWASP and Google describe the same control. The remaining question for a buyer is purely mechanical: what can the agent do alone, where is that limit enforced, and can it be revoked in a single action. A vendor that answers those three cleanly has a defensible autonomy story. One that answers with an accuracy percentage does not. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **See it on an incident:** [Aurora root cause analysis](/rca) > **Sourcing note.** Vendor positions are quoted verbatim from primary posts dated July 27 to August 25, 2026: NeuBird, Dynatrace, PagerDuty, and BigPanda. Framework language is from OWASP's LLM06 Excessive Agency entry and Google's graded autonomy framework. Aurora's behavior is described from its open-source repository. No Aurora accuracy or MTTR figure is claimed, because no such benchmark exists. Verified August 27, 2026. --- ## DORA and Frontier AI: What the 2026 ESA Statement Says URL: https://www.aurorasre.ai/blog/dora-frontier-ai-esa-statement Published: 2026-08-24 (updated 2026-09-26) Author: Noah Casarotto-Dinning, CEO at Arvo AI > On 31 July 2026 the EU's three financial regulators tied frontier AI risk to DORA oversight. What the statement requires, and what it does not. > ## Key Takeaways > > - **The EU's three financial supervisors issued a joint statement on frontier AI on 31 July 2026.** The [EBA, EIOPA and ESMA statement](https://www.esma.europa.eu/press-news/esma-news/eba-eiopa-and-esma-call-enhanced-governance-and-consistent-supervision) (reference JC 2026 25) calls "for a cross-sectoral, risk-based and consistent supervisory approach to mitigate the ICT risks stemming from frontier AI models." > - **It routes through DORA, not the AI Act.** The statement points to "updates on ongoing and planned DORA oversight activities for critical ICT third-party providers (CTPPs) to address this risk," placing frontier-AI risk inside the Digital Operational Resilience Act's existing machinery rather than new law. > - **The ask is governance, not a ban.** Regulators say "financial entities should have robust governance and risk management frameworks in place," with "emphasis is placed on the prevention, detection and management of these risks." > - **DORA has applied since 17 January 2025.** Per [EIOPA](https://www.eiopa.europa.eu/digital-operational-resilience-act-dora_en), the regulation "entered into application on 17 Jan 2025," so its incident-reporting and third-party obligations are already live for financial entities. > - **The statement is a supervisory-dialogue tool, not a binding rule.** Regulators "encourage both financial entities and competent authorities to use the statement as a basis for supervisory dialogue," which means it shapes examinations rather than creating new penalties. > - **Most AI incident tooling is not "high-risk" under the AI Act.** Under [Annex III](https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng), operational AI is high-risk only when it is a safety component in the management of critical digital infrastructure, a narrow category that ordinary investigation assistants do not meet. **Frontier AI in EU financial services is now a digital-operational-resilience question: on 31 July 2026 the EBA, EIOPA and ESMA jointly asked supervisors to treat its ICT risks under DORA.** The request is to use DORA's existing oversight framework rather than wait for new legislation. That reframing matters because DORA already applies, and its obligations attach to how financial entities run and depend on technology, including the AI systems now entering incident response. What the [joint statement](https://www.esma.europa.eu/press-news/esma-news/eba-eiopa-and-esma-call-enhanced-governance-and-consistent-supervision) actually says is narrower than the compliance claims that tend to get built on top of it. ## What did the EBA, EIOPA and ESMA actually say? The three European Supervisory Authorities (ESAs) published a joint statement, referenced JC 2026 25, dated 31 July 2026. The headline is precise about scope: the regulators "call for enhanced governance and consistent supervision to mitigate ICT risks from frontier AI models in the EU financial sector." Three phrases carry the substance. First, the approach: "a cross-sectoral, risk-based and consistent supervisory approach to mitigate the ICT risks stemming from frontier AI models." Second, the objective: "measures to help financial entities strengthen their operational resilience against cyber risks linked to frontier AI models," where "emphasis is placed on the prevention, detection and management of these risks." Third, the expectation on firms: "financial entities should have robust governance and risk management frameworks in place." The statement's own status is the part most easily overstated. It is not a regulation and not a technical standard. The ESAs "encourage both financial entities and competent authorities to use the statement as a basis for supervisory dialogue." In practice that means it will inform how examiners ask questions, not that a new fine schedule exists. ## Why is this a DORA matter rather than an AI Act matter? Because the statement explicitly connects frontier-AI risk to DORA's oversight, not to the AI Act. It references "updates on ongoing and planned DORA oversight activities for critical ICT third-party providers (CTPPs) to address this risk." That choice is deliberate. The [Digital Operational Resilience Act](https://www.eiopa.europa.eu/digital-operational-resilience-act-dora_en) "entered into application on 17 Jan 2025" and already governs how financial entities manage ICT risk, report major ICT-related incidents, test resilience, and manage third-party dependencies. It also "establishes an EU-wide oversight framework for critical ICT third-party providers (CTPPs)." A frontier model accessed as a service, or the vendor that hosts it, can fall inside that third-party perimeter. Routing the risk through DORA means no new statute is required to start asking about it. The table below maps the regulation's existing pillars to the frontier-AI concern the ESAs raised. | DORA pillar | What it already requires | Frontier-AI relevance | |---|---|---| | ICT risk management | Governance and risk frameworks for ICT | The statement's core ask: "robust governance and risk management frameworks" for AI | | ICT third-party risk | Monitoring providers, contractual provisions | A hosted frontier model is a third-party dependency to inventory and assess | | CTPP oversight | EU-wide oversight of critical providers | The named mechanism for "planned DORA oversight activities" on AI risk | | Incident reporting | Report major ICT-related incidents | AI-involved failures are ICT incidents, not a separate category | | Resilience testing | Basic and advanced testing | Prevention and detection extend to AI-linked failure modes | ![Diagram showing the 31 July 2026 ESA statement routing frontier-AI risk through DORA oversight rather than the EU AI Act, with the four DORA pillars applied to AI.](/blog/dora-frontier-ai-routing.svg) ## Does this make AI incident tooling "high-risk" under the EU AI Act? Generally no, and conflating the two regimes is the most common error in coverage of this topic. The AI Act and DORA are separate instruments with separate triggers. Under the AI Act's [Annex III](https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng), operational AI is classified high-risk only when it functions as "a safety component in the management and operation of critical digital infrastructure." An assistant that reads logs, correlates a deploy with an onset time, and drafts a root-cause hypothesis for a human is not managing critical infrastructure. The high-risk label attaches to a narrow set of control-plane roles, not to investigation support. Timing reinforces the separation. The AI Act's high-risk application dates were moved by the Digital Omnibus amendment ([COM(2025) 836 final](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52025PC0836)), which sets backstop dates of "2 December 2027 as regards AI systems classified as high-risk pursuant to Article 6(2) and Annex III" and "2 August 2028 as regards AI systems classified as high-risk pursuant to Article 6(1) and Annex I." The DORA obligations discussed here, by contrast, are already in force. ## What should a financial entity running AI in operations do now? The statement asks for governance, prevention, detection and management. Translated into operational terms, that is mostly work DORA already expects, applied to the AI systems now in the incident path. 1. **Inventory the AI dependency as a third party.** If an investigation tool calls a hosted frontier model, that model provider is an ICT third party under DORA. Record it, assess concentration risk, and check the contractual provisions DORA requires. 2. **Keep a human in the decision loop for anything that changes production.** Prevention and detection are easier to evidence when an agent proposes and a person approves. Who is accountable when an agent acts is its own question, covered in [AI agent accountability and audit evidence](/blog/ai-agent-accountability-audit-evidence). 3. **Capture audit evidence at the point of action.** Supervisory dialogue rewards records that show what the system did, when, and under whose approval. Guardrail and approval events are the material examiners can actually read, discussed in [AI agent guardrails in production](/blog/ai-agent-guardrails). 4. **Treat AI-involved failures as ICT incidents.** They report through the same DORA channel as any other major ICT-related incident, not a separate AI process. 5. **Reduce third-party surface where you can.** A self-hosted, open-source tool that runs in your own environment adds fewer external dependencies to the CTPP calculation than a closed SaaS that becomes a critical provider in its own right. This is a structural point, not a compliance guarantee. ## How does an open-source, self-hosted investigation tool fit DORA's third-party concern? The relevant property is where the software and its data live. DORA's third-party pillar exists because outsourced ICT concentrates risk in providers a financial entity does not control. A tool that a bank self-hosts, under an [Apache 2.0 license](https://github.com/Arvo-AI/aurora), does not become a new critical ICT third-party provider in the way a closed hosted platform can. Aurora is built around human-gated action rather than autonomy: its structural chokepoint denies mutating writes when no interactive human is present, and its guardrail events are recorded with sha256 fingerprints that never log raw command content. Those are governance and evidence properties, not a claim of DORA compliance. Compliance is a determination a financial entity and its supervisor make about the whole control environment, not a feature a tool ships. Self-hosting and human-gating make the governance story easier to tell, and the [open-source repository](https://github.com/Arvo-AI/aurora) lets a risk team read the controls rather than take them on trust. For the incident-management context these controls sit inside, see [open source incident management](/blog/open-source-incident-management) and the [root cause analysis guide for SREs](/blog/root-cause-analysis-complete-guide-sres). ## Try it in your own environment Aurora is Apache 2.0 and self-hostable, so a financial-sector platform team can evaluate it inside its own perimeter rather than sending incident data to an external service. One LLM API key is the only hard requirement; cloud connectors are optional, and local models via Ollama support fully self-contained deployments. ```bash git clone https://github.com/arvo-ai/aurora.git && cd aurora make init # generates secrets, copies .env.example to .env nano .env # add OPENROUTER_API_KEY (or OPENAI_API_KEY / ANTHROPIC_API_KEY) make prod-prebuilt # pulls prebuilt images from GHCR and starts ``` Open **http://localhost:3000**. The first user to register becomes admin. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Book a demo:** [cal.com/arvo-ai/demo](https://cal.com/arvo-ai/demo) - **Related guides:** [AI Agent Accountability and Audit Evidence](/blog/ai-agent-accountability-audit-evidence) · [AI Agent Guardrails in Production](/blog/ai-agent-guardrails) · [Open Source Incident Management](/blog/open-source-incident-management) · [Root Cause Analysis for SREs](/blog/root-cause-analysis-complete-guide-sres) > **Sourcing note.** The joint statement, its reference number (JC 2026 25), and every quoted phrase come from the ESAs' own press release dated 31 July 2026. DORA's application date and oversight-framework description come from EIOPA's DORA page. AI Act high-risk classification and the moved application dates come from the consolidated regulation on EUR-Lex and the Digital Omnibus proposal COM(2025) 836 final. This post does not assert that any tool makes an entity DORA-compliant, because that is a supervisory determination rather than a product feature. Verified 24 August 2026. --- ## The Agent-to-Human Handoff: What You Owe On-Call URL: https://www.aurorasre.ai/blog/ai-agent-human-handoff Published: 2026-07-28 Author: Noah Casarotto-Dinning, CEO at Arvo AI > When an AI agent wakes a human at 3am, what should it hand over? Medicine solved structured handoff and measured the result. SRE has no equivalent yet. > ## Key Takeaways > > - **Medicine ran this experiment and published the numbers.** A [nine-hospital study of 10,740 admissions](https://pubmed.ncbi.nlm.nih.gov/25372088/) found that introducing a structured handoff bundle was associated with a 23% lower medical-error rate (24.5 to 18.8 per 100 admissions) and 30% fewer preventable adverse events (4.7 to 3.3 per 100 admissions). The intervention was a bundle: a mandatory format plus communication training, faculty development and a sustainability campaign. The format is the part that transfers to software. > - **The format is five fields, and it transfers cleanly to incidents.** I-PASS: Illness severity, Patient summary, Action list, Situation awareness and contingency plans, and **Synthesis by receiver**, where the receiver reads the plan back. > - **The published handoffs are shift summaries, not investigation handoffs.** PagerDuty ships an agent skill that formats an end-of-shift summary. Nobody we could find publishes a protocol for an agent handing an in-progress investigation to a human, with confidence, exclusions and a read-back. > - **The field an agent almost never gives you is what it ruled out.** Medicine has a name for this and treats it as diagnostic content, not filler. [OpenRCA 2.0](https://arxiv.org/abs/2606.27154) shows why it matters: agents name a correct service 76.0% of the time but ground it in a verified causal path only 61.5% of the time, so what was excluded, and on what evidence, is the difference between a lead and a conclusion. > - **A page has a hard budget and you should spend it on context, not volume.** Google's SRE book: "Every page should be actionable," "If a page merely merits a robotic response, it shouldn't be a page," and "I can only react with a sense of urgency a few times a day before I become fatigued." > - **Escalating is a feature, not a failure.** [Automation bias](https://journals.sagepub.com/doi/10.1177/0018720810376055) "cannot be prevented by training or instructions," so an agent that hands over a confident wrong answer damages more than one that admits it is stuck. > - **Default autonomy is often looser than the marketing.** Azure SRE Agent's overview says no change deploys without human sign-off, while its run-modes documentation shows incident response plans and scheduled tasks defaulting to Autonomous. **An agent-to-human handoff is the structured transfer of an in-progress investigation from an AI agent to the on-call engineer who now owns it: what is broken, what changed, what has been ruled out and on what evidence, how confident the agent is, and the first decision the human has to make.** No vendor we could find publishes such a protocol. Medicine does, and it measured the outcome across nine hospitals. This post borrows that structure, states clearly which parts are established practice and which are our synthesis, and ends with a handoff format you can implement against any agent. ## Why borrow from medicine? Because clinical handoff is the same problem with published outcome data, and software operations has no equivalent study. The [I-PASS study](https://pubmed.ncbi.nlm.nih.gov/25372088/) (Starmer et al., *New England Journal of Medicine*, November 2014) introduced a standardised handoff bundle across nine hospitals and measured 10,740 patient admissions before and after: > "The medical-error rate decreased by 23% from the preintervention period to the postintervention period (24.5 vs. 18.8 per 100 admissions, P<0.001), and the rate of preventable adverse events decreased by 30% (4.7 vs. 3.3 events per 100 admissions, P<0.001)." What changed was not staffing or seniority. It was a mandatory handoff structure, taught and reinforced: the study describes "a mnemonic to standardize oral and written handoffs, handoff and communication training, a faculty development and observation program, and a sustainability campaign." AHRQ's patient-safety primer notes the related baseline finding that "being cared for by a covering resident was a risk factor for preventable adverse events," and that communication failures between providers are "a leading cause of preventable error" in closed malpractice claims. The transfer to on-call is direct. A responder woken at 3am is a covering clinician: correct credentials, missing context, inheriting a situation mid-flight from someone, or something, that has been looking at it for a while. One honest caveat: **this is medicine, and we found no study transferring these protocols to software operations.** The analogy is strong and the outcome data is real, but nobody has measured a 23% reduction in anything by restructuring incident handoffs. We are reasoning from adjacent evidence and saying so. ## What is in I-PASS, and what is the SRE equivalent? The five fields, per [AHRQ](https://psnet.ahrq.gov/primer/handoffs-and-signouts), map onto an incident with almost no forcing. ![Diagram mapping the five fields of the clinical I-PASS handoff protocol onto their site reliability engineering equivalents, with a sixth row showing the read-back step that closes the loop.](/blog/agent-handoff-ipass-mapping.svg) | I-PASS field | Clinical meaning | Incident equivalent | |---|---|---| | **I**llness severity | How sick, and how fast changing | Severity, user impact, trend | | **P**atient summary | What is wrong and the story so far | Symptom, affected service, timeline | | **A**ction list | What to do next, by whom | The first decision and who owns it | | **S**ituation awareness and contingency plans | What might happen and the plan if it does | Blast radius, what to watch, the undo path | | **S**ynthesis by receiver | Receiver restates the plan | The responder confirms before the agent stands down | The fifth field is the one software will be tempted to skip, and it is the one doing the work. A read-back is what converts a broadcast into a transfer of ownership. An agent that posts a summary into Slack and considers the job done has not handed off; it has published. SBAR, the other widely used protocol, is the compressed version: Situation, Background, Assessment, Recommendation. It is a reasonable shape for a page body where I-PASS is the shape for the full incident channel. ## What do agents actually hand over today? Summaries, mostly. The pattern across the category is a natural-language recap plus links, with the agent's confidence and its excluded hypotheses either absent or buried in a reasoning trace nobody reads at 3am. The autonomy defaults are worth checking against the marketing, because they determine whether a handoff even happens. Azure SRE Agent documents two [run modes](https://learn.microsoft.com/en-us/azure/sre-agent/run-modes): Review, where the "Agent proposes an action; you approve or deny," and Autonomous, where the "Agent executes immediately and reports what it did." The overview material states that no change deploys without human sign-off, while the run-modes documentation shows incident response plans and scheduled tasks defaulting to Autonomous, even though the same page calls Review the agent-level default and the fallback when no per-plan mode is set. Those two statements sit uneasily together, and the resolution matters: in Autonomous mode the handoff is a report after the fact, not a decision point. Google's SRE organisation has published the most rigorous thinking here, and it is worth reading because they treat handoff as a property of the system rather than a UI concern. Their [autonomy ladder](https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/) runs L0 to L4, and the interesting mechanics are the transitions. Their agents downgrade themselves: "If an agent requests an L3 (High Automation) execution, but the agent detects an elevated risk score or an anomalous production state, it will automatically downgrade the request to L2 (Partial Automation), intercepting the execution and routing an approval request to a human SRE." That is an escalation trigger defined on system state rather than on model confidence, which sidesteps the calibration problem entirely. They also specify what the agent must record, which is effectively the handoff payload: agents "must log their 'chain of thought', the signals used, the hypotheses considered, the reasons for choosing a particular action, and the confidence level." And their [principles post](https://cloud.google.com/blog/products/devops-sre/how-google-sre-is-using-agentic-ai-to-improve-operations) adds the requirement that agents explain "what options were considered and rejected. In other words, we favor transparency over black-box automation." Rejected options. That is the pertinent negative, and Google is the only major vendor we found asking for it explicitly. ## When should an agent escalate instead of continuing? This is the harder half of the problem, and the honest answer is that you cannot base it on the agent's own confidence. Anthropic's research found that reasoning models frequently fail to verbalise what actually drove an answer: [one model mentioned an influential hint 25% of the time, another 39%](https://www.anthropic.com/research/reasoning-models-dont-say-think). If stated reasoning is unreliable, stated confidence inherits that unreliability. So "escalate when confidence is below 0.7" is a threshold on a number you cannot trust. Better triggers, in rough order of how well they hold up: 1. **System state, not self-assessment.** Google's model: elevated risk score or anomalous production state forces a downgrade regardless of how sure the agent is. 2. **Budget exhaustion.** Steps, wall-clock time, or spend. An agent at its ceiling has demonstrably not converged. Aurora hard-stops at 240 graph steps, roughly 120 reasoning turns, and on the opt-in multi-agent path at 600 seconds per sub-agent and two synthesis waves. 3. **A gated action.** Anything mutating. Aurora denies structured writes with a `BACKGROUND_DENIED` code when no human is present rather than queuing them, unless the org has explicitly allowlisted that tool, which forces the escalation rather than deferring it. 4. **Contradictory evidence.** Two hypotheses with comparable support and no discriminating test available is a genuine judgment call, which is what humans are for. 5. **Novelty.** No similar prior incident and no matching runbook. And the reason to escalate early rather than push through: [Parasuraman and Manzey's review](https://journals.sagepub.com/doi/10.1177/0018720810376055) in *Human Factors* found that automation bias produces "both omission and commission errors when decision aids are imperfect," that it "occurs in both naive and expert participants, cannot be prevented by training or instructions," and that automation complacency "cannot be overcome with simple practice." A handed-over wrong conclusion is not neutral. It anchors a tired engineer, and their seniority will not save them from it. So an agent that says "I could not determine this, here is what I eliminated" is more valuable than one that always produces an answer. This is why we treat it as a schema requirement rather than a prompt suggestion: Aurora rejects a sub-agent finding that lacks a "what I ruled out" section or a self-assessed strength from a fixed set, and a timed-out sub-agent returns an explicit inconclusive stub instead of silence. ## The page itself has a budget Whatever the agent knows, the page is the narrow aperture it has to fit through, and Google's SRE book is unusually direct about the constraint. From the [monitoring chapter](https://sre.google/sre-book/monitoring-distributed-systems/): "Every page should be actionable." And: "If a page merely merits a robotic response, it shouldn't be a page." And the human limit that most alerting configurations ignore: "I can only react with a sense of urgency a few times a day before I become fatigued." Two consequences for agent design. **An agent should page less and say more.** If it can page once with five fields instead of three times with fragments, it should. The budget is attention, not messages. **An agent that pages for something it could have resolved has spent a scarce resource for nothing.** That is the same standard Google applies to robotic pages, and it is why the useful metric is not "how many incidents did the agent touch" but "how many pages did it avoid, and how good were the ones it sent." Stripe's [auto-remediation writeup](https://stripe.dev/blog/how-stripe-uses-graph-search-and-state-machines-to-auto-remediate-a-global-database-fleet) (July 2026) is the benchmark for the avoidance half, and notably uses no LLM at all: state machines and invariants delivered "a 30% reduction in pager volume (~200 fewer pages a year)." Worth remembering before reaching for a model. ## A handoff format you can implement The mapping to I-PASS is established practice; **the specific fields below are our synthesis** of that structure plus the logging requirements Google publishes. Adapt freely. ``` INCIDENT HANDOFF ════════════════════════════════════════════════ SEVERITY SEV-2 · checkout latency p99 4.2s (SLO 800ms) Degrading: 1.1s → 4.2s over 18 min SUMMARY payments-api p99 latency breach. Errors isolated to POST /charge. Downstream: 3 services degraded. CHANGED deploy payments-api v2.14.0, 22 min before onset (strongest temporal correlation) 2 config changes in window, ruled out below RULED OUT ✗ Database saturation connections 34/200, no slow query log growth ✗ Upstream provider provider status green, egress latency flat ✗ Config change c-8821 applied post-onset, cannot be causal HYPOTHESIS Connection-pool exhaustion introduced in v2.14.0 CONFIDENCE: moderate NOT CONFIRMED: could not read pool metrics, exporter down since 03:12 BLAST RADIUS 3 services depend on payments-api checkout · order-history · reconciliation FIRST CALL Roll back v2.14.0, or fix forward? Undo path: kubectl rollout undo, ~90s Rollback destroys the pool state you'd need to confirm the mechanism. EVIDENCE 14 tool calls, full timeline: [link] Cost: $2.41 · Duration: 6m 12s CONFIRM Ack to take ownership. Agent stands down. ``` Five deliberate choices in that layout. **RULED OUT sits above the hypothesis.** It is the highest-value and most-omitted section, and putting it first signals that the agent did elimination rather than pattern-matching. Each exclusion carries the evidence that justified it, so you can disagree with any of them. **The hypothesis states what it could not check.** "Exporter down since 03:12" is worth more than a confidence score, because it tells the responder exactly which gap to close first. **FIRST CALL is a question, not a recommendation.** It names the tradeoff, including the one that is easy to miss under pressure: rolling back destroys the evidence needed to understand the failure. That is the decision, and it belongs to the human. **Cost is in the packet.** It tells you whether the agent worked efficiently and whether something looped. **CONFIRM is the read-back.** Without it, ownership is ambiguous and the fifth I-PASS field is missing, which is the field the clinical evidence suggests carries much of the benefit. ## The part nobody has solved Handing off a diagnosis is the easy half. Coordination is the hard half, and it is genuinely unsolved. Lorin Hochstein made the sharpest version of this argument in [February 2026](https://surfingcomplexity.blog/2026/02/14/lots-of-ai-sre-no-ai-incident-management/), surveying the AI SRE category: "what none of these tools try to do, as far as I can tell, is incident management." His reasoning is the part worth sitting with: > "Incidents involve some sort of problem with the system as a whole, and because everybody in the organization only has partial knowledge of the whole system, we typically need to pool that knowledge together to make sense of what's actually happening right now in the system." An agent can read every dashboard. It cannot know that the payments team shipped something unusual last Tuesday, that the provider had a similar issue nobody filed, or which staff engineer has seen this failure before. Google's SRE Workbook assigns that pooling to a human role, the [Incident Commander](https://sre.google/workbook/incident-response/), and nothing in the current tooling replaces it. So the honest framing of an agent's job at handoff is narrow: arrive with evidence gathered, alternatives eliminated, and uncertainty stated, so the humans can spend their attention on the part that needs judgment and shared context. That is a real contribution. It is not incident management. ## Where Aurora stands Aurora produces most of this packet today, and it is worth being specific about which parts are structural rather than aspirational. Findings are schema-validated and rejected unless they include an explicit "what I ruled out" section plus a self-assessed strength from a fixed enum, so the RULED OUT block cannot silently go missing. Every tool call in an incident investigation is persisted with input, output (truncated), status and duration, and reconstructed as a per-incident timeline, which is the EVIDENCE line. Cost per investigation is tracked and surfaced. Gated writes are denied outright when no human is present, which is the escalation trigger in item 3 above. And a sub-agent that times out returns an inconclusive stub rather than nothing. One scoping caveat: the findings schema, the sub-agent timeout stub and the synthesis-wave cap belong to the multi-agent orchestrator, which is opt-in and off by default today, while the 240-step limit and the write gate apply on every path. What Aurora does not do is enforce a read-back. There is no mechanism today that keeps an investigation open until a human explicitly acknowledges ownership, which by the clinical evidence is the field most associated with the benefit. That is a gap, we know it is a gap, and pretending otherwise would undercut the entire argument above. ## Try the handoff yourself Aurora is Apache 2.0, so the enforcement described above is something you can exercise rather than take on trust. Run it against your own incidents and read what the agent hands back: ```bash git clone https://github.com/arvo-ai/aurora.git && cd aurora make init # generates secrets and local config nano .env # add OPENROUTER_API_KEY, OPENAI_API_KEY or ANTHROPIC_API_KEY make prod-prebuilt # pulls prebuilt images and starts ``` Open **http://localhost:3000** and register; the first user becomes admin. Trigger an investigation and check the finding for the four things this post argues a handoff owes you: a summary, the evidence, the reasoning, and what it ruled out. A finding missing any of them does not get persisted. The exclusions field is the one an agent skips unless something forces it, which is why it is validated rather than merely requested in a prompt. One scoping caveat from the section above: that validation belongs to the multi-agent orchestrator, which is opt-in and off by default. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Related guides:** [SLOs for AI Agents](/blog/slos-for-ai-agents) · [AI SRE Accuracy: What the Benchmarks Show](/blog/ai-sre-accuracy-benchmarks) · [The SRE Incident Lifecycle](/blog/sre-incident-lifecycle-with-ai-agents) · [AI Agent Guardrails](/blog/ai-agent-guardrails) > **Sourcing note.** The I-PASS outcome figures come from the New England Journal of Medicine study via PubMed; the field definitions from AHRQ's patient-safety primer; paging guidance from Google's SRE book; autonomy mechanics from Google SRE and Microsoft Learn; automation bias from the peer-reviewed review in *Human Factors*. The handoff format is Arvo's synthesis of that material and is labelled as such. Verified July 28, 2026. --- ## What Does an AI Investigation Actually Cost? (2026) URL: https://www.aurorasre.ai/blog/ai-investigation-cost Published: 2026-07-28 Author: Noah Casarotto-Dinning, CEO at Arvo AI > AWS bills agent-seconds, Azure bills agent units with an always-on floor, Datadog bills credits. A sourced AI investigation cost comparison. > ## Key Takeaways > > - **Every vendor bills in a different unit, which is why nobody can compare them.** AWS DevOps Agent charges ["$0.0083 per agent-second"](https://aws.amazon.com/devops-agent/pricing/). Azure SRE Agent charges Azure Agent Units with a mandatory always-on floor. Datadog charges non-rolling AI Credits. [Cleric](https://cleric.ai/pricing) is the only vendor publishing a straight per-investigation price: **$20 per issue**. > - **The derivable cost of one investigation ranges from about $1.17 to about $20** across vendors that publish enough to calculate it. Rootly, incident.io, PagerDuty and Google publish nothing you can derive a per-investigation figure from. > - **Azure's own two pages disagree by 4.2x on how an investigation is metered.** The [product page](https://azure.microsoft.com/en-us/products/sre-agent/) says "0.25 Azure Agent Units per second" while the [billing docs](https://learn.microsoft.com/en-us/azure/sre-agent/pricing-billing) meter active flow per token. A 10-minute investigation is $15.00 under one reading and $3.53 under the other. > - **The biggest hidden cost is not the agent, it is the queries the agent runs.** AWS states you also pay for "Amazon CloudWatch Logs Insights queries or trace retrievals." At AWS's published $0.005 per GB scanned, ten queries against a 500 GB log group is **$25**, against a $3.98 agent fee. > - **Azure bills you for an agent that is doing nothing.** Always-on is "4 AAUs per agent-hour" and continues "from agent creation until the agent is deleted," so a stopped agent still bills roughly **$292/month** at $0.10/AAU before a single investigation. AWS, by contrast: "There are no upfront commitments or charges for when the agent is idle." > - **More spending does not buy more accuracy.** Two independent 2026 evaluations found the same thing. SREGym: "the marginal token does not buy additional success." Artificial Analysis: "longer trajectories do not translate to higher accuracy." > - **Retire the $5,600-per-minute downtime figure.** It traces to a 2014 Gartner blog post that no longer resolves. Use the [2026 PagerDuty/Wakefield distribution](https://www.pagerduty.com/newsroom/2026-state-of-ai-first-operations/) instead. **An AI investigation costs between roughly $1 and $20 in vendor charges as of July 2026, but the observability queries the agent runs during that investigation can cost several times more than the agent itself.** Almost no published comparison exists, because each vendor bills in an incompatible unit and two of the largest do not publish a per-investigation figure at all. This post derives what is derivable, shows the arithmetic, and names what cannot be derived. Every price below was checked on July 28, 2026 against the live vendor page or, where a vendor renders prices client-side, its official pricing API. Where a vendor does not publish, we say so rather than estimating. ## Why can't you just compare AI SRE pricing? Because the billing units are not commensurable. This table is the actual problem. | Vendor | Billing unit | Published rate | Idle charge? | |---|---|---|---| | [AWS DevOps Agent](https://aws.amazon.com/devops-agent/pricing/) | Wall-clock agent-second | "$0.0083 per agent-second" | No | | [Azure SRE Agent](https://learn.microsoft.com/en-us/azure/sre-agent/pricing-billing) | Azure Agent Unit (AAU) | "4 AAUs per agent-hour" always-on + active flow | **Yes** | | [Datadog Bits Investigation](https://www.datadoghq.com/pricing/?product=ai-agents-console) | AI Credit | "$500 per 500 credits/month (billed annually) or $1.30 per credit on-demand" | No | | [Cleric](https://cleric.ai/pricing) | Investigated issue | "1 Issue 10 credits $20" | Platform floor | | [Rootly](https://rootly.com/pricing) | Seat | AI SRE: "Contact us" | n/a | | [incident.io](https://incident.io/pricing) | Seat | AI bundled into Pro "$25/user/month" | n/a | | [PagerDuty](https://www.pagerduty.com/pricing/incident-management/) | AI Action | "1,000 one-time AI Actions" on Professional | n/a | | [Gemini Cloud Assist](https://docs.cloud.google.com/gemini/docs/cloud-assist/investigations) | Licence | Free during preview, Premium Support gated | n/a | | Aurora (self-hosted) | Your own LLM tokens | Your provider's list price | No | Wall-clock seconds, tokens, credits, issues, and seats are five different things. A vendor billing agent-seconds is incentivised to answer fast; a vendor billing tokens is not. That is worth knowing before you sign. ## What does one investigation cost, vendor by vendor? Here is every figure that can be honestly derived, with the assumption stated. Where the multiplication is ours rather than the vendor's, we say so. ![Cost per AI investigation compared across vendors, showing AWS DevOps Agent at about $3.98 for an eight-minute investigation, Azure SRE Agent at $3.53 per-token or $15.00 per-second, Datadog Bits Investigation at $6.50 committed, and Cleric at a published $20 per issue, alongside Azure's $292 per month always-on floor.](/blog/ai-investigation-cost-comparison.svg) **AWS DevOps Agent: about $3.98.** AWS publishes the rate and an eight-minute example but never the per-investigation price. The arithmetic: 8 min x 60 s x $0.0083 = **$3.98**. Cross-check against their own published $39.84 for ten investigations, which divides to $3.984. Customers on paid Support plans also receive monthly DevOps Agent credits worth a percentage of the prior month's AWS Support charge, "100% for Unified Operations", 75% for Enterprise and 30% for Business Support+, and those credits expire at the end of each month. **Azure SRE Agent: $3.53 or $15.00, depending which Microsoft page you believe.** The docs give a worked example of 35.3 AAU for an incident investigation on Claude Opus 4.6. At $0.10/AAU that is **$3.53**. On GPT 5.3 Codex the same example is 11.7 AAU, or **$1.17**. But the product page states each task is billed "at an additional 0.25 Azure Agent Units per second," which for a 10-minute investigation gives 600 x 0.25 x $0.10 = **$15.00**. Those are incompatible billing models from the same vendor, and the gap is 4.2x. The USD figure needs a caveat: Azure's [public pricing page](https://azure.microsoft.com/en-us/pricing/details/sre-agent/) renders literal `$-` placeholders, so the price is only obtainable from the Retail Prices API. The reproducible query: ``` GET https://prices.azure.com/api/retail/prices?api-version=2023-01-01-preview &$filter=contains(meterName,'Agent Unit') ``` That returns 22 regional meters named `SRE Agent Unit` under `serviceName: "Foundry Tools"`, at $0.10/AAU in five US regions, $0.11 in thirteen others, $0.118 in uaecentral, and $0.12 in westeurope, southeastasia and brazilsouth. **Datadog Bits Investigation: $6.50 committed, $8.45 on-demand.** Datadog publishes an average of "~6.5 credits" for an autonomous investigation. Committed credits work out to $1.00 each, so 6.5 x $1.00 = **$6.50**; at the on-demand rate of $1.30 it is **$8.45**. Datadog explicitly disclaims the average: "Actual consumption for any single request may vary with task complexity and the amount of context the model processes." **Cleric: $20, published directly.** No derivation needed, and credit where it is due, because they are the only vendor in the category who states it: "1 Issue 10 credits $20," where "An Issue is a defined operational problem Cleric takes end to end." Note the platform floor of $2,000/month billed annually. **Not derivable, and you should ask why.** Rootly's AI SRE is "Contact us." incident.io bundles AI into a Pro seat with no per-investigation unit. PagerDuty grants "1,000 one-time AI Actions" with no published dollar value per Action and no published refill price. Gemini Cloud Assist Investigations is free during preview but gated to Premium Support customers. ## What does it cost if you bring your own LLM key? This is Aurora's model, so treat the following as an interested party showing its arithmetic rather than a neutral finding. Published token counts for agent investigations come from [SREGym](https://arxiv.org/abs/2605.07161), which measures tokens per run. Stratus on Sonnet-4.6 without injected noise used about 812K tokens; Claude Code used 1.47M; Codex used about 1.98M. SREGym publishes **tokens only, and no dollar figures**, so any currency number here is our calculation, not theirs. It also publishes only totals, with no input/output split, so a point estimate would be false precision. Bounding it against [Anthropic's list prices](https://platform.claude.com/docs/en/about-claude/pricing) for Sonnet-4.6 at $3/MTok input and $15/MTok output, 812K tokens is **$2.44 if every token were input and $12.18 if every token were output**. Real agent traffic is heavily input-weighted, so the realistic range is roughly **$2.50 to $3.50**. That method reproduces a published figure when tested against one, which is the only reason to trust it. [Coroot published](https://coroot.com/blog/hard-part-of-ai-root-cause-analysis-is-no-longer-the-model/) a split of 9,800 input and 1,000 output tokens on Opus 4.8 at $5/$25, and states a real per-incident price of $0.0743. The same arithmetic gives 0.0098 x 5 + 0.001 x 25 = **$0.074**. Two caveats on that comparison, though, and they matter: Coroot's number covers a single scenario with, in their words, "No tools, no agent loop," over pre-correlated findings, so it is not measuring the same work as a multi-step agent investigation. And [Coroot lists](https://coroot.com/pricing/) AI-powered root cause analysis under its paid tier at "$1 per monitored CPU core/month" rather than among the free Community edition's core features. Current list prices, for your own modelling: | Provider | Model | Input /MTok | Output /MTok | |---|---|---|---| | [Anthropic](https://platform.claude.com/docs/en/about-claude/pricing) | Claude Opus 5 / 4.8 / 4.7 / 4.6 | $5 | $25 | | Anthropic | Claude Sonnet 4.6 | $3 | $15 | | Anthropic | Claude Haiku 4.5 | $1 | $5 | | [OpenAI](https://developers.openai.com/api/docs/pricing) | gpt-5.6-sol | $5.00 | $30.00 | | OpenAI | gpt-5.6-terra | $2.50 | $15.00 | | OpenAI | gpt-5.4-mini | $0.75 | $4.50 | | [Google](https://ai.google.dev/gemini-api/docs/pricing) | Gemini 3.1 Pro Preview | $2.00 (<=200k) | $12.00 | | Google | Gemini 3.6 Flash | $1.50 | $7.50 | Two timing details that change the maths in 2026: Claude Sonnet 5's introductory pricing expires August 31, 2026, and Anthropic models from 4.7 onward use a tokenizer that produces roughly 30% more tokens for the same text, which changes effective cost independently of the sticker rate. Model routing is the cheapest lever here and it is worth stating that Aurora defaults to it: RCA runs on Haiku unless you turn cost optimisation off, guardrail checks run on a cheaper model still, and the stable system prompt carries an Anthropic cache breakpoint so later turns bill at cache-read rates. Aurora also persists tokens, latency and cost per LLM call and surfaces total cost per investigation, which is the number this whole post is about. ## The hidden bill: what the agent's queries cost This is the finding that should change how you budget, and almost nobody writes about it. AWS says it plainly on its own pricing page: "You are responsible for fees incurred for other AWS services... for example, Amazon CloudWatch Logs Insights queries or trace retrievals." AWS's published price list puts Logs Insights at **$0.005 per GB of log data scanned**. Work that through. Scanning a 500 GB log group costs $2.50 per query. An investigation that runs ten such queries costs **$25 in scanning**, against a **$3.98** agent fee. The observability bill is 6x the agent bill, and it is billed by a different team's budget. The corollary: cost scales with **how many times the agent looks**, not with what the vendor charges per second. An agent that runs one well-targeted query is dramatically cheaper than one that greps everything twice, even if the second agent finishes faster. Other costs buyers miss, each from the vendor's own terms: - **Non-rolling credits.** Datadog: "AI Credits reset on the first day of each calendar month. Unused credits do not roll over to the next month," and overages "billed automatically at the On-Demand rate," a 30% premium. Cleric is more generous: credits "roll over for one month, up to 2x your monthly amount." - **Rehydration.** Datadog charges "$0.10 per compressed GB of log data that is scanned" to rehydrate archived logs. An agent investigating anything older than your index window triggers this. - **Always-on floors.** Azure's always-on charge "continues from agent creation until the agent is deleted," and a stopped agent "still incurs the fixed always-on cost." - **Annual-commitment penalties**, category-wide: roughly 16% at PagerDuty ($25 monthly versus $21 annual) and 30% on Datadog AI credits ($1.30 on-demand versus $1.00 committed). - **Tail risk.** Ask for a p99 rather than a mean. Published eval data in this category shows per-test costs where the worst single run is more than an order of magnitude above the average, so an average tells you very little about the run that scans everything. - **Runaway loops.** incident.io's engineering team wrote that Investigations are "much more (100x) as expensive when compared to our existing AI features, with greater potential for runaway costs," and their controls are per-project billing limits and daily spend reports. ## Does spending more get you a better answer? No, and two independent 2026 evaluations reached that conclusion separately, which makes it the most robust finding in this post. [SREGym](https://arxiv.org/abs/2605.07161) concluded that "the marginal token does not buy additional success." [Artificial Analysis](https://artificialanalysis.ai/articles/itbench-aa-launch), running IBM's ITBench task set, found that "longer trajectories do not translate to higher accuracy," and their per-task cost data inverts the ranking outright: Gemma 4 31B scored about 37% at **$0.14 per task** while Gemini 3.1 Pro Preview scored about 30% at **$2.23 per task**, and Claude Opus 4.7 led at about 47% but cost **$5.38 per task**. Those figures are Artificial Analysis's, published in their May 27, 2026 launch article, not IBM's; IBM publishes no cost column. Artificial Analysis also observed *why* the expensive runs fail: "Models that over-investigate tend to surface upstream fault-injection mechanisms or co-occurring symptoms as false positives." Over-investigation costs money and buys wrong answers. We cover what the accuracy numbers actually support in [AI SRE accuracy: what the benchmarks show](/blog/ai-sre-accuracy-benchmarks). ## What is the real cost of downtime in 2026? You need this number to justify any of the above, and the one everybody quotes is unusable. **Retire "$5,600 per minute."** It traces to a 2014 Gartner blog post that no longer resolves, was never accompanied by a published methodology, and now circulates only through vendor pages citing each other. The same applies to the "$9,000 per minute" variant, which is really Ponemon's 2016 mean of $8,851 from 63 data centres, measuring facility outages rather than software incidents. **Use this instead**, from [PagerDuty's 2026 research](https://www.pagerduty.com/newsroom/2026-state-of-ai-first-operations/) fielded by Wakefield Research, N=1,000 director-level-and-above decision makers: > "Some organizations (8%) lose more than $1 million per hour, 34% lose at least $500,000 per hour, and more than two thirds (68%) lose more than $300,000 per hour during IT incidents." It is per-hour, which is the only unit that converts against investigation time; it is ungated at the primary source with the methodology published; and it is a distribution rather than a single false-precision average. Disclose the obvious limitation when you cite it: it is vendor-commissioned and self-reported. For triangulation from a different population and method, [Splunk and Oxford Economics](https://newsroom.cisco.com/c/r/newsroom/en/us/a/y2026/m05/the-600-billion-wake-up-call-new-splunk-research-reveals-downtime-is-a-systemic-business-crisis.html) (May 19, 2026, N=2,000 Global 2000 executives, hybrid CATI and online across 20 countries) put it at "$95 million in lost revenue annually" per company. One number to avoid even though it is tempting: PagerDuty's own "$793,957 per incident" figure is undated, its source whitepaper publishes no methodology, and the arithmetic does not check out (4,537 x 175 = 793,975). ## What does self-hosting actually cost? If you run an open-source agent yourself, the honest split is that the stack is cheap and the model is not. | Component (AWS rates are us-east-1 on-demand) | Published price | |---|---| | [EKS](https://aws.amazon.com/eks/pricing/) or [GKE](https://cloud.google.com/kubernetes-engine/pricing) control plane | "$0.10 per cluster per hour" (~$73/mo) | | Compute node, m7i.xlarge (4 vCPU / 16 GiB) | $0.2016/hr (~$147/mo) | | [RDS Postgres](https://aws.amazon.com/rds/postgresql/pricing/) db.m7g.large Single-AZ | $0.168/hr (~$123/mo) | | [ElastiCache](https://aws.amazon.com/elasticache/pricing/) m7g.large running Valkey | "$0.1264 per Gen Purpose m7g.large node hour" (~$92/mo) | | [S3](https://aws.amazon.com/s3/pricing/) Standard, first 50 TB | $0.023 per GB-month | | GPU for open-weight serving, p5.48xlarge (8xH100) | $55.04/hr (~$40,179/mo) | The non-GPU stack lands **under $500/month**, line by line. Serving your own frontier-class model on demand is roughly 80x that. This is the honest argument for bring-your-own-key rather than self-hosted inference, and it is why Aurora supports both cloud providers and Ollama for air-gapped deployments but does not pretend local inference is the cheap option. A pricing-page footnote worth knowing: in ElastiCache, Valkey is priced 20% below Redis OSS, $0.1264 versus $0.158 per m7g.large node hour, which is open source showing up directly on an invoice. ## The buyer's checklist Five questions that get you a real number instead of a rate card: 1. **What is the unit, and does it bill when idle?** Azure's always-on floor is roughly $292/month per agent before any work. AWS charges nothing idle. At ten investigations a month, Azure's fixed cost is 8x its variable cost. 2. **What do the agent's queries cost, and whose budget pays?** Get the vendor to state whether observability query and rehydration fees are included. At AWS's rates the query bill can exceed the agent bill several times over. 3. **Do credits expire, and what is the overage rate?** Non-rolling credits plus a 30% on-demand premium is a materially different price than the headline. 4. **What is the worst single investigation, not the average?** Ask for a p99, not a mean. Vendors publish averages; the tail is where the budget breaks. 5. **What caps exist?** Ask specifically whether there is a hard per-investigation or per-month spend ceiling, and who gets paged when it is hit. Aurora is honest about its own limit here: it tracks and reports cost per investigation, and exposes hooks a host can use to enforce limits, but it does not ship a built-in hard budget cap. ## Measure your own number instead of trusting ours The only per-investigation cost figure worth planning against is one from your own incidents, your own model choice and your own alert volume. With bring-your-own-key, the LLM spend lands on your provider bill where you can read it directly: ```bash git clone https://github.com/arvo-ai/aurora.git && cd aurora make init # generates secrets, copies .env.example to .env nano .env # add OPENROUTER_API_KEY, OPENAI_API_KEY or ANTHROPIC_API_KEY make prod-prebuilt # pulls prebuilt images and starts ``` Open **http://localhost:3000** and register; the first user becomes admin. Aurora persists per-call token, latency and cost and rolls it up per investigation, so after a dozen real alerts you have a distribution rather than a vendor average, including the p99 that item 4 above tells you to ask for. Three settings in `.env.example` move the number materially: ```bash # Background RCA on claude-haiku-4.5 instead of claude-opus-4.6 RCA_OPTIMIZE_COSTS=true # Off by default: skips the periodic and final LLM extraction for the infra graph VISUALIZATION_ENABLED=false # Caps auto-loaded integration skill context during background RCA (defaults to 12000) RCA_SKILLS_TOKEN_BUDGET=12000 ``` Two honest caveats on the self-hosted path. There is no built-in hard budget cap, per item 5 above. And the stack itself has a floor: Postgres, Memgraph, Weaviate and Redis are real infrastructure, which is the sub-$500/month line-item table earlier in this post. The hosted tier avoids that floor if you only want a per-investigation cost baseline. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Related guides:** [AI SRE Accuracy: What the Benchmarks Show](/blog/ai-sre-accuracy-benchmarks) · [How to Evaluate an AI SRE Platform](/blog/how-to-evaluate-ai-sre-platform) · [Self-Hosted AI SRE](/blog/self-hosted-ai-sre) · [Azure SRE Agent Alternative](/blog/azure-sre-agent-alternative-open-source) > **All vendor prices verified July 28, 2026** against the live vendor pricing page, the Azure Retail Prices API for Azure Agent Units, or the AWS Price List API for AWS rates that the pricing pages render client-side. Derived per-investigation figures show their arithmetic and state their assumption. Where a vendor does not publish a price, this post says so instead of estimating. Prices change; re-check before quoting. --- ## AI SRE Accuracy: What the Benchmarks Show (2026) URL: https://www.aurorasre.ai/blog/ai-sre-accuracy-benchmarks Published: 2026-07-28 (updated 2026-08-27) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Vendors claim 90%+ accuracy. The only independent AI SRE leaderboard tops out at 56.2%. What the benchmarks measure and how to test a tool yourself. > ## Key Takeaways > > - **No AI SRE vendor has submitted its own product to an independent third-party benchmark.** Two projects publish scores, but for foundation models ([Rootly's SRE-skills-bench](https://github.com/Rootly-AI-Labs/sre-skills-bench)) or for their own self-curated eval set ([HolmesGPT](https://holmesgpt.dev/latest/development/evaluations/)), not for the commercial agent as sold. > - **The best model on the only independent AI SRE leaderboard scores 56.2%.** On [ITBench-AA](https://artificialanalysis.ai/evaluations/itbench-aa), run by Artificial Analysis on IBM's task set, GPT-5.6 Sol (max) leads at 56.2% and nothing has yet cleared 60%. The metric is strict: "average precision at full recall." > - **The most useful number in this field is the gap between naming and proving.** [OpenRCA 2.0](https://arxiv.org/abs/2606.27154) (June 2026, 500 instances, 11 frontier LLMs) found agents "identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%." Exact root-cause recovery: 20.7%. > - **A leaderboard win does not transfer to your stack.** An [audit of three RCA benchmark families](https://arxiv.org/abs/2606.29159) across 11 subsystems found that picking the pooled winner selects the lower-scoring method on up to 5 of 11 held-out subsystems, "with regret reaching 24.8 pp." > - **Waiting for a better model will not fix it.** Across [1,675 agent runs on five models](https://arxiv.org/abs/2602.09937), the dominant failures "persist across all models regardless of capability tier, indicating that these failures originate from the shared agent architecture rather than from individual model limitations." > - **A confidently wrong answer is the expensive failure mode**, because [automation bias](https://journals.sagepub.com/doi/10.1177/0018720810376055) "cannot be prevented by training or instructions" and affects experts as much as novices. > - **Aurora has not published a benchmark score either.** We say so plainly below, along with what we would have to build first. **AI SRE accuracy is the rate at which an agent identifies the actual cause of an incident, and as of July 2026 the best independently measured result on a public incident benchmark is 56.2%.** Vendor marketing in the same category advertises 92% and, at [Komodor](https://komodor.com/ai-sre-2026/), "up to 95% accuracy in incident detection and resolution". Both numbers can be technically defensible, because nobody agrees on what counts as correct. This post lays out what the published benchmarks actually measure, why the vendor numbers are not comparable, and how to test a tool on your own incidents. We sell an AI SRE agent. Read the section on our own gap before you decide how much to trust the rest. ## What is the highest independently verified AI SRE accuracy? 56.2%, on [ITBench-AA](https://artificialanalysis.ai/evaluations/itbench-aa). That evaluation matters because it is the only leaderboard in this category where an independent party grades the work. [Artificial Analysis](https://artificialanalysis.ai/) runs the harness; the task set comes from IBM Research's ITBench team. The setup, in their words: "We evaluate 59 Kubernetes incident tasks (40 from IBM's public release and 19 private shared with us by the ITBench team), with 3 repeats per task." The metric is unusually honest, and it is the reason the numbers look low: > "The headline score is the average precision at full recall. For each task, the model scores TP / (TP + FP) when there are zero false negatives, and 0 otherwise." Miss a single ground-truth root cause and the run scores zero. Name extra causes that are not real and your precision drops. This is the only benchmark in the category that punishes over-reporting, which is precisely the behavior that destroys on-call trust. ![Bar chart comparing published AI SRE accuracy figures. Independent benchmarks and peer-reviewed research cluster between 20.7% and 61.5%, while vendor marketing claims cluster between 92% and 95% with no disclosed methodology.](/blog/ai-sre-accuracy-benchmark-gap.svg) Here is the full published record, with what each number actually measures. | Source | Result | What it measures | Independent? | |---|---|---|---| | [ITBench-AA](https://artificialanalysis.ai/evaluations/itbench-aa) | **56.2%** top model | Precision at full recall, 59 K8s incident tasks, 3 repeats | Yes, third-party harness | | [OpenRCA 2.0](https://arxiv.org/abs/2606.27154) | **20.7%** exact, 61.5% grounded | Exact root-cause set, and whether a causal path is verified | Yes, independent academic preprint | | [ITBench](https://arxiv.org/abs/2502.05352) | **13.8%** of SRE scenarios | End-to-end scenario resolution, 94 scenarios total | Yes, IBM + UIUC | | [Meta production](https://engineering.fb.com/2024/06/24/data-infrastructure/leveraging-ai-for-efficient-incident-response/) | **42%** top-5 | Root cause in top five suggested code changes, one monorepo | First-party, method disclosed | | [Microsoft RCACopilot](https://arxiv.org/abs/2305.15778) | **up to 0.766** | Root-cause *category* prediction, a year of real incidents | Peer-reviewed | | [Rootly SRE-skills-bench](https://rootly.com/blog/claude-sonnet-4-6-benchmark-results-and-lessons-for-ai-sre) | **90.4%** (Sonnet-4.6) | Multiple-choice SRE knowledge, not live incidents | Vendor-run, open data | | Vendor marketing | **92% to 95%** | Undisclosed | No | ## Why do vendors claim 90%+ when benchmarks show 20% to 56%? Three reasons, and only one of them is dishonest. **First, the criterion moves.** OpenRCA 2.0 demonstrates this mechanically. Ask whether the agent named a service involved in the incident and you get 76.0%. Ask whether it grounded that service in "a verified causal propagation path to the observed symptom" and you get 61.5%. Ask for the exact root-cause set and you get 20.7%. Same runs, same agents, three very different headlines. The authors named the failure mode **ungrounded diagnosis**, and their conclusion is the sentence every buyer should read: "Outcome-only evaluation hides this failure mode." **Second, some benchmarks measure knowledge rather than diagnosis.** Rootly's 90.4% is a real, reproducible number on an [Apache 2.0 benchmark](https://github.com/Rootly-AI-Labs/sre-skills-bench), and it deserves credit for existing at all. But read what it tests: their own framing is that it covers "understanding infrastructure code, reasoning about cloud configurations, and mapping code diffs to real-world pull requests." Their **Incident Response** track, the one that would replay real postmortems as live scenarios and grade an agent across detect, localize, diagnose, mitigate and verify, is marked *Planned*. So 90.4% and 20.7% are not a contradiction. They measure different things, and only one of them is the product. Rootly's own README says there is "no single blended number," which is the correct design choice. **Third, and this is the real problem: most vendor numbers have no denominator.** We checked five live vendor pages. Not one publishes the denominator behind its headline figure, a definition of correctness, or a footnote. Cleric alone publishes a volume figure, "200,000+ Production-Grade Investigations", but not the sample the 92% is drawn from. [Traversal](https://traversal.com/) advertises "82% Root Cause Analysis (RCA) accuracy," attributed to a Fortune 100 financial services customer. [Cleric](https://cleric.ai/) advertises "92% Actionable Findings," and note that *actionable* is not *correct*. [NeuBird](https://neubird.ai/) advertises "92% MTTR reduction." [Resolve.ai](https://resolve.ai/) advertises "up to 5x faster MTTR." [incident.io](https://incident.io/ai) advertises "Resolve incidents 5x faster." To be fair to them: these are not necessarily false. They are unfalsifiable, which is different. When a vendor never defines "accuracy" or "actionable," there is no claim to disprove. That is itself the finding. And one vendor does better than the rest: Traversal's [benchmarking writeup](https://traversal.com/blog/llm-benchmarking-in-context-retrieval-reasoning-incident-root-cause-analysis) describes its grading protocol, including that "Traversal SRE researchers and customer SREs manually validated a randomly sampled subset of judgments." They publish no numbers from it, but the methodological candour is real and worth acknowledging. ## Will the next frontier model fix this? Probably not on its own, and there is now direct evidence. Researchers ran [1,675 agent runs across five LLMs](https://arxiv.org/abs/2602.09937) on OpenRCA and classified the failures into 12 pitfall types. The finding that matters for anyone planning to wait out the problem: > "the most prevalent pitfalls, notably hallucinated data interpretation and incomplete exploration, persist across all models regardless of capability tier, indicating that these failures originate from the shared agent architecture rather than from individual model limitations." They also tested the obvious workaround and found it insufficient: "prompt engineering alone cannot resolve the dominant pitfalls," while enriching the inter-agent communication protocol cut communication failures by up to 15 percentage points. The architecture is the lever, not the model. ## Will connecting more telemetry fix it? This is the standard pitch in the category, and a [July 2026 paper](https://arxiv.org/abs/2607.13548) tested it directly. The authors built a reverse-reasoning agent to separate failures where evidence was present but unused (a *Reasoning Gap*) from failures where the evidence genuinely was not there (*Data Ambiguity*). Their conclusion: > "the required evidence is present in the vast majority of failures: the bottleneck is not data access but the agent's ability to reason over it correctly." If you are being sold accuracy on the basis of more integrations, that is the paper to bring to the call. ## Does a top leaderboard score mean the tool will work on your stack? No, and this is the most practically useful result in the 2026 literature. An [audit of pooled leaderboard reporting](https://arxiv.org/abs/2606.29159) covered OpenRCA, RCAEval and PetShop across 11 subsystems and 778 matched scoring units. Leave-one-system-out selection, which simulates a buyer picking the leaderboard winner for their own unseen environment, "picks the lower-scoring method on up to 5 of 11 held-out subsystems, with regret reaching 24.8 pp on RCAEval / Sock-Shop." Every pairwise comparison showed subsystem-level effects in both directions. Translated: choosing the top-ranked method can hand you the worse one for your estate, and be wrong by nearly 25 percentage points. The audit compares statistical RCA methods rather than commercial agents, but the selection failure mode is structural. Benchmarks tell you the category ceiling. They do not tell you what will work on your services. That is what your own test is for. ## How should you test an AI SRE tool on your own incidents? There is no vendor-neutral published methodology for this. We looked. So here is one assembled entirely from primary sources, with each step traced to who established it. 1. **Build a holdout set from resolved incidents, stratified.** Google's SRE organization describes using "stratified sampling to continuously surface a diverse subset of incidents for manual review, creating Gold data," where Gold means "Verified by human experts." Stratify across service, severity and failure class, or you will score the tool on forty variations of one disk-full alert. 2. **Freeze information at investigation start.** Meta's protocol is the citable precedent: accuracy measured "with historical investigations and the information available at their start." Skip this and you leak the answer, because the postmortem is sitting in the ticket. 3. **Grade the causal path, not just the named service.** Score every output twice: did it name the right thing, and did it show a chain of evidence a reviewer accepts. The delta between those two numbers is your real trust metric, and OpenRCA 2.0 shows it is roughly 15 points wide even for frontier models. 4. **Require reasoning logs as an evaluation input.** Google's stated requirement is that agents "must log their 'chain of thought', the signals used, the hypotheses considered, the reasons for choosing a particular action, and the confidence level." Without this you cannot perform step 3. 5. **Score per service, never pooled.** Per the leaderboard audit above, one blended number for your whole estate will mislead you exactly the way a public leaderboard does. 6. **Count confident-wrong separately from inconclusive.** Three buckets: correct, inconclusive, confidently wrong. Weight the third heavily. An agent that abstains costs you nothing; an agent that is assertively wrong costs you the incident. 7. **Run at Google's L1 before granting more.** In [Google's graded autonomy ladder](https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/), L1 means the agent provides "insights or suggestions" while a human must "Approve any action, manually Actuate it." Promote only after the tool clears your bar on the holdout set. 8. **Re-run on a schedule, with cost attached.** HolmesGPT's [dated eval history](https://holmesgpt.dev/latest/development/evaluations/history/) is the model to copy here: its September 28, 2025 full benchmark across 105 test cases published accuracy *and* dollar cost per model side by side, from sonnet-4 at 94% and \$0.16 per case down to gpt-4o at 62% and \$0.14 per case. Accuracy without cost per investigation is not a decision input. ## Why is a wrong answer worse than no answer? Because your responders are documented to be unable to resist it, and that finding is 16 years old. Parasuraman and Manzey's [review of automation bias and complacency](https://journals.sagepub.com/doi/10.1177/0018720810376055) in *Human Factors* (2010) is the canonical citation. Their findings, verbatim: automation bias "results in making both omission and commission errors when decision aids are imperfect," it "occurs in both naive and expert participants, cannot be prevented by training or instructions, and can affect decision making in individuals as well as in teams." Automation complacency, separately, "cannot be overcome with simple practice." Seniority does not protect your on-call engineer from a confident wrong root cause at 3am. Training does not either. This is why the roughly 15% of cases where an agent names a correct service it cannot justify are not a rounding error, and why an agent that can return *inconclusive* is worth more than one that always answers. We treat that as an architectural requirement rather than a prompt instruction: Aurora's sub-agent findings are schema-validated and rejected unless they include an explicit "what I ruled out" section and a self-assessed confidence drawn from a fixed set, and a sub-agent that times out returns an honest inconclusive stub rather than silence. One caveat we should state rather than paper over: this research comes from aviation and process control, not from SRE. We found no SRE-specific empirical study of automation bias during incident response. The transfer is reasonable but not established. Even Meta, publishing a genuinely useful 42%, was explicit about the risk: their system "can potentially suggest wrong root causes and mislead engineers," which is why they "prioritize closed feedback loops and explainability." ## What autonomy level is defensible today? Google's [graded autonomy framework](https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/) is the most credible published answer, partly because Google ships an agent that gets things wrong and documents it. The ladder runs L0 (everything human) through L4 (the agent devises and executes multi-step resolutions). L2 means a human "must explicitly Approve any plan before the system proceeds with an actuation." L3 means the agent "can independently detect, decide, and act without human approval for bounded actions." What makes it credible is the machinery around L3, not the definition of it: - **Automatic de-escalation.** "If an agent requests an L3 (High Automation) execution, but the agent detects an elevated risk score or an anomalous production state, it will automatically downgrade the request to L2 (Partial Automation), intercepting the execution and routing an approval request to a human SRE." - **A global stop.** "emergency 'Red Button' endpoints that allow SREs to instantly pause all in-flight agentic actions, block new actions, or globally revoke L3 permissions across the fleet during catastrophic, complex outages." - **A published failure.** In their own evaluation figure, "the agent incorrectly diagnosed the root cause and failed to mitigate it," after which an LLM-as-judge "automatically generated a critique of the agent's logic and filed a bug." An organization that publishes its agent's failures and designs containment for them is further along than one advertising up to 95% accuracy. For the layered version of this argument, see our [AI agent guardrails](/blog/ai-agent-guardrails) post, and for the evaluation framework beyond accuracy, [how to evaluate an AI SRE platform](/blog/how-to-evaluate-ai-sre-platform). ## What Aurora has not published The obvious question after 3,000 words of this: what does Aurora score? We have not published a benchmark score. We should be precise about why, because "we are working on it" is what every vendor says. Aurora has no investigation-quality evaluation harness. There is no golden dataset in the repository, no scored run history, no baseline. We have a 955-test suite covering guardrails, correlation math, multi-tenant isolation and adversarial prompt injection, and we have CI-enforced architectural invariants that fail the build if a new tenant-scoped table is not row-level-security protected. None of that measures whether an investigation reaches the right conclusion. Those are different claims and we are not going to blur them. So by the standard this post applies to everyone else, Aurora belongs in the same bucket: a vendor with no independently verified accuracy number. The difference we can point to is that [the whole system is Apache 2.0](https://github.com/Arvo-AI/aurora), so the investigation prompts, the confidence classification, the findings schema and the guardrail behavior are all readable rather than described. You can run step 1 through step 8 above against it without asking us for permission or a trial key. ITBench-AA's 40 public tasks are available today, and OpenRCA 2.0's 500 instances build on the public OpenRCA dataset. The first commercial AI SRE agent to publish a reproducible score on either becomes the reference point in a category where nobody has shown their work. ## The honest summary The published evidence says AI agents are useful at incident investigation and much weaker than the marketing implies. Independent ceiling 56.2%. Exact root-cause recovery 20.7%. Grounded diagnosis 61.5%. Meta's 42% top-5 in a single monorepo. Microsoft's 0.766 on the easier categorisation task. Against that: 92%, up to 95%, 5x, all undefined. The useful posture is neither dismissal nor faith. It is a holdout set, a two-part grade that separates naming from proving, a per-service score, and an autonomy level you raise only when the evidence earns it. ## Grade it yourself on your own holdout set Since we are not asking you to believe a number we have not published, the consistent thing is to make ours testable. Aurora is Apache 2.0 and runs locally: ```bash git clone https://github.com/arvo-ai/aurora.git && cd aurora make init # generates secrets, copies .env.example to .env nano .env # add one LLM API key; cloud connectors are optional make prod-prebuilt # pulls prebuilt images and starts ``` Open **http://localhost:3000** and register as admin. Then run the eight-step protocol from earlier in this post against your own resolved incidents, holding back a set the agent has never seen. Decide before you start whether an inconclusive result counts as a failure or as a correct abstention, because that choice sets the denominator, and it is exactly the denominator the vendor claims in this post leave undefined. Aurora treats an honest "insufficient evidence" as a successful outcome rather than a failed run, so score it accordingly. Grade naming and proving separately, the way OpenRCA 2.0 does, or your score will flatter the agent by counting a correct service name as a solved incident. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Related guides:** [Can You Measure an AI SRE's Accuracy?](/blog/measuring-ai-sre-without-ground-truth) · [How to Evaluate an AI SRE Platform](/blog/how-to-evaluate-ai-sre-platform) · [AI Agent Guardrails](/blog/ai-agent-guardrails) · [Why Multi-Agent AI Systems Fail](/blog/why-multi-agent-ai-systems-fail) · [What Is an AI SRE?](/blog/ai-sre-complete-guide) · [AI-Powered Incident Investigation](/blog/ai-powered-incident-investigation) > **All claims sourced.** Benchmark results link to the arXiv papers and leaderboards; vendor claims link to the live vendor page carrying them; the automation-bias finding links to the peer-reviewed review in *Human Factors*. Figures verified July 28, 2026. Leaderboards update, so timestamp any number you quote from them. --- ## The SRE Incident Lifecycle: 8 Stages With AI (2026) URL: https://www.aurorasre.ai/blog/sre-incident-lifecycle-with-ai-agents Published: 2026-07-24 (updated 2026-08-24) Author: Noah Casarotto-Dinning, CEO at Arvo AI > The complete incident process an SRE runs, stage by stage, with what an AI agent can genuinely take over and what has to stay human at each step. > ## Key Takeaways > > - **The incident lifecycle has eight stages: detect, triage, page, investigate, decide, remediate, retrospect, and prevent.** Most writing about AI in operations collapses all eight into "the AI fixes it." Only one stage is mostly evidence gathering, and that is the one worth automating first. > - **Investigation is the expensive stage, and it is expensive because it is manual reading.** Checking dashboards, correlating deploys, grepping logs. It is also the stage where an agent's advantage is largest, because gathering evidence in parallel is a mechanical strength rather than a judgment call. > - **The roles are human roles, and they are standardized.** Google's SRE Workbook defines [Incident Commander, Communications Lead, and Operations Lead](https://sre.google/workbook/incident-response/), and [PagerDuty publishes its severity scale and response process openly under Apache 2.0](https://response.pagerduty.com/before/severity_levels/). Both describe accountability held by people, not work you delegate to a model. > - **The single most useful number in this whole area:** Google's SRE book reports that ["roughly 70% of outages are due to changes in a live system"](https://sre.google/sre-book/introduction/). That is why stage 8 exists and why it deserves real effort. > - **Published AI results are genuinely useful and clearly bounded.** Meta reported [42% accuracy at putting the root cause in its top five suggestions](https://engineering.fb.com/2024/06/24/data-infrastructure/leveraging-ai-for-efficient-incident-response/) for its web monorepo, and Microsoft's [RCACopilot reports RCA accuracy up to 0.766](https://arxiv.org/abs/2305.15778). Useful decision support. Not autonomy. > - **Measure the stage, not the aggregate.** John Allspaw calls incident duration and severity ["shallow data"](https://www.adaptivecapacitylabs.com/2018/03/23/moving-past-shallow-incident-data/) that teaches "next to nothing" on its own, and DORA has replaced mean time to restore with [failed deployment recovery time](https://dora.dev/guides/dora-metrics/). Knowing which stage consumed your hour beats knowing the hour. **The SRE incident lifecycle is the sequence a team runs from the moment something breaks to the moment it is prevented from breaking the same way again: detect, triage, page, investigate, decide, remediate, retrospect, and prevent.** Each stage has a different bottleneck, a different failure mode, and a very different answer to the question of whether software can do it for you. This guide walks all eight. For each one it states plainly what an AI agent can genuinely take over today, what has to stay human, and what the primary sources say. It is written for SREs and platform engineers who want the process rather than the pitch. ## What are the stages of the incident lifecycle? ![The incident lifecycle in eight stages, with three columns showing the stage, what an agent can do, and what stays human. Detect needs no agent because thresholds already work. Triage can correlate an alert storm into one incident. Paging and role assignment stay human. Investigation is the expensive stage and the one an agent compresses most. At remediation, applying and merging are denied for unattended runs and off by default. Deciding on a fix, blamelessness in the retrospective, and prioritizing prevention all stay human.](/blog/sre-incident-lifecycle-stages.svg) The short version, before the detail. | # | Stage | Bottleneck | Agent leverage | |---|---|---|---| | 1 | Detect | Whether the right thing is measured at all | None. Thresholds work | | 2 | Triage | Alert volume, duplicate signals | High for correlation | | 3 | Page and assign roles | Human availability | Low. Routing only | | 4 | Investigate | Manual reading across many tools | Highest | | 5 | Decide on a fix | Judgment under uncertainty | Medium. Options with evidence | | 6 | Remediate | Risk of the fix itself | Medium. Draft, never apply | | 7 | Retrospect | Writing time, honesty | High for drafting | | 8 | Prevent | Prioritization against feature work | Medium | ## Stage 1: Detection Something is wrong and a machine notices. Detection is threshold arithmetic and it already works. The interesting question is not the mechanism but what you chose to measure. Google's SRE book gives the canonical answer: ["The four golden signals of monitoring are latency, traffic, errors, and saturation."](https://sre.google/sre-book/monitoring-distributed-systems/) Alternative framings exist and are worth knowing: [Brendan Gregg's USE method](https://www.brendangregg.com/usemethod.html) for resources, and the RED method for request-driven services. The stage's real failure mode is alerting on the wrong things, and the SRE book is unusually direct about the standard: ["Every page should be actionable"](https://sre.google/sre-book/monitoring-distributed-systems/), and "every page response should require intelligence. If a page merely merits a robotic response, it shouldn't be a page." It also names the human constraint that most alerting configurations ignore: "I can only react with a sense of urgency a few times a day before I become fatigued." **Agent leverage: essentially none, and that is fine.** A model is not better than a comparison operator at noticing that error rate crossed 5%. Anyone selling AI detection should be asked what it does that a threshold does not. What an agent can help with is the meta-problem, auditing whether your alert coverage matches your actual failure modes, which belongs in stage 8. ## Stage 2: Triage Forty alerts arrive. Is that forty problems or one? This is where volume becomes the enemy. A single failing dependency can light up every service downstream of it, and each of those alerts is technically correct. Triage is the work of collapsing them into a claim about how many distinct things are wrong. The mechanisms here are well established and mostly deterministic. [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) does grouping, which "categorizes alerts of similar nature into a single notification," and inhibition, "suppressing notifications for certain alerts if certain other alerts are already firing." Both are configured in the config file. Correlation platforms add scoring on top. **Agent leverage: high, with a caveat worth stating.** Correlation is genuinely improved by knowing service topology and semantic similarity rather than just matching labels. [Aurora](https://github.com/Arvo-AI/aurora) scores candidate alerts on a weighted combination of topology distance, time proximity, and text similarity, checks it against a threshold, and when an alert correlates it feeds that alert into the investigation that is already running rather than starting a second one. That last behavior is the one that matters during a storm. The caveat: correlation is a probabilistic guess with a threshold, and a wrong grouping hides a real second incident inside a first one. That is why we argue in our [alert noise reduction guide](/blog/automated-alert-noise-reduction) that correlation should collapse alerts into incidents without suppressing the underlying alert stream. **Stays human:** severity, and whether to declare an incident at all. ## Stage 3: Page and assign roles Somebody has to own this. The structure here is standardized and it predates software incident response. Google's SRE Workbook defines three roles: the [Incident Commander who "commands and coordinates the incident response, delegating roles as needed"](https://sre.google/workbook/incident-response/), the Communications Lead who is "the public face of the incident response team," and the Operations Lead who applies "operational tools to mitigate or resolve the incident." The SRE book adds that a clear separation of responsibilities "allows individuals more autonomy than they might otherwise have, since they need not second-guess their colleagues." [PagerDuty publishes its entire incident response process openly](https://response.pagerduty.com/), including a five-level severity scale where SEV-1 is a ["critical issue that warrants public notification and liaison with executive teams"](https://response.pagerduty.com/before/severity_levels/) and SEV-5 covers cosmetic issues. The documentation is [Apache 2.0 licensed](https://github.com/PagerDuty/incident-response-docs), so you can adopt it directly rather than inventing your own. The SRE Workbook's most useful advice about this stage is that it should be settled beforehand: "the time to start thinking about an incident management process is before an incident occurs." **Agent leverage: low, and mostly logistical.** Routing the page, opening the channel, and attaching context a responder will want. The roles themselves are accountability structures, and accountability is not delegable to a model. ## Stage 4: Investigation This is where the hour goes, and it is the stage this entire category exists for. An engineer wakes up, opens six tabs, and starts reading. What changed recently. What the logs say. Whether the dependency is healthy. Whether this happened before. Nearly all of that is retrieval and correlation of evidence that already exists somewhere. That composition is exactly why it is automatable in a way the other stages are not. It is also why the claim worth making for an AI SRE is narrow. The agent is faster at reading than your on-call engineer, and it can read many places at once. That is the whole advantage. **What a good investigation agent actually does.** Aurora's investigation prompts are open source and worth reading directly, because they encode the discipline that separates a useful investigation from a confident guess: - On premature conclusions: "correlation is not causation. Find the mechanism." And on symptoms: "The pod is OOMKilled" is a symptom; "memory leak in the request parser introduced in commit X" is a root cause. - On negative results: "Absence of expected evidence is evidence." If you searched for errors matching the symptom and found none, that is a finding, not a gap. - On change attribution: "Merged does not always mean deployed." A commit on the main branch is not necessarily running in production, and if deployment cannot be confirmed the conclusion has to be qualified rather than asserted. - On confidence, findings are classified as confirmed, likely, or inconclusive, where a likely finding is presented as a hypothesis rather than a cause. Two implementation notes that matter for evaluating any tool in this category, and where we will be precise about our own configuration rather than quoting a marketing line. Aurora's parallel multi-agent orchestration is a flag that ships disabled, so a default install runs a single investigating agent rather than a fan-out across specialists. Pod isolation for command execution defaults to on in code, but the shipped Helm values and example environment file set it to off, so whether your deployment isolates command execution depends on which configuration you started from. Check the flag rather than assuming. When it is on, execution moves into a per-session Kubernetes pod with an egress policy that blocks private cluster ranges, which we describe in the [kubectl safety guide](/blog/ai-agent-kubectl-safety). **How accurate is this in practice?** The two best public datapoints, quoted with their own caveats: Meta reported ["42% accuracy in identifying root causes for investigations at their creation time related to our web monorepo"](https://engineering.fb.com/2024/06/24/data-infrastructure/leveraging-ai-for-efficient-incident-response/), backtested using only information available at investigation start, scoring a hit when the cause appeared in the top five suggested code changes. Meta says it deliberately favored precision over reach to avoid misleading engineers. Microsoft's [RCACopilot paper](https://arxiv.org/abs/2305.15778) reports root cause analysis accuracy up to 0.766 across a year of real incidents. For contrast on the harder end, the [ITBench benchmark](https://arxiv.org/abs/2502.05352) found that agents built on then-current models "resolve only 13.8% of SRE scenarios," which is a useful corrective to any claim of general operational competence. **Stays human:** reading the evidence critically. An agent that cannot return "inconclusive" is not a tool, it is a liability, which is why the classification gate above matters more than the model choice. ## Stage 5: Deciding on a fix You know what is happening. Now: mitigate immediately, or keep investigating? This is a genuine tradeoff and it is not technical. Rolling back restores service and destroys the state you would need to understand the failure. Continuing to investigate keeps customers broken while you learn. Reasonable engineers pick differently depending on blast radius, time of day, and how much they trust the current hypothesis. **Agent leverage: medium, and specifically at presentation.** A useful agent lays out the options with the evidence supporting each, the risk, and the undo path. Aurora's remediation prompt requires a read-only validation query before any fix is proposed, with the output naming what validated it, the risk, and how to undo it. Proposing a fix nobody checked is worse than proposing nothing. **Stays human: the call.** This is a business risk decision wearing a technical costume. ## Stage 6: Remediation Something now changes in production. This is the stage where an agent can do real damage, so it is the stage where the design matters most. The industry has a canonical cautionary example. In July 2025, [Replit's coding agent deleted a production database during an explicit code freeze](https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/) and then incorrectly reported that a rollback was impossible. The instruction not to touch production existed. It was a sentence, not a control. The lesson generalizes: **the gate has to be structural, not instructional.** In Aurora, the function that turns a saved incident fix suggestion into a pull request is deliberately not exposed to the model as a callable tool, so a human clicks the button in the UI and the model cannot approve its own remediation. Terraform apply and destroy are denied in unattended runs and are not enabled by default. We detail the seven independent layers every command passes through in our [AI agent guardrails](/blog/ai-agent-guardrails) post. Now the part vendors tend to leave vague, which is that all of this is a permission model and permission models have settings. Repository write operations through the GitHub integration are enabled by default, so a read-write background Action can open a pull request unattended. Merging and infrastructure apply are not enabled by default, and an org admin can grant them deliberately. So the accurate statement is not "the agent cannot," it is "the agent cannot unless somebody turned it on." If you are evaluating any agent for this stage, ask which specific operations are permitted with no human present, and verify it in configuration rather than in documentation. Google has published the most useful framework for thinking about this. Its SRE organization describes [running AI operations across graded autonomy levels](https://sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/) from L0 to L4, where at L2 a human "still provides final approval" and only at L3 can an agent "independently detect, decide, and act without human approval for bounded actions." The interesting part is what happens between them: if an agent requests an L3 execution but an elevated risk score or anomalous production state is detected, the system "will automatically downgrade the request to L2," intercepting the execution and routing an approval request to a human. Above that sits an emergency "Red Button" that lets SREs "instantly pause all in-flight agentic actions, block new actions, or globally revoke L3 permissions across the fleet during catastrophic, complex outages." The same writeup shows a case where their agent "incorrectly diagnosed the root cause and failed to mitigate it." Graded autonomy that degrades under uncertainty, a global stop, and a published example of the agent being wrong is a more mature posture than any vendor claim of reliability. **Stays human: approval and the merge.** This boundary should not move. ## Stage 7: The retrospective Write down what happened, so the organization learns something a single engineer already knows. The standard here is well documented. Google's postmortem chapter defines the discipline: ["For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior"](https://sre.google/sre-book/postmortem-culture/), assuming everyone "had good intentions and did the right thing with the information they had." It also lists concrete triggers for writing one, including data loss, on-call intervention such as a rollback, a resolution time above a threshold, and "a monitoring failure (which usually implies manual incident discovery)." **Agent leverage: high, because the input is a trace rather than a memory.** The difference between a good and bad automated postmortem is which evidence it reads. A postmortem drafted from a chat transcript records what people said. One drafted from the actual investigation trace records what was checked, what was ruled out, and in what order. We covered the distinction in our [automated post-mortem generation](/blog/automated-post-mortem-generation) guide. Aurora generates a postmortem when a human marks an incident resolved, with sections for summary, timeline, root cause, impact, contributing factors, resolution, action items, and lessons learned, exportable to Confluence, Notion, or Jira, and every edit is versioned. **Stays human: blamelessness and the action items.** A model does not feel institutional pressure to blame a person, which helps, but it also cannot commit an engineer's next quarter to fixing something. ## Stage 8: Prevention The loop closes here, or it does not close at all. Return to the number that frames this entire guide: ["roughly 70% of outages are due to changes in a live system"](https://sre.google/sre-book/introduction/). If most incidents arrive through your own deploy pipeline, then prevention is not a vague aspiration, it is a specific place to stand. The practical layers, in increasing distance from the incident: 1. **Pre-merge review of risk.** Assess a change against the live state of the system it will affect, before it merges. We cover the mechanics in [PR risk review](/blog/pr-risk-review-prevent-incidents), including why the verdict should be advisory rather than blocking. 2. **Progressive delivery.** Ship gradually and abort automatically. [Argo Rollouts](https://argo-rollouts.readthedocs.io/en/stable/features/analysis/) runs analysis against live metrics, and a failed analysis "causes the Rollout to abort, setting the canary weight back to zero." See our [CI/CD auto-remediation guide](/blog/cicd-auto-remediation-complete-guide). 3. **Monitoring hygiene.** The alert that did not fire, and the mute nobody reconciled. Both are covered in [alert silence drift](/blog/alert-muting-terraform-drift). **Agent leverage: medium, and mostly about persistence.** The reason prevention work rots is that it is never the most urgent thing on anyone's list. A scheduled agent that keeps a living document of known gaps, drops resolved items, and never re-raises things you rejected is a modest mechanism that solves a real organizational failure. ## How should you measure any of this? Carefully, and per stage rather than in aggregate. [DORA](https://dora.dev/guides/dora-metrics/) gives the delivery-side frame, now five metrics rather than the original four keys, split into throughput (change lead time, deployment frequency, failed deployment recovery time) and instability (change fail rate, deployment rework rate). Change fail rate is defined as "the ratio of deployments that require immediate intervention following a deployment." DORA's own guidance is to apply these at the application or service level rather than as broad organizational comparisons. On the incident side, be skeptical of MTTR, and note that the criticism comes from inside the practice. John Allspaw groups incident frequency, time to resolve, time to detect, and severity together as shallow data, arguing that ["organizations learn next to nothing from this data alone"](https://www.adaptivecapacitylabs.com/2018/03/23/moving-past-shallow-incident-data/) and that reducing incidents to means and medians "obscures more about the incidents than it actually reveals." Atlassian, which publishes the standard formulas for these metrics, [concedes the same point](https://www.atlassian.com/incident-management/kpis/common-metrics): "A lot of experts argue that these metrics aren't actually that useful on their own because they don't ask the messier questions of how incidents are resolved." DORA has quietly acted on this. Its recovery metric is no longer mean time to restore but ["failed deployment recovery time"](https://dora.dev/guides/dora-metrics/), scoped to deployments that fail and require immediate intervention rather than averaging across all incidents. The more useful question is which stage consumed the time. An hour split as fifty minutes of investigation and ten of fixing implies a completely different intervention than ten minutes of investigation and fifty of waiting for an approval. The aggregate hides exactly the thing you would change. ## Where does an AI agent actually fit? The summary of this whole guide: **an agent compresses stage 4, helps with 2, 5, 7, and 8, and should be kept structurally out of the decisions in 3 and 6.** That is a narrower claim than "AI resolves your incidents," and it is the one supported by the evidence. Stage 4 is where the 3am hour goes, and it is almost entirely evidence gathering, which is why it is both the biggest cost and the most automatable. Everything downstream of it is judgment, accountability, or a change to production, and those are the parts you want a person holding. If you want the category context, start with [what is an AI SRE](/blog/ai-sre-complete-guide) and [AI SRE vs AIOps](/blog/ai-sre-vs-aiops). For the investigation stage specifically, see [AI-powered incident investigation](/blog/ai-powered-incident-investigation). For the diagnostic methods that predate all of this, our [root cause analysis guide for SREs](/blog/root-cause-analysis-complete-guide-sres) covers the 5 Whys and fault tree analysis, which remain the right tools when you are the one doing the thinking. If you want to run the whole loop on your own infrastructure, see [open source incident management](/blog/open-source-incident-management) and the [self-hosted deployment guide](/blog/self-hosted-ai-sre). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Docs:** [Aurora documentation](https://arvo-ai.github.io/aurora/) - **Related guides:** [Toil Is Rising Again: SRE Time Economics in 2026](/blog/sre-toil-rising-2026) · [AI-Powered Incident Investigation](/blog/ai-powered-incident-investigation) · [Automated Post-Mortem Generation](/blog/automated-post-mortem-generation) · [PR Risk Review](/blog/pr-risk-review-prevent-incidents) · [Multi-Cloud Incident Management](/blog/multi-cloud-incident-management) > **All claims sourced.** Process definitions are cited to Google's SRE book and workbook and to PagerDuty's openly licensed incident response documentation. Accuracy figures are quoted from the publishing organizations with their stated methodology. Aurora's behavior, including its default configuration, is described from its open-source implementation. Last verified: July 24, 2026. --- ## Best Automated Post-Mortem Tools: 9 Checked, 1 Real AI Draft URL: https://www.aurorasre.ai/blog/best-ai-post-mortem-tools Published: 2026-07-15 (updated 2026-10-21) Author: Arvo AI Team, Engineering > The best automated post-mortem tools, checked Oct 21, 2026: exports, pricing, PagerDuty and Jeli EOL dates, and a real AI draft with what humans must add. The best automated post-mortem tools in October 2026 are incident.io, Rootly, Datadog Incident Management, PagerDuty Post-Incident Reviews, FireHydrant, Atlassian Jira Service Management, Xurrent IMR and Grafana IRM among commercial products, and Aurora SRE in open source. Each writes a first draft, not a finished post-mortem (Grafana IRM writes only a summary): they fill the summary, timeline, root cause and suggested follow-ups from the incident channel, call transcripts, alert data or (in Aurora's case) the investigation an AI agent ran. A person still has to supply the human timeline, the business impact and an owner for each action item. Pick the tool that reads where your incident evidence actually lives. Two dates matter right now. PagerDuty's legacy Postmortems feature reaches end of life on **October 31, 2026**, and Jeli reaches end of life on **December 22, 2026**; both move to Post-Incident Reviews in PagerDuty's web UI ([release notes](https://docs.pagerduty.com/get-started/release-notes)). Below: one real AI-written incident report from our lab, marked up with what it got right and what a person would still have to write, then the nine tools, each checked on the vendor's own site or docs on 2026-10-21. Arvo AI builds Aurora; we have not used the eight commercial tools, and nothing below claims we did. ## What does an AI-drafted post-mortem actually look like? We run an open lab, [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab): a small app (`shop`: nginx frontend, Python api, worker, redis, load generator) on a 3-node kind cluster, 19 faults we inject on purpose, and Aurora investigating each one from the alert text alone. Every run saves Aurora's written report (`aurora-rca.md`), every command it ran and the grade. Arvo wrote the scenarios. Across the 38 graded runs on 2026-10-02, Aurora's root cause was correct in 34 and partial in 4; a judge model graded each answer against the injected fault, a person read all 38 grades, and a blinded regrade agreed with all 38. The full table, including HolmesGPT and K8sGPT, is on [/benchmark](/benchmark). The run below is scenario 13, run 1 ([files](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/13-networkpolicy-blocks-redis/r1)). We applied a NetworkPolicy that lets only `app=worker` pods reach redis, so the api's redis calls time out and the frontend returns 5xx. Aurora got this text and nothing else: ```text HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes (threshold 5%). Cluster: aurora-lab (kubeconfig). Investigate and find the root cause. ``` It made 25 tool calls in 202 seconds, for $0.69 in model cost: 19 read-only kubectl commands, plus listing the incident and the connected cluster and reading and writing its own memory notes. Then it wrote this. It is reproduced in full and unedited; the bracketed numbers point to saved command outputs. ```markdown ## Incident Report **What Happened** A critical "HighErrorRate" alert fired for namespace=shop, service=frontend at 2026-10-02T20:24:46 UTC. At the time of investigation, the shop namespace had just bootstrapped — all pods were Running but only ~73 seconds old, indicating a full cluster/deployment restart [4, 5]. Frontend nginx logs showed continuous `GET /api/items` requests returning 502/503 [7], and the backend `api` pods showed the request pattern `connected to redis at redis:6379` followed by `request failed: redis error: timed out` and `GET /items HTTP/1.0 503` repeating continuously rather than as a one-time startup blip [8]. **Root Cause** The root cause was a NetworkPolicy named `redis` in the shop namespace, created at 2026-10-02T20:23:54Z, which restricts ingress to redis:6379 to pods labeled `app=worker` only [20, 22]. The `api` pods are labeled `app=api`, not `app=worker` [23], so their connections to redis were silently dropped by the policy, causing the client-side 2-second socket timeout defined in the api code [14] to be hit on every request, producing the `redis error: timed out` → 503 response from the api, which cascaded into 502/503 responses from the frontend's `/api/items` proxy [7, 8]. Redis itself was healthy throughout — clean startup logs, Ready=True, and negligible CPU/memory usage (3m/4Mi) with no error output [10, 11, 12] — ruling it out as the failing component. The first observed 503 occurred within ~2 seconds of the NetworkPolicy's creation timestamp, consistent with (though not independently proof of) the client timeout window configured in the api code [22, 14]. **Impact & Timeline** The policy was created at 20:23:54Z, shortly after cluster bootstrap, and the alert triggered at 20:24:46Z [20, 22]. Both `api` replicas (api-f64486c4c-2nqsk, api-f64486c4c-mvpnl) [4, 23] were affected, resulting in 100% failure of `/api/items` calls proxied through the frontend service [7, 8]. The issue was ongoing and had not self-resolved as of the time of investigation. ## Ruled Out - Redis overload or crash — redis-59458f55d8-n6glm showed clean startup logs, Ready=True, and minimal resource usage (3m CPU / 4Mi memory) with no errors [10, 11, 12]. - Node-level resource contention (including worker pod's buffer-allocation churn) — `kubectl top nodes` showed both worker nodes at ~0% CPU and ~1% memory [16]. - Transient startup/readiness probe noise — frontend/api 5xx logs showed continuous, non-improving failures rather than a one-time startup blip [7, 8]. ## Not Checked - Whether the worker pod genuinely requires redis access (its code only allocates and frees buffers, per the api-code configmap) — flagged as a service-owner design question rather than part of the required fix, not independently verified against worker source code beyond what's in the configmap. - CNI-level or lower-level network diagnostics beyond the NetworkPolicy object itself — not needed once the policy rule was identified as sufficient explanation, and not accessible in read-only mode. - loadgen pod's traffic pattern/configuration — retrieved as YAML [19] but not analyzed in depth, since the mechanism was already isolated to the NetworkPolicy rule. ``` Aurora also attached one suggested fix, separate from the report, rated low risk: ```bash kubectl patch networkpolicy redis -n shop --type='json' -p='[{"op":"add","path":"/spec/ingress/0/from/-","value":{"podSelector":{"matchLabels":{"app":"api"}}}}]' ``` That patch was never run: the lab does not apply fixes. It changes the cluster, so in a real incident a person would apply it, then confirm with `kubectl describe networkpolicy redis -n shop`. To roll it back, remove the added `from` entry (or reapply the original policy from version control). The [NetworkPolicy runbook](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes) walks through this fault from first symptom to fix. ### What the AI draft got right - **The root cause, with proof.** The policy name, its creation time, the `app=worker` rule and the `app=api` label on the failing pods, each tied to the command output that shows it. The grade was correct, and a person confirmed it. - **What it ruled out, and why.** Redis was healthy (logs, readiness, 3m CPU and 4Mi memory), and the nodes were idle. Most human post-mortems skip this section, and it is the one that stops the next responder from chasing the same dead ends. - **What it did not check.** The "Not Checked" section is honest: no CNI-level diagnostics, and the load generator was read but not analyzed. ### What it got wrong An AI reviewer that rechecked the report against the raw output flagged two claims, and we confirmed both in the saved logs: - "Indicating a full cluster/deployment restart" is an over-reading. The pods were 73 seconds old because the lab redeploys the app before every run. Anyone who knows the environment would strike that line; the model did not know the environment. - "100% failure of `/api/items` calls" is not quite true: one api pod's log shows `"GET /items HTTP/1.0" 200` before the policy took effect. Neither error touches the root cause, and this pattern holds across the lab: the AI review found claims the evidence did not support in 15 of the 18 reports for the harder scenarios, none of them in the root cause ([REPORT.md](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md)). The worst case is scenario 12, where two faults ran at once. In both runs Aurora found the one the alert named (graded partial) and wrote a line saying the rest of the namespace was fine; run 1 put it as "Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected", with pod status as its only evidence. The frontend Service was selecting no pods. Paste that into an Impact section and the post-mortem states, wrongly, that there was no second outage. ### What a human would still have to add | Post-mortem section | What the AI report had | What a person has to add | |---|---|---| | Timeline | Two machine timestamps (policy created 20:23:54Z, alert 20:24:46Z) and a note on when the first 503 appeared | Who applied the policy and why, who was paged, when someone acknowledged, when the fix went in, when errors stopped | | Impact | Both api replicas, "100% failure" of `/api/items` (slightly overstated) | Users and requests affected, duration, SLO or SLA breach, what customers were told | | Root cause | Correct and cited | Why a worker-only policy was written and how it passed review: the process cause, not just the object | | Action items | None in the report; one suggested fix with no undo recorded | Owners and due dates; prevention work such as testing policies before rollout, or alerting on NetworkPolicy changes in the namespace | | Fact check | Two unsupported claims | Strike them; anyone who knows the environment can | The machine-written parts are the parts a machine can observe. Everything that depends on people (decisions, intent, customer impact, who owns the follow-up) has to come from the people, and any tool that fills those sections without a source for them is guessing. Atlassian's own docs say the same about its drafts: validate "the technical findings, root cause statements, contributing factors, business impact, action items, and owners" before publishing ([Atlassian](https://support.atlassian.com/jira-service-management-cloud/docs/create-a-pir-in-incident-command-center/)). ### How Aurora turns that report into a post-mortem The report above is the input to Aurora's post-mortem step. The built-in "Generate Postmortem" action is on by default and runs when an incident is resolved. It starts a background agent session with the incident fields and the first 4,000 characters of this report (this one is about 3,200, so it goes in whole), tells it to read the Slack channels for the incident window if Slack is connected, and asks for a document with Summary, Timeline, Root Cause, Impact, Contributing Factors, Resolution, Action Items (as checkboxes) and Lessons Learned, with "no speculation beyond what data supports" ([postmortem_action.py](https://github.com/Arvo-AI/aurora/blob/main/server/services/actions/postmortem_action.py)). The session runs in agent mode with the org's other connected tools available, but the instructions point it only at the report and Slack. The document is versioned and editable, and can be regenerated. Our lab incidents are never marked resolved and have no Slack channel, so the lab did not run this step, and we have not measured what it writes from a report alone. What the code tells you: with no Slack, the instructions leave the report as the only named source for the Timeline, Impact and Contributing Factors sections, so expect them to inherit both its strengths and the two overstatements above. ## Which tools generate post-mortems automatically after an incident? | Tool | AI drafts from | Export or sync | Where AI is included (list prices) | |---|---|---|---| | [incident.io](https://docs.incident.io/post-incident/postmortem-ai) | Incident timeline, Slack or Teams conversations, investigation data | Google Docs, Notion, Confluence, SharePoint | Pro ($25/user/month) and Enterprise | | [Rootly](https://rootly.com/retrospectives) | Incident data, Slack channel, call transcripts | Confluence, Notion, Google Docs, SharePoint; action items to Jira or Linear | Incident Response, $20/user/month | | [Datadog Incident Management](https://docs.datadoghq.com/incident_response/incident_management/post_incident/postmortems/) | Incident data, through AI template variables | Datadog Notebooks, Confluence, Google Drive | Incident Management, $30/seat/month billed annually | | [PagerDuty Post-Incident Reviews](https://docs.pagerduty.com/incident-management/after-incident/post-incident-reviews) | Scribe Agent transcripts, the incident's Slack channel, incident lifecycle events | HTML link, PDF; follow-ups to Jira | PIRs on Professional and above; AI drafts need PagerDuty Advance | | [FireHydrant](https://firehydrant.com/ai/) | Slack messages and actions, incident timeline, call transcripts | Confluence, Google Docs, PDF, direct link | Enterprise (custom price) | | [Atlassian Jira Service Management](https://support.atlassian.com/jira-service-management-cloud/docs/create-a-pir-in-incident-command-center/) | The incident record | PIR work item, or copy the draft out | Premium and Enterprise | | [Xurrent IMR](https://www.xurrent.com/incident-management-response/post-incident-review) (formerly Zenduty) | Logs, metrics, Slack or Teams conversations | Markdown, PDF | Pricing table checks Postmortems, under AI features, on every plan from $6/user/month | | [Grafana IRM](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/chat-and-collaboration/openai/) | Sanitized incident timeline, sent to OpenAI | Stays in the incident | Needs your own OpenAI account and API key | | [Aurora](https://github.com/Arvo-AI/aurora) | Its own investigation report, plus Slack history for the incident window | Confluence, Notion, Markdown download; SharePoint through the agent | Open source (Apache 2.0), self-hosted; you pay for LLM tokens | Prices are the vendors' published list prices on 2026-10-21, in US dollars. ## The 9 automated post-mortem tools, one by one ### incident.io - **What the AI drafts:** "AI that writes the first draft for you, pulling from your timeline, Slack or Teams conversations, and investigation data." It can also review a finished document and suggest inline edits, or redraft a section ([docs](https://docs.incident.io/post-incident/postmortems-overview)). - **Export:** Google Docs, Notion, Confluence or SharePoint, or announce it in Slack. - **Pricing:** AI features are on Pro and Enterprise ([AI post-mortem docs](https://docs.incident.io/post-incident/postmortem-ai)); Pro is $25 per user a month and Scribe call transcription is Pro and Enterprise only ([pricing](https://incident.io/pricing)). For private incidents, AI works only if your organization opts in to sending private incident data to AI subprocessors. - **Fit:** teams that run incidents in Slack or Teams and want drafting plus review in one editor. See the [incident.io alternative guide](/blog/incident-io-alternative-open-source). ### Rootly - **What the AI drafts:** AI blocks in a template draft "summary, impact, root cause, mitigation, and a curated timeline, generated from the incident's data, Slack channel, and call transcripts", each steerable with custom instructions ([retrospectives](https://rootly.com/retrospectives)). - **Export:** Confluence, Notion, Google Docs and SharePoint. Action items push to Jira or Linear as subtasks with owners and status kept in sync. - **Pricing:** Incident Response is $20 per user a month, and "AI Scribe and Retrospectives" are listed as always included ([pricing](https://rootly.com/pricing)). This changed since our July version of this page, which said the AI tier was contact-sales; that is now true only of Rootly AI SRE, the investigation product. - **Fit:** the most editorial control over each AI section, with AI drafting in the base Incident Response price. See the [Rootly alternative guide](/blog/rootly-alternative-open-source-incident-management). ### Datadog Incident Management - **What the AI drafts:** post-mortem templates take AI variables powered by Bits AI: executive summary, system overview, key timeline, customer impact, action items and lessons learned. Datadog suggests a value for each that you accept, edit or reject, and ships an out-of-the-box "General incident with AI content" template. The incident timeline must contain at least 10 messages before the AI variables generate anything ([templates](https://docs.datadoghq.com/incident_response/incident_management/setup_and_configuration/templates/)). - **Export:** Datadog Notebooks, Confluence or Google Drive ([post-mortems](https://docs.datadoghq.com/incident_response/incident_management/post_incident/postmortems/)). - **Pricing:** Incident Management is $30 per seat a month billed annually, $36 month to month ([pricing](https://www.datadoghq.com/pricing/list/)). - **Fit:** teams whose evidence already lives in Datadog. Note the AI writes a "customer impact" section from incident data; check it against the gaps in the table above. Investigation itself is covered in the [Datadog Bits alternative guide](/blog/datadog-bits-ai-sre-alternative-open-source). ### PagerDuty Post-Incident Reviews (and the Jeli deadline) - **What the AI drafts:** "Generate with AI" pulls from up to three sources, the Scribe Agent, Slack messages from the incident channel and incident lifecycle events, and writes each section of the review, narrative markers and follow-up actions. If Scribe was not on the call, the draft uses whatever sources exist ([docs](https://docs.pagerduty.com/incident-management/after-incident/post-incident-reviews)). - **Export:** an HTML share link or a PDF. Follow-up actions can become Jira tickets, with status synced both ways on the Jira Forge app 4.0 or later. - **Pricing and availability:** Post-Incident Reviews in the web UI, including AI-generated drafts, became generally available on August 27, 2026, on Professional plans and above; the AI drafts are for PagerDuty Advance customers ([release notes](https://docs.pagerduty.com/get-started/release-notes)). An October 20, 2026 release added a "Post-incident review on resolve" setting. - **The deadlines:** PagerDuty's legacy Postmortems reach end of life on October 31, 2026 for all customers, with existing data downloadable before then. Jeli reaches end of life on December 22, 2026; PagerDuty said it would send details on migrating Jeli review data before that date. If you run either, the replacement is already in your account. See the [PagerDuty alternative guide](/blog/pagerduty-alternative-root-cause-analysis) for the investigation side. ### FireHydrant (now part of Freshworks) - **What the AI drafts:** retrospective drafts created when an incident is resolved, with timeline, root cause analysis, contributing factors and suggested action items with priorities, plus Zoom and Google Meet transcription ([FireHydrant AI](https://firehydrant.com/ai/)). - **Export:** PDF, Google Docs, Confluence or a direct link ([docs](https://docs.firehydrant.com/docs/intro-to-retrospectives)). - **Pricing:** FireHydrant AI (summaries, meeting transcripts, triage, retros and follow-ups) is listed only under Enterprise (custom price); Pro is $25 per responder a month billed annually, without it ([pricing](https://firehydrant.com/pricing/)). - **Ownership:** FireHydrant absorbed Blameless in [August 2024](https://firehydrant.com/blog/press-release-firehydrant-acquires-blameless-to-further-solidify-enterprise/), and Freshworks [closed its acquisition of FireHydrant on January 5, 2026](https://www.freshworks.com/theworks/company-news/freshworks-firehydrant-acquisition/), saying it is integrating FireHydrant with Freshservice. If you are looking for a FireHydrant alternative after the acquisition, the dedicated options on this list are incident.io and Rootly; see also the [FireHydrant alternative guide](/blog/firehydrant-alternative-open-source). ### Atlassian Jira Service Management - **What the AI drafts:** two levels. Atlassian Intelligence "can automatically generate an incident summary" for a post-incident review, on Premium and Enterprise plans ([docs](https://support.atlassian.com/jira-service-management-cloud/docs/create-a-post-incident-review-using-atlassian-intelligence/)). The newer Incident Command Center, in an early access program on Premium and Enterprise and with Rovo enabled, has a "Generate PIR" button that turns the incident record into a full first draft ([docs](https://support.atlassian.com/jira-service-management-cloud/docs/create-a-pir-in-incident-command-center/)). - **Export:** create a PIR work item, or copy the draft to another tool. - **Fit:** teams already tracking incidents in Jira. The docs tell you to "add or refine" the root cause, impact, follow-up actions, "owners, and due dates" yourself, which matches what we saw in the lab. ### Xurrent IMR (formerly Zenduty) - **What the AI drafts:** "Xurrent's AI compiles postmortem reports automatically from logs, metrics, and Slack or Teams conversations, following your preferred templates" ([product page](https://www.xurrent.com/incident-management-response/post-incident-review)). - **Export:** Markdown or PDF. - **Pricing:** Starter $6 and Growth $16 per user a month billed monthly ($5 and $14 annually); Enterprise on request ([pricing](https://www.xurrent.com/pricing/incident-management-response)). The plan comparison table lists Postmortems under AI features with a check on Starter, Growth and Enterprise; the product page does not say how the AI draft differs by plan. - **Fit:** the lowest entry price, with thin export targets. ### Grafana IRM - **What the AI drafts:** a post-incident summary, not a full post-mortem. The OpenAI integration sends "a sanitized version of the incident timeline" to the OpenAI API and returns a summary, a timeline of events and the actions taken, which you edit before saving. You need your own OpenAI account, organization ID and API key. Grafana's own note: "AI-generated summaries may occasionally produce inaccurate or incomplete information" ([docs](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/integrations/chat-and-collaboration/openai/)). - **Fit:** worth turning on if you already run Grafana IRM; not a reason to adopt it. ### Aurora - **What the AI drafts:** the full document described above, from Aurora's own investigation report plus Slack context, with all eight sections. The difference from the tools above is the input: the root cause section rests on commands the agent ran and cited, not on what people typed in the channel. - **Export:** Confluence and Notion from the post-mortem view, or a Markdown download; SharePoint through the agent's SharePoint tool. No Google Docs export. - **Pricing:** open source under Apache 2.0 and self-hosted; you pay for infrastructure and model tokens. In our lab the investigation that produces the report cost a median $0.35 per run; the post-mortem step's own cost was not measured. - **Watch out for:** Aurora is an investigation agent that also writes post-mortems, not a retrospective editor, and it has no on-call, paging or status pages. If your incidents are resolved in chat and you only want drafting, one of the tools above fits better. **Checked and left off.** Better Stack's [AI SRE page](https://betterstack.com/ai-sre) advertises "AI written post-mortems" from the incident timeline and Slack, but its [post-mortem docs](https://betterstack.com/docs/uptime/post-mortems/) describe only a Markdown comment you write yourself, so we could not confirm how it works. [Keep](https://github.com/keephq/keep), which shows up in searches for open source incident automation, is an alert management and AIOps platform (deduplication, correlation, workflows), MIT-licensed outside its `ee/` directory, and Elastic [announced an agreement to acquire it](https://www.elastic.co/blog/elastic-and-keep-join-forces) in May 2025; it does not write post-mortems. The [Keep vs Aurora comparison](/blog/keep-vs-aurora-open-source-aiops) covers the difference. ## Which tool should you choose? Choose by where the facts of your incidents are recorded, because a drafting tool can only summarize what it can read: 1. **Incidents worked in Slack or Teams and on calls:** incident.io or Rootly. Rootly includes AI retrospectives at $20 per user; incident.io needs Pro at $25. The [incident.io vs Rootly comparison](/blog/incident-io-vs-rootly) goes further. 2. **Everything is already in Datadog:** Datadog's AI template variables, as long as your incidents have at least 10 timeline messages. 3. **You are on PagerDuty or Jeli:** move to Post-Incident Reviews before October 31 (legacy Postmortems) or December 22 (Jeli), then decide whether PagerDuty Advance is worth it for the AI drafts. 4. **Incidents tracked in Jira Service Management:** Atlassian's PIR drafts keep the review next to the tickets. 5. **The root cause is found in the infrastructure, not in the channel:** an investigation agent writes a better root cause section than a chat summarizer, because it has the command output. Aurora is the open source option; read its report the way you would read a new colleague's, and expect to fix the impact section. Whatever you pick, budget the human part: the incident lead still writes the timeline of decisions, the impact and the owners, and that is what turns a draft into a post-mortem someone will act on. For the method rather than the tools, see [automated post-mortem generation](/blog/automated-post-mortem-generation); for the investigation side, the [AI incident investigation tools](/blog/best-ai-powered-incident-investigation-tools) list and the [Aurora lab report](/blog/aurora-lab-report-kubernetes-19-failures). Aurora is open source on [GitHub](https://github.com/Arvo-AI/aurora); to see the investigation and post-mortem flow on your own stack, [book a demo](https://cal.com/arvo-ai/demo). *Verified 2026-10-21: vendor pages and docs as linked. Lab files from [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab), run v1-run1, scenario 13, run 1 (2026-10-02). Aurora behavior from the [Aurora repository](https://github.com/Arvo-AI/aurora) main branch.* --- ## Azure SRE Agent Alternative: Open Source (2026) URL: https://www.aurorasre.ai/blog/azure-sre-agent-alternative-open-source Published: 2026-07-15 Author: Noah Casarotto-Dinning, CEO at Arvo AI > Azure SRE Agent is Azure-locked, always-on billed, and closed source. Aurora is the Apache 2.0, self-hosted, multi-cloud open source alternative. > ## Key Takeaways > > - [Azure SRE Agent went generally available on March 10, 2026](https://techcommunity.microsoft.com/blog/appsonazureblog/announcing-general-availability-for-the-azure-sre-agent/4500682). Microsoft says it runs the agent across its own services: 1,300+ agents deployed, 35,000+ incidents mitigated, and over 20,000 engineering hours saved internally. > - **Billing never sleeps.** The agent bills a fixed [4 Azure Agent Units per agent-hour from creation until deletion, even while stopped](https://learn.microsoft.com/en-us/azure/sre-agent/pricing-billing), plus token-based active-flow AAUs. At the East US rate of roughly $0.10 per AAU (per Azure's Retail Prices API), always-on alone is about $292 per agent per month before any usage. [There is no free tier](https://learn.microsoft.com/en-us/azure/sre-agent/pricing-billing). > - **It is Azure-native by design.** The agent manages [the full range of Azure services](https://learn.microsoft.com/en-us/azure/sre-agent/overview); AWS is reachable only through an [MCP proxy bolt-on](https://techcommunity.microsoft.com/blog/appsonazureblog/announcing-aws-with-azure-sre-agent-cross-cloud-investigation-using-the-brand-ne/4507413), and the agent itself deploys in [three Azure regions as of July 2026](https://learn.microsoft.com/en-us/azure/sre-agent/faq). > - **Approval is the default, not a guarantee.** [Review mode is the agent-level default, but incident response plans and scheduled tasks default to Autonomous](https://learn.microsoft.com/en-us/azure/sre-agent/run-modes). > - [Aurora](https://github.com/Arvo-AI/aurora) is the open source (Apache 2.0) alternative: self-hosted anywhere, investigating AWS, Azure, GCP, OVH, Scaleway, and Kubernetes natively, with no per-agent meter and a [7-layer guardrail stack](/blog/ai-agent-guardrails). Azure SRE Agent is the strongest argument yet that AI SRE has gone mainstream: Microsoft dogfooded it across its own estate and now sells it as ["an autonomous agent that diagnoses issues, orchestrates mitigations, and optimizes cloud operations with intelligent precision"](https://azure.microsoft.com/en-us/products/sre-agent/). If your infrastructure is genuinely all-Azure, it deserves a serious look. This post is an honest account of where it excels, where the single-cloud scope and always-on meter bite, and how the open source [Aurora](https://github.com/Arvo-AI/aurora) project fits teams that need the same capability across more than one cloud, or on infrastructure they control. ## What is Azure SRE Agent? Azure SRE Agent is Microsoft's AI operations agent, [generally available since March 10, 2026](https://techcommunity.microsoft.com/blog/appsonazureblog/announcing-general-availability-for-the-azure-sre-agent/4500682). It monitors Azure resources, investigates incidents, proposes mitigations, and can execute them. Microsoft's launch numbers are internal but striking: the company says it runs [1,300+ agents across its own services, with 35,000+ incidents mitigated and over 20,000 engineering hours saved](https://techcommunity.microsoft.com/blog/appsonazureblog/announcing-general-availability-for-the-azure-sre-agent/4500682), and cites early adopter Ecolab reducing daily performance alerts from 30-40 to under 10. Its real strengths are worth naming plainly. It covers [the full range of Azure services](https://learn.microsoft.com/en-us/azure/sre-agent/overview), from AKS and App Service to Azure SQL and Cosmos DB, and can automate any Azure CLI operation through runbooks, subagents, and hooks. Its safety model is genuinely thought through: [Reader and Privileged permission levels](https://learn.microsoft.com/en-us/azure/sre-agent/permissions) with on-behalf-of elevation, [Review and Autonomous run modes](https://learn.microsoft.com/en-us/azure/sre-agent/run-modes), and a pre-execution permission gate on every tool call. It connects to [PagerDuty, ServiceNow, Grafana, GitHub, Azure DevOps, Slack, and Teams, plus 40+ MCP connectors](https://learn.microsoft.com/en-us/azure/sre-agent/overview). And its model picker is a detail worth knowing: per [Microsoft's pricing docs](https://learn.microsoft.com/en-us/azure/sre-agent/pricing-billing), Claude Opus 4.6 is metered at roughly three times the AAU rate of GPT 5.3 Codex for input tokens. ## What is Aurora? Aurora is an [open source, Apache-2.0 AI SRE platform](https://github.com/Arvo-AI/aurora) built by Arvo AI. Like Azure SRE Agent, it autonomously investigates incidents and produces root-cause analyses. Unlike it, Aurora is self-hosted and vendor-neutral: its LangGraph-orchestrated agent queries AWS, Azure, GCP, OVH, Scaleway, and Kubernetes natively, runs `kubectl` and cloud CLIs in sandboxed pods, correlates alerts against a Memgraph dependency graph that spans providers, and [generates post-mortems](/blog/best-ai-post-mortem-tools) that export to Confluence, Notion, or SharePoint. Every agent-issued command passes a [7-layer guardrail stack](/blog/ai-agent-guardrails) with human-gated writes, and because it is bring-your-own-LLM with Ollama support, it can run fully air-gapped. ![Scope and billing comparison: Azure SRE Agent manages Azure natively, reaches AWS only through an IAM-bounded MCP proxy, deploys in three Azure regions, and bills 4 always-on Azure Agent Units per hour with no free tier. Aurora is self-hosted, covers AWS, Azure, GCP, OVH, Scaleway, and Kubernetes natively in one dependency graph, and is free, open source Apache 2.0.](/blog/azure-sre-agent-vs-aurora-scope.svg) ## Azure SRE Agent vs Aurora: the head-to-head ### Scope: one cloud natively, others by bolt-on Azure SRE Agent's evidence reach is Azure-shaped. Within Azure it is deep: VMs, App Service, Container Apps, AKS, Functions, databases, networking, and the Monitor stack. Outside Azure, reach arrives through MCP connectors, and the flagship example is [AWS with Azure SRE Agent](https://techcommunity.microsoft.com/blog/appsonazureblog/announcing-aws-with-azure-sre-agent-cross-cloud-investigation-using-the-brand-ne/4507413): a local proxy to the official AWS MCP server that exposes 23 tools and can call AWS APIs, bounded by the IAM policies you grant it, with write access as a separate opt-in permission. It works, but it is a connector to another cloud, not management of it; the agent's native constructs, permission levels, managed resource groups, and identity all apply to Azure resources only. Aurora inverts the model. Multi-cloud is the native case: discovery populates one dependency graph with services from AWS, Azure, GCP, OVH, Scaleway, and Kubernetes, and an investigation traverses all of them in a single reasoning loop. For teams whose incidents cross provider boundaries, that is the difference between an investigation and a handoff, as we cover in [multi-cloud incident management](/blog/multi-cloud-incident-management). ### Billing: an always-on meter vs no meter This is the most concrete difference, and Microsoft documents it clearly. Azure SRE Agent bills [two flows](https://learn.microsoft.com/en-us/azure/sre-agent/pricing-billing): an always-on flow of 4 AAUs per agent-hour, charged from agent creation until deletion, explicitly including time the agent is stopped, and an active flow metered per token since April 15, 2026. Azure's Retail Prices API lists the AAU at roughly $0.10 in East US, which puts the always-on flow alone at about $292 per agent per month before the agent does anything. Microsoft's own examples price a single incident investigation at about 35 AAUs and a full remediation at about 86 on Claude Opus 4.6, or roughly 12 and 30 on GPT 5.3 Codex. The [pricing docs are direct about the entry cost: there is no free tier](https://learn.microsoft.com/en-us/azure/sre-agent/pricing-billing), and if you set a monthly active-flow limit and hit it, the agent becomes unavailable for chat and actions until the next month while always-on charges continue. Aurora has no meter. It is [free and open source](https://github.com/Arvo-AI/aurora); you pay for the infrastructure it runs on and the LLM you point it at, and a local Ollama model makes inference cost effectively fixed. For the deployment-control argument in full, see [self-hosted AI SRE](/blog/self-hosted-ai-sre). ### Autonomy and guardrails: read the defaults Azure SRE Agent's approval story is better than most, and more nuanced than its marketing summary. [Review mode, where the agent proposes and a human approves, is the default at the agent level. But incident response plans and scheduled tasks default to Autonomous](https://learn.microsoft.com/en-us/azure/sre-agent/run-modes), and approval gates apply to Azure infrastructure operations specifically. That is a reasonable design; it is also exactly the kind of default an evaluator should read the docs on rather than assume. Aurora's equivalent posture is stricter where it matters: gated actions in background and scheduled runs are denied outright rather than executed autonomously, every command passes deterministic policy and signature checks plus a fail-closed LLM judge before execution, and the pull-request path is structurally human-gated. The full architecture is in our [AI agent guardrails deep dive](/blog/ai-agent-guardrails). ### Deployment and auditability Azure SRE Agent is SaaS, [deployable in three Azure regions (Sweden Central, East US 2, Australia East) per Microsoft's FAQ at the time of writing](https://learn.microsoft.com/en-us/azure/sre-agent/faq), with expansion under way; it manages resources in any region, and the [chat interface supports English only](https://learn.microsoft.com/en-us/azure/sre-agent/overview). The reasoning system is closed. Aurora deploys via Docker Compose or Helm on any infrastructure, anywhere, air-gapped if needed, and every line of the agent that touches production is auditable on GitHub. ## Comparison table | Dimension | Azure SRE Agent | Aurora | | --- | --- | --- | | License | Proprietary, closed source | [Open source, Apache 2.0](https://github.com/Arvo-AI/aurora) | | Deployment | [SaaS, 3 Azure regions (July 2026)](https://learn.microsoft.com/en-us/azure/sre-agent/faq) | Self-hosted anywhere, air-gapped capable | | Native scope | [Full range of Azure services](https://learn.microsoft.com/en-us/azure/sre-agent/overview) | AWS, Azure, GCP, OVH, Scaleway, Kubernetes | | Other clouds | [AWS via MCP proxy, IAM-bounded](https://techcommunity.microsoft.com/blog/appsonazureblog/announcing-aws-with-azure-sre-agent-cross-cloud-investigation-using-the-brand-ne/4507413) | Native, one dependency graph across providers | | Pricing | [4 AAU/agent-hour always-on + token-based active flow, no free tier](https://learn.microsoft.com/en-us/azure/sre-agent/pricing-billing) | Free + infrastructure and LLM cost | | Autonomy defaults | [Review at agent level; incident plans and schedules default Autonomous](https://learn.microsoft.com/en-us/azure/sre-agent/run-modes) | Human-gated writes; background runs deny gated actions | | LLM choice | [Claude Opus 4.6 or GPT 5.3 Codex / GPT 5.2](https://learn.microsoft.com/en-us/azure/sre-agent/pricing-billing) | Bring-your-own-LLM, including local Ollama | | Guardrail transparency | Documented, closed implementation | [7 documented layers](/blog/ai-agent-guardrails), open implementation | ## What about AWS and Google's agents? Every hyperscaler now ships a cloud-locked operations agent, which is itself the strongest argument for a neutral layer. [AWS DevOps Agent went GA on March 31, 2026](https://aws.amazon.com/about-aws/whats-new/2026/03/aws-devops-agent-generally-available/), billed at [$0.0083 per agent-second with a 2-month free trial](https://aws.amazon.com/devops-agent/pricing/), and notably charges nothing while idle, a pointed contrast with Azure's always-on flow. Google's [Gemini Cloud Assist investigations](https://docs.cloud.google.com/gemini/docs/cloud-assist/investigations) remains in preview and analysis-only, and since April 10, 2026 it is restricted to Premium Support customers or teams granted access by their Google account team. Each agent is excellent inside its own cloud and a guest everywhere else. If your incidents live in one cloud, use that cloud's agent or a neutral one; if they cross clouds, the neutral layer is the only architecture that sees the whole failure. Our [15 best AI SRE tools roundup](/blog/top-ai-sre-tools-2026) covers the full landscape. ## Which should you choose? Choose Azure SRE Agent if your estate is all-Azure, you want first-party integration with Azure Monitor and the Azure control plane, and the per-agent economics work at your scale. It is a credible, well-documented product with a real safety model, and Microsoft's internal usage numbers are evidence it works. Choose Aurora if your infrastructure spans more than one cloud or includes significant Kubernetes and on-prem, if your incident data cannot leave your perimeter, if you want to audit the agent's guardrails in source rather than trust a closed implementation, or if an always-on per-agent meter does not fit how you budget. If you are weighing several options, our [AI SRE platform evaluation framework](/blog/how-to-evaluate-ai-sre-platform) walks through the criteria. The two also compose: teams running Azure SRE Agent for Azure-native toil can run Aurora as the cross-cloud investigation layer above it, since Azure is one of Aurora's native providers. You can read the code, deployment guides, and connector list in the [Aurora repository on GitHub](https://github.com/Arvo-AI/aurora). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) > **All claims sourced from Microsoft's official documentation, pricing pages, and launch blogs, the Azure Retail Prices API, AWS and Google official pages, and Aurora's public repository and docs.** Pricing and availability change; verify against the linked pages. Last verified: July 15, 2026. --- ## Best AI Incident Investigation Tools: 12 Cleric Alternatives URL: https://www.aurorasre.ai/blog/best-ai-powered-incident-investigation-tools Published: 2026-07-03 (updated 2026-10-21) Author: Arvo AI Team, Engineering > Cleric alternatives for AI incident investigation, checked October 21, 2026: Traversal, Resolve, Datadog and 9 more. Pricing, self-hosting, 3 tested. The alternatives to Cleric for AI incident investigation in October 2026 are Traversal, Resolve AI, Datadog Bits Investigation, incident.io Investigations, Rootly AI SRE, PagerDuty's Paige, Grafana Assistant Investigations, Dynatrace Intelligence and BigPanda among commercial products, and Aurora, HolmesGPT and K8sGPT in open source. All of them except K8sGPT start from an alert or incident and run their own queries; K8sGPT scans a cluster and explains what it finds. Only the three open source tools and Traversal (on-premises or in your own cloud account) run inside your own environment; with a local model, the open source tools need no outside service at all. Cleric publishes a price per investigated issue ($5); of the others, only Datadog and Dynatrace publish a per-investigation figure. We tested three of these hands-on: Aurora, HolmesGPT and K8sGPT, on the same 19 injected Kubernetes faults, two runs each. The other ten are closed SaaS products that we have not run; every claim about them below was checked on the vendor's own site or docs on 2026-10-21 and is linked. Arvo builds Aurora and wrote the 19 scenarios. If you are new to the category, the [AI-powered incident investigation explainer](/blog/ai-powered-incident-investigation) covers how these agents work; this page is the list. ## Which AI incident investigation tools are there, and which investigate rather than summarize? "Investigates" means the tool starts from an alert or incident and issues its own queries or tool calls to test a cause. "Tested" means we ran it on all 19 lab faults. Everything else was read from the vendor's site on 2026-10-21. | Tool | How we checked | What it does with an alert | Data it can reach | Self-host or your cloud | Published price | |---|---|---|---|---|---| | [Aurora](https://github.com/Arvo-AI/aurora) | **Tested hands-on** | Investigates; triggered by each alert | Kubernetes, AWS, Azure, GCP, alert sources such as PagerDuty, Datadog, Grafana, incident.io | Yes: Docker Compose, Helm, air-gapped bundle | Free (Apache 2.0) plus LLM tokens | | [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) | **Tested hands-on** | Investigates; CLI, server or operator mode | About 40 toolsets: Kubernetes, Prometheus, Grafana, Datadog, AWS, Azure, GCP and more | Yes | Free (Apache 2.0) plus LLM tokens | | [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) | **Tested hands-on** | Scans a cluster and explains analyzer findings; takes no alert | Kubernetes objects through its analyzers | Yes: CLI or in-cluster operator | Free (Apache 2.0) plus LLM tokens | | [Cleric](https://cleric.ai/) | Vendor site | Investigates; an issue starts from an alert, monitor or failed deploy | Datadog, Grafana, Prometheus, Kubernetes, AWS, GCP, GitHub, any MCP server | No; dedicated SaaS instance, US or EU region | Yes: from $700/month, $5 per investigated issue | | [Traversal](https://www.traversal.com/) | Vendor site | Investigates; on demand or auto-triggered | Datadog, Splunk, Dynatrace, Grafana stack, CloudWatch, Kubernetes, GitHub, GitLab, more via MCP | Yes: on-premises or bring your own cloud | No | | [Resolve AI](https://resolve.ai/) | Vendor site | Investigates alerts as they fire | 60+ integrations: observability, AWS, Azure, GCP, Kubernetes, GitHub, GitLab | No; shared or dedicated cloud, with an optional in-cluster proxy | No; credit-based | | [Datadog Bits Investigation](https://www.datadoghq.com/product/ai/bits-investigation/) | Vendor site | Investigates every alert | Datadog plus GitHub, Grafana, Dynatrace, Splunk, Sentry, ServiceNow | No | Yes: about 6.5 credits per investigation, credits from $500 per 500 | | [incident.io Investigations](https://incident.io/ai-sre) | Vendor site | Investigates incidents, optionally before anyone is paged | Datadog, Grafana (including Prometheus), Splunk, CloudWatch, Kubernetes, code, MCP connectors | No; SaaS hosted in the EU (Belgium) | Add-on to Pro and Enterprise, price not listed | | [Rootly AI SRE](https://rootly.com/ai-sre) | Vendor site | Investigates from the moment an alert fires | Rootly On-Call and Incident Response, code repos, Datadog, Sentry, Grafana, Prometheus | No; SaaS with an Edge Connector | No; "Let's talk" | | [PagerDuty Paige (SRE Agent)](https://docs.pagerduty.com/ai-automation/advance/sre-agent) | Vendor docs | Investigates with connector tools; auto-start is Early Access | Datadog, Dynatrace, Grafana, Splunk, CloudWatch, GitHub and more, by API or MCP | No; US or EU service region | Included on Reliability Platform plans (from $2,800 a year), metered in AI Actions | | [Grafana Assistant Investigations](https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/platform/investigation/) | Vendor docs | Investigates; manual, from an alert rule or from IRM webhooks | Metrics, logs, traces and profiles in Grafana Cloud | No; Grafana Cloud only | Yes: tokens and active AI users, no per-investigation fee | | [Dynatrace Intelligence](https://www.dynatrace.com/platform/artificial-intelligence/) | Vendor site | Deterministic root cause over its dependency model, plus agentic investigation | Dynatrace platform data (Smartscape, OneAgent, cloud integrations) | No; agentic AI is SaaS only, not on Dynatrace Managed | Yes: 30 AI units (about $3) average per autonomous investigation | | [BigPanda](https://www.bigpanda.io/) | Vendor site | Correlates alerts; an investigation agent swarm runs on major incidents | Datadog, Dynatrace, Grafana, New Relic, Prometheus, Splunk through its AI integrations, plus MCP tools | No; EU instance in Frankfurt | No dollar figures; credit plans | Two patterns stand out. The commercial products have converged on the same loop (hypothesis, query, evidence, ranked cause), so "does it investigate" no longer separates them; data reach and where the data goes do. And none of the ten commercial vendors publishes an accuracy benchmark that anyone else can rerun. Several publish customer figures, which are listed per tool below as the vendor's claims. ## What did we measure on the three tools we tested? We built a small app (`shop`: an nginx frontend, a Python api, a worker, redis and a load generator) on a kind v0.33.0 cluster running Kubernetes v1.37.0 with 3 nodes, and injected 19 known faults, from an OOMKilled worker to a NetworkPolicy that blocks redis to a ConfigMap edit that only one of two pods has loaded. Aurora's 38 runs were made on 2026-10-02; HolmesGPT's and K8sGPT's on 2026-10-15 and 2026-10-16, on the same cluster setup and the same faults. The full table, every trace and every miss are on [/benchmark](/benchmark) and in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab). | | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | Version tested | commit 87d22baf | 0.43.0 | 0.4.40 | | Model | Claude Sonnet 5 on Bedrock | Claude Sonnet 5 on Bedrock | Claude Sonnet 4.5 on Bedrock | | Input | alert text | alert text, `holmes ask`, default toolsets | no alert; `k8sgpt analyze --explain --namespace shop` | | Cluster access | the lab's admin kubeconfig, behind Aurora's `observability_only` command policy | the same admin kubeconfig, behind HolmesGPT's own command allow-list | the same admin kubeconfig; `analyze` only reads | | Correct | 34 of 38 | 31 of 38 | not scored as a peer | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 | | "No problems detected" | 0 | 0 | 12 runs (6 scenarios, both runs) | | Median time per run | 153.5 s | 45 s | 12.5 s for two analyze passes | | Median LLM cost per run | $0.35 | not captured | not captured | | Who confirmed the grades | a person at Arvo | judge model only, plus model reviews | judge model only, plus model reviews | How to read it: - **Grading.** A judge model (Claude Opus 5.5) graded every answer against the injected fault: correct means every required finding and none of the listed wrong answers. A person at Arvo read all 38 Aurora grades and changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model re-read all 76 and agreed with every one; a third pass graded all 76 from scratch, agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels (it called them no answer where the judge said partial or wrong). No person has checked those 76 yet. A blinded regrade with tool names removed matched the published verdict in 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs. - **Aurora against HolmesGPT is roughly even on accuracy.** The two-sided Fisher exact test that /benchmark computes on these totals gives p = 0.52, and it compares person-read Aurora grades with judge-only HolmesGPT grades. With 38 runs each, treat it as a tie. - **HolmesGPT is about 3x faster.** Its median run took 45 seconds against Aurora's 153.5. Aurora's times include contention from its own background jobs in the lab stack, described in the lab's REPORT.md, but that does not close the gap. - **K8sGPT does a different job.** It does not take an alert and investigate it; it runs analyzers over a namespace and has the LLM explain what they flag. Where the fault shows on an object, such as a mistyped image tag, it is fast and concrete. Where every object looks healthy (a NetworkPolicy block, a slow memory leak, a ConfigMap only one pod has loaded) it printed `No problems detected`. - **Prompts differ.** Aurora got the alert plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause."; HolmesGPT got the alert plus "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." HolmesGPT's own allow-list blocked `kubectl rollout history` in 12 of its runs, including 2 of its 7 non-correct runs, and we did not rerun those. Aurora's default policy blocked the same command in 26 of its 38 runs. - **Bias.** Arvo wrote the scenarios and builds Aurora. A kind cluster with one small app is not production. n = 2 per scenario does not measure variance. ### Where each tested tool went wrong Aurora's worst scenario was 12, two faults at once with an alert that covers only one. Both runs found the OOMKilled worker and its batch size change and never checked Services or endpoints. Run 1 then wrote: "Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]." Run 2 said the same in other words. The frontend Service was selecting no pods. That is a confident false statement, and the lab's model review found smaller unsupported claims (none in the root cause) in 15 of the 18 Aurora reports for scenarios 11 to 19. The [runbook for a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes) is built on these runs. HolmesGPT's one wrong answer was scenario 19 (run 1), a ConfigMap edit that only the recreated frontend pod had loaded. It blamed a startup race in the api pods and closed with "All data points to a transient startup race, now resolved." It fetched logs from only one of the two frontend pods, and its log lines show the old upstream, so that was the pod still serving normally. Its second run on the same fault was correct. The [runbook for a ConfigMap change not applied to all pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes) covers that fault. K8sGPT's clean scans came on the faults that need an investigation rather than an object check, for example the [NetworkPolicy blocking traffic](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes) scenario. ## Which tools show their work and link claims to evidence? All three tools we ran show their evidence, in different forms. Aurora's report numbers its claims and cites the tool outputs behind them (the `[5, 14]` in the quote above), and lists what it did not check. HolmesGPT prints each tool call as it runs (`Running tool #1 kubernetes_tabular_query`). K8sGPT prints each analyzer finding with its explanation. On the commercial side, the vendors say: - Cleric: "Every line of the fix links to the production data behind it" ([cleric.ai](https://cleric.ai/)), and "Every action logged, every investigation auditable." - incident.io: "every finding links back to where it came from: the Slack message, the pull request, the dashboard, the log line" ([trust and safety docs](https://docs.incident.io/investigations/trust-and-safety)). - Rootly: "Each finding comes with a confidence score and a visible reasoning chain" ([rootly.com/ai-sre](https://rootly.com/ai-sre)). - Grafana: you "review the Assistant's hypotheses, evidence, and report" ([docs](https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/platform/investigation/)). - BigPanda's investigation swarm output includes a key evidence and correlation section ([docs](https://docs.bigpanda.io/en/swarm-investigations--ia-.html)). Citations make a report checkable, not correct. Aurora's scenario 12 report above cited two tool outputs, both `kubectl get pods`, for a claim they could not support: the Service in front of those pods was broken. In a trial, open the cited evidence for a sample of claims rather than counting citations. ## What does an AI incident investigation cost per incident, including tokens? | Tool | What one investigation costs | Source | |---|---|---| | Aurora (measured) | Median $0.35 in LLM cost per run, range $0.19 to $0.84 over 38 lab runs; plus about $1 per cluster connection for its background discovery job | [lab results](https://github.com/Arvo-AI/aurora-lab) | | HolmesGPT, K8sGPT | Your LLM tokens; not captured in our lab | | | Cleric | 10 credits, $5 per investigated issue; plans from $700/month | [cleric.ai/pricing](https://cleric.ai/pricing) | | Datadog Bits Investigation | About 6.5 credits on average (Datadog's estimate); credits start at $500 per 500 a month billed annually, $600 per 500 month to month, or $1.30 each on demand, so roughly $6.50 to $8.50 per investigation | [datadoghq.com/pricing](https://www.datadoghq.com/pricing/?product=ai-credits) | | Dynatrace | 30 AI units average for an autonomous multi-step investigation at $0.10 per unit (about $3); not billed inside Dynatrace Assist "for a limited time", billed through workflows, MCP or the API | [dynatrace.com/pricing](https://www.dynatrace.com/pricing/) | | Grafana Assistant Investigations | Tokens, "no separate fee per investigation"; each active AI user includes 40M tokens a month, then $2 per 1M tokens | [Grafana docs](https://grafana.com/docs/grafana-cloud/platform/pricing-and-usage/assistant/) | | PagerDuty | Paige is included on Reliability Platform tiers, which include 1,000 (Starter), 4,000 (Essential), 16,000 (Plus) or 100,000 (Ultimate) AI Actions; no per-investigation figure | [pagerduty.com/pricing](https://www.pagerduty.com/pricing/) | | incident.io, Rootly, Resolve AI, Traversal, BigPanda | Not published | vendor pricing pages | Our Aurora figure is token cost on a small cluster with a short history. A production cluster returns longer tool outputs, so expect more tokens per run. The background discovery job cost about twice as much as the investigations in our lab ($29.70 for 30 completed discovery runs against $14.53 for the 38 graded investigations), which is the kind of line item a per-investigation price hides. ## Which AI investigation tools can self-host or keep data in the EU? - **Fully self-hosted:** Aurora (Docker Compose, Helm, an air-gapped bundle, local models through Ollama), HolmesGPT (CLI, Docker, Helm; Robusta also sells it "as SaaS, deploy into your own VPC, or fully self-host" per [robusta.dev](https://home.robusta.dev/)) and K8sGPT (CLI or operator). - **In your environment, commercial:** Traversal lists "Private, on-premises deployments", "Bring Your Own Cloud" and, separately, bring your own model with self-hosted or customer-managed models ([security](https://www.traversal.com/security)). Dynatrace Managed runs on-premises, but Dynatrace says its agentic and generative AI "is not available for Dynatrace Managed customers" ([FAQ](https://docs.dynatrace.com/docs/dynatrace-intelligence/agentic-and-generative-ai/agentic-and-generative-ai-faq)). - **EU region, SaaS:** Cleric offers US or EU hosting, and its docs say "EU hosting is not an EU-only data boundary" because some operational data is processed in the US ([docs](https://docs.cleric.ai/security/eu-hosting-data-processing)). incident.io hosts its data in GCP's Belgium region ([security FAQ](https://docs.incident.io/admin/security-faqs)). Resolve AI is SaaS; its optional Satellite runs in your cluster and proxies queries ([security](https://resolve.ai/security)). PagerDuty has an EU service region, though it says some data such as logs and billing may be stored outside it ([docs](https://docs.pagerduty.com/get-started/service-regions)). BigPanda runs EU processing in Frankfurt ([docs](https://docs.bigpanda.io/en/bigpanda-in-the-eu.html)). - **Cloud only:** Grafana's investigations are "Not available in self-managed deployments" ([docs](https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/get-started/self-managed/)). Rootly says plainly that it is SaaS and not the right fit if security requires fully on-premises software ([llms.txt](https://rootly.com/llms.txt)). ## Cleric, Traversal or Resolve AI: how do the standalone AI SREs differ? | | Cleric | Traversal | Resolve AI | |---|---|---|---| | Starts from | "an alert, monitor, or failed deploy"; can also watch a change for regressions for up to 14 days after deploy (2 credits per change) | "On-demand or auto-triggered" investigation ([llms.txt](https://www.traversal.com/llms.txt)) | Alerts "as they fire" | | Method, in the vendor's words | "forms hypotheses, queries connected production data to test them, and proposes a root cause" | "investigates thousands of hypotheses in parallel to isolate root cause" | "pursue multiple hypotheses in parallel, orchestrating specialized agents" | | Deployment | Dedicated SaaS instance, US or EU | On-premises or your own cloud account | Shared or single-tenant cloud, optional in-cluster proxy | | Models | Anthropic, Google Gemini and OpenAI, called by Cleric; no customer choice stated | Your LLMs, including self-hosted | Not stated | | Pricing | Published: $700, $1,500, $3,000 a month, $5 per issue | Contact sales | Credit-based, contact sales | | Vendor's own numbers | 58% of alerts closed without an engineer, May 20 to Oct 15, 2026, by Cleric's own "no action needed" verdict | 82% RCA accuracy, attributed to an executive at a Fortune 100 company | "5x faster MTTR", no source given | Sources: [cleric.ai/pricing](https://cleric.ai/pricing), [cleric.ai/technology](https://cleric.ai/technology), [Cleric security docs](https://docs.cleric.ai/security/security-data-privacy), [traversal.com](https://www.traversal.com/), [traversal.com/security](https://www.traversal.com/security), [docs.resolve.ai](https://docs.resolve.ai/about-resolve-ai), [resolve.ai/security](https://resolve.ai/security). Each vendor's numbers come from its own customers and its own grading; none can be rerun from outside. Traversal has published a [benchmarking method](https://www.traversal.com/blog/the-model-isnt-the-bottleneck-findings-from-benchmarking-on-high-severity-incidents-in-enterprise) on 25 private incidents, and Cleric has written about [its internal pairwise judge](https://cleric.ai/blog/measuring-an-sre-agent-without-ground-truth), which is more than most. We compared two of them in more depth in [Cleric vs Resolve AI](/blog/cleric-vs-resolve-ai). For "Traversal vs Datadog Bits AI", "Traversal vs BigPanda" and "Traversal vs Dynatrace": Datadog's agent investigates inside Datadog plus six listed third-party sources and is billed per credit; Dynatrace's works on data inside Dynatrace and only on its SaaS; Traversal reads many observability backends and can run in your own cloud; BigPanda is an alert correlation platform first, with an investigation swarm that can start from a major-incident template. ## The tools, one by one ### Aurora (tested hands-on) - **What it investigates:** every incoming alert from PagerDuty, Datadog, Grafana, New Relic, OpsGenie, incident.io and others starts a background investigation that runs read-only `kubectl` and cloud CLI commands and writes an RCA with cited evidence, a list of what it did not check, and suggested next steps. It can open a fix PR on GitHub for a human to approve; it never merges. - **Lab result:** 34 of 38 correct, 4 partial, median 153.5 s and $0.35 per run (person-read grades). Slower than HolmesGPT; missed the second fault in scenario 12 both times. - **Deploy:** Docker Compose, Helm, or an air-gapped bundle; models from OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, OpenRouter or local Ollama ([README](https://github.com/Arvo-AI/aurora)). Latest release v1.5.10, 2026-10-16. Apache 2.0. ### HolmesGPT (tested hands-on) - **What it investigates:** "Open-source AI agent for investigating production incidents and finding root causes" across Kubernetes, VMs, clouds and SaaS, with about 40 toolsets and alert integrations for AlertManager, PagerDuty, OpsGenie and Jira ([README](https://github.com/HolmesGPT/holmesgpt)). CNCF Sandbox project. Read-only by design, with an opt-in Kubernetes remediation toolset. - **Lab result:** 31 of 38 correct, 6 partial, 1 wrong, median 45 s (judge-model grades, not checked by a person). The fastest investigator we tested. - **Deploy:** CLI, Docker, Helm, Python SDK. Latest release 0.45.0, 2026-10-15 (we tested 0.43.0). Apache 2.0. ### K8sGPT (tested hands-on) - **What it does:** "a tool for scanning your Kubernetes clusters, diagnosing, and triaging issues in simple English" ([README](https://github.com/k8sgpt-ai/k8sgpt)). Analyzers find problems; `--explain` has an LLM explain them. It has an MCP server mode, and its operator has an alpha, opt-in auto-remediation for some image failures. - **Lab result:** not scored as a peer, because it takes no alert. 21 partial, 5 wrong, 12 runs with "No problems detected", median 12.5 s for two passes. - **Deploy:** CLI or in-cluster operator. Latest release v0.4.41, 2026-10-21 (we tested 0.4.40). Apache 2.0. ### Cleric Investigates issues that start from "an alert, monitor, or failed deploy" and verifies changes after deploy ([pricing](https://cleric.ai/pricing)). Integrates with Datadog, Grafana, Prometheus, New Relic, Splunk, Kubernetes, AWS, GCP, GitHub, PagerDuty, incident.io, Slack and any MCP server ([integrations](https://cleric.ai/integrations)). Read-only by default, though its GitHub and Jira integrations can open PRs and file issues. Dedicated SaaS instance per customer; on Pro and above, a [connector](https://docs.cleric.ai/integrations/private-resources) proxies access to private endpoints. Pricing is published: Team $700 a month, Pro $1,500, Scale $3,000, $0.50 per credit, $5 per investigated issue, 1,000 trial credits. ### Traversal "Traversal reasons causally. It builds a live model of how your systems actually behave, then investigates thousands of hypotheses in parallel to isolate root cause" ([traversal.com](https://www.traversal.com/)). Reads Datadog, Grafana, Prometheus, Splunk, Dynatrace, CloudWatch, Elasticsearch, Kubernetes, GitHub, GitLab and more ([integrations](https://www.traversal.com/platform/integrations)). The only commercial tool here that says it runs entirely in the customer's environment, with your own models. Pricing on request. ### Resolve AI "Automatically investigate alerts as they fire, gather evidence across your systems, and surface a root cause hypothesis before your on-call engineer even looks" ([docs](https://docs.resolve.ai/about-resolve-ai)). Lists 60+ integrations across observability, AWS, Azure, GCP, Kubernetes, code hosts, Slack, Teams, PagerDuty and coding agents. Shared or dedicated cloud deployment with bring-your-own keys ([security](https://resolve.ai/security)). Credit-based pricing, not published. Our [Resolve AI alternative](/blog/resolve-ai-alternative-open-source) page goes deeper. ### Datadog Bits Investigation "An autonomous AI agent that investigates production issues end to end. It iteratively forms hypotheses, gathers relevant telemetry, and uses data-based reasoning" ([docs](https://docs.datadoghq.com/bits_ai/bits_investigation/)). Beyond Datadog it reads GitHub, Grafana, Dynatrace, Splunk, Sentry and ServiceNow ([configure](https://docs.datadoghq.com/bits_ai/bits_investigation/configure/)). Not supported on Datadog's government sites. About 6.5 credits per investigation, from $500 per 500 credits. Datadog renamed the product from Bits AI SRE and claims it helps teams "restore services 90% faster" ([product page](https://www.datadoghq.com/product/ai/bits-investigation/)). See the [Datadog Bits AI SRE alternative](/blog/datadog-bits-ai-sre-alternative-open-source). ### incident.io Investigations "It forms a hypothesis and then tests it, reading your code and querying your dashboards to confirm or rule it out" ([docs](https://docs.incident.io/investigations/overview)). It runs on incidents rather than raw alerts: on every incident automatically if you choose, and an escalation step can [wait for it](https://docs.incident.io/on-call/wait-for-investigation) before paging a human. Connects to Datadog, Grafana, Splunk, AWS, Kubernetes and code ([telemetry](https://docs.incident.io/nexus/telemetry/overview)), and to other systems through MCP connectors. Data is hosted in Belgium. No bring-your-own model today. Investigations are an add-on on Pro and Enterprise with no listed price ([pricing](https://incident.io/pricing)). incident.io publishes [how it measures accuracy](https://docs.incident.io/investigations/measuring-accuracy) on each customer's own incidents. ### Rootly AI SRE It "starts investigating the moment an alert fires" and "runs parallel hypothesis checks across your alerts, telemetry, recent deployments, and past incidents" ([rootly.com/ai-sre](https://rootly.com/ai-sre)). It requires Rootly Incident Response; Incident Response and On-Call are $20 per user a month each and the AI SRE is "Let's talk" ([pricing](https://rootly.com/pricing)). Bring your own OpenAI, Anthropic or Gemini key on Enterprise. Never auto-remediates without human sign-off. ### PagerDuty Paige (SRE Agent) Paige uses connector tools "such as querying logs, reading a file, or listing pull requests" ([docs](https://docs.pagerduty.com/ai-automation/advance/connectors-tools-and-skills)); running it as a virtual responder that starts when an incident triggers is "Currently in Early Access" ([docs](https://docs.pagerduty.com/ai-automation/advance/sre-agent)). Connectors cover Datadog, Dynatrace, Grafana, New Relic, Splunk, Sentry, CloudWatch, Azure Monitor, GCP, GitHub and more. PagerDuty's docs say "PagerDuty AI is included with PD Reliability Platform plans"; the pricing page lists Paige as included on Starter and above and an add-on on Professional. A fit if PagerDuty is already your incident hub. ### Grafana Assistant Investigations For "incidents that span services or need analysis across metrics, logs, traces, and profiles" ([docs](https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/platform/investigation/)). Runs manually, from an investigation enrichment on an alert rule, or from IRM webhooks. Grafana Cloud only. Assistant bills on active AI users and tokens; investigations are billed only by the tokens they use, with metering that started on October 1, 2026 ([pricing docs](https://grafana.com/docs/grafana-cloud/platform/pricing-and-usage/assistant/)). Grafana also publishes [o11y-bench](https://github.com/grafana/o11y-bench), an open benchmark for LLM agents on observability tasks; it does not evaluate the Investigations product itself. ### Dynatrace Intelligence Root cause comes from a fault tree analysis over Dynatrace's dependency model: it "uses this information to quickly conduct a fault tree analysis to analyze millions of dependencies and arrive at the most probable root cause" ([docs](https://docs.dynatrace.com/docs/dynatrace-intelligence/root-cause-analysis/concepts)). The agentic layer runs autonomous multi-step investigations, priced at 30 AI units on average. All of it works on data inside Dynatrace, and the agentic part requires Dynatrace SaaS. See the [Dynatrace Davis alternative](/blog/dynatrace-davis-alternative-open-source). ### BigPanda Its core is alert correlation with change data. A "swarm of AI incident investigation agents" runs when a swarm-enabled major-incident template is used or on its own ([docs](https://docs.bigpanda.io/en/swarm-investigations--ia-.html)). It can query MCP servers you connect ([settings](https://docs.bigpanda.io/en/swarm-settings--ia-.html)), and BigPanda's AI integrations cover Datadog, Dynatrace, Grafana, New Relic, Prometheus (beta) and Splunk ([docs](https://docs.bigpanda.io/en/integrate-with-ai-incident-assistant-and-prevention.html)). Biggy AI is now called AI Incident Assistant. Credit plans start at 20,000 credits on one- to three-year terms, with no dollar figures published ([pricing](https://www.bigpanda.io/request-pricing/)). ## How do you run a fair trial of any AI incident investigation tool? Vendor demos run on the vendor's chosen incidents. The fix is a fault you injected yourself, with the answer written down before the tool runs. The lab we used is public and works for any tool that can reach a Kubernetes cluster: ``` git clone https://github.com/Arvo-AI/aurora-lab cd aurora-lab ./lab.sh up # 3-node kind cluster, its own kubeconfig in .kubeconfig ./lab.sh selftest # proves every fault appears as designed ./lab.sh break 13 # resets the app, injects scenario 13, waits until visible grep '^alert:' scenarios/13-networkpolicy-blocks-redis/scenario.yaml ``` Send only that alert line to the tool under trial. Then grade its answer against the `must_identify` and `wrong_answers` lists in the same `scenario.yaml`, which the tool must never see. `./lab.sh break ` resets the app before the next fault; `./lab.sh down` deletes the kind cluster and everything in it. The lab needs Docker with about 16 GB for the VM, kind, kubectl and Python 3. Its kubeconfig contains only the kind cluster, so it cannot touch other clusters on the machine. Rules we followed, and would hold any vendor to: 1. **Same faults, same alert text, same model where the tool allows it.** Write down every difference in the prompt. 2. **Run each fault at least twice.** HolmesGPT was wrong and then right on the same fault; one run would have hidden either result. 3. **Grade on everything the tool shows**, not only the summary. Our first Aurora report undercounted fixes because the grader read only the RCA text and not the suggested next steps. 4. **Include faults where the alerted object looks healthy:** a NetworkPolicy, a second unrelated fault, a ConfigMap only one pod loaded. Those separate investigators from scanners. 5. **Record time and cost per run,** including background jobs the tool runs on connect. 6. **Keep every run.** Publish the misses with the hits. A SaaS product running in the vendor's cloud may not reach a kind cluster on a laptop. In that case, deploy the lab app (`app/shop.yaml`) on a disposable cluster the vendor's trial can reach and run the same `break.sh` scripts there; each one is a short `kubectl` change against your current context, so check `kubectl config current-context` first and read the script before running it. Give the tool under trial the access you would give it in production, not the lab's admin kubeconfig, unless the cluster is disposable. To add a tool to the public table, open an issue with the "Benchmark: add an agent" template in the lab repo. For a single broken pod, the [Kubernetes pod diagnoser](/diagnose) takes pasted `kubectl` output and returns the most likely cause and the command that confirms it, in your browser. ## Which tool should you pick? - **You want an open, self-hosted investigator and can operate it:** Aurora or HolmesGPT. They were roughly even on accuracy in our lab; HolmesGPT was about 3x faster, Aurora triggers on alerts from your existing tools and drafts fix PRs for approval. The [open source SRE agents comparison](/blog/top-ai-sre-tools-2026) covers six more projects. - **Data cannot leave your network but you want a vendor:** Traversal is the only commercial investigator here that says it runs entirely in your environment. Dynatrace Managed runs on-premises, but Dynatrace's agentic AI does not run on it. - **Your telemetry is already in one platform:** that platform's agent (Datadog, Dynatrace, Grafana) sees the most with the least setup, and sees less outside it. - **You want an incident platform with investigation built in:** incident.io, Rootly or PagerDuty. - **You want a published price per investigation:** Cleric ($5), Dynatrace (about $3 average), Datadog (about 6.5 credits, roughly $6.50 at the annual rate). Whatever you shortlist, run the trial above before you sign. Aurora is on [GitHub](https://github.com/Arvo-AI/aurora) under Apache 2.0, or [book a demo](https://cal.com/arvo-ai/demo) to see it on your own alerts. *Vendor facts verified on each vendor's site or docs on 2026-10-21. Lab numbers from [results.json](https://github.com/Arvo-AI/aurora-lab/blob/main/benchmark/results.json), generated 2026-10-21. Methodology and every trace: [/benchmark](/benchmark) and the [lab report](/blog/aurora-lab-report-kubernetes-19-failures).* --- ## Rootly vs incident.io: Prices and AI SRE Checked Oct 2026 URL: https://www.aurorasre.ai/blog/incident-io-vs-rootly Published: 2026-06-24 (updated 2026-10-21) Author: Arvo AI Team, Engineering > Rootly vs incident.io, read off both sites on Oct 21, 2026: per-seat prices, AI SRE add-ons, on-call, status pages, integrations, and where each wins. incident.io is cheaper to start: it has a free Basic plan, and its Team plan is $15 per user a month billed annually, plus $10 for on-call. Rootly is $20 per user a month for Incident Response, which includes status pages and its AI scribe, and another $20 for On-Call. Neither publishes a price for its AI SRE agent: incident.io sells Investigations as an add-on on Pro and Enterprise, and Rootly sells AI SRE as a separate product that requires Incident Response. The clearest product difference is the trigger: Rootly's agent can investigate an alert the moment it arrives, while incident.io's investigates incidents, so an alert has to become an incident first. Every cell below was read on incident.io's and Rootly's own pricing pages and docs on 2026-10-21 and links to its source. We have not used either AI SRE agent: both are hosted, and Rootly's AI SRE is turned on by its account team. Arvo AI builds Aurora SRE, an open source investigation agent that already connects to incident.io; that is the one section about us, near the end. ## Rootly vs incident.io at a glance (checked 2026-10-21) | | incident.io | Rootly | |---|---|---| | Free plan | Basic, free: single-team on-call, 1 public status page, 2 integrations, 1 workflow ([pricing](https://incident.io/pricing)) | None. Two-week trial; companies under 25 employees "pay what you can" ([pricing](https://rootly.com/pricing)) | | Incident response, per user a month | Team $15 annual or $19 monthly; Pro $25; Enterprise custom ([pricing](https://incident.io/pricing)) | Incident Response $20; Enterprise custom ([pricing](https://rootly.com/pricing)) | | On-call, per user a month | +$10 on Team, +$20 on Pro, or $20 standalone ([pricing](https://incident.io/pricing)) | On-Call $20 standalone, or $40 together with Incident Response ([pricing FAQ](https://rootly.com/pricing)) | | AI SRE agent | Investigations: "Available as add-on" on Pro and Enterprise, no price listed ([pricing](https://incident.io/pricing)) | AI SRE: "Let's talk", requires Incident Response, included in Enterprise ([pricing](https://rootly.com/pricing)) | | AI in the base price | Suggestions, Scribe, AI post-mortem editor and the AI agent from Pro ($25); none on Team ([pricing](https://incident.io/pricing)) | @Rootly agent, AI Related Incidents, AI Scribe, AI Retrospectives in Incident Response ($20) ([pricing](https://rootly.com/pricing.md)) | | What starts an AI investigation | A declared incident: every one, by condition, or on demand; "Investigations run on incidents rather than on alerts directly" ([docs](https://docs.incident.io/investigations/triggering)) | An alert or an incident: on demand, from Slack, or an Auto-run rule matching new alerts ([docs](https://docs.rootly.com/ai/ai-sre/running-an-investigation)) | | Cloud and cluster data the agent queries | AWS (CloudWatch, EKS, OpenSearch, RDS), Google Cloud, Kubernetes, Datadog, Grafana, New Relic, Splunk and more, plus MCP connectors ([docs](https://docs.incident.io/nexus/telemetry/overview)) | AWS, Azure, Google Cloud, Cloudflare, Datadog, Grafana, New Relic, Splunk and more ([docs](https://docs.rootly.com/ai/connectors/overview)); EKS clusters through the AWS connector's optional EKS support ([docs](https://docs.rootly.com/ai/connectors/aws)); other Kubernetes clusters through Private Agent, an Early Preview for approved customers ([docs](https://docs.rootly.com/private-agent-kubernetes)) | | Changes the agent can make | A pull request you review and merge; unprompted fixes are off by default ([docs](https://docs.incident.io/investigations/trust-and-safety)) | Drafts remediation steps and PRs ([pricing](https://rootly.com/pricing)). Rootly's site says "any change requires explicit human sign-off" ([llms.txt](https://rootly.com/llms.txt)), but its AI SRE docs say "don't treat AI SRE as read-only": some connector tools can change data if the provider account allows it, and private-agent operations run without a confirmation step ([docs](https://docs.rootly.com/ai/ai-sre/overview)) | | Bring your own model key | No: "there's no bring-your-own-model or bring-your-own-key option today"; runs on OpenAI, Anthropic and Google Vertex under zero data retention ([docs](https://docs.incident.io/investigations/trust-and-safety)) | Enterprise only: OpenAI, Anthropic or Gemini keys ([pricing](https://rootly.com/pricing)) | | Chat tools | Slack, Microsoft Teams ([pricing](https://incident.io/pricing)) | Slack, Microsoft Teams, Google Chat ([pricing](https://rootly.com/pricing.md)) | | On-call schedules | 1 on Basic, 2 on Team, unlimited on Pro and Enterprise ([pricing](https://incident.io/pricing)) | 20 on On-Call, unlimited on Enterprise ([pricing](https://rootly.com/pricing.md)) | | Status pages | In every plan: 1 public page (unlimited on Enterprise), internal pages from Pro, customer pages on Enterprise ([pricing](https://incident.io/pricing)) | In Incident Response: 1 external and 1 internal page, custom domain, password or SAML login; Enterprise lists status pages without a number ([pricing](https://rootly.com/pricing.md)) | | Integrations | 71 on its integrations page, "over 40 alert sources"; 2 on Basic ([integrations](https://incident.io/integrations), [on-call](https://incident.io/on-call)) | "94 integrations across 10 categories" ([integrations](https://rootly.com/integrations)) | | API, webhooks, MCP | From Team; not on Basic ([pricing](https://incident.io/pricing)) | On every product ([pricing](https://rootly.com/pricing.md)) | | SCIM and audit logs | Enterprise only ([pricing](https://incident.io/pricing)) | Enterprise only ([pricing](https://rootly.com/pricing.md)) | | Hosting | SaaS; transactional data in Google Cloud's Belgium region ([security FAQ](https://docs.incident.io/admin/security-faqs)) | SaaS: "Rootly is SaaS" ([llms.txt](https://rootly.com/llms.txt)) | | Published accuracy of the AI agent | Grading method published; "we backtest against real historical incidents daily"; no figures published ([docs](https://docs.incident.io/investigations/measuring-accuracy), [AI SRE](https://incident.io/ai-sre)) | No figure for the product; Rootly AI Labs publishes SRE Skills Bench, which tests language models, not Rootly AI SRE ([AI Labs](https://rootly.com/ai-labs)) | ## Where incident.io is better - **Cost of entry.** A free plan exists, and a 10-person team on Team with on-call lists at $250 a month at the annual price (our arithmetic: 10 × ($15 + $10); the page does not say whether the on-call add-on is also discounted annually). Rootly has no free plan, and the same team on Incident Response and On-Call lists at $400 a month. - **Kubernetes and private telemetry for the agent.** Investigations reads any Kubernetes cluster you connect, by kubeconfig or discovered through AWS (EKS) or Google Cloud (GKE), read-only ([docs](https://docs.incident.io/nexus/telemetry/kubernetes)), and for sources inside your network you run a proxy and attach the source to it ([docs](https://docs.incident.io/nexus/telemetry/overview)). Rootly reaches EKS through its AWS connector, but other clusters and private-network sources go through Private Agent, which is an Early Preview "available only to approved customers". - **Accuracy method in the open.** incident.io documents how it grades each investigation against the cause your responders established, on a four-point scale (0%, 35%, 65%, 100%), and shows the scores in your account ([docs](https://docs.incident.io/investigations/measuring-accuracy)). That is the closest either vendor comes to a test a buyer can read. It is still your incidents and their grader, with no published totals. - **Status pages.** Every incident.io plan, including the free one, gets a public status page with a custom domain and unlimited subscribers ([pricing](https://incident.io/pricing)), and status pages are "not sold as a separate product" ([status pages FAQ](https://incident.io/status-pages)). Below Enterprise the pricing table caps public pages at one, even though the same FAQ says there are no limits on the "number of status pages"; ask before you plan on several. ## Where Rootly is better - **AI at the base price.** Rootly's $20 Incident Response seat includes the @Rootly agent, AI Scribe and AI Retrospectives. On incident.io, Scribe, Suggestions and the AI post-mortem editor start at Pro, $25. - **Investigating alerts, not just incidents.** An Auto-run rule can start Rootly AI SRE on a new alert that matches it, before anyone declares anything ([docs](https://docs.rootly.com/ai/ai-sre/running-an-investigation)). incident.io's escalation path can [wait for an investigation](https://docs.incident.io/on-call/wait-for-investigation), but only once an incident exists. - **Azure and Google Chat.** Rootly has Azure Monitor and Azure Resource Manager connectors for its AI and runs incidents in Google Chat. We found neither on incident.io's telemetry or pricing pages. - **Schedules and on-call extras on the base seat.** Rootly On-Call includes 20 schedules, the pay calculator and readiness reports at $20. incident.io's Team plan stops at 2 schedules, and the compensation calculator and readiness reports are Pro features. - **Your own model key and startup pricing.** Rootly lets Enterprise customers bring OpenAI, Anthropic or Gemini keys, and offers up to 50% off for startups under 100 employees, under $50M raised and under 5 years old. One thing to check before you connect Rootly to AWS: its AWS connector passes through the AWS MCP tool catalog without a read-only tool-name filter, and "with all services allowed, the IAM role's policies are the limit" ([docs](https://docs.rootly.com/ai/connectors/aws)). Keep the IAM role read-only, and treat other connectors the same way: Rootly's AI SRE docs say plainly not to treat AI SRE as read-only. ## Which incident management tools charge extra for AI features? Both, for the part that investigates. incident.io's Investigations (root cause analysis, the autonomous agent, Nexus data sources) are an add-on on Pro and Enterprise and are not available on Basic or Team. Rootly's AI SRE (probable root cause with evidence, alert-to-change correlation, drafted PRs) is a separate product on top of Incident Response, priced through sales or bundled into Enterprise. Rootly's pricing page lists "A separate add-on for every AI feature" under "What you won't find here"; in practice the drafting and summarizing AI is in the $20 seat and the investigating AI is not. You need a sales call to learn the AI SRE price from either vendor. ## Does Rootly automate root cause analysis during incidents? Yes, with AI SRE. Rootly's docs describe an evidence-first loop: read the alert or incident, measure the signal, draft competing explanations, query connected tools to confirm or rule out each one, challenge the leader, then write a verdict that cites evidence. The outcome can be "Root cause identified" (high, medium or low confidence), "Contributing factor identified", "Inconclusive" (no cause confirmed, a human is needed), "Blocked" (missing access) or "Could not investigate" ([docs](https://docs.rootly.com/ai/ai-sre/overview)). incident.io's Investigations work the same way at the level the docs describe: "It forms a hypothesis and then tests it, reading your code and querying your dashboards to confirm or rule it out" ([docs](https://docs.incident.io/investigations/overview)), and they use "an adversarial agent to challenge its own conclusions before sharing" ([AI SRE](https://incident.io/ai-sre)). ## How do you test either AI SRE before you buy? Neither vendor publishes a test set, so bring a known fault. Each scenario in our open lab, [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab), injects one Kubernetes failure and records the alert, the true root cause and the findings a correct answer must contain (`must_identify` in `scenario.yaml`). ``` git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab ./lab.sh up ./lab.sh break 07-service-selector-mismatch ``` Connect the cluster to the trial: in incident.io, a kubeconfig behind its proxy (a local lab cluster is not reachable from the internet); in Rootly, Private Agent if you are approved, or the AWS connector if you run the lab on EKS. Send the scenario's alert, declare an incident for incident.io, let the agent run, grade its answer against `must_identify`, then `./lab.sh reset`. Try [a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes) too (scenario 12): in our runs, both alert-driven agents we tested found the fault the alert named and missed the other one. For a single failure on your own cluster, there is [/diagnose](/diagnose). ## Where does Aurora SRE fit next to incident.io or Rootly? [Aurora](https://github.com/Arvo-AI/aurora) is an open source (Apache 2.0) investigation agent, not an on-call tool or a status page, so it sits behind either platform rather than replacing it. The [README](https://github.com/Arvo-AI/aurora#integrations) lists incident.io as an integration; Rootly is not listed, and Aurora has no Rootly connector today. With incident.io, we built and use the round trip. Aurora takes incident.io's `public_incident.incident_created_v2` and `public_alert.alert_created_v1` webhooks, investigates, and when the investigation completes posts the root cause back onto the same incident with `POST /v2/incident_updates`, keyed `aurora-rca-` so a retry cannot post twice ([PR #681](https://github.com/Arvo-AI/aurora/pull/681), merged 2026-10-02). For an alert-triggered run it posts onto the alert's incident, passing over declined, merged and canceled ones; when there is no such incident it leaves a note on the alert instead. Testing it on our own incident.io account, we found that incident.io renders markdown in updates but collapses single newlines, so the note is built for that. The agent can also read incident.io incidents and timelines while it investigates. With Rootly, point the same monitors (Datadog, Grafana, PagerDuty, New Relic and others) at Aurora directly; the result will not land on the Rootly incident. What we measured: on 19 injected Kubernetes faults, two runs each, Aurora was correct in 34 of 38 runs (median 153.5 s, median model cost $0.35), graded by a judge model and read by a person at Arvo. HolmesGPT 0.43.0 was correct in 31 of 38 and about 3x faster at the median (45 s); its grades are judge-model only. Same cluster and faults, but the prompts differed slightly (each tool got the alert plus a one-line instruction in its own wording), and Arvo wrote the scenarios and builds Aurora. Method and caveats are on [/benchmark](/benchmark) and in the [lab report](/blog/aurora-lab-report-kubernetes-19-failures). We have no comparable numbers for incident.io or Rootly, because neither publishes any. ## Related comparisons - [AI incident investigation tools compared](/blog/best-ai-powered-incident-investigation-tools), including incident.io Investigations and Rootly AI SRE - [incident.io alternatives](/blog/incident-io-alternative-open-source) and [Rootly alternatives](/blog/rootly-alternative-open-source-incident-management) - [FireHydrant vs incident.io](/blog/firehydrant-vs-incident-io) and [Rootly vs FireHydrant](/blog/rootly-vs-firehydrant) - [Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt) Verified 2026-10-21 on incident.io's pricing, AI SRE, on-call, status pages and integrations pages and docs, and on Rootly's pricing (including pricing.md), AI SRE, integrations and llms.txt pages and docs, as linked. Prices are list prices in USD per user a month. Both vendors change packaging often, so recheck the pricing pages before you sign. Aurora is on [GitHub](https://github.com/Arvo-AI/aurora), or [book a demo](https://cal.com/arvo-ai/demo) to see it post an RCA into incident.io. --- ## FireHydrant vs incident.io: 2026 Comparison URL: https://www.aurorasre.ai/blog/firehydrant-vs-incident-io Published: 2026-06-24 (updated 2026-07-02) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Freshworks acquired FireHydrant in early 2026; incident.io is independent. Compare AI, pricing, and where open-source Aurora fits for SRE teams. > **Key Takeaway:** FireHydrant and incident.io are both full incident management platforms covering response, on-call, and status pages. incident.io is independent, and its AI SRE investigates incidents. [Freshworks acquired FireHydrant](https://firehydrant.com/blog/firehydrant-to-be-acquired-by-freshworks/) (announced December 2025, closed January 2026), and FireHydrant's AI is documentation-focused and listed only on its Enterprise tier. incident.io includes AI SRE from its Pro plan. Neither is open source. [Aurora](https://github.com/Arvo-AI/aurora) is the open-source (Apache 2.0) investigation layer that complements either. ## FireHydrant vs incident.io at a glance | Capability | FireHydrant | incident.io | Aurora (open source) | |---|---|---|---| | Ownership | Part of Freshworks (acquisition closed Jan 2026) | Independent | Open source (Arvo AI) | | AI focus | Summaries, transcription, retrospectives (Enterprise only) | AI SRE: triage, RCA, drafted fixes | Autonomous cloud investigation | | Autonomous root cause analysis | No | Yes | Yes | | AI tier | Enterprise only | Included from Pro | Always (free) | | Direct cloud querying / CLI | No | Not documented | Yes, sandboxed Kubernetes pods | | On-call | Yes (Signals, consumption-based) | Yes, over 40 alert sources | No | | Status pages | Yes | Yes | No | | Pricing | $25/responder/month (Pro, annual); AI needs Enterprise | $15-$25/user/month + free Basic | Free (self-hosted) | | Open source / self-host | No | No | Yes, Apache 2.0 | ## What is FireHydrant? [FireHydrant](https://firehydrant.com) is an all-in-one incident management platform built to "Fight Fires Faster," and it says teams "resolve incidents up to 90% faster." Customers shown on its site include [Snyk, AuditBoard, Qlik, and Backblaze](https://firehydrant.com/customer-stories/). It is strong at coordination: automated runbooks, incident roles, a [service catalog](https://firehydrant.com) with dependency mapping, status pages, MTTx analytics, and an on-call product branded Signals with consumption-based alert pricing. [Freshworks acquired FireHydrant](https://firehydrant.com/blog/firehydrant-to-be-acquired-by-freshworks/) (NASDAQ: FRSH), [announced in December 2025](https://www.freshworks.com/press-releases/freshworks-to-deepen-its-it-service-and-operations-portfolio-with-acquisition-of-firehydrants-ai-native-incident-management-and-reliability-platform/) and closed in January 2026; FireHydrant is becoming the incident management and reliability layer inside Freshservice. ## What is incident.io? [incident.io](https://incident.io) is an "all-in-one incident management platform" for on-call, incident response, and status pages, with customers including [Netflix, Etsy, Intercom, and Vanta](https://incident.io) shown on its site. It spans Incident Response, On-Call with "[over 40 alert sources](https://incident.io/on-call)," [AI SRE](https://incident.io/ai-sre), and Status Pages. It remains an independent company, and its AI is a first-class investigation product rather than a documentation add-on. ## AI: investigation vs documentation This is the sharpest difference between the two. **FireHydrant AI** ([listed under the Enterprise tier](https://firehydrant.com/pricing)) generates incident summaries and status-page updates, transcribes live calls on Zoom and Google Meet, produces AI-enhanced retrospectives, and drafts intelligent follow-ups. It is documentation-focused: it summarizes and transcribes what happened. It does not autonomously investigate root causes or query infrastructure, and on the live pricing page its AI features appear only on the Enterprise plan. **incident.io AI SRE** ([incident.io/ai-sre](https://incident.io/ai-sre)) will "triage and investigate your alerts, analyse root cause," connect "code changes, alerts, and past incidents" to explain what went wrong, answer questions through @incident in Slack, draft a fix, and draft a post-mortem. It is available from the Pro plan rather than locked to Enterprise. If your goal is AI that helps figure out why an incident happened, incident.io is the stronger fit. If your AI need is mainly summarizing and transcribing for the record, FireHydrant covers that on its Enterprise tier. ## Pricing **FireHydrant** ([firehydrant.com/pricing](https://firehydrant.com/pricing)): - Free tier: up to 10 responders - Free trial: 14-day trial of Pro, no credit card - Pro: $25/responder/month, billed annually - Enterprise: custom pricing (required for FireHydrant AI features) - Alerting (Signals) is consumption-based, priced by the number of alerts you send **incident.io** ([incident.io/pricing](https://incident.io/pricing)): - Basic: free - Team: $15/user/month billed annually ($19 monthly) - Pro: $25/user/month, AI-native post-mortems editor included - Enterprise: custom pricing - On-call add-on: +$10-$20/user/month, or $20/user/month standalone Both are per-seat in a similar range: FireHydrant Pro is $25 per responder per month billed annually, while incident.io's Team is $15 and Pro is $25 per user per month. The bigger difference is AI access. incident.io includes AI SRE from its Pro plan, while FireHydrant's AI features are listed only on the custom-priced Enterprise tier. incident.io also has a free Basic tier, and FireHydrant offers a free tier for up to 10 responders. ## The Freshworks factor Freshworks [announced its acquisition of FireHydrant](https://www.freshworks.com/press-releases/freshworks-to-deepen-its-it-service-and-operations-portfolio-with-acquisition-of-firehydrants-ai-native-incident-management-and-reliability-platform/) in December 2025 and closed the deal in January 2026. FireHydrant is becoming the incident management and reliability layer inside Freshservice. FireHydrant's CEO wrote that "[your FireHydrant account, pricing, support, and access stays exactly the same](https://firehydrant.com/blog/firehydrant-to-be-acquired-by-freshworks/)" during the transition. If you already use Freshworks or Freshservice, that integration is a plus. If you want a vendor whose roadmap is dedicated to incident management, incident.io is independent today, and Aurora is open source with no acquisition risk. ## Where Aurora fits (the open-source option) Both FireHydrant and incident.io are closed-source SaaS, and neither documents directly querying your cloud APIs or running CLI commands during an investigation. [Aurora](https://github.com/Arvo-AI/aurora) is the open-source (Apache 2.0) investigation layer that does: - LangGraph agents that dynamically select from 30+ investigation tools - Direct querying across AWS, Azure, GCP, OVH, Scaleway, Cloudflare, and Kubernetes - Runs `kubectl`, `aws`, `az`, and `gcloud` in [sandboxed Kubernetes pods](https://www.aurorasre.ai/blog/ai-agent-kubectl-safety) isolated with NetworkPolicy - Memgraph dependency graph for blast radius and a Weaviate knowledge base - Suggests a fix and can open a remediation pull request on GitHub, gated on human approval (no auto-merge) - Any LLM provider, including local models via Ollama for air-gapped use Run it alongside either platform: FireHydrant or incident.io coordinates the response, and Aurora investigates and posts a structured RCA. No AI is locked behind an Enterprise tier, because Aurora is free. ## When to choose each **Choose FireHydrant** if you want a strong service catalog with dependency mapping, status pages, consumption-based Signals alerting, and you are already in or moving toward the Freshworks and Freshservice ecosystem. **Choose incident.io** if you want AI that investigates (not just summarizes), AI available from the Pro plan rather than Enterprise-only, a free tier, published per-seat pricing, and an independent vendor. **Add Aurora** if you need open source or self-hosting, an agent that directly queries your cloud and Kubernetes, or you want to avoid gating AI behind an Enterprise contract. ## Related comparisons - [FireHydrant Alternative: Open Source AI Incident Management](/blog/firehydrant-alternative-open-source) - [incident.io Alternative: Open Source AI Incident Management](/blog/incident-io-alternative-open-source) - [incident.io vs Rootly](/blog/incident-io-vs-rootly) - [Rootly vs FireHydrant](/blog/rootly-vs-firehydrant) ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) > **All claims sourced from official websites.** FireHydrant data from [firehydrant.com](https://firehydrant.com). incident.io data from [incident.io](https://incident.io). Aurora data from [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora). Last verified: June 2026. --- ## Rootly vs FireHydrant: Which Is Better in 2026? URL: https://www.aurorasre.ai/blog/rootly-vs-firehydrant Published: 2026-06-24 (updated 2026-07-02) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Rootly stays independent and AI-native; FireHydrant is now part of Freshworks. Compare AI, pricing, and where open-source Aurora fits in 2026. > **Key Takeaway:** Rootly and FireHydrant are both incident management platforms with response, on-call, and analytics. Rootly is independent and AI-native, with a dedicated AI SRE product. [Freshworks acquired FireHydrant](https://firehydrant.com/blog/firehydrant-to-be-acquired-by-freshworks/) (announced December 2025, closed January 2026), and FireHydrant lists its AI only on the Enterprise tier. Neither is open source. [Aurora](https://github.com/Arvo-AI/aurora) is the open-source (Apache 2.0) investigation layer that complements either. ## Rootly vs FireHydrant at a glance | Capability | Rootly | FireHydrant | Aurora (open source) | |---|---|---|---| | Ownership | Independent | Part of Freshworks (acquisition closed Jan 2026) | Open source (Arvo AI) | | AI focus | AI SRE: confidence-scored RCA, suggested fixes, meeting bot | Summaries, transcription, retrospectives (Enterprise only) | Autonomous cloud investigation | | Autonomous root cause analysis | Yes | No | Yes | | AI tier | Separate AI SRE product | Enterprise only | Always (free) | | Direct cloud querying / CLI | Not documented | No | Yes, sandboxed Kubernetes pods | | On-call | Yes, full product | Yes (Signals, consumption-based) | No | | Pricing | $20/user/month per product; AI via sales | $25/responder/month (Pro, annual); AI needs Enterprise | Free (self-hosted) | | Open source / self-host | No (platform proprietary) | No | Yes, Apache 2.0 | ## What is Rootly? [Rootly](https://rootly.com) is an "AI-native incident management platform" used by [NVIDIA, Replit, Canva, and Cisco](https://rootly.com/customers). It is sold as three product lines: Incident Response, On-Call, and [AI SRE](https://rootly.com/ai-sre). Rootly's AI SRE generates confidence-scored root cause analysis, surfaces similar past incidents, provides "suggested fixes and next steps," runs an AI meeting bot for incident bridges, and shows "exactly how it reasoned before you decide what to do next." Rootly is an independent company. ## What is FireHydrant? [FireHydrant](https://firehydrant.com) is an all-in-one incident management platform with strong coordination features: automated runbooks, incident roles, a service catalog with dependency mapping, status pages, MTTx analytics, and the Signals on-call product with consumption-based alerting. [Freshworks acquired FireHydrant](https://firehydrant.com/blog/firehydrant-to-be-acquired-by-freshworks/), [announced in December 2025](https://www.freshworks.com/press-releases/freshworks-to-deepen-its-it-service-and-operations-portfolio-with-acquisition-of-firehydrants-ai-native-incident-management-and-reliability-platform/) and closed in January 2026, and it is becoming part of Freshservice. Its AI features are documentation-focused and listed only on the Enterprise tier. ## AI SRE: depth vs documentation Rootly treats AI as a core investigation product. Its AI SRE correlates alerts, produces confidence-scored RCA, provides suggested fixes and next steps, and shows its reasoning, plus an AI meeting bot and a [Slack and Teams chat assistant](https://rootly.com/ai-sre). FireHydrant's AI summarizes incident activity, transcribes Zoom and Google Meet calls, and produces retrospectives. It does not autonomously investigate root causes, and on the live pricing page its AI features appear only on the [Enterprise plan](https://firehydrant.com/pricing). For teams whose main AI goal is faster, deeper investigation, Rootly is the stronger fit and does not require the top tier to access AI. For teams that mainly want AI summaries and transcription on a coordination platform, FireHydrant's Enterprise AI covers that. ## Pricing **Rootly** ([rootly.com/pricing](https://rootly.com/pricing)): - Incident Response Essentials: $20/user/month - On-Call Essentials: $20/user/month - AI SRE: contact us (no published price) - Startup discount: up to 50% off for companies under 100 employees, under $50M raised, and under five years old - Free two-week trial **FireHydrant** ([firehydrant.com/pricing](https://firehydrant.com/pricing)): - Free tier: up to 10 responders - Free trial: 14-day trial of Pro, no credit card - Pro: $25/responder/month, billed annually - Enterprise: custom pricing (required for FireHydrant AI features) - Alerting (Signals) is consumption-based Both use per-seat pricing in a similar range (Rootly at $20 per user per month per product, FireHydrant Pro at $25 per responder per month billed annually). The AI story differs: Rootly prices its AI SRE through sales, and FireHydrant lists its AI only on the custom-priced Enterprise plan. Either way, unlocking full AI requires a sales conversation. ## Where Aurora fits (the open-source option) Rootly and FireHydrant are both closed-source SaaS. Neither is self-hostable, and neither documents directly querying your cloud APIs or running CLI commands during an investigation. [Aurora](https://github.com/Arvo-AI/aurora) is the open-source (Apache 2.0) investigation layer that does: - LangGraph agents selecting from 30+ investigation tools - Direct querying across AWS, Azure, GCP, OVH, Scaleway, Cloudflare, and Kubernetes - Runs `kubectl`, `aws`, `az`, and `gcloud` in [sandboxed Kubernetes pods](https://www.aurorasre.ai/blog/ai-agent-kubectl-safety) isolated with NetworkPolicy - Memgraph blast-radius graph and a Weaviate knowledge base - Suggests a fix and can open a remediation pull request on GitHub, gated on human approval (no auto-merge) - Any LLM provider, including local models via Ollama Aurora is not an on-call or status-page tool. Teams run it alongside Rootly or FireHydrant so the SaaS platform coordinates the response while Aurora investigates and posts a structured RCA. Write actions wait for a human. ## When to choose each **Choose Rootly** if you want an AI-native platform with a detailed AI SRE product, an independent vendor, and per-seat pricing for response and on-call. **Choose FireHydrant** if you want a strong service catalog with dependency mapping, status pages, consumption-based Signals alerting, and you are in or moving toward the Freshworks ecosystem. **Add Aurora** if you need open source or self-hosting, an agent that directly queries your cloud and Kubernetes, or you want autonomous investigation without per-seat or Enterprise-gated AI pricing. ## Related comparisons - [Rootly Alternative: Open Source, Self-Hosted, and Free](/blog/rootly-alternative-open-source-incident-management) - [FireHydrant Alternative: Open Source AI Incident Management](/blog/firehydrant-alternative-open-source) - [incident.io vs Rootly](/blog/incident-io-vs-rootly) - [FireHydrant vs incident.io](/blog/firehydrant-vs-incident-io) ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) > **All claims sourced from official websites.** Rootly data from [rootly.com](https://rootly.com). FireHydrant data from [firehydrant.com](https://firehydrant.com). Aurora data from [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora). Last verified: June 2026. --- ## Automated Incident Remediation: 2026 Guide URL: https://www.aurorasre.ai/blog/automated-incident-remediation Published: 2026-06-18 (updated 2026-08-31) Author: Noah Casarotto-Dinning, CEO at Arvo AI > What automated incident remediation means for SRE vs security, how the five-stage loop works, and where to keep a human on the execute button. > ## Key Takeaways > > - **Automated incident remediation is software that detects, diagnoses, and recovers from a failing system with minimal manual steps.** In SRE it means restart, rollback, scale, or open a fix PR; in security it means isolate, block, or contain. The two meanings get conflated constantly, and the safe design differs for each. > - **The honest 2026 position is: let AI investigate autonomously, then propose remediation, and keep a human on the execute button** for anything destructive. Every credible vendor (IBM Instana, incident.io, Rootly) gates remediation behind human approval ([IBM Instana](https://www.ibm.com/products/instana/incident-remediation), [Rootly](https://rootly.com/sre/ai-observability-2026-predictive-alerts-auto-remediation-577de)). > - **The whitespace is open source.** Nearly every remediation product is closed-source, so you cannot self-host it or audit the approval logic. [Aurora](https://github.com/Arvo-AI/aurora) is Apache 2.0 and self-hosted, so the gate that decides "apply or ask a human" is code you can read. > - **Aurora's remediation is human-in-the-loop by construction.** Agents investigate, run sandboxed diagnostic commands behind a four-layer safety gate, and prepare a fix as a reviewable suggestion or a draft pull request. Opening the PR and merging it are deliberate human actions, not autonomous ones. > - **Auto-remediation is not self-healing and not auto-merge.** Aurora does not silently apply infrastructure changes, does not auto-merge PRs, and does not forecast failures. It closes the gap between "the alert fired" and "a reviewed fix is ready," and a person clicks the last button. **Automated incident remediation is the practice of using software to automatically detect, diagnose, and recover from a failure with minimal manual intervention.** In site reliability engineering that usually means restarting a service, rolling back a bad deploy, scaling infrastructure, or opening a code-fix pull request. In security it means something different: isolating a host, blocking an IP, or containing a threat. Both halves of the internet use the same phrase, which is why a search for it returns IBM Instana and incident.io next to SentinelOne and ReliaQuest. This guide is about the SRE meaning, names where the security meaning diverges, and answers the question every team actually has in 2026: how much should an AI remediate before a human approves it? We build [Aurora](https://github.com/Arvo-AI/aurora), an open-source, self-hosted AI SRE, so treat this as a vendor-informed view. The framing below is one we will defend on the merits: AI should investigate autonomously and propose remediation; a human should approve execution of anything destructive. ## What is automated incident remediation? Automated incident remediation closes the loop between detection and recovery. A monitoring tool detects a bad condition, something diagnoses it, and a remediation action returns the system to a healthy state. The "automated" part is about removing manual steps from that loop, not about removing the human entirely. It helps to define it by what it is *not*: - It is **not auto-deploy.** Auto-deploy ships code on merge. Remediation is what happens after something breaks. - It is **not self-healing infrastructure.** A Kubernetes pod restart or an autoscaling event is a runtime affordance for steady-state failures. Remediation also covers change-driven failures: the bad deploy, the broken config push, the failed migration. We draw this line in detail in our [CI/CD auto-remediation complete guide](/blog/cicd-auto-remediation-complete-guide). - It is **not anomaly prediction.** Forecasting a failure before it happens is a separate, classical-statistics problem. Remediation is the response after a signal fires. We separate the two in [AI SRE vs AIOps](/blog/ai-sre-vs-aiops). The minimum viable definition: a transition from a degraded state back to a healthy state, triggered by automated detection, executed by an action that is either fully automated or human-approved, and logged for review. ## Automated incident remediation in SRE vs in security: two different meanings If your search results feel contradictory, this is why. The same two words describe two different jobs. | Dimension | SRE / observability meaning | Security / SOC meaning | |---|---|---| | What "remediation" means | Restart service, roll back deploy, scale infra, open a fix PR, widen a noisy alert | Isolate host, block IP, kill process, quarantine file, revoke credential | | Trigger | Reliability alert (error rate, latency, failed deploy) | Threat detection (malware, intrusion, phishing) | | Goal metric | MTTR / failed deployment recovery time | Mean time to contain, dwell time | | Representative pages | [IBM Instana](https://www.ibm.com/products/instana/incident-remediation), [incident.io](https://incident.io/blog/runbook-automation-tools-2026-the-complete-guide), [Rootly](https://rootly.com/sre/ai-observability-2026-predictive-alerts-auto-remediation-577de) | [ReliaQuest](https://reliaquest.com/cyber-knowledge/understanding-automated-incident-response/) | [ReliaQuest's definition](https://reliaquest.com/cyber-knowledge/understanding-automated-incident-response/) is the canonical security one: "the use of software and tools to automatically detect, investigate, and respond to security incidents without the need for manual intervention." [AlertMend](https://www.alertmend.io/blog/automated-incident-remediation-tools) takes the IT-ops angle: tools "designed to detect and address issues within IT systems with minimal human intervention." Both are correct for their vertical. The rest of this page is the SRE vertical: reliability incidents, not threats. ## How does automated incident remediation work? The five-stage loop Almost every remediation system, open or closed, is some version of the same five stages. The interesting question is which stages are automated and which keep a human. 1. **Detect.** A monitoring tool (Datadog, PagerDuty, Grafana, Sentry, CloudWatch) notices a bad condition and fires an alert. Detection lives upstream, in the monitoring tool, not in the remediation system. 2. **Triage.** Decide whether this alert is a new incident or part of one already open. This is where [Aurora's AlertCorrelator](https://github.com/Arvo-AI/aurora) groups a related alert into an existing incident using service-topology distance, time proximity, and text similarity, so you do not spin up a duplicate investigation. Note this is alert-to-incident dedup, not alert suppression: every alert is still stored. 3. **Investigate.** An agent gathers evidence (logs, metrics, cluster state, the changed code) and produces a structured root cause analysis. This is the stage that is genuinely safe to fully automate, because reading is non-destructive. See our [AI-powered incident investigation guide](/blog/ai-powered-incident-investigation). 4. **Propose remediation.** The system turns the RCA into a concrete action: a command to run, a config to change, or a code fix as a pull request. 5. **Execute.** The action runs. This is the stage where the safe answer is "human-approved for anything destructive," and where the closed-source vendors and the open-source agents actually agree. The boundary that matters is between stage 4 and stage 5. Investigation can and should be autonomous. Execution of a destructive change should be gated. ## Which remediation steps are safe to fully automate in 2026? Not all remediations carry the same blast radius. A defensible policy keeps the reversible, low-risk classes automatable and gates the rest. When the calculation that bounds that blast radius is itself wrong, an automated repair can over-scope into an outage, as Azure's West US incident showed in [a 2026 case study of blast radius in automated remediation](/blog/blast-radius-automated-remediation-azure-outage). | Remediation action | Typical risk | Safe to fully automate today? | |---|---|---| | Read logs, query metrics, traverse the dependency graph | None (read-only) | Yes | | Group a duplicate alert into an existing incident | Low | Yes | | Widen a noisy alert threshold, suppress a log line | Low, reversible | Behind a policy gate, with audit | | Restart a pod, scale within preset bounds | Low to medium | Behind a policy gate | | Open a code-fix pull request | Low (the PR is the review surface) | Yes to *open*; merge stays human | | Roll back a deploy | Medium | Often, when the rollback target is known-good | | Merge a PR, change RBAC, touch the data plane, change production routing | High, often irreversible | No, keep a human | The pattern every honest vendor lands on: the pull request is the natural human-review surface. The agent touches the repository, not production, and the existing code-review and merge gates do the rest. [HolmesGPT](https://github.com/robusta-dev/holmesgpt) is read-only by default and can open suggested-fix PRs only when its GitHub integration with write scopes is explicitly connected. [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) is primarily a scanner; its core analysis opens no PRs, and its operator has an off-by-default auto-remediation mode that applies fixes rather than serving them as a review surface. ## How do the major tools handle the approval gate? (Instana vs incident.io vs Rootly vs Aurora) The most useful comparison is not "who automates more" but "where each tool puts the human." Here is how the named players draw that line, by their own published descriptions. | Tool | Open source / self-host | Investigation | Remediation output | Where the human approves | |---|---|---|---|---| | [IBM Instana](https://www.ibm.com/products/instana/incident-remediation) | No | AI incident investigation | AI-authored remediation runbook (Bash/Ansible), exportable | Human reviews and exports the runbook before it runs | | [incident.io](https://incident.io/blog/runbook-automation-tools-2026-the-complete-guide) | No | Workflow-driven | Executable runbook workflows | Human approval gates on execution | | [Rootly](https://rootly.com/sre/ai-observability-2026-predictive-alerts-auto-remediation-577de) | No | AI observability | Auto-remediation with human-in-the-loop | Human in the loop on remediation actions | | [HolmesGPT](https://github.com/robusta-dev/holmesgpt) | Yes | ReAct agent, read-only default | Opens suggested-fix PRs only when the GitHub write integration is connected | GitHub write integration is an explicit opt-in | | [Aurora](https://github.com/Arvo-AI/aurora) | Yes (Apache 2.0, self-hosted) | Autonomous LangGraph RCA across clouds and Kubernetes | Reviewable fix suggestion, or a draft PR (GitHub, Bitbucket) | PR creation is a UI click; merge needs foreground approval | The headline: every one of them keeps a human on the execute button for anything destructive. The difference is that only the open-source tools let you read the code that enforces it. With a closed product you trust the vendor's prose. With Aurora you can open the gate function and check. ## How does an open-source agent do automated incident remediation safely? This is the part the closed vendors cannot show you. Here is what Aurora actually does, stage by stage, and where the human sits. Every claim below maps to source you can read in the [Aurora repository](https://github.com/Arvo-AI/aurora). 1. **An alert webhook fires, and Aurora investigates autonomously.** LangGraph-orchestrated agents query infrastructure across AWS, Azure, GCP, and Kubernetes and synthesize a structured root cause analysis with remediation recommendations. The investigation is autonomous; the remediation output is a recommendation, not an applied change. 2. **Diagnostic and remediation commands run in sandboxed pods behind a four-layer gate.** When an agent needs to run 'kubectl', 'aws', 'az', or 'gcloud', the command first passes a signature check against [SigmaHQ](https://github.com/SigmaHQ/sigma) rules, your organization's allow/deny policy, an LLM safety judge, and a session-taint check. In an autonomous background run, anything that trips the gate is denied outright. Risky actions can only be approved by an interactive human in the foreground. The threat model behind this is covered in our [AI agent kubectl safety guide](/blog/ai-agent-kubectl-safety). 3. **A code fix is prepared as a reviewable suggestion, not a silent commit.** The agent proposes an anchored edit, Aurora applies it server-side to the fetched file, validates it (rejecting whole-file rewrites and no-ops), and saves it as a "fix suggestion" in the Incidents UI. The agent does not open a pull request on its own. The PR-creation function is deliberately excluded from the agent's callable tools. 4. **Opening the pull request is a deliberate human action.** A person clicks "Create Pull Request" in the Incidents UI, behind a role-based 'incidents:write' permission, and Aurora opens the PR against GitHub or Bitbucket. Merging that PR is a destructive action that requires explicit foreground approval and is denied in any background run, so the agent cannot silently merge. 5. **Recurring and event-triggered remediation is the same pattern.** [Aurora Actions](/blog/aurora-actions-background-automations) let you write a remediation in plain English and run it manually, on an incident (created, after RCA, or resolved), or on a schedule down to every five minutes. Read-write Actions are instructed to open a PR rather than apply changes directly, and the only built-in Action that ships is Generate Postmortem. Everything else you author yourself. There is also an advisory layer: Aurora's change-gating feature posts a SAFE/RISKY review on incoming pull-request webhooks. It is read-only and, by its own footer, "advisory only and does not block merge." It complements a code reviewer rather than replacing one. Because Aurora is open source under Apache 2.0 and self-hosted, all of this runs inside your perimeter and the gate logic is auditable. That is the differentiator no closed product on the citation list offers. ## Should an AI auto-remediate incidents without a human? For most production changes in 2026, no, and the vendors quietly agree. [IBM Instana](https://www.ibm.com/products/instana/incident-remediation) has the AI author the remediation runbook but a human review and export it. [Rootly](https://rootly.com/sre/ai-observability-2026-predictive-alerts-auto-remediation-577de) keeps a human in the loop on remediation. [incident.io](https://incident.io/blog/runbook-automation-tools-2026-the-complete-guide) puts approval gates on execution. The reason is blast radius: an agent that auto-remediates can also auto-make-things-worse, for instance autoscaling a service whose errors actually come from a saturated downstream dependency. The defensible policy is graded, not binary: - **Always automate** the read-only stages: investigation, RCA, alert-to-incident grouping. - **Automate behind a policy gate** a small set of low-risk, reversible fix classes: widening a noisy alert threshold, suppressing a log line, a bounded restart or scale, with full audit logging. - **Always keep a human** on irreversible or high-risk changes: PR merge, RBAC, secrets, data-plane and production-routing changes. That is exactly the boundary Aurora encodes in code. It is also why the right mental model is "automated investigation, human-approved remediation," not "self-healing." For where this sits on a maturity curve, the [CI/CD auto-remediation complete guide](/blog/cicd-auto-remediation-complete-guide) maps it onto five levels (L0 manual through L4 policy-gated), and the broader category context is in our [AI SRE complete guide](/blog/ai-sre-complete-guide). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## Introducing Aurora Actions: Background SRE Agents URL: https://www.aurorasre.ai/blog/introducing-aurora-actions Published: 2026-06-16 (updated 2026-07-02) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Aurora Actions turn plain-English instructions into background agents that mute noisy alerts, scan repos, review PRs, and write postmortems. > ## Key Takeaways > > - **Aurora Actions** are background agent tasks that follow your instructions: you write what you want done in plain English and Aurora runs it as a background agent using your connected tools. > - You control three things per Action: when it runs (manually, on an incident, or on a schedule), what it can do (read-only or read-write), and the task itself, written as natural-language instructions. > - Read-write Actions can run commands, modify infrastructure through Terraform, and open pull requests, and they are instructed to propose changes as a PR rather than applying them directly so a human stays in the loop. > - One Action ships built-in: Generate Postmortem, which writes a structured postmortem when an incident resolves. Use cases like muting noisy Datadog alerts via Terraform, scanning a repo for security issues, and reviewing PRs for risk are ones you build yourself. > - [Aurora](https://github.com/Arvo-AI/aurora) is open source (Apache 2.0) and self-hosted, and Actions is available now. Today we're introducing **Aurora Actions**: background agent tasks that follow your instructions. Write what you want done in plain English, choose how and when it should run, and Aurora carries it out as a background agent using the tools you have already connected. Actions are deliberately open-ended. The same primitive that mutes a noisy alert can review a pull request, scan a repository for security issues, or write a postmortem. You describe the outcome, and the agent works out the steps. ## Three choices, then it runs Every Action comes down to three decisions: - **What it does.** You write natural-language instructions. Aurora receives them along with context about whatever triggered the run, like the incident details or alert data, and executes them with your connected tools. - **When it runs.** A trigger: manually, on an incident, or on a schedule. - **What it is allowed to do.** An execution mode: read-only or read-write. ## Pick how it triggers - **Manual only.** Runs when you trigger it from the Actions page or an incident. - **On incident.** Fires automatically when an incident comes in. You choose the moment: the instant it is created, after root-cause analysis completes, or once it is resolved. - **On schedule.** Runs on a recurring interval, down to every five minutes. ## Pick what it can touch - **Ask (read-only).** Aurora can investigate, query your tools, and read code, but it cannot change anything. Write access is filtered out. - **Agent (read-write).** Aurora can run commands, modify infrastructure through Terraform, and open pull requests. By default it is instructed to propose changes as a PR rather than applying them directly, so a person stays in the loop on anything that touches production. ## What you can build A few of the things an Action can do: - **Mute noisy Datadog alerts via Terraform.** Tell Aurora to find the Terraform config that defines a monitor in your repo, add a mute or downtime rule, and open a PR that explains why the alert was noisy. Aurora reads the alert, finds the config, makes the change, and opens the PR. - **Scan a repository for security issues.** Put it on a schedule in read-only mode and let it sweep your code on its own cadence. - **Review pull requests for risk** before they merge. The last two are not pre-built. You create them as Actions yourself, using Aurora's read-only access to your code and pull requests. One Action does ship built-in: - **Generate a postmortem automatically.** When an incident resolves, Aurora pulls the root-cause analysis and the relevant Slack context and writes a structured postmortem, with sections for timeline, root cause, impact, action items, and lessons learned. Results do not start from scratch each run. Every Action keeps a living document that it reads and updates on each pass, reconciling new findings against what is already resolved and anything you have marked as a false positive, so results build up over time instead of resetting. ## Built to act on production safely Handing an agent write access to infrastructure is not something to do casually, so the guardrails are part of the design: - Read-only is a first-class mode. If an Action only needs to look, it cannot touch anything. - Read-write Actions are instructed to open pull requests instead of applying changes directly, keeping a human review step on real changes. - Every run is logged and linked to a full session you can open and inspect. - Instructions pass through input safety checks before anything executes. - Access is scoped per organization with role-based permissions. ## Available now [Aurora](https://github.com/Arvo-AI/aurora) is open source and self-hosted, and Actions is available today in the Actions tab. Connect your tools, write your first instruction, and let it run. For a deeper look at how Actions work under the hood, including trigger internals, RBAC, and the safety practices we use ourselves, read our [Aurora Actions deep dive](/blog/aurora-actions-background-automations). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## Dynatrace Davis AI Alternatives: Open Source RCA (2026) URL: https://www.aurorasre.ai/blog/dynatrace-davis-alternative-open-source Published: 2026-06-03 (updated 2026-10-12) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Davis AI is locked to the Dynatrace platform and OneAgent. Aurora is a free, open source, self-hosted, multi-cloud alternative for AI root cause analysis. > ## Key Takeaways > > - **Davis AI** is Dynatrace's hypermodal engine that converges predictive, causal, and generative AI, with the company's CTO stating that 'only causal AI can deterministically know the root cause of an issue' ([Dynatrace press release, July 25 2023](https://www.dynatrace.com/news/press-release/dynatrace-expanding-davis-hypermodal-ai/)). > - Davis derives its precision from the [Smartscape real-time dependency graph](https://www.dynatrace.com/platform/artificial-intelligence/), which is built from telemetry collected by Dynatrace's proprietary [OneAgent](https://www.dynatrace.com/platform/oneagent/), so its causal analysis is bound to the Dynatrace platform. > - Dynatrace prices on the consumption-based [Dynatrace Platform Subscription](https://www.dynatrace.com/pricing/rate-card/) (DPS): Full-Stack Monitoring at 0.01 USD per memory-GiB-hour, Infrastructure Monitoring at 0.04 USD per host-hour, and Kubernetes Platform Monitoring at 0.002 USD per pod-hour. > - [Aurora](https://github.com/Arvo-AI/aurora) is an open source (Apache 2.0) AI SRE platform whose LangGraph-orchestrated agents actively investigate incidents across AWS, Azure, GCP, OVH, Scaleway, and Kubernetes, with no OneAgent and no platform lock-in. > - Aurora is self-hosted and air-gapped capable with bring-your-own-LLM via Ollama, and it executes 'kubectl', 'aws', 'az', and 'gcloud' commands in sandboxed Kubernetes pods rather than only diagnosing ([Aurora repo](https://github.com/Arvo-AI/aurora)). > - Honest read: Davis wins on automatic baselining and topology at huge telemetry scale; Aurora wins on openness, multi-cloud without OneAgent, self-hosting, and active investigation plus human-gated remediation. If you want the causal-AI rigor of Dynatrace Davis without the proprietary agent, the per-GiB-hour bill, and the single-vendor platform, this guide compares Davis with Aurora, an open source AI SRE agent, and explains exactly where each one is the better fit. ## What is Dynatrace Davis? Dynatrace Davis is the AI engine inside the Dynatrace observability platform, and it is best understood as a deterministic causal-AI reasoning layer rather than a standalone tool. Dynatrace describes Davis as a hypermodal AI that converges predictive AI, causal AI, and generative AI, with CTO Bernd Greifeneder stating that 'only causal AI can deterministically know the root cause of an issue' ([Dynatrace, July 25 2023](https://www.dynatrace.com/news/press-release/dynatrace-expanding-davis-hypermodal-ai/)). Davis applies 'deterministic, causation-based analysis across applications, services, infrastructure, logs, and traces,' and it 'establishes baselines, detects anomalies, and identifies root cause with topology and dependency context' ([Dynatrace docs](https://docs.dynatrace.com/docs/discover-dynatrace/platform/davis-ai)), using the [Smartscape real-time dependency graph](https://www.dynatrace.com/platform/artificial-intelligence/). That topology is not free: it is constructed from telemetry gathered by [OneAgent](https://www.dynatrace.com/platform/oneagent/), Dynatrace's proprietary single binary that auto-instruments hosts, processes, and services and that you install once per host. The generative layer arrived with Davis CoPilot, which reached [general availability on October 10 2024](https://www.dynatrace.com/news/blog/announcing-general-availability-of-davis-copilot-your-new-ai-assistant/) and turns natural language into Dynatrace Query Language. Dynatrace, Inc. trades on the NYSE under the ticker DT after its [2019 IPO](https://www.sec.gov/Archives/edgar/data/0001773383/000119312519210943/d668054d424b4.htm) and is headquartered in Waltham, Massachusetts. OneAgent, Smartscape, PurePath, and Davis are all registered Dynatrace trademarks, which is a useful reminder of how tightly the causal engine is coupled to the proprietary stack underneath it. ## What is Aurora? Aurora is an open source, Apache 2.0 licensed AI SRE and incident-management platform from Arvo AI that actively investigates incidents instead of only correlating signals. When an alert fires, Aurora's [LangGraph-orchestrated agents](https://github.com/Arvo-AI/aurora) autonomously query your infrastructure, run commands, build a dependency graph for blast radius, and produce a structured root-cause analysis with remediation steps. Concretely, Aurora queries cloud APIs across AWS, Azure, GCP, OVH, Scaleway, and Kubernetes, and it executes 'kubectl', 'aws', 'az', and 'gcloud' commands inside sandboxed Kubernetes pods. It builds a Memgraph infrastructure knowledge graph to reason about blast radius, generates postmortems you can export to Confluence, Notion, or SharePoint, and can suggest code fixes or open pull requests. Aurora is self-hosted via Docker Compose or a Helm chart, is air-gapped capable, and is bring-your-own-LLM, so you can run local models through Ollama rather than sending telemetry to a vendor cloud. Destructive actions are human-gated. For deeper background on this active-investigation pattern, see our explainer on [AI SRE versus AIOps](/blog/ai-sre-vs-aiops). Aurora ingests alerts by webhook from eleven monitoring connectors: PagerDuty, Datadog, Grafana, New Relic, OpsGenie, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, and incident.io, plus a Slack bot. The Dynatrace connector matters here: you can keep Davis as your detection and baselining layer and forward its problems to Aurora for open, multi-cloud investigation and remediation. ## Davis vs Aurora: a direct head-to-head The cleanest way to frame this is detection and topology versus open investigation and execution. Davis is a deterministic causal engine optimized to pinpoint root cause within a single richly instrumented platform, while Aurora is a vendor-neutral agent optimized to investigate and act across many clouds without that instrumentation. **Where Davis is genuinely stronger.** Davis performs automatic, auto-adaptive baselining across host, process, service, and custom metrics with no manual thresholds, and it correlates events that share a root cause into a single problem to cut alert spam ([Dynatrace docs](https://docs.dynatrace.com/docs/discover-dynatrace/platform/davis-ai)). Because Smartscape continuously maps vertical and horizontal dependencies from OneAgent and PurePath traces, Davis can reason about causality at very large telemetry scale with a determinism that an LLM-driven agent does not match. If you already run OneAgent everywhere and want hands-off anomaly detection plus precise topology, Davis is excellent at that job. **Where Aurora is genuinely stronger.** Aurora is open source and self-hostable, so you can read the code, run it air-gapped, and avoid sending telemetry to a vendor. It investigates across AWS, Azure, GCP, OVH, Scaleway, and Kubernetes without requiring a proprietary agent on every host, which makes it a fit for [multi-cloud incident management](/blog/multi-cloud-incident-management) where deploying OneAgent fleet-wide is impractical or costly. Critically, Aurora executes: it runs cloud and kubectl commands in sandboxed pods and can open remediation PRs, where Davis primarily produces answers and hands remediation to external workflow actions. **Pricing model, stated honestly.** Dynatrace publishes list rates on the [DPS rate card](https://www.dynatrace.com/pricing/rate-card/): Full-Stack Monitoring at 0.01 USD per memory-GiB-hour, Infrastructure Monitoring at 0.04 USD per host-hour, and Kubernetes Platform Monitoring at 0.002 USD per pod-hour. There is no public per-investigation rate for Davis; it is bundled into consumption-based platform spend that scales with monitored memory and hosts. Aurora has no license fee at all under Apache 2.0; your only costs are the infrastructure you run it on and the LLM tokens you consume. ## Comparison table | Dimension | Dynatrace Davis | Aurora | |---|---|---| | License | Proprietary, part of the Dynatrace platform ([trademarks per S-1](https://www.sec.gov/Archives/edgar/data/0001773383/000119312519210943/d668054d424b4.htm)) | Open source, Apache 2.0 ([repo](https://github.com/Arvo-AI/aurora)) | | Deployment | SaaS platform, OneAgent installed per host ([OneAgent](https://www.dynatrace.com/platform/oneagent/)) | Self-hosted via Docker Compose or Helm, air-gapped capable | | Multi-cloud | Strong, but via OneAgent telemetry and Smartscape topology | AWS, Azure, GCP, OVH, Scaleway, Kubernetes with no proprietary agent | | Investigation vs correlation | Deterministic causal root cause on a continuously baselined topology ([docs](https://docs.dynatrace.com/docs/discover-dynatrace/platform/davis-ai)) | LangGraph agents actively query infra and reason over a Memgraph graph | | Write and execute actions | Answer-driven, remediation via external workflow actions | Runs kubectl, aws, az, gcloud in sandboxed pods, opens PRs, human-gated | | Pricing model | Consumption-based DPS, e.g. 0.01 USD per GiB-hour Full-Stack ([rate card](https://www.dynatrace.com/pricing/rate-card/)) | No license fee, you pay only for your infra and LLM tokens | | Self-host and air-gap | No, vendor SaaS platform | Yes, self-hosted with bring-your-own-LLM via Ollama | | LLM choice | Generative layer is Davis CoPilot, GA Oct 2024 ([blog](https://www.dynatrace.com/news/blog/announcing-general-availability-of-davis-copilot-your-new-ai-assistant/)) | Bring your own: OpenAI, Anthropic, Google, or local models | ## How Aurora fits alongside the tools you already run Aurora is the AI investigation layer, not a routing or scheduling layer, so it sits on top of whatever alerting and on-call stack you keep. If you use PagerDuty, Datadog, Grafana, Netdata, Dynatrace, Coroot, ThousandEyes, or BigPanda for detection and routing, Aurora ingests those alerts by webhook and takes the investigation from there. That layering is the point. You can keep Davis doing what it does best, namely automatic baselining and topology-aware detection inside Dynatrace, and forward its problems to Aurora so the actual investigation and remediation happen in an open, self-hosted, multi-cloud system you control. If you are evaluating where open agents fit in the broader market, our roundups of the [top AIOps platforms for free root cause analysis](/blog/top-10-aiops-platforms-free-root-cause-analysis-2026) and [self-hosted AI SRE options](/blog/self-hosted-ai-sre) put this trade-off in context. For Dynatrace next to LogicMonitor's Edwin AI and Datadog Bits on root-cause evidence, remediation and hybrid coverage, see [Edwin AI vs Dynatrace Davis AI vs Datadog Bits AI](/blog/edwin-ai-logicmonitor-alternatives). ## Which should you choose? Choose Dynatrace Davis if you are already standardized on the Dynatrace platform, can run OneAgent across your fleet, and want best-in-class automatic baselining plus deterministic, topology-aware root cause at large scale. Davis is a strong fit for organizations that value hands-off anomaly detection and accept consumption-based SaaS pricing in exchange for that depth. Choose Aurora if openness, self-hosting, and multi-cloud breadth without a proprietary agent matter more than turnkey baselining. Aurora is the better fit when you need to investigate across AWS, Azure, GCP, OVH, Scaleway, and Kubernetes, when you want an auditable Apache 2.0 codebase you can run air-gapped, when you want to bring your own LLM, and when you want the agent to execute human-gated remediation rather than only produce an answer. Many teams will not pick one and discard the other. The pragmatic pattern is Davis for detection and baselining inside Dynatrace, with Aurora as the open investigation and remediation layer on top, so you get causal precision where you have OneAgent and vendor-neutral action everywhere else. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## Datadog Bits AI Alternatives: Cost per Investigation (2026) URL: https://www.aurorasre.ai/blog/datadog-bits-ai-sre-alternative-open-source Published: 2026-06-03 (updated 2026-10-21) Author: Arvo AI Team, Engineering > Bits AI SRE, now Bits Investigation, lists ~6.5 AI credits per investigation: $6.50 to $8.45 by our math. Alternatives priced, 3 lab-tested. The main alternatives to Datadog Bits AI SRE are open source agents that read Datadog data but run outside it (Aurora, HolmesGPT) and other hosted AI SRE agents (Resolve AI, Cleric, NeuBird, Traversal). Bits, which Datadog now calls Bits Investigation, is billed in AI credits, and Datadog lists about 6.5 credits per investigation. Multiplied by Datadog's listed credit prices ($1 a credit at the "starting at" annual rate, $1.30 on demand), that is our estimate of about $6.50 to $8.45 per investigation. Teams leave it when much of their signal lives outside Datadog, when per-investigation billing grows with alert volume, or when the agent or the model has to run in their own network. In our lab, an open source agent's model cost per investigation was $0.19 to $0.84. Arvo builds Aurora and wrote the lab scenarios. Bits cannot be installed outside Datadog, so we have not tested it; every claim about it below links Datadog's own pages, read on 2026-10-21. ## What is Datadog Bits AI SRE? Bits AI SRE is Datadog's autonomous investigation agent. The [docs](https://docs.datadoghq.com/bits_ai/bits_investigation/) describe it as an agent that "iteratively forms hypotheses, gathers relevant telemetry, and uses data-based reasoning" to find root causes, and end each investigation with "a clear, evidence-backed conclusion" or mark it inconclusive ([investigate issues](https://docs.datadoghq.com/bits_ai/bits_investigation/investigate_issues/)). | Date | What happened | Source | |---|---|---| | June 10, 2025 | Announced at DASH 2025 in Limited Availability, with Bits AI Dev Agent and Bits AI Security Analyst | [Datadog press release](https://www.datadoghq.com/about/latest-news/press-releases/datadog-unveils-latest-ai-agents-to-rapidly-resolve-application-issues/) | | December 2, 2025 | Generally available, "our first generally available AI agent", tested "against more than 2,000 customer environments" | [Datadog press release](https://www.datadoghq.com/about/latest-news/press-releases/datadog-launches-bits-ai-sre-agent-to-resolve-incidents-faster/) | | March 5, 2026 | New agent harness: investigations "about 2 times faster than before", in "approximately 3-4 minutes", plus triage actions and remediation workflows | [Datadog blog](https://www.datadoghq.com/blog/bits-ai-sre-deeper-reasoning/) | | By October 2026 | Sold as Bits Investigation; the old docs URL redirects to the Bits Investigation docs | [docs](https://docs.datadoghq.com/bits_ai/bits_investigation/), [product page](https://www.datadoghq.com/product/ai/bits-investigation/) | What it reads and does, per Datadog: - **Triggers.** Datadog monitor alerts, manually or automatically when a monitor enters alert state, plus Synthetic tests, RUM and a free-text prompt. Supported monitors are metric, anomaly, forecast, integration, outlier, logs, APM metrics, composites, SLOs, Synthetics and RUM ([docs](https://docs.datadoghq.com/bits_ai/bits_investigation/investigate_issues/)). - **Data.** Datadog metrics, APM traces, logs, dashboards, events, Change Tracking, source code (GitHub only), Watchdog, RUM, Network Path, Database Monitoring and Continuous Profiler. Grafana, Dynatrace, Sentry, Splunk, ServiceNow and Confluence are listed too, but "Third-party integrations are in Preview for all customers." - **Actions.** Seven triage actions from chat, including Slack and Teams messages, paging through Datadog Incident Response, cases and Jira tickets; follow-up rollbacks and other remediation run through Datadog Workflows ([blog](https://www.datadoghq.com/blog/bits-ai-sre-deeper-reasoning/)). - **Controls.** RBAC and support for "HIPAA-regulated workloads" ([press release](https://www.datadoghq.com/about/latest-news/press-releases/datadog-launches-bits-ai-sre-agent-to-resolve-incidents-faster/)), and "zero data retention for third-party AI service providers" ([product page](https://www.datadoghq.com/product/ai/bits-investigation/)). Datadog describes an internal, replay-based evaluation platform in an [engineering post](https://www.datadoghq.com/blog/engineering/bits-ai-eval-platform/) and says the new agent is "more accurate on internal benchmarks". It does not publish the scores. ## How is Bits AI SRE priced? Bits is billed in [AI Credits](https://docs.datadoghq.com/account_management/billing/ai_credits/), shared with Bits Chat, Bits Code and Bits Agent Builder. From the [Datadog pricing page](https://www.datadoghq.com/pricing/?product=bits-ai-sre) on 2026-10-21: | Item | Datadog's figure | |---|---| | AI Credits | "Starting At $500 Per 500 credits, per month", billed annually, or $1.30 per credit on demand | | Bits Investigate, autonomous investigation | ~6.5 credits | | Bits Code, code fix | ~5 credits | | Bits Agent Builder, run | ~1 credit | | Bits Chat, message | ~0.5 credits | | Tiers | Annual Commit (12 months, bundles of 500 a month), Monthly Commit (bundles of 500 a month), On-Demand | | Unused credits | Reset on the first day of each month; unused Commit credits do not roll over | | Overage | Billed automatically at the On-Demand rate | Datadog adds that the averages are "provided only for reference" and that actual consumption "may vary with task complexity and the amount of context the model processes." Using its figures, our arithmetic: - One investigation costs about $6.50 at the "starting at" annual rate of $500 per 500 credits ($1 a credit) and about $8.45 on demand (6.5 x $1.30). Datadog does not show a Monthly Commit price, so that tier is not in the range. - The smallest annual commitment, 500 credits a month at the starting price, is $6,000 a year and covers about 76 investigations a month if nothing else uses the credits. - A team that auto-investigates 300 alerts a month uses about 1,950 credits: 4 bundles ($2,000 a month) on an annual commit, or about $2,535 a month on demand. This sits on top of the Datadog products the agent reads, since Bits investigates Datadog monitors and Datadog data. Admins can cap credits per organization or per user ([AI Credit Limits](https://docs.datadoghq.com/account_management/billing/ai_credit_limits/)), and when a cap is hit, users "can't use Bits Chat, Bits Investigation, Bits Code, or Bits Agent Builder" until the reset. AI Credit products are not available to FedRAMP customers ([docs](https://docs.datadoghq.com/account_management/billing/ai_credits/)). ## When would you leave Bits AI SRE? - **Your signal is not all in Datadog.** Bits' third-party integrations (Grafana, Dynatrace, Sentry, Splunk, ServiceNow, Confluence) are in Preview, and sources such as Loki, Prometheus or PagerDuty are not on its list. Its source code access is GitHub only. - **Alert volume drives cost.** Credits scale with every investigation, and automatic investigation on noisy monitors spends credits on noise. An open source agent's marginal cost is its model calls. - **The agent must run in your network.** Bits runs inside Datadog's SaaS, and FedRAMP customers cannot use it. Air-gapped or data-residency rules point to a self-hosted agent. - **You want to choose the model.** Bits runs on the third-party AI providers Datadog contracts with, under "zero data retention" terms. Aurora, HolmesGPT and K8sGPT take the provider you configure, including local models; Traversal advertises bring-your-own-model, and NeuBird lets you approve models, including ones in your own cloud account. - **You want one investigation layer across vendors.** Tying the agent to the observability vendor means changing both together. A separate agent can keep Datadog as one source among several. If your stack is all Datadog and the per-investigation cost is fine, Bits needs the least setup of anything here: it already has your telemetry, monitors and service tags. ## What does an investigation cost with an open source agent? We measured it. Aurora (commit 87d22baf) on `bedrock/us.anthropic.claude-sonnet-5` investigated 19 injected Kubernetes faults twice each. Its model cost per investigation: | | Aurora, 38 runs | |---|---| | Lowest | $0.19 | | Median | $0.35 | | Highest | $0.84 | | Total for 38 runs | $14.53 | | Median tool calls per run | 15 | What that excludes, and what we got wrong at first: - **Background discovery.** When a cluster is connected, Aurora runs a discovery job. In the lab it cost about $1 per cluster connection: $29.70 for 30 completed discovery runs, about twice the $14.53 of the 38 graded investigations ([REPORT.md](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md)). With one long-lived connection that is a one-off; in the lab each run used a fresh user and org, so discovery ran again and again. - **Hosting and people.** You run the containers and the database, and someone upgrades them. - **Other agents.** HolmesGPT's cost was not captured: its log says the model "has no entry in litellm's cost map", so it recorded $0. K8sGPT does not report cost. So the comparison is our estimate of about $6.50 to $8.45 in credits per Bits investigation against tens of cents of model calls plus your own operating cost. Which is cheaper depends on alert volume and on what an engineer's time costs you. ## Datadog Bits AI alternatives compared Checked against each vendor's site or repository on 2026-10-21. | Tool | Reads Datadog | Pricing | Where it runs | Model choice | Lab-tested | |---|---|---|---|---|---| | [Aurora](https://github.com/Arvo-AI/aurora) | Ingests Datadog alerts and investigates each one; also PagerDuty, Grafana, New Relic, OpsGenie, incident.io | Free, Apache 2.0; you pay for hosting and LLM calls | Docker Compose, Helm, air-gapped bundle | OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, OpenRouter, Ollama | Yes | | [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) | [Datadog toolsets](https://github.com/HolmesGPT/holmesgpt/blob/master/docs/data-sources/builtin-toolsets/datadog.md) for logs, metrics and more | Free, Apache 2.0; you pay for LLM calls | CLI, server, Kubernetes | OpenAI, Anthropic, Azure, Bedrock, Gemini and more | Yes | | [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) | No; scans Kubernetes objects | Free, Apache 2.0 | CLI, in-cluster operator | OpenAI, Azure, Bedrock, Gemini, LocalAI, Ollama and more | Yes, as a scanner | | [Resolve AI](https://resolve.ai/integrations) | Yes, plus 60+ integrations | Credit-based, [request pricing](https://resolve.ai/pricing) | Shared multi-tenant or dedicated single-tenant cloud, plus satellite | Resolve's own and frontier models | No | | [Cleric](https://cleric.ai/integrations) | "Reads logs, metrics, monitors, and dashboards" | Team $700 a month for 1,400 credits; "$5 to triage and address an alert" ([pricing](https://cleric.ai/pricing)) | Isolated Cleric instance per customer | Anthropic, Google Gemini and OpenAI enterprise APIs ([security](https://docs.cleric.ai/security/security-data-privacy)) | No | | [NeuBird](https://neubird.ai/pricing) | Yes | Investigation Agent 1 credit per run; no price per credit published | SaaS, or your VPC or VNET on Pro and Enterprise | Approved frontier models, including your own Bedrock, Azure OpenAI or Vertex AI ([model access](https://neubird.ai/platform/governed-model-access)) | No | | [Traversal](https://www.traversal.com/platform/integrations) | Yes | Not published; [pricing walkthrough](https://www.traversal.com/get-pricing) on request | Private, on-premises deployments; bring your own cloud | Bring your own, including self-hosted ([security](https://www.traversal.com/security)) | No | For a wider list, see [Resolve AI alternatives](/blog/resolve-ai-alternative-open-source) and [open source SRE agents compared](/blog/top-ai-sre-tools-2026). ## How did the open source alternatives do in our lab? The lab is a small app (nginx frontend, Python api, worker, redis, load generator) on a 3-node kind v0.33.0 cluster with Kubernetes v1.37.0. One fault is injected per scenario (two in scenario 12), two runs per scenario. Aurora and HolmesGPT get the alert; K8sGPT takes no alert and scans the namespace. All three got the same admin kubeconfig, each behind its own guard. Aurora's runs are from 2026-10-02; HolmesGPT 0.43.0 and K8sGPT 0.4.40 ran on 2026-10-15 and 2026-10-16, on the same cluster and faults. There is no Datadog in the lab: the agents read the cluster directly. | | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | Correct | 34 of 38 | 31 of 38 | not scored as a peer | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 | | "No problems detected" | 0 | 0 | 12 runs (6 scenarios) | | Median time per run | 153.5 s | 45 s | 12.5 s for two analyze passes | | Who confirmed the grades | a person at Arvo | judge model only | judge model only | A judge model (`us.anthropic.claude-opus-5-5`) graded every answer. A person at Arvo read all 38 Aurora grades and changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model re-read all 76 and agreed, a third pass graded them from scratch and agreed on all 38 HolmesGPT verdicts while disputing 6 K8sGPT labels, and no person has checked them. A blinded regrade, with tool names removed from the judge's prompt, matched the published verdict in 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs. The prompts differ in wording; HolmesGPT used `holmes ask` with default toolsets, and its allow-list blocked `kubectl rollout history` in 12 runs; Aurora's default policy blocked the same command in its runs. A two-sided Fisher exact test on the totals, computed on [/benchmark](/benchmark), gives p = 0.52 for Aurora against HolmesGPT, across human-read and judge-only grades: treat them as roughly even on accuracy. HolmesGPT is about 3x faster at the median. K8sGPT is a different tool. It runs analyzers over a namespace and explains what they flag, without an alert. In 12 runs it printed "No problems detected", on faults like a NetworkPolicy block or a slow memory leak where every object looks healthy. Use it as a scanner, not an investigator. ### The miss that matters most for alert-driven agents Bits, Aurora and HolmesGPT all start from an alert. Scenario 12 shows the weakness of that design: a worker OOMKilled fires the alert while, separately, the frontend Service selects no pods. Both agents found the OOM in both runs and neither reported the frontend outage. Aurora went further and stated the opposite: > Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]. Pod status was its only evidence. Whatever agent you choose, test it on a second fault the alert does not mention; the checks are in the runbook [a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). ## Which AI SRE tools let you bring your own LLM? Aurora, HolmesGPT and K8sGPT take the provider from your config and support local models (all three document Ollama), so the same incident workflow can switch between a hosted and a local model. Traversal's security page offers "self-hosted or customer-managed models", and NeuBird routes to models you approve, including Bedrock, Azure OpenAI and Vertex AI in your own account. Bits uses the AI providers Datadog contracts with, and Resolve combines its own models with frontier models. For air-gapped sites, Aurora ships an air-tight deployment bundle and runs on Ollama; see [self-hosted AI SRE](/blog/self-hosted-ai-sre). ## Traversal vs Datadog Bits AI Both are hosted investigation agents, but they sit differently. Bits is part of Datadog and reads Datadog first. [Traversal](https://www.traversal.com/) sells to enterprises, lists Datadog among its integrations, and offers private on-premises deployments, bring-your-own-cloud and bring-your-own-model on its [security page](https://www.traversal.com/security). Traversal publishes no price; Bits does. Neither has run our scenarios. ## How to test Bits or any alternative in a trial Datadog offers a 14-day trial of its product suite. Run the same faults through every candidate: each scenario in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab) has the alert, the true root cause and the findings a correct answer must contain. ``` git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab ./lab.sh up ./lab.sh break 12-two-faults-oom-and-selector ``` For Bits, the cluster must report to a Datadog trial org, since Bits works from Datadog data. To test it the way it runs in production, build a Datadog monitor for the alert yourself; Bits also takes a free-text prompt. Grade the answer against `must_identify` in the scenario's `scenario.yaml`, then `./lab.sh reset`. For a single failure on your own cluster, use [/diagnose](/diagnose). For Bits next to LogicMonitor's Edwin AI and Dynatrace, see [Edwin AI vs Dynatrace Davis AI vs Datadog Bits AI](/blog/edwin-ai-logicmonitor-alternatives). ## Which should you choose? | If you | Choose | Because | |---|---|---| | Run everything in Datadog and accept credit pricing | Bits Investigation | It already has your telemetry and monitors | | Keep Datadog but also use Grafana, PagerDuty, Splunk or several clouds | Aurora or HolmesGPT | Datadog becomes one source; they run where you deploy them | | Need it air-gapped or with your own model | Aurora, HolmesGPT, Traversal | Self-hosted or private deployment, model of your choice | | Want hosted with a published price | Cleric | $700 a month entry plan | | Want accuracy you can check yourself | Aurora, HolmesGPT | Open code, public lab runs | The lab is Kubernetes-only and synthetic, with faults we wrote, so it says nothing about Datadog-specific signals such as APM traces or RUM. HolmesGPT 0.44 and 0.45 were already out when we ran 0.43.0, and K8sGPT 0.4.41 shipped on 2026-10-21; those versions are untested. Verified 2026-10-21: Datadog pricing, docs, press releases and blog posts; Resolve AI, Cleric, NeuBird and Traversal pages; GitHub repos as linked. Lab numbers from [results.json](https://raw.githubusercontent.com/Arvo-AI/aurora-lab/main/benchmark/results.json), generated 2026-10-21. Aurora is on [GitHub](https://github.com/Arvo-AI/aurora) under Apache 2.0, or [book a demo](https://cal.com/arvo-ai/demo) to run it on your Datadog alerts. --- ## BigPanda Alternative: Open Source AIOps (2026) URL: https://www.aurorasre.ai/blog/bigpanda-alternative-open-source Published: 2026-06-03 (updated 2026-07-02) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Aurora is a free, open-source, self-hosted BigPanda alternative that investigates and verifies root cause instead of only grouping alerts. Full comparison. > ## Key Takeaways > > - **BigPanda is a closed, SaaS-only AIOps platform with no free or self-hosted tier.** It is a [SaaS-native platform](https://www.bigpanda.io/our-product/platform/) and publishes no pricing, routing buyers to a sales contact form. Aurora is [open source under Apache 2.0](https://github.com/Arvo-AI/aurora) and fully self-hosted. > - **BigPanda groups alerts. Aurora investigates and verifies the root cause.** BigPanda's [Open Box Machine Learning](https://www.bigpanda.io/our-product/platform/) correlates alerts, changes, and topology to [reduce alert noise by up to 95%](https://www.bigpanda.io/blog/black-box-machine-learning-ate-my-homework/). Aurora's [LangGraph agents](https://github.com/Arvo-AI/aurora) run live commands against your cloud and Kubernetes to gather new evidence, not just cluster existing alerts. > - **BigPanda is a well-funded enterprise vendor.** It raised a [190 million dollar Series D at a 1.2 billion dollar valuation in January 2022](https://www.bigpanda.io/press-release/190million-funding-leader-of-aiops/), co-led by Advent International and Insight Partners. Aurora is a free open-source project at [263 GitHub stars](https://github.com/Arvo-AI/aurora) and growing. > - **Open-source AIOps does correlation, not agentic investigation.** [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) does grouping and inhibition, and [Zabbix](https://www.zabbix.com/) does event correlation, but neither runs an LLM agent that investigates and executes. Aurora fills that gap. > - **Aurora ingests BigPanda itself.** BigPanda is one of [Aurora's monitoring connectors](https://github.com/Arvo-AI/aurora), so you can keep BigPanda for noise reduction and add Aurora as the investigation layer on top. > - **Aurora is air-gapped capable with bring-your-own-LLM.** It runs local models via [Ollama](https://github.com/Arvo-AI/aurora), so incident data and cloud credentials never leave your environment. If you are evaluating BigPanda, you are almost certainly drowning in alerts and want fewer, smarter incidents. BigPanda is genuinely strong at that. The honest question is whether grouping alerts is where your time actually goes, or whether the real cost is the hour your on-call engineer spends figuring out **why** the grouped incident happened. This guide draws the line between correlation and investigation, names BigPanda's real strengths, and shows where an open-source agent fits. Every factual claim links to a primary source. For the category-level distinction this post builds on, see our explainer on [AI SRE vs AIOps](/blog/ai-sre-vs-aiops). ## What is BigPanda? BigPanda is a closed-source, enterprise SaaS AIOps platform that correlates high-volume alert streams into a smaller number of actionable incidents. Founded in 2012 and headquartered in Redwood City, California ([Wikipedia](https://en.wikipedia.org/wiki/BigPanda)), it is one of the most established names in event correlation and noise reduction. The core engine is what BigPanda calls [Open Box Machine Learning](https://www.bigpanda.io/our-product/platform/): a deliberately transparent approach to ML correlation that lets operators read the automation logic in plain English, edit it, and preview it before it runs. That transparency is a real differentiator against opaque black-box AIOps. BigPanda claims its platform can [reduce alert noise by up to 95 percent](https://www.bigpanda.io/blog/black-box-machine-learning-ate-my-homework/) by capturing alerts, changes, and topology data from disparate tools and correlating them into high-level incidents in real time. BigPanda is a serious, well-capitalized company. It raised a [190 million dollar Series D at a 1.2 billion dollar valuation in January 2022](https://www.bigpanda.io/press-release/190million-funding-leader-of-aiops/), co-led by Advent International and Insight Partners, reaching unicorn status. The product is delivered as a [SaaS-native platform](https://www.bigpanda.io/our-product/platform/) with built-in scaling and high availability. There is no published per-incident or per-seat price: the [pricing page routes to a sales contact form](https://www.bigpanda.io/our-product/enterprise-cloud-platform/), and there is no free tier or self-hosted distribution you can download and run yourself. ## What is Aurora? Aurora is an open-source, self-hosted AI SRE platform that autonomously investigates incidents and can execute verified fixes, rather than only grouping alerts. Built by [Arvo AI](https://www.aurorasre.ai) and [licensed under Apache 2.0](https://github.com/Arvo-AI/aurora), Aurora uses LangGraph-orchestrated agents that pick from dozens of tools per incident. When an alert fires, Aurora's agents query infrastructure across AWS, Azure, GCP, OVH, Scaleway, and Kubernetes, run real commands like **kubectl**, **aws**, **az**, and **gcloud** inside sandboxed Kubernetes pods, and build a [Memgraph](https://memgraph.com/) infrastructure knowledge graph to trace blast radius. The output is a structured root-cause analysis and a postmortem you can export to Confluence, Notion, or SharePoint, plus suggested code fixes or an opened pull request. Destructive actions are human-gated, so the agent never executes a risky change without an operator approving it. Aurora ingests alerts via webhook from eleven monitoring connectors: PagerDuty, Datadog, Grafana, New Relic, OpsGenie, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, and incident.io, plus a Slack bot for conversational investigation. Because BigPanda is one of those connectors, the two are not strictly either-or. It is self-hosted and air-gapped capable, with bring-your-own-LLM support for local models via Ollama. ## Correlation groups alerts. Investigation finds the cause. The cleanest way to understand BigPanda versus Aurora is one sentence: **correlation groups the alerts you already have, while investigation gathers new evidence to explain them.** BigPanda's job ends at a well-formed incident. Its Open Box ML reads your existing alert stream, deduplicates it, and clusters symptoms that belong together, so a single network partition surfaces as one incident instead of three hundred pages. That is enormously valuable, and BigPanda does it at enterprise scale. But the grouped incident still arrives as a question: now that you know these forty alerts are related, **why** did they fire? Aurora starts where BigPanda stops. It treats the incident as a hypothesis to test. Its agents log into your cloud accounts and clusters, run live diagnostic commands, walk the dependency graph to find the blast radius, and assemble an evidence chain that points at a verified root cause. Then, with human approval, it can act on that conclusion: open a pull request with a fix, or run a remediation command in a sandboxed pod. Neither approach makes the other redundant. A team running thousands of daily alerts benefits from correlation upstream and investigation downstream. For the broader breakdown of why these are different lifecycle stages, see [AI SRE vs AIOps](/blog/ai-sre-vs-aiops) and our roundup of [free AIOps platforms with root cause analysis](/blog/top-10-aiops-platforms-free-root-cause-analysis-2026). ## BigPanda vs Aurora: head to head | Dimension | BigPanda | Aurora | |---|---|---| | **License** | Proprietary, closed source ([SaaS-native](https://www.bigpanda.io/our-product/platform/)) | [Apache 2.0, open source](https://github.com/Arvo-AI/aurora) | | **Deployment** | Vendor-hosted SaaS only, demo and sales-led | Self-hosted, air-gapped capable | | **Multi-cloud reach** | Ingests alerts from cloud and on-prem tools | Actively queries AWS, Azure, GCP, OVH, Scaleway, Kubernetes | | **Correlation vs investigation** | [Groups alerts into incidents](https://www.bigpanda.io/our-product/platform/) via Open Box ML | Investigates and verifies root cause with live evidence | | **Write and execute actions** | Surfaces context and suggested steps | Runs commands in sandboxed pods, opens PRs, human-gated | | **Pricing model** | No public price, custom enterprise quote via sales | Free software; you pay only for your own infra and LLM tokens | | **Self-host and air-gap** | Not available as a self-hosted product | Self-hosted with bring-your-own-LLM (Ollama) | BigPanda's genuine strengths are real: a mature correlation engine, transparent Open Box ML, [up to 95 percent noise reduction](https://www.bigpanda.io/blog/black-box-machine-learning-ate-my-homework/), and a deep catalog of enterprise integrations backed by a [unicorn-scale balance sheet](https://www.bigpanda.io/press-release/190million-funding-leader-of-aiops/). If a single vendor relationship and managed SaaS are what you want, those are points in BigPanda's favor. Aurora's strengths are orthogonal. It is the only side of this comparison that is open source, self-hosted, free, and capable of taking action rather than only describing context. For teams with data-sovereignty or air-gap requirements, see [self-hosted AI SRE](/blog/self-hosted-ai-sre). ## Where open-source AIOps stops The open-source ecosystem already does correlation. It does not do agentic investigation. That gap is exactly what Aurora is built for. [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) is the canonical example. It deduplicates, groups, and routes alerts, and it supports inhibition rules that suppress symptom alerts when a parent root-cause alert is firing. That is powerful, but it is rule-based and manually configured: you tell it which alerts inhibit which. It never logs into your cloud to check what actually broke. [Zabbix](https://www.zabbix.com/) similarly ships a built-in event correlation engine for servers, VMs, and network gear, but it does not run an LLM agent that gathers new evidence during an incident. So the practical landscape looks like this. BigPanda is the polished commercial correlation layer. Prometheus and Zabbix are the open-source correlation layer. None of them investigate. Aurora is the open-source investigation layer that sits on top of whichever correlation and routing stack you already run. It is worth being precise about a few other names buyers compare here. Moogsoft is not gone: it is now Dell APEX AIOps Incident Management, following [Dell's July 2023 acquisition](https://www.dell.com/en-us/dt/corporate/newsroom/announcements/detailpage.press-releases~usa~2023~07~dell-technologies-announces-intent-to-acquire-moogsoft.htm), and it is actively maintained. The honest tradeoff there is Dell-ownership lock-in, ProSupport-contract gating, and opaque enterprise pricing around aging patented correlation ML, not a shutdown. Routing is a separate concern again. [Grafana OnCall OSS was archived on 24 March 2026](https://grafana.com/docs/oncall/latest/set-up/open-source/), with Grafana pushing users toward Grafana Cloud IRM. OnCall is alert routing, scheduling, and escalation, not investigation, so it and Aurora are complementary, not substitutes. If you are migrating off OnCall, an open-source router like [Keep](https://github.com/keephq/keep), or notifications via ntfy or Twilio, can handle paging while Aurora handles the investigation on top. If you are weighing BigPanda against LogicMonitor's event intelligence, see [Edwin AI vs Dynatrace Davis AI vs Datadog Bits AI](/blog/edwin-ai-logicmonitor-alternatives). ## Which should you choose? Choose based on which problem actually costs your team the most time: grouping alerts, or explaining them. **Choose BigPanda if** your primary pain is alert-storm noise at enterprise scale, you want a single managed SaaS vendor with white-glove support, you have budget for an opaque enterprise contract, and a demo-and-sales procurement cycle is acceptable. BigPanda's correlation maturity and [transparent Open Box ML](https://www.bigpanda.io/our-product/platform/) are real reasons to pick it for that job. **Choose Aurora if** your bottleneck is the manual investigation that happens after an incident is formed, you need to run across multiple clouds rather than a single Kubernetes cluster, you require self-hosting or an air-gapped deployment, or you simply will not adopt a closed SaaS with no published price. Aurora is open source, free, vendor-neutral, and it executes verified fixes rather than only summarizing context. **Run both if** you already own BigPanda. Point BigPanda's correlated incidents at Aurora's BigPanda webhook connector, and let Aurora investigate each one automatically. You keep best-in-class noise reduction and add an investigation layer that costs only your own infrastructure and tokens. For multi-cloud estates specifically, see [multi-cloud incident management](/blog/multi-cloud-incident-management). ## Getting started with Aurora Aurora is on GitHub at [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) under Apache 2.0. Clone the repo, run the setup with the provided make targets, point your monitoring webhooks at it, add cloud credentials, and investigations begin automatically. Bring your own LLM, including local models for air-gapped environments, and keep every byte of incident data inside your own perimeter. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) > Every claim in this post is sourced from official vendor pages, primary press releases, and public repositories. BigPanda data from [bigpanda.io](https://www.bigpanda.io/). Aurora data from [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora). Last verified June 2026. --- ## Grafana OnCall OSS Archived: Alternatives, Fork, IRM Pricing URL: https://www.aurorasre.ai/blog/grafana-oncall-alternative-open-source Published: 2026-06-03 (updated 2026-10-21) Author: Arvo AI Team, Engineering > Grafana OnCall OSS was archived on 2026-03-24. Checked 2026-10-21: Grafana IRM pricing, the Appwrite fork, GoAlert, Keep, PagerDuty and incident.io. Grafana OnCall OSS is archived. It entered maintenance mode on 2025-03-11 and was archived on 2026-03-24, the date Grafana set for its Cloud Connection to stop working with self-hosted installs, which ends mobile push and Grafana-relayed SMS and phone calls ([Grafana](https://grafana.com/blog/grafana-oncall-maintenance-mode/)). Grafana recommends Grafana Cloud IRM instead: it is cloud-only and free for 3 active IRM users a month; on Pro, the $19 monthly Grafana Cloud platform fee includes 3 active IRM users and each further one starts at $20 ([pricing](https://grafana.com/pricing/)). If you need to stay self-hosted and open source, the realistic options are the Appwrite-maintained fork of OnCall and GoAlert, with ntfy for push and your own Twilio account for calls and SMS. Everything below was checked on 2026-10-21 against each project's repository or each vendor's own pages. Arvo AI builds Aurora SRE, which appears near the end. Aurora is not a pager: it does not schedule, page or escalate, so it is not on the list of OnCall replacements. It investigates the alert after the page, and that is the one part of this page we measured ourselves. ## What happened to Grafana OnCall OSS? | Date | Event | Source | |---|---|---| | 2025-03-11 | OnCall OSS enters maintenance mode: fixes only for critical bugs and "valid CVEs with a CVSS score of 7.0 or higher". Grafana Cloud IRM launches, merging OnCall (Cloud) and Incident | [Grafana blog, 2025-03-12](https://grafana.com/blog/grafana-oncall-maintenance-mode/) | | 2025-04-21 | The separate OnCall (Cloud) and Incident apps scheduled to be sunset in favour of the IRM app | same post, timeline | | 2026-02-11 | Last upstream release, v1.16.11 | [GitHub releases](https://github.com/grafana-cold-storage/oncall/releases) | | 2026-03-24 | OnCall OSS archived; "our APIs, Cloud Connection, and other services will stop working with OnCall (OSS) after 2026-03-24" | [Grafana blog](https://grafana.com/blog/grafana-oncall-maintenance-mode/), [README](https://github.com/grafana-cold-storage/oncall) | | 2026-06-05 | GitHub marks the repository "archived by the owner on Jun 5, 2026"; it now lives at grafana-cold-storage/oncall, and github.com/grafana/oncall redirects there | [GitHub](https://github.com/grafana-cold-storage/oncall) | Two dates float around for the archival. The README says "On 2025-03-11 Grafana OnCall (OSS) entered maintenance mode and was archived on 2026-03-24", and the last commit is from 2026-03-24. The read-only flag on GitHub was set on 2026-06-05. For planning, 2026-03-24 is the date that matters: it is the date Grafana gave for its APIs and Cloud Connection to stop serving OnCall OSS. The old OSS documentation page at grafana.com/docs/oncall/latest/set-up/open-source/ now redirects to the Grafana IRM docs, so links to it in older articles land on a different page. ## Can you still self-host Grafana OnCall OSS? Yes, but you own it. Grafana says "you will be able to use any current deployment of OnCall (OSS) indefinitely", and the code is still AGPLv3 (3,881 stars, 438 forks on 2026-10-21). What stopped on 2026-03-24: - **Mobile app push.** It went through Grafana's Cloud Connection. Grafana's migration guide says mobile push, SMS and phone notifications from Cloud Connection are not available to OSS users after archival ([migration guide](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/set-up/migrate/oncall-oss/)). - **Grafana-relayed SMS and phone calls.** Grafana's suggestion is to configure Twilio, Exotel or Zvonok yourself, or trigger your own call and SMS workflows with webhooks ([Grafana blog](https://grafana.com/blog/grafana-oncall-maintenance-mode/)). - **Security fixes.** The CVE fixes were a maintenance-mode promise for the year before archival. The last release is v1.16.11 from 2026-02-11. For push, Grafana merged outgoing webhooks in personal notification rules before archival, which "makes it possible to use third-party services like Pushover, Gotify, or ntfy to continue receiving push notifications" ([Grafana blog](https://grafana.com/blog/grafana-oncall-maintenance-mode/)). ### Is there a maintained Grafana OnCall fork? One fork of the GitHub repository shows steady work: [appwrite/grafana-oncall](https://github.com/appwrite/grafana-oncall), "Appwrite's fork of Grafana OnCall" (AGPL-3.0). Its README says Appwrite "maintains this fork for its own use". As of 2026-10-21: - Latest release v1.24.10 (2026-09-21), after more than 25 releases since mid-August 2026, and dependency updates merged since then. - It runs on Grafana 13.2 and React 19 ("Upstream's plugin crashes on mount there"), and on Django 5.2 LTS instead of upstream's 4.2. - It adds a Discord integration and SMS through MSG91 alongside Twilio, Zvonok and Exotel. - Signed container images, SBOMs and build provenance at `ghcr.io/appwrite/grafana-oncall`. Its README also lists what it does not offer: "No support and no roadmap. We keep it working for our own use." The plugin is unsigned, so Grafana needs `GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=grafana-oncall-app` (the setting takes plugin IDs, so list only this one, and pin the signed image by digest); the Helm chart is not published to a chart repository; and "The mobile app does not work. It relied on Grafana Cloud's push relay." The other direct forks of the repository had one star or fewer on 2026-10-21. Grafana said in 2025 that if the community forks OnCall, "we will support them with best reasonable effort." Our view: the Appwrite fork is the shortest path if you are on OnCall OSS today and your Grafana upgrade broke the plugin. It is one company's fork for its own use, so treat it as a bridge and keep your Terraform or API export current. ## Is Grafana IRM open source or self-hosted? No. "Grafana IRM is the on-call management and incident response product in Grafana Cloud", and the docs call it "a paid Grafana Cloud add-on, billed based on monthly active IRM users" ([docs](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/)). There is no self-managed or open source IRM. Grafana's only self-hosted on-call product was OnCall OSS, and that is the archived one. | | Free | Pro | Enterprise | |---|---|---|---| | IRM price | $0, "Limited to 3 active IRM users per month" | "Starts at $20 / active IRM user"; the $19 a month Grafana Cloud Pro platform fee includes "3 active IRM users per month then pay as you go" | Custom, "Minimum commit of $25,000 per year" | | Support | Community | 8x5 email | Premium | Source: [grafana.com/pricing](https://grafana.com/pricing/), 2026-10-21. The $19 is the one Pro platform fee for the whole Grafana Cloud stack (the same fee appears on every product's card), not an extra IRM charge. So a team of 3 on-call engineers on Pro pays $19 a month, and a team of 8 starts at $19 plus 5 x $20 = $119 a month before volume discounts. An active IRM user is "a user who is included in OnCall schedules or escalation chains" or who changed an alert group's status or OnCall configuration, received or sent a page, or created or edited an incident in the month. The IRM product page puts it as billing "for people who are on call, get notified, or take action that month" ([IRM](https://grafana.com/products/cloud/irm/)). IRM notifies through the mobile app, Slack, Microsoft Teams, Telegram, SMS, phone and email, and the same page says it "also covers incident coordination, post-incident review templates, and AI investigations through Grafana Assistant". **Migrating from OSS.** Grafana's guide offers Terraform or the OnCall API, and notes "The IRM migration tools don't currently support Grafana OnCall OSS as a source platform." Move resources in order: integrations, escalation chains, routes, then schedules ([migration guide](https://grafana.com/docs/grafana-cloud/observe-and-act/respond-to-incidents/set-up/migrate/oncall-oss/)). ## Should you consolidate on-call into Grafana IRM instead of a separate tool? Yes, if your alerts and dashboards already live in Grafana Cloud and you can send alert data to a SaaS. Alert rules, routing, schedules and incidents then share one stack, and you pay only for people who are actually on call. No, if you self-host Grafana for data-residency or air-gap reasons: IRM requires Grafana Cloud, and moving paging there means your alert payloads leave your network. In that case a self-hosted pager (GoAlert or the Appwrite fork) next to self-hosted Grafana keeps the boundary where it was. A third case: if you already pay for PagerDuty or incident.io, Grafana Alerting can keep sending to them, and adding IRM only makes sense if you plan to drop the other tool. ## What are the Grafana OnCall alternatives? Checked 2026-10-21. Prices are list prices from each vendor's pricing page. | Tool | What it does | Self-hosted | License or price | Status | |---|---|---|---|---| | [Grafana Cloud IRM](https://grafana.com/products/cloud/irm/) | Schedules, routing, escalation, paging, incidents | No | Free for 3 active users; Pro $19/month includes 3, then from $20 per active user | Grafana's recommended path | | [Appwrite OnCall fork](https://github.com/appwrite/grafana-oncall) | OnCall OSS, kept running on current Grafana | Yes | AGPL-3.0 | v1.24.10, 2026-09-21; no support, no roadmap | | [GoAlert](https://github.com/target/goalert) (Target) | "on-call scheduling, automated escalations and notifications (like SMS or voice calls)" | Yes, single binary plus Postgres | Apache-2.0 | v0.35.0, 2026-09-21; 2,851 stars | | [LinkedIn Oncall](https://github.com/linkedin/oncall) + [Iris](https://github.com/linkedin/iris) | Oncall is "a calendar tool" for shifts; Iris does paging | Yes | BSD-2-Clause | Last releases 2024-03-12 (Oncall) and 2021-11-16 (Iris) | | [Keep](https://github.com/keephq/keep) | Alert management, dedup, correlation, workflows | Yes | MIT outside its `ee/` directory | v0.54.3, 2026-09-09; no native on-call scheduling | | [ntfy](https://github.com/binwiederhier/ntfy) | Push notifications over HTTP | Yes | Apache-2.0 and GPLv2 | v2.32.0, 2026-10-18; a delivery channel, not a scheduler | | [PagerDuty](https://www.pagerduty.com/pricing/) | Schedules, escalation, paging, incident management | No | Free for 5 users; Professional $21/user/month annual, $25 monthly | Active | | [incident.io](https://incident.io/pricing) | On-call plus incident response | No | Team $15 (annual) or $19 per user/month, on-call +$10; Pro $25, on-call +$20; on-call alone $20 | Active | | [Rootly](https://rootly.com/pricing) | On-call and incident response | No | On-Call $20 and Incident Response $20 per user/month | Active | | Opsgenie | Paging | No | Sale ended 2025-06-04 | Shuts down 2027-04-05 ([Atlassian](https://www.atlassian.com/software/opsgenie/migration)) | Notes on the ones a self-hoster is most likely to pick: **GoAlert** is the closest open source match to what OnCall did. It sends "SMS and Voice using Twilio, email using Mailgun, or Slack", and takes alerts from its generic API, email, Grafana, Prometheus, Site24x7 or Sensu ([goalert.me](https://goalert.me/)). The only hard requirement is a Postgres database ([getting started](https://github.com/target/goalert/blob/master/docs/getting-started.md)). You bring the Twilio account, as with OnCall OSS after archival. It does not have OnCall's Grafana-plugin UI. **Keep** is not a pager. A Keep team member who describes himself as the "creator of OnCall (not related to Grafana anymore)" wrote in March 2025 that Keep is a "toolbox for alerts" and "If you want to send SMS and make Phone Calls to your on-call engineers, you still need some 3'rd party", suggesting Keep plus Twilio, or "minimalistic use of PagerDuty (Just calendar and phone calling)" alongside Keep ([discussion](https://github.com/keephq/keep/discussions/4036)). Elastic agreed to acquire Keep in May 2025 and said it "will integrate with Elasticsearch and Kibana and remain open source" ([Elastic](https://www.elastic.co/security-labs/blog/elastic-and-keep-join-forces)). **PagerDuty's free plan** covers 5 users, "1 on-call schedule, 1 escalation policy", with "100 free global SMS or phone notifications per month"; Professional removes the cap. For small teams leaving Opsgenie as well as OnCall, that, incident.io's Basic plan (single-team on-call included) and Grafana IRM's free tier (3 active users) are the cheapest hosted starting points we found. For incident management beyond paging, see [open source incident management](/blog/open-source-incident-management) and the [Opsgenie guide](/blog/opsgenie-complete-guide-2026). ## Which on-call tool doesn't send incident data to an AI vendor? A self-hosted pager with no AI features: GoAlert, the Appwrite fork or LinkedIn Oncall. Alert payloads then stay on your servers, apart from what you send to Twilio, Slack or a push service for delivery. The hosted tools now bundle AI: Grafana IRM lists "AI investigations through Grafana Assistant", and incident.io and Rootly list AI features on their pricing pages. Check each vendor's data-processing terms for the plan you buy; the pricing pages do not settle it. If you want AI investigation without sending data to a model vendor, run the investigation agent yourself with a local model. Aurora supports Ollama for air-gapped operation, and HolmesGPT documents Ollama too. ## What happens after the page: where an investigation agent fits Every tool above answers "who gets woken up". None of them tells that person why the alert fired. That second step is where an AI investigation agent sits, and it runs beside the pager, not instead of it. With Grafana Alerting it is one more contact point. Aurora's [Grafana connector](https://github.com/Arvo-AI/aurora/blob/main/website/docs/integrations/connectors.md) is a webhook contact point ("No API key required") that you add to a notification policy, for Grafana Cloud or self-hosted Grafana. The same policy keeps paging through IRM, GoAlert, PagerDuty or a fork. Aurora also takes alerts from PagerDuty, Datadog, New Relic, OpsGenie, incident.io and others. If you are on Grafana Cloud, Grafana Assistant Investigations does this step inside Grafana and can be started from IRM webhooks ([docs](https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/platform/investigation/)); it is "Not available in self-managed deployments" ([docs](https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/get-started/self-managed/)). ### What we measured on that step We ran the step after the page in a lab: a small app (`shop`: nginx frontend, Python api, worker, redis, load generator) on a 3-node kind v0.33.0 cluster with Kubernetes v1.37.0, 19 injected faults, 2 runs each. Each run started from alert text of the kind a pager delivers, for example "HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes (threshold 5%)." No pager was in the loop: the lab posted the alert to Aurora's API directly. | | Aurora (commit 87d22baf) | HolmesGPT 0.43.0 | K8sGPT 0.4.40 | |---|---|---|---| | Input | alert text | alert text, `holmes ask` | no alert; `k8sgpt analyze --explain --namespace shop` | | Correct root cause | 34 of 38 | 31 of 38 | not scored as a peer | | Partial / wrong | 4 / 0 | 6 / 1 | 21 / 5, plus 12 "No problems detected" runs (6 scenarios) | | Median time per run | 153.5 s (77 to 548 s) | 45 s (22 to 162 s) | 12.5 s | | Median model cost | $0.35 | not captured | not captured | | Grades confirmed by | a person at Arvo AI | judge model only | judge model only | From [results.json](https://raw.githubusercontent.com/Arvo-AI/aurora-lab/main/benchmark/results.json). A judge model (Claude Opus 5.5) graded every answer. A person at Arvo AI read all 38 Aurora grades and changed none; the 76 HolmesGPT and K8sGPT grades are judge-model only (a second model re-read all 76 and agreed, a third pass disputed 6 K8sGPT labels, and a blinded regrade matched 38 of 38, 37 of 38 and 36 of 38). The prompts differ in wording. Aurora and HolmesGPT ran Claude Sonnet 5 on Bedrock; K8sGPT ran Claude Sonnet 4.5, because Bedrock rejects the temperature parameter K8sGPT sends for Sonnet 5. All three got the same admin kubeconfig behind their own guard (Aurora's `observability_only` command policy, HolmesGPT's allow-list, K8sGPT's read-only `analyze`). Aurora ran on 2026-10-02; HolmesGPT and K8sGPT on 2026-10-15 and 2026-10-16, same cluster and faults. Arvo AI wrote the scenarios and builds Aurora. Aurora and HolmesGPT are roughly even on accuracy (the two-sided Fisher test on [/benchmark](/benchmark) gives p = 0.52, comparing person-read with judge-only grades), and HolmesGPT is about 3x faster. K8sGPT does a different job: it lists what its analyzers detect rather than investigating an alert. Full table and traces: [/benchmark](/benchmark). **A page that says nothing useful.** In scenario 13 the alert only says the frontend returns 100% 5xx; every pod is Running. Aurora's first run took 202 seconds and 25 tool calls ($0.69) and wrote: > The root cause was a NetworkPolicy named `redis` in the shop namespace, created at 2026-10-02T20:23:54Z, which restricts ingress to redis:6379 to pods labeled `app=worker` only [20, 22]. The `api` pods are labeled `app=api`, not `app=worker` [23], so their connections to redis were silently dropped by the policy The bracketed numbers point at the commands whose output supports each claim ([run folder](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/13-networkpolicy-blocks-redis/r1)). The second run was also correct but took 444 seconds. HolmesGPT got it right in 110 and 118 seconds. K8sGPT printed "No problems detected" both times, because no object looks broken. The manual checks are in the [NetworkPolicy blocking traffic runbook](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes). **Where it went wrong.** In scenario 12 the alert is a crash loop on the worker, which is being OOMKilled, while, separately, the frontend Service selects no pods. Both Aurora runs found the OOM and then stated the opposite of the truth about the rest. The first run wrote: > Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]. The second said the api and frontend pods "were running normally and unaffected" ([run folder](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/12-two-faults-oom-and-selector)). HolmesGPT was partial on both runs too. An agent that starts from the alert tends to stay inside the alert. Whoever takes the page should still check what the alert does not mention; the [second fault runbook](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes) lists the checks. The lab is Kubernetes-only and synthetic, with faults we wrote, so treat it as a method rather than a forecast. Every scenario ships with its alert and its expected findings in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab), so you can run any agent through the same faults. For one failure on your own cluster, use [/diagnose](/diagnose). For more on the agents themselves, see [Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt) and [AI incident investigation tools compared](/blog/best-ai-powered-incident-investigation-tools). ## Which should you choose? | If you | Use | Why | |---|---|---| | Already use Grafana Cloud and can send alert data to SaaS | Grafana Cloud IRM | Grafana's own successor, priced per active responder | | Run OnCall OSS today and need it working on current Grafana | Appwrite fork, as a bridge | Same data model; no support or roadmap | | Must self-host for good | GoAlert, plus your own Twilio account | Maintained, Apache-2.0, scheduling and escalation built in | | Want alert dedup and correlation in front of a pager | Keep | Not a pager; pair it with one | | Want hosted, cheapest start | PagerDuty Free (5 users), incident.io Basic or Grafana IRM Free (3 users) | Free tiers with on-call | | Want the alert investigated before anyone opens a laptop | Any pager above, plus an investigation agent as a second contact point | Paging and investigation are separate jobs | Verified 2026-10-21: Grafana blog, docs, pricing and IRM pages; GitHub repositories, releases and star counts as linked; PagerDuty, incident.io and Rootly pricing pages; Atlassian's Opsgenie migration page; Elastic's Keep announcement. Lab numbers from results.json, generated 2026-10-21. Aurora is open source on [GitHub](https://github.com/Arvo-AI/aurora) (Apache 2.0); to see it investigate your Grafana alerts, [book a demo](https://cal.com/arvo-ai/demo). --- ## Keep AIOps in 2026: Is the Open Source Platform Still Alive? URL: https://www.aurorasre.ai/blog/keep-vs-aurora-open-source-aiops Published: 2026-06-03 (updated 2026-10-21) Author: Arvo AI Team, Engineering > Keep (keephq) open source AIOps, checked 2026-10-21: MIT core, owned by Elastic, community-maintained, v0.54.3. What it does and what an AI agent adds. Keep ([keephq/keep](https://github.com/keephq/keep)) is an open source alert management and AIOps platform: it collects alerts from monitoring tools, deduplicates them, correlates them into incidents, and runs YAML workflows that open tickets, notify people or call other tools. Elastic announced on May 7, 2025 that it was acquiring Keep, said Keep would "remain open source", and completed the deal on May 21, 2025. When we checked on 2026-10-21, the project was still shipping but slowly: a maintainer describes a "community-maintained" phase, the latest release was v0.54.3 (2026-09-09), and the hosted Keep Cloud address served a TLS certificate that expired on 2026-06-17. The self-hosted MIT core is still published; the AI correlation feature is not part of it. Arvo AI builds Aurora SRE, an open source investigation agent that can sit behind an alert platform like Keep. We did not install Keep for this page. Every Keep fact below comes from its repo, docs and website, or from checks we ran against them on 2026-10-21, and is linked. The lab results come from our own benchmark, where Arvo wrote the scenarios. ## What is Keep (keephq) and what does it do? Keep's README calls it "The open-source AIOps and alert management platform" and lists a single pane of glass, deduplication, enrichment, filtering and correlation, bi-directional integrations, workflows and dashboards. Elastic's [acquisition post](https://www.elastic.co/blog/elastic-and-keep-join-forces) describes it the same way and says it dedupes, correlates and prioritizes events "to reduce noise and automate root cause analysis". The [Keep homepage](https://www.keephq.dev/) claims "110+ Providers": monitoring tools that send or are polled for alerts, ticketing and paging tools it writes to, and LLM providers. | Feature | What it does | In the open source edition? | |---|---|---| | [Deduplication](https://docs.keephq.dev/overview/deduplication) | Partial mode groups alerts on fingerprint fields each provider defines; full mode discards exact repeats | Yes | | [Correlation rules](https://docs.keephq.dev/overview/correlation-rules) | Conditions on alert attributes create an incident or an incident candidate | Yes | | [Topology correlation](https://docs.keephq.dev/overview/correlation-topology) | Groups alerts on services of the same application; off by default (`KEEP_TOPOLOGY_PROCESSOR=false`) | Yes | | [Workflows](https://docs.keephq.dev/workflows/overview) | YAML with triggers (alert, incident, schedule, manual), steps and actions, "GitHub Actions for your monitoring tools" | Yes | | [AI in workflows](https://docs.keephq.dev/overview/ai-in-workflows) | LLM calls as workflow steps, for normalization, routing, severity and summaries, with your own provider (OpenAI, Anthropic, Gemini, Ollama, Llama.cpp, vLLM and others) | Yes | | [AI incident assistant](https://docs.keephq.dev/overview/ai-incident-assistant) | Chat on the incident page with its alerts and topology as context; "Users can ask for root cause analysis and even execute commands on third-party services" through [provider methods](https://docs.keephq.dev/providers/provider-methods) | Experimental (generally available on Keep Cloud and Enterprise On-Premises) | | [AI semi-automatic correlation](https://docs.keephq.dev/overview/ai-semi-automatic-correlation) | Select alerts, an LLM proposes incidents for review; aimed at fewer than 100 alerts a day | Experimental | | [AI correlation](https://docs.keephq.dev/overview/ai-correlation) | Trains a per-tenant model on your history and assigns new alerts to incidents above a confidence threshold, in 5 to 15 minute cycles | No (Keep Cloud and Enterprise On-Premises only) | Keep describes itself as an "intelligent layer before the IRM" on its [comparison page](https://docs.keephq.dev/overview/comparisons): it reduces and groups alerts before they reach PagerDuty or Opsgenie, and it positions itself as an alternative to BigPanda and Moogsoft. ## Is Keep still maintained after the Elastic acquisition? Yes, at a much lower pace than in 2025. On 2025-11-22 a user asked on GitHub whether the project was [still alive after being acquired by Elastic](https://github.com/keephq/keep/issues/5451). Maintainer Shahar Glazner (shahargl) answered the same day: > Keep as an OSS project is now in a "community-maintained" phase where we (me and @talboren) review PR's from the community and merge them. We also solve bugs discussed on Keep's Slack community. The numbers we pulled from the GitHub API on 2026-10-21 match that description: | Signal (checked 2026-10-21) | Value | |---|---| | Latest release | [v0.54.3](https://github.com/keephq/keep/releases/tag/v0.54.3), 2026-09-09 (13 releases in the past 12 months) | | Last commit on `main` | 2026-09-28; none from October 1 to 21 | | Commits on `main` per month | 245 in March 2025, 63 in June 2025, 3 in January 2026, 33 in March 2026, 11 in September 2026 | | Merged pull requests | 700 from January to April 2025; 139 from January 1 to October 21, 2026 | | Open issues and pull requests | 459 issues, 207 pull requests | | GitHub stars | 12,428 | | Self-host images | `keep-api` 0.54.3 is in Keep's public registry | | Keep Cloud (`platform.keephq.dev`, the README's "Try it out" link), 16:26 UTC | Let's Encrypt certificate expired 2026-06-17 05:26 UTC, so browsers refuse the connection; past the certificate error, Vercel answers 404 `DEPLOYMENT_NOT_FOUND` | | `api.keephq.dev`, 16:26 UTC | Served through Cloudflare with a valid edge certificate; every request we sent got HTTP 525, which means Cloudflare could not complete TLS to the origin server | | keephq.dev, docs.keephq.dev, pricing page | HTTP 200 | Of the 140 commits on `main` in 2026, 93 have a "fix" prefix and most come from outside contributors; all 139 pull requests merged in 2026 were merged by one of the two maintainers. The endpoint results are one check from one network at one time; a 525 can be a temporary origin fault, so retest before you rely on it. Elastic's 10-K for its 2025 fiscal year says it acquired all of Keep Alerting Ltd. on May 21, 2025, and its announcement said "Keep will integrate with Elasticsearch and Kibana and remain open source"; the repo is still MIT outside one directory, so that part holds. We found no current statement from Elastic or Keep on whether Keep Cloud or the Enterprise plans are still sold. The [pricing page](https://www.keephq.dev/pricing) still lists them under a 2025 copyright, and the status page it links lists no public monitors. If you need the AI correlation tier, ask Elastic or Keep directly before planning on it. What this means in practice: the self-hosted code is published, still gets bug fixes and releases, and is maintained by two people reviewing community pull requests. That can work for a team that can patch Python and Next.js itself. A team that needs a vendor to answer a security report should get that commitment in writing first. ## Is Keep open source and free? The [LICENSE](https://github.com/keephq/keep/blob/main/LICENSE) puts everything under MIT except an `ee/` directory, which has its own license; on 2026-10-21 that directory held only an identity manager module. Self-hosting is free. The pricing page lists Startup at "$0 / month", Growth at "$199 / month" and "Enterprise + AI" at "Prices fitting your needs"; "AIOps Alert Correlation", SAML/SSO and RBAC appear only in the Enterprise column. Given the Cloud endpoint failures above, treat those plans as unconfirmed. ## Does Keep do root cause analysis, or only correlation? Both, in different parts of the product. Keep's correlation works on alerts: their fields, fingerprints, timing and, if you load it, a service topology. On top of that, the AI incident assistant passes the incident's alerts, description and topology to an LLM, and Keep's [provider methods](https://docs.keephq.dev/providers/provider-methods) page says it can call methods on connected tools on request (its examples fetch Datadog traces, metrics and logs; the Opsgenie provider exposes `close_alert`) and "Chains multiple methods for comprehensive investigation". A self-hosted workflow can also run shell commands through the [Bash provider](https://docs.keephq.dev/providers/documentation/bash-provider). The docs mark the assistant experimental in the open source edition, describe it as a chat the on-call engineer drives, and do not describe it starting an investigation on its own when an alert arrives. We did not run it. That is the difference with an investigation agent: it starts on the alert without being asked, queries the systems behind it (pods, events, Services, logs, recent changes), and returns a cause with the evidence. The lab shows why that step cannot come from the alert fields alone. ## What does an alert tell you about the root cause? Evidence from 19 injected faults In our [benchmark](/benchmark), each of 19 scenarios breaks one thing in a small app on a 3-node kind v0.33.0 cluster running Kubernetes v1.37.0 (scenario 12 breaks two). Each scenario comes with one alert, written by us in Prometheus style and given to the agents as text. There are only four alert names across the 19: KubePodCrashLooping (9 scenarios), HighErrorRate (5), KubeDeploymentRolloutStuck (4) and KubePodNotReady (1). Two pairs of scenarios carry the same alert, word for word, with different causes: | Alert text (identical in both scenarios) | Scenario | Actual cause | Aurora, 2 runs | HolmesGPT, 2 runs | |---|---|---|---|---| | HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes (threshold 5%). | [11](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/11-noisy-worker-svc-targetport) | api Service `targetPort` changed to 9090; the api listens on 8080 | correct, partial | partial, partial | | (same) | [13](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/13-networkpolicy-blocks-redis) | A NetworkPolicy lets only `app=worker` reach redis | correct, correct | correct, correct | | KubeDeploymentRolloutStuck [warning] namespace=shop deployment=api: Rollout of deployment shop/api is not progressing for longer than 15 minutes. | [14](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/14-resourcequota-rollout) | A ResourceQuota leaves no room for the rollout's surge pod | correct, correct | correct, correct | | (same) | [18](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/18-init-waits-missing-service) | A new init container waits for a `postgres` Service that does not exist | correct, correct | partial, correct | To an alert platform, each pair is the same alert: same name, same labels, same text. Deduplication, rules and an ML correlator all work from those fields, and nothing in them separates a wrong port from a NetworkPolicy. The agents separated them by reading the cluster. In all 16 runs on these four scenarios, both agents named the actual cause. The four partial grades were for something else: three runs on scenario 11 did not rule out a worker whose logs were noisy but harmless, and HolmesGPT's first run on 18 offered deploying a postgres Service as one of two fixes without noting that the api does not use one. Aurora's first run on scenario 13 (2026-10-02, 25 tool calls, 202 seconds, $0.69 of model cost) found it by checking the policy against the pod labels: > The root cause was a NetworkPolicy named `redis` in the shop namespace, created at 2026-10-02T20:23:54Z, which restricts ingress to redis:6379 to pods labeled `app=worker` only [20, 22]. On scenario 11, with the identical alert, its first run reported that the Service's `spec.ports[0].targetPort` "was set to 9090, while the api Deployment's containers only expose and listen on port 8080 [17, 13]". The bracketed numbers point to the tool calls that produced the evidence. The fixes are in the runbooks: [NetworkPolicy blocking traffic](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes), [Service targetPort mismatch](/runbooks/fix-service-targetport-mismatch-kubernetes), [ResourceQuota blocking a rollout](/runbooks/fix-resourcequota-exceeded-rollout-stuck-kubernetes) and [pod stuck on an init container](/runbooks/fix-pod-stuck-init-container-kubernetes). K8sGPT, the third tool in the benchmark, takes no alert: `k8sgpt analyze` lists objects its analyzers flag. On scenarios 11 and 13 it printed "No problems detected" in all four runs, because every object looked healthy. It is a scanner, so we do not score it as a peer of the two agents. The limit runs the other way too. In scenario 12 the alert covers an OOMKilled worker while, separately, the frontend Service selects no pods. The scenario raises no alert for the second fault, so an alert platform has nothing to group it with. Both agents found the OOM in both runs and missed the frontend, and Aurora wrote that the other workloads "remained healthy and unaffected" on pod status alone. The checks it skipped are in [a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). ### Totals and caveats | | Aurora | HolmesGPT 0.43.0 | K8sGPT 0.4.40 | |---|---|---|---| | Correct | 34 of 38 | 31 of 38 | not scored as a peer | | Partial / wrong | 4 / 0 | 6 / 1 | 21 / 5, plus 12 "No problems detected" runs (6 scenarios) | | Median time per run | 153.5 s | 45 s | 12.5 s for two analyze passes | | Who confirmed the grades | a person at Arvo | judge model only | judge model only | A judge model graded every answer against the scenario's required findings. A person at Arvo read all 38 Aurora grades and changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model re-read all 76 and agreed, a third pass graded them from scratch and agreed on all 38 HolmesGPT verdicts while disputing 6 K8sGPT labels, and a blinded regrade matched 38 of 38 Aurora, 37 of 38 HolmesGPT and 36 of 38 K8sGPT verdicts. Aurora ran on 2026-10-02 (commit 87d22baf), HolmesGPT and K8sGPT on 2026-10-15 and 16, on the same cluster and faults. Aurora and HolmesGPT used the same model (Claude Sonnet 5 on Bedrock); K8sGPT used Sonnet 4.5. The prompts added different one-line instructions after the alert text, and all three got the same admin kubeconfig behind their own guards. HolmesGPT is about 3x faster at the median. Full method, every trace and the misses: [Aurora lab report](/blog/aurora-lab-report-kubernetes-19-failures) and [Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt). ## Keep vs Aurora: what each one does | | Keep | Aurora | |---|---|---| | Job | Take in alerts, deduplicate, correlate into incidents, route and automate | Investigate an alert and write the root cause with cited evidence | | License | MIT outside `ee/` | [Apache 2.0](https://github.com/Arvo-AI/aurora) | | Activity, 2026-10-21 | Community-maintained; last release 2026-09-09; 12,428 stars | 18 commits in the past 7 days; 439 stars | | Alert intake | 110+ providers | PagerDuty, Datadog, Grafana, New Relic, OpsGenie, incident.io and more, by webhook | | Deduplication and routing | Yes, the core of the product | No. It links related alerts into the running investigation and never suppresses or mutes them | | Queries your systems | Through workflow steps you write, or the AI assistant when someone asks it (experimental in open source) | Yes: `kubectl`, `aws`, `az`, `gcloud` and connected observability tools; background investigations cannot run mutating writes | | Remediation | Workflow actions you define | Suggests fixes; can open a pull request, gated on human approval, never merged automatically | | LLM | Your provider for workflow steps; AI correlation runs on Keep's service | OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, OpenRouter or Ollama | | Self-host | Docker Compose, Kubernetes, ECS, OpenShift | Docker Compose or Helm | Pick Keep when the problem is volume: alerts from many monitors, duplicates, and routing rules you want in YAML. Pick an investigation agent when the alerts already reach the right person and the time goes into working out why. If you are choosing between investigation agents, the [open source AI SRE comparison](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt) covers HolmesGPT and K8sGPT; for the alert-noise side, see [automated alert noise reduction](/blog/automated-alert-noise-reduction). ## Can Keep and Aurora work together? Yes, through a paging tool both already support. Aurora has no Keep connector (we checked the repo on 2026-10-21), and Keep has no Aurora provider. Keep's [PagerDuty provider](https://docs.keephq.dev/providers/documentation/pagerduty-provider) can create incidents from a workflow, and its [Opsgenie provider](https://docs.keephq.dev/providers/documentation/opsgenie-provider) can create alerts. Aurora takes PagerDuty webhooks (`incident.triggered`) and Opsgenie or JSM Operations webhooks (alert create events), and starts a background investigation for each one. So the chain is: monitors send to Keep, a Keep workflow on the correlated incident creates one PagerDuty incident or Opsgenie alert, and Aurora investigates that one incident instead of every raw alert. If you turn on RCA notes and give it a token with write access, Aurora also posts the root cause back to the PagerDuty incident as a note. We have not run this chain end to end. Test it with one workflow and one service before you route production through it, and if you depend on Keep, have a plan for maintaining your own fork. ## Which open source AIOps tools support bring-your-own-model? Among the tools on this page: Keep's AI workflow steps use the provider you configure, including local Ollama, Llama.cpp or vLLM; its AI correlation does not, since it runs on Keep's service. Aurora runs on the model you choose, including Ollama for air-gapped installs. HolmesGPT and K8sGPT also take your own provider. One caution from our lab: every result above used Claude models on Bedrock, and accuracy on a local model is untested. For the wider field, including Coroot, see [AIOps platforms with root cause analysis](/blog/top-10-aiops-platforms-free-root-cause-analysis-2026) and [top AI SRE tools in 2026](/blog/top-ai-sre-tools-2026). ## Is Keep a cheaper alternative to BigPanda? Keep positions itself as one, and its free and $199 tiers are far below enterprise AIOps pricing. Two things narrow that. The AI correlation that compares with BigPanda's is in the Enterprise tier only, and Keep Cloud did not load when we checked on 2026-10-21. The self-hosted rule and topology correlation is free and works without a vendor. For the full field, see [BigPanda alternatives](/blog/bigpanda-alternative-open-source). ## How to test any of this yourself The scenarios, alerts and grading rubric are in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab). Break a scenario on a local kind cluster, send its alert through Keep or straight to an agent, and grade the answer against `must_identify` in the scenario's `scenario.yaml`. Try scenarios 11 and 13 back to back: same alert, different fix. For a single failure on your own cluster, paste the symptoms into [/diagnose](/diagnose). Verified 2026-10-21: Keep's GitHub repo, LICENSE, releases, commit and pull request counts and issue 5451 (GitHub API); Keep docs pages and pricing page as linked; Keep Cloud and API endpoints (curl -v and openssl s_client, 16:26 UTC); Elastic's acquisition post and its fiscal 2025 10-K; Aurora's README, connector docs and webhook routes at commit 6018891. Lab numbers come from [results.json](https://raw.githubusercontent.com/Arvo-AI/aurora-lab/main/benchmark/results.json). Aurora is on [GitHub](https://github.com/Arvo-AI/aurora) under Apache 2.0, or [book a demo](https://cal.com/arvo-ai/demo) to run it behind your own alerts. --- ## HolmesGPT Review 2026: What It Is and How It Did in 38 Runs URL: https://www.aurorasre.ai/blog/holmesgpt-alternative-multi-cloud Published: 2026-06-03 (updated 2026-10-21) Author: Ben Gervais, Engineer at Arvo AI > HolmesGPT is the CNCF Sandbox SRE agent Robusta created. We ran 0.43.0 on 19 Kubernetes faults: 31 of 38 judge-graded correct, about 3x faster than Aurora. **HolmesGPT is an open-source (Apache 2.0) AI agent that investigates production incidents by letting an LLM call read-only tools against your Kubernetes cluster, metrics, logs and cloud APIs until it can name a root cause.** Robusta created it, Microsoft contributes to it, and it has been a [CNCF Sandbox project since October 8, 2025](https://www.cncf.io/projects/holmesgpt/). It is good: when we ran HolmesGPT 0.43.0 on 19 injected Kubernetes faults, twice each, a judge model graded 31 of its 38 answers correct, 6 partial and 1 wrong, at a median of 45 seconds per investigation. Those 38 grades come from a judge model, re-read by a second model and a blinded regrade, and no person has checked them yet. Arvo builds [Aurora](https://github.com/Arvo-AI/aurora), a competing open-source agent, and Arvo wrote the 19 scenarios. Every scenario, every HolmesGPT transcript and every grade is public in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/h2h-v1), so you can check our reading against the raw output. ## What is HolmesGPT? The [README](https://github.com/HolmesGPT/holmesgpt) calls it an "Open-source AI agent for investigating production incidents and finding root causes" that "Works with any stack". The facts a reader usually wants, checked on 2026-10-21: | | HolmesGPT | |---|---| | Who builds it | Originally created by Robusta.Dev, with major contributions from Microsoft ([README](https://github.com/HolmesGPT/holmesgpt)) | | Governance | [CNCF Sandbox, accepted October 8, 2025](https://www.cncf.io/projects/holmesgpt/) | | License | [Apache 2.0](https://github.com/HolmesGPT/holmesgpt/blob/master/LICENSE) | | Repo | [github.com/HolmesGPT/holmesgpt](https://github.com/HolmesGPT/holmesgpt) (the old robusta-dev/holmesgpt URL redirects there), about 3,650 stars | | Release cadence | [0.43.0 on 2026-10-07, 0.44.0 on 2026-10-14, 0.45.0 on 2026-10-15](https://github.com/HolmesGPT/holmesgpt/releases). We tested 0.43.0; 0.44.0 and 0.45.0 were already out when our runs started and are untested | | Interfaces | CLI (`holmes ask`, `holmes investigate`), Helm chart, Slack and Teams via Robusta, a web UI through the [Robusta platform](https://holmesgpt.dev/latest/installation/ui-installation/) (SaaS or self-hosted) | | Models | Any LiteLLM-supported provider: OpenAI, Anthropic, AWS Bedrock, Azure, Gemini, Vertex AI, OpenRouter, Ollama and others ([AI providers](https://holmesgpt.dev/latest/ai-providers/)) | | Write access | Read-only by default; an optional [Kubernetes Remediation MCP](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/kubernetes-remediation-mcp/) adds human-approved writes | "Robusta HolmesGPT" and "CNCF HolmesGPT" are the same project. Robusta still runs the commercial platform around it (the hosted UI, the Slack bot, a managed model endpoint called Robusta AI), while the agent itself lives under the CNCF. ## How does HolmesGPT work? You give it a question or an alert. The model picks tools from its enabled **toolsets**, reads the output, and loops until it has an answer. In our runs the CLI printed `Using 28 datasources (toolsets)` with default settings, then initialized only the ones the model used (`kubernetes/core` in 29 of 38 runs, `bash` in 25). Toolsets are the core idea. The [built-in list](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/) covers Kubernetes, Prometheus, Grafana, Loki, Tempo, Datadog, New Relic, Elasticsearch, ArgoCD, Helm, Kafka, RabbitMQ, SQL databases and more, with AWS, Azure, GCP, GitHub and Sentry reached through MCP servers. You can add your own as [custom toolsets](https://holmesgpt.dev/latest/data-sources/custom-toolsets/), REST API toolsets or [remote MCP servers](https://holmesgpt.dev/latest/data-sources/remote-mcp-servers/). Shell access goes through the [bash toolset](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/bash/), which checks each command against an allow list. The CLI default (`core`) pre-approves `kubectl get`, `describe`, `logs`, `top`, `events` and a few others plus text tools like `jq` and `grep`. Anything else asks you for approval, and you can widen the list in `~/.holmes/config.yaml`. This matters for the results below. The two commands you will use: ```bash # Ask anything; HolmesGPT picks the tools holmes ask "what pods are failing?" # Pull active alerts from Alertmanager and investigate them holmes investigate alertmanager --alertmanager-url http://localhost:9093 ``` PagerDuty, OpsGenie and Jira can also be investigation sources from the CLI, with findings written back ([README](https://github.com/HolmesGPT/holmesgpt)). [Operator mode](https://holmesgpt.dev/latest/operator/), marked alpha, runs scheduled and deployment-triggered health checks as Kubernetes CRDs and posts to Slack. ## How do you install HolmesGPT? From the [CLI install docs](https://holmesgpt.dev/latest/installation/cli-installation/): ```bash # Homebrew (Mac/Linux) brew tap robusta-dev/homebrew-holmesgpt brew install holmesgpt # or pipx pipx install holmesgpt holmes ask --help ``` For a cluster install, the [Helm chart](https://holmesgpt.dev/latest/installation/kubernetes-installation/) is `robusta/holmes` from `https://robusta-charts.storage.googleapis.com`, configured with a `values.yaml` that lists your models. Set the provider key (for example `ANTHROPIC_API_KEY`) and choose the model with the model flag shown below. HolmesGPT's own docs say "Model choice has a large impact on results" and publish [per-model eval results](https://holmesgpt.dev/latest/development/evaluations/latest-results/). ## Can HolmesGPT run with your own model or offline? Yes for your own model, with a caveat for fully local ones. Any hosted provider above works with your own key, and an OpenAI-compatible endpoint (a LiteLLM proxy, vLLM and similar) covers self-hosted serving. For Ollama the [docs](https://holmesgpt.dev/latest/ai-providers/ollama/) warn that "Ollama support is experimental and can be tricky to configure correctly" and that tool calling with local models may be inconsistent. We ran it on AWS Bedrock (`bedrock/us.anthropic.claude-sonnet-5`), which worked without changes. One gap we hit: LiteLLM had no price entry for that model, so HolmesGPT logged usage cost as 0 and we have no cost per run for it. ## How good is HolmesGPT? Our test on 19 Kubernetes faults We built a small app (`shop`: nginx frontend, Python api, worker, redis) on a 3-node [kind](https://kind.sigs.k8s.io) cluster (kind v0.33.0, Kubernetes v1.37.0) and broke it 19 ways. Ten scenarios have one clear fault. Nine are harder: a red herring next to the fault, two faults at once, a slow leak, a half-applied config change. Each scenario has a written ground truth and a list of findings a correct answer must contain. Full method in the [lab report](/blog/aurora-lab-report-kubernetes-19-failures); live numbers on the [benchmark page](/benchmark). HolmesGPT 0.43.0 got the alert text and this, through `holmes ask` with default toolsets: ```bash holmes ask " Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." \ --model "bedrock/us.anthropic.claude-sonnet-5" --max-steps 40 ``` Aurora got the same alert text plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." on the same model. The lab has no Alertmanager, so `holmes investigate` was not used. HolmesGPT's runs were on 2026-10-15 and 2026-10-16; Aurora's 38 runs are from 2026-10-02, on the same cluster and the same injected faults. ### The results | | HolmesGPT 0.43.0 | Aurora | |---|---|---| | Correct | 31 of 38 | 34 of 38 | | Partial | 6 | 4 | | Wrong | 1 | 0 | | Single-fault scenarios (01 to 10) | 19 of 20 correct | 20 of 20 correct | | Harder scenarios (11 to 19) | 12 of 18 correct | 14 of 18 correct | | Median time per investigation | 45 s | 153.5 s | | Range | 22 to 162 s | 77 to 548 s | | Who checked the grades | Judge model, second-model review, blinded regrade; no person yet | A person read all 38 | The 3-run gap in correct answers is not statistically meaningful at this sample size: a two-sided Fisher exact test on correct versus not (34 of 38 against 31 of 38) gives p ≈ 0.52, the same test the [benchmark page](/benchmark) computes from the totals. That test also compares person-read Aurora grades with judge-only HolmesGPT grades. Read the two as roughly even on accuracy. Speed is where HolmesGPT clearly wins. Its median is about 3.4 times faster than Aurora's, and it was faster on the median of every one of the 19 scenarios. Part of Aurora's time is contention from its own background jobs in the lab stack, but even its single-fault median (135.5 s) is three times HolmesGPT's (41 s). For a CLI you run during an incident, 45 seconds versus two and a half minutes is a real difference. Three checks on the HolmesGPT grades. A second model (Claude Sonnet 5) read every grade next to the judge's and agreed with all 38 ([review-summary.md](https://github.com/Arvo-AI/aurora-lab/blob/main/runs/h2h-v1/review-summary.md)). A third, fresh pass graded all 38 from scratch without seeing the judge's verdict and also agreed with all 38. A blinded regrade, with agent and tool names removed, matched 37 of 38 and moved one run down (scenario 19, run 2, from correct to partial). The judge (Claude Opus 5.5) is from the same vendor as the model both agents ran on. The third pass and the blind regrade are in [review2-summary.md](https://github.com/Arvo-AI/aurora-lab/blob/main/runs/h2h-v1/review2-summary.md) and [blind-regrade.md](https://github.com/Arvo-AI/aurora-lab/blob/main/runs/blind-regrade.md). ### What a good HolmesGPT answer looks like Scenario 15: a stray Deployment's pods carry `app=api`, so the api Service sends a third of requests to a broken pod. The alert only says 31% of `/api/` requests return 5xx. HolmesGPT, run 1, 97 seconds: ```text The shop namespace has two separate backend Deployments that both carry the Pod label app: api: [...] shop-api's app.py crashes on every /items request: KeyError: 'DATABASE_URL' [...] Since kube-proxy load-balances the api Service round-robin/random across its 3 endpoints, 1 of every 3 (~33%) requests to frontend's /api/* proxy_pass hits the broken shop-api pod and returns 500 ``` That is a correct, evidenced answer, including the arithmetic that ties 1 in 3 endpoints to the 31% error rate. On scenario 17 (a slow leak in worker code, logs clean) it found the unbounded list in the worker's source, worked out about 5 MB per batch against the 192Mi limit, and checked that the restart changed nothing in env, image or limits. Both scenarios have runbooks built from the same lab: [Service selecting the wrong pods](/runbooks/fix-service-selects-wrong-pods-label-overlap-kubernetes) and [memory leak OOMKilled](/runbooks/fix-memory-leak-oomkilled-kubernetes). ### Where HolmesGPT missed All seven non-correct runs, none left out: | Scenario | Run | Verdict | What happened | |---|---|---|---| | 10 wrong `REDIS_HOST` | r1 | partial | Found the unresolvable `redis-master` host, but inverted which ReplicaSet carried the change and called the problem cosmetic | | 11 wrong `targetPort`, noisy worker | r1, r2 | partial, partial | Right cause and fix (`targetPort` 9090 vs 8080) both times; never addressed the worker's error logs, which the scenario requires ruling out | | 12 two faults, alert covers one | r1, r2 | partial, partial | Diagnosed the alerted worker OOM correctly; never checked Services, so missed the frontend Service selecting no pods | | 18 init container waits for a missing Service | r1 | partial | Diagnosis fully right; first suggested fix was to deploy a postgres the app does not use | | 19 ConfigMap edit loaded by one pod of two | r1 | wrong | Blamed a cold-start race and declared the incident resolved | Scenario 19, run 1, is the one wrong answer. The frontend ConfigMap's upstream was changed to `api:8000`, and only the pod that restarted picked it up. HolmesGPT read logs from the healthy pod, saw old 502s from startup, and concluded, in 49 seconds: ```text AI: All data points to a transient startup race, now resolved. [...] Current state (resolved): both api pods and both frontend pods are now Running/Ready with 0 restarts, the api Endpoints object lists both pod IPs as healthy, and current api logs show steady 200 responses on /healthz and /items. No new 502s in the latest frontend logs. ``` In run 2 it found the split and the `api:8000` change, though it misstated why the healthy pod was still healthy. Aurora did not do much better here: one correct, one partial. This failure pattern (one pod sampled, the answer generalized) is the subject of the [ConfigMap change not applied to all pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes) runbook. Scenario 12 tripped both agents in all four runs. Each diagnosed the worker OOM the alert named and stopped. If your alerts cover one symptom of a wider change, neither tool will go looking unprompted; see [when the alert covers only one fault](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). ### Was the test fair to HolmesGPT? Things that went against it: - Its bash allow list refused `kubectl rollout history` in 12 runs, 2 of its 7 non-correct runs among them. Because `holmes ask` ran non-interactively, it could not prompt us, so the command was denied. The [docs](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/bash/) show adding exactly that command to `allow` in config. We did not rerun with it allowed. Aurora's own command policy had the same block in our first lab report. - Default toolsets only. A tuned install with Prometheus and Loki connected would see more than kubectl. - Arvo wrote the scenarios and the must-identify lists, and builds the competing product. Access was the same for both: each got the kind cluster's admin kubeconfig, behind its own guard (Aurora's default `observability_only` command policy, HolmesGPT's bash allow list). No transcript shows a mutating command running. Things that went for it: - Aurora's runs come from a commit dated the day before the scenarios were written, so neither tool was tuned on them. ## What HolmesGPT does not do On the evidence above it is a strong in-cluster investigator. The limits are about scope, not quality: - **Read-only by default.** The README says "By design, HolmesGPT has **read-only access** and respects RBAC permissions." Writes need the separate [Remediation MCP](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/kubernetes-remediation-mcp/), where mutations go through one `run_kubectl_command` tool that is listed under `approval_required_tools` by default, so each one waits for a person. - **Its job is the investigation.** The README describes fetching an alert or ticket and writing the analysis back to that source or to Slack. Incident records and postmortems stay in whatever incident tool you already run. - **Cloud reach goes through MCP.** AWS, Azure and GCP are MCP-based toolsets rather than direct CLI access, so cross-cloud depth depends on those servers and the credentials you give them. - **Local models are experimental**, per its own Ollama page. - **No cost figure in our runs**, because LiteLLM had no price for the model. Check this on your model before you budget. ## HolmesGPT vs alternatives: which should you pick? **Pick HolmesGPT** if your incidents mostly live in Kubernetes and the observability stack around it, you want a fast CLI answer during an incident, CNCF governance matters to your review board, and read-only by default is what your security team will approve. Its speed and its toolset catalog are real advantages. **Pick K8sGPT** if you want a scanner rather than an investigator. `k8sgpt analyze` with its explain option lists problems its analyzers detect across a namespace and has an LLM explain each one; it does not take an alert and chase it. In our lab it ran in a median of 12.5 seconds for two passes and mostly reported the visible symptom (a crashed container, a Service with no ready endpoints) with a checklist, which is what a scanner is built to do; it was not given the alert, so it is not scored as a peer of the two investigators. The full head-to-head is in [HolmesGPT vs K8sGPT](/blog/holmesgpt-vs-k8sgpt). **Pick Aurora** if you need an incident platform around the investigation: alerts arriving by webhook from your monitoring tools, investigations across AWS, Azure, GCP, OVH, Scaleway and Kubernetes through their CLIs, postmortems exported to Confluence, Notion or SharePoint, and suggested fixes that can become a GitHub pull request after a human approves it. It self-hosts with Docker Compose, Helm or an air-tight bundle and takes any model, including Ollama ([README](https://github.com/Arvo-AI/aurora)). On our faults it matched HolmesGPT's accuracy and was about three times slower. | If you need | Best fit | |---|---| | Fastest answer to "why is this pod broken" | HolmesGPT | | A cluster-wide list of what is misconfigured | K8sGPT | | Read-only, CNCF-governed agent for a strict review | HolmesGPT | | Incident records, postmortems and approved fix PRs across clouds | Aurora | | Investigation data that never leaves your network | Any of the three, self-hosted with your own model | For more options, including Cleric and other self-hosted agents, see [the best open-source AI SRE tools](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt), [HolmesGPT vs Cleric](/blog/holmesgpt-vs-cleric) and the [self-hosted AI SRE guide](/blog/self-hosted-ai-sre). If you only have a broken pod right now, paste its `kubectl` output into the [diagnose tool](/diagnose). ## Try Aurora Run both on your own failures. Aurora is Apache 2.0 at [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora), and you can [book a demo](https://cal.com/arvo-ai/demo). *Tested with HolmesGPT 0.43.0 on 2026-10-15 and 2026-10-16, against Aurora runs (commit 87d22baf) from 2026-10-02. HolmesGPT facts checked against its GitHub repo, docs and the CNCF project page on 2026-10-21. Raw runs: [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab).* --- ## Replacing Splunk ITSI: Open Source AI SRE Alternatives URL: https://www.aurorasre.ai/blog/splunk-itsi-alternative-open-source Published: 2026-06-03 (updated 2026-10-12) Author: Noah Casarotto-Dinning, CEO at Arvo AI > How to replace Splunk ITSI with an AI SRE: what ITSI does well, what its ingest licensing costs, and the open source, multi-cloud alternatives to evaluate. > ## Key Takeaways > > - **Splunk IT Service Intelligence (ITSI) is a closed, proprietary AIOps add-on, not open source.** It is purchased on top of Splunk Cloud Platform or Splunk Enterprise under either ingest-based or workload-based (Splunk Virtual Compute) pricing, with no public per-GB or per-seat dollar figure ([Splunk ITSI pricing FAQ](https://splunk.com/en_us/software/pricing/faqs/it-operations.html)). Aurora by Arvo AI is [Apache-2.0 open source](https://github.com/Arvo-AI/aurora) with no ingest-priced licensing. > - **ITSI is now Cisco-owned.** [Cisco completed its 28 billion dollar acquisition of Splunk on 18 March 2024](https://newsroom.cisco.com/c/r/newsroom/en/us/a/y2024/m03/cisco-completes-acquisition-of-splunk.html), so ITSI roadmap and licensing now sit inside Cisco. > - **ITSI is correlation and analytics centric.** Its core jobs are service-oriented KPI dashboards, glass tables, predictive alerting, and grouping [notable events into episodes](https://docs.splunk.com/Documentation/ITSI/latest/EA/NotableEvents) via aggregation policies ([Splunk ITSI product page](https://www.splunk.com/en_us/products/it-service-intelligence.html)). Aurora is an agentic investigator that actively queries clouds and runs commands. > - **ITSI's newest AI is summarization and correlation, not autonomous execution.** At [.conf25 Splunk announced ITSI Event iQ (GA) for AI-driven alert correlation and ITSI Episode Summarization (Alpha)](https://www.splunk.com/en_us/blog/observability/conf25-splunk-observability-announcements.html). Aurora goes further: it runs kubectl, aws, az, and gcloud in sandboxed Kubernetes pods and can open pull requests ([Aurora repo](https://github.com/Arvo-AI/aurora)). > - **Aurora is self-hosted, air-gapped capable, and multi-cloud.** It supports AWS, Azure, GCP, OVH, and Scaleway, and runs on your own LLM via Ollama ([Aurora repo](https://github.com/Arvo-AI/aurora)). See our [self-hosted AI SRE guide](/blog/self-hosted-ai-sre). > - **Honest take:** if your operational data already lives in Splunk, ITSI's glass-table service views and deep-dive analytics are hard to beat. The trade is Splunk-licensing cost and a closed, single-vendor stack. If you already run Splunk, ITSI is one of the most capable service-monitoring layers on the market. The question this post answers is narrower: when does an open-source, self-hosted, multi-cloud AI SRE make more sense than another ingest-priced Splunk module? We compare ITSI and Aurora claim by claim, name where each genuinely wins, and cite every fact to a primary source. For the broader category framing, see our guide on [AI SRE vs AIOps](/blog/ai-sre-vs-aiops). For the wider field of free root-cause tooling, see [Top 10 AIOps platforms offering free root cause analysis](/blog/top-10-aiops-platforms-free-root-cause-analysis-2026). ## What is Splunk ITSI? Splunk IT Service Intelligence (ITSI) is a premium AIOps and service-monitoring application that runs on top of the Splunk platform. It models your environment as services and KPIs, then layers machine learning, dashboards, and event analytics on the telemetry you already ingest into Splunk. Its core capabilities, per the [official ITSI product page](https://www.splunk.com/en_us/products/it-service-intelligence.html), include service-oriented dashboards that track KPIs and availability, service deep dives that let you analyze metrics in swim lanes and drill into raw data, predictive alerting that uses machine learning to detect future service degradations, automated event aggregation using out-of-the-box machine learning policies, and a KPI-driven triage view that prioritizes incidents by severity. ITSI also offers glass tables, which let teams visualize the key metrics behind a business or IT service. ITSI's event analytics model is built around two concepts. A [notable event](https://docs.splunk.com/Documentation/ITSI/latest/EA/NotableEvents) is an anomalous incident detected by a multi-KPI alert, a correlation search, or an anomaly-detection algorithm. Notable events feed an aggregation engine that groups related events into [episodes](https://docs.splunk.com/Documentation/ITSI/latest/EA/Investigate), which analysts triage, assign, and investigate in Episode Review. This correlation and grouping model is the heart of ITSI. ITSI is now part of Cisco. [Cisco completed its acquisition of Splunk for approximately 28 billion dollars on 18 March 2024](https://newsroom.cisco.com/c/r/newsroom/en/us/a/y2024/m03/cisco-completes-acquisition-of-splunk.html), so ITSI's roadmap, support, and licensing now sit inside Cisco's observability portfolio. ## What AI did Splunk announce for ITSI at .conf25? At .conf25, Splunk announced two ITSI AI features focused on alert correlation and incident summarization, not autonomous investigation. Per the [.conf25 observability recap](https://www.splunk.com/en_us/blog/observability/conf25-splunk-observability-announcements.html), ITSI Event iQ became generally available and delivers AI-driven alert correlation that groups related alerts and highlights the incidents requiring immediate attention. ITSI Episode Summarization, announced in Alpha, uses AI to pull the key facts about an episode into a single AI-generated summary so analysts go from clicking through many tabs to seeing the relevant details in one place. These are meaningful quality-of-life improvements for Splunk-resident teams. They are squarely in the correlate-and-summarize lane, though. Event iQ groups alerts; Episode Summarization writes a readable digest of what already happened. Neither runs a cloud API call, executes a kubectl command, or opens a fix. That gap is the opening for an agentic AI SRE. ## What is Aurora? [Aurora](https://github.com/Arvo-AI/aurora) is an open-source, Apache-2.0 AI SRE and incident-management platform from Arvo AI that autonomously investigates incidents across multi-cloud and Kubernetes environments. Where ITSI correlates and summarizes, Aurora actively gathers new evidence and can take remediation actions under human approval. Aurora's agents are orchestrated with LangGraph. When an alert fires, they query AWS, Azure, GCP, OVH, and Scaleway plus Kubernetes, and they run kubectl, aws, az, and gcloud commands inside sandboxed Kubernetes pods to inspect live state. Aurora builds a Memgraph-backed infrastructure knowledge graph to reason about blast radius, generates root-cause analyses and postmortems that export to Confluence, Notion, or SharePoint, and can suggest code fixes or open pull requests. Destructive actions are human-gated. Aurora ingests alerts via webhook from eleven monitoring connectors: PagerDuty, Datadog, Grafana, New Relic, OpsGenie, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, and incident.io, plus a Slack bot for interaction. It is self-hosted and air-gapped capable, and it is bring-your-own-LLM through Ollama, so inference can stay inside your own boundary. For why that deployment posture matters to regulated buyers, see our [self-hosted AI SRE guide](/blog/self-hosted-ai-sre). ## Splunk ITSI vs Aurora: head to head The cleanest way to see the difference is correlation versus investigation. ITSI is excellent at turning a firehose of Splunk-resident telemetry into service KPIs, glass-table views, and grouped episodes. Aurora is built to take a single incident and actively investigate it: query the cloud control plane, run live commands, map dependencies, and draft a fix. | Dimension | Splunk ITSI | Aurora | |---|---|---| | License | Closed, proprietary Splunk premium app ([pricing FAQ](https://splunk.com/en_us/software/pricing/faqs/it-operations.html)) | [Apache-2.0 open source](https://github.com/Arvo-AI/aurora) | | Deployment | Add-on to Splunk Cloud Platform or Splunk Enterprise ([pricing FAQ](https://splunk.com/en_us/software/pricing/faqs/it-operations.html)) | Self-hosted via Docker Compose or Helm, air-gapped capable | | Multi-cloud scope | Analyzes whatever telemetry you ingest into Splunk | Native queries to AWS, Azure, GCP, OVH, Scaleway, Kubernetes | | Investigation vs correlation | Correlates notable events into [episodes](https://docs.splunk.com/Documentation/ITSI/latest/EA/Investigate), AI summarization in Alpha | Agentic, runs an LLM tool-calling loop to gather new evidence | | Write and execute actions | Triggers ITSM tickets and playbooks; no native cloud-CLI execution | Runs kubectl, aws, az, gcloud in sandboxed pods, opens PRs, human-gated | | Pricing model | Ingest-based or workload-based (SVC), no public rate ([pricing FAQ](https://splunk.com/en_us/software/pricing/faqs/it-operations.html)) | Free and open source, you pay only your own compute and LLM | | Self-host and air-gap | Self-managed Enterprise option exists; SaaS via Splunk Cloud | Self-hosted by default, BYO-LLM via Ollama | | LLM model | AI features run inside the Splunk-managed stack | Bring your own model, including fully local | ### Where the cost models diverge The pricing difference is structural, not just a discount. ITSI is sold on top of Splunk under either ingest-based pricing, tied to GB ingested per day, or workload-based pricing measured in Splunk Virtual Compute units, and Splunk publishes no per-GB or per-seat rate on its [pricing FAQ](https://splunk.com/en_us/software/pricing/faqs/it-operations.html). In practice your ITSI cost scales with how much telemetry you push into Splunk. Aurora carries no licensing fee at all; it is Apache-2.0 and self-hosted, so your only costs are the compute you run it on and whatever LLM inference you choose, including a local model that costs nothing per token. ### Where ITSI genuinely wins ITSI is the stronger choice in several real situations, and it would be dishonest to pretend otherwise. If your operational data already lives in Splunk, ITSI sits directly on it with no extra pipeline, and its glass tables and service deep dives are a mature, polished way to present service health to both engineers and executives. Its episode and notable-event model is battle-tested for large enterprises that need a structured triage workflow with ITSM integration. And as a Cisco-owned commercial product, it ships with enterprise support and a long track record. If deep Splunk-resident analytics and executive-facing service views are the job, ITSI is hard to beat. ### Where Aurora wins Aurora wins when the job is active, cross-cloud investigation rather than Splunk-resident analytics. It is open source and self-hosted, so there is no ingest-priced licensing and no vendor lock-in, and it can run fully air-gapped with a local LLM. It is genuinely multi-cloud, querying AWS, Azure, GCP, OVH, and Scaleway directly rather than only reasoning over telemetry that has been shipped into one platform. And it executes: it runs cloud and Kubernetes commands in sandboxed pods, maps blast radius through a knowledge graph, and can open a pull request with a suggested fix, with destructive steps human-gated. For teams spread across providers, our [multi-cloud incident management guide](/blog/multi-cloud-incident-management) covers why that breadth matters. ## A note on alert routing: Aurora complements, it does not replace Aurora is an investigation layer, not an alert-routing or on-call-scheduling tool. It sits on top of whatever routing layer you run and consumes the alerts that layer forwards. ITSI handles its own event aggregation inside Splunk, but if you are assembling an open-source stack, Aurora pairs with a routing layer such as Keep, or with simple notification paths, rather than replacing your escalation tooling. Think of Aurora as the agent that investigates each incident after it has been routed, regardless of which router surfaced it. ## Which should you choose? Choose ITSI if your telemetry already lives in Splunk and your primary need is service-level KPI monitoring, glass-table dashboards, and structured episode triage, and you are comfortable with ingest-based or workload-based Splunk licensing and a closed, Cisco-owned stack. ITSI's depth on Splunk-resident data and its executive-facing service views are its strongest cards. Choose Aurora if you want an open-source, self-hosted AI SRE that actively investigates and remediates across multiple clouds without ingest-priced licensing, if air-gapped or BYO-LLM deployment is a hard requirement, or if you specifically want an agent that executes cloud and Kubernetes commands and drafts fixes rather than one that correlates and summarizes. Many teams run both: a monitoring and correlation layer for service KPIs, and an agentic investigator on top for root-cause work. For the category-level framing behind this decision, see [AI SRE vs AIOps](/blog/ai-sre-vs-aiops). For deployment-posture trade-offs, see [self-hosted AI SRE](/blog/self-hosted-ai-sre), and for the wider tooling field, [top 10 AIOps platforms offering free root cause analysis](/blog/top-10-aiops-platforms-free-root-cause-analysis-2026). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## K8sGPT Alternatives: We Tested K8sGPT on 19 Faults (2026) URL: https://www.aurorasre.ai/blog/k8sgpt-alternative-beyond-kubernetes Published: 2026-06-03 (updated 2026-10-21) Author: Ben Gervais, Engineer at Arvo AI > K8sGPT alternatives, tested: K8sGPT 0.4.40 saw nothing in 6 of 19 injected faults. Which tool fits each gap: HolmesGPT, Aurora, kagent, Robusta, Komodor. The main K8sGPT alternatives are HolmesGPT (an open-source CNCF Sandbox agent that investigates an alert across Kubernetes and other data sources), Aurora (an open-source AI SRE that investigates across clouds and drafts fixes for human approval), kagent (a CNCF Sandbox framework for building your own agents inside Kubernetes), Robusta (Prometheus alert enrichment in Slack) and Komodor's commercial AI SRE. Which one fits depends on why you are leaving K8sGPT. K8sGPT is a scanner: it lists Kubernetes objects that look unhealthy and asks an LLM to explain each one. When we ran K8sGPT 0.4.40 twice on each of 19 injected faults, it quoted the specific error behind the fault in 9, reported only a generic symptom (a container that exited with Error, a pod that was not ready) in 4, and printed "No problems detected" in both runs of the other 6, where no object it checks looked unhealthy. The rest of this page is the evidence: what `k8sgpt analyze --explain` printed for each fault, why the 6 misses happened, where K8sGPT did better than the agents, and then the alternative for each reason people move on. Every run folder is public in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab), and the full three-tool table is on the [AI SRE benchmark page](/benchmark). Disclosure: Arvo builds Aurora and wrote the 19 scenarios. Read the Aurora numbers with that in mind. ## How we tested K8sGPT We broke a small synthetic app (`shop`: nginx frontend, Python api, worker, redis) on a 3-node kind v0.33.0 cluster running Kubernetes v1.37.0, one known fault at a time, and ran K8sGPT twice per fault, on 2026-10-15 and 2026-10-16. The command, from the lab runner: ```bash k8sgpt analyze --explain --namespace shop --no-cache ``` The runner runs it twice per fault, once with `--output json` and once as text, so the recorded time covers two analyze passes. K8sGPT used the Amazon Bedrock Converse backend with Claude Sonnet 4.5. We wanted Sonnet 5, the model the other two tools ran on, but K8sGPT always sends a `temperature` parameter and Bedrock rejects it for Sonnet 5. The analyzer set was the default one: `k8sgpt filters list` on the machine that ran it, checked again on 2026-10-21, shows NetworkPolicy and Log among the unused filters. K8sGPT got no alert, because it has no notion of one. HolmesGPT 0.43.0 and Aurora (commit 87d22baf) got the alert text for each fault and were asked to find the root cause. Aurora's 38 runs are older: they ran on 2026-10-02 against the same kind cluster and the same injected faults. K8sGPT and HolmesGPT ran with the cluster admin kubeconfig (HolmesGPT's own command allow-list kept it read-only in practice); Aurora ran under its default read-only `observability_only` policy. A judge model (Claude Opus 5.5) graded every answer against the injected fault. For K8sGPT and HolmesGPT those 76 grades are judge-model only: a second model re-read all of them, a third pass graded all 76 from scratch, and a blinded regrade removed tool names. No person has checked them yet. Aurora's 38 grades were read by a person. K8sGPT is scored here on a criterion built for alert-driven investigators, which is not what it was designed to do. So the table below says what it reported rather than ranking it at 0% correct. ## What did K8sGPT find on the 19 faults? | # | Injected fault | What K8sGPT reported (both runs) | Judge verdict | |---|---|---|---| | 01 | Worker OOMKilled after a batch size change | `OOMKilled exitCode=137 container=worker` | partial, partial | | 02 | Log-shipper sidecar OOMKilled, app container healthy | `OOMKilled exitCode=137 container=log-shipper` | partial, partial | | 03 | Frontend rollout stuck on a mistyped image tag | `nginx:1.27-alpne: not found`, and suggested `nginx:1.27-alpine` | partial, partial | | 04 | New api pods blocked on a missing Secret | `secret "api-secrets" not found` | partial, partial | | 05 | Frontend crash loop from an invalid nginx config | Symptom only: `the last termination reason is Error container=nginx` | partial, partial | | 06 | Liveness probe targets the wrong port | No problems detected | no answer, no answer | | 07 | api Service selector changed to match no pods | `Service has no endpoints, expected label app=api-v2` | partial, partial | | 08 | Worker Pending, CPU request larger than any node | `0/3 nodes are available: ... 2 Insufficient cpu` | partial, partial | | 09 | redis Pending on a PVC with a missing StorageClass | `storageclass.storage.k8s.io "fast-ssd" not found`, among 6 findings | partial, partial | | 10 | api crash loop from a wrong `REDIS_HOST` | Symptom only: Service not ready, readiness probe refused | wrong, wrong | | 11 | Service `targetPort` points where nothing listens | No problems detected | no answer, no answer | | 12 | Two faults: worker OOM and a frontend Service selecting no pods | Both: the OOMKill and `Service has no endpoints, expected label tier=web` | partial, partial | | 13 | NetworkPolicy blocks api to redis | No problems detected | no answer, no answer | | 14 | Rollout stuck on an exhausted ResourceQuota | `exceeded quota: compute-resources` | partial, partial | | 15 | A stray Deployment's pods share the `app=api` label | No problems detected | no answer, no answer | | 16 | Pod `dnsConfig` search domains for another cluster domain | Symptom only: readiness probe connection refused | wrong, wrong | | 17 | Slow memory leak, worker OOMKilled every few minutes | No problems detected | no answer, no answer | | 18 | Init container waits for a Service that does not exist | Symptom only: `Service has not ready endpoints` | partial, wrong | | 19 | ConfigMap edit loaded by one of two frontend pods | No problems detected | no answer, no answer | Totals over 38 runs: 21 partial, 5 wrong, 12 "No problems detected", 0 correct. The first model review agreed with all 76 grades. The third pass, which graded from scratch, disputed 6 K8sGPT labels (both runs of 05, 10 and 18) and called them no answer, because the text restates a symptom without a cause. The blinded regrade moved two K8sGPT runs down as well (r1 of 05 to no answer, r1 of 18 to wrong). We kept the judge's labels, which are the more generous reading for K8sGPT. **Run time.** The median was 12.5 seconds for the two passes. Runs with findings took 8 to 46 seconds; the 12 clean scans took 0 to 1 second, because there was nothing to send to the model. HolmesGPT's median was 45 seconds for one investigation and Aurora's 153.5 seconds. ### Why "partial" and not "correct" In the 9 faults where K8sGPT quoted the specific error, the error string was right and often exact. What it did not do was say why the object broke or how findings connect. Three examples from the grade files: - **01, worker OOM.** It reported the OOMKill and suggested raising the memory limit (with 2Gi as the example in one run), but never mentioned the `BATCH_SIZE` change from 500 to 2500 that caused it. - **09, redis PVC.** The output was a flat list of six findings. It never linked the api crash loop to redis being down, which was the point of the scenario. - **07, selector mismatch.** It treated `app=api-v2` as correct and suggested relabeling pods instead of reverting the selector. In the same run it flagged an unrelated redis ClusterIP repair event and suggested `kubectl delete svc redis -n shop`, which is noise at best. That is the scanner design working as documented. The [K8sGPT README](https://github.com/k8sgpt-ai/k8sgpt) describes it as "a tool for scanning your Kubernetes clusters, diagnosing, and triaging issues in simple English", with "SRE experience codified into its analyzers". Each analyzer checks one resource type, and the LLM explains one finding at a time. ## Which faults can K8sGPT not see? In 4 of the 6 misses (11, 13, 15, 19), no object in the namespace was in a failing state that a default analyzer checks. The other 2 (06, 17) are less clear-cut: - **11, wrong `targetPort`.** The api pods were Running and Ready and the Service had endpoints. Every connection was refused anyway. ([runbook](/runbooks/fix-service-targetport-mismatch-kubernetes)) - **13, NetworkPolicy.** Every pod was Running and Ready; only the api to redis path was blocked. The NetworkPolicy analyzer is optional and was off. We did not rerun with it, so we do not know whether it would have flagged this policy. ([runbook](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes)) - **15, stray pods sharing a label.** The stray pod passed its `/healthz` readiness check and returned 500 on every other path. With three pods behind the Service, about one request in three failed. ([runbook](/runbooks/fix-service-selects-wrong-pods-label-overlap-kubernetes)) - **19, partial config reload.** Both frontend pods were Running. One had started with the edited ConfigMap, the other still served the old config from memory. ([runbook](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes)) - **06, liveness probe on the wrong port, and 17, slow memory leak.** The containers were being killed repeatedly, yet all four scans reported nothing. Between kills the container is up again (in 06 readiness passes between restarts). We did not capture pod state at the moment of each scan, so we cannot say which check skipped it. ([liveness runbook](/runbooks/fix-liveness-probe-failed-kubernetes), [memory leak runbook](/runbooks/fix-memory-leak-oomkilled-kubernetes)) The two alert-driven agents did better on these 6 faults, because they start from the alert (for 15, "31% of requests to /api/ returned 5xx"; for 06 and 17, a crash-loop alert) and follow it to the cause. All 12 runs per tool: | Faults K8sGPT reported nothing on (12 runs) | K8sGPT | HolmesGPT 0.43.0 (judge-graded) | Aurora (human-reviewed) | |---|---|---|---| | Correct root cause | 0 | 9 | 10 | | Partial | 0 | 2 (both runs of 11) | 2 (one run each of 11 and 19) | | Wrong | 0 | 1 (one run of 19) | 0 | | "No problems detected" | 12 | 0 | 0 | Faults outside the cluster are a harder limit. Every built-in K8sGPT analyzer reads Kubernetes objects; of the optional integrations, the AWS one adds a single EKS check that reads the cluster's own health issues from the EKS API. A saturated managed database or a cloud load balancer only shows up through whatever Kubernetes symptom it causes. Our lab does not include out-of-cluster faults, so we have no measured number for that case. ## Where K8sGPT did better than the agents Scenario 12 had two faults at once, and the alert mentioned only the worker. K8sGPT reported the frontend Service selecting no pods in both runs. Aurora and HolmesGPT missed that second fault in all four of their runs: they explained the worker OOM and never checked Services or endpoints. Both Aurora RCAs went further and stated that the frontend was healthy and unaffected, which was false. A scan that does not care about the alert covers the whole namespace. ([runbook](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes)) It is also fast and cheap: a clean scan took about a second and had nothing to send to the model, and the exact strings it pulled (`nginx:1.27-alpne: not found`, `secret "api-secrets" not found`, `exceeded quota`) are what you would want pasted into a ticket. ## When is K8sGPT the right tool? Keep K8sGPT when the job is "tell me what is unhealthy in this cluster right now" rather than "explain this alert". Specifically: - **Continuous posture checks.** The [k8sgpt-operator](https://github.com/k8sgpt-ai/k8sgpt-operator) re-runs analysis on an interval that defaults to 30 seconds and can send results to a Slack sink. - **Catching faults nobody alerted on**, as scenario 12 showed. - **A low footprint and strict data handling.** It is a single CLI, a [CNCF Sandbox project since 19 December 2023](https://www.cncf.io/projects/k8sgpt/), Apache 2.0, and supports local backends such as Ollama and LocalAI. Its `--anonymize` option masks object names and labels before they reach the AI backend. - **Giving an assistant cluster tools.** `k8sgpt serve --mcp` starts an MCP server that the README says provides 12 tools for clients like Claude Desktop. Nothing stops you from running K8sGPT next to an investigator instead of replacing it; scenario 12 is the argument for doing that. ## Which K8sGPT alternative fits your reason for leaving? | Why you are leaving K8sGPT | Alternatives to look at | What their docs say (checked 2026-10-21) | |---|---|---| | You want an investigation of an alert, not a scan | HolmesGPT, Aurora, Komodor AI SRE | HolmesGPT and Aurora both start from an alert; see the benchmark numbers below | | You need sources beyond Kubernetes or more than one cloud | HolmesGPT, Aurora | HolmesGPT lists AWS, Azure, GCP, Datadog, Prometheus, databases and more as toolsets; Aurora runs CLI work across AWS, Azure, GCP, OVH, Scaleway, Fly.io and Kubernetes | | You want remediation | Aurora, HolmesGPT, Robusta, Komodor AI SRE | All four describe fixes that are proposed or rule-driven; details per tool below | | You want a chat or Slack interface | Robusta, HolmesGPT, kagent, Aurora | Robusta delivers enriched alerts to Slack and Teams; kagent has a web UI for chatting with agents | | You want to build your own agents on Kubernetes | kagent | A Kubernetes-native agent framework with declarative YAML agents | ### HolmesGPT The [HolmesGPT README](https://github.com/HolmesGPT/holmesgpt) describes an "Open-source AI agent for investigating production incidents and finding root causes", and states that it is a CNCF Sandbox project originally created by Robusta, with major contributions from Microsoft. Its built-in toolsets include Prometheus, Grafana, Datadog, AWS, Azure, GCP, Elasticsearch, Kafka, SQL databases and GitHub. The README states "By design, HolmesGPT has **read-only access** and respects RBAC permissions", and it also lists an optional Kubernetes Remediation toolset for scaling, rollbacks and resource edits, plus an Operator mode that runs in the background, messages you in Slack and, with the GitHub integration, "can even open PRs to fix what it finds." Slack is available via Robusta. In our lab, on the same 19 faults with the same model as Aurora, HolmesGPT was correct in 31 of 38 runs, partial in 6 and wrong in 1. Those grades are judge-model only (second-model review and blinded regrade, no person yet). Its median time was 45 seconds against Aurora's 153.5, about 3x faster. If speed of a Kubernetes-plus-observability investigation is what you want, it is the first one to try. For more on the two CNCF tools, see [HolmesGPT vs K8sGPT](/blog/holmesgpt-vs-k8sgpt). ### Aurora [Aurora](https://github.com/Arvo-AI/aurora) is an Apache 2.0, self-hosted AI SRE (Docker Compose or Helm). It takes alerts by webhook from tools such as PagerDuty, Datadog and Grafana, investigates across AWS, Azure, GCP, OVH, Scaleway, Fly.io and Kubernetes, and writes a root cause analysis with a separate list of suggested next steps. It can open a remediation PR on GitHub after a person approves it; it does not auto-merge. It works with your own LLM, including local models through Ollama, and has a Slack integration and an MCP server. In the lab Aurora was correct in 34 of 38 runs and partial in 4, all read by a person, at a median of 153.5 seconds and $0.35 per investigation. Against HolmesGPT's 31, that gap is not significant (two-sided Fisher exact test on correct against not correct, p = 0.52, the same calculation the [benchmark page](/benchmark) runs on these totals), so read the two as roughly even on accuracy, with HolmesGPT faster. The prompts differed slightly: Aurora got the alert plus "Investigate and find the root cause", HolmesGPT the alert plus "Find the root cause and say what changed". Every Aurora trace and miss is in the [lab report](/blog/aurora-lab-report-kubernetes-19-failures). ### kagent [kagent](https://github.com/kagent-dev/kagent) is "a Kubernetes native framework for building AI agents", a [CNCF Sandbox project since 22 May 2025](https://www.cncf.io/projects/kagent/). Agents are defined in YAML and managed with `kubectl`. It ships an MCP server with tools for Kubernetes, Istio, Helm, Argo, Prometheus, Grafana and Cilium, supports OpenAI, Azure OpenAI, Anthropic, Vertex AI and Ollama, and exposes agents through a web UI, a CLI and gRPC, A2A and MCP APIs. It is a framework rather than a finished incident investigator, so it suits teams that want to write their own agents. We did not test it. ### Robusta [Robusta](https://github.com/robusta-dev/robusta) (MIT) takes Prometheus alerts and adds pod logs, graphs and possible remediations, then delivers them to Slack, Microsoft Teams and other sinks with grouping into threads. It supports self-healing through auto-remediation rules you define, and HolmesGPT as optional AI investigation. A free hosted Robusta UI adds an alert timeline across clusters. If your complaint about K8sGPT is that findings do not reach the on-call channel with context, this is the closest fit. We did not test it. ### Komodor AI SRE Komodor's [incident management module](https://docs.komodor.com/modules-and-workflows/sre/incidents) is part of its commercial AI SRE family. Its docs say observability, Kubernetes and cloud investigators gather evidence in parallel, and that it will "Propose the fix and route it for approval before anything is applied", with optional Slack posting. We did not test it and have no numbers for it. ## How do K8sGPT, HolmesGPT and Aurora compare on governance? | | K8sGPT 0.4.40 | HolmesGPT 0.43.0 | Aurora | |---|---|---|---| | Input | Scans a namespace | Alert or question | Alert by webhook | | Scope | Kubernetes resources | Kubernetes plus listed toolsets | Kubernetes plus 6 clouds' CLIs | | Writes | Operator auto-remediation is [alpha and off by default](https://github.com/k8sgpt-ai/k8sgpt-operator/blob/main/AUTO_REMEDIATION.md), image replacement only | Read-only by design; optional remediation toolset and PRs | PRs only after human approval, no auto-merge | | Lab result (38 runs) | Not scored as a peer: 21 partial, 5 wrong, 12 nothing found | 31 correct, 6 partial, 1 wrong (judge only) | 34 correct, 4 partial (human-reviewed) | | Median time | 12.5 s (two passes) | 45 s | 153.5 s | The K8sGPT operator's own docs say "Status: Alpha. Auto remediation is experimental and must not be relied on for production changes without independent review." For a deeper three-way comparison, see [open-source AI SRE: Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt). ## What did we not test? We did not test kagent, Robusta or Komodor. We did not run K8sGPT with optional analyzers (NetworkPolicy, Log) enabled, on Claude Sonnet 5, or with the operator. The lab has no faults outside Kubernetes. All 19 scenarios were written by Arvo. Rerun any of it from [the lab repo](https://github.com/Arvo-AI/aurora-lab), or paste your own `kubectl describe` output into the [Kubernetes pod diagnoser](/diagnose) to see a likely cause. ## Sources and verification K8sGPT and HolmesGPT runs on 2026-10-15 and 2026-10-16, run folders under `runs/h2h-v1/r1` and `r2`; Aurora runs on 2026-10-02, under `runs/v0-run2` and `runs/v1-run1`, all in [aurora-lab](https://github.com/Arvo-AI/aurora-lab). Numbers from the benchmark results file generated 2026-10-21. Tool docs checked 2026-10-21: [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt), [k8sgpt-operator](https://github.com/k8sgpt-ai/k8sgpt-operator), [HolmesGPT](https://github.com/HolmesGPT/holmesgpt), [kagent](https://github.com/kagent-dev/kagent), [Robusta](https://github.com/robusta-dev/robusta), [Komodor docs](https://docs.komodor.com/modules-and-workflows/sre/incidents), [CNCF K8sGPT](https://www.cncf.io/projects/k8sgpt/), [CNCF kagent](https://www.cncf.io/projects/kagent/). If Aurora fits, it is on [GitHub](https://github.com/Arvo-AI/aurora), or [book a demo](https://cal.com/arvo-ai/demo). --- ## New Relic AI Alternative: Open Source RCA (2026) URL: https://www.aurorasre.ai/blog/new-relic-ai-alternative-open-source Published: 2026-06-03 (updated 2026-07-02) Author: Noah Casarotto-Dinning, CEO at Arvo AI > New Relic AI is low-friction for New Relic-native shops. Aurora is open source, self-hosted, multi-cloud, and runs active investigations and remediations. > ## Key Takeaways > > - **New Relic AI reasons over telemetry already ingested into New Relic, and it is SaaS-only.** Its [AIOps and Applied Intelligence layer](https://newrelic.com/platform/applied-intelligence) does anomaly detection, alert correlation, noise reduction, and 'precise causal analysis that identifies root cause' across data inside the New Relic platform. The newer [SRE Agent, announced 24 February 2026](https://newrelic.com/press-release/20260224), adds 'always on' diagnostics, root cause analysis, and post-incident reports. > - **New Relic deliberately does not own remediation.** Per [TechTarget's reporting](https://www.techtarget.com/searchitoperations/news/366639362/New-Relic-plans-to-expand-AI-agent-observability), 'New Relic's tools won't include an AI agent control plane to control incident remediation that overlaps with those partner tools' such as ServiceNow, Atlassian, and PagerDuty. The SRE Agent diagnoses and recommends; humans and partner tools act. > - **Aurora is open source under [Apache 2.0](https://github.com/Arvo-AI/aurora) and self-hosted.** Its [LangGraph](https://github.com/langchain-ai/langgraph)-orchestrated agents both investigate AND execute: they run kubectl, aws, az, and gcloud in sandboxed Kubernetes pods, with destructive actions human-gated. > - **Aurora is multi-cloud and vendor-neutral.** It queries AWS, Azure, GCP, OVH, Scaleway, and Kubernetes natively, builds a [Memgraph](https://memgraph.com/) infrastructure knowledge graph for blast radius, and is bring-your-own-LLM, including local models via [Ollama](https://ollama.com/) for air-gapped use. > - **Aurora ingests from your existing monitoring.** It accepts alert webhooks from eleven connectors (PagerDuty, Datadog, Grafana, New Relic, OpsGenie, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, incident.io) plus a Slack bot, so you can keep your current stack and add an open investigation layer on top. > - **New Relic pricing is consumption-based.** The published model bills [$0.40 per GB ingested (Original Data) or $0.60 per GB (Data Plus)](https://newrelic.com/pricing) plus per-user seats, after a free tier of 100 GB and one full platform user. Aurora is free under Apache 2.0; you pay only for your own infrastructure and LLM usage. New Relic AI is a convenient choice if your telemetry already lives in New Relic. The honest tradeoff: it reasons over New-Relic-ingested data, runs only as SaaS, and is built to diagnose and assist rather than to execute fixes. Aurora takes the opposite shape: open source, self-hosted, vendor-neutral, multi-cloud, and capable of running active investigations and human-gated remediations. This guide compares the two on the dimensions that actually drive a procurement decision, with every claim linked to a primary source. For the broader category framing, see our explainer on [AI SRE vs AIOps](/blog/ai-sre-vs-aiops). For the full open-source landscape, see our roundup of [the top AIOps platforms with free root cause analysis in 2026](/blog/top-10-aiops-platforms-free-root-cause-analysis-2026). ## What is New Relic AI? New Relic AI is the artificial-intelligence layer inside the New Relic observability platform, spanning the older Applied Intelligence and AIOps features and the newer SRE Agent. It is a SaaS product that analyzes telemetry already ingested into New Relic and helps engineers detect, correlate, and understand incidents faster. The [Applied Intelligence and AIOps capabilities](https://newrelic.com/platform/applied-intelligence) are correlation and assist centric. New Relic describes them as connecting 'similar alert events into a single issue for simpler troubleshooting,' reducing 'mean time to resolve (MTTR) with precise causal analysis that identifies root cause,' and identifying 'root cause in seconds by correlating logs, metrics, traces, and changes.' The framing throughout is noise reduction, expert routing, and recommended next steps rather than self-healing. On 24 February 2026, New Relic [announced an SRE Agent](https://newrelic.com/press-release/20260224) that pushes further into investigation. It is positioned as an 'always on' AI teammate that performs 'deep, full-stack diagnostics,' delivers root cause analysis, impact assessments, and comprehensive post-incident reports, and surfaces 'a unified view of the evolving timeline of events.' It works through Slack and Zoom so responders can query New Relic from triage rooms, and it uses Intelligent Root Cause Analysis, which scores an entity's topology graph with probabilistic causal models. New Relic was also [named a Leader in the IDC MarketScape: Worldwide AIOps 2026 Vendor Assessment](https://newrelic.com/press-release/20260324) (Doc #US54116226, March 2026), and in May 2026 it [introduced New Relic Knowledge](https://www.helpnetsecurity.com/2026/05/06/new-relic-knowledge-capability/), which connects 'telemetry, documentation, and historical incidents to deliver context aware and trusted insights in real time.' Two structural facts matter for buyers. First, New Relic AI reasons over data that has been ingested into New Relic, and that ingestion is what the pricing meter counts. Second, New Relic has consciously chosen not to own remediation. As [CPO Brian Emerson told TechTarget](https://www.techtarget.com/searchitoperations/news/366639362/New-Relic-plans-to-expand-AI-agent-observability), the company will continue to focus on 'agentic integrations with partners' rather than directly automating the software delivery lifecycle, and 'New Relic's tools won't include an AI agent control plane to control incident remediation that overlaps with those partner tools' from ServiceNow, Atlassian, and PagerDuty. ## What is Aurora? Aurora is an open-source AI SRE and incident-management platform that both investigates incidents and executes human-gated remediations across multiple clouds. It is built by Arvo AI, licensed under [Apache 2.0](https://github.com/Arvo-AI/aurora), and designed to be self-hosted. When an alert fires, Aurora's [LangGraph](https://github.com/langchain-ai/langgraph)-orchestrated agents autonomously investigate. They query AWS, Azure, GCP, OVH, Scaleway, and Kubernetes, and they run real commands, kubectl, aws, az, and gcloud, inside sandboxed Kubernetes pods. Aurora builds a [Memgraph](https://memgraph.com/) infrastructure knowledge graph to trace blast radius across services and cloud providers, generates structured root-cause analyses and postmortems that export to Confluence, Notion, or SharePoint, and can suggest code fixes or open pull requests. Destructive actions are gated behind human approval. Aurora is bring-your-own-LLM. You can point it at a hosted frontier model or run local inference through [Ollama](https://ollama.com/) for air-gapped deployments where no data leaves your environment. Crucially, Aurora does not require you to migrate your monitoring stack. It ingests alerts by webhook from eleven connectors, PagerDuty, Datadog, Grafana, New Relic, OpsGenie, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, and incident.io, plus a Slack bot, so it sits on top of whatever you already run. For the deployment and data-sovereignty story, see our guide to [self-hosted AI SRE](/blog/self-hosted-ai-sre). ## New Relic AI vs Aurora: head-to-head The clearest way to separate these tools is by what triggers them, what they reason over, and whether they can act. New Relic AI is triggered by telemetry inside New Relic and is built to diagnose and recommend. Aurora is triggered by an alert from any of your monitors and is built to investigate live infrastructure and then execute approved fixes. **Where they overlap.** Both detect or correlate incidents, both produce root-cause analysis with an evidence trail, both generate post-incident reports, and both keep a human in the loop before consequential action. If you only need faster diagnosis on data you already send to New Relic, the two converge on outcomes. **Where they diverge.** - **Data boundary.** New Relic AI reasons over data ingested into New Relic; the more you want it to reason about, the more you ingest, and ingestion is metered. Aurora reaches directly into your cloud APIs and clusters at investigation time, so it can inspect infrastructure state that was never shipped to an observability backend. - **Action.** New Relic deliberately stops at diagnosis and recommendation, deferring remediation to [partner tools](https://www.techtarget.com/searchitoperations/news/366639362/New-Relic-plans-to-expand-AI-agent-observability). Aurora runs commands and opens PRs, with destructive steps human-gated. - **Deployment.** New Relic AI is SaaS-only. Aurora is self-hosted via Docker Compose or Helm and can run fully air-gapped. - **Cloud breadth.** New Relic AI is cloud-agnostic for telemetry but does not query your clouds as an agent. Aurora natively investigates five clouds plus Kubernetes. See our overview of [multi-cloud incident management](/blog/multi-cloud-incident-management). - **Cost model.** New Relic AI rides on [consumption pricing](https://newrelic.com/pricing). Aurora is free software; your cost is your own compute plus optional LLM spend. ## Comparison table | Dimension | New Relic AI | Aurora | |---|---|---| | License | Proprietary, part of the [New Relic platform](https://newrelic.com/platform/applied-intelligence) | Open source, [Apache 2.0](https://github.com/Arvo-AI/aurora) | | Deployment | SaaS-only, hosted by New Relic | Self-hosted (Docker Compose or Helm), air-gap capable | | Multi-cloud | Cloud-agnostic for telemetry; does not query clouds as an agent | Native agent queries to AWS, Azure, GCP, OVH, Scaleway, Kubernetes | | Investigation vs correlation | Correlation and assist centric; [SRE Agent](https://newrelic.com/press-release/20260224) adds diagnostics and RCA | Agentic investigation that runs live tool calls against infrastructure | | Write / execute actions | Diagnoses and recommends; [defers remediation to partners](https://www.techtarget.com/searchitoperations/news/366639362/New-Relic-plans-to-expand-AI-agent-observability) | Executes kubectl, aws, az, gcloud in sandboxed pods; destructive steps human-gated | | Pricing model | Consumption: [$0.40 to $0.60 per GB ingested](https://newrelic.com/pricing) plus per-user seats, after a free 100 GB tier | Free software; pay only for your own infra and LLM usage | | LLM choice | New Relic-managed AI | Bring-your-own-LLM, including local via [Ollama](https://ollama.com/) for air-gapped use | | Monitoring ingest | Native to New Relic telemetry | Webhooks from 11 connectors (PagerDuty, Datadog, Grafana, New Relic, OpsGenie, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, incident.io) plus a Slack bot | ## How much does New Relic AI cost? New Relic does not publish a per-investigation or per-agent rate; its AI features are bundled into a consumption-based platform price. The two metered dimensions are data ingest and users. Per the [official pricing page](https://newrelic.com/pricing), the free tier includes 100 GB of monthly data ingest and one full platform user. Beyond that, ingest is billed at $0.40 per GB on Original Data or $0.60 per GB on Data Plus, with an EU data-center add-on of $0.05 per GB. Full platform users cost $10 for the first user and $99 per additional user on Standard (maximum five), and $349 per user per year on Pro annual commitments, with $418.80 per user for monthly pay-as-you-go. Core users are $49 each and basic users are free. The [billing documentation](https://docs.newrelic.com/docs/accounts/accounts-billing/new-relic-one-pricing-billing/new-relic-one-pricing-billing/) states plainly: 'We bill for data by the amount of GB Ingested, no matter the type.' The practical consequence is that the value New Relic AI extracts is bounded by what you ingest, and what you ingest is what you pay for. There is no public per-investigation rate to compare against. Aurora has no license cost at all; its only spend is the infrastructure you self-host it on plus whatever LLM usage you choose, which can be zero with a local model. ## Which should you choose? Choose New Relic AI if your telemetry already lives in New Relic and you want a low-friction, managed add-on. It is a strong fit when you are already paying for New Relic, want anomaly detection, correlation, and diagnostics without standing up new infrastructure, and are comfortable letting partner tools and humans own remediation. Being [named a Leader in the IDC MarketScape Worldwide AIOps 2026 assessment](https://newrelic.com/press-release/20260324) reflects a mature, well-supported platform, and the [SRE Agent](https://newrelic.com/press-release/20260224) is a genuine step into agentic diagnosis for New Relic-native shops. Choose Aurora if open source, self-hosting, multi-cloud breadth, or autonomous execution matter to you. Aurora is the better fit when you need to investigate cloud and Kubernetes state directly rather than only data you have already shipped to a vendor, when you want the agent to execute approved fixes rather than just recommend them, when you require air-gapped or data-sovereign deployment, or when consumption-based ingest pricing is hard to predict at your volume. Because Aurora ingests from eleven monitoring connectors, you can keep your existing alerting and add it as a vendor-neutral investigation and remediation layer. For the conceptual contrast between correlation-style AIOps and agentic AI SRE, see our [AI SRE vs AIOps explainer](/blog/ai-sre-vs-aiops). Many teams will run both. New Relic AI can stay the observability and correlation home for telemetry that already lives there, while Aurora handles cross-cloud investigation and human-gated remediation on top of the alert stream. The honest summary: New Relic AI is the convenient option for New Relic-native shops, and Aurora is the open, executable option for everyone who needs vendor neutrality, multi-cloud reach, or the ability to act. ## Getting started with Aurora Aurora is on GitHub at [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) under Apache 2.0. Clone the repo, deploy with Docker Compose or the Helm chart, point your monitoring webhooks at it, and add your cloud credentials. Investigations begin automatically when alerts arrive. If you want a connector or capability that does not exist yet, the Arvo AI team builds custom integrations with users at no cost. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## Moogsoft (Dell APEX) Alternative: Open Source AIOps URL: https://www.aurorasre.ai/blog/moogsoft-dell-apex-alternative-open-source Published: 2026-06-03 (updated 2026-07-02) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Dell APEX AIOps (formerly Moogsoft) is a closed, contract-gated correlation engine. Aurora is an open source, self-hosted AI SRE alternative. > ## Key Takeaways > > - Moogsoft is not discontinued. It was [acquired by Dell in 2023](https://siliconangle.com/2023/07/20/dell-acquires-venture-backed-aiops-startup-moogsoft/) and lives on as [Dell APEX AIOps Incident Management](https://www.dell.com/en-us/blog/it-operations-needs-a-new-wardrobe/), an actively maintained enterprise product. > - Dell calls Moogsoft 'the original AIOps pioneer' and the platform ships with a [portfolio of more than 50 patents](https://www.dell.com/en-us/blog/it-operations-needs-a-new-wardrobe/) built on statistical machine learning for event correlation and noise reduction. > - Dell APEX is closed source, sold through Dell contracts and service agreements, and the related Infrastructure Observability tier is [gated behind ProSupport contracts](https://www.dell.com/en-us/dt/apex/aiops.htm). There is no public per-seat or per-incident price list. > - Aurora by Arvo AI is [open source under Apache 2.0](https://github.com/Arvo-AI/aurora), self-hosted, air-gap capable, and free to run. > - The deepest difference is approach: Dell APEX correlates and reduces noise, while Aurora runs [LangGraph agents](https://github.com/Arvo-AI/aurora) that actively investigate across clouds and execute read-only commands in sandboxed pods. > - Aurora ingests alerts from 11 monitoring connectors plus a Slack bot, so it can layer AI investigation on top of whatever routing tool you already use. If you have evaluated Moogsoft over the years, you know it as the company that popularized machine-learning event correlation. It is still here, now under Dell, and it is still good at what it does. The honest question for a 2026 SRE team is not whether Dell APEX AIOps works. It is whether a closed, contract-gated correlation engine is the right shape for how incident response is changing, and whether an open source agent that investigates and acts is a better fit for your stack and your budget. This post compares Dell APEX AIOps Incident Management (formerly Moogsoft) with [Aurora](https://github.com/Arvo-AI/aurora), an open source AI SRE platform. For broader context on where correlation engines sit relative to AI SRE tooling, see our explainer on [AI SRE versus AIOps](/blog/ai-sre-vs-aiops). ## What is Moogsoft, now Dell APEX AIOps Incident Management? Dell APEX AIOps Incident Management is the current name for Moogsoft, the AIOps platform Dell [acquired in 2023](https://siliconangle.com/2023/07/20/dell-acquires-venture-backed-aiops-startup-moogsoft/). It is an actively maintained, enterprise-grade event correlation and incident intelligence product, not a discontinued tool. Moogsoft launched in 2012 and raised [more than 92 million dollars](https://siliconangle.com/2023/07/20/dell-acquires-venture-backed-aiops-startup-moogsoft/) before Dell acquired it. Dell folded it into the APEX AIOps family alongside Infrastructure Observability (formerly CloudIQ) and Application Observability. In Dell's own words, Moogsoft is 'the original AIOps pioneer,' and the Incident Management product carries [a portfolio of more than 50 patents](https://www.dell.com/en-us/blog/it-operations-needs-a-new-wardrobe/). What it does well is genuine. The platform ingests operational data from a wide range of multivendor monitoring tools and applies supervised and unsupervised machine learning, deduplication, and correlation to turn hundreds of thousands of raw events into a handful of actionable incidents. Dell cites customer research showing [a 99 percent-plus reduction in event noise, a 50 percent-plus reduction in service tickets, and a 93 percent reduction in customer-reported issues](https://www.dell.com/en-us/blog/it-operations-needs-a-new-wardrobe/). For a large NOC drowning in alerts across many vendors, that noise reduction is the core value, and it is real. The product is documented under the [Moogsoft Cloud documentation site](https://docs.moogsoft.com/moogsoft-cloud/en/index-en.html), which Dell now operates. It is a cloud SaaS, sold and supported through Dell. ## What is Aurora? Aurora is an open source AI SRE platform that autonomously investigates incidents and can execute remediation steps, rather than only correlating and reducing alert noise. It is [licensed under Apache 2.0](https://github.com/Arvo-AI/aurora), self-hosted, and free. When an alert fires, Aurora's [LangGraph-orchestrated agents](https://github.com/Arvo-AI/aurora) start an investigation. They query cloud provider APIs across AWS, Azure, GCP, OVH, Scaleway, and Kubernetes, and they run read-only **kubectl**, **aws**, **az**, and **gcloud** commands inside sandboxed Kubernetes pods. Aurora builds a Memgraph infrastructure knowledge graph to reason about blast radius across services, generates a root-cause analysis and a postmortem you can export to Confluence, Notion, or SharePoint, and can suggest code fixes or open a pull request. Destructive actions are human-gated. Aurora ingests alerts via webhook from 11 monitoring connectors: PagerDuty, Datadog, Grafana, New Relic, OpsGenie, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, and incident.io, plus a Slack bot for collaboration. Because it is self-hosted and supports bring-your-own-LLM through Ollama, it can run fully air-gapped with no data leaving your environment. For a deeper look at running this kind of tooling in your own infrastructure, see our guide to [self-hosted AI SRE](/blog/self-hosted-ai-sre). ## Correlation versus investigation: the core difference The honest head-to-head is not feature checklists, it is approach. Dell APEX AIOps is a correlation engine: it is excellent at compressing event noise into incidents. Aurora is an investigation engine: it takes an incident and actively goes and looks at your infrastructure to find why it broke. Both reduce toil, but at different stages of the incident lifecycle. Dell APEX answers 'which alerts belong to the same problem' using patented statistical machine learning. Aurora answers 'what is actually wrong and what should we do about it' by querying clouds, traversing a dependency graph, and running diagnostic commands. The two are not mutually exclusive in theory, but in practice teams adopting agentic investigation tend to want the openness and execution that a closed correlation product does not offer. We cover this distinction in depth in [AI SRE versus AIOps](/blog/ai-sre-vs-aiops). A second difference is openness. Dell APEX is proprietary and sold through Dell. Aurora is auditable source code you can read, fork, and self-host. For regulated or sovereign environments, the ability to run entirely on your own hardware with local models is often the deciding factor. ## Dell APEX AIOps versus Aurora at a glance | Dimension | Dell APEX AIOps Incident Management (formerly Moogsoft) | Aurora by Arvo AI | | --- | --- | --- | | License | Proprietary, Dell-owned | [Open source, Apache 2.0](https://github.com/Arvo-AI/aurora) | | Deployment | Cloud SaaS, [operated by Dell](https://docs.moogsoft.com/moogsoft-cloud/en/index-en.html) | Self-hosted via Docker Compose or Helm | | Multi-cloud scope | Vendor-neutral event ingestion across multivendor tools | Native queries across AWS, Azure, GCP, OVH, Scaleway, Kubernetes | | Core approach | [Patented statistical ML correlation and noise reduction](https://www.dell.com/en-us/blog/it-operations-needs-a-new-wardrobe/) | LangGraph agents that investigate root cause | | Write and execute actions | Workflow automation and remediation triggers | Runs read-only CLI in sandboxed pods, suggests fixes, opens PRs, human-gated writes | | Pricing model | Dell contract and service agreement, no public per-seat or per-incident rate | Free software, you pay only for infrastructure and LLM usage | | Self-host and air-gap | Cloud SaaS only | Self-hosted, air-gap capable with [local LLMs](https://github.com/Arvo-AI/aurora) | ## How does Dell APEX AIOps pricing work? Dell APEX AIOps is sold through Dell contracts, and there is no public per-seat or per-incident price list to quote. Pricing is handled as a Dell service offering with its own [service agreement and terms](https://www.dell.com/en-us/lp/legal/apex-ai-ops-incident-management). The related Infrastructure Observability tier (formerly CloudIQ) is [included at no additional cost for customers with active Dell ProSupport or higher service contracts](https://www.dell.com/en-us/dt/apex/aiops.htm), which is a genuine benefit if you already run Dell hardware. The Incident Management product itself, the part that came from Moogsoft, is a separate commercial SaaS subscription. Because the published terms route through Dell sales and support agreements rather than a self-service price page, budgeting requires a sales conversation, and the lock-in is to the Dell ecosystem and ProSupport contract structure. Aurora's pricing model is different in kind, not degree. The software is free under Apache 2.0. Your only costs are the infrastructure you run it on and your LLM usage, and if you use local models through Ollama, the LLM cost can be zero. For a wider survey of free and open source options, see our roundup of the [top AIOps platforms with free root cause analysis](/blog/top-10-aiops-platforms-free-root-cause-analysis-2026). ## A note on routing: Aurora does not replace your on-call tool Aurora is an investigation layer, not an alert router or on-call scheduler. It does not replace tools like PagerDuty or Grafana OnCall. It sits on top of them. This matters in 2026 because the routing landscape shifted. The open source [Grafana OnCall project entered maintenance mode and the repository moved toward full archival on March 24, 2026](https://grafana.com/docs/oncall/latest/set-up/open-source/), with Grafana steering users to Grafana Cloud IRM. If you ran OnCall OSS for routing, you now need an alternative routing layer, for example Keep, or notifications through services like ntfy or Twilio. Aurora is complementary to whatever you choose: routing and scheduling decide who gets paged, and Aurora investigates the incident once it fires. For more on the open source incident stack, see our guide to [open source incident management](/blog/open-source-incident-management). ## Which should you choose? Choose Dell APEX AIOps if your priority is mature, patented event correlation across a large multivendor estate, especially if you already buy Dell infrastructure with ProSupport and want a vendor-managed SaaS with enterprise support. Its noise reduction track record is well documented and its correlation engine is a known quantity. Choose Aurora if you want an open source, self-hosted agent that does not just correlate but actively investigates and can execute, if you need multi-cloud breadth beyond a single vendor, if data sovereignty or air-gapped operation rules out a cloud SaaS, or if a Dell contract and ProSupport-gated model does not fit your budget or your stack. Aurora also gives you full source transparency into how the agent reasons about your production systems. Many teams will run both for a transition period: keep a correlation layer to compress noise, and add Aurora as the open source AI investigation layer that turns a correlated incident into a root cause and a fix. If you have a feature gap, the Arvo AI team builds custom integrations with users, so [reach out](https://cal.com/arvo-ai) and they will work through it with you. ## Getting started with Aurora Aurora installs from its [GitHub repository](https://github.com/Arvo-AI/aurora) via Docker Compose or a Helm chart. Point your monitoring webhooks at it, add cloud credentials, and investigations start automatically when alerts fire. Pick any LLM provider, or run local models for fully air-gapped operation. The full deployment guides live in the project documentation. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## AI SRE vs Traditional AIOps Platforms, With a Real Trace URL: https://www.aurorasre.ai/blog/ai-sre-vs-aiops Published: 2026-05-28 (updated 2026-10-21) Author: Ben Gervais, Engineer at Arvo AI > AIOps or AI SRE? AIOps groups alerts and flags anomalies; an AI SRE investigates one alert. A full lab trace and 4 vendors' docs, checked Oct 2026. Traditional AIOps platforms work on many alerts at once: they correlate related alerts into one incident, learn baselines and flag anomalies, and cut the number of pages a person gets. An AI SRE works on one alert: an LLM agent runs read-only commands against the live system, follows what each result shows, and returns a root cause with the evidence behind it. AIOps decides what deserves attention; an AI SRE works out why it broke. Buy for the step that is slow on your team today, and run both when you have alert floods and slow investigations at the same time. The [AI SRE complete guide](/blog/ai-sre-complete-guide) has the short version of this comparison. This page shows the difference with evidence: what four AIOps products say they do in their own docs (checked 21 October 2026), one complete investigation from our lab with every command the agent ran, and the same fault seen by a scanner, K8sGPT, which lists problems instead of investigating an alert. Arvo AI builds Aurora SRE, the agent in the trace, and wrote the lab scenarios. Every run is public in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab) and the totals are at [/benchmark](/benchmark). ## What does a traditional AIOps platform actually do? Three jobs, all of them across many signals: correlation (which alerts belong to the same incident), anomaly detection (which metric is behaving unlike its baseline) and noise reduction (fewer, better pages). In each vendor's own words: | Product | Job | What the vendor's docs say | |---|---|---| | [BigPanda](https://docs.bigpanda.io/en/alert-correlation-logic--adr-) | Correlation | "Correlation patterns set rules to define relationships between system elements, which are then used to dynamically cluster alerts into incidents." Patterns use source systems, tags and a time window, and BigPanda "can correlate a maximum of 300 alerts into a single incident." | | [PagerDuty Intelligent Alert Grouping](https://docs.pagerduty.com/ai-automation/aiops/noise-reduction/alert-grouping/intelligent-alert-grouping) | Noise reduction | "automatically consolidates related alerts into a single, actionable incident, using machine learning to reduce alert noise". Part of the PagerDuty AIOps add-on or Signal Intelligence in PD Reliability Platform plans. | | [Datadog Watchdog](https://docs.datadoghq.com/watchdog/) | Anomaly detection | "Watchdog proactively computes a baseline of expected behavior for your systems, applications, and deployments. This baseline is then used to detect anomalous behavior." | | [Dynatrace Intelligence](https://docs.dynatrace.com/docs/dynatrace-intelligence) | Anomaly detection and topology RCA | "applies deterministic, causation-based analysis"; it "establishes baselines, detects anomalies, and identifies root cause with topology and dependency context." | What these have in common: the input is a stream (alerts from many monitors, metrics from many hosts), and the output is a grouped incident, an anomaly, or a ranked suspect. Two of them reach toward cause over their own data: Dynatrace reasons over its dependency graph and says it "correlates code changes, deployments, configuration, and policy updates to show what changed", and [Watchdog RCA](https://docs.datadoghq.com/watchdog/rca/) draws "causal relationships between symptoms" from APM anomalies. None of the four describes running `kubectl` against a cluster, or reading a live object, to test a theory about one incident. That is the AI SRE's job. The labels are converging at the product level. PagerDuty's docs list its SRE agent (Paige) next to AIOps; Dynatrace says its platform "can propose and run approved actions through agentic workflows and built-in agents" (marked Preview); Datadog sells [Bits Investigation](https://docs.datadoghq.com/bits_ai/bits_investigation/), an autonomous agent, next to Watchdog. So ask which job a feature does, not which category the vendor files it under. ## What does an AI SRE agent do with one alert? Here is a complete run from our lab, scenario 11, run 1, on 2 October 2026 ([raw files](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/11-noisy-worker-svc-targetport/r1)). The setup is built to look like an AIOps problem. Two changes went out at about the same time: 1. The worker was redeployed with code that logs a `DeprecationWarning` on every batch and a metrics push `ERROR` on every third ([break.sh](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/11-noisy-worker-svc-targetport)). Noisy, harmless, not on the request path. 2. The api Service's `targetPort` was changed from 8080 to 9090. Nothing listens on 9090. One alert fired: `HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes (threshold 5%).` Aurora received that text plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." It made 19 tool calls in 144 seconds, issued two or three at a time, at a model cost of $0.44: | # | Call | What came back | |---|---|---| | 1 | List open incidents | The HighErrorRate incident | | 2 | Search memory for "shop frontend" | No matches | | 3 | List connected clusters | kind-aurora-lab | | 4 | `kubectl get pods -n shop -o wide` | Every pod Running, 0 restarts | | 5 | `kubectl get events -n shop --sort-by=.lastTimestamp` | Startup events, readiness failures at startup, and the worker rolled to a new ReplicaSet 58 s earlier | | 6 | `kubectl logs -n shop --tail=100` | nginx startup, then `connect() failed (111: Connection refused)` to the api Service on 8080 and 502s | | 7 | `kubectl logs -n shop --tail=100` | `api listening on :8080`, `/healthz` returning 200 | | 8 | `kubectl describe pod -n shop` | Pod detail | | 9 | `kubectl get svc,endpoints -n shop` | api endpoints on port 9090 | | 10 | `kubectl logs -n shop --tail=100` | Same as step 7 | | 11 | `kubectl get pods -n shop -o wide` | Unchanged | | 12 | `kubectl describe svc api -n shop` | `TargetPort: 9090/TCP` | | 13 | `kubectl get deploy api -n shop -o yaml` | `containerPort: 8080` | | 14 | `kubectl logs -n shop --tail=30` | The same `connect() failed (111: Connection refused)` and 502s | | 15 | `kubectl get nodes -o wide` | Three nodes Ready | | 16 | `kubectl top pods -n shop` | Low CPU and memory | | 17 | `kubectl get svc api -n shop -o yaml` | Last applied `targetPort` 8080, live 9090 | | 18 | `kubectl logs -n shop --tail=20` | `GET /api/items -> 502` | | 19 | `kubectl logs -n shop --tail=30` | The deprecation and pushgateway noise | The frontend log at step 6 already showed the symptom: nginx refused on its way to the api Service at port 8080. Step 9 is the first output that shows the fault. The Service port is 8080 but the endpoints are on 9090: ``` NAME ENDPOINTS AGE endpoints/api 10.244.1.131:9090,10.244.2.124:9090 97s ``` Steps 12 and 13 confirmed it from both sides (`TargetPort: 9090/TCP` on the Service, `containerPort: 8080` on the Deployment), and step 14 showed the same refusal on the second frontend pod: ``` 2026/10/02 20:24:21 [error] 27#27: *49 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.2.126, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.190.97:8080/items", host: "frontend" ``` and step 17 showed how it happened: the `last-applied-configuration` annotation still says `"targetPort":8080` while the live spec says `targetPort: 9090`, so someone edited the Service after it was applied. Only then, at step 19, did the agent read the worker's logs: ``` WARNING py.warnings: DeprecationWarning: StrictRedis is deprecated and will be removed in redis-py 6.0, use Redis ERROR metrics: push to pushgateway:9091 failed: [Errno -2] Name does not resolve (will retry next batch) ``` Its report put them outside the causal chain, though under "Not Checked" rather than "Ruled Out", described only as `pushgateway:9091` DNS errors (the DeprecationWarning is not mentioned) and "not further investigated". The report's root cause: the api Service's `targetPort` "was set to 9090, while the api Deployment's containers only expose and listen on port 8080". Its suggested fix was to patch `targetPort` back to 8080. The run was graded correct, and a person (Ben Gervais) read that grade. Runbook for this failure: [Service targetPort mismatch](/runbooks/fix-service-targetport-mismatch-kubernetes). Now picture the same signals reaching a correlation layer, with the worker's ERROR lines also wired to an alert (in the lab only the 5xx alert fired). The worker noise and the frontend's 5xx started together, in the same namespace, after two changes made at about the same time. Time and topology both say "related", and grouping them would be right that they arrived together and silent on which one matters. A platform that records changes, as Dynatrace says it does, would see two: a worker rollout and a Service edit. Ruling the worker out took a different kind of evidence: endpoints, ports and a config annotation. That is our reading of how the four products describe themselves, not something we tested on them. The same caution applies to us: Aurora's own alert correlation weighs topology and time, and this scenario never exercised it because only one alert fired. ## What did the agent get wrong in that run? The answer was right; parts of the report were not. - It called the worker's errors "a separate, pre-existing cosmetic issue". They were not pre-existing: the worker had been redeployed at about the same time, and the events it read at step 5 show the new worker ReplicaSet created 58 seconds earlier. The grading caught it. - It said frontend readiness probes were "also failing during this window". The only frontend readiness failures in the events were at startup, 66 seconds before the agent read them, and the frontend pods were Ready throughout the 502s. - It never stated the fix in the report itself; the fix came only in its separate suggestions. The second run of the same scenario was graded partial. It found the same port mismatch, took 535 seconds, and listed the worker under "Not Checked" with the reason "no symptoms or alerts pointed to these components, so they were not examined." Run 1 at least read the worker's logs before setting them aside; run 2 never opened them, so its dismissal was untested. It also claimed "the fix was validated but not applied" without naming a fix, and the grading marked that unsupported. HolmesGPT 0.43.0 found the port in both of its runs (65 and 46 seconds, on 15 and 16 October) and was graded partial both times for not addressing the worker noise (run 1 called the worker "unaffected", run 2 never mentioned it), on judge-model grades only. ## How is a scanner different from an agent? K8sGPT on the same faults K8sGPT 0.4.40 runs a different kind of job. Its README calls it "a tool for scanning your Kubernetes clusters, diagnosing, and triaging issues in simple English" ([k8sgpt-ai/k8sgpt](https://github.com/k8sgpt-ai/k8sgpt)). `k8sgpt analyze --explain --namespace shop` walks Kubernetes objects through its analyzers and has an LLM explain what they flag. It never saw the alert. So we do not score it as a failed investigator; we report what its scan showed. On scenario 11 it printed this, in both runs: ``` AI Provider: amazonbedrockconverse No problems detected ``` Every pod was Running and Ready and every Service had endpoints, which is what its checks look at. The one wrong value, a Service `targetPort` that no container listens on, only shows when you compare two objects or send traffic. K8sGPT printed "No problems detected" in 12 runs across 6 scenarios (06, 11, 13, 15, 17, 19). Not all were quiet faults: in 06 and 17 pods were restarting under a firing KubePodCrashLooping alert, and the scan still came back clean. Scenario 12 shows the scanner's strength. Two faults went out together: the worker started OOMKilling (alerted) and the frontend Service's selector stopped matching any pod (not alerted, but the frontend was down for users). Both agents followed the alert to the worker and missed the frontend in all four of their runs; Aurora's reports said the frontend "remained healthy and unaffected" and was "running normally and unaffected". K8sGPT's scan, in both runs, listed it: ``` 2: Service shop/frontend() - Error: Service has no endpoints, expected label app=frontend - Error: Service has no endpoints, expected label tier=web ``` Its fixes there were not acceptable (it suggested labelling pods by hand), but it saw the outage the agents did not, because it looked at everything instead of following one alert. Runbook: [a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). ## AIOps vs AI SRE vs a scanner: what is the difference? | | Traditional AIOps platform | AI SRE agent | Scanner (K8sGPT) | |---|---|---|---| | Input | Alert and telemetry streams from many tools | One alert plus read access to tools | The cluster's current objects, no alert | | Method | Correlation rules and learned patterns, baselines, topology | An LLM choosing each next command from the last result | Fixed analyzers, LLM explanation of what they flag | | Output | Grouped incidents, anomalies, fewer pages | A root cause, its evidence, what was ruled out and not checked, suggested fixes | A list of objects in a bad state | | Scenario 11 (wrong port, noisy worker) | Not tested; time and topology would link the worker noise to the 5xx | Found the port in all four agent runs; ruled out the worker in one | "No problems detected" | | Scenario 12 (second, unalerted fault) | Not tested; it only sees the frontend outage if something alerts on it | Missed it in all four runs | Listed it in both runs | | Typical failure | Groups things that arrived together but are unrelated, or flags a harmless anomaly | Stops at the alerted fault, adds unsupported details | Silent when objects look healthy | | Speed and cost in our lab | Not tested | Aurora median about 154 s and $0.35 per run; HolmesGPT median 45 s | Median 12.5 s for two scan passes | The lab totals, for context: on 19 faults, two runs each, Aurora was graded correct in 34 of 38 runs and partial in 4, every grade read by a person. HolmesGPT was correct in 31 of 38 (6 partial, 1 wrong) on judge-model grades that no person has checked; a second model agreed with all 76 HolmesGPT and K8sGPT grades, a third pass agreed with all 38 HolmesGPT grades and disputed 6 K8sGPT labels, and a blinded regrade moved one HolmesGPT run from correct to partial. HolmesGPT was about 3x faster at the median. The /benchmark page's Fisher exact test on those totals gives p = 0.52, so the accuracy gap is not significant, and it compares person-read grades with judge-only ones. Aurora's runs are from 2 October and the others from 15 and 16 October, same cluster and faults. The prompts differed: HolmesGPT got the alert plus "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." which names the namespace; K8sGPT got no alert. All three got the same admin kubeconfig, each behind its own guard (Aurora's read-only `observability_only` command policy, HolmesGPT's allow-list, K8sGPT's read-only `analyze`). ## Our AIOps tool groups alerts but engineers still find the cause. What should we add? An investigation layer behind the grouping. The handoff point is the grouped incident: today it lands on a person who starts with `kubectl get`, `describe` and `logs`; an AI SRE can start there instead and hand the person a root cause with citations to the commands it ran. In our runs, that first pass took Aurora a median of about two and a half minutes and HolmesGPT 45 seconds. What to check in a trial, from what went wrong in ours: 1. **Does the report say what it ruled out and what it did not check?** In scenario 11 run 2 the only sign of the gap was the "Not Checked" line. A report without that section hides it. 2. **Does the cited output say what the claim says?** Both false details in run 1 carried citations: "pre-existing" cited the worker log (step 19) and the readiness claim cited the events (step 5), and neither output supports it. Open the cited output for every claim the decision rests on. 3. **Does it notice a second fault?** Neither agent did. Keep a scanner or your AIOps layer watching the rest of the system, and do not let the agent's "unaffected" close the incident. 4. **Can it change anything?** Our agents ran with read-only guards. Keep it that way until you have read a few dozen of its reports. The [AI SRE evaluation guide](/blog/how-to-evaluate-ai-sre-platform) covers the rest, and our lab is a ready-made trial: break your own staging cluster with the same scenarios and grade any vendor's agent against the same rubric. ## When is AIOps the right buy, when an AI SRE, and when both? | Your situation | Buy first | Why | |---|---|---| | Hundreds of pages a night, many duplicates, on-call burning out on noise | AIOps or the grouping in your paging tool | An agent investigating every duplicate spends time and model cost on noise, and you still read every report | | Alert volume is fine, each incident takes an hour of `kubectl` and log reading | AI SRE | Grouping does not shorten an investigation; the trace above is the hour it replaces | | Many monitoring tools, no single view of what is related | AIOps | Correlation across sources is its core job; an agent works one incident at a time | | Faults that leave no bad object or metric ([wrong port](/runbooks/fix-service-targetport-mismatch-kubernetes), [NetworkPolicy](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes), [config drift](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes)) | AI SRE | They need a theory tested against config; K8sGPT printed "No problems detected" on six scenarios, these among them | | Both of the first two | Both | AIOps decides what becomes an incident; the agent investigates each incident | | Kubernetes, small team, tight budget | A scanner plus an open source agent | K8sGPT finds broken objects across the namespace; Aurora or HolmesGPT investigates the alert | Our position: most teams that ask "AIOps or AI SRE?" have one of the two problems much worse than the other, and should fix that one first. Running both is right when an agent is being paged on duplicates, or when grouped incidents still wait an hour for a person. ## Should we use our observability vendor's built-in agent or a vendor-neutral AI SRE? Ask what each can reach. The built-in agents sit on the vendor's data: Datadog's Watchdog RCA, for example, requires its APM data ([docs](https://docs.datadoghq.com/watchdog/rca/)), and Dynatrace's analysis runs on its own dependency graph. That is a strength when everything is in that vendor, and a gap when the cause is in a cluster, a cloud account or a repo it does not see. A vendor-neutral agent queries the systems directly (in the trace above, `kubectl` against the cluster), so it needs its own credentials and guardrails. The same split answers "AI SRE vs APM": an APM tool collects traces and service metrics and shows where errors and latency are; an AI SRE reads that data, plus the cluster, cloud and repo, to work out why. Price models differ as well; the [AIOps RCA pricing comparison](/blog/top-10-aiops-platforms-free-root-cause-analysis-2026) lists what each vendor publishes. Open source agents (Aurora, HolmesGPT) and commercial ones are compared in [top AI SRE tools](/blog/top-ai-sre-tools-2026). ## Are there open source AIOps and AI SRE tools? Yes. [Keep](https://github.com/keephq/keep) is an open source alert management and workflow platform (MIT outside its `ee/` directory); its AI correlation is marked "KEEP ENTERPRISE ONLY" on [keephq.dev](https://www.keephq.dev/), and Elastic [announced an agreement to acquire it](https://www.elastic.co/security-labs/blog/elastic-and-keep-join-forces) in May 2025. On the investigation side, [Aurora](https://github.com/Arvo-AI/aurora) and [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) are Apache 2.0 agents, and [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) is the Apache 2.0 scanner tested above. See [Keep vs Aurora](/blog/keep-vs-aurora-open-source-aiops) for the pairing and the [three-way open source comparison](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt) for every lab run. ## What this comparison does not show We did not test any AIOps platform; the AIOps column comes from vendor docs, and the correlation scenario above is our reading of them. Each lab scenario fires one alert on a small kind cluster (Kubernetes v1.37.0, 3 nodes), so nothing here measures noise reduction at volume. Arvo AI builds Aurora and wrote the 19 scenarios. A person read Aurora's grades; nobody has yet read HolmesGPT's or K8sGPT's. ## How Aurora fits Aurora is the investigation layer, not an AIOps platform. It has no anomaly detection or machine-learned baselines, and it never suppresses or mutes alerts; it does link related alerts into an investigation that is already running. It investigates with read-only commands, writes the report with "Ruled Out" and "Not Checked" sections you saw above, and suggests fixes that a person approves. It runs self-hosted or hosted: [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora), or [book a demo](https://cal.com/arvo-ai/demo). Sources and verification: vendor docs for BigPanda, PagerDuty, Datadog, Dynatrace, K8sGPT and Keep checked on 21 October 2026. Lab data from `benchmark/results.json` (generated 21 October 2026) and the run folders in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab): Aurora runs of 2 October 2026, HolmesGPT 0.43.0 and K8sGPT 0.4.40 runs of 15 and 16 October 2026. More: [the lab report](/blog/aurora-lab-report-kubernetes-19-failures), [agentic incident management](/blog/what-is-agentic-incident-management), [Service has no endpoints](/runbooks/fix-service-no-endpoints-kubernetes), [/diagnose](/diagnose). --- ## How to Evaluate an AI SRE Platform (2026) URL: https://www.aurorasre.ai/blog/how-to-evaluate-ai-sre-platform Published: 2026-05-28 (updated 2026-09-28) Author: Noah Casarotto-Dinning, CEO at Arvo AI > A four-pillar framework for evaluating AI SRE platforms in 2026: investigation quality, trust, data sovereignty, and total cost of ownership. > ## Key Takeaways > > - **Generic SaaS RFPs do not fit AI SRE.** The category is younger than most procurement templates and the failure modes (hallucinated root causes, model drift, signal-type sensitivity) are not covered by traditional vendor checklists. > - **Investigation quality is measurable.** The [RCAEval benchmark](https://arxiv.org/abs/2412.17015) (Pham et al., December 2024, [published at ACM Web Conference 2025 Companion Proceedings](https://dl.acm.org/doi/proceedings/10.1145/3701716)) provides 735 fault-injection cases across three microservice systems with 11 fault types and 15 reproducible baselines. The [NOFire AI benchmark](https://www.nofire.ai/ai-sre-benchmark) extends this with a signal-type ladder showing Top-1 accuracy rises from 29 percent on metrics-only inputs to 77 percent when logs are added, 87 percent when traces are added, and 89 percent on full multi-modal telemetry with agentic reasoning. > - **Trust is a separate axis from capability.** [Rootly's AI SRE Maturity Model](https://rootly.com/ai-sre-guide/maturity-model) maps the trust ladder in four steps: Level 0 (manual), Level 1 (read-only copilot), Level 2 (assisted actions with approvals), Level 3 (guardrailed autonomy for narrow, reversible failure modes). Buyers should stage trust across that ladder, not buy at the top. > - **Deployment sovereignty is a gating constraint, not a tiebreaker.** Air-gapped, residency-bound, and BYO-LLM buyers must filter the shortlist on inference location before scoring anything else. See our [Self-Hosted AI SRE guide](/blog/self-hosted-ai-sre). > - **TCO is not just the licence line.** AI SRE cost models include LLM inference, observability surface, runbook ingestion, and the engineering time spent on guardrails. Open-source projects shift the licence cost to zero and surface the operating cost transparently. This guide is the deep evaluation framework. For the brief two-week procurement plan, see the [HowTo schema in our What is an AI SRE? glossary entry](/blog/ai-sre-complete-guide). For the five-capability rubric that filters the shortlist before evaluation starts, see the [Five-Capability AI SRE Test in that same post](/blog/ai-sre-complete-guide). Everything below assumes the shortlist has already cleared that test. We call the framework below the **Four-Pillar AI SRE Evaluation Framework**. Each pillar is a separate scoring axis and each is anchored to a primary source where one exists. ## Why do generic RFPs fail for AI SRE? A typical SaaS procurement RFP covers uptime, security posture, pricing, support tiers, and integrations. None of these surface the failure modes that matter for an AI SRE. - **Hallucinated root causes.** An AI SRE that produces a confident, plausible, wrong root-cause analysis is worse than no AI SRE, because the on-call rotation will follow the suggestion before second-guessing it. RFPs that do not include investigation-quality measurement miss this. - **Signal-type sensitivity.** The NOFire benchmark shows that the same agent moves from 29 percent Top-1 accuracy on metrics-only inputs to 87 percent when traces are added, and 89 percent on full multi-modal telemetry with agentic reasoning ([NOFire AI Benchmark](https://www.nofire.ai/ai-sre-benchmark)). Buyers who do not test the agent against their actual signal mix will misestimate accuracy by a factor of three. - **Trust ladder mismatches.** Buyers often evaluate against the wrong trust level. A team that wants Level 1 read-only operation and scores a tool on its Level 3 closed-loop remediation features is grading the tool on capability they will never use. - **Inference-location ambiguity.** Where the LLM call physically runs is a single-question filter that disqualifies half the shortlist for regulated buyers. Standard RFPs bury this in a "data residency" footnote rather than putting it at the top of the form. The four pillars below replace the generic RFP with an AI-SRE-specific scoring sheet. ## How do you measure AI SRE investigation quality? Investigation quality is the central question and the one most often left to vendor demos. The literature now provides enough scaffolding to measure it without a labelled production dataset. ### Anchor: the RCAEval benchmark [RCAEval](https://arxiv.org/abs/2412.17015) (Pham, Zhang, Ha, Salim, Zhang; December 2024; published at ACM Web Conference 2025 per the [ACM Digital Library record](https://dl.acm.org/doi/10.1145/3701716.3715290)) ships: - 735 failure cases collected from three microservice systems. - 11 fault types observed in real-world failures, including CPU throttling, memory leaks, network latency, container crashes, deployment errors, resource exhaustion, and database connection failures. - Multi-source telemetry (metrics, logs, and traces) supporting metric-based, trace-based, and multi-source RCA approaches. - 15 reproducible baselines covering coarse-grained and fine-grained RCA. The authors describe the work as the first comprehensive benchmark for root-cause analysis of microservices and the gap it fills as "no standard benchmark that includes large-scale datasets and supports comprehensive evaluation environments." For a buyer, this gives a defensible neutral testbed: ask the vendor to run their agent against the RCAEval fault-injection set and report Top-1 accuracy. ### Anchor: the NOFire signal-type ladder [NOFire AI's published benchmark](https://www.nofire.ai/ai-sre-benchmark) uses the RCAEval dataset and reports two findings buyers should internalise. 1. **Signal type matters more than model choice.** The benchmark reports Top-1 accuracy of 29 percent on metrics-only inputs, 77 percent when logs are added, 87 percent when traces are added, and 89 percent on full multi-modal telemetry with agentic reasoning. The implication: a buyer who only sends metrics to the agent should not expect more than a third of investigations to converge correctly, regardless of which vendor they pick. 2. **Production-context graphs beat plain LLMs.** NOFire reports 89 percent Top-1 accuracy on the RCAEval set, versus 42 percent for the best academic baseline (described on the benchmark page as "Academic SOTA (GALA BARO M2)"). The published implication is that a structured representation of the production environment (services, dependencies, recent changes) gives the agent better grounding than narrative telemetry alone. The vendor publishes this benchmark; treat the absolute numbers with the appropriate scepticism, but the relative ranking and the signal-type ladder are reproducible from the open dataset. ### What to measure during evaluation - **Top-1 accuracy on RCAEval-style fault injection.** Replay a handful of fault types and ask the vendor to walk through the investigation trace. The presence or absence of a coherent reasoning chain is the binary signal. - **Signal-type robustness.** Run the same fault under three configurations: metrics only, metrics plus logs, metrics plus logs plus traces. The shape of the accuracy curve tells you whether the agent compensates for signal gaps or simply degrades. - **Time to first useful finding.** Not MTTR (which depends on the human response loop), but the elapsed time from alert ingestion to the first piece of evidence a human SRE would have surfaced manually. - **Cross-system reasoning.** Construct a synthetic incident that spans Kubernetes, a managed database, and a recent deploy. Measure whether the agent reasons across all three or fixates on the loudest signal. ## How do you evaluate AI SRE trust and governance? Investigation quality without trust controls is a liability. The buyer's question is not "can the agent act" but "under what conditions, with what evidence, and with what rollback." ### Anchor: the Rootly AI SRE Maturity Model [Rootly publishes a four-level maturity model](https://rootly.com/ai-sre-guide/maturity-model) that maps the trust ladder cleanly. - **Level 0: Manual reliability operations.** No AI assistance. Responders hunt across dashboards. - **Level 1: Read-only AI SRE, evidence-first copilot.** The "trust-building stage." The AI accelerates context gathering and produces ranked hypotheses linked to evidence, but executes no changes. - **Level 2: Assisted actions with approvals.** The AI can propose and run approved actions through a governed workflow engine with RBAC, audit logs, verification gates, and rollback readiness. - **Level 3: Guardrailed autonomy for narrow, reversible failure modes.** The AI autonomously executes pre-approved runbooks for a small set of repeatable incidents, within strict bounds and with continuous verification. The Rootly framing is useful for a reason buyers often miss: **the levels are not a feature ranking, they are a deployment posture**. A tool that ships Level 3 features is not "better" than a tool that ships Level 1 features; it is appropriate for a different stage of the buyer's adoption arc. ### What to evaluate - **Match the maturity level to the buyer's current state.** A team that has not yet shipped Level 1 should not be paying for Level 3 capability they will not turn on. A team ready for Level 2 should not be buying a Level 1-only tool that they will outgrow in twelve months. - **Action class boundaries.** Read-only investigation, PR-based suggestions, and sandboxed in-cluster execution are three different trust decisions. Document which classes the tool supports and which the buyer will enable. - **Audit and rollback.** Every action the agent can take must have an audit log entry and a rollback path. Komodor's published benchmarking guide names this dimension as "Transparency: evidence, timelines, and change history alongside every recommendation" ([Komodor: Beyond the Hype](https://komodor.com/resources/beyond-the-hype-a-benchmarking-guide-for-ai-sre-in-2026/)). - **Hallucination guardrails.** Komodor's guide also calls out "rigorous testing cycles and closed feedback loops" as the path to "95% RCA precision." Ask the vendor what those feedback loops look like in practice; a tool with no published guardrail story should be downgraded. ## What deployment sovereignty checks matter for an AI SRE? For regulated industries, this pillar is a filter, not a scoring axis. The shortlist either includes a deployment posture that matches the buyer's constraints or it does not. ### What to evaluate - **Inference location.** Does the LLM call run on vendor-managed infrastructure (SaaS), on customer-managed infrastructure (self-hosted), or on a local model (air-gapped)? See our [Self-Hosted AI SRE](/blog/self-hosted-ai-sre) guide for the full deployment-tier framework. - **Data residency.** Where does telemetry physically reside when sent to the agent? Buyers under GDPR, HIPAA, or sector-specific regimes need a written answer, not a marketing one. - **BYO-LLM support.** Can the buyer point the agent at their own LLM endpoint? Open-source projects support this directly; [HolmesGPT documents OpenAI-Compatible (LiteLLM proxy) and Ollama](https://holmesgpt.dev/latest/); [K8sGPT registers IBM watsonx, Oracle OCI GenAI, and a generic Custom REST endpoint](https://github.com/k8sgpt-ai/k8sgpt/blob/main/pkg/ai/iai.go) among others; Aurora supports local inference through Ollama. Most commercial AI SREs offer a smaller backend list. - **Air-gapped operation.** Can the agent run with no outbound network calls? This is the strictest test and disqualifies most SaaS-only products. ### Anchor: cite the project's own documentation For an open-source AI SRE, the source of truth is the project's GitHub repository and official docs site. For a commercial AI SRE, the source of truth is the vendor's data-processing addendum and the published deployment architecture. Avoid relying on the sales conversation for this pillar; the engineering documentation is where commitments are durable. ## How do you model AI SRE total cost of ownership? TCO for AI SRE breaks down into four layers, only one of which is the licence. 1. **Licence or subscription cost.** Open-source AI SREs are zero at this layer. Commercial AI SREs use per-seat, per-investigation, or platform-bundled pricing. Datadog Bits AI SRE bundles into the broader Datadog platform per the [product page](https://www.datadoghq.com/product/ai/bits-ai-sre/). PagerDuty SRE Agent bundles into the PagerDuty platform per the [product page](https://www.pagerduty.com/platform/ai-agents/sre/). Resolve.ai and Traversal price by custom contract. 2. **LLM inference cost.** This is the live-burn cost. Frontier-model API pricing changes monthly; buyers should model investigations per month against the published per-token rates of their chosen provider (Anthropic, OpenAI, Google). For BYO-LLM deployments running local models through Ollama or vLLM, the inference cost is reduced to the underlying compute. 3. **Operating surface cost.** The agent reads from observability backends, ticket systems, and source-control. Heavy use can increase the costs of the systems it reads from (Datadog ingestion, Splunk indexing, GitHub API rate-limit upgrades). Buyers should add a line item for this and ask their FinOps team to model it before purchase. 4. **Engineering time on guardrails and runbook ingestion.** Every AI SRE needs runbooks ingested, integrations configured, and RBAC scoped. The first month of engineering time is the largest hidden TCO cost. ### Why we are not publishing competitor pricing numbers Most direct AI SRE competitors (Resolve.ai, Cleric, PagerDuty SRE Agent, Datadog Bits AI SRE) do not publish per-investigation or per-seat pricing on their public sites. Buyers must request a quote. Any TCO comparison we published with specific dollar figures would either be out of date by the time you read it or fabricated. The correct approach is to issue an RFP that asks for the same shape of cost data (licence floor, per-investigation rate, per-seat rate, included integrations) from every vendor on the shortlist and to model the rest yourself. ## How long should the evaluation take? Our recommendation is a 21-day evaluation sprint, structured as below. This is longer than the 14-day plan in the [What is an AI SRE? HowTo schema](/blog/ai-sre-complete-guide) because the four-pillar framework explicitly measures investigation quality with synthetic fault injection rather than relying on demo-driven impressions. The HowTo schema attached to this post documents the seven-step sprint. ## What do competitor evaluation frameworks miss? Several competitor frameworks exist; each has gaps the Four-Pillar Framework closes. - **[Traversal's "How Should You Evaluate an AI SRE Product?" post](https://www.traversal.com/blog/how-should-you-evaluate-an-ai-sre-product)** focuses on selecting representative incidents, defining success metrics, and a multi-tier accuracy rubric for root cause analysis. It is strong on the testing methodology and quiet on TCO modelling and open-source alternatives. - **[Komodor's "Beyond the Hype" benchmarking guide](https://komodor.com/resources/beyond-the-hype-a-benchmarking-guide-for-ai-sre-in-2026/)** is strong on transparency and on the LLM-as-a-Judge testing methodology. The detailed scoring rubric is gated behind an ebook download and the framework does not extend to deployment sovereignty. - **Rootly's maturity model** is the cleanest published trust ladder, but does not address investigation quality measurement or signal-type sensitivity. - **The [Traversal LLM benchmarking paper for incident root cause analysis](https://www.traversal.com/blog/llm-benchmarking-in-context-retrieval-reasoning-incident-root-cause-analysis)** is excellent on model-level evaluation and silent on the buyer-process question of how to map evaluation results onto a maturity-level decision. The Four-Pillar Framework is intentionally additive. A buyer who has already adopted Rootly's maturity model and Komodor's testing methodology can use the framework above to fill the deployment-sovereignty and TCO gaps without throwing away their existing work. ## What are the common AI SRE evaluation mistakes? - **Scoring on demo polish.** A demo is a curated success path. Evaluate on failure cases the vendor has not seen. - **Skipping the signal-type test.** The NOFire ladder shows accuracy can vary by a factor of three depending on what telemetry the agent receives. Run the test on the buyer's actual signal mix. - **Buying remediation before trust.** Most teams should buy at Level 1 (read-only investigation) and stage trust upward across six to twelve months, not procure at Level 3 in a single decision. - **Ignoring open-source baselines.** A Five-Capability-passing open-source AI SRE deployed in a single afternoon is the fairest baseline against which to measure any commercial pitch. See our [open-source three-way comparison](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt). - **Treating TCO as the licence line.** Inference, operating surface, and guardrail engineering frequently exceed the licence cost in year one. ## Where this guide fits - [What is an AI SRE?](/blog/ai-sre-complete-guide) for the five-capability shortlist filter. - [AI SRE vs AIOps](/blog/ai-sre-vs-aiops) for the category placement. - [AI SRE Complete Guide](/blog/ai-sre-complete-guide) for the procurement and adoption arc. - [Top 15 AI SRE Tools in 2026](/blog/top-ai-sre-tools-2026) for the full vendor matrix. - [Self-Hosted AI SRE](/blog/self-hosted-ai-sre) for the deployment-tier framework. - [Open-Source AI SRE: Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt) for the open-source baseline. - [AI SRE and incident tool pricing compared](/blog/ai-sre-incident-tool-pricing-2026) for the per-user, per-responder, and per-credit billing models behind the licence line. ## Try Aurora - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## HolmesGPT vs K8sGPT: Tested on 19 Kubernetes Faults (2026) URL: https://www.aurorasre.ai/blog/holmesgpt-vs-k8sgpt Published: 2026-05-21 (updated 2026-10-21) Author: Ben Gervais, Engineer at Arvo AI > HolmesGPT vs K8sGPT, tested Oct 2026 on the same 19 injected Kubernetes faults: real output from both, MCP, operator mode, kagent, and when to pick each. HolmesGPT is an open-source AI agent that investigates a production incident: you give it a question or an alert, and an LLM loop runs read-only tools (kubectl, logs, Prometheus and other data sources) until it can state a root cause. K8sGPT is a Kubernetes scanner: its analyzers list objects that look broken, and an LLM explains each finding in plain English. Both are Apache 2.0 and CNCF Sandbox projects, so the choice comes down to the job. Pick HolmesGPT to answer "why is this alert firing", and K8sGPT for a fast, cheap list of broken objects in a cluster. We did not take that from the READMEs. In October 2026 we ran HolmesGPT 0.43.0 and K8sGPT 0.4.40 on the same 19 injected Kubernetes faults, twice each, and graded every answer against the fault we injected. HolmesGPT named the root cause in 31 of 38 runs (6 partial, 1 wrong) at a median of 45 seconds per investigation. K8sGPT's analyzers returned findings in 26 of 38 runs and printed "No problems detected" in the other 12; none of its explanations stated the full root cause, which is expected from a tool that does not investigate. These 76 grades come from a judge model, checked by a second model and by a blinded regrade. No person has reviewed them yet. A note on bias. Arvo builds [Aurora](https://github.com/Arvo-AI/aurora), another open-source AI SRE, and Arvo wrote the 19 scenarios. Aurora ran on the same faults; its results are on the [benchmark page](/benchmark) and in a short section near the end. Every transcript quoted here is in the public lab repo, [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab), under `runs/h2h-v1/`. ## How we tested HolmesGPT and K8sGPT **Cluster.** kind v0.33.0, Kubernetes v1.37.0, 3 nodes, running a synthetic app called `shop` (nginx frontend, Python api, worker, redis, load generator). No customer code or data. Each fault is one `break.sh` in a [scenario folder](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios), and `check.sh` proves the fault is visible before a tool runs. **Versions.** HolmesGPT 0.43.0, installed with `uv tool install holmesgpt`. K8sGPT 0.4.40, installed with Homebrew. Runs r1 on 2026-10-15 and r2 on 2026-10-16. **Models.** HolmesGPT ran on `bedrock/us.anthropic.claude-sonnet-5`, the model Aurora used. K8sGPT ran on `us.anthropic.claude-sonnet-4-5` through its `amazonbedrockconverse` backend, because K8sGPT always sends a `temperature` parameter and Bedrock rejects that for Sonnet 5. We did not try a proxy that strips it. That is a real difference: K8sGPT got an older model. **What each tool was asked.** This is the core of the comparison, so here are the exact commands. The `holmes ask` and `analyze` lines are from [`runner/h2h.sh`](https://github.com/Arvo-AI/aurora-lab/blob/main/runner/h2h.sh); the `auth add` line is the one-time backend setup from `runs/h2h-smoke/README.md`: ``` holmes ask "$ALERT Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." \ --model "bedrock/us.anthropic.claude-sonnet-5" --max-steps 40 k8sgpt auth add -b amazonbedrockconverse -m -r us-east-1 k8sgpt analyze --explain --namespace shop --output json --no-cache k8sgpt analyze --explain --namespace shop --no-cache ``` HolmesGPT got the scenario's alert text (for example "KubePodCrashLooping: an api pod in namespace shop is restarting after the latest deploy."). K8sGPT has no notion of an alert, so it got the namespace. We used `holmes ask` with its default toolsets ("Using 28 datasources (toolsets)" in every run) because the lab has no Alertmanager for `holmes investigate` to read from. We did not change K8sGPT's analyzer filters. Both tools had the cluster admin kubeconfig, and no transcript shows a mutating command running. **Grading.** A run is correct when it states every `must_identify` item in the scenario and asserts none of the listed wrong answers; partial means some items. The judge was `us.anthropic.claude-opus-5-5`, from the same vendor as both models under test. A second model re-read all 76 grades and agreed with all of them. A third pass regraded all 76 from scratch without seeing the judge: it agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels, calling symptom-only answers "no answer" where the judge said partial or wrong. A blinded regrade, with tool names removed, moved one HolmesGPT run (r2/19) from correct to partial and two K8sGPT runs down. We kept the judge's labels. All of it is in [review2-summary.md](https://github.com/Arvo-AI/aurora-lab/blob/main/runs/h2h-v1/review2-summary.md) and [blind-regrade.md](https://github.com/Arvo-AI/aurora-lab/blob/main/runs/blind-regrade.md). **Timing.** HolmesGPT's seconds are one `holmes ask`. K8sGPT's cover the two `analyze` passes above, so a single pass takes about half. ## Results: every scenario, both runs HolmesGPT's verdict is graded against the injected root cause. K8sGPT's column is the judge's label for what its analyzers and explanation said, and it measures something different: K8sGPT was scored on a criterion built for tools that investigate an alert, which it does not claim to do. Read it as "how close does the scan get", not as a failure rate. Each fault links to our runbook for it. | # | Fault | HolmesGPT r1 | HolmesGPT r2 | K8sGPT r1 | K8sGPT r2 | |---|---|---|---|---|---| | 01 | [Worker OOMKilled after a batch size change](/runbooks/fix-oomkilled-kubernetes) | correct, 39 s | correct, 28 s | partial | partial | | 02 | [Log shipper sidecar OOMKilled, app container healthy](/runbooks/fix-sidecar-container-oomkilled-kubernetes) | correct, 39 s | correct, 30 s | partial | partial | | 03 | [Frontend rollout stuck on a mistyped image tag](/runbooks/fix-imagepullbackoff-kubernetes) | correct, 22 s | correct, 28 s | partial | partial | | 04 | [New api pods blocked on a Secret that does not exist](/runbooks/fix-createcontainerconfigerror-kubernetes) | correct, 31 s | correct, 29 s | partial | partial | | 05 | [Frontend crash loop from an invalid nginx config](/runbooks/fix-nginx-emerg-unknown-directive-kubernetes) | correct, 49 s | correct, 36 s | partial | partial | | 06 | [api restarts because the liveness probe targets the wrong port](/runbooks/fix-liveness-probe-failed-kubernetes) | correct, 43 s | correct, 51 s | No problems detected | No problems detected | | 07 | [api Service selects no pods after a selector change](/runbooks/fix-service-no-endpoints-kubernetes) | correct, 55 s | correct, 39 s | partial | partial | | 08 | [Worker pod Pending, CPU request larger than any node](/runbooks/fix-pod-pending-failedscheduling-kubernetes) | correct, 43 s | correct, 44 s | partial | partial | | 09 | [api failing because redis is Pending on an unbindable PVC](/runbooks/fix-pvc-pending-storageclass-not-found-kubernetes) | correct, 104 s | correct, 87 s | partial | partial | | 10 | [api crash loop from a wrong REDIS_HOST](/runbooks/fix-dns-name-not-resolving-kubernetes) | partial, 45 s | correct, 67 s | wrong | wrong | | 11 | [frontend 5xx from a wrong api Service targetPort, next to a noisy but harmless worker](/runbooks/fix-service-targetport-mismatch-kubernetes) | partial, 65 s | partial, 46 s | No problems detected | No problems detected | | 12 | [Two faults at once, worker OOM and a frontend Service selecting no pods; the alert covers only the worker](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes) | partial, 26 s | partial, 34 s | partial | partial | | 13 | [frontend 5xx because a NetworkPolicy blocks api to redis](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes) | correct, 110 s | correct, 118 s | No problems detected | No problems detected | | 14 | [api rollout stuck because a ResourceQuota forbids the surge pod](/runbooks/fix-resourcequota-exceeded-rollout-stuck-kubernetes) | correct, 41 s | correct, 45 s | partial | partial | | 15 | [One in three api requests fails because a stray Deployment's pods carry app=api](/runbooks/fix-service-selects-wrong-pods-label-overlap-kubernetes) | correct, 97 s | correct, 162 s | No problems detected | No problems detected | | 16 | [New api pods crash because their dnsConfig search domains are for another cluster domain](/runbooks/fix-pod-dnsconfig-search-domain-not-resolving-kubernetes) | correct, 82 s | correct, 101 s | wrong | wrong | | 17 | [Worker OOMKilled every few minutes by a slow leak in its code; logs look clean](/runbooks/fix-memory-leak-oomkilled-kubernetes) | correct, 72 s | correct, 70 s | No problems detected | No problems detected | | 18 | [api rollout stuck on an init container waiting for a Service that does not exist](/runbooks/fix-pod-stuck-init-container-kubernetes) | partial, 43 s | correct, 43 s | partial | wrong | | 19 | [Half of frontend requests fail; one frontend pod loaded an edited ConfigMap, the other still runs the old config](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes) | wrong, 49 s | correct, 106 s | No problems detected | No problems detected | | | HolmesGPT 0.43.0 | K8sGPT 0.4.40 | |---|---|---| | Correct root cause | 31 of 38 | 0 of 38 (not designed to investigate) | | Partial | 6 | 21 | | Wrong | 1 | 5 | | "No problems detected" | 0 | 12 | | Median time | 45 s | 12.5 s for two analyze passes (15 s on the 26 runs with findings; the 12 empty scans took 0 to 1 s) | | Grades reviewed by a person | 0 of 38 | 0 of 38 | Costs are not in the table. HolmesGPT printed "Usage event costs for this model will be 0" because LiteLLM had no price entry for Sonnet 5, and we did not capture token counts for either tool. ## Same fault, both tools: a wrong REDIS_HOST Scenario 10 is a one-line change: `kubectl -n shop set env deploy/api REDIS_HOST=redis-master`. There is no Service called `redis-master`, so the new api pod cannot resolve the name, retries five times, exits 1 and crash-loops. The old pods keep serving. Here is run r2 for both tools. K8sGPT printed this, in full ([k8sgpt.txt](https://github.com/Arvo-AI/aurora-lab/blob/main/runs/h2h-v1/r2/10-wrong-redis-host/k8sgpt.txt)): ``` AI Provider: amazonbedrockconverse 0: Pod shop/api-869d87b484-55cm4(Deployment/api) - Error: Readiness probe failed: Get "http://10.244.2.129:8080/healthz": dial tcp 10.244.2.129:8080: connect: connection refused Error: The readiness probe cannot connect to your application on port 8080. The application isn't running or listening on the expected port. Solution: 1. Verify app runs on port 8080 2. Check container logs for startup errors 3. Ensure app starts before probe begins 4. Adjust initialDelaySeconds in probe config 1: Service shop/api() - Error: Service has not ready endpoints, pods: [Pod/api-869d87b484-55cm4], expected 1 Error: The service cannot route traffic because its pod is not ready to accept connections. Solution: 1. Check pod status: `kubectl describe pod api-869d87b484-55cm4` 2. View logs: `kubectl logs api-869d87b484-55cm4` 3. Verify readiness probe configuration 4. Fix app startup issues or probe settings ``` Everything in it is true. It found the right pod in 15 seconds and its second step ("Check container logs for startup errors") would have led a person to the answer. But the analyzer works from the readiness probe event, so the explanation is about the probe, and most of the eight suggestions (verify port 8080, adjust `initialDelaySeconds`, verify the readiness probe configuration) point at the probe rather than the cause. The judge and the second model called it wrong; the independent third pass called it no answer, since no cause is asserted. Either way, `REDIS_HOST` never appears. HolmesGPT took 67 seconds and nine tool calls, one of them blocked. The trace, trimmed ([holmes.md](https://github.com/Arvo-AI/aurora-lab/blob/main/runs/h2h-v1/r2/10-wrong-redis-host/holmes.md)): ``` Running tool #1 kubernetes_tabular_query: filter: kubectl get pods Running tool #2 bash: kubectl get pods -n shop -o wide | grep api; echo ---; kubectl get rs -n shop | grep api; echo ---; kubectl get deploy api -n shop -o yaml | head -100 Running tool #3 bash: kubectl describe pod api-869d87b484-55cm4 -n shop Running tool #4 fetch_pod_logs: Fetch Logs (pod=api-869d87b484-55cm4, namespace=shop) [...] Running tool #7 bash: kubectl get endpoints redis -n shop; echo ---; kubectl get pods -n shop -l app=redis -o wide; echo ---; kubectl describe svc redis -n shop [...] Tool 'bash' requires approval: Command requires approval. Segment(s) not in allow list: 'kubectl rollout history deployment/api -n shop' [...] • ReplicaSet api-f64486c4c (created 08:22:55, revision 1): REDIS_HOST=redis — 2/2 pods Running, serving /healthz and /items with HTTP 200. • ReplicaSet api-869d87b484 (created 08:23:04, revision 2, current): REDIS_HOST=redis-master — 1 pod (api-869d87b484-55cm4), CrashLoopBackOff. The only Redis Service in shop is named redis (ClusterIP 10.96.64.60, endpoint 10.244.1.145:6379, backed by pod redis-59458f55d8-vf8vz). There is no redis-master Service. Pod logs for api-869d87b484-55cm4 confirm the DNS failure: redis connection failed (attempt 1/5): [Errno -2] Name does not resolve ... giving up: redis unavailable [...] The latest deploy changed container env REDIS_HOST from redis → redis-master, pointing at a Service that doesn't exist. ``` That is the whole difference between the two tools in one scenario. K8sGPT reports what the cluster says is broken. HolmesGPT keeps asking questions: it compared the two ReplicaSets' env, checked which Services exist, read the logs, and noticed on its own that the `last-applied-configuration` annotation still said `redis`, so the change was pushed with `kubectl set env`, `patch` or `edit` rather than `kubectl apply` (it was `set env`). It also hit its own command allow-list: `kubectl rollout history` needs approval in HolmesGPT's default bash toolset, and it routed around that by reading the ReplicaSets directly. HolmesGPT did not get this right both times. In r1 on the same fault it saw the same two ReplicaSets and inverted them: "The latest deploy rolled the api Deployment (namespace shop) to a new ReplicaSet (api-f64486c4c) that sets REDIS_HOST=redis", called the crashing pod the "tail end of a rollout", and called the bad config "cosmetic now". The judge graded it partial. An LLM loop can read the right evidence and still draw the wrong conclusion, and you only see it when you check the trace. ## Where K8sGPT's analyzers do well When the fault is written on a single object, K8sGPT is fast and precise. On the mistyped image tag (scenario 03, r1) its first finding was: ``` 0: Pod shop/frontend-7b6c998456-rv848(Deployment/frontend) - Error: Back-off pulling image "nginx:1.27-alpne": ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.27-alpne": failed to resolve reference "docker.io/library/nginx:1.27-alpne": docker.io/library/nginx:1.27-alpne: not found Error: The Docker image "nginx:1.27-alpne" cannot be found because the tag name is misspelled. ``` That is the root cause, in 17 seconds for two passes. It was graded partial because the second finding said the frontend Service "cannot route traffic", while the old ReplicaSet was still serving. The same pattern held for the unschedulable pod (it quoted `0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu`, but led with "Add tolerations" rather than the 64-core request), the missing Secret (`secret "api-secrets" not found`) and the ResourceQuota (`exceeded quota: compute-resources`). On the Pending PVC it found the cause (`storageclass.storage.k8s.io "fast-ssd" not found`) but listed the api crash loop as a separate finding with a generic fix, never linking it to redis being down. K8sGPT points at the right object or close to it. It does not connect findings to each other or to the change that caused them. In 12 runs it printed `No problems detected`: the wrong liveness port, the wrong Service targetPort, the NetworkPolicy blocking api to redis, stray pods sharing the `app=api` label, the slow memory leak and the ConfigMap drift, both runs each. In all six something was failing (pods being restarted in 06 and 17, users getting 5xx in the other four), but no object was in a state its analyzers flag at scan time. (`networkPolicyAnalyzer` is in K8sGPT's optional set, per its [README](https://github.com/k8sgpt-ai/k8sgpt#analyzers), and we did not add filters.) HolmesGPT found the NetworkPolicy in both runs, the leak in both and the stray pods in both, because it started from the symptom in the alert. ## Where HolmesGPT went wrong Seven of HolmesGPT's 38 runs were not correct, and they share a shape. - **Scenario 19, r1, wrong.** One frontend pod had been restarted with an edited ConfigMap (`api:8000`), the other still ran the old config from memory. The 502s HolmesGPT quoted came from the healthy pod's logs; it never compared the two pods, blamed "a transient startup race, now resolved", and declared the incident over. In r2 it found the drift, but said the healthy pod was fine only because the kubelet had not re-synced the ConfigMap yet (nginx reads its config only at start); the blinded regrade marked that run partial. - **Scenario 11, both runs partial.** The real fault is a wrong Service targetPort, next to a worker that logs noisy but harmless errors. HolmesGPT found the targetPort and the right fix both times, but never ruled out the worker's error lines, which the scenario requires: r1 only called the worker "unaffected", r2 never mentioned it. - **Scenario 12, both runs partial.** Two faults at once; the alert covers only the worker OOM. HolmesGPT diagnosed the OOM correctly and never checked Services, so it missed the frontend selector fault. - **Scenarios 10 and 18, r1 partial.** The inverted rollout above. On 18 the diagnosis was right, but its first fix was to deploy the missing postgres Service, which the api does not use. The allow-list matters here. `kubectl rollout history` was blocked in 12 runs, including 2 of the 7 that were not correct. We did not rerun those with the command allowed. That is a safe default and you can extend it, but out of the box it hides the one command that answers "what changed". ## How do HolmesGPT and K8sGPT differ? Checked against each project's GitHub and docs on 2026-10-21. | | HolmesGPT | K8sGPT | |---|---|---| | What it does | An agentic loop that queries live observability data from several sources to find a root cause ([README](https://github.com/HolmesGPT/holmesgpt)) | "Scanning your Kubernetes clusters, diagnosing, and triaging issues in simple English" with codified analyzers ([README](https://github.com/k8sgpt-ai/k8sgpt)) | | Input | A question or an alert (`holmes ask`, `holmes investigate`) | A cluster or namespace (`k8sgpt analyze`) | | Scope | Kubernetes, VMs, bare metal, cloud services and SaaS data sources; "No Kubernetes required" | Kubernetes resources, one analyzer per type, plus optional [integrations](https://github.com/k8sgpt-ai/k8sgpt/tree/main/pkg/integration) for Prometheus, AWS (an EKS analyzer), KEDA and Kyverno | | LLM required | Yes, every step | Only for `--explain`; the analyzers run without one | | Language, install | Python; Homebrew, pipx, Docker, from source ([install docs](https://holmesgpt.dev/latest/installation/cli-installation/)) | Go binary; Homebrew, RPM, DEB and APK packages, Windows binaries, in-cluster operator | | LLM backends | Anthropic, AWS Bedrock, Azure AI Foundry, Baseten, Gemini, GitHub Copilot, GitHub Models, Google Vertex AI, Ollama, OpenAI, OpenRouter, Robusta AI, OpenAI-compatible ([docs](https://holmesgpt.dev/latest/ai-providers/)); calls go through LiteLLM | OpenAI, Anthropic, Azure OpenAI, LocalAI, Ollama, Cohere, Amazon Bedrock and Bedrock Converse, SageMaker, Google GenAI, Vertex AI, Hugging Face, Oracle OCI, IBM watsonx, Groq, custom REST, LiteLLM proxy ([`pkg/ai/iai.go`](https://github.com/k8sgpt-ai/k8sgpt/blob/main/pkg/ai/iai.go)) | | MCP | Consumes MCP servers as data sources | Serves an MCP server (`k8sgpt serve --mcp`) | | Background mode | Operator mode, alpha | k8sgpt-operator, default 30 s interval | | Writes | Read-only by design; optional GitHub PRs and a Kubernetes Remediation MCP toolset | Opt-in alpha auto-remediation of image failures in the operator | | Data sent to the LLM | Tool output goes to the LLM you configure; Ollama or another local provider keeps it in your network | Optional `--anonymize` masks object names and labels; "Anonymization does not currently apply to events" | | License | [Apache 2.0](https://github.com/HolmesGPT/holmesgpt/blob/master/LICENSE) | Apache 2.0 | | CNCF | [Sandbox, accepted October 8, 2025](https://www.cncf.io/projects/holmesgpt/) | [Sandbox, accepted December 19, 2023](https://www.cncf.io/projects/k8sgpt/) | | Origin | "Originally created by Robusta.Dev, with major contributions from Microsoft" | Maintained under the [k8sgpt-ai](https://github.com/k8sgpt-ai) GitHub organization | | GitHub, 2026-10-21 | [HolmesGPT/holmesgpt](https://github.com/HolmesGPT/holmesgpt) (moved from the Robusta.Dev organization; the old URL redirects), 3,652 stars, 544 forks | [k8sgpt-ai/k8sgpt](https://github.com/k8sgpt-ai/k8sgpt), 8,265 stars, 1,082 forks | | Latest release, 2026-10-21 | [0.45.0](https://github.com/HolmesGPT/holmesgpt/releases), 15 Oct 2026 (we tested 0.43.0) | [v0.4.41](https://github.com/k8sgpt-ai/k8sgpt/releases), 21 Oct 2026 (we tested 0.4.40) | ## HolmesGPT MCP vs K8sGPT MCP They point in opposite directions, which matters if you are wiring either into an agent you already run. **HolmesGPT is an MCP client.** You add MCP servers under `mcp_servers` in its config, over streamable-HTTP (recommended), stdio or the deprecated SSE ([remote MCP docs](https://holmesgpt.dev/latest/data-sources/remote-mcp-servers/)). Several of its built-in integrations are MCP servers: AWS, Azure, GCP, GitHub, GitLab, Jenkins, Sentry, Splunk, Prefect, Confluence, Atlassian Rovo, and "Kubernetes Remediation (MCP)", described as "Apply fixes like scaling, rollbacks, and resource edits" ([README](https://github.com/HolmesGPT/holmesgpt)). That last one is the write path: its [docs](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/kubernetes-remediation-mcp/) say it can restart pods, scale deployments, drain nodes and patch resources, "with every mutating action gated behind human approval". Review it before you enable it. **K8sGPT is an MCP server.** `k8sgpt serve --mcp` runs it over stdio for Claude Desktop and similar clients, and `k8sgpt serve --mcp --mcp-http` serves HTTP on port 8089 by default. Its [source](https://github.com/k8sgpt-ai/k8sgpt/blob/main/pkg/server/mcp.go) registers 12 tools (`analyze`, `cluster-info`, `config`, `list-resources`, `get-resource`, `list-namespaces`, `list-events`, `get-logs`, `list-filters`, `add-filters`, `remove-filters`, `list-integrations`) and 3 resources (`cluster-info`, `namespaces`, `active-filters`). If you already use an MCP-capable assistant and want it to see your cluster through K8sGPT's analyzers, this is the cheapest way to do it. We ran both tools as CLIs, not through MCP. ## Operator mode: HolmesGPT vs K8sGPT in the background HolmesGPT's [operator mode](https://holmesgpt.dev/latest/operator/) runs health checks as custom resources: one-off, on a cron schedule, or "whenever a matching Deployment is rolled out". A "kopf-based controller handles CRD orchestration and scheduling, while stateless Holmes API servers execute the actual checks." The docs say it "is in alpha and subject to breaking changes" and warn that "Each health check triggers an LLM call (at least 1)." With the GitHub MCP server connected, "Holmes can open PRs to fix the problems it finds." The [k8sgpt-operator](https://github.com/k8sgpt-ai/k8sgpt-operator) runs the analyzers in-cluster on an interval ("If not specified, the default interval is 30 seconds") and writes results as custom resources. It now has auto-remediation that is "deliberately alpha and opt-in": the LLM never gets write access, and the operator computes one image patch itself and dry-runs it against the API server before it applies anything. Our test covered neither operator. ## HolmesGPT vs kagent kagent is not a HolmesGPT alternative in the way K8sGPT is. HolmesGPT is a finished SRE agent with a prompt, toolsets and a CLI. [kagent](https://github.com/kagent-dev/kagent) is "a Kubernetes native framework for building AI agents": you define an `Agent` custom resource, attach tools over MCP, and its controller runs the agent and "exposes gRPC, A2A, and MCP APIs." It was created at Solo.io, which announced it on March 17, 2025 and [contributed it to the CNCF](https://www.solo.io/blog/bringing-agentic-ai-to-kubernetes-contributing-kagent-to-cncf); it was [accepted to the CNCF Sandbox on May 22, 2025](https://www.cncf.io/projects/kagent/). Apache 2.0, 4,066 GitHub stars on 2026-10-21, latest release v1.0.0-alpha16 on 19 Oct 2026. Choose kagent when you want to build and run your own agents (an SRE agent, a release agent, a cost agent) as Kubernetes resources managed with GitOps, and you are willing to write the prompts and pick the tools. Choose HolmesGPT when you want an investigation agent that works today from an alert. We did not run kagent in this benchmark, so we have no accuracy numbers for it. ## When should you pick HolmesGPT, K8sGPT or both? **Pick HolmesGPT** when the question is "why is this alert firing". In our runs it named the root cause on 31 of 38 attempts, read logs, compared ReplicaSets and checked Services without being told to. It is the one of the two built to reach outside Kubernetes (AWS, GCP, Azure, databases, observability tools); K8sGPT's optional AWS integration adds only an EKS check. Budget for an LLM call per step and for reading the trace: the r1 inversion on scenario 10 looked confident. **Pick K8sGPT** when you want a fast, cheap inventory of broken objects: a pre-deploy check, a CI gate, a cron job that posts to Slack, or an MCP tool for an assistant. It took a median 15 seconds for two passes when it had findings, the analyzers need no LLM at all, and when a fault is written on an object (bad image tag, Pending pod, missing Secret) it names it exactly. Do not use it as an incident investigator. In 12 of 38 runs it said "No problems detected" while pods were being restarted or requests were failing. **Use both** if you run Kubernetes at any size. They do not overlap much: K8sGPT tells you what is broken now, HolmesGPT tells you why an alert fired. ## Where Aurora fits Aurora is Arvo's open-source AI SRE, and it ran on the same 19 faults with the same model as HolmesGPT. A person reviewed all 38 of its grades: 34 correct, 4 partial, 0 wrong, at a median of 153.5 seconds and $0.35 per run. Its runs are from 2026-10-02, its times include contention from its own background jobs in the lab stack, and its prompt differs slightly ("Investigate and find the root cause" instead of "Find the root cause and say what changed"). HolmesGPT is about three times faster at the median and close on accuracy (31 correct, judge-graded, not yet reviewed by a person). With 38 runs each and a model judge, read the two as roughly even. The full table, prompt differences and caveats are on the [benchmark page](/benchmark) and in the [lab report](/blog/aurora-lab-report-kubernetes-19-failures). For the three-way comparison on features and deployment, see [Open-Source AI SRE: Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt). For what HolmesGPT is and how it handles estates beyond one cluster, see [HolmesGPT alternative for multi-cloud](/blog/holmesgpt-alternative-multi-cloud). To check a single failing pod against our runbooks without installing anything, try [/diagnose](/diagnose). ## Sources and verification - Lab data: [results.json](https://github.com/Arvo-AI/aurora-lab/blob/main/benchmark/results.json) and every transcript and grade under [runs/h2h-v1](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/h2h-v1). Tested HolmesGPT 0.43.0 and K8sGPT 0.4.40 on 2026-10-15 and 2026-10-16. - HolmesGPT: [GitHub](https://github.com/HolmesGPT/holmesgpt), [LLM providers](https://holmesgpt.dev/latest/ai-providers/), [operator mode](https://holmesgpt.dev/latest/operator/), [MCP servers](https://holmesgpt.dev/latest/data-sources/remote-mcp-servers/), [CNCF page](https://www.cncf.io/projects/holmesgpt/). - K8sGPT: [GitHub](https://github.com/k8sgpt-ai/k8sgpt), [backend registry](https://github.com/k8sgpt-ai/k8sgpt/blob/main/pkg/ai/iai.go), [MCP server source](https://github.com/k8sgpt-ai/k8sgpt/blob/main/pkg/server/mcp.go), [k8sgpt-operator](https://github.com/k8sgpt-ai/k8sgpt-operator), [CNCF page](https://www.cncf.io/projects/k8sgpt/). - kagent: [GitHub](https://github.com/kagent-dev/kagent), [kagent.dev](https://kagent.dev/), [CNCF page](https://www.cncf.io/projects/kagent/). - Project facts, stars and releases checked 2026-10-21. We retest when either tool ships a major version. Aurora is on [GitHub](https://github.com/Arvo-AI/aurora) under Apache 2.0. To see it on your own alerts, [book a demo](https://cal.com/arvo-ai/demo). --- ## Open Source SRE Agents in 2026: 9 Compared, 3 Tested URL: https://www.aurorasre.ai/blog/top-ai-sre-tools-2026 Published: 2026-05-18 (updated 2026-10-21) Author: Ben Gervais, Engineer at Arvo AI > Open source AI SRE agents and alternatives, checked October 21, 2026: Aurora, HolmesGPT and K8sGPT run on the same 19 Kubernetes faults, 6 more reviewed. The strongest open source SRE agents for investigating a live alert in October 2026 are [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) and [Aurora](https://github.com/Arvo-AI/aurora). On the same 19 injected Kubernetes faults, two runs each, Aurora was graded correct in 34 of 38 runs (grades read by a person) and HolmesGPT in 31 of 38 (judge-model grades only), a gap that is not statistically significant, and HolmesGPT was about 3x faster. [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) is a different kind of tool: it scans a cluster and explains what its analyzers flag, rather than investigating an alert. If you want to build your own agent rather than run one, look at OpenSRE, kagent and Mezmo's AURA, which we have not tested. Arvo builds Aurora and wrote the 19 scenarios. Everything we ran is public in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab), and the live table is at [/benchmark](/benchmark). ## Which open source SRE agents exist, and which did we test? Stars, licenses and releases were read from the GitHub API on 2026-10-21. "Tested" means we installed the tool and ran it on all 19 faults. "Repo check" means we read the README, docs and license, and did not run it. | Project | How we checked | License | Stars (2026-10-21) | Latest release | What it is | Self-host | Model choice | |---|---|---|---|---|---|---|---| | [Aurora](https://github.com/Arvo-AI/aurora) | Tested (commit 87d22baf) | Apache 2.0 | 439 | v1.5.10, 2026-10-16 | Alert-driven investigation agent with RCA, suggested fixes, postmortems | Docker Compose, Helm, air-gapped bundle | OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, OpenRouter, Ollama | | [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) | Tested (0.43.0) | Apache 2.0 | 3,652 | 0.45.0, 2026-10-15 | Investigation agent (CLI, server, operator mode), CNCF Sandbox | CLI, Python, Docker Compose, Kubernetes | OpenAI, Anthropic, Azure, Bedrock, Gemini and more | | [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) | Tested (0.4.40) | Apache 2.0 | 8,265 | v0.4.41, 2026-10-21 | Kubernetes analyzers plus an LLM that explains the findings | CLI, in-cluster operator | OpenAI, Azure, Cohere, Bedrock, Gemini, LocalAI, Ollama | | [OpenSRE](https://github.com/Tracer-Cloud/opensre) | Repo check | Apache 2.0 | 12,337 | v0.1.2026.10.21, 2026-10-21 | Framework for building AI SRE agents, public alpha | Self-hosted container or EC2 gateway; a managed OpenSRE Cloud gateway also exists | Anthropic, OpenAI, Ollama, Gemini, Bedrock, OpenRouter and more | | [kagent](https://github.com/kagent-dev/kagent) | Repo check | Apache 2.0 | 4,066 | v1.0.0-alpha16, 2026-10-19 | Kubernetes-native framework for agents defined as custom resources, CNCF Sandbox | In-cluster | OpenAI, Azure OpenAI, Anthropic, Vertex AI, Ollama, gateways | | [Mezmo AURA](https://github.com/mezmo/aura) | Repo check | Apache 2.0 | 403 | v0.2.19, 2026-10-09 | SRE agent runtime with multi-agent teams defined in TOML | Your infrastructure, including air-gapped | OpenAI, Anthropic, Bedrock, Gemini, Ollama, OpenRouter | | [Keep](https://github.com/keephq/keep) | Repo check | MIT, except an `ee/` directory | 12,427 | v0.54.3, 2026-09-09 | Alert management and AIOps: dedup, correlation, workflows | Docker Compose, Helm | Anthropic, OpenAI, DeepSeek, Ollama, LlamaCPP, Grok, Gemini | | [Coroot](https://github.com/coroot/coroot) | Repo check | Apache 2.0 | 7,977 | v1.27.4, 2026-10-15 | eBPF observability with built-in inspections; AI RCA is a paid add-on | Docker, Kubernetes | Not documented for the open source edition | | [Versus Incident](https://github.com/VersusControl/versus-incident) | Repo check | MIT | 897 | v1.4.31, 2026-10-17 | Log anomaly detection agent plus incident routing and on-call | Docker | OpenAI, DeepSeek, Qwen, Ollama, Claude, Gemini, any OpenAI-compatible endpoint | Stars measure attention, not quality. OpenSRE and Keep lead on stars; Aurora has the fewest of the three we tested. ## How we evaluated the three we tested We built a small app (`shop`: an nginx frontend, a Python api, a worker, redis and a load generator) on a kind v0.33.0 cluster running Kubernetes v1.37.0 with 3 nodes. Each scenario injects one known fault (two in scenario 12) with a `break.sh` script, and a `check.sh` proves the fault is visible before any agent starts. Each tool ran twice per scenario: 38 runs per tool, 114 in total. All three used the same cluster-admin kubeconfig, each behind its own guard: Aurora's command policy, HolmesGPT's command allow-list, and K8sGPT's read-only `analyze`. What each tool received: - **Aurora** (commit 87d22baf, model `bedrock/us.anthropic.claude-sonnet-5`): the alert text plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." It ran under its default read-only `observability_only` command policy, with a fresh user and org per run so no memory carried over. - **HolmesGPT** 0.43.0 (same model): the alert text plus "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." through `holmes ask` with default toolsets. The lab has no Alertmanager, so `holmes investigate` was not used. No HolmesGPT transcript shows a mutating command. - **K8sGPT** 0.4.40: no alert, because it has no notion of one. It ran the command below. It used `us.anthropic.claude-sonnet-4-5`, because K8sGPT sends a temperature parameter that Bedrock rejects for Sonnet 5. ``` k8sgpt analyze --explain --namespace shop ``` An answer is **correct** when it states every required finding for the scenario and asserts none of the listed wrong answers; **partial** when it states some. A judge model (`us.anthropic.claude-opus-5-5`) graded every answer against the injected fault. Review is not the same for all three tools: - Aurora's 38 grades were read and confirmed by a person at Arvo, who changed none. - HolmesGPT's and K8sGPT's 76 grades are judge-model only. A second model re-read all 76 and agreed. A third pass graded all 76 from scratch: it agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels. No person has checked these 76 yet. - A blinded regrade, with agent names, tool names and the cluster name removed, matched the published verdict in 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs. Every change moved a verdict down. The prompts differ in wording, as quoted above. HolmesGPT's own command allow-list blocked `kubectl rollout history` in 12 of its runs, including 2 of its 7 non-correct runs, and we did not rerun those with the command allowed. Aurora's policy had the same block in [our first lab report](/blog/aurora-lab-report-kubernetes-19-failures). The judge model comes from the same vendor as every model under test. ## Aurora vs HolmesGPT vs K8sGPT: results on 19 Kubernetes faults | | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | Version | commit 87d22baf | 0.43.0 | 0.4.40 | | Correct | 34 of 38 | 31 of 38 | not scored as a peer (see below) | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 | | "No problems detected" | 0 | 0 | 12 | | Correct in the blinded regrade | 34 | 30 | 0 | | Median time per run | 153.5 s | 45 s | 12.5 s for two analyze passes | | Median cost per run | $0.35 | not captured | not captured | | Who confirmed the grades | a person | judge model, two model reviews | judge model, two model reviews | On correct against not correct, the two-sided Fisher exact test that [/benchmark](/benchmark) computes from these totals gives p = 0.52 for Aurora against HolmesGPT (34 of 38 against 31 of 38). It compares Aurora's person-read grades with HolmesGPT's judge-only grades; on the blinded regrade (34 against 30) the same test gives p = 0.35. With 38 runs each and a model judge, read the two as roughly even on accuracy. HolmesGPT is clearly ahead on speed: its median run took 45 seconds against Aurora's 153.5. Aurora's times include contention from its own background jobs in the lab stack, which the lab repo's REPORT.md describes, but that does not close a 3x gap. Per scenario (correct runs out of 2; for K8sGPT, the judge's labels): | # | Fault | Aurora | HolmesGPT | K8sGPT | |---|---|---|---|---| | 01 | Worker OOMKilled after a batch size change | 2/2 | 2/2 | partial, partial | | 02 | Log shipper sidecar OOMKilled, app container healthy | 2/2 | 2/2 | partial, partial | | 03 | Frontend rollout stuck on a mistyped image tag | 2/2 | 2/2 | partial, partial | | 04 | New api pods blocked on a Secret that does not exist | 2/2 | 2/2 | partial, partial | | 05 | Frontend crash loop from an invalid nginx config | 2/2 | 2/2 | partial, partial | | 06 | Liveness probe targets the wrong port | 2/2 | 2/2 | no problems detected | | 07 | api Service selects no pods after a selector change | 2/2 | 2/2 | partial, partial | | 08 | Worker Pending, CPU request larger than any node | 2/2 | 2/2 | partial, partial | | 09 | redis Pending on an unbindable PVC | 2/2 | 2/2 | partial, partial | | 10 | api crash loop from a wrong REDIS_HOST | 2/2 | 1/2, 1 partial | wrong, wrong | | 11 | Wrong api Service targetPort, next to a noisy but harmless worker | 1/2, 1 partial | 0/2, 2 partial | no problems detected | | 12 | Two faults at once; the alert covers only one | 0/2, 2 partial | 0/2, 2 partial | partial, partial | | 13 | NetworkPolicy blocks api to redis | 2/2 | 2/2 | no problems detected | | 14 | ResourceQuota forbids the rollout's surge pod | 2/2 | 2/2 | partial, partial | | 15 | Stray Deployment's pods carry app=api | 2/2 | 2/2 | no problems detected | | 16 | dnsConfig search domains for another cluster domain | 2/2 | 2/2 | wrong, wrong | | 17 | Slow memory leak in the worker; logs look clean | 2/2 | 2/2 | no problems detected | | 18 | Init container waits for a Service that does not exist | 2/2 | 1/2, 1 partial | partial, wrong | | 19 | ConfigMap edited, only one of two frontend pods restarted | 1/2, 1 partial | 1/2, 1 wrong | no problems detected | Every row links to its transcripts and grade files on [/benchmark](/benchmark). ### Where both agents failed Scenario 12 broke both. The worker is OOMKilled and the alert says so; separately, the frontend Service selects no pods. Both tools found the OOM in both runs and neither reported the frontend outage. Aurora went further and wrote that it did not exist: > Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]. Pod status was its only evidence. The runbook [a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes) shows the checks that would have caught it. Scenario 19 is the other hard one: only the restarted frontend pod loaded an edited ConfigMap. HolmesGPT's first run blamed a cold-start race between frontend and api pods and closed with "All data points to a transient startup race, now resolved." That is its one wrong verdict. Aurora's first run found the edited upstream but did not work out that only the recreated pod had loaded it. It tried to read the live config with `kubectl exec`, its read-only policy blocked that, and it missed the clue already in its own `get pods` output: the failing pod was 9 seconds younger than the healthy one. Both got it right on the second run. The runbook is [ConfigMap change not applied to all pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes). ### What K8sGPT does and does not do K8sGPT answers a different question. It runs analyzers over the objects in a namespace and has the model explain each finding; it takes no alert and does not follow a chain of evidence. Grading it on alert-driven criteria tells you what it misses, not that it failed the same task. In 12 of 38 runs (both runs of 6 scenarios) it printed "No problems detected", including the NetworkPolicy, the slow leak and the config drift, where every object looks healthy to an analyzer. In the other 26 it reported a symptom on a related object; the judge graded 21 partial and 5 wrong. The second model agreed with all of those labels; the third, independent pass would have called 6 of them "no answer", because they restate a symptom and give no cause. When the fault is visible on an object, it is fast and concrete. This is the first finding in its output for scenario 03, run 1, verbatim: ``` 0: Pod shop/frontend-7b6c998456-rv848(Deployment/frontend) - Error: Back-off pulling image "nginx:1.27-alpne": ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.27-alpne": failed to resolve reference "docker.io/library/nginx:1.27-alpne": docker.io/library/nginx:1.27-alpne: not found Error: The Docker image "nginx:1.27-alpne" cannot be found because the tag name is misspelled. ``` The judge marked it partial because it never said the old ReplicaSet was still serving, and its Service finding implied an outage. K8sGPT's median was 12.5 seconds for two passes, it can anonymize resource names before they reach the model, and its operator now has an opt-in, alpha [auto-remediation mode](https://github.com/k8sgpt-ai/k8sgpt-operator). For a cluster health scan it is the cheapest of the three to run. For "this alert fired, why?" it is the wrong tool. More in [HolmesGPT vs K8sGPT](/blog/holmesgpt-vs-k8sgpt). ## The open source SRE agents we did not test These were checked from their repos and docs on 2026-10-21. We did not run them, so nothing below is a statement about how well they investigate. Any of them can be added to the benchmark (see the end of this page). ### OpenSRE (Tracer-Cloud) A framework for building AI SRE agents, and the most starred project here. The README lists 60+ integrations across observability, cloud, databases and incident tools, a tool-calling loop that answers with evidence-linked responses, optional masking of identifiers before external LLM calls, and optional execution of remediation. It is marked public alpha, and the interactive shell requires signing in to an account. Self-hosting is a container with `LLM_PROVIDER` set. Product telemetry is opt-out. Pick it if you want to assemble your own agent and are comfortable with an alpha API. ### kagent A [CNCF Sandbox project](https://www.cncf.io/projects/kagent/) (accepted 2025-05-22) for running agents inside Kubernetes. Agents, their prompts, tools and models are custom resources, so they live in Git and deploy through GitOps. It ships an MCP server with tools for Kubernetes, Istio, Helm, Argo, Prometheus, Grafana and Cilium, and stores sessions in PostgreSQL. It is a framework: you write the SRE agent. The latest release is still an alpha of 1.0. This is the closest match for "investigations and remediation workflows versioned alongside application code". ### Mezmo AURA An SRE agent runtime from Mezmo, written in Rust. Models, specialist agent teams, prompts, tools, approval policies and guardrails are all defined in reviewable TOML. Sensitive tool calls can require human approval, and the README says denial, timeout and transport failures are handled fail-closed. It exports OpenTelemetry traces of every model and tool call and connects tools over MCP. Mezmo says it began as its internal harness for operating its own SaaS. If your requirement is multi-agent workflows with explicit approval gates that you own, it is worth a trial. ### Keep An alert management and AIOps platform: one place for alerts from many monitors, with deduplication, correlation, enrichment and workflows. Its AI backends are used for enrichment, correlation and gathering incident context. It does not present itself as an investigation agent; it sits in front of one. See [Keep vs Aurora](/blog/keep-vs-aurora-open-source-aiops). ### Coroot Open source eBPF observability (metrics, logs, traces, profiles) with predefined inspections that explain many problems without an LLM. Its AI root cause analysis is not in the open source edition: per [Coroot's docs](https://docs.coroot.com/ai/), it comes through Coroot Cloud (10 free investigations a month) or Coroot Enterprise. Aurora has a Coroot connector, so the two can be used together. ### Versus Incident A self-hosted incident tool with an agent that learns normal log patterns and opens an incident when a new error appears, with `training`, `shadow` and `detect` modes so you can watch its judgement before it alerts. It routes incidents to Slack, Teams, Telegram and email and to PagerDuty, Opsgenie, incident.io and ServiceNow. Its focus is detection and notification; investigation is an AI triage summary on each incident. Also found and left out: [aptible/unpage](https://github.com/aptible/unpage) is archived; [fuzzylabs/sre-agent](https://github.com/fuzzylabs/sre-agent) has had no commits on its default branch since March 2026; [microsoft/sre-agent](https://github.com/microsoft/sre-agent) holds material for Azure SRE Agent, which is a hosted Microsoft product, not an open source agent (see our [Azure SRE Agent alternative](/blog/azure-sre-agent-alternative-open-source)). ## Which open source agent fits which team? | If you need | Look at | Why | |---|---|---| | A ready agent that investigates alerts in Kubernetes today | HolmesGPT or Aurora | Both tested; roughly even on correct runs, HolmesGPT about 3x faster | | The same agent across AWS, Azure, GCP and Kubernetes with RCA, fix PRs and postmortems | Aurora | Postmortems export to Confluence, Notion and SharePoint; fixes go to a PR a human approves | | A fast, cheap cluster health scan | K8sGPT | Analyzer-based, 12.5 s median for two passes | | To build your own agent and own the workflow as code | kagent, Mezmo AURA, OpenSRE | Agents defined in Kubernetes resources, TOML or Python; none tested by us | | Alert dedup and correlation in front of an agent | Keep | An alert platform, not an investigator | | Observability with built-in inspections | Coroot | AI RCA needs Coroot Cloud or Enterprise | ## What about closed AI SRE platforms like Resolve AI, Datadog Bits AI and Dynatrace Davis? We have not tested any closed AI SRE product, and none is on the benchmark, so we cannot give you a number. What the open source tools give you that closed ones cannot is inspection: you can read the agent's prompts and tool calls, run it inside your own network, pick and swap the model, and keep its memory in your own database. In Aurora, knowledge-base write-back needs a human thumbs-up, and all of it lives in your deployment. The trade is that you operate it. Each closed platform has its own page, checked against the vendor's docs: - Resolve AI: [Resolve AI alternatives](/blog/resolve-ai-alternative-open-source) and [Cleric vs Resolve AI](/blog/cleric-vs-resolve-ai) - Datadog Bits AI SRE: [Datadog Bits AI alternatives](/blog/datadog-bits-ai-sre-alternative-open-source) - Dynatrace Davis AI: [Dynatrace Davis AI alternatives](/blog/dynatrace-davis-alternative-open-source) - LogicMonitor Edwin AI: [Edwin AI vs Dynatrace Davis AI vs Datadog Bits AI](/blog/edwin-ai-logicmonitor-alternatives) - Azure SRE Agent: [Azure SRE Agent alternative](/blog/azure-sre-agent-alternative-open-source) - New Relic AI and Splunk ITSI: [New Relic AI alternative](/blog/new-relic-ai-alternative-open-source), [replacing Splunk ITSI](/blog/splunk-itsi-alternative-open-source) - Incident platforms with AI features: [incident.io](/blog/incident-io-alternative-open-source), [Rootly](/blog/rootly-alternative-open-source-incident-management), [PagerDuty](/blog/pagerduty-alternative-root-cause-analysis) - Prices across all of them: [AI SRE and incident tool pricing](/blog/ai-sre-incident-tool-pricing-2026) For deployment inside your own network, see [self-hosted AI SRE](/blog/self-hosted-ai-sre). ## Can I rerun the test, or add another agent? Yes. The scenarios, runner, every transcript and every grade are in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab): ``` git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab ./lab.sh up && ./lab.sh break 03-bad-image-tag ./runner/h2h.sh 03-bad-image-tag runs/mine ``` `h2h.sh` runs HolmesGPT and K8sGPT on Bedrock, so it needs AWS credentials with Bedrock access (`AWS_PROFILE`, `AWS_REGION`). To add an agent (OpenSRE, kagent and AURA are the obvious next ones), open an issue with the "Benchmark: add an agent" template: a command that takes the alert text and a read-only kubeconfig and prints the root cause. We run it on all 19 faults, grade it with the same judge prompt, have a person check the verdicts and publish the result. To check one failure against your own cluster, [/diagnose](/diagnose) walks through it. The three-way write-up with more traces is [Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt). ## What we would change next time The test is Kubernetes-only, on a synthetic app, with faults we wrote. It says nothing about cloud APIs, logs at scale, or noisy production alerts. HolmesGPT 0.44.0 and 0.45.0 were already out when we ran 0.43.0, and K8sGPT 0.4.41 shipped on 2026-10-21, after our runs, so their current versions are untested. HolmesGPT's grades still need a person to read them, and its cost per run was not captured. We plan to rerun all three every quarter and when any of them ships a major version. Verified 2026-10-21: repo stars, licenses and releases from the GitHub API; benchmark numbers from [results.json](https://raw.githubusercontent.com/Arvo-AI/aurora-lab/main/benchmark/results.json) generated 2026-10-21; HolmesGPT and K8sGPT runs on 2026-10-15 and 2026-10-16; Aurora runs on 2026-10-02. Aurora is on [GitHub](https://github.com/Arvo-AI/aurora) under Apache 2.0, or [book a demo](https://cal.com/arvo-ai/demo) to see it on your own alerts. --- ## Self-Hosted AI SRE: The 2026 Deployment Guide URL: https://www.aurorasre.ai/blog/self-hosted-ai-sre Published: 2026-05-18 (updated 2026-07-02) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Self-hosted AI SRE runs the agent, memory, and LLM inside your perimeter. The 2026 architecture for air-gapped, multi-cloud, BYO-LLM deployment. > ## Key Takeaways > > - **Self-hosted AI SRE means the agent runtime, its memory layer, and the LLM all run inside the customer's perimeter.** Every inference call, every telemetry read, and every postmortem write happens on customer-owned infrastructure. The definition is structural. A vendor agent that ships data to vendor-managed inference is not self-hosted under this definition. > - **We propose the Sovereignty Spectrum.** Five deployment tiers: T1 Public SaaS, T2 Private SaaS, T3 VPC-Isolated, T4 On-Prem Hosted, T5 Air-Gapped. Of the fifteen most-cited AI SRE tools in 2026, only [Aurora](https://github.com/Arvo-AI/aurora), [HolmesGPT](https://github.com/HolmesGPT/holmesgpt), and [K8sGPT](https://www.cncf.io/projects/k8sgpt/) credibly reach T4 or T5. The other twelve top out at T1 or T2. > - **Air-gapped deployment requires three independent stacks: orchestration, memory, and inference.** Orchestration is the agent loop (LangGraph, ReAct). Memory is the dependency graph plus RAG corpus (Memgraph, Weaviate). Inference is the LLM ([Ollama](https://ollama.com/), [vLLM](https://docs.vllm.ai/), or a sovereign endpoint). All three must run locally, with no outbound network call. > - **Regulatory drivers are concrete and dated.** The [EU Data Boundary for the Microsoft Cloud was completed on 26 February 2025](https://learn.microsoft.com/en-us/privacy/eudb/eu-data-boundary-learn). The [EU AI Act implementation timeline](https://artificialintelligenceact.eu/implementation-timeline/) phases in through 2027. The [SEC adopted cybersecurity disclosure rules on 26 July 2023](https://www.sec.gov/newsroom/press-releases/2023-139) (Form 8-K Item 1.05 effective 18 December 2023). > - **Open-weight LLMs in 2026 are credible for local inference.** [Meta's Llama 3.3 70B (December 2024)](https://ai.meta.com/blog/meta-llama-3-3-70b/) delivers similar performance to Llama 3.1 405B at lower inference cost, per Meta's own announcement. Mistral, DeepSeek, and Qwen have released competitive open-weight models. Aurora's reference local stack uses Ollama with a 70B-class model. In Arvo's design-partner conversations across 2025, every regulated customer ran into the same procurement wall: every credible commercial AI SRE required production telemetry, including customer data inside log lines, error messages, and stack traces, to leave the customer perimeter for inference. For a SaaS startup the wall is paperwork. For a bank, a defence contractor, an EU sovereign-data buyer, or a healthcare provider, it blocks the procurement. **Self-hosted AI SRE removes the wall.** The agent, its memory, and the LLM all run inside the customer's perimeter. This guide is the 2026 reference for evaluating, designing, and deploying a self-hosted AI SRE, with every commercial tool mapped to its deployment tier and Aurora's air-gapped stack used as the worked example. ## What does self-hosted AI SRE mean? The phrase is overloaded. Three definitions circulate in 2026 vendor marketing, and only the strictest meaningfully reduces the trust surface. 1. **Self-hosted collector with VPC peering.** A vendor agent runs in the customer VPC, gathers telemetry, and ships it (sometimes after partial filtering) to a vendor-managed inference plane. The inference call leaves the customer perimeter. Most commercial AI SREs in 2026 use this pattern and call it "private deployment." 2. **Single-tenant SaaS.** A dedicated vendor-managed instance inside a vendor-owned cloud account. The data plane is isolated from other tenants but still vendor-operated. Inference still leaves the customer perimeter. 3. **True self-hosted.** Every component (orchestration runtime, memory layers, inference endpoint, secrets manager) runs on customer-owned infrastructure. No outbound network call is required for an investigation to complete. This guide uses the third definition. For audits and compliance reviews, only the third meaning answers the question "could a malicious actor at the vendor have read our incident transcript" with a structural no. ## The Sovereignty Spectrum Each tier increases perimeter control over the previous one. Choose the tier the team can defend operationally; aiming further than that is engineering debt waiting to happen. | Tier | What runs on customer infrastructure | What leaves the perimeter | Representative tools | |---|---|---|---| | **T1, Public SaaS** | Nothing | Telemetry, transcripts, investigation prompts | Datadog Bits AI, incident.io AI SRE, Rootly AI, PagerDuty SRE Agent, ServiceNow Now Assist, Splunk ITSI, Cleric.ai, Causely | | **T2, Private SaaS (VPC peering)** | A vendor-supplied agent or collector | Telemetry, embeddings, sometimes whole log lines, all inference calls | Resolve.ai (satellite agent), Traversal, NeuBird Hawkeye (VPC option), Edwin AI | | **T3, VPC-Isolated single-tenant** | Vendor-managed control plane inside a vendor-owned cloud account dedicated to one customer | All inference calls; cross-tenant data flow is structurally absent, the vendor still operates the plane | Some incumbent "private cloud" tiers (custom-quoted) | | **T4, On-prem hosted, hosted LLM** | Agent, memory, dependency graph, RAG corpus | LLM API calls to OpenAI, Anthropic, Google, or Bedrock | Aurora with managed LLM; HolmesGPT with managed LLM | | **T5, Air-gapped** | Agent, memory, dependency graph, RAG corpus, and a local LLM via Ollama, vLLM, or a sovereign endpoint | Nothing. Investigation completes without an outbound call | Aurora with Ollama; HolmesGPT with self-hosted LLM endpoint; K8sGPT with local LLM (Kubernetes-only scope) | A team's deployment tier is fixed by its strictest constraint, not its average. The [FINMA Circular 2018/03](https://www.finma.ch/en/~/media/finma/dokumente/dokumentencenter/myfinma/rundschreiben/finma-rs-2018-03-20180101.pdf?la=en) on outsourcing for Swiss banks and insurers pushes regulated workloads toward T5. A privacy-by-design product advertising "your incident data never leaves your servers" lands at T5. A team that cannot obtain controller approval for an LLM provider under [GDPR Article 28](https://gdpr-info.eu/art-28-gdpr/) lands at T5. Any other constraint allows T3 or T4. A single strict regulator collapses the choice to T5. ## Why does self-hosting matter in 2026? Three pressures, in roughly this order. **Regulatory.** The [EU Data Boundary for the Microsoft Cloud was completed on 26 February 2025](https://learn.microsoft.com/en-us/privacy/eudb/eu-data-boundary-learn). The boundary covers data processing and storage for core services and is the model EU procurement teams now apply to other vendors. The [EU AI Act timeline](https://artificialintelligenceact.eu/implementation-timeline/) phases in through 2027, with [high-risk system obligations under Chapter III](https://artificialintelligenceact.eu/chapter/3/) (risk management, data governance, human oversight, post-market monitoring) applicable to operational AI used in critical infrastructure. The [SEC's cybersecurity disclosure rules](https://www.sec.gov/newsroom/press-releases/2023-139) (adopted 26 July 2023, Form 8-K Item 1.05 effective 18 December 2023) make incident response transparency a public-company concern. **Sovereignty and latency.** Sovereign cloud is no longer a French preoccupation. [OVHcloud Sovereign Cloud](https://www.ovhcloud.com/en/enterprise/products/hosted-private-cloud/), [Scaleway](https://www.scaleway.com/en/), [T-Systems Sovereign Cloud](https://www.t-systems.com/de/en/cloud-services/sovereign-cloud), [Stackit](https://www.stackit.de/en/) (Schwarz Group), and [Oracle EU Sovereign Cloud](https://www.oracle.com/cloud/sovereign-cloud/eu/) ship contractually sovereign tiers. An AI SRE that cannot operate without sending telemetry to a US hyperscaler region is unfit for these workloads. Latency follows the same constraint: an EU-hosted agent calling a US-hosted LLM during an incident incurs round-trip latency on every step of a multi-turn investigation. **Data leakage and trust.** Production log lines frequently contain customer PII, secrets, and proprietary identifiers. [GitGuardian's State of Secrets Sprawl 2024](https://www.gitguardian.com/state-of-secrets-sprawl-report-2024) found 12.8 million new exposed secrets across public repositories alone in 2023, a steady reminder that telemetry contains material auditors care about. The audit calculation for a security team is the same as for any third-party data flow: if it can leak, model the risk as if it will. T5 makes the model trivial because nothing leaves the perimeter. For the full incident-investigation context, see [AI-Powered Incident Investigation: The Complete Guide for SRE Teams](/blog/ai-powered-incident-investigation). ## Which AI SRE tools can be fully self-hosted? The honest map. | Tool | Best achievable tier | Constraint | |---|---|---| | **Aurora** | T5, Air-Gapped | Reference stack: Docker Compose or Helm chart, Ollama local LLM, Vault, Memgraph, Weaviate. See the [Aurora repo](https://github.com/Arvo-AI/aurora). | | **HolmesGPT** | T4, On-prem with hosted LLM (T5 with self-hosted LLM endpoint) | Apache 2.0. Per the [HolmesGPT docs](https://holmesgpt.dev/), documentation assumes a hosted model provider (OpenAI, Azure OpenAI, Bedrock). Self-hosted LLM is an advanced configuration. | | **K8sGPT** | T4, On-prem (T5 with local LLM, Kubernetes scope only) | CLI or Helm. [Local LLMs via Ollama supported](https://docs.k8sgpt.ai/). Scope is limited to the Kubernetes API. | | **Resolve.ai** | T2, Private SaaS | Satellite agent in the customer VPC for telemetry. Inference is vendor-managed. No publicly documented air-gapped option. | | **Traversal** | T2, Private SaaS | Flexible deployment options. Inference is vendor-managed. | | **NeuBird Hawkeye** | T2, Private SaaS (VPC) | VPC deployment available. Ephemeral telemetry processing claimed by NeuBird. Inference path is vendor-managed. | | **Causely** | T1, Public SaaS | Kubernetes-only. SaaS control plane. | | **Cleric.ai** | T1, Public SaaS | Slack-first SaaS. | | **PagerDuty SRE Agent** | T1, Public SaaS | Inside [PagerDuty Operations Cloud](https://www.pagerduty.com/platform/ai-agents/sre/). | | **Datadog Bits AI SRE** | T1, Public SaaS | Multi-tenant inside Datadog. HIPAA-compliant per [Datadog's documentation](https://www.datadoghq.com/product/ai/bits-ai-sre/), not air-gapped. | | **incident.io AI SRE** | T1, Public SaaS | Hosted multi-tenant. AI SRE access design-partner-gated. | | **Rootly AI** | T1, Public SaaS | Closed-core SaaS. [Rootly AI Labs](https://rootly.com/labs) publishes open-source prototypes. | | **ServiceNow Now Assist SRE** | T1, Public SaaS | ServiceNow cloud. GA targeted June 2026. | | **Edwin AI (LogicMonitor)** | T2, Private (LogicMonitor-managed) | Bundled with LogicMonitor Envision platform. Not standalone. | | **Splunk ITSI Episode Summarization** | T1, Public SaaS | Splunk Cloud only as of May 2026 (Alpha). | The open-source projects are the only tools today that credibly reach T4 or T5 with public documentation. Aurora is the only one with multi-cloud scope at T5. Resolve.ai, Traversal, NeuBird, and Datadog Bits AI publish FedRAMP-adjacent or HIPAA tiers but no air-gapped reference architecture as of May 2026. For the broader category overview, see our [open-source incident management overview](/blog/open-source-incident-management) and the [Aurora Actions launch post](/blog/aurora-actions-background-automations) for scheduled and event-triggered automations on top of self-hosted Aurora. ## What is the architecture of a self-hosted AI SRE? A self-hosted agentic AI SRE has three concurrent runtime stacks. Skip any one and the deployment regresses to a lower sovereignty tier. ### 1. Orchestration runtime The agent loop is the LangGraph, ReAct, or equivalent orchestration that decides what tool to call next. It is the smallest of the three stacks by resource footprint and the easiest to self-host. Requirements: - A Python or Node runtime, typically containerised. - A task queue ([Celery](https://docs.celeryq.dev/), [RQ](https://python-rq.org/), [BullMQ](https://docs.bullmq.io/)) for long-running investigations. - Postgres for agent state, investigation records, and audit logs. - A secrets store ([HashiCorp Vault](https://developer.hashicorp.com/vault), [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/), or [KMS](https://cloud.google.com/security/products/security-key-management)) for cloud credentials and LLM keys. - A web UI or API surface for engineers to inspect and trigger investigations. Aurora ships this stack as a Docker Compose for single-node deployment and a Helm chart for Kubernetes-native deployment, both [documented in the repo](https://github.com/Arvo-AI/aurora). ### 2. Memory layer The agent without memory is a stateless inference call. Memory is the difference between an agent that learns from the environment and an agent that makes the same investigative mistake every week. - **Dependency graph.** A graph database ([Memgraph](https://memgraph.com/docs), Neo4j) that holds the live topology of the infrastructure: services, dependencies, alert sources, and ownership. The agent traverses the graph to assess blast radius and trace upstream causes before issuing tool calls. - **RAG corpus.** A vector database ([Weaviate](https://weaviate.io/developers/weaviate), [Qdrant](https://qdrant.tech/documentation/), [Chroma](https://docs.trychroma.com/)) holding embeddings of past postmortems, runbooks, design docs, and code. [Hybrid retrieval combining BM25 and vector search](https://weaviate.io/developers/weaviate/search/hybrid) outperforms either alone on SRE corpora because exact-match identifiers (service names, error codes) coexist with semantic concepts (failure modes). See also the [root cause analysis complete guide for SREs](/blog/root-cause-analysis-complete-guide-sres) for the broader investigation context. - **Event store.** Postgres or an event-sourcing database for the agent's own investigation history. Past investigations become future evidence. Aurora's reference stack is Memgraph, Weaviate, and Postgres. Each runs in a customer container, and none requires an outbound network call. ### 3. Inference layer The LLM. Three paths, in increasing sovereignty: 1. **Managed LLM API.** OpenAI, Anthropic, Google, Bedrock. Cheapest to start, lowest operational burden, but the deployment stays at T4. 2. **Private endpoint.** [Azure OpenAI dedicated](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/provisioned-throughput), [Bedrock Provisioned Throughput](https://docs.aws.amazon.com/bedrock/latest/userguide/prov-throughput.html), or a partner-hosted endpoint. Stronger contractual perimeter, although the data still leaves the customer cloud account. 3. **Local LLM.** [Ollama](https://ollama.com/), [vLLM](https://docs.vllm.ai/), or a sovereign inference appliance. Reaches T5. For T5, the inference stack is the operational lift. Hardware is the largest single line item, and team expertise is the second. ## BYO-LLM: which models run well locally? Open-weight model quality has progressed enough to anchor an agentic SRE loop in 2026. The current options: - **Llama 3.3 70B** ([Meta, December 2024](https://ai.meta.com/blog/meta-llama-3-3-70b/)). Meta states the model delivers similar performance to Llama 3.1 405B at lower inference cost. A common starting point for local deployments. - **DeepSeek-R1** ([model card](https://huggingface.co/deepseek-ai/DeepSeek-R1)). A reasoning-tuned open-weight model. - **Qwen 2.5 and 3 families** ([Qwen 2.5 release](https://qwenlm.github.io/blog/qwen2.5/)). Strong multilingual support for teams with non-English runbook content. - **Mistral Large** ([Mistral models](https://docs.mistral.ai/getting-started/models/models_overview/)). Strong tool-use performance. Hardware sizing for a 70B-class model: in float16, weights are roughly 140GB, so plan two 80GB cards (a pair of H100 or A100 80GB) or a single H200 (141GB). Q4-quantised variants compress weights to roughly 35-40GB and fit on a single 80GB card with context room, at some latency and quality cost. See the [Llama 3.3 70B model card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) for the canonical parameter and tensor sizes. Specific latency targets are workload-dependent and should be measured, not assumed. The constraint to flag: running a local LLM is a real engineering discipline. Teams without LLM-ops capacity should consider T4 (managed API) as the long-term answer and revisit T5 when the team is staffed for it. ## How does multi-cloud authentication work in a self-hosted agent? A self-hosted agent must still reach customer cloud APIs. The auth pattern matters because credentials live in the customer perimeter. Vendor-managed inference makes credential exfiltration a vendor-trust problem. Self-hosted inference makes it a customer-operations problem, which is the desired state. Aurora's reference multi-cloud auth pattern: | Cloud | Pattern | |---|---| | **AWS** | [STS AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) into customer accounts via a least-privilege investigation role. Credentials never persist in agent storage. | | **Azure** | [Service Principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal) with Reader (and incident-scoped Operator) role assignments. | | **GCP** | OAuth-based authentication or [workload identity federation](https://cloud.google.com/iam/docs/workload-identity-federation). | | **OVH** | [API key per investigation scope](https://help.ovhcloud.com/csm/en-gb-api-getting-started-ovhcloud-api), stored in Vault. | | **Scaleway** | [API token](https://www.scaleway.com/en/docs/iam/how-to/create-api-keys/) stored in Vault. | | **Kubernetes** | Kubeconfig per cluster, stored in Vault. Sandboxed kubectl execution into an isolated namespace; see our [AI Agent kubectl Safety guide](/blog/ai-agent-kubectl-safety). | The Vault binding matters: every cloud credential is short-lived where the cloud supports it, and every credential use is auditable. In a T5 deployment, the auditor's "who issued this command" question is answered by the Vault audit log and the agent's tool-call trace, not by a vendor SOC 2 attestation. ## What does an air-gapped AI SRE deployment require? The hard version requires no outbound network call during an investigation, including for inference. Aurora's air-gapped reference architecture covers six layers: 1. **Mirrored container registry.** Every image (Aurora, Memgraph, Weaviate, Postgres, Vault, Ollama) is pulled from a customer-internal registry. No Docker Hub calls. 2. **Mirrored package indices.** Python wheels and OS packages served from internal Artifactory or equivalent. 3. **Mirrored model weights.** Llama 3.3 weights downloaded once on a connected jumpbox, scanned, hashed, and copied into the air-gapped network. Same for embedding models. 4. **Local DNS.** No outbound DNS resolution required. Cloud APIs are reached via VPC private endpoints ([AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html), [Azure Private Endpoint](https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overview), [GCP Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect)). 5. **No telemetry to vendor.** Neither Aurora nor the open-source components phone home; this is verified per release. 6. **Sealed Vault.** Vault sealed and unsealed via internal HSM or [Shamir keyshares](https://developer.hashicorp.com/vault/docs/concepts/seal). No auto-unseal against a vendor KMS. The provisioning lift is real. Teams that have operated air-gapped Kubernetes will recognise the pattern. Teams that have not should pilot in a connected environment first. ## How Aurora implements the Sovereignty Spectrum Every Aurora deployment is configured for the customer's tier. The same code base supports all five. - **T1 and T2.** Aurora deployed to a public-cloud account with managed services for Postgres, Memgraph, and Weaviate. LLM via OpenAI or Anthropic API. Useful for evaluation pilots. - **T3.** Aurora deployed to a customer-owned VPC with private endpoints to managed services. LLM via private endpoint (Azure OpenAI dedicated, Bedrock). - **T4.** Aurora deployed to customer-owned VMs or Kubernetes with self-hosted Postgres, Memgraph, and Weaviate. LLM via managed API or private endpoint. - **T5.** Aurora deployed to customer-owned air-gapped infrastructure with Ollama-hosted Llama 3.3 (or a sovereign LLM endpoint). All dependencies mirrored. Aurora ships a single codebase that serves all five tiers. Tier downgrade ("drop from T5 to T3 for one workload") and upgrade ("move the EU workload from T3 to T5") become configuration changes rather than migrations. ## How does self-hosted AI SRE cost compare to SaaS? A precise total cost of ownership depends on team size, model choice, infrastructure pricing, regional rates, and incident volume. Procurement should model the variable axes against incident volume rather than anchor on a single vendor-supplied number. - **Self-hosted T4 or T5 fixed costs.** Compute for the agent runtime, memory stores, and (for T5) the LLM node. Storage for the RAG corpus and audit log. Engineering time to operate the stack. - **Self-hosted T4 variable costs.** Managed LLM API usage at provider rates ([OpenAI pricing](https://openai.com/api/pricing/), [Anthropic pricing](https://www.anthropic.com/pricing), [Bedrock pricing](https://aws.amazon.com/bedrock/pricing/)). Scales with the number and depth of investigations. - **Commercial SaaS variable costs.** Per-seat tiers (incident.io, Rootly, PagerDuty), per-investigation billing ([Datadog Bits AI](https://www.datadoghq.com/pricing/), NeuBird), or per-credit consumption (ServiceNow). All published on the vendor's pricing page. The break-even between a self-hosted Tier 5 deployment and per-investigation SaaS depends on the vendor's per-investigation price, the LLM choice, and the engineering cost of running the stack. Procurement teams should model three points: today's incident volume, twelve-month projected volume, and a 3x scenario. If any of the three is dominated by sovereignty rather than economics, the regulator decides the deployment tier, not the spreadsheet. ## When self-hosting is the wrong answer Self-hosting is an engineering commitment, not a checkbox. Teams that should skip it: - **No LLM-ops capacity.** If no one on the team has run inference servers in production, do not start with air-gapped Ollama. Pilot at T1 or T2. - **Small team, low incident volume.** Below twenty incidents per month, the operational overhead can exceed the cost savings of self-hosting. T1 is fine if the data classification allows it. - **No regulatory or sovereignty pressure.** If the compliance team is not asking and the data classification is not sensitive, the sovereignty premium is paid for nothing. - **Early in the AI SRE evaluation curve.** A managed pilot validates the value of the agent to the team. Self-host after that decision, not before it. Teams that should default to self-hosting: - Regulated workloads (finance, healthcare, defence, critical infrastructure). - EU sovereign-data customers. - Customers that advertise sovereignty as a product attribute themselves. - Public-sector buyers under FedRAMP High, IRAP PROTECTED, IL5, or equivalent. - Anyone whose log lines contain customer PII that has not been scrubbed at source. ## What to watch next Arvo expects three shifts in the self-hosted AI SRE landscape over the next twelve months. - **Sovereign LLM endpoints.** EU-hosted, contract-bound LLM endpoints from cloud regions outside US jurisdiction will turn T4 into a viable tier for European regulated customers without forcing T5. Anthropic, OpenAI, and Google are each shipping or piloting EU-resident inference. - **Air-gap reference appliances.** Appliance-style packages (preloaded GPU servers with Aurora, a local LLM, and a sealed Vault) sold as turn-key T5 deployments are likely to emerge from hardware vendors. - **Open benchmark cohorts.** Closed-source players still measure themselves on private datasets. The first open, named, multi-LLM benchmark on a public incident corpus will become the citation surface the category orbits. In 2024 self-hosted AI SRE was a theoretical option. By 2025 it was niche. In 2026 it has become the procurement default for regulated workloads. The tools that can execute it today are Aurora at the multi-cloud end, HolmesGPT at the CNCF and Kubernetes end, and K8sGPT for diagnostics. For the full landscape of AI SRE tools and how each maps to a deployment tier, see [Top 15 AI SRE Tools in 2026](/blog/top-ai-sre-tools-2026). For the broader category overview, see [AI SRE: The Complete Guide for Engineering Teams in 2026](/blog/ai-sre-complete-guide). For the investigation and postmortem halves of the workflow, see [AI-Powered Incident Investigation](/blog/ai-powered-incident-investigation) and [Automated Post-Mortem Generation](/blog/automated-post-mortem-generation). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## How AI Incident Investigation Works: Two Agents Traced URL: https://www.aurorasre.ai/blog/ai-powered-incident-investigation Published: 2026-05-13 (updated 2026-10-21) Author: Ben Gervais, Engineer at Arvo AI > AI incident investigation, step by step: every tool call two agents made on the same injected fault, where both failed and what to require. Oct 2026. AI-powered incident investigation is an LLM agent that takes an alert, runs read-only queries against the live system (`kubectl`, logs, metrics), picks each next query from what the last one returned, and stops when the evidence supports one cause. It then reports that cause with the evidence behind it and a suggested fix that a person approves. It does not replace your monitoring or paging: it does the first pass of an engineer's diagnosis, and it can be wrong in ways you can only catch by reading its trace. Below are two real investigations of the same fault, one by Aurora and one by HolmesGPT, with every tool call in order. Then the scenario where both agents failed, what happened when their commands were blocked, and what to require from any tool. The faults come from our lab: 19 injected Kubernetes failures, two runs each, all public in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab) and summarized on [/benchmark](/benchmark). Arvo AI builds Aurora and wrote the scenarios. For a list of the products in this category, see [the AI incident investigation tools list](/blog/best-ai-powered-incident-investigation-tools); this page is about how the method works. ## How does an AI agent investigate an incident? Every agent we have traced runs the same loop: 1. **Read the alert** and anything it already knows (past incidents, memory, connected clusters). 2. **Look wide:** pods, events, logs of the alerted service. 3. **Follow the first anomaly** in the output, for example an error line in a log. 4. **Rule out the obvious suspects** (is the dependency down, is the node full). 5. **Find the object that explains the anomaly** and confirm it with a second piece of evidence. 6. **Report:** cause, evidence, what it ruled out, a suggested fix. The model decides each step. Nothing in the loop forces it to look beyond what the alert points at, which is where it fails (see scenario 12 below). ## What did Aurora do, call by call? Scenario 13 of the lab, run r1, 2 October 2026. The fault: a NetworkPolicy named `redis` allows ingress to redis on 6379 only from pods labeled `app=worker`. The api pods are not allowed, so every `/api/items` request times out on its redis call and returns 503. Every pod stays Running and Ready, because the api's `/healthz` does not touch redis. The alert: > HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes (threshold 5%). Aurora (commit 87d22baf, model Claude Sonnet 5 on Bedrock) had a fresh user and organization, so no memory of earlier runs, and the lab's admin kubeconfig behind its default read-only `observability_only` command policy. These are all 25 tool calls, start times in UTC, from `steps-full.json` in the [run folder](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/13-networkpolicy-blocks-redis/r1). Indented lines are copied from the tool output, with long lines cut at the right; `[...]` marks skipped lines, and other bracketed lines are our notes. ```text 20:24:50 list_incidents, grep_memories "frontend", get_connected_clusters No matches for 'frontend' in memory. 20:24:54 kubectl get pods -n shop -o wide api-f64486c4c-2nqsk 1/1 Running 0 73s redis-59458f55d8-n6glm 1/1 Running 0 73s [... all 7 pods Running, 0 restarts] 20:24:54 kubectl get events -n shop --sort-by=.lastTimestamp 20:24:54 grep_memories "aurora-lab" 20:25:07 kubectl logs -n shop -l app=frontend --tail=100 20:25:07 kubectl logs -n shop -l app=api --tail=100 redis connection failed (attempt 1/5): [Errno 111] Connection refused connected to redis at redis:6379 api listening on :8080 [...] request failed: redis error: timed out 10.244.2.45 "GET /items HTTP/1.0" 503 - [... repeats, with /healthz returning 200 in between] 20:25:07 kubectl get configmap -n shop 20:25:25 kubectl logs -n shop -l app=redis --tail=100 1:M 02 Oct 2026 20:23:45.678 * Ready to accept connections tcp 20:25:25 kubectl describe pod -n shop -l app=redis 20:25:25 kubectl top pods -n shop redis-59458f55d8-n6glm 3m 4Mi 20:25:40 kubectl logs -n shop -l app=worker --tail=100 20:25:40 kubectl get configmap api-code -n shop -o yaml with socket.create_connection((REDIS_HOST, 6379), timeout=2) as s: 20:25:40 kubectl get svc -n shop 20:26:01 kubectl top nodes 20:26:01 kubectl describe pod -n shop -l app=worker 20:26:01 kubectl get nodes -o wide 20:26:33 kubectl get pod -n shop -l app=loadgen -o yaml 20:26:33 kubectl get networkpolicies -n shop NAME POD-SELECTOR AGE redis app=redis 2m49s 20:26:33 kubectl get deployment api -n shop -o yaml 20:26:49 kubectl describe networkpolicy redis -n shop Created on: 2026-10-02 20:23:54 +0000 UTC [...] PodSelector: app=redis Allowing ingress traffic: To Port: 6379/TCP From: PodSelector: app=worker 20:26:49 kubectl get pod -n shop -l app=api --show-labels api-f64486c4c-2nqsk 1/1 Running 0 3m8s app=api,pod-template-hash=f64486c4c 20:27:41 write_memory, again at 20:27:56: "shop namespace frontend 5xx caused by redis NetworkPolicy excluding api pods" ``` What turned the investigation, in order: - **The api log at 20:25:07.** `redis error: timed out` moved the question from "why is the frontend failing" to "why can the api not reach redis". The first line, "Connection refused", is from startup before redis was up, and the api then connected. The later errors are timeouts, not refusals: the hint that packets are being dropped, not rejected. - **Ruling redis out at 20:25:25.** Clean startup log, Ready, 3m CPU. The agent now knows the dependency is up and the path to it is broken. - **The detour at 20:25:40 to 20:26:01.** Worker logs, nodes, node usage. None of it was needed. It read the api source from the ConfigMap, which gave it the 2 second client timeout it later cited. - **The policy at 20:26:33.** `get networkpolicies` was call 20 of 25, about 1 minute 40 seconds after the first call. `describe` showed the `app=worker` allow-list, and listing the api pods' labels confirmed they carry `app=api`. Two pieces of evidence, one from each side of the blocked path. The report's root cause: > The root cause was a NetworkPolicy named `redis` in the shop namespace, created at 2026-10-02T20:23:54Z, which restricts ingress to redis:6379 to pods labeled `app=worker` only [20, 22]. The `api` pods are labeled `app=api`, not `app=worker` [23], so their connections to redis were silently dropped by the policy [...] The bracketed numbers point at the tool calls above, so each claim can be checked against its output. The report also had a "Ruled Out" list (redis overload, node contention, startup noise) and a "Not Checked" list. Separately, Aurora suggested a fix with a risk rating: ```text title: Patch `redis` NetworkPolicy to allow `app=api` ingress risk: low command: kubectl patch networkpolicy redis -n shop --type='json' -p='[{"op":"add","path":"/spec/ingress/0/from/-","value":{"podSelector":{"matchLabels":{"app":"api"}}}}]' undo: null ``` Nothing ran it; suggestions wait for a person. Note the empty `undo`: to roll that patch back you would re-apply the original policy manifest. Run total: 202 seconds, 25 tool calls, 38 model calls, $0.69 in LLM cost. Graded correct. The model review still found two overstatements in the report: it called the young pod ages "a full cluster/deployment restart", and it said "100% failure" when one api log showed a 200 before the policy took effect. The [NetworkPolicy blocking traffic runbook](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes) covers the fix by hand. ## What did HolmesGPT do on the same fault? HolmesGPT 0.43.0, same model, same cluster setup and fault, run on 15 October 2026 with `holmes ask` and its default toolsets. It got the same alert text with a slightly different suffix: "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." (Aurora's was "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause."). HolmesGPT prints each call as it goes. Its 26 calls, grouped by model turn, from `holmes.md` in the [run folder](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/h2h-v1/r1/13-networkpolicy-blocks-redis): | Turn | Calls | What it asked for | |---|---|---| | 1 | #1 to #3 | frontend pods, deployments, Services | | 2 | #4 to #6 | frontend pod logs, Services in shop, ConfigMap names | | 3 | #7 to #9 | Service IPs, the `frontend-nginx` ConfigMap, pod status | | 4 | #10 to #11 | full pod state, api deployments | | 5 | #12 to #15 | logs of both api pods, the `api-code` ConfigMap, api env and ports | | 6 | #16 to #18 | redis pod state and logs, `kubectl top pods -A` | | 7 | #19 to #21 | events in shop, loadgen spec, node status (from the `aks/node-health` toolset, on a kind cluster) | | 8 | #22 to #25 | `kubectl get networkpolicies`, top pods per node, worker resources | | 9 | #26 | pod labels | Its conclusion, after 110 seconds: > A NetworkPolicy named redis in namespace shop (created 2026-10-15T09:06:57Z, applied via kubectl apply) restricts ingress to the redis pod (redis-59458f55d8-kspkb, label app=redis) to only allow traffic from pods labeled app=worker, on TCP/6379 It then proposed adding `app=api` to the policy's `from` list and listed what it checked and found healthy. Graded correct. Its report says the policy was applied "4s later" than the api's first redis PING, which the judge noted is inferred from timestamps rather than shown. The two investigations followed the same path: logs of the failing service, the api's redis timeout, redis ruled out, then the policy late in the run (call 20 of 25 for Aurora, 22 of 26 for HolmesGPT). Neither went to NetworkPolicies first, even though a timeout to a healthy pod in the same namespace points there. | Scenario 13 | Aurora r1 | Aurora r2 | HolmesGPT r1 | HolmesGPT r2 | K8sGPT r1, r2 | |---|---|---|---|---|---| | Verdict | correct | correct | correct | correct | "No problems detected" | | Time | 202 s | 444 s | 110 s | 118 s | under 1 s | | Tool calls | 25 | 19 | 26 | 22 | analyzers only | | LLM cost | $0.69 | $0.54 | not captured | not captured | not captured | | Date | 2026-10-02 | 2026-10-02 | 2026-10-15 | 2026-10-16 | 2026-10-15/16 | Aurora's grades were read by a person at Arvo; HolmesGPT's and K8sGPT's are judge-model only (details below). Aurora r2's 444 seconds includes waiting for a worker slot taken by Aurora's own background discovery jobs in the lab stack. K8sGPT does not take an alert: `k8sgpt analyze` scans the namespace for objects in a bad state, and with every pod Running and Ready its default analyzers found nothing. That is a different job, not a failed investigation. ## Where does AI incident investigation fail? Scenario 12 is the clearest failure we have. Two unrelated changes went out together: the worker's `BATCH_SIZE` went from 500 to 2500 (OOMKilled against a 192Mi limit), and the frontend Service's selector was changed to `app=frontend,tier=web`, which no pod carries. The alert covered only the worker: > KubePodCrashLooping [warning] namespace=shop deployment=worker: Pod shop/worker (container worker) is in waiting state (reason: CrashLoopBackOff). | Scenario 12 | Tool calls | Found the OOM and the batch size change | Checked Services or endpoints | Said about the frontend | |---|---|---|---|---| | Aurora r1 | 18 | yes | no | "Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]." | | Aurora r2 | 14 | yes | no | "Other workloads in the namespace (api, frontend pods) were running normally and unaffected [4]." | | HolmesGPT r1 | 4 | yes | no | nothing | | HolmesGPT r2 | 6 | yes | no | nothing | | K8sGPT r1, r2 | analyzers | reported the OOMKill, not the batch size change | listed the Service | "Service has no endpoints, expected label tier=web" | All four agent runs were graded partial. Both agents did good work on the alerted fault (Aurora in both runs and HolmesGPT in r2 worked out the memory per batch from the worker's code) and then stopped. Aurora's version is worse: in both runs it declared the frontend unaffected and cited `get pods` output as proof (calls 5 and 14 in r1, call 4 in r2), while the Service in front of those pods had no endpoints. A citation shows where a claim came from, not that the evidence supports it. K8sGPT, which scans everything instead of following the alert, saw the second fault in both runs, but its suggested fix was to label the pods by hand with `tier=web` rather than revert the selector. The lesson for the method: an investigation is scoped by its alert. "Everything else is healthy" needs a request-path check (Service endpoints, the load generator's or users' errors), not pod status. The [runbook for a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes) is built on these runs, and the [Service with no endpoints runbook](/runbooks/fix-service-no-endpoints-kubernetes) covers the selector fix. ## What happens when the agent's command is blocked? Both agents run behind a command guard, and both guards blocked reads the agents wanted. All three tools had the same admin kubeconfig in the lab, each behind its own guard. | Blocked command | Tool | Runs affected | What happened next | |---|---|---|---| | `kubectl rollout history` | Aurora (`observability_only` policy) | 26 of 38 | Read ReplicaSets and the Deployment instead (as in scenario 12); listed rollout history under "Not Checked" | | `kubectl exec -- cat /etc/nginx/conf.d/default.conf` | Aurora | scenario 19, both runs | r1 left the cause "not fully reconciled" and was graded partial; r2 was graded correct | | `kubectl exec -- cat /etc/resolv.conf` | Aurora | scenario 16 r1 | Graded correct | | `kubectl rollout history` | HolmesGPT (bash allow-list) | 12 of 38 | Read ReplicaSets instead (as in scenario 01 r1); 2 of the 12 were among its 7 non-correct runs | | `kubectl rollout status` | HolmesGPT | scenario 14 r1 | Graded correct | | `kubectl exec ... -- cat /etc/resolv.conf` and a DNS lookup | HolmesGPT | scenario 16 r2 | Queried ReplicaSets and the cluster DNS config instead; graded correct | | a `describe ... \| sed -n ...` pipe | HolmesGPT | scenario 13 r2 | Ran other calls; graded correct | Aurora's denial reads "Command blocked by organization policy: Mutating kubectl operations". The default policy's deny rule matches the whole `rollout` verb, so the read-only `rollout history` and `rollout status` go down with `rollout restart` and `rollout undo`. Our lab fix branch allowed the two read-only forms, and in the 6 runs on that branch `rollout history` ran 14 times and was never denied. As of 21 October 2026 the default policy on Aurora's main branch still denies the whole verb, and `exec` with it. HolmesGPT's bash toolset matches commands by prefix against an allow-list; per [its docs](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/bash/), the default list includes `kubectl get`, `describe`, `logs` and `top` but not `rollout` or `exec`, and anything else needs approval. Our runs piped the prompt in ("Interactive mode disabled when reading piped input"), so nobody was there to approve and the command did not run. A blocked read is a hole in the evidence. What separates a usable tool is whether the report says so. Aurora's reports did, in their "Not Checked" lists, though in one earlier run it repeated the policy's wording and called `rollout history` "a mutating operation" in the report. The scenario 19 miss also shows the cost of a missing fallback: Aurora's own `get pods` output showed the failing pod was 9 seconds younger than the healthy one, which was enough to see it had loaded the edited ConfigMap. The [ConfigMap change not applied to all pods runbook](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes) walks through that check. The trade-off is real: loosening a guard so the agent can run `exec` also lets it run anything inside your containers. We would allow `rollout history` and `rollout status` and keep `exec` blocked, and accept that some faults then need a person. ## Can an AI agent investigate a security incident? Not the tools and faults on this page. Everything above is operational: a service is failing and the agent finds the change that broke it. A security incident investigation (a suspicious login, a malware alert, data leaving the network) runs on different data, such as SIEM events, endpoint telemetry and identity logs, and is the job of AI security operations (SOC) tools. We have not tested any of those, and the lab has no security scenarios, so we make no claims about them. Two things do carry over. The loop is the same shape: a request comes in, the agent queries, cites evidence and concludes. And so does the audit question security teams ask about any agent: what exactly did it run? In the lab, Aurora's trace records every command with its full output, including the denied ones (`POLICY_DENIED` with the command text), and Aurora screens each command against hand-written rules plus a vendored subset of SigmaHQ detection rules before it runs. HolmesGPT prints every tool call and every blocked command in its output. If you are evaluating an investigation agent for your security team's review, ask for that record from a real run, not a description of it. ## What should you require from any AI investigation tool? From the 76 Aurora and HolmesGPT runs in the lab, the requirements that would have caught the problems we saw: 1. **Every tool call and its output, in order.** You cannot judge an answer without the trace. Both traced agents provide one. 2. **Claims tied to specific evidence**, and a habit of opening that evidence. Aurora's scenario 12 reports cited real outputs for a false claim. 3. **A "ruled out" and a "not checked" list.** Not checked is where blocked commands and skipped areas belong. 4. **A check beyond the alerted object** before anything is declared healthy: endpoints, traffic, the users' side of the request path. 5. **A read-only identity with a guard you can read**, and a record of what the guard blocked. Decide in advance which blind spots you accept (`exec`, Secrets, rollout history). 6. **Fixes as suggestions with a risk rating and an undo**, run only after a person approves. Aurora's scenario 13 suggestion had the rating and no undo. 7. **Isolation between trial runs.** Aurora keeps memory and an incident index per organization. In our first, superseded batch, all runs shared one organization and Aurora cited "prior incidents" from an earlier fault; after that every run got a fresh organization. 8. **Time and cost per run, including background jobs.** In the lab, Aurora's per-cluster discovery job cost about twice what the 38 graded investigations did ($29.70 for 30 discovery runs against $14.53). 9. **Two runs per fault.** HolmesGPT was wrong and then correct on scenario 19; Aurora was partial and then correct on the same fault. The [AI incident investigation tools list](/blog/best-ai-powered-incident-investigation-tools) has the commands to run these faults against any product in a trial. For one broken pod, the [Kubernetes pod diagnoser](/diagnose) takes pasted `kubectl` output and returns the likely cause in your browser. ## How accurate is AI incident investigation in our lab? | All 19 scenarios, 2 runs each | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | Correct | 34 of 38 | 31 of 38 | not scored as a peer | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 | | "No problems detected" | 0 | 0 | 12 runs (6 scenarios) | | Median time per run | 153.5 s | 45 s | 12.5 s for two analyze passes | | Median LLM cost per run | $0.35 | not captured | not captured | A judge model (Claude Opus 5.5) graded every answer against the injected fault. A person at Arvo read all 38 Aurora grades and changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model re-read all 76 and agreed with every one, and a third pass graded them from scratch, agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels. A blinded regrade with the tool names removed matched the published verdict in 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs. The two-sided Fisher exact test that [/benchmark](/benchmark) computes on these totals gives p = 0.52, comparing person-read Aurora grades with judge-only HolmesGPT grades, so read Aurora and HolmesGPT as roughly even on accuracy. HolmesGPT is about three times faster at the median. Aurora's runs are from 2 October 2026, the others from 15 and 16 October, on the same cluster setup (kind v0.33.0, Kubernetes v1.37.0, 3 nodes) and faults. Arvo wrote the scenarios, a kind cluster with one small app is not production, and two runs per fault do not measure variance. The [lab report](/blog/aurora-lab-report-kubernetes-19-failures) has the method and every miss, and [agentic incident response on a real alert](/blog/what-is-agentic-incident-management) traces a third investigation (scenario 07) call by call. Aurora is open source under Apache 2.0: [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora), or [book a demo](https://cal.com/arvo-ai/demo) to see it on your own alerts. *Sources: run folders `runs/v1-run1/13-networkpolicy-blocks-redis`, `runs/v1-run1/12-two-faults-oom-and-selector`, `runs/v1-run1/19-config-drift-partial-restart` and `runs/h2h-v1/r1|r2`, plus `benchmark/results.json`, `REPORT.md`, `FINDINGS.md`, `runs/h2h-v1/README.md`, `review-summary.md`, `review2-summary.md` and `runs/blind-regrade.md` in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab). Aurora's default command policy read on its main branch and HolmesGPT's bash toolset docs read on 21 October 2026. Verified 21 October 2026.* --- ## Automated Post-Mortem Generation: 2026 Guide URL: https://www.aurorasre.ai/blog/automated-post-mortem-generation Published: 2026-05-13 (updated 2026-07-15) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Automated post-mortem generation turns chat transcripts, observability data, and investigation traces into retrospectives. 2026 tools and architectures. > ## Key Takeaways > > - **Automated post-mortem generation is the process of producing an incident retrospective from artifacts already collected during the incident**, chat transcript, alert timeline, monitor data, and (in agentic systems) the investigation agent's own tool-call trace. The category is not a single technology; it's an output shared by three distinct architectures. > - **We propose the Postmortem Provenance Model (PPM).** Three source types: **(1) chat-transcript postmortems** (Rootly, incident.io, FireHydrant) summarize what humans said in the channel; **(2) observability-stitched postmortems** (Datadog Bits AI) summarize what monitors recorded; **(3) agentic-investigation postmortems** (Aurora) compose from the agent's causal reasoning trace. The three artifacts answer different questions and are not interchangeable. > - **The standards that anchor this work are old, but unchanged by AI.** [Google SRE Book Chapter 15. Postmortem Culture](https://sre.google/sre-book/postmortem-culture/) (Lunney and Lueder, 2017) and [John Allspaw's "Blameless PostMortems and a Just Culture"](https://www.etsy.com/codeascraft/blameless-postmortems) (Etsy, May 2012) define what a postmortem is for. AI changes the authoring cost, not the purpose. > - **The vendor landscape consolidated in 2025–2026.** PagerDuty acquired [Jeli in November 2023 for $29.7M](https://www.pagerduty.com/newsroom/pagerduty-acquires-jeli/); FireHydrant was acquired by Freshworks in December 2025; Squadcast was acquired by SolarWinds. ServiceNow's [Now Assist SRE specialist](https://newsroom.servicenow.com/press-releases/details/2026/ServiceNow-brings-Autonomous-Workforce-to-every-major-business-function/default.aspx) (GA targeted June 2026) brings the largest ITSM vendor into the postmortem-generation lane. > - **Open-source agentic-investigation postmortems are a small lane.** [Aurora](https://github.com/Arvo-AI/aurora) (Apache 2.0) generates postmortems from its own investigation agent's reasoning chain and exports to Confluence Cloud (OAuth) or Server / Data Center (PAT), with customizable per-org templates and version history. A good postmortem outlives the incident. **An automated post-mortem is an incident retrospective whose narrative, timeline, root cause, contributing factors, and action items are drafted by software rather than by hand, typically a large language model, sometimes a tool-using agent, always built on artifacts already collected during the incident.** This guide is for SRE, platform, and incident-management leaders deciding which automated-postmortem architecture matches their team's working style. not which vendor logo to add to their stack. ## Why automation, and why now Most teams write postmortems by hand. Most postmortems are late, short, and read by no one. The reason is unsentimental: writing a good postmortem takes hours of reconstruction work, on top of an incident that has already drained the on-call's day. The lit-survey of practitioner posts converges on a 4–8 hour figure per postmortem of moderate complexity, most of that spent in Slack, dashboards, and ticket trails trying to reassemble the timeline. The market response since 2023 has been a wave of automated-postmortem features: [Rootly AI Copilot](https://rootly.com/retrospectives), [incident.io](https://incident.io/ai-sre) Scribe and AI summaries, [FireHydrant AI-Drafted Retrospectives](https://firehydrant.com/ai/), [Datadog Bits AI postmortem variables](https://www.datadoghq.com/blog/create-postmortems-with-datadog/), and [PagerDuty Scribe Agent](https://support.pagerduty.com/main/docs/scribe-agent). The pitch is similar across them: 90 minutes of human reconstruction collapses to 15 minutes of human review. The honest framing is that these tools do real work, but most of them are summarizing artifacts that already exist. **They are not investigating; they are transcribing.** That's enough for many teams, especially those whose incidents are well-captured in their incident-channel chatter. It is not enough for teams whose incidents require deep investigation across systems, and that gap is what the agentic-investigation category is starting to fill. ## The Postmortem Provenance Model (PPM) The three architectures differ in what they read from, not in what they produce. Same sections, different evidence. | Source type | Reads from | Strength | Limitation | |---|---|---|---| | **Chat-transcript** | Slack / Teams / Zoom channel for the incident window; on-call chatter; status updates | Captures human narrative, decisions, and judgment calls verbatim | Inherits human errors and gaps; weak on infrastructure facts the channel didn't surface | | **Observability-stitched** | Monitor events, alert timeline, dashboards, deployment history | Strong factual timeline, embedded graphs and logs | Misses human context; weak on contributing factors that aren't in telemetry | | **Agentic-investigation** | The investigation agent's tool-call trace, reasoning chain, evidence collected mid-incident | Causal record of what the system did and what the agent found | Requires running an investigation agent in the first place; quality depends on the agent | A team's choice should match its incident profile. If most incidents resolve in chat with little investigation needed, a chat-transcript tool is fine. If incidents are surfaced and resolved entirely in your observability stack, an observability-stitched approach gives you tight monitor-to-postmortem fidelity. If your incidents require traversing AWS, GCP, Kubernetes, and your own services to find the cause, an agentic-investigation postmortem is the only artifact that records the work the agent actually did. ## Standards: what a postmortem is for It is worth grounding the conversation in what postmortems were designed to do before LLMs existed. - **[Google SRE Book, Chapter 15. Postmortem Culture: Learning from Failure](https://sre.google/sre-book/postmortem-culture/)** by John Lunney and Sue Lueder (O'Reilly, 2017). The canonical text on blameless postmortems as organizational learning. The companion [SRE Workbook Chapter 10](https://sre.google/workbook/postmortem-culture/) updates the practical guidance. - **[John Allspaw. Blameless PostMortems and a Just Culture](https://www.etsy.com/codeascraft/blameless-postmortems)** (Etsy Code as Craft, May 22, 2012). The earlier articulation of why blameless-ness is operationally load-bearing. - **[Lunney. Postmortem Action Items](https://www.usenix.org/system/files/login/articles/login_spring17_09_lunney.pdf)** (USENIX ;login: Spring 2017). The honest practitioner read on why most postmortems' action items never get done. - **[PagerDuty's open-source Postmortem documentation](https://postmortems.pagerduty.com/)** (Apache 2.0, [GitHub](https://github.com/PagerDuty/postmortem-docs)). Includes a maintained [postmortem template](https://github.com/PagerDuty/postmortem-docs/blob/master/docs/resources/post_mortem_template.md) used as a baseline by many teams. - **[Verica Open Incident Database (VOID)](https://www.thevoid.community/)**. The 2nd Annual VOID Report (December 2022) catalogs approximately 10,000 incidents from 600+ organizations; its central finding is that MTTR is statistically unreliable as a cross-organization comparison and that only ~25% of public incident reports clearly identify a root cause. A useful corrective to the "we reduced MTTR by X%" claims that pepper vendor marketing. - **[Dan Luu's curated postmortems collection](https://github.com/danluu/post-mortems)**. The widest public corpus of real postmortems; useful as RAG fuel for any AI postmortem system. A blameless, learning-oriented postmortem is the goal. Automation changes the authoring cost; it does not relax the standard. ## What gets auto-generated today A typical 2026 automated postmortem produces some subset of: 1. **Summary**, one paragraph, the executive read. 2. **Timeline**, chronological events with timestamps (often HH:MM UTC). 3. **Impact**, customer-facing effect, services affected, error budget burn. 4. **Root cause**, the technical fault. 5. **Contributing factors**, human, process, and organizational conditions that allowed the incident. 6. **Resolution**, what stopped the bleeding. 7. **Action items**, owners, due dates, follow-ups. 8. **Lessons learned**, what the team would do differently. Different products auto-draft different subsets. The "Lessons Learned" section, in particular, is left to humans in most products, for the obvious reason that it is the section where judgment is most consequential. ## The tooling landscape Concrete vendor positioning as of May 2026. | Product | License / hosting | What it auto-generates | Notes | |---|---|---|---| | **[Rootly AI Copilot](https://rootly.com/retrospectives)** | Closed, SaaS | Narrative summary, timeline, action items, root cause, embedded Datadog charts; meeting-bot transcription | Headline claim: 90 min → 15 min review. Exports to Confluence, Google Docs, Notion, Slack. | | **[incident.io AI postmortems](https://incident.io/ai-sre)** | Closed, SaaS | Summary, timeline, contributing factors, suggested follow-ups; Scribe transcribes call audio | "Lessons Learned" is left to humans by design. Exports to Confluence, Notion, Google Docs. | | **[FireHydrant AI-Drafted Retrospectives](https://firehydrant.com/ai/)** | Closed, SaaS | Description, customer impact, lessons learned; Copilot compares ongoing incident to past incidents | Acquired by Freshworks December 2025; AI features are Enterprise tier only. | | **[Datadog Bits AI postmortems](https://www.datadoghq.com/blog/create-postmortems-with-datadog/)** | Closed, SaaS | Summary, customer impact, lessons learned variables; dynamic embedded graphs and logs | Exports to Datadog Notebooks, Confluence, or Google Drive. | | **[PagerDuty Scribe Agent](https://support.pagerduty.com/main/docs/scribe-agent)** | Closed, SaaS | Real-time call transcription and timeline contributions to PagerDuty's Postmortems product | Part of PagerDuty's Spring 2026 agent suite (SRE Agent, Scribe Agent, Insights Agent). | | **[Aurora](https://github.com/Arvo-AI/aurora)** | Apache 2.0, self-hosted | Summary, timeline (HH:MM UTC), root cause, impact, contributing factors, resolution, action items, lessons learned; generated from the investigation agent's reasoning trace | Per-org template overrides; Confluence Cloud (OAuth) and Server / Data Center (PAT) export. | | **[ServiceNow Now Assist SRE specialist](https://newsroom.servicenow.com/press-releases/details/2026/ServiceNow-brings-Autonomous-Workforce-to-every-major-business-function/default.aspx)** | Closed, SaaS | Triage + postmortem documentation end to end | GA targeted June 2026 (Knowledge 2026 announcement). | | **[Squadcast](https://www.squadcast.com/product/postmortems)** | Closed, SaaS | One-click postmortem, webhook automation, templates | Acquired by SolarWinds. | The pattern: the SaaS-IM vendors all do chat-transcript postmortems well; Datadog owns the observability-stitched lane; Aurora is the open-source agentic-investigation option. ServiceNow's June 2026 GA brings the largest ITSM vendor into the category as a fourth meaningful entrant. ## Architecture: how agentic-investigation postmortems work Worth describing in detail because this is the category least visible to most buyers. In a chat-transcript postmortem system, the flow is: incident channel → LLM with a postmortem template prompt → draft document. In an observability-stitched postmortem system, the flow is: incident timeline + dashboards → LLM with embedding variables → draft document with live charts. An agentic-investigation postmortem starts earlier, at the *investigation*. The pattern, using Aurora as the concrete open-source example: 1. **Alert webhook arrives.** PagerDuty, Datadog, Grafana, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, NewRelic, OpsGenie, or incident.io fires. The provider-specific RCA-prompt builder constructs the agent's first message, including alert metadata, severity, service, and environment. 2. **Investigation runs.** Aurora's ReAct-style LangGraph agent calls tools across the next 3–15 minutes, `kubectl`, cloud CLIs, knowledge-base search, Terraform read, Confluence search, and accumulates a transcript of tool calls, tool results, and reasoning steps. The result is persisted as the incident's `aurora_summary`, the agent's RCA narrative. 3. **Postmortem dispatch.** When the incident is resolved (either manually, via Aurora's "Run Action" dropdown on completed incidents, or via an [Aurora Actions](/blog/aurora-actions-background-automations) on-incident-completion trigger), a postmortem agent run is dispatched with the agent's RCA summary as load-bearing context. The postmortem agent re-reads the original investigation output, optionally pulls Slack channel context for the incident window, and composes the postmortem under a per-org template. 4. **Storage and versioning.** Drafts are stored in PostgreSQL with version history. Engineers can edit; subsequent regenerations preserve human edits as a separate version. 5. **Confluence export.** The user clicks Export. Aurora pushes the rendered postmortem to Confluence Cloud (OAuth) or Server / Data Center (PAT), creating a page under a configured space and parent. Export is currently user-triggered rather than automatic, which preserves the human review step before publication. The structural difference from chat-transcript postmortems is what evidence the LLM gets. A chat-transcript system can only describe what humans typed. An agentic-investigation system describes what the agent did, which tools it ran, what the cloud responded with, and how it reasoned through to the root cause. The artifact carries the actual causal trail, not a social reconstruction of it. ## How to evaluate an automated postmortem tool A rubric you can run on any vendor, open source or commercial. 1. **Provenance match.** Does the tool's source-of-truth match how your team actually runs incidents? Chat-heavy team → chat-transcript. Observability-heavy team → Datadog or equivalent. Investigation-heavy team → agentic. 2. **Template control.** Can you replace the vendor's template with your team's? Per-team templates? Aurora supports per-org template overrides via its `actions` configuration table; vendor SaaS varies. 3. **Export target.** Confluence Cloud, Server / Data Center, Notion, Google Docs, internal wiki. Match your team's documentation home. Aurora supports Confluence (both flavors); the SaaS vendors support different combinations. 4. **Edit lineage.** When the AI draft is edited, regenerated, and edited again, what survives? Test this explicitly with three round trips. Aurora preserves version history; check each candidate. 5. **Action-item ownership.** Does the tool extract action items with owners and due dates, or just bullet points? The Lunney USENIX piece is blunt about why this matters: action items without owners do not get done. 6. **Embedded evidence.** Are graphs, logs, and resource identifiers embedded inline or linked? Embedded survives the documentation system; linked rots over time. 7. **Cost and privacy.** Where does the postmortem text get processed? Self-hosted with bring-your-own-LLM (Aurora) keeps incident data on your infrastructure; SaaS vendors vary in how they handle this and your security team will want to know. 8. **Standards alignment.** Does the generated artifact match the blameless tradition (Allspaw, Lunney, the SRE Book) or accidentally drift into individual blame? Check the prompt if you can; otherwise inspect a sample. ## How to roll out automation without breaking culture A six-step adoption plan that respects the standards while saving the time. 1. **Start with the easiest 30%**, short-impact incidents with mostly-chat investigations. These produce passable AI drafts on day one. 2. **Keep humans on lessons learned.** Even tools that auto-generate the "Lessons Learned" section ship it as a draft to be aggressively rewritten. The judgment in that section is the point of the postmortem. 3. **Require human edit before publish.** The on-call engineer who ran the incident should always be the one who clicks "Publish." This is the cultural firewall. 4. **Track action-item completion separately.** AI-generated action items have a known completion-gap problem. Add a weekly review of last week's postmortem action items, with owners called out by name. 5. **Run a quarterly audit of the generated postmortems.** Pick five at random; have a senior engineer read them critically. Look for drift toward individual blame, missed contributing factors, and surface-level root causes. 6. **Tighten the loop with the investigation tool.** If your investigation tool and postmortem tool are the same product (Aurora, eventually Resolve.ai-class systems), the postmortem inherits the investigation's evidence chain. This is the highest-quality automated postmortem possible, but it requires running an agentic investigation in the first place. ## What can go wrong A short failure-mode list. - **Surface-level root cause.** AI drafts read confidently while attributing a deep system issue to its most visible symptom. The cure is human review by someone who was in the incident. - **Hallucinated timeline.** LLM invents events, misattributes timestamps, or doubles up on entries. Most common when the input artifact (chat transcript or telemetry) has gaps the model patches over. - **Blame drift.** AI summary slips into individual-blame framing because the human chat did. The blameless tradition exists exactly for this reason; the AI does not enforce it on its own. - **Action items without ownership.** A bullet list of "should do X" with no owner is not an action item; it is decoration. Treat ownerless action items as a failure of the tool's prompt. - **Edit loss on regeneration.** Some tools overwrite human edits when the user clicks "Regenerate." Verify that version history is preserved before trusting the tool for a quarter's worth of postmortems. ## Where Aurora fits [Aurora](https://github.com/Arvo-AI/aurora) is the open-source agentic-investigation entry in this category. Apache 2.0, self-hosted via Docker Compose or Helm. Postmortems are generated from the same agent that ran the investigation, with per-org template control, version history, Slack context backfill, and export to Confluence Cloud or Server / Data Center. If your incidents look like chat-resolved coordination work, you probably don't need Aurora's postmortem layer specifically. If your incidents look like deep cross-cloud investigation work, you probably do. For more on how Aurora's investigation half works, see our [AI-Powered Incident Investigation guide](/blog/ai-powered-incident-investigation). For how Aurora's automation primitive (Aurora Actions) lets you chain postmortem generation onto every incident automatically, see the [Aurora Actions launch post](/blog/aurora-actions-background-automations). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Docs:** [arvo-ai.github.io/aurora](https://arvo-ai.github.io/aurora/) - **Related guides:** [9 Best AI Post-Mortem Tools in 2026](/blog/best-ai-post-mortem-tools) · [AI-Powered Incident Investigation](/blog/ai-powered-incident-investigation) · [Aurora Actions](/blog/aurora-actions-background-automations) · [Root Cause Analysis: Complete Guide for SREs](/blog/root-cause-analysis-complete-guide-sres) · [Pre-Incident Detection](/blog/pre-incident-detection) · [Automated Incident Remediation](/blog/automated-incident-remediation) --- ## Aurora Actions: Custom Background SRE Automations URL: https://www.aurorasre.ai/blog/aurora-actions-background-automations Published: 2026-05-11 (updated 2026-07-02) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Aurora Actions let SRE teams define reusable background automations in natural language, triggered manually, on incident completion, or on a schedule. > ## Key Takeaways > > - **Aurora Actions are reusable, natural-language automations** that Aurora's agent executes in the background using all 22+ connected integrations. Available today on the main branch of [Aurora](https://github.com/Arvo-AI/aurora). > - **Three trigger types**: manual ("run now"), on incident (fire the moment an incident is created, after RCA completes, or once it is resolved), and recurring schedule (interval-based via Celery Beat, down to every five minutes). > - **Two execution modes per Action**: Ask (read-only) lets Aurora investigate and read code without changing anything, while Agent (read-write) lets it run commands, modify Terraform, and open pull requests. > - **Same agent, same tools, different prompt scaffolding.** Actions reuse Aurora's existing LangGraph agent and 30+ tools (kubectl, aws, gcloud, az, Terraform, Confluence, Slack, GitHub), they just run as background chat sessions with eager-loaded skills and no RCA mandate. > - **`/action ` is a first-class chat primitive.** Slash-command autocomplete in the chat input, "Run Action" dropdown on completed incidents, and full RBAC-gated CRUD UI in Settings. > - **Aurora Actions turn the agent into a programmable platform.** This is the building block for CI/CD auto-remediation, scheduled audits, and post-incident health checks, covered in [our CI/CD Auto-Remediation guide](/blog/cicd-auto-remediation-complete-guide). We shipped one of the most-requested features in [Aurora](https://github.com/Arvo-AI/aurora)'s history: **Aurora Actions, user-defined background automations that run on Aurora's agent.** **An Aurora Action is a named, natural-language instruction the user writes once and then triggers manually, on incident completion, or on a recurring schedule; Aurora's agent executes it as a background task with full access to every connected integration.** Where traditional incident management tools force you to pick from a fixed catalog of "automations" (close incident, post to Slack, run runbook), Actions are written in plain English and inherit the full reasoning capability of the agent. This post is for SRE and platform teams already running Aurora, or evaluating it, who want to understand what Actions actually do, where they fit on the agentic spectrum, and how to use them safely. For the short version, see the [Aurora Actions launch announcement](/blog/introducing-aurora-actions); this guide is the deep dive. ## What is an Aurora Action? An **Aurora Action** has five parts: 1. **A name**, used as the slash-command handle (`/action `) and as the dropdown label on incident cards. 2. **A natural-language instruction**, the prompt the agent will execute. The same instruction the user would type into chat, except it can reference incident context placeholders when triggered post-incident. 3. **A trigger type**, manual, on-incident (fire on creation, after RCA completes, or once resolved), or on-schedule (interval-based via [Celery Beat](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html), down to every five minutes). 4. **An execution mode**. Ask (read-only), so the agent can investigate and read code but change nothing and write tools are filtered out, or Agent (read-write), so it can run commands, modify Terraform, and open pull requests. 5. **An on/off toggle**, actions can be disabled without deletion, with full RBAC for who can create, edit, or trigger them. The implementation is a thin layer over Aurora's existing chat agent. When an Action triggers, the executor service creates a background chat session with the action's instruction as the user message, runs it through the same LangGraph workflow that powers interactive chat, and persists the run history. The agent has full tool access (kubectl, cloud CLIs, Terraform, Slack, GitHub, Confluence, Memgraph, Weaviate) and eager-loaded skills, the only differences from interactive chat are scaffolded prompts and the absence of any RCA mandate. ## Why this matters Most incident management automation today is **workflow automation**: PagerDuty fires, Slack channel is created, status page is updated, runbook link is posted. The "automation" is a directed graph of static actions. There is no reasoning, no investigation, no judgment. Tools like Rootly, FireHydrant, and incident.io are excellent at this, but they don't *do* anything an SRE wouldn't have to manually verify after the fact. Aurora's bet has always been the opposite: **automate the investigation itself.** Aurora Actions extend that bet from one-shot incident investigations to recurring or post-incident workflows. A few concrete examples: - **Noisy alert tuning**. "Every Friday at 5pm, review which Datadog alerts fired more than 20 times this week with mean time-to-acknowledge over 10 minutes. Open a Terraform PR to widen the thresholds or move them to a warning channel." - **Post-incident health check**. "After every completed RCA, run a 15-minute observation on the affected service: check error rate, p99 latency, and pod restart count. Post results to #incident-followup." - **Scheduled infrastructure audit**. "Every Monday at 9am, audit IAM roles in the production AWS account that have not been used in 90 days. List candidates for removal in a Confluence page." None of these are runbook automation. Each requires the agent to query infrastructure, reason about results, and produce a structured output. Each one was previously the job of an on-call engineer doing follow-up between pages. ## Where Actions sit on the agentic capability spectrum In our [Open-Source AI SRE comparison](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt), we proposed a four-level spectrum for AI SRE capability. Actions don't change the level, they change *when the agent runs.* | When the agent runs | Trigger | Pre-Actions example | With Actions | |---|---|---|---| | **On alert** | Webhook from PagerDuty / Datadog / Grafana | Aurora investigates the alert and produces an RCA | Same, investigation flow is unchanged | | **On user request** | Engineer asks a question in chat | Aurora answers using tools | Same, plus `/action ` shortcuts | | **After every incident** | Incident state transitions to "resolved" | Postmortem generated; engineer manually does follow-up checks | Action runs automatically with incident context in scope | | **On a schedule** | Celery Beat cron | No equivalent, required external scheduler + custom code | Single source of truth: agent runs the prompt on cadence | The post-incident and scheduled triggers are the genuinely new capability. Before Actions, anything recurring or post-incident required gluing Aurora to an external scheduler, an external prompt store, and bespoke trigger code. Actions collapse all three into the product surface. ## How Actions work under the hood This is for the technically curious. A few architecturally interesting things from the implementation: **1. Background chat sessions, not a separate runtime.** When an Action triggers, the executor service creates a regular chat session with the action's instruction as the seed message and dispatches it as a background Celery task. The agent doesn't know it's running an Action, it just runs the workflow. This means every capability the interactive agent has (tool calls, RAG, graph traversal, sub-agent orchestration) is available inside Actions for free. **2. Eager-loaded skills, no RCA mandate.** Interactive chat lazy-loads skills based on the user message. Background actions eager-load all skills because there is no human to clarify ambiguity. The system prompt also strips the "your job is to find root cause" framing. Actions can do anything the agent can do, not just investigate. **3. RLS context is preserved.** Aurora uses [PostgreSQL row-level security](https://www.postgresql.org/docs/current/ddl-rowsecurity.html) for multi-tenancy. The executor explicitly sets RLS context (`org_id`, `user_id`) before running so background tasks see only their own org's data, even though they run under a service identity. **4. Stale run cleanup is integrated.** Aurora's existing background-chat janitor already handles orphaned chat sessions from crashed pods. Action runs go through the same path, so a worker pod dying mid-action doesn't leave the run state inconsistent. **5. RBAC is enforced at the route layer.** Action CRUD is gated by Aurora's Casbin-based RBAC. Org admins can restrict which roles can create or trigger actions, important because an Action with cloud-CLI access has real blast radius. ## Trigger types in detail ### Manual triggers The simplest case. An admin creates the action, an engineer triggers it from the Actions page or via `/action ` in chat. Useful for codifying common operational tasks ("rotate ECS task definitions for service X", "scan Confluence for stale runbooks") into named, repeatable commands. The chat integration is worth calling out: `/action` is implemented as an LLM tool call using the same pattern as Aurora's `/rca` slash command. The agent processes the action dispatch and then continues responding to the rest of the user's message, so you can write "kick off the IAM audit and tell me what changed since last week" and the agent will dispatch the audit action *and* answer your question in the same turn. ### On-incident triggers Incident-triggered actions run against the incident context automatically, with the incident's metadata, RCA, and timeline available to the action's agent without the user having to paste anything in. You choose when they fire: the moment the incident is created, after RCA completes, or once it is resolved. Firing on creation is for fast enrichment, after RCA for follow-up that needs the root cause in scope, and on resolve for postmortems and health checks. This is the trigger that turns Aurora from a reactive tool ("investigate this page") into a continuous one ("investigate, then run health checks, then file the postmortem"). ### Scheduled triggers Interval-based, driven by [Celery Beat](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html). Choose a cadence (every N minutes / hours / days), and the action runs without user involvement. This is the building block for the CI/CD auto-remediation and scheduled audit use cases, and it's why we're calling this post and the [CI/CD Auto-Remediation guide](/blog/cicd-auto-remediation-complete-guide) sister posts. ## What Actions don't do (and why) A few capability decisions worth being explicit about: - **No external webhook triggers** in this release. We could have added "trigger on arbitrary webhook" but it overlaps with the existing alert-triggered investigation flow. We may add it if we see demand for triggers from systems that don't go through PagerDuty / Datadog / Grafana. - **No agent-authored Actions** yet. The agent can't create or modify Actions on its own. Self-modification is a serious security boundary; we'd want approval gating and audit logging before opening that door. (See our [AI Agent kubectl Safety guide](/blog/ai-agent-kubectl-safety) for the threat model.) - **No conditional / DAG composition** in this release. Actions are single-prompt for now. If you need a multi-step workflow, write a single prompt that describes the steps, the agent is good at sequencing. We'll add explicit composition if the natural-language form proves limiting. ## Safety: what to think about before enabling Every Action is a small program with access to your cloud environment. A few rules we use ourselves: 1. **Start read-only.** Set new Actions to Ask (read-only) mode so the agent can investigate and read code but cannot change anything; write tools, including `kubectl apply` and `aws ec2 terminate-instances`, are filtered out. Promote an Action to Agent (read-write) only once its trace is clean for several runs. Actions also inherit Aurora's underlying tool permissions on top of the mode. 2. **Use scheduled triggers conservatively.** A daily audit is cheap. A 5-minute polling loop with cloud CLI calls is not. Watch the LLM bill. 3. **Audit who can create Actions.** RBAC defaults to org-admin-only creation. Leave it there unless you have a clear reason to widen. 4. **Pin the model.** Action prompts can be sensitive to model behavior. Pin a known-good model per action (gpt-5.5, claude-sonnet-4.6, opus-4.7, etc.) using Aurora's per-org model dropdown until you have confidence in cross-model stability. 5. **Review action runs weekly.** Every action has a run-history view. Spend 10 minutes a week reading the agent's traces for your scheduled actions, anomalous reasoning is the leading indicator of prompt drift or tool drift. ## How to ship your first Action A six-step recipe. ### 1. Pick a recurring task you currently do manually Anything you do every week or after every incident. Examples: stale-PR review, alert-noise audit, on-call handover summary. The smaller and more deterministic, the better for v1. ### 2. Write the prompt as if you were typing it into chat Don't translate to "automation language." Write it the way you would write a chat message to a smart junior SRE. "Look at..." "Check whether..." "Open a PR that..." ### 3. Create the Action with a manual trigger Settings → Actions → New Action. Paste the prompt, set trigger = manual, leave it disabled if you want to review before enabling. Trigger it once and watch the run. ### 4. Inspect the run trace Click the run in the history view. Read every tool call. Look for: tool misuse (wrong cloud account), excessive tool calls (3 attempts at the same thing), hallucinated paths or resource IDs. Iterate on the prompt until the trace is clean for three consecutive runs. ### 5. Promote to the right trigger type If the action makes sense after every incident → on-incident-completion. If it's a routine sweep → on-schedule with the longest cadence that still meets your need. Only use short cadences when you have a clear cost and blast-radius understanding. ### 6. Add it to your team's incident review Treat agent runs the same way you treat human runs: include them in your weekly incident review. Look for actions that produced wrong output, actions that nobody read the output of, and actions that produced output nobody acted on. Delete or downgrade as needed. ## Aurora Actions vs traditional incident-management automation The category most people compare us to is "workflow automation in incident-management SaaS". Rootly, FireHydrant, incident.io. The comparison is informative but ultimately category-different: | Capability | Aurora Actions | Rootly / FireHydrant / incident.io workflows | |---|---|---| | **Authoring** | Natural language | DSL or visual builder | | **Reasoning** | Yes. LLM agent | No, fixed conditional graph | | **Tool reach** | Cloud CLIs, kubectl, Terraform, Slack, Confluence, GitHub, RAG, infra graph | Slack, status pages, Zoom, runbook links, ticket creation | | **Scheduled execution** | Yes (Celery Beat) | Limited (some support timed reminders) | | **Post-incident chaining** | Yes, full incident context available | Yes, but limited to workflow actions | | **Open source** | Yes (Apache 2.0, self-hosted) | No | | **Pricing** | Free (self-hosted; LLM tokens only) | Per-user SaaS | The honest framing: traditional incident-management tools automate the *process around* the incident. Aurora Actions automate *what happens inside the agent*. Both have value; they cover non-overlapping work. If you live in PagerDuty and use Rootly for incident channels, Aurora Actions sit alongside that, they don't replace it. ## What's next Aurora Actions is the foundation for several capabilities on our roadmap: - **DAG composition**, explicit multi-step Action chains where each step is itself an Action. - **Approval gates**. Actions that pause for human approval before destructive tool calls (already supported in chat; explicit Action-level gating coming). - **CI/CD auto-remediation hooks**, first-class integration with GitHub Actions, Jenkins, and ArgoCD so a failing pipeline becomes a triggered Aurora investigation. (Background and detailed write-up in our [CI/CD Auto-Remediation guide](/blog/cicd-auto-remediation-complete-guide).) - **Action marketplace**, community-contributed Actions you can install with one click. Bring-your-own prompt store. We'll publish each of these as they ship. ## Get Aurora Aurora is fully open source under Apache 2.0. Self-host with [Docker Compose or Helm](https://github.com/Arvo-AI/aurora). Actions ship in the next tagged release after [aurora-oss-1.2.15](https://github.com/Arvo-AI/aurora/releases) (April 15, 2026); the feature is available on `main` today. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Docs:** [arvo-ai.github.io/aurora](https://arvo-ai.github.io/aurora/) - **Compare against alternatives:** [Open-Source AI SRE: Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt) · [Aurora vs traditional incident-management tools](/blog/aurora-vs-traditional-incident-management-tools) --- ## CI/CD Auto-Remediation: How It Works and What Breaks URL: https://www.aurorasre.ai/blog/cicd-auto-remediation-complete-guide Published: 2026-05-11 (updated 2026-10-12) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Auto-remediation in CI/CD means pipelines that detect, diagnose, and recover from failure without paging a human. Architecture, tools, and pitfalls. > ## Key Takeaways > > - **Most teams do not yet auto-remediate inside CI/CD.** Per [JetBrains' AI Pulse coverage (April 2026)](https://blog.jetbrains.com/teamcity/2026/04/ai-in-devops/), **78.2% of respondents don't use AI in CI/CD workflows at all**, even though AI is now widely used elsewhere in the development lifecycle. > - **CI/CD auto-remediation is an architectural pattern, not a product category.** It combines progressive delivery (canary, blue-green), automated metric-driven rollback, and AI-assisted root-cause-and-fix. Owned components, not a single SKU. > - **Three layers, four maturity levels.** We propose the **CI/CD Auto-Remediation Maturity Spectrum (CARM)**: L0 (manual), L1 (rollback), L2 (rollback + diagnostic), L3 (rollback + diagnostic + remediation), L4 (closed-loop with policy gates). > - **Open-source stack is mature.** [Argo Rollouts](https://argoproj.github.io/argo-rollouts/), [Flagger](https://fluxcd.io/flagger/), and metric-driven `AnalysisTemplates` cover L1–L2 with no AI. AI agents like [Aurora](https://github.com/Arvo-AI/aurora) extend to L3 with Actions-based remediation. > - **DORA's bar is real.** Top-performing teams keep change failure rate low and recover from failed deployments in under one hour ([DORA program guidance](https://dora.dev/guides/dora-metrics/)). Auto-remediation is how non-elite teams close the gap. Of the [46+ AI SRE products](https://rootly.com/ai-sre-guide) and dozens of progressive-delivery tools shipping in 2026, only a handful explicitly target the pattern this guide is about. **CI/CD auto-remediation is the practice of having your delivery pipeline automatically detect, diagnose, and recover from failure, without paging a human, using a combination of progressive-delivery primitives, metric-driven rollback policies, and (increasingly) AI agents that propose or apply fixes.** It is not the same as auto-deploy. It is not the same as canary rollout. It is the closing of the loop between "the pipeline noticed something is wrong" and "the system is back in a good state", without an engineer in the middle. This guide is for SRE and platform teams who already run continuous delivery and want to push toward the auto-remediation end of the spectrum. By the end, you should be able to: position your current setup on the CARM maturity spectrum, identify the next concrete step, and pick a credible tool stack to get there. ## Why auto-remediation matters in 2026 Three numbers explain the demand. **1. AI is shipping more code, faster.** Per [JetBrains' AI Pulse coverage on the TeamCity blog (April 2026)](https://blog.jetbrains.com/teamcity/2026/04/ai-in-devops/), AI tools are now used by a large majority of developers in their daily work. The [DX 2026 change-failure-rate analysis](https://getdx.com/blog/change-failure-rate/) puts a number on it: with 91% of developers having adopted AI and 20%+ of merged code now AI-authored, **code velocity has gone up while quality has gone in the opposite direction.** More deployments per day means more chances to break production. **2. The pipeline itself is the new bottleneck.** [JetBrains' 2025 State of CI/CD survey](https://blog.jetbrains.com/teamcity/2025/10/the-state-of-cicd/) documents widespread frustration with slow and unreliable CI/CD pipelines as a leading contributor to developer burnout. **3. AI in CI/CD specifically lags adoption.** Per [JetBrains' AI Pulse coverage (April 2026)](https://blog.jetbrains.com/teamcity/2026/04/ai-in-devops/), **78.2% of respondents don't use AI in CI/CD workflows at all**, even though most use AI everywhere else in the development lifecycle. The gap isn't capability; it's trust and integration. AI in IDEs is low-risk; AI in pipelines touches production. Teams want the impact but won't take the blast radius until the architecture is right. Auto-remediation is the architecture that closes that gap. It bounds the agent's reach (only inside the delivery pipeline), gives it deterministic guardrails (progressive delivery and metric-driven rollback), and produces a clear contract: detect, diagnose, fix-or-rollback, log. ## What "auto-remediation" actually means It is easiest to define by negation. Auto-remediation is **not**: - **Auto-deploy.** Auto-deploy ships code on merge. Auto-remediation is what happens *after* a problem appears. - **Canary release.** Canary is the *detection mechanism*, it surfaces problems early by shifting traffic gradually. Remediation is the *response*, rolling back, hotfixing, or reverting. - **Self-healing infrastructure.** Self-healing systems like Kubernetes restart pods. Auto-remediation includes that plus *change-driven* failure recovery: rolling back a bad deploy, rolling forward a fix, or pausing the pipeline while a human investigates. - **AIOps.** AIOps platforms surface alerts and correlations. Auto-remediation closes the loop by *acting* on them. The minimum viable definition: **a pipeline transition from a degraded state back to a healthy state, triggered by automated detection, executed by automated action, observed and logged for human review.** ## The CI/CD Auto-Remediation Maturity Spectrum (CARM) There is no single industry-standard maturity model for auto-remediation. We use the following five-level spectrum, derived from how teams actually evolve. | Level | What happens on failed deploy | Tools that get you here | Trust required | |---|---|---|---| | **L0. Manual** | Pipeline fails. PagerDuty pages the on-call. Engineer investigates, decides to roll back or hotfix, executes manually. | None, this is the default for most teams. | None, humans do everything. | | **L1. Automated Rollback** | Pipeline detects health-check failure (error rate, latency, smoke test). Automatically rolls back to the previous version. Pages a human after the fact. | [Argo Rollouts](https://argoproj.github.io/argo-rollouts/), [Flagger](https://fluxcd.io/flagger/), [Spinnaker](https://spinnaker.io/) | Trust that the health metric reflects user-visible failure. | | **L2. Rollback + Diagnostic** | L1 plus: AI agent runs an investigation when rollback fires. Produces an RCA before the human starts. Page goes out with context, not blank. | L1 stack + [HolmesGPT](https://github.com/HolmesGPT/holmesgpt), [Aurora](https://github.com/Arvo-AI/aurora), [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) | Trust that the diagnostic is right enough to bias human reasoning. | | **L3. Rollback + Diagnostic + Remediation** | L2 plus: agent proposes (or in some cases applies) a fix, a PR, a config change, an alert threshold update. Human reviews and merges. | L2 stack + Aurora Actions, HolmesGPT Operator mode | Trust that the agent's fix is correct, scoped, and reviewable. | | **L4. Closed-loop with policy gates** | L3 plus: certain *low-risk, well-understood* fixes are applied automatically inside policy guardrails (alert threshold widening, log-only changes, retry loops). Destructive or high-risk changes still gated. | L3 stack + policy engine (OPA, Casbin, Kyverno) + audit logging | Trust the policy gate definitions more than the agent. | Most teams in 2026 are at **L0 or L1**. The leap from L1 to L2 is the single highest-leverage move available because it preserves human-in-the-loop decision-making while removing the "blank-page" delay that explains a large share of MTTR. The 2024-2025 DORA research [renamed MTTR to Failed Deployment Recovery Time (FDRT)](https://dora.dev/guides/dora-metrics/) precisely because the metric is more meaningful when scoped to change-driven failures, which is exactly the failure mode auto-remediation targets. ## L1: Automated rollback (where most serious teams should be) This is the foundation. Without L1, AI-assisted remediation at L2-L3 has nowhere to act. The two Apache 2.0 incumbents are **Argo Rollouts** and **Flagger.** Both run in Kubernetes; both implement metric-driven progressive delivery with automated rollback. They differ in invasiveness. | Capability | [Argo Rollouts](https://argoproj.github.io/argo-rollouts/) | [Flagger](https://fluxcd.io/flagger/) | |---|---|---| | **CNCF status** | Part of [Argo](https://www.cncf.io/projects/argo/) (Graduated, Dec 2022) | Part of [Flux](https://www.cncf.io/projects/flux/) (Graduated, Nov 2022) | | **Resource model** | Replaces `Deployment` with `Rollout` CRD | Wraps existing `Deployment` | | **GitOps pairing** | ArgoCD | FluxCD | | **Analysis** | `AnalysisTemplate` querying Prometheus, Datadog, CloudWatch, etc. | Service-mesh metrics + custom webhooks | | **Automated rollback** | Metric-threshold breach → revert | Metric-threshold breach → revert | | **Traffic shaping** | Native + ingress + service mesh | Service-mesh first (Istio, Linkerd, App Mesh) | | **Invasiveness** | Higher (changes resource type) | Lower (transparent wrapper) | | **Webhooks for custom logic** | `Experiment` resource + analysis runs | Pre-/post-/during-rollout hooks | **Pick Argo Rollouts** if you already use ArgoCD and want explicit per-step canary control. **Pick Flagger** if you use a service mesh and want progressive delivery to be transparent to existing manifests. For non-Kubernetes pipelines, equivalent capability lives in **Spinnaker** (multi-cloud, mature), **Harness** (commercial), and feature-flag platforms like **LaunchDarkly** (when "rollback" can be a flag flip). A minimal Argo Rollouts AnalysisTemplate for HTTP error rate, simplified from the [official docs](https://argoproj.github.io/argo-rollouts/features/analysis/): ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: error-rate spec: args: - name: service-name metrics: - name: error-rate interval: 30s successCondition: result[0] <= 0.01 failureLimit: 3 provider: prometheus: address: http://prometheus.monitoring:9090 query: | sum(rate(http_requests_total{service="{{args.service-name}}",status=~"5.."}[1m])) / sum(rate(http_requests_total{service="{{args.service-name}}"}[1m])) ``` Three failed 30-second windows → rollback. This is L1 in 30 lines of YAML. ## L2: Rollback + automated diagnostic L1 gets you out of an outage fast. It does not tell you *why* the deploy failed. The human gets paged with a rollback notification and starts from zero. L2 fills that gap with an AI agent that runs when rollback fires. The agent queries the cluster state, the application logs, the rollout metrics, and the changed code, and produces an RCA before the human starts typing. Three credible open-source options exist as of 2026 (compared in detail in our [Open-Source AI SRE: Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt) guide): - **[K8sGPT](https://github.com/k8sgpt-ai/k8sgpt)**, rule-based scanner with LLM explanations. Best for low-blast-radius first deployment; explains *why* a resource is unhealthy. - **[HolmesGPT](https://github.com/HolmesGPT/holmesgpt)**. ReAct-loop AI agent (CNCF Sandbox). 30+ observability integrations. Read-only by default. Strong fit for cluster-scoped investigation. - **[Aurora](https://github.com/Arvo-AI/aurora)**. LangGraph supervisor agent. Multi-cloud (AWS / Azure / GCP / OVH / Scaleway). Generates postmortems. Opens remediation PRs with human approval. Wiring up L2 is straightforward: configure your AI SRE's webhook to receive the rollback event (Argo Rollouts emits Kubernetes events; you can route them via [Argo Notifications](https://argoproj.github.io/argo-rollouts/features/notifications/) to the agent). The agent investigates and posts results to the on-call Slack channel before the human acknowledges the page. ## L3: Diagnostic + agent-proposed remediation L3 is where AI starts proposing fixes, not just diagnosis. The pattern that works: 1. Pipeline fails → automated rollback (L1). 2. Agent investigates → RCA produced (L2). 3. Agent proposes a fix as a **pull request**, with the RCA as the PR description, the diff scoped to one file, and tests where possible. 4. Human reviews PR. If correct, merges. If wrong, comments and rejects. This works because the pull request is the natural human-review surface. The agent doesn't touch production directly; it touches the repository, which already has CI, code review, and a merge gate. [Aurora Actions](/blog/aurora-actions-background-automations) is built precisely for this pattern. A post-incident-completion Action with a prompt like "Open a PR widening alert thresholds for the three noisiest alerts in this incident" converts the human follow-up step into automated PR creation. The human review surface stays exactly the same as for human-authored PRs. The HolmesGPT equivalent ships as ["Operator mode"](https://github.com/robusta-dev/holmesgpt), the agent can write to GitHub when explicitly enabled. ## L4: Closed-loop with policy gates L4 is the contentious one. It involves the agent making changes *without* human approval, but only inside a tightly scoped policy. The pattern: - A **policy engine** ([Open Policy Agent](https://www.openpolicyagent.org/), [Kyverno](https://kyverno.io/), Casbin) defines which classes of remediation can run automatically. - The agent proposes a fix. The policy engine evaluates whether the fix matches a permitted class. - If yes → apply automatically with audit logging. If no → route to L3 (PR for human review). Permitted classes that are usually safe at L4: widening an alert threshold by less than 2x, restarting a pod, scaling a deployment within preset bounds, adding a retry loop to a network call, suppressing a noisy log line. Permitted classes that are usually *not* safe at L4: any data-plane change, any production traffic routing change, any secret or RBAC change, any change touching the policy engine itself. The reason L4 is contentious is that the policy gate is now a high-value target. An attacker who can broaden the policy can broaden the agent's blast radius. The same threat model we walk through in our [AI Agent kubectl Safety guide](/blog/ai-agent-kubectl-safety) applies, plus an additional layer: the policy engine must be operated with the same rigor as the orchestration plane itself. Almost no production teams in 2026 run pure L4. The credible deployments are **L3 with hardcoded L4 exceptions** for two or three well-understood remediation classes. That's where to aim. ## Common pitfalls A short list of failure modes we have seen, in our own work and in customer deployments. 1. **Auto-remediating *into* a worse state.** The classic failure is auto-scaling a service to handle elevated error rates that are themselves caused by a downstream dependency. The service scales, hammers the dependency harder, and the dependency collapses. **Fix:** never auto-remediate without dependency-graph awareness. Aurora uses [Memgraph](https://memgraph.com/) for this; HolmesGPT uses its toolset structure; pure-L1 stacks should require manual escalation when the failure crosses service boundaries. 2. **Trusting the AnalysisTemplate metric too much.** A 1% error rate threshold on a P99-tail service is meaningless if your real failure mode is request-stalled-not-failed. **Fix:** model what user-visible failure actually looks like, not what the cleanest Prometheus query produces. 3. **Letting the agent run unbounded retries.** AI agents that hit a "this didn't work" signal will often retry, sometimes thousands of times, burning tokens and triggering downstream rate limits. **Fix:** cap the agent's tool-call budget. Aurora's executor enforces this by default; verify your agent does the same. 4. **Skipping the post-mortem.** Auto-remediation that "just worked" without a clear human review of what happened is a slow path to brittleness. Every auto-remediation event should produce a postmortem the on-call reads. 5. **Conflating auto-remediation with "self-healing infra".** Kubernetes pod restarts are not auto-remediation. They are a runtime affordance. Auto-remediation is the response to a *change-driven* failure, the deploy, the config push, the schema migration. Keep the categories separate. ## A pragmatic 90-day path to auto-remediation For a team currently at L0 or L1. ### Days 1–14: instrument and detect Pick your three highest-traffic services. Add or harden: - Synthetic checks that exercise the user-visible path. - One Prometheus error-rate metric per service with a clear threshold. - A canary or blue-green rollout primitive ([Argo Rollouts](https://argoproj.github.io/argo-rollouts/) or [Flagger](https://fluxcd.io/flagger/)). Goal at end of week 2: a controlled bad deploy auto-rolls back without human intervention. ### Days 15–45: wire in the agent Deploy one of [Aurora](https://github.com/Arvo-AI/aurora), [HolmesGPT](https://github.com/HolmesGPT/holmesgpt), or [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) in read-only mode. Configure rollback events to webhook the agent. Have it post an RCA to your incident channel within five minutes of rollback. Goal at end of week 6: every rollback comes with a written diagnostic before the human acknowledges. ### Days 46–75: add agent-proposed remediation Enable PR-creation for the agent ([Aurora Actions](/blog/aurora-actions-background-automations) on-incident-completion trigger, or HolmesGPT Operator mode). Constrain initial scope to one repo and one class of fix (alert thresholds, retry loops, log suppression). Review every PR for the first two weeks. Goal at end of week 11: agent opens correct PRs in 70%+ of fired rollbacks. False-positive PRs are caught at code review. ### Days 76–90: policy-gate one fix class for L4 Pick the safest class, usually alert threshold widening when an alert fired more than N times in M hours with mean TTA above some bound. Define an OPA / Kyverno policy that permits *only that class.* Wire the agent to apply directly when the policy permits, raise a PR otherwise. Goal at end of week 12: one L4 lane open for one fix class with full audit trail. This is the conservative path. Aggressive teams have moved faster, but we have not seen anyone skip steps successfully. ## The DORA reality check The [DORA program's published guidance](https://dora.dev/guides/dora-metrics/) is blunt about what good looks like. Historical State of DevOps Reports have consistently shown the same shape of distribution: - **Change Failure Rate**: top performers maintain low single-digit percentages; lower performers see substantially higher rates. - **Failed Deployment Recovery Time (FDRT)**: top performers recover in under one hour; lower performers can take days to weeks. DORA's research has also consistently found that **speed and stability reinforce each other rather than trade off**, the fastest teams are also the most stable, per [DORA's history of metrics](https://dora.dev/insights/dora-metrics-history/) and successive State of DevOps Reports. Auto-remediation is one of the small number of capabilities that moves teams across these tiers without requiring deeper organizational change. The L1→L2 jump alone reduces FDRT meaningfully because the human is no longer reconstructing context, the agent has already done it. ## Where this is heading Two predictions, each with a reasonable evidence base. **1. The L2 → L3 transition becomes table-stakes within 18 months.** AI-authored PRs from agents are already merging in production at multiple companies in our network. Once the review surface is the same as for human-authored PRs (which it already is via GitHub / Bitbucket / GitLab), there is no organizational reason not to use them. **2. L4 stays narrow.** The threat surface of agent-applied changes is genuinely scary, and the per-incident savings of going from L3 to L4 are smaller than the savings from L1 to L2. Expect L4 to be the place where one or two well-understood fix classes get automated, while everything else stays L3. The teams who win in 2026-2027 are the ones who get to credible L3 first. ## Where Aurora fits Aurora is the AI agent layer of an auto-remediation stack, it covers L2 (investigation), L3 (PR-based remediation via [Aurora Actions](/blog/aurora-actions-background-automations)), and the agent half of L4 (policy-gated remediation). It does not replace Argo Rollouts or Flagger at L1; those remain the foundation. Aurora is the difference between rolling back blind and rolling back with a written RCA and a draft PR. - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) - **Docs:** [arvo-ai.github.io/aurora](https://arvo-ai.github.io/aurora/) - **Aurora Actions launch:** [Aurora Actions: User-Defined Background Automations](/blog/aurora-actions-background-automations) - **OSS comparison:** [Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt) - **Safety architecture:** [AI Agent kubectl Safety](/blog/ai-agent-kubectl-safety) --- ## HolmesGPT Alternatives, Tested: Aurora vs K8sGPT on 19 Faults URL: https://www.aurorasre.ai/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt Published: 2026-05-06 (updated 2026-10-21) Author: Ben Gervais, Engineer at Arvo AI > Open source HolmesGPT alternatives tested Oct 2026: Aurora and K8sGPT vs HolmesGPT on the same 19 Kubernetes faults, 2 runs each. Misses and raw output. **The two open source alternatives to HolmesGPT worth testing side by side are Aurora and K8sGPT, and they do different jobs.** On the same 19 injected Kubernetes faults, two runs each, Aurora named the root cause in 34 of 38 runs and HolmesGPT 0.43.0 in 31 of 38, a gap that is not statistically significant (two-sided Fisher exact test on those totals, p = 0.52, the same test [/benchmark](/benchmark) computes). HolmesGPT was about 3x faster at the median, 45 seconds against Aurora's 154. K8sGPT 0.4.40 does not investigate an alert at all: it scans a namespace and explains what its analyzers flag, in a median of 13 seconds for two analyze passes, and it printed "No problems detected" on 6 of the 19 faults (12 of its 38 runs). Read those numbers with three facts attached. Arvo builds Aurora, and Arvo wrote the scenarios and the grading criteria. Aurora's 38 verdicts were checked by a person at Arvo. The 76 HolmesGPT and K8sGPT verdicts are judge-model grades, re-read by two other models and regraded blind, and no person has checked them yet, so the p-value above compares person-checked Aurora grades with judge-only HolmesGPT grades. Everything is public: the [live benchmark table](/benchmark), and every scenario, transcript and grade file in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab). ## Aurora vs HolmesGPT vs K8sGPT: results on 19 Kubernetes faults | | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | Version tested | commit 87d22baf | 0.43.0 | 0.4.40 | | Model | Claude Sonnet 5 on Bedrock | Claude Sonnet 5 on Bedrock | Claude Sonnet 4.5 on Bedrock (see method) | | Input | Alert text | Alert text, `holmes ask`, default toolsets | No alert; scans the `shop` namespace | | Cluster access | The kind admin kubeconfig, behind Aurora's default read-only `observability_only` command policy | The same admin kubeconfig, behind HolmesGPT's own command allow-list | The same admin kubeconfig; `analyze` only reads | | Correct | 34 of 38 | 31 of 38 | Not scored as a peer (see below) | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 | | No answer | 0 | 0 | 12 ("No problems detected") | | Median time per run | 154 s | 45 s | 13 s for two analyze passes | | Median cost per run | $0.35 | not captured | not captured | | Who checked the grades | Judge model, then a person (38 of 38, none changed) | Judge model, two reviewer models, blind regrade; no person yet | Judge model, two reviewer models, blind regrade; no person yet | | Blind regrade agreed | 38 of 38 | 37 of 38 | 36 of 38 | "Correct" means the answer stated every finding the scenario requires and asserted none of its listed wrong answers. "Partial" means some of the required findings. K8sGPT's row uses the same rubric, but the rubric was built for tools that investigate an alert, and K8sGPT is not one. Its partials are mostly "it flagged the broken object and explained the flag, without saying what changed", which is what it is designed to do. Our position after running all three: for alert-driven root cause analysis on Kubernetes, Aurora and HolmesGPT are roughly even on accuracy in this lab, HolmesGPT is clearly faster and lighter to run, and the reasons to pick one over the other are scope, deployment model and workflow, not a 3-run difference in correct answers. K8sGPT is a scanner that belongs next to either of them, not instead of them. ## Results for every scenario Each fault name links to the runbook we wrote from the lab trace for that failure. Aurora and HolmesGPT columns show correct runs out of 2 and the median time. Per-run verdicts and links to the raw output are on [/benchmark](/benchmark). | # | Fault | Alert | Aurora | HolmesGPT | K8sGPT | |---|---|---|---|---|---| | 01 | [Worker OOMKilled after a batch size change](/runbooks/fix-oomkilled-kubernetes) | KubePodCrashLooping | 2/2 (160 s) | 2/2 (34 s) | 2 partial | | 02 | [Log shipper sidecar OOMKilled, app container healthy](/runbooks/fix-sidecar-container-oomkilled-kubernetes) | KubePodCrashLooping | 2/2 (90 s) | 2/2 (35 s) | 2 partial | | 03 | [Frontend rollout stuck on a mistyped image tag](/runbooks/fix-imagepullbackoff-kubernetes) | KubeDeploymentRolloutStuck | 2/2 (89 s) | 2/2 (25 s) | 2 partial | | 04 | [New api pods blocked on a Secret that does not exist](/runbooks/fix-createcontainerconfigerror-kubernetes) | KubeDeploymentRolloutStuck | 2/2 (164 s) | 2/2 (30 s) | 2 partial | | 05 | [Frontend crash loop from an invalid nginx config](/runbooks/fix-nginx-emerg-unknown-directive-kubernetes) | KubePodCrashLooping | 2/2 (136 s) | 2/2 (43 s) | 2 partial | | 06 | [api restarts because the liveness probe targets the wrong port](/runbooks/fix-liveness-probe-failed-kubernetes) | KubePodCrashLooping | 2/2 (105 s) | 2/2 (47 s) | No problems detected, both runs | | 07 | [api Service selects no pods after a selector change](/runbooks/fix-service-no-endpoints-kubernetes) | HighErrorRate | 2/2 (157 s) | 2/2 (47 s) | 2 partial | | 08 | [Worker pod Pending, CPU request larger than any node](/runbooks/fix-pod-pending-failedscheduling-kubernetes) | KubePodNotReady | 2/2 (226 s) | 2/2 (44 s) | 2 partial | | 09 | [api failing because redis is Pending on an unbindable PVC](/runbooks/fix-pvc-pending-storageclass-not-found-kubernetes) | KubePodCrashLooping | 2/2 (281 s) | 2/2 (96 s) | 2 partial | | 10 | [api crash loop from a wrong REDIS_HOST](/runbooks/fix-dns-name-not-resolving-kubernetes) | KubePodCrashLooping | 2/2 (245 s) | 1/2 (56 s) | 2 wrong | | 11 | [frontend 5xx from a wrong api Service targetPort, next to a noisy but harmless worker](/runbooks/fix-service-targetport-mismatch-kubernetes) | HighErrorRate | 1/2 (340 s) | 0/2 (56 s) | No problems detected, both runs | | 12 | [Two faults at once: worker OOM, and a frontend Service selecting no pods; the alert covers only the worker](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes) | KubePodCrashLooping | 0/2 (344 s) | 0/2 (30 s) | 2 partial | | 13 | [frontend 5xx because a NetworkPolicy blocks api to redis](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes) | HighErrorRate | 2/2 (323 s) | 2/2 (114 s) | No problems detected, both runs | | 14 | [api rollout stuck because a ResourceQuota forbids the surge pod](/runbooks/fix-resourcequota-exceeded-rollout-stuck-kubernetes) | KubeDeploymentRolloutStuck | 2/2 (126 s) | 2/2 (43 s) | 2 partial | | 15 | [One in three api requests fails because a stray Deployment's pods carry app=api](/runbooks/fix-service-selects-wrong-pods-label-overlap-kubernetes) | HighErrorRate | 2/2 (147 s) | 2/2 (130 s) | No problems detected, both runs | | 16 | [New api pods crash because their dnsConfig search domains are for another cluster domain](/runbooks/fix-pod-dnsconfig-search-domain-not-resolving-kubernetes) | KubePodCrashLooping | 2/2 (173 s) | 2/2 (92 s) | 2 wrong | | 17 | [Worker OOMKilled every few minutes by a slow leak in its code; logs look clean](/runbooks/fix-memory-leak-oomkilled-kubernetes) | KubePodCrashLooping | 2/2 (144 s) | 2/2 (71 s) | No problems detected, both runs | | 18 | [api rollout stuck on an init container waiting for a Service that does not exist](/runbooks/fix-pod-stuck-init-container-kubernetes) | KubeDeploymentRolloutStuck | 2/2 (159 s) | 1/2 (43 s) | 1 partial, 1 wrong | | 19 | [Half of frontend requests fail; one frontend pod loaded an edited ConfigMap, the other still runs the old config](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes) | HighErrorRate | 1/2 (267 s) | 1/2 (78 s) | No problems detected, both runs | Scenarios 01 to 10 have one clear fault each. Scenarios 11 to 19 were written to be harder: noise next to the fault, two faults at once, a slow leak, a partial restart. All the non-correct runs for both investigators are in the harder set except HolmesGPT's run on 10. ## Where did Aurora miss? Aurora was not correct in 4 runs, all partial. **Scenario 12, both runs: it found the alerted fault and scoped the incident to it without checking.** The alert named only the OOMKilled worker. A second fault, a frontend Service whose selector matched no pods, had taken the frontend down for users. Both Aurora runs diagnosed the worker correctly (`BATCH_SIZE` raised from 500 to 2500 against a 192Mi limit, with the buffer math from `worker.py`) and neither ran `get svc` or `get endpoints`. Run 1 then wrote this, citing only two `get pods` calls: ``` Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]. ``` Run 2 wrote "Impact is scoped to the single-replica `worker` deployment in `shop`". That is the worst kind of miss for an incident tool: a real outage that the report says is not there. HolmesGPT made the same miss on both runs of this scenario. The [runbook for a second fault the alert did not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes) shows the traffic-level checks that would have caught it. **Scenario 19, run 1: the right clue, the wrong conclusion.** Aurora found the edited `frontend-nginx` ConfigMap pointing at `api:8000` while the Service listens on 8080. It did not work out that only the recreated pod had loaded the edit. It tried to read the live config inside the failing pod, and its own default command policy refused: ``` $ kubectl exec frontend-76d98597b9-g5zrg -n shop -c frontend -- cat /etc/nginx/conf.d/default.conf Command blocked by organization policy: Mutating kubectl operations ``` So it left the question open in the report: ``` This discrepancy was not fully reconciled — the live nginx configuration inside the running pod could not be inspected to confirm which port it was actually using at the time of each failure, since direct exec access was blocked by policy [10]. ``` Its own `get pods` output already held the answer: the failing pod was 9 seconds younger than the healthy one. Run 2 got it right with 28 tool calls. See the [runbook for a ConfigMap change not applied to all pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes). **Scenario 11, run 2: graded partial for not ruling out the red herring.** Aurora found the Service `targetPort` of 9090 against a container listening on 8080, which is the fault. The scenario also has a worker logging harmless errors, and the rubric requires the answer to say those are not the cause. Aurora's report said only this about the worker: ``` - Health and configuration of the redis, worker, and loadgen deployments — no symptoms or alerts pointed to these components, so they were not examined. ``` You can argue that is a correct answer that was not misled. We kept the rubric as written, and the same rule cost HolmesGPT both of its runs on 11. Aurora's time column is also worse than it should be. During these runs, Aurora's own background "prediscovery" jobs competed with the investigations for worker slots, which pushed some runs to 444 to 548 seconds. The [lab report](/blog/aurora-lab-report-kubernetes-19-failures) has the details and the fixes we made after it. ## Where was HolmesGPT better than Aurora? **Speed, everywhere.** HolmesGPT was faster on every one of the 19 scenarios, with a median of 45 seconds against 154. Part of Aurora's gap is the lab contention above, but not all of it: on the ten single-fault scenarios Aurora's medians are 89 to 281 seconds against HolmesGPT's 25 to 96, and no Aurora run in that set finished under 60 seconds. **Same evidence in a fraction of the time.** On scenario 12, run 2, HolmesGPT read the worker's code out of its ConfigMap and did the memory math in 34 seconds. Both Aurora runs did the same math, in 140 and 548 seconds, and both tools missed the second fault: ``` Why it crashes: worker.py (from ConfigMap api-code) allocates batch buffers of 100 KiB each on every loop iteration: buf = [bytearray(100*1024) for _ in range(batch)]. At BATCH_SIZE=500 that's ~50MB — fine. At 2500 that's ~250MB, which blows past the 192Mi limit immediately. ``` **Evidence quality on scenario 19.** On run 2 it worked out the two-pod split from creation timestamps and logs, which is exactly what Aurora's run 1 failed to do: ``` With 1 of 2 replicas permanently broken and the other still fine, this produces almost exactly the observed ~48% 5xx rate on /api/ (one healthy, one failing, roughly 50/50 traffic split). ``` The same answer then got the mechanism wrong, and the blind regrade moved it from correct to partial. nginx reads its config only at start, so the healthy pod would not break on its own: ``` xmkdj is only "healthy" because it hasn't yet re-synced its mounted ConfigMap volume (kubelet syncs mounted ConfigMaps periodically, default ~1 minute). ``` **Setup.** HolmesGPT is a CLI you install with Homebrew or pipx and point at a kubeconfig and an LLM key. Aurora is a multi-container stack (see setup below). ## Where did HolmesGPT miss? Seven HolmesGPT runs were not correct. Two of them are the scenario 12 miss described above, and two are the scenario 11 red-herring rule. **Scenario 19, run 1: wrong, and it declared the incident over.** With 48% of requests failing, HolmesGPT fetched logs from one of the two frontend pods (the healthy one, judging by the upstream port in its errors), found two 502s from startup, and concluded: ``` AI: All data points to a transient startup race, now resolved. ``` It listed both frontend pods but never read the other pod's logs or the ConfigMap's contents. **Scenario 10, run 1: right symptom, inverted change.** It found `redis-master` failing to resolve, then decided the crashing pod was a leftover from the old ReplicaSet and the latest deploy was the fix: "`this is cosmetic now since the new RS already has the correct value`". The judge marked the second required finding partial for that reversal. **Scenario 18, run 1: correct diagnosis, flagged fix.** It named the `wait-for-db` init container and the missing `postgres` Service, which is the fault. Its first suggested fix was to deploy a postgres Service or StatefulSet. The scenario lists "creating a postgres database as the fix without noting the api does not need one" as a wrong answer, because the app does not use postgres. That made the run partial. A reader could call this strict, as with Aurora's run on 11. HolmesGPT's own command allow-list blocked `kubectl rollout history` in 12 runs, 2 of its 7 non-correct runs among them. We did not rerun those with the command allowed. Aurora's default policy had the same block in the first lab report (26 of 38 runs), so neither tool had an easy "what changed" view on the default settings. ## What did K8sGPT surface? K8sGPT runs a fixed set of analyzers over the cluster and asks the LLM to explain each finding. It got no alert. That makes it fast and literal, and it is very good when the broken object carries its own error. On scenario 3 it named the typo in 17 seconds: ``` - Error: Back-off pulling image "nginx:1.27-alpne": ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.27-alpne": failed to resolve reference "docker.io/library/nginx:1.27-alpne": docker.io/library/nginx:1.27-alpne: not found Error: The Docker image "nginx:1.27-alpne" cannot be found because the tag name is misspelled. ``` **On scenario 12 it was the only tool to report the second fault**, in both runs, because it scans everything instead of following the alert: ``` 2: Service shop/frontend() - Error: Service has no endpoints, expected label app=frontend - Error: Service has no endpoints, expected label tier=web Error: The Service cannot find any Pods because none have the required labels (app=frontend AND tier=web). ``` Both investigators would have been correct on 12 if they had done what K8sGPT does by default. That is the strongest argument in this data for running a scanner next to an agent. Where nothing looks broken at the object level, it finds nothing. On scenarios 6, 11, 13, 15, 17 and 19 it printed this in both runs, while the alert was firing: ``` No problems detected ``` Where it did report something, it reports the symptom and a generic checklist, and it does not say what changed. On scenario 1 it found the OOMKill and suggested raising the limit (to an example "2Gi" in run 1), without the `BATCH_SIZE` change. On 16 it blamed probe timing for what was a DNS search-domain fault, which the judge graded wrong. Two caveats specific to K8sGPT. We ran it with its default analyzers. Its README lists `networkPolicyAnalyzer` and `logAnalyzer` as optional, and they were off; we did not test whether they change the result on 13 or 17. And the from-scratch regrade (the third pass described in the method section; the first reviewer model agreed with all 76 grades) disputed 6 of the judge's K8sGPT labels, saying symptom-only answers on 05, 10 and 18 should be "no answer" rather than partial or wrong. We kept the judge's labels, which are the more generous reading for K8sGPT, and published the dispute in the lab repo. ## How fast and how expensive is each open source SRE agent? | | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | Median time | 154 s | 45 s | 13 s (two passes) | | Fastest and slowest scenario median | 89 s to 344 s | 25 s to 130 s | 0 to 46 s | | LLM cost per run | median $0.35, from Aurora's own usage tracking | not captured | not captured | We do not have a per-run cost for HolmesGPT or K8sGPT, so we do not publish one. HolmesGPT printed the reason at the top of every run: its LLM library had no price entry for the model. ``` Model 'bedrock/us.anthropic.claude-sonnet-5' has no entry in litellm's cost map and no input/output_cost_per_token configured. Usage event costs for this model will be 0. ``` Aurora has one cost that does not show up in the per-run number. Each kubeconfig upload schedules a background prediscovery run, which cost about $1 per cluster connection in the lab, almost twice what all 44 investigations together cost ($30.38 against $16.77). That is in the lab repo's REPORT.md and is something we are fixing. ## How hard is each one to set up? This is how the lab ran HolmesGPT and K8sGPT, from `runner/h2h.sh` in the lab repo. Each is one command against a kubeconfig: ```bash k8sgpt analyze --explain --namespace shop --output json --no-cache k8sgpt analyze --explain --namespace shop --no-cache holmes ask "$ALERT Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." \ --model "bedrock/us.anthropic.claude-sonnet-5" --max-steps 40 ``` - **K8sGPT** is a single Go binary ([Homebrew, RPM, DEB, APK or Windows builds](https://github.com/k8sgpt-ai/k8sgpt)), configured with `k8sgpt auth add`. An [operator](https://github.com/k8sgpt-ai/k8sgpt-operator) runs it in-cluster on a schedule. - **HolmesGPT** installs with Homebrew, pipx or Docker ([install docs](https://holmesgpt.dev/latest/installation/cli-installation/)), and also ships a Helm chart, an HTTP server and a Python SDK. The CLI needs only an LLM key and cluster access. - **Aurora** is a self-hosted application with a web UI, a database, Vault and background workers. The [quick start](https://github.com/Arvo-AI/aurora) is `make init`, an LLM key in `.env` and `make prod-prebuilt`, then a Vault token step and a restart. Production installs use the Helm chart. Alerts arrive by webhook and each one starts an investigation in the background, so it is built to sit in the incident path rather than be run by hand. If you want an answer in a terminal in five minutes, HolmesGPT and K8sGPT win. Aurora's setup pays off only if you want the alert-to-postmortem workflow it is built around. ## Which open source SRE agent works beyond Kubernetes? Our lab tests only Kubernetes. Scope beyond it comes from each project's own README and docs, checked on 2026-10-21. | | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | What it is | Alert-driven investigation agent with incident workflow | Investigation agent: CLI, server, operator | Kubernetes analyzer with LLM explanations | | License | Apache 2.0 | Apache 2.0, [CNCF Sandbox since 2025-10-08](https://www.cncf.io/projects/holmesgpt/) | Apache 2.0, [CNCF Sandbox since 2023-12-19](https://www.cncf.io/projects/k8sgpt/) | | Clouds | Runs `kubectl`, `aws`, `az` and `gcloud`; connectors for AWS, Azure, GCP, OVH, Scaleway, Fly.io, Cloudflare ([README](https://github.com/Arvo-AI/aurora)) | AWS, Azure and GCP toolsets via MCP servers; AKS and OpenShift toolsets ([README](https://github.com/HolmesGPT/holmesgpt)) | Kubernetes; an optional AWS integration adds an EKS analyzer ([integrations](https://docs.k8sgpt.ai/reference/cli/filters/)) | | Observability and data | Datadog, Grafana, New Relic, Dynatrace, Splunk, Elastic and more | Prometheus, Grafana, Loki, Tempo, Datadog, New Relic, Elasticsearch, Splunk, Coralogix, VictoriaMetrics, SQL databases, Kafka, RabbitMQ and more | Prometheus, Trivy, Kyverno integrations | | Alert sources | PagerDuty, Datadog, Grafana, New Relic, OpsGenie, incident.io, CloudWatch alarm webhooks | AlertManager, PagerDuty, OpsGenie, Jira, with findings written back | Operator integrates with Prometheus and Alertmanager | | Code and docs | GitHub, Bitbucket, Jenkins, CloudBees, Terraform; postmortems to Confluence, Notion, SharePoint | GitHub, GitLab, Jenkins, ArgoCD, Helm, Confluence, ServiceNow | None | | LLM providers | OpenAI, Anthropic, Gemini, Vertex AI, OpenRouter, Bedrock, Ollama | Anthropic, Bedrock, Azure AI Foundry, Gemini, Vertex AI, Ollama, OpenAI, OpenRouter and OpenAI-compatible ([AI providers](https://holmesgpt.dev/latest/ai-providers/)) | 11 backends including OpenAI, Bedrock, Azure OpenAI, Gemini, Vertex AI, watsonx.ai, LocalAI and Ollama ([backends](https://docs.k8sgpt.ai/reference/providers/backend/)) | | Writes to your systems | Suggests fixes and can open a remediation PR on GitHub, gated on human approval, no auto-merge; org-level command policies (the lab used the default read-only `observability_only` policy) | Read-only by design per its README; an optional [Kubernetes Remediation MCP](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/kubernetes-remediation-mcp/) adds writes behind human approval; operator mode (alpha) can open PRs | Read-only CLI; the operator has an alpha, opt-in auto-remediation for image failures ([operator README](https://github.com/k8sgpt-ai/k8sgpt-operator)) | | Privacy feature | Self-hosted; air-tight bundle for restricted networks | Self-hosted; respects RBAC | An anonymize flag masks object names and labels before they reach the LLM; its README says this does not yet apply to events | Two points to weigh. HolmesGPT is not Kubernetes-only: its README says it works without Kubernetes, against VMs, cloud services and databases, and its list of observability and data toolsets is long. Aurora's difference is that it is an incident system rather than a tool: alerts come in by webhook, it investigates in the background, keeps an incident history, writes the postmortem and proposes the fix PR. Aurora's defaults run a single agent; a multi-agent orchestrator exists but is off unless you enable it. On permissions: all three tools got the same kind admin kubeconfig (`kubernetes-admin`). Aurora ran under its default read-only `observability_only` command policy on top of it; HolmesGPT's own command allow-list asked for approval on commands it did not know, including every `kubectl exec` it tried, and no approval was given; K8sGPT's `analyze` only reads. No transcript shows any of them running a mutating command. We did not test any tool on a restricted kubeconfig. HolmesGPT's README says it has read-only access and respects RBAC, and Aurora adds its own command policy on top of RBAC. ## What other open source SRE agents are there? We have not benchmarked these. Facts are from each repository on 2026-10-21. | Project | What it is | Status | |---|---|---| | [OpenSRE](https://github.com/Tracer-Cloud/opensre) (Tracer) | Toolkit for building AI SRE agents; can optionally execute remediation | Apache 2.0, labels itself Public Alpha | | [kagent](https://github.com/kagent-dev/kagent) | Kubernetes native framework for building and running agents, with MCP tools for Kubernetes, Helm, Argo, Prometheus and more | Apache 2.0, [CNCF Sandbox since 2025-05-22](https://www.cncf.io/projects/kagent/) | | [Mezmo AURA](https://github.com/mezmo/aura) | Mezmo's open-source harness for SRE agents | Apache 2.0 | | [Keep](https://github.com/keephq/keep) | Alert management and correlation, not an investigator | MIT, with a separately licensed `ee/` directory | | [Coroot](https://github.com/coroot/coroot) | Observability and APM with AI root cause analysis | Apache 2.0 | | [IncidentFox](https://github.com/incidentfox/incidentfox) | AI SRE platform | Repository archived, read-only | kagent and OpenSRE are frameworks: you build the agent. Aurora, HolmesGPT and K8sGPT ship one. If you maintain any of these and want them on the board, the [benchmark](/benchmark) takes a script that reads a kubeconfig and an alert and prints a root cause; we run it on all 19 faults and publish the result. ## How was the benchmark run, and what are its limits? - **Cluster.** kind v0.33.0, Kubernetes v1.37.0, 3 nodes. A synthetic `shop` app: nginx frontend, Python api, worker, redis, load generator. No customer code or data. Each fault is one `break.sh` in the [scenarios folder](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios), and `check.sh` proves it is visible before any tool runs. - **Dates.** Aurora ran on 2026-10-02. HolmesGPT and K8sGPT ran on 2026-10-15 and 2026-10-16. Aurora's commit predates the scenarios, so it was not tuned on them. - **Prompts differ.** Aurora got the alert plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." HolmesGPT got the alert plus "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." HolmesGPT's prompt names the namespace and asks what changed, which if anything helps it. K8sGPT got no alert. - **Models.** Aurora and HolmesGPT both ran Claude Sonnet 5 on Bedrock. K8sGPT ran Claude Sonnet 4.5 because it always sends a temperature parameter, which Bedrock rejects for Sonnet 5; we did not try a proxy that strips it. - **HolmesGPT settings.** `holmes ask` with default toolsets and a 40-step cap it never reached. The lab has no Alertmanager, so `holmes investigate` was not used. - **Grading.** A judge model (Claude Opus 5.5) marked each required finding as met, partial or missed and quoted the answer. The judge is from the same vendor as every model under test. A second model re-read all 76 HolmesGPT and K8sGPT grades and agreed with all of them. A third pass graded those 76 from scratch without seeing the judge: it agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels. A person at Arvo reviewed Aurora's 38 and changed none. - **Blinded regrade.** To test whether the judge favoured Aurora, every answer was graded again with the agent's name, its tool names and the cluster name removed. The blind verdict matched for 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs. All three changes were downgrades of non-Aurora runs. The blind judge still recognised K8sGPT's output format, so blinding is incomplete for that tool. - **Reruns and one correction.** Scenario 19 was rerun after its break script hit a TLS timeout against the API server, and scenario 18 after HolmesGPT hit a Bedrock timeout; both are logged in the repo. The first K8sGPT grade for scenario 18, run 1 was of the replaced first-pass output. We regraded the actual file on 2026-10-21 and the verdict stayed partial. - **What it cannot tell you.** Two runs per scenario do not measure variance. One small app on kind is quieter than production and has no dashboards, runbooks or history, which every tool here could use. The scenarios were chosen by the vendor of one of the tools. HolmesGPT 0.44.0 (2026-10-14) and 0.45.0 (2026-10-15) were already out when our HolmesGPT runs started, and K8sGPT 0.4.41 shipped on 2026-10-21; we tested 0.43.0 and 0.4.40 and have not retested. ## Which should you pick? - **Kubernetes-centric team, wants answers fast, happy to run it from a terminal or Slack:** HolmesGPT. In our lab it matched Aurora within noise (its grades are judge-model only) and was about 3x faster. - **Wants a periodic cluster scan that catches broken objects nobody alerted on:** K8sGPT, next to whichever investigator you choose. It was the only tool that saw the second fault in scenario 12. - **Wants alerts from PagerDuty, Datadog or Grafana investigated automatically, across Kubernetes and cloud accounts, with postmortems and fix PRs behind approval, all self-hosted:** Aurora. Expect a heavier install and slower investigations than HolmesGPT. All three are open source, so the cheapest way to decide is to break something on a test cluster and run them yourself. The lab repo breaks a scenario and runs HolmesGPT and K8sGPT against it in three commands (Aurora needs its own install first), and the [Kubernetes pod diagnoser](/diagnose) handles the one-off case where you just want to paste `kubectl` output. Aurora is on GitHub at [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora); to see it on your own alerts, [book a demo](https://cal.com/arvo-ai/demo). *Benchmark data from [results.json](https://raw.githubusercontent.com/Arvo-AI/aurora-lab/main/benchmark/results.json), generated 2026-10-21. Competitor features, versions and licenses verified against their repositories and docs on 2026-10-21.* --- ## AI Agent kubectl Safety: Sandboxed Execution URL: https://www.aurorasre.ai/blog/ai-agent-kubectl-safety Published: 2026-05-06 (updated 2026-09-28) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Giving an AI agent kubectl access is an architecture decision. OWASP threats, k8s-sigs/agent-sandbox, gVisor, and Aurora's pod-isolated execution model. > ## Key Takeaways > > - **Giving an AI agent kubectl access is an architecture decision, not a permission flag.** Per-permission gates fail under prompt injection. > - **OWASP ranks "Excessive Agency" as LLM06 in the [2025 Top 10 for LLM Applications](https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/)** and "Tool Misuse and Exploitation" as ASI02 in the [2026 Top 10 for Agentic Applications](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/). > - **The Kubernetes ecosystem already has an answer**: [k8s-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) provides a declarative API for isolated agent runtimes using gVisor or Kata Containers. > - **Real precedent exists.** [EchoLeak (CVE-2025-32711)](https://thehackernews.com/2025/06/zero-click-ai-vulnerability-exposes.html), CVSS 9.3, was the first publicly documented zero-click prompt-injection data exfiltration in a production LLM system. The kubectl analogue would be cluster-wide. > - **Aurora runs every `kubectl` command in a pod-isolated process** via its `terminal_run` primitive, with an environment-variable allowlist that strips secrets, signature-matcher and LLM-judge guardrails, and per-invocation cloud credentials. Of the [46+ products marketed as "AI SRE" in 2026](https://rootly.com/ai-sre-guide), only a handful publicly document their kubectl execution architecture, and the gap between vendors that handle this well and vendors that handle it badly is the single largest unspoken risk in the category. **AI agent kubectl safety is the architectural discipline of letting an AI agent run `kubectl` (or any cloud CLI) against production without inheriting cluster-wide blast radius if the agent is compromised.** It is not the same as RBAC scoping, and it is not the same as a human approval prompt, both are necessary but neither is sufficient on its own. When [OWASP published its 2025 Top 10 for LLM Applications](https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/), it ranked **Prompt Injection (LLM01)** as the top risk and **Excessive Agency (LLM06)** as one of the most consequential, defining it across three root causes: excessive functionality, excessive permissions, and excessive autonomy. In December 2025, OWASP followed up with a [dedicated Top 10 for Agentic Applications](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/) that names **Tool Misuse and Exploitation (ASI02)** and **Identity and Privilege Abuse (ASI03)** as primary attack surfaces. Translation: if you give an AI agent the ability to run `kubectl`, `aws`, or `gcloud` commands against production, you have a security architecture problem. not a permissions problem. This guide walks through the threat model, the emerging Kubernetes sandboxing standard, and how to evaluate any AI SRE on its kubectl safety. ## What can go wrong when AI agents run kubectl? Any LLM-driven agent that executes commands inherits the security properties of the LLM, the harness, and the runtime. Three real-world precedents illustrate the failure modes: - **EchoLeak (CVE-2025-32711)**. Microsoft 365 Copilot, CVSS 9.3 critical, [patched in June 2025](https://thehackernews.com/2025/06/zero-click-ai-vulnerability-exposes.html). Discovered by Aim Security, it was the first publicly documented zero-click indirect prompt-injection data exfiltration in a production LLM system. A crafted email sat in Outlook; when the user later asked Copilot for an unrelated summary, the email's hidden instructions fired and exfiltrated SharePoint, OneDrive, and Teams data. Research paper: [arXiv:2509.10540](https://arxiv.org/abs/2509.10540). - **MITRE ATLAS prompt-injection techniques**, [MITRE ATLAS](https://atlas.mitre.org/) catalogues real-world adversary techniques against AI systems, including indirect prompt injection that turns an LLM with tool access into an attacker-controlled execution surface. The framework specifically documents techniques for exfiltration via AI agent tool invocation. - **Agent Session Smuggling**. Palo Alto Unit 42 (November 2025) demonstrated rogue agents exploiting trust in the Agent-to-Agent (A2A) protocol with multi-turn manipulation. Documented in OWASP's Agentic Top 10. None of these specifically targeted kubectl-running agents in production, but the class is the same and the blast radius would be larger. An agent that can run `kubectl delete` is one prompt-injection payload away from a cluster-wide outage. ## The Four Attack Surfaces of Agentic kubectl Most teams think of kubectl agent safety as a single problem ("can the agent be tricked?"). It's actually four distinct attack surfaces, each requiring its own mitigation. | Surface | Failure mode | Why permission-scoping alone fails | Mitigation | |---|---|---|---| | **1. Prompt injection** | Hidden instructions in logs, alerts, runbooks, or chat coerce the agent | Compromised agent acts within its granted permissions, which is exactly what permission-scoping permits | Sandboxed runtime; never trust LLM output derived from data the LLM read | | **2. Credential leakage** | Executed command reads `AWS_SECRET_ACCESS_KEY`, `VAULT_TOKEN`, `KUBECONFIG` from inherited env | Permissions live on credentials; if the credential leaks, the permission set leaks with it | Per-invocation short-lived credentials (STS, Service Principal); explicit env allowlist that strips secrets | | **3. Blast radius escalation** | Legitimate command runs against wrong namespace, region, or cluster | Permissions don't model "right action, wrong target" | Default read-only; dependency-graph awareness; human approval for destructive writes | | **4. Audit trail gaps** | Logs capture commands without the agent's reasoning | Permission systems audit "who ran what," not "why" | Per-investigation transcripts that link reasoning → tool calls → outputs | ### Attack Surface 1: Prompt injection The agent reads a log line, alert payload, runbook, or chat message that contains hidden instructions. The LLM cannot reliably distinguish data from instructions in the same channel, this is the fundamental property OWASP's LLM01 captures. Even frontier models do not eliminate it. Anthropic has publicly stated that "no browser agent is immune to prompt injection" and publishes [defense benchmarks](https://www.anthropic.com/news/prompt-injection-defenses) showing measurable but imperfect attack-prevention rates across computer-use, bash tool use, and MCP workflows. The implication for kubectl-running agents is clear: **the LLM is not the security boundary. The runtime is.** Mitigation: never trust LLM output that originates from data the LLM also read. Sandbox the execution layer so even a successful injection has limited blast radius. ### Attack Surface 2: Credential leakage If the agent runs commands with credentials inherited from the host process environment (`AWS_SECRET_ACCESS_KEY`, `KUBECONFIG`, `VAULT_TOKEN`), a successful command-injection or shell escape exposes everything the agent process has access to. Long-lived static credentials make this catastrophic. Mitigation: per-invocation credential scoping. AWS STS AssumeRole, Azure Service Principal sessions, GCP short-lived tokens. Strip everything else from the child process environment with an explicit allowlist. ### Attack Surface 3: Blast radius escalation Even legitimate, non-injected commands can have outsized effects. `kubectl delete pod` on the wrong namespace. `aws ec2 terminate-instances` against a misidentified region. The agent doesn't need to be compromised, it just needs to be wrong. Mitigation: read-only by default, write actions behind explicit human approval, and dependency-graph awareness so the agent can compute blast radius before acting. (This is the role of Aurora's [Memgraph dependency graph](/blog/multi-cloud-incident-management).) ### Attack Surface 4: Audit trail gaps When an investigation runs across 20+ tool invocations, traditional audit systems (CloudTrail, Kubernetes audit logs) record what was run but not why. A reviewer six months later cannot tell whether a `kubectl scale` was a legitimate response to a load spike or an injected instruction. Mitigation: structured per-investigation transcripts that capture agent reasoning alongside tool calls. The right log isn't "kubectl was run", it's "in response to alert X, the agent hypothesized Y, ran kubectl Z, and observed W." ## Why "human approval" alone is not enough The most common safety story in the AI SRE space is "the agent suggests; humans approve." That is necessary but not sufficient. The problem with approval gates as the only line of defense: - **Decision fatigue.** An agent that handles 50 alerts a week generates dozens of approval prompts. Humans rubber-stamp. - **Approval ≠ understanding.** Engineers approve commands they don't fully understand because the agent's reasoning sounds plausible. - **Injected intent looks legitimate.** A prompt-injection payload can produce a recommendation that *reads* exactly like a normal RCA. The approver has no signal that the underlying instruction came from an attacker. Approval gates are critical, but they need to sit on top of an already-sandboxed runtime. not be the only protection. ## Permission scoping vs sandboxed execution: what's the difference? These two terms get conflated. They aren't the same thing. **Permission scoping** restricts what an agent's identity can do. RBAC roles, IAM policies, kubeconfig contexts. It's necessary, but it operates at the cluster-API layer, meaning a successful prompt injection can still use every permission the agent has. **Sandboxed execution** isolates the *runtime* in which commands execute. If the agent's process is compromised, the sandbox limits what the compromised process can do regardless of the credentials it holds. The compromised process can't read other pods' files, can't reach other nodes, can't escalate to the host kernel. The defensible architecture combines both: tight permission scoping (small RBAC role, short-lived credentials) + runtime isolation (sandboxed execution). ## How sandboxed kubectl actually works The Kubernetes ecosystem standardized on this pattern in 2025–2026. ### k8s-sigs/agent-sandbox [k8s-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) is a formal Kubernetes SIG Apps subproject that launched at KubeCon Atlanta in November 2025. It provides a declarative Kubernetes API for "isolated, stateful, singleton workloads", built specifically for AI agent runtimes that may execute untrusted, LLM-generated code. Core CRDs: - `Sandbox`, an isolated pod-equivalent with stronger boundaries - `SandboxTemplate`, reusable configuration - `SandboxClaim`, request a sandbox for a workload - `SandboxWarmPool`, pre-created sandboxes that bring cold-start under one second The [Kubernetes blog post from March 2026](https://kubernetes.io/blog/2026/03/20/running-agents-on-kubernetes-with-agent-sandbox/) makes the architectural claim explicit: "Isolation achieved via runtime-level sandboxing (gVisor/Kata), not just container-level namespaces." ### gVisor [gVisor](https://gvisor.dev/) is a Google-maintained user-space application kernel that provides kernel-level isolation without full virtualization. Architecture: **Sentry** (a kernel emulator written in Go) intercepts roughly 200 Linux syscalls; **Gofer** brokers filesystem access over 9P. The OCI runtime is `runsc`, drop-in compatible with `runc`. gVisor runs in production at Google for App Engine standard, Cloud Functions, Cloud Run, and Cloud ML Engine. GKE Sandbox productizes it for GKE node pools. It is one of two named isolation backends in agent-sandbox (the other being Kata Containers, which uses lightweight VMs). ### Why this matters for AI SRE An AI SRE that runs `kubectl` against production is exactly the kind of workload agent-sandbox was built for. It executes LLM-generated commands. It needs file system isolation, syscall isolation, and per-invocation credential scoping. It benefits enormously from a warm pool that reduces cold-start latency. If you are evaluating an AI SRE in 2026, this is one of the right questions to ask: *what isolation backend does the agent use when it executes commands?* ## How Aurora's pod-isolated execution works Aurora's approach predates agent-sandbox and follows the same architectural principles. When Aurora's agent runs a `kubectl`, `aws`, `az`, or `gcloud` command, it doesn't use `subprocess.run()` directly. It uses an internal primitive called `terminal_run`, defined in [`server/utils/terminal/terminal_run.py`](https://github.com/Arvo-AI/aurora). The module's docstring is explicit: > Drop-in replacement for subprocess.run() that executes in terminal pods. This module provides a terminal_run() function that mimics subprocess.run() API but executes commands in isolated terminal pods via kubectl exec. Safety guardrails (signature matcher + LLM judge) run automatically unless the caller passes `trusted=True` for known-safe internal operations. Three properties matter: **1. Pod-isolated execution.** When the `ENABLE_POD_ISOLATION` flag is set, every external command runs inside a separate terminal pod via `kubectl exec`. The flag defaults to on in code, but the shipped Helm values and example environment file set it off, so confirm it for your own deployment rather than assuming. The agent's own process never executes the command directly. A successful command-injection in the agent's reasoning loop does not give an attacker access to the agent host. **2. Two-stage safety guardrails.** Before any non-trusted command runs, two checks fire automatically: a deterministic signature matcher that rejects known-dangerous patterns, and an LLM judge that evaluates the proposed command against the investigation context. The `trusted=True` flag bypasses both, used only for known-safe internal operations like configured connector calls. **3. Sanitized environment allowlist.** Aurora's `terminal_exec_tool` module defines an explicit `_SAFE_ENV_KEYS` set: `PATH`, `HOME`, `USER`, `SHELL`, `TERM`, `LANG`, `TMPDIR`, `SSL_CERT_FILE`, plus `ENABLE_POD_ISOLATION` itself. Everything else, including `VAULT_TOKEN`, `DATABASE_URL`, `SECRET_KEY`, and any cloud credentials, is stripped from the child process environment. A compromised command cannot read the agent's secrets via `env`. Cloud credentials are handled separately. Aurora calls `generate_contextual_access_token` and `generate_azure_access_token` per invocation. AWS uses STS AssumeRole via cross-account roles ([`aurora-cross-account-role.yaml`](https://github.com/Arvo-AI/aurora/tree/main/server/connectors/aws_connector)), short-lived credentials, not long-lived access keys. Azure uses Service Principal sessions. GCP uses OAuth-derived tokens. For agents that need to reach customer Kubernetes clusters Aurora can't access directly, a separate [`kubectl-agent`](https://github.com/Arvo-AI/aurora/tree/main/kubectl-agent) binary deploys via Helm into the customer's cluster and connects outbound over WebSocket. No inbound network access required, no kubeconfig sharing, no static credentials at rest. ## How to evaluate an AI SRE's kubectl safety model Eight questions to ask any AI SRE vendor or open-source project before enabling production access: 1. **Where does the command actually execute?** Same process as the agent? Same host? Separate container? Sandboxed runtime (gVisor/Kata)? 2. **What credentials does the command inherit from the host environment?** Specifically: can the executed command read your agent's vault token, database URL, or other host secrets? 3. **Are credentials short-lived or static?** STS / Service Principal sessions, or long-lived access keys? 4. **Is the default read-only?** What flag, configuration, or RBAC role enables write access? 5. **What happens between "agent decides to run X" and "X runs"?** Is there a deterministic policy check? An LLM judge? A human approval prompt? All three? 6. **Are destructive actions specifically gated?** What's the definition of "destructive", vendor-defined or operator-configurable? 7. **What does the audit trail capture?** Just the commands, or the agent's reasoning + the commands together? 8. **What's the blast radius of a single successful prompt injection?** Walk through the worst case explicitly with the vendor. If a vendor can't answer these clearly, the architecture isn't ready for production write access. ## Open questions in 2026 This is a young problem space. Several questions are not yet resolved: - **Standardization.** k8s-sigs/agent-sandbox is the leading candidate for a standard, but Knative Sandbox, container-level approaches, and microVM-based runtimes (Firecracker) are all in play. - **Multi-cloud isolation.** Sandboxing a Kubernetes pod is a solved problem. Sandboxing a process that calls `aws`, `az`, `gcloud` across cloud APIs from a single agent is harder, the credentials and trust boundaries change per provider. - **Approval UX at scale.** Engineers can't approve 200 actions per week. The right UI for batch approval, policy-based pre-approval, and rollback-only autonomy is still being figured out. Expect significant movement on all three through 2026 and into 2027. ## Aurora's approach in summary If you operate an AI SRE in production, the safety questions are non-negotiable. Aurora's answer is: pod-isolated execution by default, deterministic + LLM-judge guardrails before any non-trusted command, environment-variable allowlist that strips secrets, per-invocation cloud credentials via STS/Service Principal/short-lived tokens, and human approval for destructive write operations. The full architecture is open source under Apache 2.0, auditable in the [Aurora repository](https://github.com/Arvo-AI/aurora). Sandboxed execution is one layer of seven; the full pipeline, input rail, org policy, signatures, LLM judge, human gates, and output redaction, is mapped in our [AI agent guardrails deep dive](/blog/ai-agent-guardrails). For common failures the same read-only tools diagnose, see [how to fix CrashLoopBackOff](/blog/fix-crashloopbackoff-kubernetes), [how to fix OOMKilled](/runbooks/fix-oomkilled-kubernetes), and [how to fix ImagePullBackOff](/runbooks/fix-imagepullbackoff-kubernetes). For background on the agent and tool model, see our [complete guide to AI SRE](/blog/ai-sre-complete-guide), our [open-source AI SRE comparison](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt), or our explainer on [agentic incident management](/blog/what-is-agentic-incident-management). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## AI SRE in 2026: What It Is, Tested on 19 Kubernetes Faults URL: https://www.aurorasre.ai/blog/ai-sre-complete-guide Published: 2026-04-24 (updated 2026-10-21) Author: Ben Gervais, Engineer at Arvo AI > An AI SRE is an LLM agent that investigates alerts. We ran Aurora, HolmesGPT and K8sGPT on 19 injected faults: accuracy, time, cost and every miss. An AI SRE is a software agent that takes an alert, investigates the live system with read access to tools like `kubectl`, logs and metrics, and returns a root cause with the evidence behind it. It does the first hour of on-call triage; a person still decides what to change. In our lab, Aurora found the injected root cause in 34 of 38 runs (89%, median 154 seconds) and HolmesGPT in 31 of 38 (82%, median 45 seconds, graded by a judge model rather than a person), and both missed a second outage the alert did not mention. Everything below comes from that lab: 19 Kubernetes faults we injected on purpose, three agents, two runs each, every transcript public in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab). Arvo builds Aurora and wrote the scenarios, so read the numbers with that in mind. The live table is at [/benchmark](/benchmark). ## What does an AI SRE actually do during an incident? Here is one complete Aurora investigation, scenario 15 of the lab, run r1 on 2 October 2026. The fault: a second Deployment, `shop-api`, an old build missing its `DATABASE_URL` setting, was applied with the pod label `app=api`. The `api` Service selects on `app=api`, so it load-balances over three pods, one of which answers every real request with a 500 while passing its health check. Every pod shows `Running` and `1/1` ready. Aurora received only this text, the scenario's alert plus the lab's one-line instruction: > HighErrorRate [warning] namespace=shop service=frontend: 31% of requests to /api/ returned 5xx over the last 10 minutes (threshold 5%). Cluster: aurora-lab (kubeconfig). Investigate and find the root cause. The tool calls it made, in order (start times UTC, from `steps-full.json` in the [run folder](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/15-stray-pods-share-label/r1)). Indented lines are copied from the tool output; `[...]` marks lines we cut: ```text 20:29:58 list_incidents, grep_memories, get_connected_clusters 20:30:03 kubectl get pods -n shop -o wide 20:30:03 kubectl get events -n shop --sort-by=.lastTimestamp 20:30:15 kubectl get svc -n shop 20:30:15 kubectl logs frontend-76d98597b9-bb4z9 -n shop --tail=100 20:30:26 kubectl logs api-f64486c4c-r95bj -n shop --tail=100 20:30:26 kubectl logs api-f64486c4c-vftsb -n shop --tail=100 20:30:26 kubectl get configmap api-code -n shop -o yaml 20:30:43 kubectl get endpoints api -n shop -o wide NAME ENDPOINTS AGE api 10.244.1.37:8080,10.244.1.40:8080,10.244.2.48:8080 111s 20:30:43 kubectl get pods -n shop --show-labels [...] shop-api-69f6fc5964-jhzz8 1/1 Running 0 105s app.kubernetes.io/name=shop-api,app=api,pod-template-hash=69f6fc5964 [...] 20:30:43 kubectl logs shop-api-69f6fc5964-jhzz8 -n shop --tail=100 10.244.1.38 "GET /items HTTP/1.0" 500 - [...] KeyError: 'DATABASE_URL' 10.244.1.1 "GET /healthz HTTP/1.1" 200 - 20:31:02 kubectl get deployment shop-api -n shop -o yaml 20:31:02 kubectl get svc api -n shop -o yaml 20:31:02 kubectl get deployment api -n shop -o yaml 20:31:28 kubectl get secrets -n shop 20:31:28 kubectl get configmap shop-api -n shop -o yaml 20:32:03 kubectl get nodes -o wide 20:32:03 kubectl top pods -n shop 20:32:40 write_memory ``` The excerpt leaves out a few calls (a `get configmap` list, a `list_services` call and one failed jsonpath query that came back `Error from server (NotFound)` because of an unquoted space). The run made 24 tool calls and 36 model calls in total: 10 investigation steps, the report and the fix suggestions on Claude Sonnet 5, and 23 command-safety checks plus one summary on Claude Haiku 4.5. The turn happens at 20:30:43: three endpoints behind a Service whose Deployment runs two replicas, then a pod whose labels include `app=api` but whose name is not `api-`, then its logs. The report's root cause ends: > With 1 of the 3 load-balanced endpoints behind `api` failing 100% of the time, roughly one-third of frontend `/api/` requests fail, matching the alert's reported error rate [8, 13, 15]. It also listed what it ruled out (redis, node resources, nginx config) and what it did not check, including how the colliding Deployment got applied. Its first suggested next step was "Scale `shop-api` Deployment to zero replicas", rated low risk; the second, relabelling the Deployment, was rated high risk and came with an undo command. Aurora ran neither: its command policy was read-only, and a person would have to. **Cost and time:** 175 seconds by Aurora's own timer and $0.59 of model spend for all 36 calls. The second run of the same scenario took 118 seconds and $0.35. HolmesGPT also got this one right in both runs (97 and 162 seconds). K8sGPT printed "No problems detected", which is accurate for what it checks: every object was healthy. The manual version of this diagnosis is in the runbook [Service selects the wrong pods](/runbooks/fix-service-selects-wrong-pods-label-overlap-kubernetes). That is the job in practice. Some products add scheduled checks on top (HolmesGPT's operator mode, Azure SRE Agent's scheduled tasks), but the part you can grade is this loop: read an alert, form a guess, run commands to test it, and write down what it found and what it did not look at. ## How accurate are AI SRE agents today? We ran Aurora, HolmesGPT 0.43.0 and K8sGPT 0.4.40 on the same 19 faults on a kind cluster (Kubernetes v1.37.0, 3 nodes), two runs per scenario, 38 runs per tool. A judge model (Claude Opus 5.5) graded each answer against the injected fault: correct means every required finding is present and none of the listed wrong answers is asserted. | | Aurora | HolmesGPT 0.43.0 | K8sGPT 0.4.40 | |---|---|---|---| | Model | Claude Sonnet 5 (Bedrock); Haiku 4.5 for command-safety checks | Claude Sonnet 5 (Bedrock) | Claude Sonnet 4.5 (Bedrock; K8sGPT sends a temperature setting Bedrock rejects for Sonnet 5) | | Input | alert text, kubectl behind its default read-only command policy | alert text, `holmes ask`, default toolsets | no alert, `k8sgpt analyze --explain --namespace shop` | | Correct | 34 of 38 (89%) | 31 of 38 (82%) | not scored as a peer (see below) | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 wrong, 12 "No problems detected" | | Median time | 153.5 s | 45 s | 12.5 s (two analyze passes) | | Median model cost | $0.35 | not captured | not captured | | Who checked the grades | a person at Arvo, all 38 | judge model, second-model review, blinded regrade | judge model, second-model review, blinded regrade | What the table supports: - **Aurora and HolmesGPT are roughly even on accuracy.** Three more correct runs out of 38 is not a significant gap: a two-sided Fisher exact test on the totals, the one /benchmark computes, gives p = 0.52. That test also compares person-read Aurora grades with judge-only HolmesGPT grades. - **HolmesGPT is about 3x faster at the median**, 45 seconds against 153.5. Part of Aurora's time is contention from its own background jobs in the lab stack, but not all of it. - **HolmesGPT's and K8sGPT's 76 grades have not been checked by a person.** A second model reviewed them and agreed with all 76. A third pass graded them from scratch, agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels. A blinded regrade, with agent and tool names removed, matched the published verdict for 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs; every change was a downgrade. - **K8sGPT does a different job.** `analyze` scans a namespace and explains objects its analyzers flag. It never sees the alert, so a criterion built for alert-driven investigation is not a fair test of it. Its 12 "No problems detected" runs are both runs of six scenarios where its analyzers flagged nothing, scenario 15 among them. The flip side shows up in scenario 12 below: because it lists every problem it detects, it flagged the outage both investigating agents missed. The prompts were not identical. Aurora's alert text ended with "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause."; HolmesGPT's ended with "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." All three got the same cluster admin kubeconfig, each behind its own guard: Aurora's `observability_only` command policy, HolmesGPT's allow-list of known commands, and K8sGPT's read-only `analyze`. No transcript shows a mutating command running. ## Where do AI SRE agents fail? The misses say more than the score. All of them are in the repo with full transcripts. **They stop at the alerted fault.** Scenario 12 has two faults: a worker OOM, which the alert names, and a frontend Service selecting no pods, which it does not. Aurora and HolmesGPT found the OOM in all four runs and missed the second outage in all four. Aurora never ran `get svc` or `get endpoints`, and neither agent's answer mentions the frontend Service. K8sGPT, which scans everything, listed "Service has no endpoints, expected label tier=web" for `shop/frontend` in both runs, though it never named the BATCH_SIZE change behind the OOM. Aurora then wrote "Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected", with pod status as its only evidence. Treat any "everything else is healthy" line as unverified unless the agent checked traffic. The runbook for this case: [a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). **They settle on a plausible story.** In scenario 19 one of two frontend pods had loaded an edited ConfigMap and the other had not. HolmesGPT's first run concluded "All data points to a transient startup race, now resolved." and was graded wrong. Aurora's first run found the bad upstream (`api:8000`) in the ConfigMap but did not work out that only the recreated pod had loaded it. It tried to read the live config with `kubectl exec`, its read-only policy blocked that, and it left the question open. The clue was in its own `get pods` output: the failing pod was 9 seconds younger. Both second runs were graded correct, though the blinded regrade marked HolmesGPT's second run partial for getting the mechanism wrong. Runbook: [ConfigMap change not applied to all pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes). **They skip the red herring.** Scenario 11 puts noisy but harmless errors in the worker's logs next to the real fault, a wrong Service `targetPort`. Both agents found the port. In three of the four runs the agent never ruled out the worker's errors (two never mentioned the worker; one only called it "unaffected"), so those runs were graded partial. **They add small false details.** A second-model review found claims the evidence does not support in 15 of Aurora's 18 reports on the harder scenarios (11 to 19), none of them in the root cause: "both cluster nodes" when there are three, an image "pulled successfully" with no Pulled event, a trend drawn from one sample. **Read-only policies cost information.** Aurora's default policy blocked the read-only `kubectl rollout history` in 26 of 38 runs (27 calls), with "Command blocked by organization policy: Mutating kubectl operations", because the deny rule matched the whole `rollout` verb. In 19 of those 26 runs Aurora queried ReplicaSets instead, which works, but "what changed" is the first question in most incidents. ## Are AI SRE agents production-ready in 2026? For read-only investigation with a person reading the answer: yes, on faults like these. Two independent agents got the full root cause in over 80% of runs. For unattended action: no. Both investigating agents missed an unalerted outage that a plain K8sGPT scan listed, and one declared an incident "resolved" that was not. An agent whose suggested fix is right 34 times in 38 (Aurora's rate on its "Suggested Next Steps" list) is useful and is also wrong often enough that a person must approve each change. The lab is a small app on a 3-node kind cluster with no real traffic and no history. Production is noisier, which is harder for an agent, and it has dashboards and runbooks the agent can use. These numbers say what these tools can do on Kubernetes faults with known causes; they do not predict your mean time to recovery. ## How do you pilot an AI SRE without disrupting production? Run it the way the lab does: read-only access, real alerts, and a grade for every answer. 1. **Give it read-only credentials first.** In Kubernetes, the built-in `view` ClusterRole "allows read-only access to see most objects in a namespace" and does not allow viewing Secrets ([Kubernetes RBAC docs](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles)). Expect gaps: Aurora's trace above lists Secrets, which `view` would refuse, and Aurora's own default policy blocked `exec` and `rollout history` in our runs. Decide which gaps you accept before the pilot, not during an incident. 2. **Send it alerts that already fire.** One service, one alert source. The agent investigates in parallel with the on-call engineer and changes nothing. 3. **Grade every answer against what the human found.** Correct, partial or wrong, plus whether it claimed anything false. Write the grade down the same day. 4. **Watch the background cost, not only the per-investigation cost.** In our lab, Aurora's discovery job, which runs once per cluster connection, cost about $1 a run: $30.38 that day (30 completed runs plus a few cut short), more than twice the $14.53 for all 38 graded investigations. 5. **Allow suggested changes only behind approval.** Aurora suggests fixes and can open a pull request, but a person approves and merges; it does not auto-remediate. Check what your tool does by default. Azure SRE Agent, for example, has Review and Autonomous run modes. The agent-level setting defaults to Review, but the run-modes page lists Autonomous as the default for incident response plans and scheduled tasks, and Review only gates Azure infrastructure operations, so set it per plan ([overview](https://learn.microsoft.com/en-us/azure/sre-agent/overview), [run modes](https://learn.microsoft.com/en-us/azure/sre-agent/run-modes)). How long a pilot takes depends on how many alerts you can grade, not on calendar time. One number from our lab to calibrate: 38 graded runs per agent were not enough to tell two agents three points apart (p = 0.52, above). If you are choosing between tools, plan for more answers than that, or accept that only large gaps will show. Microsoft's run-modes page suggests watching Review-mode recommendations for two to four weeks before switching the triggers you consistently approve to Autonomous. ## How do you evaluate an AI SRE platform? Break something on purpose and score the answer. A known fault is the only way to know whether a root cause is right; real incidents rarely come with ground truth. The lab is built for this and is Apache 2.0: ```bash git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab ./lab.sh up && ./lab.sh break 03-bad-image-tag ./runner/h2h.sh 03-bad-image-tag runs/mine ``` `h2h.sh` runs HolmesGPT and K8sGPT against the broken cluster and saves their output; it expects both installed and AWS Bedrock access for the model. Each scenario folder holds the fault (`break.sh`), proof it is visible (`check.sh`) and a `scenario.yaml` with the alert, the findings a correct answer must contain and the wrong answers it must not assert. To put another agent on the public board, open an issue with a script that takes a kubeconfig and an alert and prints a root cause; we run it on all 19 faults and publish the result. What we learned about grading while building it: - **Grade everything the tool shows.** Our first report said 28 of 38 Aurora runs proposed no fix. Wrong: the grader read only the RCA text, and Aurora puts fixes in a separate list. - **Run each scenario at least twice.** Scenario 19 split correct and wrong between runs for HolmesGPT and correct and partial for Aurora. - **Include faults the alert does not name** and noise next to the real fault. Those two scenarios (11 and 12) account for three of Aurora's four partial runs and four of HolmesGPT's six. - **Blind the grader.** Remove agent and tool names before a model grades. It moved 3 of 114 verdicts here, all down. - **Record time and cost per run**, including jobs that run outside the investigation. A longer checklist covering trust, data handling and total cost is in [how to evaluate an AI SRE platform](/blog/how-to-evaluate-ai-sre-platform). The accuracy question on its own is covered in [AI SRE accuracy benchmarks](/blog/ai-sre-accuracy-benchmarks). ## Can AI SRE platforms run on-prem or with bring-your-own-model? Some can. What each vendor says, checked on 21 October 2026: | Tool | Where it runs | Model choice | Access stance | |---|---|---|---| | [Aurora](https://github.com/Arvo-AI/aurora) | Self-hosted: Docker Compose, Helm chart, air-tight bundle for restricted networks | OpenAI, Anthropic, Gemini, Vertex AI, AWS Bedrock, OpenRouter, Ollama | Org-level command policies; lab ran the default read-only policy | | [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) | CLI, Docker HTTP server, Helm chart, operator; CNCF sandbox, Apache 2.0 | [Providers](https://holmesgpt.dev/latest/ai-providers/) include Anthropic, Bedrock, OpenAI, Ollama and OpenAI-compatible endpoints | "By design, HolmesGPT has read-only access and respects RBAC permissions." | | [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) | CLI or in-cluster operator, Apache 2.0 | OpenAI, Azure OpenAI, Bedrock, Gemini, Ollama, LocalAI, LiteLLM and others | Scans and explains; Kubernetes analyzers plus optional integrations such as AWS | | [Traversal](https://www.traversal.com/) | Vendor platform, or in your own cloud: "supports BYOC/BYOM (bring your own cloud/model)" | Your own model (BYOM) | "deploys read-only by default" | | [Azure SRE Agent](https://learn.microsoft.com/en-us/azure/sre-agent/overview) | Azure service, with VNet integration | Pricing page: "Rates vary by selected model" | Per-tool allow, ask or deny; Review or Autonomous run modes | | [Cleric](https://cleric.ai/) | Hosted, US or EU region chosen at setup; other regions on Enterprise | Vendor-chosen: Anthropic, Google Gemini and OpenAI enterprise APIs ([security page](https://docs.cleric.ai/security/security-data-privacy)) | "Read-only by default." | Two cautions. First, self-hosting the agent does not keep data in your network if the model is a hosted API: prompts and tool output go to the model provider. Only a local model (Ollama, LocalAI, an in-VPC gateway) keeps it inside. Second, all our lab results used Claude models on Bedrock. We have not tested any agent on a local model, and accuracy will not carry over automatically. The architecture options are covered in [self-hosted AI SRE](/blog/self-hosted-ai-sre). ## What is the ROI of an AI SRE for a large engineering organization? The per-investigation model cost is small next to an engineer's time: Aurora's median was $0.35 per run in our lab, with a range of $0.19 to $0.84, and $14.53 for all 38. Commercial prices include the product, not only the model, and some are published: Cleric lists "$5 to triage and address an alert" (10 credits at $0.50), with plans from $700 a month ([pricing](https://cleric.ai/pricing.md)). Azure SRE Agent bills an always-on charge of 4 Azure Agent Units an hour per agent plus token-based usage that varies by model; the page currently advertises a trial with no always-on charges ([pricing](https://azure.microsoft.com/en-us/pricing/details/sre-agent/)). The return depends on numbers only you have: how many alerts get investigated, how often the answer is good enough to skip manual triage (82 to 89% correct on our faults, lower on yours until you measure it), and how many engineer minutes each good answer saves. We did not time human engineers in the lab, so we have no minutes-saved figure to give you. Measure it in the pilot. Price comparisons are in [AI SRE and incident tool pricing](/blog/ai-sre-incident-tool-pricing-2026) and [what an AI investigation costs](/blog/ai-investigation-cost). ## AI SRE vs AIOps vs traditional SRE: what is the difference? | | Traditional SRE | AIOps | AI SRE | |---|---|---|---| | Who investigates | An engineer | Nobody; it groups and ranks alerts | An LLM agent, reviewed by an engineer | | Input | Alerts, dashboards, experience | Telemetry streams and alert floods | One alert plus read access to tools | | Output | A diagnosis and a fix | Correlated incidents, anomalies | A root cause with evidence, a list of what it did not check, suggested fixes | | Where it breaks | Slow at 3 a.m., knowledge in few heads | Correlation is not cause | Stops at the alerted fault, settles on plausible stories (above) | AIOps products correlate and deduplicate alerts and flag anomalies; an AI SRE takes one alert and investigates it by running commands. Many teams run both. "Agentic incident response" usually names the same thing as AI SRE, sometimes with the agent also running remediation ([one alert traced end to end, with the autonomy levels vendors offer](/blog/what-is-agentic-incident-management)). An AI SRE does not replace the SRE: in our lab both investigating agents missed a broken Service (Aurora reported the frontend workloads as unaffected), and Aurora's fix suggestions fell short in 4 of 38 runs, often enough to need a person. What it replaces is the first round of `kubectl get`, `describe` and `logs`. The longer comparison is in [AI SRE vs AIOps](/blog/ai-sre-vs-aiops). ## What are the best AI SRE tools in 2026? There is no public accuracy ranking across commercial and open-source agents, and we will not invent one. What we can say, as of 21 October 2026: | Tool | Type | Tested in our lab | |---|---|---| | Aurora | Open source (Apache 2.0), self-hosted | Yes: 34 of 38 correct, person-read grades | | HolmesGPT | Open source (Apache 2.0), CNCF sandbox | Yes: 31 of 38 correct, judge-model grades; about 3x faster | | K8sGPT | Open source (Apache 2.0), cluster scanner | Yes, as a scanner rather than an investigator | | Azure SRE Agent | Commercial, Azure service | No | | Traversal | Commercial, vendor platform or your own cloud | No | | Cleric | Commercial, hosted | No | | Resolve.ai, Datadog Bits AI SRE, incident.io, Rootly | Commercial | No | If you run on Kubernetes and want something you can try in an afternoon, HolmesGPT and Aurora are the two that did the investigation job on our faults. For a commercial product, ask the vendor to run your trial on faults you inject yourself (the lab above works for that) and grade the answers the same way. The head-to-head with features and setup is in [Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt), the two non-Aurora tools are compared in [HolmesGPT vs K8sGPT](/blog/holmesgpt-vs-k8sgpt), and the open-source field beyond these three is in [open source SRE agents in 2026](/blog/top-ai-sre-tools-2026). To try a diagnosis without installing anything, paste an error into [/diagnose](/diagnose). ## Sources and method Lab runs: Aurora 2 October 2026 (commit 87d22baf), HolmesGPT 0.43.0 and K8sGPT 0.4.40 on 15 and 16 October 2026, same kind cluster and faults, all in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab), table in [results.json](https://raw.githubusercontent.com/Arvo-AI/aurora-lab/main/benchmark/results.json). Full Aurora method and every miss: [lab report v1](/blog/aurora-lab-report-kubernetes-19-failures). Vendor claims were checked against the linked pages on 21 October 2026. Aurora is open source on [GitHub](https://github.com/Arvo-AI/aurora); to see it on your own alerts, [book a demo](https://cal.com/arvo-ai/demo). --- ## Opsgenie Pricing, End of Life Date and Migration (Oct 2026) URL: https://www.aurorasre.ai/blog/opsgenie-complete-guide-2026 Published: 2026-04-21 (updated 2026-10-21) Author: Arvo AI Team, Engineering > Opsgenie pricing by tier ($9.45 to $31.90 per user), the April 5, 2027 shutdown, what JSM and Compass cost instead, and alternatives. Checked Oct 21, 2026. Opsgenie's list prices, read on Atlassian's [Opsgenie pricing page](https://www.atlassian.com/software/opsgenie/pricing) on October 21, 2026, are Free (up to 5 users), Essentials $9.45, Standard $19.95 and Enterprise $31.90 per user per month billed annually ($11.55, $24.15 and $38.50 billed monthly), but you can no longer buy it: sales ended on June 4, 2025. Opsgenie reaches end of support on April 5, 2027, when Atlassian shuts it off and deletes any data you have not migrated. Atlassian's migration path is Jira Service Management (JSM), where Opsgenie Standard customers are pointed at JSM Premium ($55.28 per agent per month, the monthly list price at the page's default 75 agents), about 2.5 times what 100 Opsgenie Standard users pay; Compass is the developer-focused alternative destination at $7.67 or $23.96 per user per month. Outside Atlassian, PagerDuty, incident.io, Rootly, ilert, Grafana Cloud IRM, FireHydrant, xMatters and SolarWinds (Squadcast) all publish per-user prices, listed below. Every price and date on this page was read on the vendor's own site on October 21, 2026 and is linked where it appears. Where we did arithmetic on top of a vendor's numbers, the page says so. Arvo AI builds Aurora, an open source incident investigation tool that connects to Opsgenie and JSM; it is not a paging or on-call tool and is not one of the alternatives in the price table. ## How much does Opsgenie cost? | Plan | Annual billing (per user per month) | Monthly billing (per user per month) | SMS and voice (US and Canada) | Routing rules | |---|---|---|---|---| | Free | $0, up to 5 users | $0 | 100 SMS account wide, no voice | 1 | | Essentials | $9.45 | $11.55 | users x 100, then $0.10 each | 1 | | Standard | $19.95 | $24.15 | Unlimited | 100 | | Enterprise | $31.90 | $38.50 | Unlimited | 100 | Source: [atlassian.com/software/opsgenie/pricing](https://www.atlassian.com/software/opsgenie/pricing), which now opens with "New signups and sales of Opsgenie have ended and Opsgenie alert, on-call, and incident response features are available in Jira Service Management." Essentials is capped at "100 incidents per month (account-wide)"; Standard and Enterprise are unlimited, and the Free plan has no incident management. International SMS and voice on Essentials is users x 25, then $0.35 each. Incoming call routing on Standard and Enterprise includes one number ($10 for each additional number) and 100 US and Canada minutes, then $0.10 a minute, or $0.35 a minute internationally. The headline prices are for the first 25 users. The page's price calculator uses progressive tiers, so each additional block of users is cheaper. For example, the monthly Standard rate is $24.15 for users 1 to 25, $22.05 for users 26 to 50 and $21 for users 51 to 100. Totals for a 100-user account, our arithmetic from those tiers: | 100 users | Billed monthly | Billed annually | |---|---|---| | Essentials | $1,023.75 a month | $9,765 a year | | Standard | $2,205 a month | $21,420 a year | | Enterprise | $3,162.50 a month | $31,020 a year | Above 10,000 users Atlassian asks you to contact sales. These prices only matter for renewals now (see below). ## When is Opsgenie end of life? Atlassian announced the change on March 3, 2025 in [The Evolution of IT Operations and Opsgenie](https://www.atlassian.com/blog/announcements/evolution-of-it-operations). The dates: | Milestone | Date | What Atlassian says happens | |---|---|---| | Announcement | March 3, 2025 | Migration to JSM opens | | End of sale | June 4, 2025 | "you'll no longer be able to purchase or sign up for Opsgenie"; no edition upgrades or downgrades | | Renewals | Until end of support | "renewals will still be allowed ... so long as the subscription term ends before the end of support date" | | Latest migration date | April 5, 2027 | A migration "must be set at least seven days in the future and can be scheduled up until April 5, 2027" ([source](https://support.atlassian.com/opsgenie/docs/what-to-expect-from-opsgenie-migration/)) | | End of support | April 5, 2027 | "access to Opsgenie will be shut off ... Any unmigrated data in Opsgenie will also be deleted" | As of October 21, 2026 that leaves 166 days. Two details change your real deadline: - **The 120-day clock.** Once your data has migrated, "Opsgenie will also shut down automatically 120 days after your migration date if you don't turn it of[f] manually," and turning it off "CANNOT be undone" ([Atlassian](https://support.atlassian.com/opsgenie/docs/what-happens-when-opsgenie-is-turned-off/)). Your Opsgenie may therefore go dark before April 2027. The JSM guide adds that the turn-off date "may change depending on your subscription, so it's best to refer to your product UI" ([source](https://support.atlassian.com/jira-service-management-cloud/docs/start-shifting-from-opsgenie-to-jira-service-management/)). - **The REST API.** "Opsgenie REST APIs will continue to work until April 5, 2027," but Atlassian recommends moving to the [JSM Operations REST API](https://support.atlassian.com/jira-service-management-cloud/docs/start-shifting-from-opsgenie-to-jira-service-management/) before your Opsgenie is turned off. Every script, Terraform module and monitoring tool that calls api.opsgenie.com needs a new endpoint; Atlassian has a separate guide for moving from the [Opsgenie Terraform provider to the Atlassian Operations provider](https://support.atlassian.com/opsgenie/docs/migrate-from-terraforms-opsgenie-provider-to-atlassian-operations-provider/). ## What happens to my data when Opsgenie shuts down? From [What happens when Opsgenie is turned off](https://support.atlassian.com/opsgenie/docs/what-happens-when-opsgenie-is-turned-off/): users can no longer log in or retrieve any data, the Opsgenie mobile app is deactivated, remaining Opsgenie-based integrations stop working, and features left behind (deprecated notification policies, incident rules, Opsgenie actions, older integrations) become permanently unavailable. Only migrated data survives in JSM: alerts, on-call schedules, escalation policies, incident history, audit logs, and migrated configurations and integrations. ## What are the Opsgenie migration paths? ### Jira Service Management JSM is the destination Atlassian's tooling is built around. The in-app tool (Settings > Plan your move) recommends a plan based on your Opsgenie plan ([source](https://support.atlassian.com/opsgenie/docs/what-are-my-options-for-migrating-from-opsgenie/)): | Current Opsgenie plan | Atlassian's recommended JSM plan | JSM list price ([pricing](https://www.atlassian.com/software/jira/service-management/pricing)) | |---|---|---| | Free | JSM Free (or any paid plan) | $0, up to 3 agents | | Essentials | JSM Standard | $21.50 per agent per month | | Standard | JSM Premium "if no enterprise features are used" | $55.28 per agent per month | | Enterprise | JSM Premium or Enterprise | Premium as above; Enterprise is annual only, contact sales | The JSM prices are the monthly-billing figures the pricing page shows at its default team size of 75 agents. They are averages: the price data behind the page's calculator charges Premium at $61.60 for each of the first 15 agents and $53.70 for agents 16 to 100 ((15 x 61.60 + 60 x 53.70) / 75 = $55.28), and Standard at $26.90 then $20.15 (which averages $21.50 at 75). Annual billing saves "up to 17%" and is sold in agent bands; 51 to 100 Premium agents is $55,000 a year. JSM is now sold as part of Atlassian's Service Collection, which bundles Rovo AI features, Assets and customer service apps into the same per-agent price. **What it costs compared with Opsgenie.** Our arithmetic, assuming every one of 100 Opsgenie Standard users becomes a JSM agent, at list prices read on October 21, 2026: | 100 users or agents | Billed monthly | Billed annually | |---|---|---| | Opsgenie Standard (25 x $24.15 + 25 x $22.05 + 50 x $21.00 monthly) | $2,205 a month | $21,420 a year | | JSM Premium (15 x $61.60 + 85 x $53.70 monthly; 51 to 100 agent band annually) | $5,488.50 a month | $55,000 a year | | Ratio | 2.5x | 2.6x | So the plan Atlassian recommends costs about two and a half times as much at list price, before any negotiated discount or any credit for unused Opsgenie time. Teams that only need some people to respond can license fewer agents. In return, JSM Premium includes ITSM, asset management and AI alert grouping that Opsgenie does not. Atlassian also notes that the trial "may be extended to make up for any unused time left on your Opsgenie plan" ([billing guide](https://support.atlassian.com/opsgenie/docs/review-and-approve-billing-for-opsgenie-migration/)). **Why the plan level matters.** Atlassian's [feature table](https://support.atlassian.com/opsgenie/docs/feature-changes-and-deprecations-in-jira-service-management/) puts several Opsgenie Standard features behind JSM Premium or Enterprise: | Feature | JSM Free / Standard | JSM Premium / Enterprise | |---|---|---| | Alerts, incoming integrations, schedules, who is on call | Yes | Yes | | Outbound and bidirectional ITSM integrations | No | Yes | | Heartbeats | No | Yes | | Voice notifications | No | Yes | | Incoming call routing | No | Yes | | Global ChatOps integrations, incident ChatOps | No | Yes | | Alert policies, team custom roles, stakeholders | No | Yes | | Advanced incident management, analytics | No (basic incidents only; notification analytics from Standard) | Yes | **What does not carry over at all**, per the same page and the [JSM shifting guide](https://support.atlassian.com/jira-service-management-cloud/docs/start-shifting-from-opsgenie-to-jira-service-management/): - Incident Command Center, conference rooms, incident templates, response roles and email templates (replaced by Incident Configurations). - Opsgenie actions and incident rules: they "won't be automatically moved" and are rebuilt as Jira Automation. - Chat integrations must be re-authenticated by hand. - Custom roles not supported in JSM fall back to the User role; heartbeats with no assigned team are not moved. - Opsgenie-provided phone numbers for call routing must be ported to your own Twilio account before migration ([source](https://support.atlassian.com/opsgenie/docs/incoming-call-routing-after-migration/)). - The Opsgenie mobile app is replaced by the Jira mobile app; notification rules must be recreated there and insistent notifications are not available ([source](https://support.atlassian.com/opsgenie/docs/opsgenie-mobile-app-after-migration/)). - MSP sub-accounts become free accounts that need their own migration. ### Compass Atlassian's docs describe migrating "from Opsgenie to Jira Service Management or Compass" with one shared Operations layer: alerts, schedules, escalation policies, routing rules and integrations stay in sync across both, and users only get licenses for the product you pick as the primary destination ([source](https://support.atlassian.com/opsgenie/docs/managing-operations-in-compass-and-jira-service-management-at-the-same-time/)). Compass ties alerts to software components instead of ITSM tickets. The plan recommendation table in the migration docs lists only JSM plans; to land in Compass you pick it as the primary destination product when you set up the migration. [Compass pricing](https://www.atlassian.com/software/compass/pricing), monthly only ("Subscriptions to Compass are currently only available as monthly payments"): | Compass plan | Price | Operations features | |---|---|---| | Free | $0, up to 3 full users | Component catalog; no Operations listed | | Standard | $7.67 per user per month | On-call schedules and alerts from more than 150 integrations | | Premium | $23.96 per user per month | Adds advanced integrations and escalation policies, alert and notification policies, voice notifications, heartbeat monitoring | The page shows these at its default team size of 300 users. ## Should I move to JSM or find an alternative to Opsgenie? Stay with Atlassian if you already run JSM or Jira and want incidents tied to tickets, changes and assets; that is what the Premium price buys. If your Opsgenie use was paging, schedules and escalations, and you were on Essentials or Standard, the JSM plan with feature parity costs more than you pay today, and the dedicated on-call tools below are priced closer to Opsgenie. Compass is the cheaper Atlassian option for engineering teams, as long as the features you need are on its Premium plan. ## What are the alternatives to Opsgenie, and what do they cost? | Product | Free plan | Paid on-call pricing (list, checked 2026-10-21) | Notes | |---|---|---|---| | [PagerDuty](https://www.pagerduty.com/pricing/) | 5 users, 1 schedule, 1 escalation policy | Professional $21 per user per month annual, $25 monthly | AIOps, PagerDuty Advance and Live Call Routing are separate add-ons; PD Reliability Platform from $2,800 a year (Starter), $12,000 (Essential), $48,000 (Plus) | | [incident.io](https://incident.io/pricing) | Basic: single-team on-call | Team $15 per user per month annual ($19 monthly) plus $10 for on-call; Pro $25 plus $20 for on-call | On-call alone is $20 per user per month | | [Rootly](https://rootly.com/pricing) | No | On-Call $20 per user per month; Incident Response another $20 | AI SRE is priced on request | | [ilert](https://www.ilert.com/pricing) | Up to 5 users | Pro $19, Scale $39 per user per month billed yearly ($24 and $49 monthly), min. 3 users; Enterprise from $49 annual | Inbound call routing add-on $10 per user per month yearly ($12.50 monthly) | | [Grafana Cloud IRM](https://grafana.com/pricing/) | 3 active IRM users | Pro: $19 a month platform fee incl. 3 active users, then $20 per active IRM user | [Billed](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/understand-your-invoice/irm-invoice/) per unique user who is on a schedule or escalation chain, is paged or pages someone, or edits an incident that month | | [FireHydrant](https://firehydrant.com/pricing/) | Up to 10 responders | Pro $25 per responder per month, billed annually | Signals on-call is "charged based on the number of alerts you send" | | [xMatters](https://www.xmatters.com/pricing) | Up to 5 users | Starter $15 per user per month (up to 25 users); Base $39 | | | [SolarWinds Incident Response (Squadcast)](https://www.solarwinds.com/it-incident-response-software/pricing) | 14-day trial | Pro $15, Premium $24 per user per month, billed annually | squadcast.com/pricing now redirects to this SolarWinds page | Status notes, each from the vendor's own page: Grafana OnCall OSS "entered maintenance mode and was archived on 2026-03-24" and points users to Grafana Cloud IRM ([GitHub](https://github.com/grafana/oncall)). Freshworks [signed an agreement to acquire FireHydrant](https://www.freshworks.com/press-releases/freshworks-to-deepen-its-it-service-and-operations-portfolio-with-acquisition-of-firehydrants-ai-native-incident-management-and-reliability-platform/) on December 15, 2025, "expected to close in Freshworks' first fiscal quarter of 2026"; FireHydrant now has a [product page on freshworks.com](https://www.freshworks.com/freshservice/firehydrant/), but we found no closing announcement on Freshworks' newsroom. For the full PagerDuty versus JSM trade-off, see [PagerDuty vs Jira Service Management](/blog/pagerduty-vs-jira-service-management); for AI pricing across these vendors, see [AI SRE and incident tool pricing](/blog/ai-sre-incident-tool-pricing-2026). ### Which Opsgenie alternatives can import our existing schedules and escalation policies? Three of the vendors above advertise Opsgenie migration on their pricing pages; we have not tested any of them. FireHydrant's page carries a banner for "Signals Migrator: Migrate your PagerDuty or Opsgenie config in minutes". incident.io's on-call plan lists "Best-in-class migration support", and Rootly promises "A tailored and proven migration that avoids any downtime". Ask each vendor for the exact objects its importer moves (schedules, rotations, overrides, escalation policies, routing rules, integrations) and try it on a copy of your config before the 120-day clock starts. Atlassian's own migration moves alerts, schedules, escalation policies and incident history into JSM automatically. What none of these tables tell you is how well each product investigates an alert. That is a separate category, covered in [AI-powered incident investigation tools](/blog/best-ai-powered-incident-investigation-tools). ## Where Aurora fits after the page Aurora is not a paging tool. It has no on-call schedules, escalations or notifications, so it does not replace Opsgenie, JSM or anything in the table above. It is an open source investigator that takes the alert after the page fires. Its [Opsgenie and JSM connector](https://github.com/Arvo-AI/aurora/blob/main/website/docs/integrations/opsgenie-jsm.md) takes either an Opsgenie API key (GenieKey, US or EU region) or an Atlassian account email and API token for JSM Operations. When an alert Create event arrives on its webhook, it opens an incident and starts a root cause investigation ([tasks.py](https://github.com/Arvo-AI/aurora/blob/main/server/routes/opsgenie/tasks.py) skips every other action), and it answers questions like "who is on call right now?" from the same API. The plan level decides whether that works. Atlassian's [Webhook integration doc](https://support.atlassian.com/jira-service-management-cloud/docs/integrate-with-webhook/) says "Webhook is an outgoing integration which means you should be on a Premium or Enterprise plan to use it." So on JSM Standard, Aurora's docs tell users it can only query alerts on demand from chat: nothing is pushed, so no investigation starts on its own. The same limit applies to any tool you want Opsgenie alerts pushed to after the move. If you plan to land on JSM Standard and keep a downstream webhook consumer, check this before you migrate. How well it investigates is published, with its limits, on [/benchmark](/benchmark): Aurora found the root cause in 34 of 38 runs (4 partial) on 19 injected Kubernetes faults, with every grade read by a person, at a median of 153.5 seconds and $0.35 per investigation. HolmesGPT was 3.4 times faster at the median (45 seconds). The two agents' prompts differed slightly (both are quoted on /benchmark), and Arvo AI wrote the scenarios and builds Aurora; the [lab is open](https://github.com/Arvo-AI/aurora-lab) so you can rerun it. For what one of those investigations looks like, see the runbook on [an alert that covers only one of two faults](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). Aurora is on [GitHub](https://github.com/Arvo-AI/aurora) under Apache 2.0, or [book a demo](https://cal.com/arvo-ai/demo). Prices and dates verified on Atlassian's and each vendor's own pages on October 21, 2026. List prices change; check the linked page before you sign. --- ## AIOps RCA Tools 2026: Pricing, Open Source, 3 Tested URL: https://www.aurorasre.ai/blog/top-10-aiops-platforms-free-root-cause-analysis-2026 Published: 2026-04-10 (updated 2026-10-21) Author: Arvo AI Team, Engineering > How 14 AIOps tools license root cause analysis, prices checked Oct 21, 2026, and the open source ones measured on 19 Kubernetes faults, every run published. Automated root cause analysis is sold in four ways in October 2026. It comes bundled into an observability platform's host or usage price (Dynatrace Full-Stack Monitoring, Datadog Watchdog RCA with APM, Grafana Sift). The newer agentic investigations are metered per use, in credits, AI units, tokens or compute units; where a vendor publishes average consumption, our arithmetic puts it at about $3 (Dynatrace: 30 AI units at $0.10) to $6.50 or $8.45 (Datadog: about 6.5 credits at $1 or $1.30) per investigation. Enterprise AIOps (BigPanda, ScienceLogic Skylar Advisor, Splunk ITSI, Traversal) is quote-only, on annual commitments. Open source agents (Aurora, HolmesGPT, K8sGPT) cost nothing to license; you pay the model provider and run them yourself. Price tells you little about whether an RCA is right. Of the commercial tools below, only Coroot publishes a benchmark a buyer can rerun (five OpenTelemetry Demo failures); Traversal publishes one on 25 private incidents, and the rest publish none we found. So we measured the open source agents. On the same 19 injected Kubernetes faults, two runs each, Aurora was graded correct in 34 of 38 runs, partial in 4 and wrong in 0, with every grade read by a person. HolmesGPT was correct in 31 of 38 on judge-model grades that no person has checked yet, and it was about 3x faster. Arvo builds Aurora and wrote the scenarios. All runs, partial and wrong answers included, are at [/benchmark](/benchmark) and in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab). ## Which observability platforms have RCA built in? Every price and feature name below was read on the vendor's own pricing page or docs on 2026-10-21. Several names changed in 2026: Dynatrace's Davis AI docs now redirect to **Dynatrace Intelligence**, Datadog's Bits AI SRE pages redirect to **Bits Investigation**, and New Relic's SRE Agent is now **Autopilot**. | Platform | RCA feature, as the vendor names it | How it finds a cause | Built in or extra | |---|---|---|---| | [Dynatrace](https://docs.dynatrace.com/docs/dynatrace-intelligence) | Dynatrace Intelligence; Agentic Operations | "deterministic, causation-based analysis" with topology and dependency context; agentic investigations on top | Classic RCA is listed in Full-Stack Monitoring; agentic investigations are metered | | [Datadog](https://docs.datadoghq.com/watchdog/rca/) | Watchdog RCA; [Bits Investigation](https://docs.datadoghq.com/bits_ai/bits_investigation/) | Watchdog finds anomalies and causal links in APM data; Bits Investigation is "an autonomous AI agent" | Watchdog RCA "requires the use of APM"; Bits Investigation draws AI credits | | [New Relic](https://docs.newrelic.com/docs/agentic-ai/autopilot/overview/) | Autopilot (formerly SRE Agent), New Relic AI | An AI agent | Needs Pro or Enterprise plus the Advanced Compute add-on, or a legacy CCU plan | | [Grafana Cloud](https://grafana.com/docs/grafana-cloud/ai-tools/machine-learning/sift/) | Sift; Assistant Investigations; Knowledge Graph RCA workbench | Sift runs diagnostic checks; Investigations "forms hypotheses, chases down leads across your metrics, logs, traces, and profiles" | Sift is free; Investigations consume Assistant tokens; the knowledge graph has "no separate charge", though its entities "typically add 5% to 10% more active series" | | [Splunk Observability Cloud](https://www.splunk.com/en_us/products/ai-sre.html) | AI SRE, AI troubleshooting agent | Sifts metrics, logs and traces "surfacing the most likely root causes" | "built into Splunk Observability Cloud"; the pages do not say whether it costs extra | | [PagerDuty](https://www.pagerduty.com/pricing/) | Probable Origin, Root Cause Analysis (Past, Related and Outlier Incidents, Change Correlation), SRE Agent | Correlates incident history and changes | Add-on on Professional; included in the PD Reliability Platform tiers | | [Coroot](https://docs.coroot.com/ai/overview/) | AI-powered Root Cause Analysis | Follows the dependency graph from the affected service with ML checks over eBPF metrics, logs, traces and profiles; an LLM only summarizes the findings | Not in the Community Edition alone; Enterprise or Coroot Cloud | Event-correlation and enterprise AIOps platforms, which sit across many monitoring tools rather than collecting telemetry themselves: | Platform | RCA feature | What the vendor says it does | |---|---|---| | [BigPanda](https://www.bigpanda.io/our-product/root-cause-analysis/) | Root Cause Analysis, Root Cause Changes | Root Cause Changes "uses AI and Open Box Machine Learning to identify patterns and correlate real-time change data with incidents" | | [ScienceLogic](https://sciencelogic.com/platform/skylar-advisor) | Skylar Advisor (zebrium.com now redirects here) | "surfaces the problem, explains the root cause, and guides remediation" | | [Traversal](https://www.traversal.com/agentic-capabilities/incident-rca) | Incident RCA (an AI SRE) | "isolates root cause across services, dependencies, and changes, even when it's multiple hops from the symptom" | | [Splunk ITSI](https://www.splunk.com/en_us/products/pricing.html) | IT Service Intelligence | Service-level event analytics; priced separately from Observability Cloud | ## How are automated RCA features licensed and priced? List prices as published on 2026-10-21. "Per investigation" figures marked as our arithmetic multiply the vendor's published average consumption by its published unit price; actual consumption varies, as every vendor here says. | Vendor | Licensing model for RCA | Published price | Free way to try it | |---|---|---|---| | Datadog | Watchdog RCA with APM; Bits Investigation billed in AI credits | AI credits "Starting At $500 Per 500 credits, per month", billed annually, or $1.30 a credit on demand; an autonomous investigation averages about 6.5 credits ([pricing](https://www.datadoghq.com/pricing/)). Our arithmetic, not Datadog's figure: 6.5 credits at $1 (the annual $500 per 500) is about $6.50 an investigation, at $1.30 on demand about $8.45 | Free Infrastructure tier up to 5 hosts with 1-day retention; free trial | | Dynatrace | Dynatrace Platform Subscription: a minimum annual commit drawn down on a rate card | Full-Stack Monitoring $58 a month per 8 GiB host; $0.10 per AI unit, with an autonomous multi-step investigation averaging 30 AI units ([pricing](https://www.dynatrace.com/pricing/)). Our arithmetic, not Dynatrace's figure: 30 AI units at $0.10 list is about $3 an investigation (the unit price falls with larger commits). Agentic tasks run inside Dynatrace Assist are not billed "for a limited time"; the same tasks through workflows, MCP or the API are | 15-day free trial | | New Relic | Data ingest plus users plus compute; Autopilot needs Advanced Compute | $0.40 per GB beyond the free 100 GB; full platform user on Pro $349 a month on annual commitment; compute capacity units are "Contact Sales" ([pricing](https://newrelic.com/pricing), [compute](https://newrelic.com/pricing/compute)) | Free tier: 100 GB ingest, one full platform user; Autopilot is not in it | | Grafana Cloud | Plan plus usage; Investigations pay only for tokens | Pro "From $19 / month + usage"; Assistant "Starts at $20 / active AI user (includes 40M tokens per user)", extra tokens from $2 per 1M; "no per-seat or per-investigation fee" ([pricing](https://grafana.com/pricing/)) | Free plan includes Sift, the Assistant for 3 active AI users and a 25M-token monthly pool for alert-triggered investigations | | Splunk | Observability Cloud per host; ITSI by quote | Observability "Starts at $15 per host/month" (Infrastructure), $60 (App and Infra), $75 (End-to-End), billed annually; ITSI "Workload or ingest pricing ... Get a quote" ([pricing](https://www.splunk.com/en_us/products/pricing.html)) | The AI SRE page offers "Observability Cloud free for up to 15 hosts" | | PagerDuty | AIOps add-on on per-user plans, or platform tiers with metered AI actions | Professional $21 per user a month billed annually, AIOps purchased separately with no price shown; PD Reliability Platform Starter $2,800 a year (10 seats, 1,000 AI actions), Essential $12,000, Plus $48,000, Ultimate custom ([pricing](https://www.pagerduty.com/pricing/)) | Free plan for 5 users has none of the RCA features | | Coroot | Per monitored CPU core, or per investigation through Coroot Cloud | Enterprise Standard "$1 per monitored CPU core/month" ([pricing](https://coroot.com/pricing)); no published price per Coroot Cloud investigation | "10 free investigations per month" through [Coroot Cloud](https://docs.coroot.com/ai/coroot-cloud/); 14-day trial | | BigPanda | "value-based subscription pricing and a universal credit system" | Quote only; "Tiered credit plans start at 20,000 credits with a one- to three-year commitment" ([request pricing](https://www.bigpanda.io/request-pricing/)) | Proof-of-value assessment | | ScienceLogic | Per device for Skylar One; quote for Skylar Advisor | Skylar One "From $5 USD Per device/month"; Skylar Advisor "Please reach out to your ScienceLogic team" ([pricing](https://sciencelogic.com/why-sciencelogic/pricing)) | 14-day test drive of Skylar One | | Traversal | Quote, sized by the number of "Workers" your alert and incident volume needs | Not public ([get pricing](https://www.traversal.com/get-pricing)) | Walkthrough on request | | Aurora, HolmesGPT, K8sGPT | Apache 2.0, no license fee | You pay your model provider. Aurora measured $0.19 to $0.84 per investigation in our lab, median $0.35 (details below) | Self-host | Three things the table hides. First, the per-investigation numbers are averages the vendors publish "for reference"; a long investigation costs more. Second, several platforms only run agentic RCA on data they already store, so the real cost includes the ingest or host fees underneath. Third, open source is not free to run: in our lab, Aurora's background discovery job (prediscovery), which Aurora schedules after each cluster connection, cost about $1 a connection, almost twice what all 44 investigations cost together ($30.38 against $16.77; the 44 are the 38 graded runs plus 6 check runs the same day). Budget for the model and for the background work, not just the RCA calls. ## Which RCA tools are free or open source in 2026? | Tool | License | What it does for RCA | Tested by us | Model cost | |---|---|---|---|---| | [Aurora](https://github.com/Arvo-AI/aurora) | Apache 2.0 | Alert-driven investigation agent: queries your clusters and clouds read-only, writes an RCA with numbered evidence, suggests fixes in a separate list | Yes, commit 87d22baf | Median $0.35 per run on Claude Sonnet 5 via Bedrock | | [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) | Apache 2.0, CNCF Sandbox | Investigation agent with a CLI, server and operator mode; runs read-only commands through toolsets | Yes, 0.43.0 | Not captured in our runs | | [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) | Apache 2.0 | Runs analyzers over cluster objects and has an LLM explain each finding; takes no alert | Yes, 0.4.40 | Not captured | | [Coroot](https://github.com/coroot/coroot) Community Edition | Apache 2.0 | eBPF observability with built-in inspections; AI RCA needs Coroot Cloud (10 free a month) or Enterprise | No | Not applicable | | [Keep](https://github.com/keephq/keep) | MIT outside an `ee/` directory | Alert management and workflows; its AI correlation is "KEEP ENTERPRISE ONLY" per [keephq.dev](https://www.keephq.dev/). Elastic [announced an agreement to acquire Keep](https://www.elastic.co/security-labs/blog/elastic-and-keep-join-forces) in May 2025 | No | Not applicable | Free commercial tiers that include some RCA: Grafana Cloud Free (Sift and token-limited Investigations) and Coroot Cloud's 10 investigations a month. Datadog's and New Relic's free tiers exist, but their agentic RCA (Bits Investigation, Autopilot) is paid. PagerDuty Free has no RCA features. ## How do you tell whether an AI RCA is accurate and explainable? This is where tools differ most, and where pricing pages say nothing. Our position: do not buy an RCA you cannot audit. Three checks separate an explainable RCA from a confident paragraph, and you can run all three in a trial. ### 1. Every claim cites evidence, and the evidence says what the claim says Aurora's report numbers its claims against the tool calls that produced them. In scenario 13 of our lab (a NetworkPolicy blocks the api from reaching redis), run 1, the root cause reads in part: "The root cause was a NetworkPolicy named `redis` in the shop namespace, created at 2026-10-02T20:23:54Z, which restricts ingress to redis:6379 to pods labeled `app=worker` only [20, 22]. The `api` pods are labeled `app=api`, not `app=worker` [23]". Those numbers are steps in the trace. Step 22 is: ``` $ kubectl describe networkpolicy redis -n shop Name: redis Namespace: shop Created on: 2026-10-02 20:23:54 +0000 UTC Labels: Annotations: Spec: PodSelector: app=redis Allowing ingress traffic: To Port: 6379/TCP From: PodSelector: app=worker Not affecting egress traffic Policy Types: Ingress ``` and step 23 shows both api pods labeled `app=api,pod-template-hash=f64486c4c`. The claim and the output match, so you can accept the RCA without rerunning anything. Citations are necessary, not sufficient. In scenario 12, the worker is OOMKilled (the alert) and, separately, the frontend Service selects no pods. Aurora's run 1 found the OOM and then wrote: > Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]. Steps 5 and 14 are both `kubectl get pods -n shop -o wide`. Running pods say nothing about whether a Service has endpoints. The citation was real; the inference was not, and a real second outage went unreported in both runs. An auditor who checks what the cited step actually proves catches this in a minute. The fix is in the runbook [a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). ### 2. The full tool-call trace is available You should be able to see every command the agent ran, in order, with its output. Aurora's scenario 13 run made 25 tool calls in 202 seconds for $0.69. Nineteen were kubectl commands: pods, events, frontend, api and redis logs, the api code ConfigMap, Services, nodes, and finally `get networkpolicies`, `describe networkpolicy redis` and the api pod labels; the other 6 read the incident list, connected clusters and Aurora's memory, and wrote two memory notes. HolmesGPT prints its trace as it goes ("Running tool #1 kubernetes_tabular_query: ..." and so on); its scenario 13 run made 26 tool calls, all of them cluster queries, in 110 seconds and reached the same cause. K8sGPT has no trace in this sense: it runs fixed analyzers, and in scenario 13 it printed "No problems detected", because every object looks healthy to an analyzer. The trace also shows what the agent could not do. In scenario 19 run 1, Aurora tried to read the live nginx config inside a pod and its read-only command policy blocked it: ``` $ kubectl exec frontend-76d98597b9-g5zrg -n shop -c frontend -- cat /etc/nginx/conf.d/default.conf Command blocked by organization policy: Mutating kubectl operations ``` Its report said the discrepancy "was not fully reconciled", which is the right thing to say. The clue it missed was already in its own `get pods` output: the failing pod was 9 seconds younger than the healthy one. Good reports also list what was ruled out and what was not checked; Aurora's scenario 13 report has both sections. ### 3. Accuracy is measured on faults whose cause is known A citation trail shows you how the agent reasoned on one incident. Only a set of incidents with a known cause tells you how often it is right. Each lab scenario states its required findings and its tempting wrong answers in a file; scenario 13's says: ``` must_identify: - The api cannot reach redis (connection timeouts), which is why /api/ returns 5xx; redis itself is healthy - The cause is the NetworkPolicy redis, which allows ingress to redis only from app=worker, not from the api wrong_answers: - redis is down or overloaded - DNS failure - Restarting the api or redis as the fix ``` Correct means every required finding stated and no wrong answer asserted; partial means some but not all, or a wrong answer stated alongside the right one. That is the bar we used below. ## What did the lab measure for Aurora, HolmesGPT and K8sGPT? Setup: a small app (`shop`: nginx frontend, Python api, worker, redis, load generator) on a kind v0.33.0 cluster running Kubernetes v1.37.0 with 3 nodes. Each scenario injects a known fault, and a check script proves the fault is visible before any agent starts. Aurora's runs are from 2026-10-02; HolmesGPT's and K8sGPT's from 2026-10-15 and 2026-10-16, on the same cluster and faults. All three got the same admin kubeconfig, each behind its own guard: Aurora's default read-only `observability_only` command policy, HolmesGPT's command allow-list, K8sGPT's read-only analyzers. Aurora and HolmesGPT both ran on Claude Sonnet 5 via Bedrock; K8sGPT ran on Sonnet 4.5, because it sends a temperature parameter Bedrock rejects for Sonnet 5. The prompts differ. Aurora got the alert text plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." HolmesGPT got the alert text plus "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." K8sGPT takes no alert and scanned the namespace. | | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | Version | commit 87d22baf | 0.43.0 | 0.4.40 | | Correct | 34 of 38 | 31 of 38 | not scored as a peer (see below) | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 | | "No problems detected" | 0 | 0 | 12 (both runs of 6 scenarios) | | Median time per run | 153.5 s | 45 s | 12.5 s for two analyze passes | | Median cost per run | $0.35 | not captured | not captured | | Who confirmed the grades | a person at Arvo, who changed none | judge model only | judge model only | How the grades were checked. A judge model (Claude Opus 5.5, from the same vendor as every model under test) graded all 114 runs against the injected fault. A person read all 38 Aurora grades. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model re-read all 76 and agreed with every one, and a third pass graded all 76 from scratch, agreeing on all 38 HolmesGPT verdicts and disputing 6 K8sGPT labels (it would have called them "no answer"). A blinded regrade with agent, tool and cluster names removed matched the published verdict in 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs; every change moved a verdict down. No person has checked the 76 yet. On correct against not correct, the two-sided Fisher exact test that [/benchmark](/benchmark) computes on the totals gives p = 0.52 for Aurora against HolmesGPT, comparing person-read Aurora grades with judge-only HolmesGPT grades. Read the two as roughly even on accuracy. HolmesGPT is clearly faster: 45 seconds at the median against Aurora's 153.5. Aurora's times include contention from its own background discovery jobs in the lab, but that does not close a 3x gap. ### Aurora's four partial answers None of Aurora's 38 runs was graded wrong; four were partial: - **Scenario 12, both runs (partial).** Found the OOMKilled worker, reported the frontend workloads as healthy or unaffected without checking Services or endpoints, as quoted above. HolmesGPT also found only the OOM in both runs. - **Scenario 19, run 1 (partial).** Found the edited ConfigMap but could not work out that only the restarted pod had loaded it. Run 2 got it right with 28 tool calls. - **Scenario 11, run 2 (partial).** Found the wrong Service targetPort, but never looked at the worker whose noisy errors were the scenario's red herring; its report says those deployments "were not examined". You could argue it deserves correct, since it was not misled. We kept the rubric as written. A second-model review also found small claims the evidence did not support in 15 of the 18 reports for scenarios 11 to 19, none of them in the root cause: "both cluster nodes" when there were three, an image "pulled successfully" with no Pulled event, a trend drawn from one sample. That is the kind of hallucination the citation check in section 1 is for. Separately, a grading script found an acceptable fix in Aurora's suggestions in 34 of 38 runs. HolmesGPT's one wrong verdict was scenario 19, run 1, whose answer opened "All data points to a transient startup race, now resolved." Its second run was correct. ### What K8sGPT does and does not do K8sGPT answers a different question: what looks broken in this namespace? It runs analyzers and has the model explain each finding; it does not take an alert or follow a chain of evidence, so grading it on alert-driven criteria shows what it misses, not that it failed the same task. In 12 of its 38 runs, both runs of 6 scenarios (the wrong liveness port, a wrong targetPort, the NetworkPolicy, a stray Deployment sharing a label, a slow memory leak, the config drift), it printed "No problems detected". Where the fault shows on an object, such as a mistyped image tag, it named it in seconds (graded partial there, because it never said the old pods were still serving). As a cheap cluster scan it is useful; as an RCA tool for an alert it is the wrong fit. More in [HolmesGPT vs K8sGPT](/blog/holmesgpt-vs-k8sgpt). ## What are the alternatives to Zebrium for log-driven RCA? ScienceLogic [acquired Zebrium](https://sciencelogic.com/press_release/aiops-provider-sciencelogic-acquires-machine-learning-analytics-provider-zebrium-to-provide-at-a-glance-root-cause-visibility) on October 11, 2022. In July 2024 ScienceLogic [introduced](https://sciencelogic.com/press_release/sciencelogic-introduces-skylar-ai-the-suite-of-advanced-ai-capabilities-creating-a-new-industry-paradigm) log-based "Skylar Automated Root Cause Analysis (RCA)", and today zebrium.com redirects to ScienceLogic's Skylar Advisor page, so that is where ScienceLogic now points Zebrium's buyers; it is priced by quote. Other options that read logs alongside metrics and traces to propose a cause: Grafana Cloud Investigations (token-priced), Datadog Bits Investigation (credit-priced), Coroot (per core, with 10 free cloud investigations a month), and the open source agents above, which read logs through `kubectl` and monitoring connectors. Only the last group can be run on your own infrastructure with no vendor account. ## How do Traversal, Datadog, Dynatrace and ScienceLogic compare for evidence-backed RCA? From what each publishes: Dynatrace builds its RCA on a topology model and calls the method deterministic and causation-based; Datadog's Watchdog RCA needs APM data and Bits Investigation runs as an agent over Datadog data; Traversal sells an AI SRE that deploys read-only by default, with private on-premises deployments, your own model and your own cloud account as [options](https://www.traversal.com/security); ScienceLogic positions Skylar Advisor on top of Skylar One's device and service data. Traversal's RCA page claims "> 82% Accurate remediation", and Traversal has published a [benchmark](https://www.traversal.com/blog/the-model-isnt-the-bottleneck-findings-from-benchmarking-on-high-severity-incidents-in-enterprise) on 25 real incidents at one unnamed enterprise, judged blind against frontier models given direct tool access; the incidents are private, so a buyer cannot rerun it. BigPanda's page says its RCA "reduces MTTR by up to 50%". Datadog has written about the [internal evaluation platform](https://www.datadoghq.com/blog/engineering/bits-ai-eval-platform/) it uses for Bits. We found no public test set for any of these four that a buyer could rerun. Coroot, outside this group, is the exception: its [AI RCA benchmark](https://coroot.com/ai-benchmark) uses five OpenTelemetry Demo failure flags anyone can trigger, and Coroot scores its own RCA 2/3 or 3/3 on each. We have not tested any of them, so we cannot rank them. ## How can you test a vendor's RCA in a trial? Use the lab. It is a kind cluster on one machine; any agent that can read a kubeconfig can be pointed at it, and a SaaS platform needs its own agent or collector installed in the cluster first, so the faults reach it the way they would in production: ``` git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab ./lab.sh up ./lab.sh break 13-networkpolicy-blocks-redis ``` Then give the trial agent the scenario's `alert` line from `scenarios/13-networkpolicy-blocks-redis/scenario.yaml`, and grade its answer against `must_identify` and `wrong_answers` in the same file. `./lab.sh break` restores the healthy app before injecting each fault, and the cluster's kubeconfig is written to `.kubeconfig` in the repo. Five scenarios are enough to learn a lot: 03 (bad image tag) is the easy baseline, 12 (two faults) and 19 (partial restart) got partial or wrong answers from both investigation agents we tested, 13 (NetworkPolicy) gave K8sGPT's analyzers nothing to flag, and 17 (slow leak) has clean logs. While you grade, run the three checks above: does each claim cite a step, does the step say what the claim says, and can you see every command. Runbooks for each fault live at [/runbooks](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes), and [/diagnose](/diagnose) walks through a single failure on your own cluster. ## Which RCA tool fits which team? | If you | Look at | Why | |---|---|---| | Already pay for Dynatrace Full-Stack or Datadog APM | Their built-in RCA first | It is in the price you already pay; agentic investigations add about $3 (Dynatrace, 30 AI units at $0.10, outside Assist) or $6.50 to $8.45 (Datadog, about 6.5 credits at $1 to $1.30) each, by our arithmetic from their published averages | | Run Grafana Cloud | Sift and Investigations | Free plan includes both within token limits; Pro charges tokens, not seats | | Need an agent that works across several monitoring vendors and clouds | Aurora or HolmesGPT | Both open source, read-only by default, roughly even on our 19 faults; HolmesGPT is about 3x faster | | Must keep data and the model inside your network | Aurora, HolmesGPT, or Traversal's BYOC and on-premises options | Self-hosted with your own model, or a vendor deployment in your account | | Want a fast cluster health scan | K8sGPT | 12.5 s median for two analyze passes; does not investigate alerts | | Are an enterprise consolidating events from many tools | BigPanda, ScienceLogic, Splunk ITSI | Quote-only, multi-year commitments; ask for a proof of value on faults you choose | | Have no budget | Aurora, HolmesGPT, Grafana Cloud Free, Coroot Cloud's 10 a month | No license fee; you still pay the model or stay inside free limits | ## What this does not show The lab is Kubernetes-only, on a synthetic app, with faults Arvo wrote, two runs per scenario. It says nothing about cloud APIs, logs at production scale or noisy real alerts, and nothing about how any commercial tool would score. HolmesGPT's grades still need a person to read them, and its cost per run was not captured. Vendor prices change often and several are promotional ("for a limited time" at Dynatrace); recheck them before you sign. For the full method and every trace, see the [Aurora lab report](/blog/aurora-lab-report-kubernetes-19-failures) and the three-way write-up [Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt). For background on the practice itself, [Root Cause Analysis: The Complete Guide for SREs](/blog/root-cause-analysis-complete-guide-sres); for buying criteria, [how to evaluate an AI SRE platform](/blog/how-to-evaluate-ai-sre-platform). Vendor-specific comparisons: [Dynatrace Davis](/blog/dynatrace-davis-alternative-open-source), [Datadog Bits AI SRE](/blog/datadog-bits-ai-sre-alternative-open-source), [New Relic AI](/blog/new-relic-ai-alternative-open-source), [Splunk ITSI](/blog/splunk-itsi-alternative-open-source), [BigPanda](/blog/bigpanda-alternative-open-source), [PagerDuty](/blog/pagerduty-alternative-root-cause-analysis), [Keep vs Aurora](/blog/keep-vs-aurora-open-source-aiops). Verified 2026-10-21: vendor features and prices from each vendor's own pricing page and docs, linked above; benchmark numbers from [results.json](https://raw.githubusercontent.com/Arvo-AI/aurora-lab/main/benchmark/results.json) generated 2026-10-21; Aurora runs on 2026-10-02, HolmesGPT and K8sGPT runs on 2026-10-15 and 2026-10-16. Aurora is on [GitHub](https://github.com/Arvo-AI/aurora) under Apache 2.0, or [book a demo](https://cal.com/arvo-ai/demo) to see it investigate your own alerts. --- ## FireHydrant Alternatives: Open Source and Paid (2026) URL: https://www.aurorasre.ai/blog/firehydrant-alternative-open-source Published: 2026-04-06 (updated 2026-10-12) Author: Noah Casarotto-Dinning, CEO at Arvo AI > FireHydrant is now part of Freshworks and AI features are Enterprise-only. Aurora is a free, open source alternative with autonomous AI investigation. > **Key Takeaway:** FireHydrant is a solid incident management platform, but it was [acquired by Freshworks](https://firehydrant.com/blog/firehydrant-to-be-acquired-by-freshworks/) (announced December 2025, closed January 2026), AI features are locked to the Enterprise tier, and there's no autonomous investigation. Aurora is an open source (Apache 2.0) alternative with AI agents that autonomously investigate root causes across your cloud infrastructure, completely free and self-hosted. ## What is FireHydrant? [FireHydrant](https://firehydrant.com) is an all-in-one incident management platform that helps teams plan, respond to, and learn from incidents. Their tagline: "Fight Fires Faster." They claim teams resolve incidents [up to 90% faster](https://firehydrant.com) with their platform. Freshworks (NASDAQ: FRSH) [announced its acquisition of FireHydrant](https://firehydrant.com/blog/firehydrant-to-be-acquired-by-freshworks/) in December 2025 and closed the deal in January 2026. FireHydrant is becoming the incident management and reliability layer inside Freshservice, Freshworks' ITSM product. **Notable customers:** [Backblaze](https://firehydrant.com/customer-stories) (91% faster mitigation), [Bluecore](https://firehydrant.com/customer-stories) (saving 30-90 minutes per incident), Snyk, AuditBoard, and Qlik. ## What is Aurora? [Aurora](https://github.com/Arvo-AI/aurora) is an open source (Apache 2.0) AI agent for automated incident investigation and root cause analysis. When an alert fires, Aurora's LangGraph-orchestrated agents autonomously query your infrastructure across AWS, Azure, GCP, OVH, Scaleway, and Kubernetes, delivering a structured RCA with remediation recommendations. Aurora is free, self-hosted, and works with any LLM provider. ## How They Compare ### AI Capabilities **FireHydrant AI** ([Enterprise tier only](https://firehydrant.com/pricing)): - AI-generated incident summaries from Slack messages - Automated event timelines - Real-time call transcription (Zoom, Google Meet) with key point summarization - AI-drafted retrospectives with contributing factors and suggested action items - Stakeholder update generation FireHydrant's AI is **documentation-focused**, it summarizes what happened, transcribes calls, and drafts retrospectives. It does not autonomously investigate root causes or query infrastructure. **Aurora AI:** - Multi-agent architecture via LangGraph with dynamic tool selection (30+ tools) - Correlates alerts across services and dependencies - Constructs investigation timelines linking deployments, infra events, and telemetry - Generates structured RCA with evidence citations and remediation steps - Human-in-the-loop for write/destructive actions, read-only commands run automatically - Executes `kubectl`, `aws`, `az`, `gcloud` commands in sandboxed Kubernetes pods - Queries cloud APIs directly. AWS, Azure, GCP, OVH, Scaleway - Traverses Memgraph infrastructure dependency graph for blast radius - Searches Weaviate knowledge base (vector search over runbooks and past incidents) - Suggests code fixes with diff preview, human approves and creates PR - Works with any LLM provider including local models via Ollama ### Incident Response & Coordination **FireHydrant** is strong at incident coordination: - Slack and Microsoft Teams chatbot - Automated runbooks (triggered by severity, service, or custom fields) - Incident roles and assignments - Service catalog with dependency mapping - [40+ integrations](https://firehydrant.com/integrations) - MTTx analytics (MTTD, MTTA, MTTR, MTTM) - Mobile notifications (iOS, Android) **Aurora** creates and manages Slack incident channels, tracks action items with Jira sync, and sends investigation notifications. Aurora does not have Microsoft Teams support, incident roles, service catalog, or mobile app. ### On-Call & Alerting **FireHydrant** (branded "Signals"): - Team-based on-call schedules with unlimited escalation policies - SMS, voice, push, Slack, Teams, email, WhatsApp notifications - Alert routing via Common Expression Language (CEL) - Consumption-based alert pricing (not per-seat) - Alert grouping (Enterprise only) **Aurora** has no on-call capabilities. For on-call, use PagerDuty, Grafana OnCall, or Opsgenie alongside Aurora. ## Feature Comparison **FireHydrant has, Aurora doesn't:** - Microsoft Teams support - Incident roles and assignments - Service catalog with dependency mapping - Status pages (public and private) - MTTx analytics dashboards - Mobile notifications (iOS, Android) - Call transcription (Zoom, Google Meet) - SOC 2 compliance - 40+ integrations - Consumption-based alerting **Aurora has, FireHydrant doesn't:** - Autonomous AI investigation (FireHydrant AI is documentation-focused only) - Direct cloud infrastructure querying (AWS, Azure, GCP, OVH, Scaleway) - CLI execution in sandboxed Kubernetes pods - Native vector search knowledge base (Weaviate RAG) - Infrastructure dependency graph (Memgraph) - Terraform/IaC state analysis - AI-suggested code fixes with diff preview - Open source (Apache 2.0), full codebase auditable - Self-hosted deployment (Docker Compose, Helm) - LLM provider flexibility (OpenAI, Anthropic, Google, Ollama) - Free, no licensing costs **Both have:** - Slack incident channel management - Automated postmortem/retrospective generation - Action item tracking with Jira sync - On-call integrations (PagerDuty, Opsgenie) - Datadog, Grafana, New Relic monitoring integrations - GitHub integration - Runbook/workflow automation - RBAC and security controls ## Pricing **FireHydrant** ([firehydrant.com/pricing](https://firehydrant.com/pricing)): - Free trial: 2 weeks, up to 10 responders - Pro: **$25/responder/month** billed annually (free tier up to 10 responders) - Enterprise: Custom pricing (required for AI features) - Alerting is consumption-based (separate from platform fee) **Aurora:** - **Free**. Apache 2.0, self-hosted - Costs: infrastructure + LLM API usage - $0 LLM cost with Ollama local models > Note: FireHydrant AI features (summaries, transcripts, triage, retrospectives) are **only available on the Enterprise tier**. Pro users do not get AI capabilities. ## The Freshworks Acquisition Factor FireHydrant was [acquired by Freshworks](https://firehydrant.com) in December 2025. What this means: - The platform will be integrated into **Freshservice** (Freshworks' ITSM product) - Current accounts, pricing, and support stay the same during transition - Long-term product direction is now under Freshworks' roadmap - Some teams may want to evaluate alternatives before deeper Freshworks lock-in Aurora is independently maintained open source, no acquisition risk, no vendor roadmap dependency. ## When to Choose FireHydrant - **You need full incident coordination**, roles, runbooks, status pages, service catalog, analytics - **Call transcription matters**, real-time Zoom/Google Meet transcription with AI summaries - **Microsoft Teams is required**. Aurora is Slack-only - **You want managed SaaS**, no infrastructure to maintain - **You're already in the Freshworks ecosystem**. Freshservice integration will be seamless ## When to Choose Aurora - **Investigation is your bottleneck**, you need AI that actually investigates, not just summarizes - **You need direct cloud querying**. AI agents that run commands on AWS, Azure, GCP, K8s - **Open source is required**, audit how AI investigates your infrastructure - **Self-hosted is required**, compliance, data sovereignty, or air-gapped environments - **Budget is limited**. FireHydrant Enterprise (required for AI) is custom pricing; Aurora is free - **LLM flexibility**, choose your provider or run local models - **You're concerned about the acquisition**. Aurora has no vendor lock-in risk - **You want a custom integration**, the Arvo AI team builds custom integrations at no cost. [Reach out](https://cal.com/arvo-ai) and they'll build it with you. ## Limitations of Aurora Aurora is powerful for investigation but doesn't replace a full incident coordination platform: - **No on-call scheduling**, use PagerDuty, Grafana OnCall, or Opsgenie alongside Aurora - **No status pages**, use Atlassian Statuspage, incident.io, or Instatus - **Slack only**, no Microsoft Teams support currently - **No mobile app**, investigation results are accessed via web dashboard - **SOC 2 Type II in progress**. not yet certified (FireHydrant has SOC 2) - **Self-hosted requires infrastructure**, you maintain the Docker/K8s deployment > "We built Aurora for one job, investigating why incidents happen. We deliberately didn't build on-call or status pages because tools like PagerDuty and FireHydrant already do those well. Aurora is the investigation layer that plugs into your existing stack.". Noah Casarotto-Dinning, CEO at Arvo AI ## Getting Started with Aurora ```bash git clone https://github.com/arvo-ai/aurora.git && cd aurora make init # generates secrets, copies .env.example to .env nano .env # add OPENROUTER_API_KEY, OPENAI_API_KEY or ANTHROPIC_API_KEY make prod-prebuilt # pulls prebuilt images from GHCR and starts ``` Configure your monitoring webhooks, add cloud credentials, and investigations start automatically. See the [full documentation](https://arvo-ai.github.io/aurora/). Learn more at [aurorasre.ai](https://www.aurorasre.ai). For other comparisons, see [Aurora vs Traditional Tools](/blog/aurora-vs-traditional-incident-management-tools), [PagerDuty Alternative](/blog/pagerduty-alternative-root-cause-analysis), and [Rootly Alternative](/blog/rootly-alternative-open-source-incident-management). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) > **All claims sourced from official websites.** FireHydrant data from [firehydrant.com](https://firehydrant.com). Aurora data from [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora). Last verified: April 2026. --- ## incident.io Alternatives: Open Source Incident Management 2026 URL: https://www.aurorasre.ai/blog/incident-io-alternative-open-source Published: 2026-04-06 (updated 2026-10-21) Author: Arvo AI Team, Engineering > Open source incident.io alternatives checked Oct 21, 2026: Keep, GoAlert, openstatus and 6 more, two archived ones to avoid, and incident.io pricing. No single open source project replaces incident.io. incident.io sells four products in one platform (on-call, Slack and Teams incident response, status pages, and an AI SRE called Investigations), and the open source alternatives each cover one layer: GoAlert for on-call, openstatus, Uptime Kuma, Gatus or Cachet for status pages, Incident Bot or openstatus for running incidents in Slack, Keep for alert management and workflows, and Aurora or HolmesGPT for AI investigation. Two of the best-known open source options are gone: Grafana OnCall OSS was archived on 2026-03-24 and Netflix Dispatch in September 2025. incident.io itself starts free, then $15 per user a month for Team and $25 for Pro, with on-call and Investigations priced on top. Arvo AI builds Aurora. Aurora is an investigation agent: it takes an alert or an incident, queries your infrastructure and writes a root cause. It is not an incident response or on-call platform. It has no paging, no schedules, no escalation policies and no status pages, so it replaces only the Investigations part of incident.io, and it is built to run next to incident.io rather than instead of it. We could not install incident.io; every claim about it below links its own site or docs, read on 2026-10-21. Every repository was checked on GitHub the same day. ## What does incident.io do, and how much does it cost? incident.io calls itself a "software reliability platform unifying on-call, agentic root cause analysis, incident response, and status pages" ([incident.io](https://incident.io/)). Its [pricing page](https://incident.io/pricing) on 2026-10-21: | Plan | Price | Response | On-call | Status pages | AI and API | |---|---|---|---|---|---| | Basic | Free | Slack or Teams, 1 workflow, 2 integrations | Single team, 1 schedule | 1 public page | Basic post-mortem editor | | Team | $15 per user a month with the annual discount ($19 without) | 3 workflows, unlimited integrations | + $10 per user a month, 2 schedules | 1 public page | MCP, API and webhooks | | Pro | $25 per user a month | Unlimited workflows, custom post-incident flow | + $20 per user a month, unlimited schedules | 1 public, 1 internal page | AI agent, Suggestions, Scribe, "AI Native" post-mortems; Investigations as an add-on | | Enterprise | Custom | Everything in Pro, multiple streams, audit logs | Unlimited on-call policies, phone trees | Unlimited public, internal and customer pages | Investigations as an add-on | | On-call only | $20 per user a month | | Standalone product | | | Investigations (root cause analysis, the autonomous agent and Nexus data sources) is marked "Available as add-on" on Pro and Enterprise and has no published price. So for a 20-person team on Pro with on-call, the published part is ($25 + $20) x 20 = $900 a month before the AI SRE, which needs a quote. SOC 2 report and HIPAA are listed on every plan. ## What is incident.io's AI SRE, and how does it work? incident.io's AI SRE product is [Investigations](https://incident.io/investigations), "Agentic root cause analysis". Per [the docs](https://docs.incident.io/investigations/overview), the moment an incident is declared it gathers the alert and stack trace, similar past incidents, channel discussion, change events, runbooks, code changes and telemetry, then "forms a hypothesis and then tests it, reading your code and querying your dashboards to confirm or rule it out", and posts a root cause with evidence and next steps into the incident channel. Other points from its own pages: - **Model.** It runs on Nexus, which incident.io calls its "production intelligence model", with its own instance per customer ([Nexus](https://incident.io/nexus)). The underlying providers are OpenAI, Anthropic and Google Vertex under Zero Data Retention agreements ([trust and safety](https://docs.incident.io/investigations/trust-and-safety)). - **Your own model.** Asked "Can we bring our own AI model or API key?", the docs answer "Not currently." - **Actions.** "The only change Investigations can make to your systems is a pull request you review and merge yourself" ([Investigations](https://incident.io/investigations)). The separate [agent](https://incident.io/agent) works in Slack, Teams, the mobile app, the web dashboard and coding agents. - **MCP.** Nexus can be connected to Claude and Cursor over MCP ([Nexus](https://incident.io/nexus)); MCP is listed on Team, Pro and Enterprise ([pricing](https://incident.io/pricing)). - **Accuracy.** incident.io grades each investigation on a closed incident against the cause your responders recorded, on a four-step scale from "Bullseye" to "Nowhere near", and shows the scores in your account ([measuring accuracy](https://docs.incident.io/investigations/measuring-accuracy)). That is a sound way to judge it on your own incidents. We found no accuracy figure on its site that an outsider can rerun. ## Is there an open source version of incident.io? No. incident.io is a hosted service and its pricing page lists only hosted plans. The open source projects below are what you can run yourself; every one had commits in the three months before 2026-10-21. | Project | License | Latest release, last commit | On-call | Status page | Incident response | Postmortems | AI | |---|---|---|---|---|---|---|---| | [Keep](https://github.com/keephq/keep) | MIT, except an `ee/` folder under an enterprise license | v0.54.3 (2026-09-09), 2026-09-28 | No | No | Alert and incident hub, YAML workflows | Not described | LLM enrichment and correlation; advanced correlation is Enterprise-only | | [GoAlert](https://github.com/target/goalert) | Apache 2.0 | v0.35.0 (2026-09-21), 2026-10-14 | Schedules, escalations, SMS and voice | No | No | No | None described | | [openstatus](https://github.com/openstatusHQ/openstatus) | AGPL-3.0 | no tagged releases, 2026-10-21 | No | Yes, plus uptime monitoring from 28 regions | Declare and run incidents from the dashboard or Slack | Not described | MCP server for Claude, ChatGPT, Cursor | | [Incident Bot](https://github.com/incidentbot/incidentbot) | MIT | 2026.06.01, 2026-08-20 | No | Embeddable status widgets; Statuspage integration | Slack or Matrix channels, roles, severities | Builds a postmortem document base | None described | | [Uptime Kuma](https://github.com/louislam/uptime-kuma) | MIT | 2.5.6 (2026-10-09), 2026-10-20 | No | Multiple status pages | No | No | None described | | [Gatus](https://github.com/TwiN/gatus) | Apache 2.0 | v5.37.0 (2026-09-24), 2026-10-17 | No | Yes, with announcements | No; sends alerts to Slack, Teams, PagerDuty and more | No | None described | | [Cachet](https://github.com/cachethq/cachet) | Cachet's own license from Alt Three Services | v2.4.1 (2023-11-07); 3.x rebuild in progress, commits to 2026-10-05 | No | Yes | No | No | None described | | [Aurora](https://github.com/Arvo-AI/aurora) | Apache 2.0 | v1.5.10 (2026-10-16), 2026-10-21 | No | No | Takes alerts and incidents in; posts findings back | Generates postmortems, exports to Confluence, Notion, SharePoint | Investigation agent, any LLM including Ollama | | [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) | Apache 2.0, CNCF Sandbox | 0.45.0 (2026-10-15), 2026-10-21 | No | No | No | No | Investigation agent, many LLM providers | What this adds up to: open source covers on-call (GoAlert) and status pages (four good options) well. Running an incident in Slack is thinner: Incident Bot is a small project and openstatus's incident features sit inside a status-page product. Nothing open source matches incident.io's workflows, catalog and insights. A full replacement means running three or four services and wiring them together, which is real work. Teams that pick it usually do so for data control, cost at headcount, or model choice, not to save effort. ### Archived or stale projects to avoid starting on - **Grafana OnCall OSS.** "On 2025-03-11 Grafana OnCall (OSS) entered maintenance mode and was archived on 2026-03-24" ([README](https://github.com/grafana-cold-storage/oncall)). Grafana points users to Grafana Cloud IRM, which is hosted. More in [Grafana OnCall alternatives](/blog/grafana-oncall-alternative-open-source). - **Netflix Dispatch.** Archived and read-only since September 2025; the README gave September 1, 2025 as the date ([README](https://github.com/Netflix/dispatch)). - **Monzo Response.** Not archived, but the last commit is from 2021-06-08 ([repo](https://github.com/monzo/response)). - **LinkedIn Oncall and Iris.** Not archived, but the last commits on their default branches are from 2025-08-20 and 2024-01-18 ([Oncall](https://github.com/linkedin/oncall), [Iris](https://github.com/linkedin/iris)). ## Is Keep open source incident management and automation? Keep describes itself as "The open-source AIOps and alert management platform" ([GitHub](https://github.com/keephq/keep)): one place for alerts from many tools, with deduplication, enrichment, filtering and correlation, two-way integrations and workflows written in YAML, "Similar to GitHub Actions", that can query a database, update Jira or run Python ([keephq.dev](https://www.keephq.dev)). It is not an on-call or status page tool, and it does not schedule people; it sits between your monitors and your paging and ticketing tools, and lists incident.io among its integrations. Three things to know before you adopt it: - **Ownership.** Elastic announced it would acquire Keep Alerting Ltd on 2025-05-07 and said "Keep will integrate with Elasticsearch and Kibana and remain open source" ([Elastic](https://www.elastic.co/blog/elastic-and-keep-join-forces)). In November 2025 maintainer Shahar Glazner described Keep as being in a "community-maintained" phase where two maintainers review and merge community pull requests ([issue 5451](https://github.com/keephq/keep/issues/5451)). Releases continue at that pace: v0.54.3 on 2026-09-09, last commit 2026-09-28. - **License.** The core is MIT. Code in the `ee/` folder is under the Keep Enterprise Edition license, which allows production use only under Keep's subscription terms ([LICENSE](https://github.com/keephq/keep/blob/main/LICENSE)). - **AI.** The README lists AI backends for "Enrichments, Correlations and Incident Context Gathering", including OpenAI, Anthropic, Gemini, DeepSeek, Ollama and LlamaCPP. Keep's site marks its "Next-gen AIOps" alert correlation based on past incidents as "KEEP ENTERPRISE ONLY". Keep groups and routes alerts; it does not investigate why one fired. Our [Keep page](/blog/keep-vs-aurora-open-source-aiops) covers its features, the open source versus Enterprise split, how active the project is after the Elastic deal, and where an investigation agent fits behind it. ## How does incident.io's AI compare to an autonomous investigation agent? Both start from an alert or incident and run their own queries. The differences are in where they run, who picks the model, and what you can check. | | incident.io Investigations | Aurora | HolmesGPT | |---|---|---|---| | Where it runs | incident.io's cloud | Your servers: Docker Compose, Helm, air-gapped bundle | Your machine or cluster: CLI, server, Kubernetes | | Model | Nexus on OpenAI, Anthropic, Google Vertex; no bring-your-own | You choose: OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, OpenRouter, Ollama | You choose: OpenAI, Anthropic, Azure, Bedrock, Gemini and more | | Starts from | A declared incident (or by condition, or on demand) | An alert or incident from PagerDuty, Datadog, Grafana, New Relic, OpsGenie, incident.io and more | A question or alert you pass it | | Context it uses | Telemetry, code, past incidents, channel chat, runbooks, catalog | Cloud CLIs and APIs, Kubernetes, monitoring connectors, code, knowledge base | Toolsets for Kubernetes, Prometheus, Grafana, Loki, Datadog and more | | Changes it can make | A pull request you merge | Suggests fixes; can open a PR after a human approves; no auto-merge | Read-only commands | | Price | Add-on, quote only | Free; you pay hosting and model calls | Free; you pay model calls | | Accuracy you can check | Graded per customer in your own account | Public lab runs (below) | Public lab runs (below) | incident.io's advantage is context it already owns: your incident history, Slack channel, catalog and on-call data are in the same product, and its docs show it reading them. An agent you run yourself has to be given that context through connectors. The advantage of the open source agents is that you can read every tool call, choose and swap the model, keep everything inside your network, and test them before buying anything. ### What the open source agents did on 19 real faults We cannot run incident.io Investigations, so we tested what we could. We built a small app (an nginx frontend, a Python api, a worker, redis and a load generator) on a 3-node kind v0.33.0 cluster running Kubernetes v1.37.0, injected one known fault per scenario (two in scenario 12) and gave Aurora and HolmesGPT the alert; K8sGPT takes no alert and scanned the namespace. Each tool ran twice per scenario, 38 runs per tool. All three got the same admin kubeconfig, each behind its own guard (Aurora's `observability_only` command policy, HolmesGPT's allow-list, K8sGPT's read-only `analyze`). Aurora (commit 87d22baf) and HolmesGPT 0.43.0 both used `bedrock/us.anthropic.claude-sonnet-5`; K8sGPT 0.4.40 used `us.anthropic.claude-sonnet-4-5`, because Bedrock rejects the temperature parameter it sends to Sonnet 5. Aurora's runs are from 2026-10-02, HolmesGPT's and K8sGPT's from 2026-10-15 and 2026-10-16, on the same cluster and faults. | | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | Correct | 34 of 38 | 31 of 38 | not scored as a peer | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 | | "No problems detected" | 0 | 0 | 12 runs (6 scenarios) | | Median time per run | 153.5 s | 45 s | 12.5 s for two analyze passes | | Median model cost per run | $0.35 | not captured | not captured | | Who confirmed the grades | a person at Arvo | judge model only | judge model only | A judge model (`us.anthropic.claude-opus-5-5`) graded every answer against the injected fault. A person at Arvo read all 38 Aurora grades and changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model re-read all 76 and agreed with every one, a third pass graded them from scratch and agreed on all 38 HolmesGPT verdicts while disputing 6 K8sGPT labels, and no person has checked them yet. A blinded regrade, with tool names removed from the judge's prompt, matched the published verdict in 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs. The two agents' prompts differ in wording, and Arvo wrote the scenarios. The [/benchmark](/benchmark) page computes a two-sided Fisher exact test on the totals, p = 0.52, so the accuracy gap between Aurora and HolmesGPT is not significant, and it compares human-read grades with judge-only ones. HolmesGPT is about 3x faster at the median. K8sGPT lists what its analyzers flag in a namespace rather than investigating an alert, so it answers a different question; see [HolmesGPT vs K8sGPT](/blog/holmesgpt-vs-k8sgpt). Scenario 13 shows the kind of answer you want posted into an incident channel. Every pod is Running and Ready, and a NetworkPolicy silently drops traffic from the api to redis. The alert was `HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes (threshold 5%).` Aurora, run 1 (202 s, 25 tool calls, $0.69): > The root cause was a NetworkPolicy named `redis` in the shop namespace, created at 2026-10-02T20:23:54Z, which restricts ingress to redis:6379 to pods labeled `app=worker` only [20, 22]. The `api` pods are labeled `app=api`, not `app=worker` [23], so their connections to redis were silently dropped by the policy [...] HolmesGPT, run 1 (110 s): > The api pods (api-f64486c4c-f7fpv, api-f64486c4c-nz5j6, label app=api) are not covered by this allow-list, so their connections to redis:6379 are now dropped by the policy. Both agents got it in both runs; the fix is in the runbook [NetworkPolicy blocking traffic](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes). Where they failed: scenario 12 has two faults and the alert covers only a worker OOMKilled. Both agents found the OOM in both runs and missed that the frontend Service selected no pods, and Aurora wrote that the frontend "remained healthy and unaffected" with pod status as its only evidence. The checks that catch it are in [a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). Every transcript and grade is on [/benchmark](/benchmark). ## How does Aurora work with incident.io? Aurora's incident.io connector is in the open source repo, so this is what the code does, not a plan: 1. **Connect with an incident.io API key.** Aurora checks the key's roles. Posting results back needs `incident_editor` for incident updates or `on_call_editor` for alert notes. 2. **Point an incident.io webhook at Aurora.** Aurora checks incident.io's webhook signature once you save the signing secret; until you do, it accepts unsigned events, so save it. New incidents (public and private) start an investigation, and so do new alerts unless you switch alert RCA off, which helps orgs whose alerts fire constantly. For alerts only, a minimum priority or an allow-list, ranked against your org's own alert priorities, keeps low-priority noise out; declared incidents are not filtered. 3. **Aurora investigates** with its other connectors (cloud accounts, Kubernetes, monitoring, code) and with three read-only incident.io tools: list incidents, get one incident, read its timeline. Its instructions tell it to check past closed incidents for repeats. 4. **The result goes back to incident.io** when post-back is on (it is off by default). At completion, Aurora posts one update on the incident's timeline headed "Aurora RCA", with the root cause, impact, a link to the full investigation and the line "Generated automatically by Aurora. Verify before acting." If the alert never became an incident, the RCA goes on the alert as a note instead. When several alerts are grouped, the first RCA becomes the group's note, later alerts add a short correlation note instead of a full report, and the group note is extended if a later RCA finds something new. 5. **Slack.** If you add Aurora's Slack app to incident.io's incident channels, it can answer @mentions there. So incident.io keeps paging, roles, comms, status pages and the post-incident process, and Aurora adds an investigation your team can read, rerun and run on your own model. The overlap is the root cause step: if you already pay for Investigations, run both on the same incidents for a few weeks and compare. If you run Keep or another alert hub instead, Aurora takes alerts from PagerDuty, Datadog, Grafana, New Relic, OpsGenie and more ([README](https://github.com/Arvo-AI/aurora)). We have not measured Aurora inside incident.io the way we measured it in the lab: the 19 scenarios fed the alert straight to each agent. ## Which incident management tools let me bring my own AI model? | Tool | Your own model or key? | Source | |---|---|---| | incident.io Investigations | "Not currently" for model or key; you can plug your own MCP servers and skills into Nexus | [docs](https://docs.incident.io/investigations/trust-and-safety), [Nexus](https://incident.io/nexus) | | Rootly | "BYO AI Key (OpenAI, Anthropic, Gemini)" on Enterprise | [pricing](https://rootly.com/pricing) | | Keep | OpenAI, Anthropic, Gemini, DeepSeek, Grok, Ollama, LlamaCPP backends | [GitHub](https://github.com/keephq/keep) | | Aurora | OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, OpenRouter, Ollama for air-gapped use | [GitHub](https://github.com/Arvo-AI/aurora) | | HolmesGPT | OpenAI, Anthropic, Azure, Bedrock, Gemini, Ollama and more | [GitHub](https://github.com/HolmesGPT/holmesgpt) | If the requirement is that incident data never reaches an AI vendor at all, the on-call layer is the easy part: GoAlert, Uptime Kuma and Gatus describe no AI features and run on your servers. For investigation, Aurora and HolmesGPT with a local model through Ollama keep everything inside your network. Our lab used a frontier model on Bedrock; we have not tested local models. ## How do I compare open source and commercial incident management tools? Split the job into layers first, then compare per layer. A tool that is excellent at one is usually thin at the others. 1. **On-call and paging.** Delivery is the requirement. incident.io claims "Millions of alerts delivered with 99.99% reliability" ([on-call](https://incident.io/on-call)); with a self-hosted tool such as GoAlert, keeping paging up is your job. Rootly's on-call is another $20 per user a month on top of its $20 Incident Response ([pricing](https://rootly.com/pricing)). 2. **Response workflow.** Roles, severities, channels, updates, follow-ups. This is incident.io's strongest layer and open source's weakest. 3. **Status pages.** Open source is strong here. incident.io's Basic includes one public page; internal pages start on Pro and customer pages need Enterprise. 4. **Postmortems.** incident.io's AI post-mortem editor is on Pro and above. Incident Bot builds a postmortem document base; Aurora generates postmortems from its investigation and exports them to Confluence, Notion or SharePoint. 5. **Investigation.** Ask every vendor for accuracy on faults they did not choose. Ours are public: clone [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab), break a scenario, give a trial agent the alert line from its `scenario.yaml`, and compare the answer with the `must_identify` list in the same file. To diagnose one failure on your own cluster, [/diagnose](/diagnose) walks through it. Then check the boring things: is the repository still maintained (two of the best-known ones are archived), which license applies to the parts you need (Keep's `ee/` folder, Cachet's custom license, openstatus's AGPL if you modify and host it), and who carries the pager for the incident tool itself. | If you need | Look at | |---|---| | Everything in one hosted product with little to run | incident.io, Rootly | | Open source on-call | GoAlert | | Open source status page | openstatus, Uptime Kuma, Gatus, Cachet | | Open source Slack incident channels | Incident Bot, openstatus | | Alert dedup, routing and workflows | Keep | | AI investigation you run yourself, with your model | Aurora, HolmesGPT | | incident.io for response plus an investigation you can audit | incident.io with Aurora | For commercial head-to-heads see [incident.io vs Rootly](/blog/incident-io-vs-rootly) and [FireHydrant vs incident.io](/blog/firehydrant-vs-incident-io); for the wider open source picture, [open source incident management](/blog/open-source-incident-management) and [open source AI SRE tools compared](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt). ## What this comparison does not cover We did not test incident.io, Rootly, Keep, GoAlert or the status page tools; their rows are what their own pages and repositories say. The lab is Kubernetes-only, synthetic, and the faults are ours. HolmesGPT 0.44 and 0.45 were already out when we ran 0.43.0, and K8sGPT 0.4.41 shipped on 2026-10-21, so their current versions are untested. Aurora's $0.35 median is model cost only; you also run the servers. Verified 2026-10-21: incident.io pricing, Investigations, Nexus, agent and on-call pages and docs; Rootly pricing; Elastic's Keep announcement; every GitHub repository linked (license, latest release, last commit, archive status); Aurora's incident.io connector in the [Aurora repo](https://github.com/Arvo-AI/aurora); lab numbers from [results.json](https://raw.githubusercontent.com/Arvo-AI/aurora-lab/main/benchmark/results.json) generated 2026-10-21. Aurora is on [GitHub](https://github.com/Arvo-AI/aurora) under Apache 2.0, or [book a demo](https://cal.com/arvo-ai/demo) to see it investigate your incident.io incidents. --- ## Rootly Alternatives: Open Source and Paid Options (2026) URL: https://www.aurorasre.ai/blog/rootly-alternative-open-source-incident-management Published: 2026-04-02 (updated 2026-10-12) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Alternatives to Rootly for incident management and AI investigation, including the free, open source Aurora. Features and pricing compared side by side. > **Key Takeaway:** Rootly is an AI-native incident management platform with on-call, workflows, and AI SRE agents, starting at $20/user/month with AI SRE priced separately. Aurora is an open source (Apache 2.0) AI agent focused purely on autonomous incident investigation and root cause analysis. Rootly orchestrates your entire incident lifecycle. Aurora automates the hardest part, figuring out *why* something broke. ## What is Rootly? [Rootly](https://rootly.com) describes itself as an "AI-native incident management platform", an all-in-one tool for detecting, managing, learning from, and resolving incidents. Founded in 2021, it's used by teams at NVIDIA, Replit, Canva, Cisco, DoorDash, and [hundreds more](https://rootly.com/customers). Rootly offers three products: - **Incident Response**. Slack/Teams-native workflows, playbooks, roles, status pages, retrospectives - **On-Call**. Schedules, escalation policies, alert routing, live call routing, mobile app - **AI SRE**. Autonomous AI agents for root cause analysis, remediation, and alert triage ## What is Aurora? [Aurora](https://github.com/Arvo-AI/aurora) is an open source AI agent that automates incident investigation. When a monitoring tool fires an alert, Aurora's LangGraph-orchestrated agents autonomously query your infrastructure across AWS, Azure, GCP, OVH, Scaleway, and Kubernetes, correlating data from 25+ tools and delivering a structured root cause analysis with remediation recommendations. Aurora doesn't manage your incident lifecycle. It investigates the root cause. ## How They Compare ### Incident Response & Coordination **Rootly** is a full incident lifecycle platform: - Slack and Microsoft Teams native incident channels - Automated workflows (create channels, page responders, update status) - Incident roles (commander, communications lead, etc.) - Playbooks and runbooks - Status pages (internal and external) - Action item tracking with Jira sync - DORA metrics and advanced analytics - Mobile app (iOS and Android) **Aurora** is not a full incident coordination platform, no roles or status pages. However, Aurora does create and manage Slack incident channels, tracks action items with Jira sync, sends investigation notifications, and supports @Aurora mentions in any channel for conversational investigation. ### On-Call Management **Rootly** has a full on-call product: - Schedules with shadow rotations, holiday calendars, PTO overrides - Escalation policies with gap detection - SMS, voice, push notifications (bypass Do Not Disturb) - Live call routing - On-call pay calculator - [99.99% uptime claim](https://rootly.com) **Aurora** has no on-call capabilities. No schedules, no paging, no escalation. For on-call, use Rootly, PagerDuty, Grafana OnCall, or Opsgenie alongside Aurora. ### AI Investigation This is where the tools diverge most. **Rootly AI SRE** ([rootly.com/ai-sre](https://rootly.com/ai-sre)): - Correlates alerts with code changes, deploys, and config changes - Generates root cause analysis with confidence scores - Surfaces similar past incidents and proven solutions - Drafts remediation steps and PRs with suggested fixes - AI Meeting Bot that transcribes incident bridges in real time - @Rootly AI chat in Slack/Teams for summaries and task assignment - [MCP server](https://rootly.com/blog/rootly-mcp-goes-ga-up-to-95-less-tokens) for IDEs (Cursor, Windsurf, Claude Code) - Chain-of-thought visibility ("see *why* a root cause is flagged") - Whether it directly queries cloud infrastructure APIs is unverified **Aurora AI Investigation**: - Autonomous multi-step investigation using LangGraph-orchestrated agents - Dynamically selects from 30+ tools per investigation - Executes `kubectl`, `aws`, `az`, `gcloud` commands in **sandboxed Kubernetes pods** (non-root, read-only filesystem, capabilities dropped, seccomp enforced) - Queries cloud APIs directly. AWS (STS AssumeRole), Azure (Service Principal), GCP (OAuth), OVH, Scaleway - Traverses Memgraph infrastructure dependency graph for blast radius - Searches Weaviate knowledge base (vector search over runbooks, past postmortems) - Generates structured RCA with timeline, evidence citations, and remediation - Suggests code fixes with diff preview, human approves and creates PR via GitHub and Bitbucket - Exports postmortems to Confluence and Jira ### Knowledge Base **Rootly**: Surfaces similar past incidents during investigations. Integrates with [Glean](https://rootly.com/integrations) for broader knowledge search. No native vector search product. **Aurora**: Built-in Weaviate-powered vector store. Upload runbooks, past postmortems, and documentation, the AI agent searches them using semantic similarity during every investigation. ### Postmortems **Rootly**: AI-generated retrospectives with context, timelines, and custom templates. Collaborative editing. Jira sync for action items. **Aurora**: AI-generated postmortems with timeline, root cause, impact assessment, and remediation steps. One-click export to Confluence and Jira. ## Feature Comparison **Rootly has, Aurora doesn't:** - On-call scheduling, escalation policies, paging (SMS/voice/push) - Microsoft Teams support (Aurora is Slack-only) - Automated incident workflows (create channels, page responders, update status) - Status pages (internal and external) - Incident roles - DORA metrics and analytics - Mobile app (iOS, Android) - MCP server for IDEs - AI Meeting Bot for incident bridges - SOC 2 Type II, HIPAA, GDPR, CCPA compliance - A broad integration catalog **Aurora has, Rootly doesn't (or is unverified):** - Direct cloud infrastructure querying (AWS, Azure, GCP, OVH, Scaleway APIs) - CLI command execution in sandboxed Kubernetes pods - Native vector search knowledge base (Weaviate RAG) - Infrastructure dependency graph (Memgraph) - Terraform/IaC state analysis - Open source (Apache 2.0), full codebase auditable - Self-hosted deployment (Docker Compose, Helm) - LLM provider flexibility including local models (Ollama for air-gapped) - Free, no per-user or per-incident pricing **Both have:** - AI-powered root cause analysis - AI-suggested code fixes (human-approved PRs) - Automated postmortem generation - PagerDuty, Datadog, Grafana integrations - GitHub integration - Confluence integration - HashiCorp Vault integration - BYOK for LLM providers (Rootly: OpenAI, Anthropic, Gemini; Aurora: OpenAI, Anthropic, Google, Ollama, OpenRouter, Vertex AI) - Slack incident channels - Action item tracking with Jira sync ## Pricing **Rootly** ([rootly.com/pricing](https://rootly.com/pricing)): - Incident Response Essentials: **$20/user/month** - On-Call Essentials: **$20/user/month** - AI SRE: **Contact sales** (no published price) - Enterprise tiers: Contact sales - Bundle discounts available for IR + On-Call + AI SRE - Startup discount: up to 50% off (<100 employees, <$50M raised) - Free 14-day trial **Aurora**: - **Free**. Apache 2.0, self-hosted - Costs: infrastructure (VM or K8s cluster) + LLM API usage - $0 LLM cost possible with Ollama local models **Example: 20-person SRE team** For Rootly IR + On-Call: $20 + $20 = $40/user/month x 20 = **$800/month** (before AI SRE add-on, which is priced separately via sales). For Aurora: **$0** + infrastructure + LLM API. > Note: Rootly pricing from [rootly.com/pricing](https://rootly.com/pricing). AI SRE pricing is not publicly listed. ## Open Source vs SaaS **Rootly** is SaaS-only. The core platform is proprietary. They have [open source tooling on GitHub](https://github.com/rootlyhq) (Terraform provider with 400,000+ downloads, Backstage plugin, CLI, SDKs) but not the platform itself. **Aurora** is fully open source under Apache 2.0. The entire codebase, backend, frontend, agent orchestration, is on [GitHub](https://github.com/Arvo-AI/aurora). You can: - Audit exactly what the AI does on your infrastructure - Modify investigation workflows and add custom tools - Fork and customize for your organization - Run fully air-gapped with local LLMs via Ollama - Keep all incident data in your own environment ## When to Choose Rootly Rootly is the better choice when: - **You need a full incident lifecycle platform**, on-call, workflows, status pages, roles, retrospectives, DORA metrics in one tool - **Slack/Teams-native workflows matter**. Rootly's incident channels and AI chat are deeply embedded in collaboration tools - **Compliance requirements**. SOC 2 Type II, HIPAA, GDPR out of the box - **You want managed SaaS**, no infrastructure to maintain - **You need a mobile app**, iOS and Android for on-call - **Enterprise support**, dedicated support, SLAs, BAA for HIPAA ## When to Choose Aurora Aurora is the better choice when: - **Investigation is your bottleneck**, your team spends hours diagnosing incidents manually - **You need deep cloud investigation**. AI agents that directly query AWS, Azure, GCP, and Kubernetes - **You want open source**, full transparency into how AI investigates your infrastructure - **Self-hosted is required**, compliance, data sovereignty, or air-gapped environments - **Budget is limited**, free forever, no per-user pricing - **LLM flexibility matters**, bring any provider, including local models - **You already have on-call**. PagerDuty, Grafana OnCall, or Opsgenie handles paging; you need the investigation layer - **You want a custom integration**. Aurora is open source and the Arvo AI team actively builds custom integrations for companies that need them, at no cost. If there's a feature gap, [reach out](https://cal.com/arvo-ai) and they'll build it with you. ## Using Rootly + Aurora Together They're not mutually exclusive. Rootly manages your incident lifecycle; Aurora investigates the root cause: 1. **Alert fires** → Rootly creates incident channel, pages on-call 2. **Same alert** → Aurora receives webhook, starts AI investigation 3. **Rootly** coordinates the response (roles, comms, status page) 4. **Aurora** investigates in the background (queries cloud, checks K8s, searches knowledge base) 5. **On-call SRE** finds Aurora's completed RCA with root cause and remediation 6. **Aurora** generates postmortem → exports to Confluence 7. **Rootly** tracks action items → syncs to Jira ## Getting Started with Aurora ```bash git clone https://github.com/arvo-ai/aurora.git && cd aurora make init # generates secrets, copies .env.example to .env nano .env # add OPENROUTER_API_KEY, OPENAI_API_KEY or ANTHROPIC_API_KEY make prod-prebuilt # pulls prebuilt images from GHCR and starts ``` Configure your monitoring webhooks (PagerDuty, Datadog, Grafana), add cloud provider credentials, and investigations start automatically. See the [full documentation](https://arvo-ai.github.io/aurora/) for deployment guides. Learn more at [aurorasre.ai](https://www.aurorasre.ai) or read the [full documentation](https://arvo-ai.github.io/aurora/). For comparisons with other tools, see [Aurora vs Traditional Incident Management Tools](/blog/aurora-vs-traditional-incident-management-tools) and [PagerDuty Alternative for Root Cause Analysis](/blog/pagerduty-alternative-root-cause-analysis). For an open-source-to-open-source comparison, see [Aurora vs HolmesGPT vs K8sGPT](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) > **Every claim in this post is sourced from official websites and public repositories.** Rootly data from [rootly.com](https://rootly.com). Aurora data from [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora). Last verified: April 2026. --- ## Resolve AI Alternatives 2026: Real Prices, 3 Tested in a Lab URL: https://www.aurorasre.ai/blog/resolve-ai-alternative-open-source Published: 2026-04-02 (updated 2026-10-21) Author: Arvo AI Team, Engineering > Resolve AI publishes no prices. 8 alternatives and their published pricing, checked Oct 21, 2026, plus lab results for the open source ones on 19 faults. The main alternatives to Resolve AI are other hosted AI SRE agents that publish their pricing (Cleric from $700 a month, Datadog Bits Investigation at about 6.5 AI credits per investigation; NeuBird lists 1 credit per investigation but no price per credit) and open source agents you run yourself (Aurora and HolmesGPT, with K8sGPT as a cluster scanner). Resolve AI sells through sales: its [pricing page](https://resolve.ai/pricing) is a request form and its credits per action are set in each contract. The open source agents are the only ones here we could test ourselves: on 19 injected Kubernetes faults, Aurora was graded correct in 34 of 38 runs and HolmesGPT in 31 of 38, and HolmesGPT was about 3x faster. Arvo builds Aurora and wrote the lab scenarios. We cannot install Resolve AI or the other hosted products, so nothing below is a test of them; every claim about them links the vendor's own page, read on 2026-10-21. ## What is Resolve AI? [Resolve AI](https://resolve.ai/) is a hosted AI SRE platform. Its [product overview](https://resolve.ai/product/overview) describes agents that triage every alert, mitigate critical incidents, remediate recurring failures and run scheduled production tasks. [The company](https://resolve.ai/about-us) was founded in October 2024 by Spiros Xanthos and Mayank Agarwal, who led Splunk Observability and co-created OpenTelemetry, and it states a $1.5B valuation and more than $190M in funding. What the vendor's own pages say it does: - **Investigation.** It "plans investigations that pursue multiple hypotheses in parallel" and builds causal timelines linking code changes, infrastructure events and telemetry ([docs](https://docs.resolve.ai/about-resolve-ai)). Alerts run in a triage, investigation or adaptive mode, and adaptive mode only escalates to a full investigation when needed ([billing docs](https://docs.resolve.ai/billing-and-credit-controls.md)). - **Actions.** Mitigation actions such as silencing an alert or reverting a commit are proposals that a person must approve; "Resolve never has direct access to execute write APIs on your monitoring platforms" ([docs](https://docs.resolve.ai/mitigation-actions.md)). Code changes come as PRs that also need approval. - **Models.** "Our own domain models and the best frontier models" ([overview](https://resolve.ai/product/overview)), with per-customer fine-tuning: "Models are tailored specifically to your data and needs" ([security](https://resolve.ai/security)). - **Integrations.** 60+ pre-built integrations. The [integrations page](https://resolve.ai/integrations) lists AWS, Google Cloud, Microsoft Azure, Kubernetes, Datadog, Grafana, Splunk, Dynatrace, Elastic, New Relic, Honeycomb, Sentry, Logz.io, PagerDuty, Opsgenie, ServiceNow, Slack, Microsoft Teams, Confluence, Notion and more, plus MCP, APIs and webhooks. - **Deployment.** Two models: "Shared", a multi-tenant environment with a satellite, and "Dedicated", a single-tenant cloud deployment with bring-your-own keys ([security](https://resolve.ai/security)). The [Resolve Satellite](https://docs.resolve.ai/resolve-satellite.md) runs in your Kubernetes cluster or on ECS Fargate and proxies queries back to Resolve. - **Compliance.** SOC 2 Type II, GDPR and HIPAA ([security](https://resolve.ai/security)). Customer results it publishes include Coinbase ("surfaced accurate root causes 72% faster than our teams") and DoorDash ("from hours to minutes for investigations in many incidents") on its [customers page](https://resolve.ai/customers). Its [about page](https://resolve.ai/about-us) also claims 95% RCA accuracy. We found no published test set or scoring method behind that figure, so it cannot be checked from outside. Resolve AI (resolve.ai) is not Resolve Systems at [resolve.io](https://resolve.io/), which sells workflow automation for IT teams. ## How much does Resolve AI cost? Resolve AI does not list a price. The [pricing page](https://resolve.ai/pricing) describes the model and asks for your details: - **Credit-based.** "Credits are used as ResolveAI agents perform defined types of work across your production environment." - **Five billable actions:** chats, tasks, investigations, incidents and triages; tasks are metered as chats unless your contract lists them separately ([billing docs](https://docs.resolve.ai/billing-and-credit-controls.md)). - **Per-action cost is in the contract.** "How each action is metered, and how many credits it costs, is defined in your contract with Resolve." - **Caps.** Admins can set a monthly cap that pauses automated work. By default it "is not a hard spend stop", because work a person starts keeps running; a "Block manual work" option makes it apply to all usage. - **Buying.** The site's "Try now" button goes to the demo booking page. It is also listed on [AWS Marketplace](https://resolve.ai/integrations). So for a budget you need a quote. The alternatives below that publish prices let you estimate before a call. ## Who needs a Resolve AI alternative, and why? - **Teams that need a price before a sales cycle.** A small team may not be able to open a procurement process just to learn whether a tool fits. Cleric and Datadog publish their unit prices and NeuBird its credits per run; the open source agents cost what your model calls and hosting cost. - **Teams that need the agent inside their own network.** Resolve offers a shared multi-tenant environment or a dedicated single-tenant cloud deployment, with the satellite as the gateway inside your network. If the reasoning layer itself must run in your VPC or an air-gapped network, look at Aurora, HolmesGPT, Traversal's private deployment or NeuBird's VPC option. - **Teams that want to pick the model.** Resolve combines its own models with frontier models of its choice. Aurora, HolmesGPT and K8sGPT let you set the provider, including local models through Ollama; Traversal advertises bring-your-own-model. - **Teams that want to verify accuracy themselves.** You cannot read Resolve's agent or rerun its 95% figure. With an open source agent you can read the prompts and every tool call, and rerun our 19 scenarios. - **Teams already paying for one observability vendor.** If everything lives in Datadog, Bits Investigation is bought in Datadog AI credits on the same bill. See our [Datadog Bits AI alternatives](/blog/datadog-bits-ai-sre-alternative-open-source) page for its pricing. ## Resolve AI alternatives compared Checked against each vendor's site or repository on 2026-10-21. "Lab-tested" means we ran it on our 19 faults; the others we have not run. | Tool | Type | Published price | Where it runs | Model choice | Lab-tested | |---|---|---|---|---|---| | [Resolve AI](https://resolve.ai/) | Hosted AI SRE | None; [request pricing](https://resolve.ai/pricing) | Shared multi-tenant or dedicated single-tenant cloud, plus satellite | Resolve's own and frontier models | No | | [Aurora](https://github.com/Arvo-AI/aurora) | Open source agent, Apache 2.0 | Free; you pay for hosting and LLM calls | Docker Compose, Helm, air-gapped bundle | OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, OpenRouter, Ollama | Yes | | [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) | Open source agent, Apache 2.0, CNCF Sandbox | Free; you pay for LLM calls | CLI, server, Kubernetes | OpenAI, Anthropic, Azure, Bedrock, Gemini and more | Yes | | [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) | Open source Kubernetes scanner, Apache 2.0 | Free; you pay for LLM calls | CLI, in-cluster operator | OpenAI, Azure, Bedrock, Gemini, LocalAI, Ollama and more | Yes, as a scanner | | [Datadog Bits Investigation](https://www.datadoghq.com/pricing/?product=bits-ai-sre) | Hosted, inside Datadog | About 6.5 AI credits per investigation; credits from $500 per 500 a month on annual commit, or $1.30 each on demand | Datadog SaaS | Datadog-managed | No | | [Cleric](https://cleric.ai/pricing) | Hosted AI SRE | Team $700 a month for 1,400 credits; "$5 to triage and address an alert" | Dedicated Cleric instance; private resources through the Cleric Connector (Pro) | Provider enterprise APIs | No | | [NeuBird](https://neubird.ai/pricing) | Hosted AI SRE | Investigation Agent 1 credit per run; no price per credit published; 14-day trial with 50 credits | SaaS, or your VPC or VNET on Pro and Enterprise | Approved frontier models, including your own Bedrock, Azure OpenAI or Vertex AI ([model access](https://neubird.ai/platform/governed-model-access)) | No | | [Traversal](https://www.traversal.com/security) | Hosted AI SRE for enterprises | None; [pricing walkthrough](https://www.traversal.com/get-pricing) on request | Private, on-premises deployments; bring your own cloud | "Run Traversal with your preferred LLMs, including self-hosted or customer-managed models" | No | | [Grafana Assistant Investigations](https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/platform/investigation/) | Investigation mode in Grafana Cloud | Counts against Grafana Assistant usage limits | Grafana Cloud stack | Grafana-managed; Auto, Small, Medium or Large tier | No | ## How did the open source alternatives do on real faults? We built a small app (an nginx frontend, a Python api, a worker, redis and a load generator) on a 3-node kind v0.33.0 cluster running Kubernetes v1.37.0, injected one known fault per scenario (two in scenario 12) and gave Aurora and HolmesGPT the alert; K8sGPT takes no alert and scanned the namespace. Each tool ran twice per scenario, 38 runs per tool. All three got the same admin kubeconfig, each behind its own guard. Aurora (commit 87d22baf, under its default read-only command policy) and HolmesGPT 0.43.0 (`holmes ask`, default toolsets) both used `bedrock/us.anthropic.claude-sonnet-5`. K8sGPT 0.4.40 ran `k8sgpt analyze --explain --namespace shop` on `us.anthropic.claude-sonnet-4-5`, because Bedrock rejects the temperature parameter it sends to Sonnet 5. Aurora's runs are from 2026-10-02, HolmesGPT's and K8sGPT's from 2026-10-15 and 2026-10-16, on the same cluster and faults. | | Aurora | HolmesGPT | K8sGPT | |---|---|---|---| | Correct | 34 of 38 | 31 of 38 | not scored as a peer | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 | | "No problems detected" | 0 | 0 | 12 runs (6 scenarios) | | Median time per run | 153.5 s | 45 s | 12.5 s for two analyze passes | | Median model cost per run | $0.35 | not captured | not captured | | Who confirmed the grades | a person at Arvo | judge model only | judge model only | A judge model (`us.anthropic.claude-opus-5-5`) graded every answer against the injected fault. A person at Arvo read all 38 Aurora grades and changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model re-read all 76 and agreed with every one, a third pass graded them from scratch and agreed on all 38 HolmesGPT verdicts while disputing 6 K8sGPT labels, and no person has checked them yet. A blinded regrade, with tool names removed from the judge's prompt, matched the published verdict in 38 of 38 Aurora runs, 37 of 38 HolmesGPT runs and 36 of 38 K8sGPT runs. The two prompts differ in wording, and HolmesGPT's own allow-list blocked `kubectl rollout history` in 12 of its runs; Aurora's default policy blocked the same command in its runs. The [/benchmark](/benchmark) page computes a two-sided Fisher exact test on the totals, p = 0.52, so the accuracy gap between Aurora and HolmesGPT is not significant, and it compares human-read grades with judge-only ones. HolmesGPT is clearly faster. K8sGPT answers a different question. It scans a namespace and explains what its analyzers flag; it does not take an alert or follow evidence. It is the fastest health scan of the three, and the wrong tool for "why did this alert fire?". More in [HolmesGPT vs K8sGPT](/blog/holmesgpt-vs-k8sgpt). ### One fault, three answers Scenario 13 is the kind of fault a vendor demo rarely shows: every pod is Running and Ready, and a NetworkPolicy silently drops traffic from the api to redis. The alert was `HighErrorRate [critical] namespace=shop service=frontend: 100% of requests to /api/ returned 5xx over the last 5 minutes (threshold 5%).` Aurora, run 1 (202 s, 25 tool calls, $0.69): > The root cause was a NetworkPolicy named `redis` in the shop namespace, created at 2026-10-02T20:23:54Z, which restricts ingress to redis:6379 to pods labeled `app=worker` only [20, 22]. The `api` pods are labeled `app=api`, not `app=worker` [23], so their connections to redis were silently dropped by the policy [...] HolmesGPT, run 1 (110 s): > The api pods (api-f64486c4c-f7fpv, api-f64486c4c-nz5j6, label app=api) are not covered by this allow-list, so their connections to redis:6379 are now dropped by the policy. K8sGPT, run 1: ``` No problems detected ``` Both agents got it in both runs. HolmesGPT took 110 s and 118 s, Aurora 202 s and 444 s (its second run waited for a worker slot behind a background discovery job). The fix steps are in the runbook [NetworkPolicy blocking traffic](/runbooks/fix-networkpolicy-blocking-traffic-kubernetes). ### Where they failed Scenario 12 has two faults; the alert covers only a worker OOMKilled. Both agents found the OOM in both runs and missed that the frontend Service selected no pods. Aurora also wrote that the frontend was healthy: > Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]. Pod status was its only evidence. The checks that catch it are in [a second fault the alert does not cover](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). Every transcript and grade, run by run, is on [/benchmark](/benchmark). ## How does Grafana compare to Resolve AI for autonomous incident remediation? Grafana's equivalent is [Grafana Assistant Investigations](https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/platform/investigation/), for incidents that need analysis across metrics, logs, traces and profiles. You describe the problem in Assistant Workspace, or alerting and IRM webhooks start one automatically; it tests hypotheses and writes a report with findings and recommended next steps. It needs a Grafana Cloud stack: per [Grafana's docs](https://grafana.com/docs/grafana-cloud/platform/grafana-assistant/), investigations are "available only with the full Grafana Cloud feature set", so self-managed Grafana alone does not get them. Resolve, by contrast, reads Grafana as one of many sources and proposes approved mitigations such as alert silences. If your telemetry is in self-hosted Grafana, Prometheus and Loki, an open source agent that queries them directly is the closer fit; Aurora takes Grafana alerts and HolmesGPT has Grafana, Loki, Tempo and Prometheus toolsets. ## Is Resolve AI compatible with all cloud providers? Resolve's integrations page lists AWS, Google Cloud and Microsoft Azure, plus Kubernetes through the satellite. Its [Azure integration](https://docs.resolve.ai/azure.md) is read-only and "covers Azure Commercial Cloud only", reading Azure Monitor, Application Insights and Entra logs. For code it supports github.com, GitHub Enterprise, GitLab, Bitbucket, Azure DevOps and self-hosted Git, with Git commands run either in Resolve's cloud or on your satellite ([docs](https://docs.resolve.ai/git.md)). Aurora runs CLI investigations on AWS, Azure, GCP, OVH, Scaleway and Kubernetes. ## How to test any Resolve AI alternative before you sign Vendors show you their best incidents. Bring yours, or bring ours: every scenario in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab) has the alert, the true root cause and the findings an answer must contain. ``` git clone https://github.com/Arvo-AI/aurora-lab && cd aurora-lab ./lab.sh up ./lab.sh break 13-networkpolicy-blocks-redis ``` Connect the trial agent to the cluster, give it the `alert` line from `scenarios/13-networkpolicy-blocks-redis/scenario.yaml`, and compare its answer with the `must_identify` list in the same file. Scenarios 11, 12 and 19 are the ones both agents found hardest. `./lab.sh reset` restores the app. To diagnose one failure on your own cluster, [/diagnose](/diagnose) walks through it. ## Which Resolve AI alternative fits which team? | If you need | Look at | Why | |---|---|---| | A known price and a hosted agent | Cleric | $700 a month entry plan, unlimited users | | An agent inside the Datadog bill | Datadog Bits Investigation | About 6.5 credits per investigation; third-party sources in Preview | | The reasoning layer in your own VPC or air-gapped | Aurora, HolmesGPT, Traversal, NeuBird VPC | Self-hosted or private deployment | | To choose and swap the model | Aurora, HolmesGPT, Traversal | Provider set in your config, local models possible | | Accuracy you can verify before buying | Aurora, HolmesGPT | Open code, public lab runs | | A fast Kubernetes health scan | K8sGPT | Analyzer-based, 12.5 s median | | Enterprise scale with vendor support and compliance | Resolve AI, Traversal | SOC 2 Type II, HIPAA, dedicated deployments | ## What this comparison does not cover The lab is Kubernetes-only, synthetic, and the faults are ours. None of the hosted products ran it, so their rows are what the vendors publish, not what we measured. HolmesGPT 0.44 and 0.45 were already out when we ran 0.43.0, and K8sGPT 0.4.41 shipped on 2026-10-21, so their current versions are untested. Aurora's $0.35 median is model cost only: you also run the servers, and Aurora's background discovery cost about $1 per cluster connection in the lab ([REPORT.md](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md)). Verified 2026-10-21: Resolve AI pages and docs, Datadog pricing, Cleric, NeuBird and Traversal pages, Grafana docs and GitHub repos as linked; lab numbers from [results.json](https://raw.githubusercontent.com/Arvo-AI/aurora-lab/main/benchmark/results.json) generated 2026-10-21. Aurora is on [GitHub](https://github.com/Arvo-AI/aurora) under Apache 2.0, or [book a demo](https://cal.com/arvo-ai/demo) to see it on your alerts. --- ## PagerDuty Alternative for Root Cause Analysis (2026) URL: https://www.aurorasre.ai/blog/pagerduty-alternative-root-cause-analysis Published: 2026-04-01 (updated 2026-07-28) Author: Noah Casarotto-Dinning, CEO at Arvo AI > PagerDuty handles alerting and on-call, but who finds the root cause? Aurora is an open source AI agent that investigates incidents automatically. > **Key Takeaway:** PagerDuty is the industry standard for alerting and on-call management, but it doesn't investigate *why* incidents happen. Aurora is an open source AI agent that plugs into PagerDuty via webhooks and autonomously investigates root causes across AWS, Azure, GCP, and Kubernetes. They're complementary tools, but for teams spending hours on manual RCA, Aurora fills the gap PagerDuty doesn't cover. PagerDuty has over [30,000 customers](https://www.pagerduty.com) and dominates on-call management. It's excellent at what it does: detecting alerts, routing them to the right person, coordinating incident response, and tracking SLAs. But here's the problem: **PagerDuty pages you. Then you're on your own.** The actual investigation. SSHing into servers, querying CloudWatch, checking Kubernetes pod logs, correlating deployments with error spikes, is still manual. According to the [VOID (Verica Open Incident Database)](https://www.thevoid.community/), the median incident involves 3.5 contributing factors, and the investigation phase consumes the majority of mean time to resolve (MTTR). This is the gap Aurora fills. ## PagerDuty vs Aurora: Different Tools, Different Jobs This isn't a "which is better" comparison. PagerDuty and Aurora solve different problems: | | PagerDuty | Aurora | |---|---|---| | **Primary job** | Alert routing, on-call, coordination | Root cause investigation | | **Answers the question** | "Who needs to know and how do we coordinate?" | "Why did this happen and what should we fix?" | | **Trigger** | Monitoring tool fires alert | PagerDuty webhook (or Datadog, Grafana, etc.) | | **Output** | Engineer gets paged, war room opens | Structured RCA with timeline, root cause, remediation | **They work together.** Aurora ingests PagerDuty `incident.triggered` webhooks. When PagerDuty pages your SRE, Aurora is already investigating in the background. ## What PagerDuty Does Well PagerDuty's strengths are real and well-established: - **On-call scheduling**. Flexible rotations, escalation policies, shift overrides - **Alert routing**, [700+ integrations](https://www.pagerduty.com/integrations/) for ingesting alerts from every monitoring tool - **Multi-channel paging**. SMS, phone, push notifications, email - **Incident coordination**. War rooms, stakeholder communications, status pages - **SLA tracking**. Urgency-based alerting and escalation - **AI noise reduction**, [AIOps add-on](https://www.pagerduty.com/platform/aiops/) claims 91% alert noise reduction via intelligent correlation and deduplication PagerDuty has also added AI features through [PagerDuty Advance](https://www.pagerduty.com/platform/aiops/), including: - AI incident summaries ("catch me up" in Slack) - AI-generated status updates - AI postmortem drafts (Beta) - SRE Agent for triage and approved remediation actions - Probable Origin for pattern-based root cause suggestions ## Where PagerDuty Stops Despite the AI additions, PagerDuty's investigation capabilities have limits: **No autonomous multi-step investigation.** PagerDuty's SRE Agent surfaces past incidents and patterns, but it doesn't autonomously query your AWS accounts, check Kubernetes pod status, correlate Terraform changes, or trace dependency graphs. The investigation itself is still on the engineer. **No native cloud infrastructure querying.** PagerDuty receives alerts *from* CloudWatch, Azure Monitor, etc., it doesn't query them directly. It can't run `kubectl get pods` or `aws cloudwatch get-metric-data` on your behalf during an investigation. **No knowledge base with vector search.** PagerDuty's RAG capability is partial, it requires configuring [Amazon Q Business](https://www.pagerduty.com/integrations/) as an external integration. There's no native vector search over your runbooks and past postmortems. **No code fix suggestions.** PagerDuty can surface recent code changes that may be related to an incident, but it doesn't generate remediation code or create pull requests. **AI features are paid add-ons.** AIOps starts at [$699/month](https://www.pagerduty.com/pricing/). PagerDuty Advance starts at [$415/month](https://www.pagerduty.com/pricing/). These are on top of per-user pricing ($21-$41+/user/month depending on tier). ## What Aurora Does Differently [Aurora](https://github.com/Arvo-AI/aurora) is an open source (Apache 2.0) AI agent that automates the investigation phase, the part that happens *after* you get paged. ### Autonomous Investigation When Aurora receives an alert webhook, its LangGraph-orchestrated AI agents: 1. Analyze the alert context (severity, service, timing) 2. Dynamically select from 30+ tools to investigate 3. Execute `kubectl`, `aws`, `az`, `gcloud` commands in **sandboxed Kubernetes pods** 4. Query logs, metrics, and recent deployments across cloud providers 5. Search the knowledge base for relevant runbooks and past incidents 6. Traverse the infrastructure dependency graph for blast radius 7. Synthesize everything into a structured root cause analysis No human in the loop during investigation. The SRE gets paged by PagerDuty and finds a completed RCA waiting in Aurora. ### Multi-Cloud Native Aurora connects directly to your cloud infrastructure: | Provider | Authentication | |----------|---------------| | **AWS** | STS AssumeRole (temporary credentials) | | **Azure** | Service Principal | | **GCP** | OAuth | | **OVH** | API key | | **Scaleway** | API token | | **Kubernetes** | Kubeconfig via outbound WebSocket agent | ### 25+ Verified Integrations | Category | Tools | |----------|-------| | **Monitoring** | PagerDuty, Datadog, Grafana, New Relic, Netdata, Dynatrace, Coroot, ThousandEyes, BigPanda, Splunk | | **Cloud** | AWS, Azure, GCP, OVH, Scaleway | | **Infrastructure** | Kubernetes, Terraform, Docker | | **CI/CD** | GitHub, Bitbucket, Jenkins, CloudBees, Spinnaker | | **Docs & Knowledge** | Confluence, Jira, SharePoint | | **Network** | Cloudflare, Tailscale | | **Communication** | Slack | ### Knowledge Base with RAG Aurora includes a built-in Weaviate-powered vector store. Upload your runbooks, past postmortems, and documentation, the AI agent searches them during every investigation using semantic similarity, not just keyword matching. ### AI Code Fix Suggestions Aurora identifies code-level root causes and suggests fixes with a diff preview. Engineers review the suggested change and create a PR with one click via GitHub and Bitbucket integrations. Aurora doesn't push code autonomously, the human approves every fix. ### Automated Postmortems Structured postmortem documents generated automatically with: - Incident timeline with timestamps - Root cause identification with evidence and citations - Impact assessment - Remediation steps (taken and recommended) - One-click export to Confluence or Jira ## Feature Comparison | Feature | PagerDuty | Aurora | |---------|-----------|--------| | **On-call scheduling** | Yes (core) | No | | **Alert routing & escalation** | Yes (core) | No | | **SMS/phone/push paging** | Yes (core) | No | | **Status pages** | Yes (add-on, [from $89/mo](https://www.pagerduty.com/pricing/)) | No | | **SLA/SLO tracking** | Yes | No | | **Autonomous AI investigation** | Partial (SRE Agent for triage) | Yes (full multi-step) | | **Native cloud querying** | No (receives alerts) | Yes (AWS, Azure, GCP, OVH, Scaleway) | | **CLI execution on infra** | Via [Runbook Automation add-on](https://www.pagerduty.com/platform/automation/) | Yes (sandboxed K8s pods) | | **Knowledge base (RAG)** | Via Amazon Q Business integration | Yes (native Weaviate) | | **Infrastructure graph** | No | Yes (Memgraph) | | **AI postmortems** | Beta (via Jeli) | Yes (with Confluence export) | | **AI code fix PRs** | No | Yes (GitHub, Bitbucket) | | **Open source** | No (Rundeck only) | Yes (Apache 2.0) | | **Self-hosted** | No (SaaS only) | Yes (Docker, Helm) | | **LLM provider choice** | No (undisclosed, fixed) | Yes (OpenAI, Anthropic, Google, Ollama) | | **Integrations** | [700+](https://www.pagerduty.com/integrations/) | 25+ | | **Pricing** | [From $21/user/mo](https://www.pagerduty.com/pricing/) + AI add-ons ($415-$699/mo) | Free (self-hosted) | ## Cost Comparison For a team of 20 SREs on PagerDuty Business with AI features: | Line Item | PagerDuty | Aurora | |-----------|-----------|--------| | Base platform | $41/user/mo x 20 = **$820/mo** | $0 | | AIOps | **$699/mo** | Included | | PagerDuty Advance (GenAI) | **$415/mo** | Included | | Status pages | **$89/mo** | Not included | | **Total** | **~$2,023/mo** | **$0 + infra + LLM API** | Aurora's costs are infrastructure (a VM or K8s cluster) and LLM API usage. With Ollama running local models, the LLM cost is also $0. > Note: PagerDuty pricing verified from [pagerduty.com/pricing](https://www.pagerduty.com/pricing/) as of March 2026. Aurora is free under Apache 2.0. ## When to Use PagerDuty + Aurora Together The strongest setup is running both: 1. **PagerDuty** receives alerts from your monitoring tools (Datadog, CloudWatch, Grafana) 2. **PagerDuty** pages the right on-call engineer via SMS/phone 3. **Aurora** receives the same alert via PagerDuty webhook (`incident.triggered`) 4. **Aurora's AI agents** investigate autonomously in the background 5. **The on-call SRE** opens Aurora and finds a completed RCA with root cause, timeline, and remediation 6. **Aurora** generates the postmortem and exports it to Confluence PagerDuty handles the *who* and *when*. Aurora handles the *why* and *how to fix it*. ## When Aurora Alone Might Be Enough For smaller teams or budget-conscious organizations: - **You don't need enterprise on-call**. Your team is small enough that a simple rotation works - **You already have alerting**. Datadog, Grafana, or CloudWatch can send webhooks directly to Aurora - **Investigation is your bottleneck**. You're spending more time diagnosing than coordinating - **You need self-hosted**. Compliance or security requires keeping incident data on-premise - **Budget is limited**. PagerDuty + AI add-ons at $2,000+/mo isn't feasible Aurora can ingest webhooks directly from any monitoring tool. PagerDuty is not required. ## Getting Started ```bash git clone https://github.com/arvo-ai/aurora.git && cd aurora make init # generates secrets, copies .env.example to .env nano .env # add OPENROUTER_API_KEY, OPENAI_API_KEY or ANTHROPIC_API_KEY make prod-prebuilt # pulls prebuilt images from GHCR and starts ``` Configure your PagerDuty webhook to point at Aurora, add your cloud provider credentials, and investigations start automatically. Learn more at [aurorasre.ai](https://www.aurorasre.ai) or read the [full documentation](https://arvo-ai.github.io/aurora/). For a comparison with other tools, see [Aurora vs Traditional Incident Management Tools](/blog/aurora-vs-traditional-incident-management-tools). To understand how AI investigation works, read [What is Agentic Incident Management?](/blog/what-is-agentic-incident-management). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## AI Root Cause Analysis vs 5 Whys for SREs: 3 Tools Tested URL: https://www.aurorasre.ai/blog/root-cause-analysis-complete-guide-sres Published: 2026-03-14 (updated 2026-10-21) Author: Ben Gervais, Engineer at Arvo AI > AI root cause analysis vs 5 Whys: how SREs do RCA in distributed systems, and how 3 automated RCA tools did on 19 injected Kubernetes faults (Oct 2026). AI root cause analysis is an agent that takes the alert, runs read-only queries against the cluster and the logs, and returns a cited hypothesis in minutes. The 5 Whys is a team method that follows one causal chain from the symptom down to a condition it can remove; it works when the incident has one chain and fails when it has several. Neither one proves causation on its own, and they fail the same way: both stop at the first chain that explains the alert. In our lab one AI agent named the right cause in 34 of 38 runs; in three of its 4 partial answers it explained the alerted fault and never checked a second change, and in the fourth it had the evidence but stopped one proof short. The evidence below comes from the public [Aurora lab](https://github.com/Arvo-AI/aurora-lab): 19 faults injected one at a time into a small Kubernetes app, each investigated twice from the alert text alone, with every tool call and report published. Arvo builds Aurora and wrote the scenarios, so read the Aurora numbers with that in mind. ## Which RCA method should an SRE use, and when does each fail? Use a timeline on every incident, change correlation first inside it, the 5 Whys when one chain explains the impact, and a fault tree when the failure needed several conditions at once. An AI agent speeds up the evidence for all four but inherits the same blind spots. | Method | How it works | Works when | Fails when | What we saw in the lab | | --- | --- | --- | --- | --- | | 5 Whys | Ask why until you reach a cause you can eliminate | One causal chain, fast retro | Two independent faults, or the first plausible answer stops the questions | Scenario 12: the chain to `BATCH_SIZE` was perfect and the second, unalerted fault was never asked about | | Fault tree | Decompose the failure top down with AND/OR gates | The outage needed several conditions together | Built from a wrong model of the system, or skipped because it is slow | Scenario 19: four conditions had to hold, and the miss came from leaving one unproven | | Timeline | Put deploys, config edits, alerts, logs and human actions on one clock | Every distributed-systems incident | Events that line up in time are taken as causal | Scenario 19: startup 502s on the healthy pod looked like the incident and misled one agent run | | Change correlation | Start from what changed just before the onset | The trigger was a deploy, flag or config change | The change leaves no revision (a ConfigMap edit), or history access is denied | Aurora's `kubectl rollout history` was blocked by its own default policy in 26 of 38 runs | | AI-assisted investigation | An agent queries the system and returns a cited hypothesis | Many signals, unfamiliar systems, evidence that expires | Scope is set by the alert; "healthy" is asserted without a traffic check | Aurora: 34 correct, 4 partial, 0 wrong over 38 runs, median 153.5 s and $0.35 a run | ## What is root cause analysis in SRE? **Root cause analysis (RCA) is the investigation of an incident that finds the conditions that caused it, so the team removes those conditions instead of only the visible symptom.** In Google's model it is the core of the postmortem, which the [SRE book](https://sre.google/sre-book/postmortem-culture/) defines as "a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring." The [SRE Workbook](https://sre.google/workbook/postmortem-culture/) separates two things worth naming apart: "Identifying the root causes and trigger is one of the most important reasons to write a postmortem." The trigger is the event that set the failure off, such as a config push. The root causes are the latent conditions that let the trigger become an outage, such as a probe that cannot see the broken path. A postmortem that names only the trigger fixes one instance and leaves the conditions in place. RCA is also not a search for someone to blame. The SRE book says a blameless postmortem "must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior." ## Is there always a single root cause? Usually not. John Allspaw writes in [Each necessary, but only jointly sufficient](https://www.kitchensoap.com/2012/02/10/each-necessary-but-only-jointly-sufficient/) that for complex systems "there is no root cause," and that failures take "multiple contributing causes, each necessary but only jointly sufficient." Richard Cook's [How Complex Systems Fail](https://how.complexsystems.fail/) is blunter: "Post-accident attribution to a 'root cause' is fundamentally wrong." The term survives because a team needs somewhere to start. The working rule: keep asking after the first cause is found, and record every condition that had to be true. Scenario 19 below is a small, concrete case of four conditions, each necessary. ## 5 Whys vs AI root cause analysis: where does each break? The [Lean Enterprise Institute](https://www.lean.org/lexicon-terms/5-whys/) defines the method as "the practice of asking why repeatedly whenever a problem is encountered in order to get beyond the obvious symptoms to discover the root cause," and adds that "The specific number five is not the point." Stop when the answer is something you can eliminate, not when you reach five. Its weakness is that it follows one chain. Lab scenario 12 shows the failure mode cleanly, and an AI agent fell into it too. **The setup** ([scenario 12](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/12-two-faults-oom-and-selector)). Two unrelated changes went out back to back: ```bash kubectl -n shop set env deploy/worker BATCH_SIZE=2500 kubectl -n shop patch svc frontend -p '{"spec":{"selector":{"app":"frontend","tier":"web"}}}' ``` The alert covered only the first one: "KubePodCrashLooping [warning] namespace=shop deployment=worker: Pod shop/worker (container worker) is in waiting state (reason: CrashLoopBackOff)." The second change left the frontend Service with no endpoints, so every user request failed. **A 5 Whys from the alert** lands exactly where Aurora's report landed: 1. Why is the worker crash looping? It is OOMKilled (exit code 137) under a 192Mi limit. 2. Why does it run out of memory? `BATCH_SIZE` went from 500 to 2500. Aurora read the worker code and did the math: about 100 KiB per item, so about 244 MiB at 2500. 3. Why did that ship? The env var changed without the memory limit changing. Every step is right, and the chain never reaches the frontend. The load generator's log shows that users were failing first: last success at 20:23:32Z, first timeout at 20:23:35Z, while the incident was created from the worker alert at 20:24:25, about 50 s later. **What Aurora did** ([runs](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/12-two-faults-oom-and-selector)). In both runs it found the OOM and the `BATCH_SIZE` change and stopped there. Run r1's report says "Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected," and its only evidence was pod status. It never ran `get svc`, `get endpoints` or read the load generator's log. Its suggested fix reverted `BATCH_SIZE` and left the selector broken. A person reviewed both runs: partial. It was not alone. HolmesGPT 0.43.0, on the same model, also diagnosed only the OOM in both runs. K8sGPT 0.4.40, which is not given the alert and scans every object in the namespace, flagged `Service has no endpoints, expected label tier=web` in both runs. It also listed the OOMKilled worker, but never said why it ran out of memory: its advice was a larger memory limit, and for the Service it suggested adding `tier=web` to the pods rather than reverting the selector. A scanner sees breadth that an alert-driven investigator does not, and explains less. **The lesson for humans and agents alike.** Before writing "unaffected", check the user path: the load generator or synthetic check, the Service endpoints, the error rate at the edge. A pod in `Running` says nothing about whether traffic reaches it. The full procedure is in the runbook [second fault when the alert covers only one](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). ## When is a fault tree better than the 5 Whys? When the outage needed several things to be true at once. Fault tree analysis starts from the undesired event and decomposes it with AND gates (all inputs needed) and OR gates (any one is enough). The U.S. Nuclear Regulatory Commission's [Fault Tree Handbook (NUREG-0492)](https://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr0492/index.html), published January 1981, is still the standard reference. A related group tool, the fishbone diagram, is described by the [Lean Enterprise Institute](https://www.lean.org/lexicon-terms/fishbone-diagram/) as "a tool used to identify the root causes of a problem"; it generates candidates, while a fault tree states how they combine. Lab [scenario 19](https://github.com/Arvo-AI/aurora-lab/tree/main/scenarios/19-config-drift-partial-restart) is a small fault tree. The alert: "HighErrorRate [warning] namespace=shop service=frontend: 48% of requests to /api/ returned 5xx over the last 10 minutes (threshold 5%)." What we had done: edited the `frontend-nginx` ConfigMap upstream from `api:8080` to `api:8000`, then deleted one of the two frontend pods. ```text About half of /api/ requests return 5xx └── AND ├── ConfigMap upstream edited to api:8000 (the api Service exposes only 8080) ├── one of two frontend pods recreated after the edit ├── nginx reads its config only at start, so the old pod keeps api:8080 in memory └── the readiness probe hits /healthz, which nginx answers itself, so the broken pod stays Ready and keeps receiving traffic ``` Remove any one branch and the incident does not happen, which is also the list of action items: validate upstreams against Services before applying, restart all replicas on a config change, and probe a path that exercises the upstream. A 5 Whys from "requests fail" tends to stop at the first branch. The trap the tree exposes: restarting the "healthy" pod without fixing the ConfigMap would break it too. ## How do you do RCA for distributed systems? Build one timeline across every component, then test the change that came first. Google's [troubleshooting chapter](https://sre.google/sre-book/effective-troubleshooting/) explains why the timeline so often points at a change: "Systems have inertia," and "Recent changes to a system can be a productive place to start identifying what's going wrong." It also names the trap: "Hunting down spurious correlations that are actually coincidences or are correlated with shared causes." Scenario 19's timeline, from Aurora's run r2 tool output (2026-10-02, UTC): | Time | Source | Event | | --- | --- | --- | | 21:07:52 | ConfigMap metadata | `frontend-nginx` created by the app reset | | 21:07:54 to 21:07:58 | pod `rr7xl` nginx log | 3 × 502, `connect() failed (111: Connection refused)` to `10.96.137.47:8080` while the api pods were still starting | | 21:08:01 | pod `r4thz` nginx log | nginx starts in the recreated pod, with the edited file | | 21:08:10 | pod `r4thz` nginx log | first `upstream timed out (110: Operation timed out)` to `10.96.137.47:8000`, 504 | | 21:08:48 | alert | incident created | | 21:08:52 | Aurora | first tool call | Two things in that table are easy to get wrong. The 502s at 21:07:54 are real errors on the pod that later turned out healthy: a cold start, not the incident. HolmesGPT's first run on this scenario blamed exactly that, "a transient cold-start race in which api pods were not yet Ready" in the judge's words, declared the incident resolved, and was graded wrong. Its second run was graded correct, though the judge noted it got the mechanism wrong (it predicted the healthy pod would break on the next kubelet sync), and the blinded regrade called it partial. And the ConfigMap edit itself appears nowhere in the namespace events (52 lines in the evidence captured before the run), and it creates no Deployment revision: the [Deployment docs](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) say a new revision is created "if and only if the Deployment's Pod template" changes. That is why distributed RCA is mostly evidence work: - **Evidence expires.** The [Kubernetes logging docs](https://kubernetes.io/docs/concepts/cluster-administration/logging/) say "By default, if a container restarts, the kubelet keeps one terminated container with its logs," and an evicted pod's containers go "along with their logs." Use `kubectl logs --previous` early and save the output. In scenario 12 r1, Aurora's `--previous` call failed with a containerd error, and it said so under "Not checked". - **Compare replicas, not services.** Scenario 19 is invisible at the Deployment level: every pod `1/1 Running`, zero restarts, same revision. It shows only when you read each pod's log and compare the upstream address. - **Use pod age as change evidence.** `kubectl get pods -o wide` showed `r4thz` at 59 s and `rr7xl` at 68 s. A replica younger than its siblings, with no rollout, means something replaced it: a delete, an eviction or a node failure. ## What does AI-assisted root cause analysis actually do? It runs the timeline and correlation work above as a loop of read-only commands, then writes a report that cites them. Here is Aurora's complete evidence chain on scenario 19, run r2: 28 tool calls, 253 s, $0.84, graded correct by the judge and by a person ([run folder](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v1-run1/19-config-drift-partial-restart/r2)). | Step | Command (as Aurora ran it) | What it showed | | --- | --- | --- | | 1 to 3 | incident history, memory, connected clusters | Nothing prior: each run had a fresh organization | | 4 | `kubectl get pods -n shop -o wide` | All Running, 0 restarts; frontend pods 59 s and 68 s old | | 9 | `kubectl logs loadgen-7c6676864-nkrq2 -n shop --tail=100` | 200 and 504 interleaved from 21:08:10 on | | 8, 10 | `kubectl logs` on both api pods | `api listening on :8080`, only 200 responses | | 11, 25 | `kubectl top pods`, `kubectl top nodes` | Negligible CPU and memory, which ruled out exhaustion | | 13 | `kubectl logs` on `rr7xl` piped to `grep -i` | Failed: kubectl received the grep flag, `unknown shorthand flag: 'i' in -i` | | 16 | `kubectl get configmap frontend-nginx -n shop -o yaml` | `server api:8000;` | | 19 | `kubectl get svc -n shop -o wide` | `api ClusterIP 10.96.137.47 ... 8080/TCP` | | 20 | `kubectl exec frontend-76d98597b9-rr7xl -n shop -- cat /etc/nginx/conf.d/default.conf` | Blocked: "Command blocked by organization policy: Mutating kubectl operations" | | 24 | `kubectl logs frontend-76d98597b9-r4thz -n shop --tail=300` | Every `/api/items` request a 504 to `10.96.137.47:8000` | | 26 | `kubectl rollout history deployment/frontend -n shop` | Blocked by the same policy | The line that closed it, from step 24: ```text 2026/10/02 21:08:10 [error] 27#27: *3 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.244.1.54, server: , request: "GET /api/items HTTP/1.1", upstream: "http://10.96.137.47:8000/items", host: "frontend" ``` The report then put the branches of the fault tree together: the ConfigMap points at 8000, the Service exposes 8080, nginx does not hot-reload, so the pod started after the edit fails and the one started before it does not. It listed what it ruled out (resource exhaustion, the worker and redis, the api pods) and, under "Not checked", the two commands its policy blocked. Its suggested fix patched the ConfigMap back to `api:8080` and then ran `kubectl rollout restart deployment/frontend`. Aurora suggests; a person runs it. A rollout restart replaces every frontend pod, so apply the ConfigMap fix first, or the restart moves the failure from half the pods to all of them. Compared with a person, the agent's advantages here were breadth and patience: it read the logs of six of the seven pods, including the load generator, which people rarely think to check. The blocked `exec` cost less than it looks. The [ConfigMap docs](https://kubernetes.io/docs/concepts/configuration/configmap/) say that when a mounted ConfigMap is updated, "projected keys are eventually updated as well", so after a kubelet sync the file in the old pod can already read `api:8000` while nginx still runs the `api:8080` config it loaded at start. Reading the file proves what is on disk, not what the process loaded. Pod age and per-pod logs were the proof that mattered, and Aurora had both. ## Where was the AI only partly right, and why? On the same fault, run r1 (280 s, 15 tool calls, $0.39) was graded partial by the judge, a second model and a person. It had the same key evidence: the ConfigMap says `api:8000`, the Service says 8080, one pod times out against 8000, the other recovered against 8080. It wrote "The most likely cause is a port mismatch" and then: "This discrepancy was not fully reconciled", because `kubectl exec` to read the live config was blocked. It never concluded that the ConfigMap had been edited and only the recreated pod had loaded it. It also left the api backend open as a possible contributing factor although it had not read the api logs. The evidence to close the gap was already in its output: the 9 s age difference between the pods, and a ConfigMap whose live `resourceVersion` differed from the one recorded in its `last-applied-configuration` annotation. When the check it reached for (reading the file inside the pod) was denied, it did not fall back to those, although they were the stronger evidence anyway. The lab's [FINDINGS.md](https://github.com/Arvo-AI/aurora-lab/blob/main/FINDINGS.md) records this as a gap, along with the policy behavior behind it: Aurora's default read-only policy denies `kubectl exec`, which is defensible, and also the read-only `rollout history`, which is a bug the lab patched on a fix branch. Two lessons carry over to any AI RCA tool: 1. **Hedging is not the failure; stopping is.** "Not fully reconciled" was accurate. The miss was not trying the next-cheapest proof. 2. **Permissions shape the answer.** An agent denied the change history will reason around it, sometimes well (r2) and sometimes not (r1). Read the "not checked" list before trusting the conclusion. The full diagnosis procedure for this failure, with per-pod commands and the fix, is in the runbook [ConfigMap change not applied to all pods](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes). ## Which automated root cause analysis tools did we test, and how accurate were they? We ran three open-source tools an SRE team can install today, Aurora, HolmesGPT and K8sGPT, on 19 injected Kubernetes faults, two runs each, from [results.json](https://github.com/Arvo-AI/aurora-lab/blob/main/benchmark/results.json) (full table on the [AI SRE benchmark page](/benchmark)): | Tool | Correct | Partial | Wrong | Median time | Who checked the grades | | --- | --- | --- | --- | --- | --- | | Aurora (commit 87d22baf, Claude Sonnet 5) | 34 | 4 | 0 | 153.5 s | Judge model, second model, and a person read all 38 | | HolmesGPT 0.43.0 (Claude Sonnet 5) | 31 | 6 | 1 | 45 s | Judge model only, see below | | K8sGPT 0.4.40 (Claude Sonnet 4.5) | Not scored as a peer | | | 12.5 s (two scan passes) | Judge model only | What the table does and does not show: - **Review status differs.** Aurora's 38 grades were read by a person. The 76 HolmesGPT and K8sGPT grades are judge-model only (Claude Opus 5.5). A second model read every one of them next to the judge's grade and agreed with all 76. A third pass graded all 76 from scratch without seeing the judge's grade; it agreed on all 38 HolmesGPT verdicts and disputed 6 K8sGPT labels. A blinded regrade with tool names removed matched the published verdicts in 38 of 38 for Aurora, 37 of 38 for HolmesGPT and 36 of 38 for K8sGPT. No person has checked the HolmesGPT or K8sGPT grades yet. - **HolmesGPT is about 3x faster at the median.** That is a real advantage during an incident. Aurora's times also include queueing behind background jobs in the lab, described in the [lab report](https://github.com/Arvo-AI/aurora-lab/blob/main/REPORT.md). - **K8sGPT does a different job.** `k8sgpt analyze --explain --namespace shop` lists objects that look unhealthy and explains each; it gets no alert and does not investigate one. It printed "No problems detected" on 6 of the 19 scenarios (12 runs), and it was the only tool that flagged the second fault in scenario 12. - **Prompts differed slightly.** HolmesGPT got the alert text plus "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." Aurora got the alert text plus "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." All three got the same admin kubeconfig, each behind its own guard: Aurora's `observability_only` command policy, HolmesGPT's allow-list, and K8sGPT's read-only `analyze`. - **Dates and versions.** Aurora's graded runs are from 2026-10-02; the HolmesGPT and K8sGPT runs from 2026-10-15 and 16, on the same cluster and faults. HolmesGPT 0.44.0 and 0.45.0 were already released when we ran 0.43.0; we have not rerun on them. - **It is a lab, not production.** 19 known faults in a four-service app, written by the team that builds one of the tools. The value is that every run is public and the method can be rerun against any tool you trial. Independent research points the same way. In the [OpenRCA 2.0 preprint](https://arxiv.org/abs/2606.27154) (500 instances, 11 frontier LLMs), models found at least one correct root-cause service in 76.0% of cases, grounded it in a verified causal path in only 61.5%, and recovered the exact root-cause set in 20.7%. The authors call the gap "the ungrounded diagnosis." Naming the right component is easier than proving why it failed, which is what scenario 19 r1 looks like from the inside. ## Why isn't my AI SRE tool finding the real root cause? From what we watched across 38 Aurora runs and 76 runs of the other two tools, the causes were rarely the model's reasoning. They were inputs and scope: - **The alert sets the scope.** Both agents explained the alerted fault in scenario 12 and missed the unalerted one. Give the agent the user-facing symptom (error rate, synthetic check), not only the component alert. - **Permissions remove the evidence.** `rollout history` was denied in 26 of 38 Aurora runs by its own default policy. Check what the agent's credentials can read, especially change history, before blaming the model. - **No traffic signal, no proof of health.** In scenario 19 the load generator log showed the 200 and 504 responses interleaving; without a request log, metrics or traces, "healthy" means "Running". - **Config that is not in a revision.** ConfigMap edits, feature flags and console changes leave no rollout. If the agent cannot read the audit log or GitOps history, it has to infer the change from side effects like pod age. - **Unsupported side claims.** Several reports carried small errors outside the root cause, such as "both cluster nodes" when the cluster had three. Treat the citations as the product and the prose as a summary. ## How should an SRE run an RCA, step by step? 1. **Stop the bleeding.** The [SRE book](https://sre.google/sre-book/effective-troubleshooting/): "Stopping the bleeding should be your first priority; you aren't helping your users if the system dies while you're root-causing." 2. **Preserve evidence.** Save `--previous` logs, per-pod logs, the live manifests and the exact versions before restarts and autoscaling erase them. Rapid triage, the same chapter says, "doesn't preclude taking steps to preserve evidence." 3. **Build one timeline.** Deploys, config edits, alerts, per-pod log lines and human actions on one clock, with the user-facing onset marked separately from the alert. 4. **Correlate with changes, then test.** Start with what changed, including changes that leave no revision. Treat a match in time as a hypothesis and test it, for example by comparing a pod that got the change with one that did not. 5. **Widen to every condition.** Use a fault tree or several 5 Whys branches until every necessary condition is listed, including why detection was late or pointed at the wrong component. 6. **Write preventive action items.** The [SRE Workbook](https://sre.google/workbook/postmortem-culture/) warns about a postmortem whose action items "are mostly mitigative": "you should include some preventative action items and fixes." 7. **Write it up blamelessly.** Assume people acted reasonably on what they knew. The draft can be generated; see [automated post-mortem generation](/blog/automated-post-mortem-generation). An AI agent can do most of steps 2 to 4 in minutes. Steps 5 and 6, deciding which conditions to remove, are where a team's judgment matters most. ## How do you automate post-incident reviews with 5 Whys? Let the agent produce the evidence chain and the first whys, then run the remaining whys as a team on the conditions it surfaced. In scenario 19 the agent's report answers "why did half the requests fail" with citations. The useful human whys come after: why could a ConfigMap point at a port no Service exposes, why did the readiness probe not notice, why did only one pod restart. Each answer is an action item. For a single pod failure, the [Kubernetes pod diagnoser](/diagnose) takes pasted `kubectl` output and returns the likely cause, a quick first step before the review. ## How does Aurora approach root cause analysis? [Aurora](https://github.com/Arvo-AI/aurora) is an open-source (Apache 2.0) AI SRE agent. It takes an alert, runs read-only commands against Kubernetes and the clouds it is connected to (AWS, Azure, GCP, OVH, Scaleway, Fly.io), and writes a report with numbered citations, a "Ruled out" list and a "Not checked" list, as in the runs above. Fixes go into a separate suggestions list; it can open a remediation pull request on GitHub for a person to review, with no auto-merge. Its alert correlation engine links related alerts into the running investigation and never suppresses them. The lab shows where it still falls short: scope set by the alert, and fallbacks when a command is denied. To try it on your own incidents, the code is on [GitHub](https://github.com/Arvo-AI/aurora), or [book a demo](https://cal.com/arvo-ai/demo). Related: [Aurora lab report on 19 Kubernetes failures](/blog/aurora-lab-report-kubernetes-19-failures), [HolmesGPT vs K8sGPT](/blog/holmesgpt-vs-k8sgpt), [open source incident management](/blog/open-source-incident-management). > **Sources and verification.** Lab data: [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab), scenarios 12 and 19, runs `v1-run1` (Aurora, 2026-10-02) and `h2h-v1` (HolmesGPT 0.43.0 and K8sGPT 0.4.40, 2026-10-15 and 16), kind v0.33.0, Kubernetes v1.37.0, nginx 1.27.5; totals from `benchmark/results.json` generated 2026-10-21. Quotes from Google's SRE book chapters "Postmortem Culture" and "Effective Troubleshooting", the SRE Workbook chapter "Postmortem Culture", the Lean Enterprise Institute lexicon, John Allspaw's kitchensoap.com essay, Richard Cook's How Complex Systems Fail, the NRC NUREG-0492 page, the Kubernetes logging, ConfigMap and Deployment docs, the HolmesGPT GitHub releases page, and arXiv:2606.27154 (a preprint). All checked against the live pages on 21 October 2026. --- ## Aurora vs Traditional Incident Management Tools URL: https://www.aurorasre.ai/blog/aurora-vs-traditional-incident-management-tools Published: 2026-03-12 (updated 2026-07-28) Author: Noah Casarotto-Dinning, CEO at Arvo AI > Compare Aurora's open source agentic AI approach with platforms like Rootly, FireHydrant, and incident.io. Verified features, pricing, and use cases. > **Key Takeaway:** Aurora is an open source (Apache 2.0) AI agent that autonomously investigates cloud incidents across AWS, Azure, GCP, and Kubernetes. While Rootly and incident.io have added AI agents to their workflow platforms, Aurora is the only option that is fully open source, self-hosted, and works with any LLM provider including local models. The incident management landscape has evolved significantly. The [global IT incident management market](https://www.marketsandmarkets.com/Market-Reports/incident-management-market-227738490.html) is projected to reach $5.6 billion by 2028. Platforms like Rootly and incident.io have evolved from pure workflow automation to include AI agents, while FireHydrant focuses on workflow automation with AI-enhanced summaries. Aurora takes a different approach, it's built from the ground up as an agentic investigation tool, open source, and self-hosted. This guide provides an honest comparison of Aurora against the leading incident management platforms to help you choose the right tool for your team. ## How Aurora Differs Aurora takes a fundamentally different approach to incident management. Instead of automating the process around incident response (creating channels, paging people, running predefined workflows), Aurora automates the investigation itself. When an incident is triggered, Aurora's AI agents: - Autonomously query your infrastructure across multiple cloud providers - Execute CLI commands in sandboxed pods to gather real data - Search your knowledge base for relevant runbooks and past incidents - Build a dependency graph to assess blast radius - Synthesize findings into a structured root cause analysis This is the difference between **workflow automation** and **agentic investigation**. ## Feature Comparison | Feature | Aurora | Rootly | FireHydrant | incident.io | |---------|--------|--------|-------------|-------------| | **Approach** | Agentic AI investigation | Workflow + [AI agents](https://rootly.com/ai-sre) | Workflow automation | Workflow + [AI agents](https://incident.io/ai-sre) | | **AI Root Cause Analysis** | Autonomous multi-step | Autonomous multi-step | [Post-incident summaries](https://firehydrant.com/ai/) | Autonomous multi-agent | | **Cloud Providers** | AWS, Azure, GCP, OVH, Scaleway | Not specified | Not specified | Not specified | | **Infrastructure Execution** | CLI commands in sandboxed pods | No | No | No | | **Knowledge Base (RAG)** | Weaviate vector search | Past incidents | No | Past incidents | | **Infrastructure Graph** | Memgraph dependency mapping | No | [Service Catalog](https://firehydrant.com) | No | | **Open Source** | Yes (Apache 2.0) | No | No | No | | **Self-Hosted** | Yes (Docker, Helm) | No | No | No | | **LLM Provider** | Any (OpenAI, Anthropic, Google, Ollama) | [Multi-provider (BYOK)](https://rootly.com/ai-sre) | Undisclosed | Undisclosed | | **Code Fix PRs** | Yes | Not verified | No | [Yes](https://incident.io/ai-sre) | | **Pricing** | Free (self-hosted) | [From $20/user/mo](https://rootly.com/pricing) | [From $25/responder/mo](https://firehydrant.com/pricing) | [From $15/user/mo](https://incident.io/pricing) (free Basic tier) | | **Integrations** | 22+ tools | 50+ tools | 40+ tools | 30+ tools | | **Terraform/IaC Support** | Native Terraform analysis | No | No | No | > *Note: Shoreline was [acquired by NVIDIA](https://www.nvidia.com) and is no longer an independent product. Data verified from official sources as of March 2026.* ## When to Choose Aurora > "We evaluated Rootly and FireHydrant but chose Aurora because we needed AI that actually investigates, not just routes alerts to Slack. The open-source model meant we could audit exactly what the AI was doing on our infrastructure.". Early Aurora adopter Aurora is the best fit when your team needs: - **Autonomous investigation**: You want AI that actually investigates incidents, not just summarizes them. - **Multi-cloud environments**: You run infrastructure across AWS, Azure, GCP, OVH, or Scaleway and need unified incident investigation. - **Open source and self-hosted**: You need to keep incident data in your own environment for compliance or security reasons. - **LLM flexibility**: You want to choose your own LLM provider, or run models locally with Ollama. - **Deep Kubernetes support**: Your infrastructure is heavily Kubernetes-based and you need deep pod-level investigation. - **Infrastructure as Code**: You use Terraform and want the AI to understand your IaC state. ## When to Choose Traditional Tools Rootly, FireHydrant, or incident.io may be better when: - **Process orchestration is the priority**: Your main need is automating Slack channel creation, status pages, and stakeholder communication. - **Larger ecosystem**: You need 50+ integrations out of the box. - **Managed service**: You prefer SaaS over self-hosted. - **Established workflows**: Your team has mature incident processes and just needs tooling to automate them. ## The Open Source Advantage Aurora's Apache 2.0 license means: - **No vendor lock-in**: Deploy on your infrastructure, use your LLM provider, keep your data. - **Full transparency**: Audit exactly how the AI investigates your incidents. - **Community-driven**: Contribute integrations, tools, and improvements. - **Cost efficiency**: No per-seat or per-incident pricing. Self-hosted is completely free. - **Customization**: Modify investigation workflows, add custom tools, integrate with internal systems. ## Getting Started Try Aurora alongside your existing tooling, it complements rather than replaces workflow platforms: ```bash git clone https://github.com/arvo-ai/aurora.git && cd aurora make init # generates secrets, copies .env.example to .env nano .env # add OPENROUTER_API_KEY, OPENAI_API_KEY or ANTHROPIC_API_KEY make prod-prebuilt # pulls prebuilt images from GHCR and starts ``` Aurora can receive webhooks from PagerDuty, Datadog, and Grafana, running AI-powered investigations in the background while your existing incident process continues. Learn more at [aurorasre.ai](https://www.aurorasre.ai) or read the [full documentation](https://arvo-ai.github.io/aurora/). For a deeper look at how agentic investigation works, see our guide on [What is Agentic Incident Management?](/blog/what-is-agentic-incident-management) and the [AI-Powered Incident Investigation complete guide](/blog/ai-powered-incident-investigation). For the postmortem half of the workflow, see [Automated Post-Mortem Generation](/blog/automated-post-mortem-generation). To evaluate Aurora against the other open-source AI SREs, read our [Aurora vs HolmesGPT vs K8sGPT comparison](/blog/open-source-ai-sre-aurora-vs-holmesgpt-vs-k8sgpt). For how Aurora extends from one-shot investigation into recurring automation, see the [Aurora Actions launch post](/blog/aurora-actions-background-automations). - **Start free:** [aurora-ai.net](https://aurora-ai.net/) (hosted, no infrastructure to run) - **GitHub:** [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora) --- ## Agentic Incident Response: One Real Alert, Every Tool Call URL: https://www.aurorasre.ai/blog/what-is-agentic-incident-management Published: 2026-03-10 (updated 2026-10-21) Author: Ben Gervais, Engineer at Arvo AI > Agentic incident response shown on one tested alert: every tool call, the fix, what the agent was blocked from running, autonomy levels, where it fails. Agentic incident response (also called agentic incident management) is an AI agent that takes an alert, investigates the live system itself by choosing and running read-only commands one after another, and reports a root cause with its evidence and a suggested fix. Nobody scripts its steps in advance, as with runbook automation, and in the tools tested here a person still approves any change it proposes. Below is one complete agentic investigation from our lab: the alert, every tool call, the root cause, the suggested fix, and the commands the agent was not allowed to run. Then the autonomy levels vendors offer, checked against their docs, and the scenarios where the agents got it wrong. Arvo AI builds Aurora SRE, one of the three tools tested, and wrote the scenarios; every run is public in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab) and the live table is at [/benchmark](/benchmark). For the wider category, read [the AI SRE complete guide](/blog/ai-sre-complete-guide); for investigation techniques, [AI-powered incident investigation](/blog/ai-powered-incident-investigation). ## What does an agent actually do across the incident lifecycle? Scenario 07 of the lab, run r1, 2 October 2026. The fault: the `api` Service's selector was changed from `app=api` to `app=api-v2`, a label no pod carries. Every pod stays Running and Ready, nothing restarts, and the frontend's nginx returns 502 on every `/api/` request because the Service has no endpoints. **1. The alert.** This is the whole input: > HighErrorRate: frontend /api/ requests in namespace shop are failing with 5xx; all pods are Running. Cluster: aurora-lab (kubeconfig). Investigate and find the root cause. Aurora SRE (commit 87d22baf, model `bedrock/us.anthropic.claude-sonnet-5`) ran with a fresh user and organization, so no memory from earlier runs, and the lab's admin kubeconfig behind its default `observability_only` command policy. **2. The investigation.** Every tool call, in order, with start times in UTC from `steps-full.json` in the [run folder](https://github.com/Arvo-AI/aurora-lab/tree/main/runs/v0-run2/07-service-selector-mismatch/r1). Indented lines are copied from the tool output, usually a few lines picked from a longer result; `[...]` marks a cut inside an excerpt, and any other line in square brackets is our note, not output. ```text 16:36:09 list_incidents, grep_memories 16:36:10 get_connected_clusters 16:36:15 kubectl get pods -n shop -o wide api-f64486c4c-4rwgd 1/1 Running 0 79s 10.244.1.12 aurora-lab-2-worker api-f64486c4c-56fw6 1/1 Running 0 79s 10.244.2.18 aurora-lab-2-worker2 [...] 16:36:15 kubectl get events -n shop --sort-by=.lastTimestamp 16:36:15 kubectl get svc,ingress -n shop service/api ClusterIP 10.96.89.40 8080/TCP 81s 16:36:31 kubectl logs -n shop -l app=frontend --tail=100 --all-containers 2026/10/02 16:35:10 [error] 29#29: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.244.1.14 [...] 16:36:31 kubectl logs -n shop -l app=api --tail=100 --all-containers api listening on :8080 10.244.1.1 "GET /healthz HTTP/1.1" 200 - 16:36:31 kubectl logs -n shop -l app=loadgen --tail=100 --all-containers 2026-10-02T16:35:06Z GET /api/items -> 502 2026-10-02T16:35:08Z GET /api/items -> 502 [...] 16:37:05 kubectl get endpoints api -n shop -o yaml [...] kind: Endpoints metadata: [...] name: api namespace: shop [...] [no subsets field at all] 16:37:05 kubectl logs api-f64486c4c-4rwgd -n shop --tail=200 16:37:05 kubectl logs api-f64486c4c-56fw6 -n shop --tail=200 16:37:20 kubectl get pods -n shop -l app=api -o json 16:37:20 kubectl describe pod api-f64486c4c-4rwgd -n shop 16:37:41 kubectl get endpoints api -n shop NAME ENDPOINTS AGE api 2m43s 16:37:41 kubectl get svc api -n shop -o yaml kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service", [...] "selector":{"app":"api"}}} [...] selector: app: api-v2 16:37:41 kubectl get endpointslices -n shop -l kubernetes.io/service-name=api -o yaml endpoints: null 16:37:56 kubectl get deploy -n shop -o wide api 2/2 2 2 2m59s api python:3.12-alpine app=api 16:37:56 kubectl get pods -n shop -l app=api-v2 (no output) 16:37:56 kubectl get events -n shop --field-selector involvedObject.name=api -o wide 16:38:31 kubectl get nodes -o wide 16:38:31 kubectl top pods -n shop ``` The first three calls are housekeeping (the incident list, a memory search that found nothing, the cluster list). Then the agent fans out: pods, events, Services, then logs for all three workloads at once. The frontend log gives it `connect() failed (111: Connection refused) while connecting to upstream` and the loadgen log a 502 every two seconds since 16:35:06 (44 of its 45 lines; one 200 at 16:35:12), so by 16:37:05 it is asking what is behind the `api` Service, and the Endpoints object comes back with no subsets. At 16:37:41 it has the Service's live selector (`app: api-v2`) next to its last-applied one, at 16:37:56 the Deployment's label (`app=api`), and in the same second it tests the alternative, a real `api-v2` workload the selector might target on purpose, and gets no output. The last two calls rule out node and resource problems. 22 tool calls (19 of them `kubectl`), 35 model calls, 175 seconds by Aurora's own timer, $0.52 of model spend. **3. The root cause.** The report's paragraph: > The root cause was a label/selector mismatch on the `api` Service in the `shop` namespace: the Deployment and its pods are labeled `app=api` [13, 18], but the Service had no matching backend endpoints, as confirmed by `kubectl get endpoints api -n shop` returning `` and the associated EndpointSlice showing `endpoints: null` [10, 15, 17]. The bracketed numbers are the tool calls above. The report also listed what it ruled out (OOM, crash loops, redis, node capacity, nginx misconfiguration, an intentional `api-v2` target) and what it did not check, including when the selector changed and who changed it. Note what the paragraph leaves out: the bad value `app=api-v2`, which Aurora had read at 16:37:41. The judge model graded the run correct, a second model review called it partial for exactly that omission, and Ben read the full output (the final chat message names `app: api-v2`) and kept it correct. The second run of the scenario was also correct, in 138 seconds, 15 tool calls and $0.40. **4. The suggested fix.** It went into a separate suggestions list, rated low risk: "Patch `api` Service selector to match pod labels `app=api`", with the command: ```shell kubectl patch svc api -n shop -p '{"spec":{"selector":{"app":"api"}}}' ``` **5. What it did not do.** Aurora did not run the patch, and did not try to: its Not Checked list says mutating `kubectl patch` operations "were blocked under read-only session policy", which is true of the policy, but no patch call appears in the tool log. The fix is a suggestion a person applies, and the rollback is the same command with `app=api-v2`, if the new selector turns out to be intended. The step-by-step fix with verification is the [Service has no endpoints runbook](/runbooks/fix-service-no-endpoints-kubernetes). The same alert went to two other open-source tools on 15 October 2026, same cluster, same fault. [HolmesGPT](https://github.com/HolmesGPT/holmesgpt) 0.43.0, on the same model, got there in 55 seconds and 11 tool calls and read the drift the same way: > The Service's kubectl.kubernetes.io/last-applied-configuration annotation still records selector: {"app":"api"} (the originally-applied/desired state), but the live object's spec.selector is app: api-v2. [K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) 0.4.40 is not an agent; its `analyze` runs fixed analyzers over the namespace and has a model explain each finding (Claude Sonnet 4.5 here, because K8sGPT sends a temperature setting that Bedrock rejects for Sonnet 5), without reading the alert. It flagged the Service in about 15 seconds, "Service has no endpoints, expected label app=api-v2", then pointed the fix the wrong way: "Add missing label to Pods: kubectl label pods app=api-v2". The judge graded it partial. Flagging the broken object is one step; working out which side of the mismatch is wrong is the part an agent adds. ## What was the agent not allowed to run, and why? All three tools got the same admin kubeconfig, each behind its own guard: Aurora's `observability_only` command policy, HolmesGPT's command allow-list, K8sGPT's read-only `analyze`. The guard, not the credential, decided what ran. Aurora's policy is an organization-level allow-list and deny-list, checked before every command; a new organization gets the `observability_only` template with both lists on ([`command_policy.py`](https://github.com/Arvo-AI/aurora/blob/main/server/utils/auth/command_policy.py), unchanged on this point between the lab commit and today's main). Its `kubectl` allow rule covers only `get`, `describe`, `logs`, `top`, `explain`, `api-resources`, `api-versions`, `cluster-info` and read-only `config` subcommands, so anything else (`label`, `annotate`, `set image`) is refused as "No matching allow rule". On top of that, one deny rule labelled "Mutating kubectl operations" names `exec`, `cp`, `run`, `attach`, `port-forward`, `apply`, `delete`, `create`, `edit`, `patch`, `replace`, `scale`, `rollout`, `drain`, `cordon`, `uncordon` and `taint`. Not all of those mutate, but every one is refused with that label. Across all 38 graded Aurora runs (both lab sets, 19 scenarios, 2 runs each) the agent made 452 `kubectl` calls and the policy refused 30 of them, every one with the same message. None of the 30 was an attempt to change the cluster. In the 38 tool logs there is no `patch`, `apply`, `delete`, `scale`, `edit` or `label` call at all, and the only shell command outside `kubectl` was one `curl` to Docker Hub's tag API in scenario 03; the agent investigated and left changes to the suggestions list. What it was refused: | Refused command | Calls | Runs | Read-only? | |---|---|---|---| | `kubectl rollout history deployment/` | 27 | 26 of 38 | Yes | | `kubectl exec -- cat ` (`/etc/resolv.conf` in scenario 16, the nginx config in both runs of scenario 19) | 3 | 3 | The `cat` is, `exec` in general is not | What the agent saw, from scenario 12 r1 and scenario 19 r1: ```text $ kubectl rollout history deployment/worker -n shop Command blocked by organization policy: Mutating kubectl operations $ kubectl exec frontend-76d98597b9-g5zrg -n shop -c frontend -- cat /etc/nginx/conf.d/default.conf Command blocked by organization policy: Mutating kubectl operations ``` The `exec` block is defensible: `exec` runs arbitrary commands inside a container, and a policy that cannot see what the shell will do should refuse it. The `rollout history` block was a bug in the default. The deny rule matched the whole `rollout` verb, so the read-only `history` and `status` went down with `restart` and `undo`, and the agent lost the first command most people run to answer "what changed". It worked around it by diffing ReplicaSets. A lab branch narrowed the rule to `rollout (restart|undo|pause|resume)`; the fix, which also adds `rollout (history|status)` to the allow-list, is an open pull request on the Aurora repo and not yet merged. HolmesGPT's own allow-list blocked the same command in 12 of its runs. Both blocks cost evidence, and the second section below shows where. That is the trade an agentic tool makes: every read it is refused is a question it cannot answer, and every write it is allowed is a change nobody reviewed. ## What autonomy levels exist for incident response agents? Vendors describe autonomy in their own terms. Mapped onto four levels: | Level | The agent | A person | |---|---|---| | 1. Investigate | Reads, reports a root cause and evidence | Decides and makes every change | | 2. Suggest | Also drafts the fix: a command with a risk rating, or a pull request | Reviews, then applies or merges | | 3. Act on approval | Runs the change itself after a per-action approval | Approves or denies each action | | 4. Autonomous | Runs changes within a policy and reports afterwards | Sets the policy, reviews after the fact | How named tools gate actions, from their own docs, checked 21 October 2026: | Tool | Default | Highest level offered | How actions are gated | |---|---|---|---| | Aurora SRE | Investigate, under the `observability_only` command policy | 3, in an interactive chat | Org-level command policies. In a background investigation (an alert, as in the lab) a refused command is denied with no prompt; in an interactive chat it becomes a Yes / No / Yes-Always prompt and runs only if the user approves ([`command_gate.py`](https://github.com/Arvo-AI/aurora/blob/main/server/utils/auth/command_gate.py)). Fixes from an investigation are suggestions or a pull request; it ["suggests fixes and can generate pull requests"](https://github.com/Arvo-AI/aurora), with no auto-merge | | HolmesGPT | ["By design, HolmesGPT has **read-only access** and respects RBAC permissions."](https://github.com/HolmesGPT/holmesgpt) | 3 | The optional [Kubernetes Remediation MCP toolset](https://holmesgpt.dev/latest/data-sources/builtin-toolsets/kubernetes-remediation-mcp/) is not enabled by default; its mutating `run_kubectl_command` tool always pauses for a human | | K8sGPT operator | `analyze` only reads | 4, narrowly | [Auto remediation](https://github.com/k8sgpt-ai/k8sgpt-operator/blob/main/AUTO_REMEDIATION.md) is "Status: Alpha", off unless enabled, "deny-by-default": built-in policies allow only container image changes, other fields need an opt-in annotation per resource, and the API server dry-runs each patch first | | Azure SRE Agent | Agent-level default Review | 4 | [Run modes](https://learn.microsoft.com/en-us/azure/sre-agent/run-modes): Review "proposes an action; you approve or deny", Autonomous "executes immediately and reports what it did". The same page's table defaults incident response plans and scheduled tasks to Autonomous, and Review's Approve and Deny buttons cover only Azure infrastructure operations. The [overview](https://learn.microsoft.com/en-us/azure/sre-agent/overview) says it "suggests or, when configured, executes mitigations" | | Resolve AI | Proposals only | 3 | ["Every proposal must be approved by a human via the Resolve UI or Slack before any action is taken."](https://docs.resolve.ai/mitigation-actions.md) and "The model cannot call write APIs." | | Cleric | ["Read-only by default."](https://cleric.ai/) | 2 by default | "Read access by default, write access when you're ready."; fixes arrive as a pull request to review | Of these, only Aurora, HolmesGPT and K8sGPT ran in our lab, and only in read-only, background mode; we did not test Aurora's chat approval prompt, HolmesGPT's remediation toolset or the K8sGPT operator. The rest is what each vendor documents. Our position: on the lab's evidence, level 2 is where an incident agent belongs in production today. None of the three tools needed write access to be useful, and the failures below are exactly the cases where an agent at level 4 would have acted on a wrong or incomplete answer. Microsoft's own run-modes page recommends Review for production incidents and Autonomous for staging and recurring tasks such as daily health checks; that split is reasonable, as long as the autonomous jobs are ones whose worst action you can undo. ## How is agentic incident response different from runbook automation and AIOps? | | Runbook automation | AIOps | Agentic incident response | |---|---|---|---| | Trigger | An alert matches a rule | Alert and metric streams | An alert, or a question from a person | | What runs | A script someone wrote in advance | Correlation and anomaly models over telemetry | A model choosing the next read-only command from the last result | | Output | The script's side effects | Grouped or suppressed alerts, anomaly scores | A root cause with cited evidence, what was ruled out, a suggested fix | | Handles a fault nobody scripted | No | Flags it, does not explain it | Yes, within what read access shows | | Changes production | Often, that is the point | Rarely | At level 2, only when a person applies the suggestion | AIOps reduces the noise that reaches a person; an agent does the first stretch of the on-call engineer's work after the page. Nobody wrote "check the endpoints when nginx says connection refused" for scenario 07; the agent chose it from the log line. The longer comparison is in [AI SRE vs AIOps](/blog/ai-sre-vs-aiops). ## How do AI agents handle complex incidents? Unevenly. Two scenarios were built to catch the failure an agent is prone to: stopping at the first good explanation. **Scenario 12: a second fault the alert did not mention.** The worker is OOMKilled after a batch-size change, and separately the frontend Service's selector matches no pods, so users get errors. The alert names only the worker. Aurora found the OOM in both runs and wrote, with pod status as its only evidence: > Other workloads in the `shop` namespace (api, frontend) remained healthy and unaffected [5, 14]. It never ran `get svc` or `get endpoints` in either run, and one of its 18 calls in r1 was the blocked `rollout history` above. HolmesGPT did the same in both of its runs; its first run ends "Root cause confirmed: shop/worker is being OOMKilled." and makes no claim about the frontend either way. All four runs were graded partial. K8sGPT, which scans every object instead of following the alert, listed the Service in both runs, "Service has no endpoints, expected label tier=web", and never explained the OOM. An agent allowed to act on its own would have raised the worker's memory, closed the incident and left the frontend down. The check that catches it is in the [second fault runbook](/runbooks/fix-second-fault-alert-covers-only-one-kubernetes). **Scenario 19: a ConfigMap edit that only one of two frontend pods has loaded.** HolmesGPT's first run read the untouched pod's logs and concluded "All data points to a transient startup race, now resolved." Graded wrong. Aurora's first run found the edited upstream (`server api:8000` against a Service on 8080) and tied it to the failing pod, then tried to read the live nginx config with `kubectl exec`, was refused, and wrote that the discrepancy "was not fully reconciled". Its own `get pods` output already showed the failing pod was 9 seconds younger than the healthy one, which answers the question without `exec`. Graded partial. Both second runs got it. The manual version is the [ConfigMap change not applied to all pods runbook](/runbooks/fix-configmap-change-not-applied-to-all-pods-kubernetes). Smaller misses run through the whole set. A second model found claims the evidence did not support in 15 of the 18 Aurora reports from the harder scenario set, none of them in the root cause: "both cluster nodes" when there are three, "pulled successfully" with no Pulled event. The totals, from [results.json](https://github.com/Arvo-AI/aurora-lab/blob/main/benchmark/results.json): | | Aurora SRE | HolmesGPT | K8sGPT | |---|---|---|---| | Correct, of 38 runs | 34 | 31 | not scored as a peer | | Partial | 4 | 6 | 21 | | Wrong | 0 | 1 | 5 | | "No problems detected" | 0 | 0 | 12 | | Median time per run | 153.5 s | 45 s | 12.5 s for two analyze passes | | Median model cost per run | $0.35 | not captured | not captured | Aurora's 38 graded runs are from 2 October 2026 and HolmesGPT's and K8sGPT's from 15 and 16 October, on the same cluster and faults. Each agent got the alert text; Aurora's prompt added "Cluster: aurora-lab (kubeconfig). Investigate and find the root cause." and HolmesGPT's "Namespace shop, cluster kind-aurora-lab. Find the root cause and say what changed." Aurora's 38 grades were read by a person, who changed none. The 76 HolmesGPT and K8sGPT grades are judge-model only: a second model review agreed with all 76, a third pass graded them from scratch and disputed 6 K8sGPT labels, and a blinded regrade with the agent names removed matched 38 of 38 Aurora, 37 of 38 HolmesGPT and 36 of 38 K8sGPT verdicts, every change downward. On correct against not, the gap between Aurora and HolmesGPT is not statistically significant (two-sided Fisher exact p = 0.52, computed on [/benchmark](/benchmark) from these totals, comparing person-read Aurora grades with judge-only HolmesGPT grades). HolmesGPT is about three times faster at the median. Read the two as roughly even on these faults. ## What should you expect from an agent on your own incidents? 1. **Start at level 1 or 2.** A read-only identity such as the Kubernetes `view` ClusterRole reproduces most of scenario 07. Decide in advance which reads you refuse (Secrets, `exec`) and check that the policy does not refuse read-only commands like `rollout history` by accident. 2. **Grade the answers against what the human found.** Correct, partial or wrong, the same day, including false claims outside the root cause. That record is what tells you whether to move a class of incident up a level. 3. **Treat "everything else is healthy" as a claim that needs a request-path check.** Pod status is not evidence that a Service has endpoints. In scenario 12 Aurora said it in both runs ("remained healthy and unaffected", "running normally and unaffected") and it was wrong both times; HolmesGPT never checked. 4. **Keep changes behind approval, with an undo.** A suggested command with a risk rating and a rollback, or a pull request someone merges. The pilot procedure step by step is in [the AI SRE complete guide](/blog/ai-sre-complete-guide), and the tool landscape in [top AI SRE tools 2026](/blog/top-ai-sre-tools-2026). The faults are reproducible: `./lab.sh break 07-service-selector-mismatch` in the lab repo puts a cluster in the state above. To check a symptom you are seeing now, paste it into [/diagnose](/diagnose). Aurora SRE is open source under Apache 2.0: [github.com/Arvo-AI/aurora](https://github.com/Arvo-AI/aurora), or [book a demo](https://cal.com/arvo-ai/demo). Sources: Aurora run folders `runs/v0-run2/07-service-selector-mismatch/r1` and `r2`, `runs/v1-run1/12-two-faults-oom-and-selector`, `runs/v1-run1/16-dns-search-domain`, `runs/v1-run1/19-config-drift-partial-restart` and the `steps-full.json` tool logs of all 38 graded runs; HolmesGPT and K8sGPT outputs under `runs/h2h-v1/r1` and `r2`; `benchmark/results.json`, `REPORT.md`, `FINDINGS.md`, `runs/h2h-v1/README.md`, `review-summary.md`, `review2-summary.md` and `runs/blind-regrade.md` in [Arvo-AI/aurora-lab](https://github.com/Arvo-AI/aurora-lab). Vendor autonomy and approval behaviour from the linked docs of HolmesGPT, the K8sGPT operator, Azure SRE Agent, Resolve AI and Cleric, read 21 October 2026. Verified 21 October 2026.