Alert Fatigue: Causes, Fixes, and What Our Lab Measured

Alert fatigue is what happens when on-call engineers get so many alerts that are false, duplicated or not actionable that they start to skim, delay or ignore pages, including the real ones. Google's SRE book puts it as "employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a 'real' page that's masked by the noise." The fix is fewer alerts, each on a user-facing symptom, grouped so that one problem pages once. Our lab adds one condition: whatever you cut, every user-facing path must still have an alert. In our runs, the two investigators that start from the alert found the cause when the alert named the wrong object, and missed the second fault in all 4 runs where that fault had no alert.

That last claim comes from our own data, and the rest of this page shows it. It comes from two sources: a stock Prometheus alerting install we ran on 2026-10-29 against a healthy lab cluster and two injected faults, and the 114 graded investigations in our Kubernetes lab, where we know the true cause of every failure because we injected it. For noise reduction tooling (suppression, correlation, dedup) see automated alert noise reduction. For the human side (shift load, rotation size) see on-call burnout: what the data says.

What is alert fatigue?

The problem is not unique to software: Google's SRE book cites "Applying Cardiac Alarm Management Techniques to Your On-Call" as "an example of alert fatigue in another context" (Monitoring Distributed Systems). For operations, the same chapter states the limit plainly: "I can only react with a sense of urgency a few times a day before I become fatigued."

Three things are easy to mix up:

Most advice treats fatigue as a noise problem only. Coverage is the other half, and it is the half that cutting alerts can break.

What causes alert fatigue in SRE and DevOps teams?

The SRE Workbook ties fatigue to "a low signal-to-noise ratio" and caps load at "a maximum of two incidents per on-call shift," counting one problem as one incident however many alerts it fired (SRE Workbook, On-Call; the human side is in on-call burnout).

What does a stock Kubernetes alerting install fire on a healthy cluster?

We installed kube-prometheus-stack 92.15.0 (Prometheus v3.15.1, Alertmanager v0.34.2, operator v0.94.2) with its default rules on a 3-node kind cluster (kind v0.33.0, Kubernetes v1.37.0) running our healthy shop app. The only changes from chart defaults: Grafana off, and the default Alertmanager receiver posts to a small webhook logger instead of null, so every notification that would have paged someone is logged. We saved Prometheus's /api/v1/alerts every 30 seconds: 20 minutes healthy, then 18 minutes each on two injected faults. This is a real alerting run, separate from the lab's scripted alerts further down.

At the end of the healthy 20 minutes, 11 alerts were firing (Watchdog, the always-on heartbeat, excluded):

alertname                                  severity  job
KubeControllerManagerInstanceUnreachable   warning   kube-controller-manager
KubeProxyInstanceUnreachable               warning   kube-proxy        (x3, one per node)
KubeSchedulerInstanceUnreachable           warning   kube-scheduler
TargetDown                                 warning   kube-etcd
TargetDown                                 warning   kube-scheduler
TargetDown                                 warning   kube-proxy
TargetDown                                 warning   kube-controller-manager
etcdInsufficientMembers                    critical  kube-etcd
etcdMembersDown                            warning   kube-etcd

(Formatted from the 19:57:04 UTC snapshot in run2/alerts.jsonl; the three kube-proxy rows are collapsed into one line.)

Every one is about Prometheus failing to scrape a control-plane component, not about the component being broken: the expressions test up for the scrape job (up == 0, or for etcd a count of up targets). No alert touched the shop namespace, and the lab's reset had confirmed the app was serving 200s just before the window. The chart's README explains one of the four: kube-proxy's metrics bind address defaults to 127.0.0.1:10249, "that prometheus instances cannot access to." Alertmanager grouped the 11 by namespace and sent our webhook one notification with 10 of them at 19:55:27 UTC, including etcdInsufficientMembers at severity critical, then a second at 20:00:27 when the 11th (etcdMembersDown, for: 20m) joined the group. The chart ships its default route pointed at a null receiver; a team that points it at a pager gets a critical etcd page on a cluster like ours before it has deployed anything. One gap in our record: Prometheus marked etcdInsufficientMembers firing at 19:39:57 (for: 3m), but the webhook logged nothing before 19:55:27. We read the logger's output with kubectl logs after the run, so earlier notifications may have been lost with a container restart after the VM restart; we cannot tell which.

An earlier attempt on the same day, on a machine shared with several other kind clusters, had the same 11 firing plus 27 node alerts pending (NodeFilesystemAlmostOutOfSpace, NodeFilesystemSpaceFillingUp, NodeSystemSaturation), none of which had fired by the end of its 20-minute healthy window. Those were true, the VM disk had under 5% free, and useless to the owner of shop. That machine then stopped responding, so we discarded the attempt and reran on a quiet VM; both are in the lab folder.

How many alerts does one fault produce, and when do they page?

Scenario 01 from the lab: kubectl -n shop set env deploy/worker BATCH_SIZE=2500, applied at about 19:58:25 UTC (the lab resets the app first, starting at 19:57:34, and logged the fault as visible at 19:59:15 after a fixed 45-second settle). The worker now exceeds its 192Mi limit and is OOM-killed on every start. Every change in the shop namespace's alerts, from the 30-second snapshots:

19:59:15Z  KubePodNotReady                 pending (active since 19:58:59)
20:03:16Z  KubePodCrashLooping             pending (active since 20:02:59)
20:08:47Z  KubeDeploymentReplicasMismatch  pending (active since 20:08:29)
20:14:18Z  KubePodNotReady                 firing
[20:14:29  webhook notification, group namespace=shop: KubePodNotReady]
20:16:49Z  end of window: KubePodCrashLooping and KubeDeploymentReplicasMismatch still pending

(Formatted from run2/alerts.jsonl and run2/pager-log.jsonl: one line per change of state; all three alerts are for the worker pod worker-86f68b9bf6-2x9hs or deployment=worker.)

One fault, three rules. All three have for: 15m in this chart version: Prometheus waits until the condition has held at every evaluation for that long before the alert counts as firing (alerting rules docs). That is the stock defence against flapping, and its cost is in the timeline: the first page arrived about 16 minutes after the change (15.5 minutes after Prometheus first saw the pod not ready), and the alert that names the actual symptom (crash looping) had not fired when the window ended 18 minutes after the change, because the CrashLoopBackOff state appeared four minutes after the pod went not-ready. It fired at 20:17:59, after the lab's next reset had already deleted the pod; more on that below. In the discarded first attempt the same fault showed a second flapping cost: KubePodNotReady dropped out for one evaluation at 16:17:12 and came back at 16:17:29 with a new 15-minute clock (either the pod reported Ready between crashes or a scrape was missed on that overloaded machine; the snapshots cannot tell which).

Grouping did less here than it looks. Alertmanager's default grouping in this chart (group_by: ['namespace'], group_wait: 30s, group_interval: 5m) merged the 10 healthy-cluster alerts into one notification. For the worker, what kept three rules from becoming three pages inside the window was for: 15m: only one had fired. When the crash-loop alert fired later, it went out as a new notification for the same group at the next 5-minute tick. Grouping "categorizes alerts of similar nature into a single notification" (Alertmanager docs), but every alert that joins a group later still produces another notification.

What do stock alert rules miss?

Scenario 12 next, under the same rules: the same worker change plus a frontend Service selector that matches no pods, so the frontend has no endpoints and every user request fails. Both changes went in at about 20:18:10 UTC, after the lab's reset recreated the namespace. For the whole 18-minute window that followed (snapshots 20:19:00 to 20:36:36 UTC) the alerts in shop were KubePodCrashLooping and KubePodNotReady (both fired) and KubeDeploymentReplicasMismatch (pending), all for the worker. No alert in any snapshot, in any namespace, mentioned the frontend. The webhook got three notifications for shop in that window, all about the worker:

20:19:29  firing    KubePodCrashLooping  worker-86f68b9bf6-2x9hs  (scenario 01's pod, deleted by the reset)
                    resolved: KubePodNotReady  worker-86f68b9bf6-2x9hs
20:24:29  resolved  KubePodCrashLooping, KubePodNotReady  worker-86f68b9bf6-2x9hs
20:34:29  firing    KubePodNotReady  worker-86f68b9bf6-fzgrs  (scenario 12's pod)
                    resolved: KubePodCrashLooping  worker-86f68b9bf6-2x9hs (repeated)

(Formatted from run2/pager-log.jsonl; every alert in each notification is listed.)

Two of those three are about a pod that no longer existed. KubePodCrashLooping takes max_over_time(...[5m]) of the waiting reason, so the old pod's alert finished its 15 minutes and fired at 20:17:59, less than a minute after the reset (started 20:17:19) deleted the pod, and dropped out once the 5-minute window emptied. The "resolved" at 20:24 went out while the new worker pod was crash looping; its own KubePodCrashLooping had been pending since 20:21:29 and fired at 20:36:29, too late for a notification inside the window. The alert is keyed on the pod name, so from reading the rule, any rollout or pod replacement in production does the same: the page clears, and the new pod starts its own 15-minute clock.

The frontend outage got nothing, and not by chance. We read the 139 distinct alert names in the chart's rule templates: none of the expressions reads kube_endpoint*, kube_endpointslice* or kube_service* metrics, and none can see your application's HTTP status codes, because the chart does not know your application's metrics. Six of the lab's nineteen faults look like that from the user's side, requests failing while the pods stay Running (scenarios 07, 11, 12's second fault, 13, 15, 19). We measured only scenario 12. For the other five we have not run the stock rules; reading the rule expressions against each fault, we expect none of them to fire, but that is inference, not measurement.

This is the Prometheus project's own advice turned around: "Aim to have as few alerts as possible, by alerting on symptoms that are associated with end-user pain rather than trying to catch every possible way that pain could be caused" (Prometheus alerting practices). The stock set is almost all causes. Cause alerts are both the noise and the gap.

What happens to triage when the alert covers only one of two faults?

This is the part of our data that changed how we think about alert fatigue. In scenario 12 two unrelated changes go out together: the worker's BATCH_SIZE goes from 500 to 2500 (OOM, crash loop), and the frontend Service selector gains tier=web, which no pod carries, so the frontend has no endpoints and is down for users. The lab sends a scripted alert that names only the worker, the same text to Aurora and HolmesGPT (K8sGPT takes no alert):

KubePodCrashLooping [warning] namespace=shop deployment=worker: Pod shop/worker (container worker) is in waiting state (reason: CrashLoopBackOff).

(The lab's alerts are fixed text from the scenario file, not evaluated rules. They are a controlled input, not a measurement of alerting.)

All six runs were graded partial. The two agents that start from the alert (Aurora, HolmesGPT) went deep on the worker and never listed Services or endpoints. Aurora said the frontend was fine because its pods were Running, which is true of the pods and false for users. K8sGPT does not read the alert; k8sgpt analyze scans the namespace, so it saw the empty Service, but it never explained the OOM and suggested labelling pods by hand instead of reverting the selector. Aurora's grades were read by a person; the HolmesGPT and K8sGPT grades are from a judge model with a second model review and a blinded regrade (benchmark).

Compare that with the scenarios where the alert pointed at the wrong object. In six scenarios (07, 09, 11, 13, 14, 15) the cause sat in a different object from the one the alert named: the api Service's selector or targetPort, redis's PVC, a NetworkPolicy, a ResourceQuota, a stray Deployment. Aurora was correct in 11 of 12 of those runs and HolmesGPT in 10 of 12, both following the dependency chain from the symptom to the cause; the three misses are partial grades on scenario 11, where all three runs still found the cause. In the twelve scenarios where the alert named the broken workload itself, Aurora was correct in 23 of 24 and HolmesGPT in 21 of 24. K8sGPT, which does not read the alert, had no run graded correct in either group (0 of 12, 0 of 24). The grouping is ours, made from each scenario's root cause: we counted 05 and 19, where the cause is the frontend's own nginx ConfigMap, as "the alert named the broken workload". Counting them as "cause elsewhere" instead gives Aurora 20 of 20 and 14 of 16, HolmesGPT 18 of 20 and 13 of 16. Scenario 12 is 0 of 4 for the alert-led tools either way (counts from results.json).

A misleading alert cost little. A missing alert cost the whole second fault in all 4 runs of the two investigators that start from the alert. It is one scenario, so read it as a strong signal, not a rate. Our position on alert fatigue follows from it: when you cut alerts, cut the cause alerts and keep at least one symptom alert per user-facing path, because neither people nor agents look where no alert points. The runbook for this case is when the alert covers only one of two faults.

Do noisy logs mislead investigators?

Less than we expected. In scenario 11 the worker logs a DeprecationWarning on every batch and a "metrics push failed" ERROR every third, while the real fault is an api Service targetPort of 9090 where nothing listens. All four alert-led runs (Aurora r1 and r2, HolmesGPT r1 and r2) found the targetPort and none blamed the worker. Only Aurora r1 explicitly ruled the worker out; Aurora r2 and HolmesGPT r1 got partial credit for that point and HolmesGPT r2 never mentioned it. K8sGPT reported "No problems detected" in both runs, which fits the fault: every pod was Ready and the Service had endpoints. Log noise wastes the reader's time; in our runs it did not change the answer. A missing alert did.

How do you reduce alert fatigue?

In the order we would do it:

1. Count pages per shift and the action each one led to. Use the SRE Workbook's yardstick: at most two incidents per shift, counting one problem as one incident however many alerts it fired. Anything that led to no action is a candidate for deletion.
2. Fix or remove rules that fire on a healthy system. On our kind cluster, all 11 healthy-cluster alerts came from scrape targets Prometheus could not reach. Fix the scrape first (for kube-proxy, the chart README's metricsBindAddress: 0.0.0.0:10249). If a component cannot be scraped at all, for example a control plane run by your cloud provider, turn the rule off in values (defaultRules.disabled, e.g. KubeControllerManagerDown: true) and write down why in the same change. Do not silence it by hand; a hand-made silence is runtime state that drifts (why muted alerts come back).
3. Page on symptoms, ticket on causes. Page on user-facing error rate and latency at the edge; send KubePodCrashLooping-style cause alerts to a ticket queue or dashboard. Prometheus: "For error rates, page on user-visible errors. If there are errors further down the stack that will cause such a failure, there is no need to page on them separately."
4. Make sure every user-facing path has a symptom alert. This is the scenario 12 lesson. A Service with no endpoints fired nothing under the stock rules for 18 minutes; an error-rate or probe alert at the frontend would have caught it.
5. Group and inhibit in Alertmanager. Group by namespace or service so alerts for one problem that fire together arrive as one notification, and inhibit lower severities while a higher one fires for the same thing. Inhibition is "suppressing notifications for certain alerts if certain other alerts are already firing" (Alertmanager docs).
6. Tune for and keep_firing_for for flapping conditions, knowing that a long for delays real pages too. Ours delayed the first page for the OOM-killed worker to about 16 minutes after the change.
7. Review every page in the weekly handoff. Each one either led to action, gets fixed, or gets deleted.

Can AI triage fix alert fatigue?

It can cut the cost of each alert, and it cannot see what no alert points at. An agent that investigates every page returns a cause and evidence in minutes instead of waking someone to do the first pass, so a noisy rule costs less. That is a real reduction in toil. It does not change the number of alerts, and it inherits the alert's scope: in scenario 12, both agentic investigators were thorough about the worker and silent about the outage next to it. A tool that scans everything without reading the alert (K8sGPT here) saw the second fault and did not explain either one. For SRE teams overwhelmed with alerts, the order is: fix coverage and noise in the rules first, then let an agent take the first look at what remains.

Where this evidence is thin

Arvo AI builds Aurora and wrote the lab scenarios. The lab alerts are scripted, so the investigation results say how investigators respond to an alert's scope, not how real alerting behaves. The stock-rules run covers 20 healthy minutes and two faults for 18 minutes each, once, on kind; a managed cluster would not show the same control-plane scrape failures, the webhook log has a gap before 19:55, and the other 17 faults have not been run under real rules yet. Scenario 12 is one scenario with six graded investigations. We would rather publish small, exact numbers than large borrowed ones.

How Aurora handles this

Aurora takes alerts from its monitoring connectors and runs an investigation on each one; in our lab runs it used read-only commands under the observability_only command policy. Its alert correlator links related alerts that arrive close together into the open incident and its running investigation, so one problem gets one investigation; it never suppresses or mutes an alert. It does not do paging, on-call scheduling or anomaly detection; those stay with your alerting stack. Scenario 12 shows its limit: it reported the alerted fault correctly and called the unalerted frontend unaffected. The full trace is in the lab report. Source: github.com/Arvo-AI/aurora; to see it on your own alerts, book a demo.

Related: automated alert noise reduction, on-call burnout, silence drift, root cause analysis guide, Service has no endpoints, benchmark.

Sources and verification

Verified 2026-10-29. Stock-rules run, 2026-10-29 19:37 to 20:37 UTC: kube-prometheus-stack 92.15.0, Prometheus v3.15.1, Alertmanager v0.34.2, prometheus-operator v0.94.2, kind v0.33.0, Kubernetes v1.37.0; snapshots, webhook log and scripts in the lab repo under runs/search-2026-10-29/alert-fatigue/run2/ (the discarded first attempt is in run1-aborted/). Investigation results: aurora-lab benchmark/results.json (Aurora runs of 2026-10-02, HolmesGPT and K8sGPT runs of 2026-10-15 and 16). Quotes: Google SRE book, Monitoring Distributed Systems, Google SRE Workbook, On-Call, Prometheus alerting practices, Prometheus alerting rules, Alertmanager.