Why Your Muted Alert Came Back: Silence Drift (2026)

Key Takeaways

• Alert silence drift is what happens when a mute created by hand in a UI disappears, because the mute is runtime state and the monitor it belongs to is declared in version control. The two stores never exchange information, so the alert comes back and nobody decided that it should.
• The gap is documented by the tool vendors themselves. The Datadog Terraform provider began deprecating its monitor silenced attribute in v2.0.0 (June 2019) and removed it in v3.0.0 (May 2021), moving muting to separate downtime resources. Prometheus Alertmanager configures silences in the web interface rather than the config file, and garbage-collects them a default 120 hours after they end.
• Google's SRE book treats this class of failure as postmortem-worthy. It lists "a monitoring failure (which usually implies manual incident discovery)" as a trigger for writing a postmortem, alongside data loss and user-visible downtime.
• The durable fix is to move the decision into the repository, as a reviewed downtime or silence resource with an owner, an expiry, and a diff. Terraform's ignore_changes is the documented escape hatch for shared ownership, and it is a tradeoff, not a solution.
• Aurora can do the codifying work as a background agent, and this post is precise about the boundary. Aurora does not mute, silence, or suppress anything. It edits the infrastructure code and opens a pull request. A human merges it, and the customer's own apply is what changes the monitor.
• The part that makes it stick is memory. Aurora Actions maintain a persistent, versioned artifact that the agent reads before each run and reconciles afterward, so a mute you already codified is not proposed twice and the agent is instructed to leave anything you filed under "won't fix" alone.

Alert silence drift is the failure mode where an alert someone deliberately muted starts firing again, because the mute was created as runtime state in a monitoring tool while the monitor itself is declared in an infrastructure-as-code repository, and neither store knows the other exists. Your alerting tool is not malfunctioning when this happens. It is the predictable result of keeping one decision in two places, and it is one of the few on-call annoyances with a clean, verifiable, well-documented cause.

This post is for SRE and platform teams who manage monitors in Terraform and mute them in a browser. It explains exactly why the mute vanishes, what each major vendor's documentation says about it, and what to do instead.

Why does a muted alert come back?

Because you wrote the mute in a place your repository cannot see, and the monitor in a place your browser cannot change.

Look at what each vendor actually documents.

Tool	Where a manual mute lives	What ends it
Datadog	The Mute button on the monitor status page, or a scheduled downtime	The downtime window closing. Datadog notes that "muting or un-muting a monitor from the monitor status page does not delete scheduled downtimes associated with the monitor"
Prometheus Alertmanager	A silence, and the docs are explicit: "Silences are configured in the web interface of the Alertmanager"	Its end time, then garbage collection. The data.retention flag defaults to 120h
Grafana	A silence with a fixed start and end time, which stops notifications without interrupting alert evaluation	Its end time. Expired silences are deleted after five days

Every one of those is runtime state with a clock attached. None of them is in your repository. So there are two independent ways the alert returns:

1. The mute expires on its own schedule. This is the intended behavior, working exactly as designed. A silence is temporary by construction.
2. The next apply reconciles the monitor with no knowledge of the mute. HashiCorp documents that "Terraform plan and apply operations run an implicit in-memory refresh as part of their operations, reconciling any drift" against declared configuration. If your mute is expressed as a monitor attribute rather than a separate downtime, the declared config wins.

The second path is the one people find genuinely surprising, and it has a specific history worth knowing.

Why is there no mute field in the Datadog Terraform provider?

Because it was deliberately removed, and the removal was correct.

The provider's own changelog records the sequence. In v2.0.0 (June 18, 2019), the note reads: "datadog_monitor: The silence attribute is beginning its deprecation process, please use datadog_downtime instead." Then in v3.0.0 (May 27, 2021): "datadog_monitor: Remove the silenced deprecated property." There is an intermediate entry in v2.7.0 that is a perfect fossil of the pain: "improve monitor example with ignoring changes on silenced."

That middle step is the tell. Practitioners were reaching for lifecycle { ignore_changes = [...] } to stop Terraform fighting their manual mutes. HashiCorp's documentation frames that meta-argument honestly: it exists for when "a remote object's settings are modified by processes outside of Terraform" and you want "to let Terraform share management responsibilities of a single object with a separate process."

Sharing management of one field between a declarative system and a human with a mouse is a workable tradeoff for some attributes. For muting, it is a bad one, because it makes the muted state permanently invisible to code review. Deleting the attribute and giving muting its own resource, datadog_downtime_schedule, was the right call: a downtime has a scope, a schedule with an rrule, and a lifecycle of its own, which is what a mute genuinely is.

The consequence is the thing to internalize. In the current provider, muting a monitor from the UI is by design not representable in the monitor resource. If you want a mute in code, it has to be a different resource that somebody writes.

Why does this matter more than an annoyance?

Because a mute is a decision, and this pattern throws away the decision while keeping its effect.

When an engineer mutes an alert at 3am, they are recording a judgment: this is firing, it is not worth waking someone for, and here is roughly why. The mute persists the effect of that judgment for a few hours. The reasoning evaporates immediately. Nobody can review it, nobody inherits it, and when the alert returns the next person starts from zero.

Two authoritative sources make the stakes concrete.

Google's SRE book sets the standard the alert failed to meet in the first place. Its monitoring chapter states that "every page should be actionable" and that "every page response should require intelligence. If a page merely merits a robotic response, it shouldn't be a page." It also names the human limit directly: "I can only react with a sense of urgency a few times a day before I become fatigued." An alert you keep muting is an alert that fails this test. Muting treats the symptom of a monitoring design problem.

The same book's postmortem chapter puts a monitoring failure on the list of things that warrant a written postmortem, naming "a monitoring failure (which usually implies manual incident discovery)" alongside data loss and user-visible downtime. A mute that outlives its reason is a monitoring failure waiting for its turn: the alert is suppressed, the underlying condition is not, and the discovery path becomes a customer.

There is a real risk in the other direction too, and it deserves equal weight. We covered it in our alert noise reduction guide: a mute rule written for last quarter's flapping alert can silently mute this quarter's genuine outage that happens to match the same pattern. Suppression is quiet by nature. A dashboard looks identical whether you eliminated noise or stopped seeing signal.

If you want the canonical example of this failure mode at scale, it is not from software. The official U.S. and Canadian investigation into the August 2003 Northeast blackout records that an analyst troubleshooting a problem "had turned off the automatic trigger that runs the state estimator every five minutes. After fixing the problem he forgot to re-enable it," and that the deactivation was discovered hours later. The report names that alongside a separate control-room alarm failure where operators were "running a complex power system without adequate indicators" and "without awareness that they were running the system without these alarms." A monitoring capability turned off by a human for a good reason, and never turned back on, is the exact shape of the problem, and it scaled to 55 million people.

The software version is quieter and more common. GitLab's January 2017 database incident postmortem records that backup failure notifications were being sent but rejected in transit because DMARC was not configured for those emails, so nobody learned the backups were broken until they were needed. The check existed. The notification path had silently stopped working, which is functionally a mute nobody chose.

So drift cuts both ways. A mute that disappears wakes you up for nothing. A mute that persists too long stops waking you when it matters. Both are the same root problem: the mute has no owner, no expiry you chose deliberately, and no written reason.

What does the fix actually look like?

Move the decision into the repository. Concretely, that means four properties every mute should have.

1. A resource, not a button. Express the mute as datadog_downtime_schedule, a Grafana mute timing, or an Alertmanager inhibition rule where the relationship is genuinely symptom-to-cause. The Alertmanager docs note that unlike silences, "inhibitions are configured through the Alertmanager's configuration file," which makes them reviewable by default.
2. An explicit expiry. A downtime with an rrule or a one-time window states when the exception ends. Indefinite mutes are how alerting decays.
3. A written reason in the diff. The pull request body is the only place the 3am judgment survives contact with next quarter.
4. An owner. Version control gives you this for free through blame.

One caveat: this is more work than clicking Mute, and at 3am you should still click Mute. The fast path is worth keeping, as long as it gets reconciled into code before the mute silently expires or silently persists. That reconciliation is recurring, boring, easily-forgotten work, which is exactly the kind of work worth handing to an agent.

How can a background agent maintain muting as code?

This is where Aurora fits, and it is worth being exact about what it does and does not do, because the distinction is the whole design.

What Aurora does not do. Aurora has no mute, silence, snooze, or downtime feature. It does not call the Datadog mute API. It does not create Alertmanager silences. It does not suppress alerts, and as we wrote in the alert noise reduction guide, it deliberately does not mute, snooze, rate-limit, or flap-detect anything.

What Aurora does. It edits infrastructure code and opens a pull request, and that is the whole mechanism. Your merge and your apply are what change the monitor.

The primitive is an Aurora Action: a named natural-language instruction that runs as a background agent, triggered manually, on an incident, or on a recurring schedule down to a five minute floor. For this use case the instruction is close to what you would tell a teammate: find the Terraform that defines this monitor, add or update the downtime resource, explain in the PR why the alert was noisy, and do not touch anything else.

The safety boundary is a permission model rather than a prompt instruction. In a background run, an Aurora agent can write Terraform files, commit, and open a pull request. Running terraform apply or destroy, and merging a pull request, are denied for unattended runs and are not enabled by default, so an org admin has to grant them deliberately. The Actions deep dive covers the trigger and permission model, and our AI agent guardrails post covers the seven independent layers every command passes through. On the default configuration the agent's output is a diff for a human to read, which is the correct output for a change to your alerting posture.

Why memory is the part that matters

A one-shot agent that proposes mutes is not useful for long. Run it weekly and it re-proposes the same three downtimes forever, including the one you already merged and the one you explicitly rejected. That agent gets turned off in a fortnight.

Aurora Actions maintain a persistent artifact: a titled, org-scoped, fully versioned document stored alongside the agent's other state. Every action run is instructed to read the prior version first, then write the same document back, reconciling rather than regenerating. Add what is new, drop what is now resolved, and preserve what a human changed. Anything filed under a false-positives or won't-fix heading the agent is instructed to preserve verbatim and never re-raise, even if a fresh scan surfaces it again, and if you edited the document by hand your version is treated as authoritative. That instruction is steering rather than a hard constraint, which is why the harder guarantee matters: every write creates a new version, so a bad overwrite is visible and restorable.

That is a small mechanism with a large behavioral consequence. It converts a recurring agent from something that repeats itself into something that accumulates a decision log. The artifact becomes the written record the manual mute never had: which alerts you chose to silence, why, and which ones you decided were not worth the effort.

Where does this sit next to your existing alerting stack?

It complements every layer rather than replacing one.

Layer	Job	Where drift bites	What to do
Alertmanager inhibition and routing	Deterministic symptom suppression, grouping, routing	Config-file rules are reviewable; UI silences are not	Prefer inhibition for true symptom-to-cause relationships
Correlation tools like Keep or BigPanda	Cluster related alerts into one notification	Correlation rules drift like any other rule	Keep the rules in version control
Monitor definitions in Terraform	Declare what is watched and at what threshold	Manual mutes are invisible here by design	Express mutes as downtime resources
Investigation agents	Decide whether a firing alert is real	Not a drift surface	See AI-powered incident investigation

The ordering that works: suppress only what you can deterministically prove is a symptom, correlate the rest, codify every deliberate exception as a reviewed resource, and let something investigate what remains. Drift is what you get when step three is done in a browser.

The uncomfortable part: a returning alert is usually correct

There is a version of this post that ends with "so automate your mutes and enjoy the silence." That would be the wrong ending.

If an alert keeps coming back and you keep muting it, the drift is not really the problem. The problem is an alert that fails Google's actionability test and has been failing it for months, and the mute is how the team avoids the harder conversation about deleting it, rewriting its threshold, or fixing the thing it keeps noticing. Codifying that mute makes the avoidance durable and reviewable, which is a genuine improvement, but it is still avoidance.

The reason to put mutes in code is not to mute more comfortably. It is that a mute in a pull request is visible, so somebody eventually reads the diff and asks the question a UI mute lets everyone skip: should this alert exist at all?

• GitHub: github.com/Arvo-AI/aurora
• Docs: Aurora documentation
• Related guides: Automated Alert Noise Reduction · Aurora Actions · Open Source Incident Management

All claims sourced. Vendor behavior is cited to provider changelogs, official documentation, and upstream source code. Aurora's capabilities and its limits are described from its open-source implementation. Last verified: July 24, 2026.