How a Retry Loop Took Down GitHub (2026 Postmortem)

Key Takeaways

• On 13 September 2026, GitHub degraded across roughly 28 services for about two hours, and the trigger was self-inflicted: a routine background job, not an attack or a traffic spike. The public incident report resolved the incident at 10:44 UTC.
• The safeguard meant to pace the job watched the wrong signal. It monitored database replica lag, which "stayed low the whole time," while the primary database quietly ran toward its connection limit.
• An unbounded retry loop turned saturation into an outage. When token creation began failing, "a retry loop around token creation kept re-sending the writes that were already failing," holding the database saturated.
• The blast radius was wide but the trigger was narrow. At peak, "8.8% of requests to create GitHub App installation access tokens failed," about 4% of Actions workflows were affected, and web issue creation failed for roughly 96% of attempts.
• This is a textbook cascade: shared dependency, wrong health signal, no fast timeout, retry amplification. Each is a general reliability failure mode, not a GitHub-specific bug.
• The investigation lesson is the distance between symptom and cause. The visible failures spanned signup, Issues, and Actions; the trigger was one cleanup job on a shared permissions database.
• Fixes are structural. GitHub's remediation includes pacing background jobs on primary load rather than replica lag, bounding retries, adding request timeouts, and splitting the shared cluster.

A retry storm is a failure mode where clients retrying failed calls add load to a struggling system and keep it saturated, turning a brief degradation into a sustained outage. GitHub's 13 September 2026 incident is a clean public example, triggered by an internal background job. For SREs, the incident is worth reading not for the specifics of GitHub's stack but because every link in its chain, a shared dependency, a health check on the wrong metric, a missing timeout, and an unbounded retry, is a pattern that recurs across systems.

What happened in the GitHub incident on 13 September 2026?

A background data-cleanup job saturated a shared database that nearly every authenticated request depends on. According to GitHub's incident report, the job "began writing to a shared database cluster" that "holds permission data read on nearly every authenticated request." The incident window ran roughly 08:43 to 10:44 UTC, with GitHub declaring the incident shortly after monitoring caught it and marking full recovery at 10:44 UTC.

The effect fanned out across about 28 services because the saturated resource was shared. Users saw failures in Issues, Pull Requests, Actions, Codespaces, Pages, and signup, among others. That fan-out is the first lesson: when a single shared dependency degrades, the symptom surface is enormous and points everywhere except the actual cause. An on-call engineer watching 28 services alert at once has to work backward to one cleanup job, which is exactly the kind of correlation an investigation has to perform under pressure.

Why did GitHub's safeguard fail to stop the job?

Because it watched a signal that stayed healthy while the real resource was exhausted. The job had a pacing safeguard, but per GitHub it "watched only one health signal, how far the database replicas were lagging," and that signal "stayed low the whole time." Replica lag measures how far behind read replicas are; it says little about write pressure on the primary. So the safeguard "did not account for the load building on the primary itself," and the job kept writing "while the primary quietly ran toward its limit."

This is a precise, transferable lesson: a safeguard is only as good as the signal it watches, and the intuitive signal is often not the limiting one. Replica lag is a natural thing to monitor and a common proxy for database health, but it is a lagging and indirect measure of the primary's connection capacity. A pacing control that throttles on the resource actually being consumed, primary connections or write throughput, would have slowed the job before saturation. Watching an adjacent metric gave false confidence.

How did a retry loop turn saturation into an outage?

By re-sending failing writes fast enough to keep the database from recovering. Once the primary "ran out of available connections," two amplifiers turned a slowdown into a sustained outage. First, calls had no fast timeout, so handlers "waited on the stalled database instead of failing fast," tying up shared capacity. Second, and decisively, "a retry loop around token creation kept re-sending the writes that were already failing," which kept the database saturated.

Link in the chain	What happened	General failure mode
Shared dependency	Cleanup job wrote to a permissions DB on the auth path	Single point of failure
Wrong health signal	Pacing watched replica lag, not primary load	Monitoring the wrong metric
No fast timeout	Handlers waited on the stalled DB	Missing timeout / bulkhead
Retry amplification	Failed token writes were re-sent in a loop	Retry storm

A retry storm is one of the most studied cascade patterns in distributed systems, and the defense is well known: bound retries, add jitter and backoff, and shed load rather than queue it. GitHub's own recovery step was internal load-shedding, after which it reported signs of recovery. The general principle is that retries are a client's way of adding load precisely when a system can least absorb it, so they must be capped and backed off, never unbounded.

What was the actual impact of the outage?

Wide but partial, and GitHub published specific numbers rather than vague ranges. At peak, per the report, "8.8% of requests to create GitHub App installation access tokens failed." Token issuance problems hit "approximately 4% of workflows" in Actions, creating issues through the web interface "failed for about 96% of attempts," and "signup failures were above 90%." The spread of those numbers is itself instructive: the token-creation failure rate was single digits while issue creation and signup, which depend more directly on the saturated write path, failed almost entirely. Same root cause, very different blast radius per feature, because each feature leaned on the degraded resource to a different degree.

For anyone writing a postmortem, this is the way to state impact: name the metric, the peak percentage, and the affected feature, and resist rounding a 4% failure rate and a 96% failure rate into a single "degraded" label. Aurora never publishes invented impact figures, and neither did GitHub here; the numbers above are quoted from GitHub's own report.

What are the reliability lessons from the GitHub retry storm?

Four, each general enough to apply to almost any production system.

1. Rate-limit background work against shared customer-serving stores. A cleanup job should not be able to saturate a database on the request path. GitHub's stated fix is "rate-limiting background jobs against shared customer-serving databases by default."
2. Pace on the resource you are actually consuming. Watch primary load, not replica lag. GitHub is moving to auto-pause and page "on primary-server load rather than replication lag alone."
3. Bound every retry and add fast timeouts. An unbounded retry loop is a load amplifier. The fixes include "bounding retries in the token-issuing path" and adding request-level timeouts so calls fail fast instead of piling up.
4. Remove single points of failure. GitHub is "breaking apart this database cluster to remove the single point of failure," moving service-specific data off the shared cluster.

The investigation angle ties these together. When 28 services fail at once, the work is to trace the fan-out back to the one saturated resource and the one job that saturated it. An AI-powered incident investigation agent that correlates a spread of symptoms against a dependency graph is built for exactly that reverse walk, from a wall of alerts to a single background job on a shared primary. Aurora runs that correlation as a single investigating agent by default and performs a bounded traversal of infrastructure dependencies to reason about why so many surface services degraded from one cluster, without claiming to have predicted the failure or to remediate it unattended. How that blast-radius reasoning behaves is covered in blast radius in automated remediation, and the reverse walk from symptom to cause in root cause analysis for SREs.

The summary

GitHub's 13 September 2026 degradation was a self-inflicted retry storm: a background cleanup job wrote to a shared permissions database, a pacing safeguard watched replica lag instead of primary load and never throttled it, missing timeouts let handlers stall, and an unbounded retry loop on token creation kept the database saturated. The result touched about 28 services with per-feature failure rates from 8.8% to above 90%. The fixes are structural: pace background work on the right signal, bound retries, add timeouts, and split the single point of failure. The general reliability lessons outlast the specific incident.

Try Aurora

• Start free: aurora-ai.net (hosted, no infrastructure to run)
• GitHub: github.com/Arvo-AI/aurora
• Book a demo: cal.com/arvo-ai/demo
• See it on an incident: Aurora root cause analysis

Sourcing note. The trigger, the shared permissions database, the replica-lag safeguard, the retry loop, the timeline, the roughly 28 affected services, and every impact percentage (8.8% token creation, about 4% of Actions workflows, about 96% of issue creation, above 90% of signups) are quoted or paraphrased from GitHub's public incident report at githubstatus.com, incident 0rn90wk115q9, resolved 13 September 2026, verified 24 September 2026. No customer counts, dollar figures, or MTTR numbers are asserted because GitHub's report does not state them. Aurora's investigation behavior is described from its open-source repository.