Demo incident: package installs failing for some users
- Started
- Resolved
- Duration
- 1 h 2 min
Affected components
- GitHub: Degraded performance
- npm registry: Partial outage
Updates
- Resolved
Demo incident: resolved. Package installs and GitHub have worked normally for 20 minutes. A short postmortem is below.
- Monitoring
Demo incident: a fix is in place and installs are succeeding again. We're watching error rates before we call this resolved.
- Identified
Demo incident: we've found the cause, an overloaded cache in front of the package mirror, and are moving traffic to healthy nodes.
- Investigating
Demo incident: this is an example written to show how updates look, and it did not happen. We're investigating failed package installs from the npm registry. GitHub is loading slowly for some users.
Postmortem
This is a demo postmortem. The incident it describes is an example written for this page and did not happen.
What happened: a cache tier in front of the package mirror ran out of memory after a traffic spike, so some requests timed out.
Impact: for about 40 minutes, roughly one in five package installs failed and pages loaded slowly for some visitors.
What we're changing:
- Alert on cache memory pressure well before it becomes critical.
- Shed load gradually instead of timing out.
- Run a quarterly failover drill for the cache tier.