When a production incident hits, the natural instinct is to scramble. Everyone jumps in, people duplicate effort, communication breaks down, and the resolution takes longer than it should. A defined incident response workflow prevents this by giving every team member a clear role and a predictable sequence of steps.
The workflow follows six phases: detection, acknowledgment, investigation, resolution, communication, and postmortem.
Detection is the starting point. Your monitoring system identifies an anomaly, confirms it from multiple regions, and triggers an alert through your escalation policy.
Automated detection is the goal, but not every incident gets caught by monitors. Customer reports, support tickets, and internal team observations also surface problems. Regardless of the source, every reported issue should follow the same workflow from this point forward.
Hyperping's incident manager provides a centralized place to track incidents from the moment they are detected.
The on-call responder receives the alert and acknowledges it. Acknowledgment tells the rest of the team that someone is actively looking at the problem and stops the alert from escalating further.
Acknowledgment should happen within your defined response window, typically 5-15 minutes for critical incidents. If the on-call responder does not acknowledge, the alert escalates to the secondary responder (see Chapter 5: On-Call Scheduling).
At this stage, the responder makes an initial severity assessment:
| Severity | Impact | Examples |
|---|---|---|
| Critical | Complete outage or data integrity risk | Database unreachable, payment processing down |
| High | Major feature broken, large user impact | Login failing for 50%+ of users |
| Medium | Degraded performance, partial impact | Elevated latency, one region affected |
| Low | Minor issue, limited user impact | Non-critical background job failing |
Critical and high-severity incidents trigger the full response workflow. Medium and low incidents might be handled by the on-call engineer alone.
For critical incidents, assemble the response team and assign roles:
The IC owns the incident from start to finish. They coordinate the response, make decisions about next steps, and track the overall progress. The IC does not need to be the most senior engineer. They need to be organized, calm under pressure, and able to delegate effectively.
The IC's responsibilities:
The comms lead handles all external and internal communication. They update the status page, post to internal Slack channels, notify customer support, and draft customer-facing messages. Separating communication from investigation lets the technical responders focus on fixing the problem.
The engineers doing the actual debugging and remediation. They investigate logs, metrics, and traces, test hypotheses, and implement fixes. They report their findings and progress to the IC.
Follow a structured troubleshooting approach:
Resolution means the service is restored to normal operation. This might involve:
Once the fix is in place, verify recovery by checking your monitoring dashboards and confirming that alerts have cleared. Do not declare an incident resolved until your monitors confirm the service is healthy from all regions.
If a temporary workaround restores service but the root cause is not yet fixed, document the workaround and create a follow-up ticket for the permanent fix.
Communication runs in parallel with every phase, but it deserves special attention as its own step. During the incident:
See Chapter 7: Status Pages and Incident Communication for detailed guidance on what to communicate and how often.
Every critical and high-severity incident should have a postmortem. Schedule it within 48 hours of resolution while the details are fresh.
A good postmortem answers:
Postmortems should be blameless. The goal is to improve systems and processes, not to assign fault. An engineer who deployed a buggy change is not the root cause. The lack of automated testing, canary deployments, or rollback procedures that allowed the bug to reach production is.
Track postmortem action items in your project management tool and review them regularly. An action item without a deadline and an owner is just a wish.
Runbooks are pre-written guides for handling specific types of incidents. A runbook for "database connection pool exhaustion" might include steps to check active connections, identify long-running queries, restart connection pools, and scale the database if needed.
Write runbooks for your most common incident types. Keep them in a location the on-call team can access quickly during an incident (a shared wiki, a docs repo, or directly linked from your monitoring alerts). Review and update runbooks after each postmortem.
The next chapter covers how to communicate with users during incidents through status pages.