Incident management and postmortems

In short: The orderly handling of outages: detect quickly, fix, communicate and afterwards learn from it without blame.

In more detail: An incident is an event that unexpectedly impairs a service, such as an outage or sharply increased error rates. Incident management regulates roles, procedures and communication so that people do not have to improvise under pressure. The postmortem (debrief) records what happened and how to prevent it in future.

In Depth

Process during an incident

1. Detect: alert from monitoring or a report from users.
2. Classify: how severe is it (severity, affected users)?
3. Coordinate: one person leads (incident commander), others investigate, one communicates.
4. Mitigate: first stop the impact (roll back, switch off, fail over), only then look for the cause.
5. Inform: status page and customer notices with an honest state of affairs.
6. Close: confirm normal operation.

The postmortem

It contains a summary, a timeline with times, the impact (duration, number affected), the causes (usually several, not just “one error”), what went well and badly, and concrete actions with owners and dates. It helps to ask “why?” several times in a row (“five whys”).

Without blame

A “blameless” postmortem does not ask who was at fault but why the system allowed a person to make a mistake. People who fear accusations conceal things, and the organisation learns less. Human errors are usually a sign of missing safeguards such as reviews, tests or fallbacks.

Metrics

Common ones are MTTD (time to detect), MTTR (time to restore) and the number of incidents. They should give hints, not serve as a competition. Even small projects benefit from a short note after every outage.

See also: SLI, SLO and error budget, SLA, support level and ticket system, Chaos engineering