Runbooks: Step-by-step guides for operations

In short: Short, tested guides describing how to recognise and fix a known problem or carry out a routine task.

In more detail: A runbook answers the question “What do I do now?” at three in the morning. Instead of relying on one person’s memory, the procedure is written down and ready. Good runbooks reduce time to resolution and make on-call duty fairer, because new team members can help too.

In Depth

Structure of a good runbook

• Trigger: which alert or symptom does it belong to?
• Impact: who is affected, how urgent?
• Diagnosis: which commands or dashboards show the cause?
• Actions: numbered, copyable steps, ideally with the expected result.
• Undo: how do you reverse a step?
• Escalation: whom to turn to if it does not work?
• Status: date of the last review.

Example (excerpt)

Symptom: alert "disk full" on web-01
1. Show usage:          df -h
2. Largest folders:     du -sh /var/* | sort -h
3. Delete old logs:     sudo journalctl --vacuum-time=7d
4. Check result:        df -h   (expect below 80 %)
Escalation: infrastructure on-call if not below 90 %

Maintenance

An outdated runbook is dangerous. After every incident, check whether it was correct and adjust it. Link it directly in the alert. Even better is to automate recurring steps (scripts, “runbook automation”) so that the human only confirms.

Distinction

A playbook usually describes a higher-level procedure (for example the approach to a security incident), a runbook concrete technical steps. In practice the boundary is blurred.

See also: Incident management and postmortems, Logging, SLA and ticket system