Incident Leadership

⏱ ~9 min read · 🎧 9 min listen · ✎ 3 quizzes · 🧪 ~60 min lab

Nothing makes staff+ operation more visible than a major incident. The whole org is watching one Slack channel; executives who've never read your ADRs are reading every message. Whoever brings order to that room is doing leadership without authority in its purest form — which is why incident stories are among the strongest promotion evidence and interview stories there are. But here's the counter-intuitive part this lesson is built on: in a well-run incident, the leader is not the person fixing the system. Leading and debugging are different jobs, and mixing them breaks both.

The Incident Commander model

PagerDuty open-sourced its real internal major-incident process — adapted from the Incident Command System that fire departments and emergency services have used for decades.¹ Its center is the Incident Commander (IC): the single source of truth and decision-maker during the incident. And the guide is blunt about what the IC does not do: "You should not be performing any actions or remediations, checking graphs, or investigating logs" — deep technical knowledge is explicitly not required.¹ The IC's job is coordination: keep a shared picture of what's happening, decide what happens next, and keep everyone else's hands free to work. The worst IC is often the best engineer on the failing service — because they dive into a terminal, and the moment they do, nobody is commanding the incident.

On small incidents one person may wear several hats; the structure flexes. What never flexes is the separation of commanding from fixing. And note what this means for you: you don't need any title to take the IC role — you just need to be the person who brings the structure when nobody else does.

Severity is a decision rule, not a debate

The second load-bearing idea: severity levels are pre-agreed decision rules, set in peacetime — not something to negotiate mid-fire. Each level maps to a response: who gets paged, whether execs are woken, whether you post publicly. PagerDuty's rule for ambiguity is the one to memorize: if you're unsure whether it's a SEV-2 or a SEV-1, treat it as the higher one — and re-grade calmly in the review afterwards.²

Talk like a commander

Calm is a skill, and it lives in specific sentence shapes. Decisions use propose-and-object: "I propose we roll back the 14:02 deploy — any strong objections?" — implicit consensus in seconds, no committee. Tasks are assigned by name, with a time-box: "Priya, please check the connection pool change; I'll come back to you in ten minutes — understood?" Unnamed tasks evaporate into the bystander effect. Even executives get a script: if one starts directing the call, the IC asks "do you wish to take command?" — and if not, they observe quietly.¹ Practice these shapes on small incidents so they're automatic on big ones; steadiness under pressure is rehearsed, not innate.

The blameless postmortem

The incident ends; the leadership work doesn't. The Google SRE book's chapter on postmortem culture sets the rule: a blameless postmortem assumes everyone involved acted with good intentions on the information they had — the failure belongs to the system, and the fix is better systems, not "fixed" people.³ John Allspaw's canonical essay sharpens the method: "human error" is where the analysis starts, never where it ends.⁴ If an engineer skipped the canary, the real questions are: why was skipping possible, why did it look reasonable at the time, why did nothing downstream catch it? That's also why strong reviews list contributing factors, plural — real failures are a conjunction of code, process, and detection gaps, and hunting a single "root cause" is mostly theater that stops at the first comfortable answer.

Here's the Principal angle: the postmortem document is org-level leverage. The incident cost you once; the document collects the return — it changes review checklists, deploy pipelines, alert thresholds, and how teams you've never met build things. That's why Google treats postmortems as artifacts to be reviewed by senior engineers, shared across the company, and visibly rewarded by leadership.³ And blamelessness isn't only ethics — it's engineering: people who fear blame hide facts, and a postmortem without facts fixes nothing. A calm IC plus a rigorous blameless review, witnessed by the whole org, is staff+ evidence money can't buy.

🧪 Lab: write a blameless incident review

This lesson's artifact is a one-to-two-page blameless incident review of a real past incident you were near — as responder, bystander, or cleanup crew. Old is fine; real is mandatory. Use this template, in order:

1. Timeline — timestamped facts only: first signal, detection, escalation, key decisions, resolution. No interpretation yet.
2. Impact in user and SLO terms — what did users actually experience, for how long, and how much error budget did it burn? (Lesson 07's vocabulary, applied.)
3. Contributing factors — at least three, spanning more than one layer: code, process, detection. The phrase "root cause" is banned from the doc.
4. What went well — the alert that fired, the rollback that worked, the person who coordinated. Reviews that skip this teach half the lesson.
5. Action items — each systemic and owner-shaped: "platform team adds a canary gate to the deploy pipeline," never "engineers should be more careful."

Feedback loop: bring it back to me in chat. I'll review it against a Principal-level rubric — no names attached to blame anywhere; at least three contributing factors that span more than one layer; action items that change the system rather than exhort the humans. This review joins your evidence trail for Lesson 12's promotion packet — and it's a ready-made answer to the interview staple "tell me about a production incident."

You are a seasoned Principal Engineer reviewing my work: a one-to-two-page blameless incident review of a real production incident (timeline, impact, contributing factors, what went well, action items). Grade it against this rubric:

- Blameless throughout: no names attached to blame anywhere; analysis never stops at "human error."
- At least three contributing factors, spanning more than one layer — code, process, detection — with no single "root cause" framing.
- Impact stated in user and SLO terms: what users experienced, for how long, and the error budget burned.
- Action items are systemic and owner-shaped — they change the system, never exhort humans to "be more careful."

Grade each criterion Strong / Adequate / Missing with one sentence of evidence. Be skeptical — challenge the weakest claim first. Then ask me 2–3 follow-up questions a senior postmortem reviewer or interviewer would ask. Finish with the single highest-leverage improvement.

My blameless incident review follows below.

Check yourself — command the scenario

Three scenarios. Decide from the mental model — don't scroll up. Wrong picks stay live.