When an incident becomes confusing, teams often add communication: another call, another channel, another person asking for an update. The result can be less information and fewer decisions.
In one recurring incident shape, several people investigated different components while nobody owned the customer outcome. Each observation was locally useful. Together they formed a noisy, leaderless system.
The bottleneck #
The bottleneck was serialized decision-making. Someone needed to maintain the current hypothesis, choose the next safe action, record what changed, and decide when evidence justified rollback. Investigators needed freedom to test bounded hypotheses without simultaneously explaining the entire incident.
We separated responsibilities:
- incident lead: owns outcome, priority, and decisions;
- investigators: gather evidence and report concise findings;
- scribe: preserves timeline, actions, and open questions;
- communications owner: translates facts without inventing certainty.
One person can hold multiple roles in a small team. The responsibilities still need names.
Reversibility beats cleverness #
Under pressure, the best action often reduces harm and preserves the next decision: pause a risky worker, disable one feature path, route traffic away from a cell, or roll back when rollback is safer than diagnosing live.
We used a small decision record:
time | observation | hypothesis | action | owner | expected signalThe expected signal matters. Without it, an action becomes activity rather than an experiment.
A leader joining an incident can accidentally reset the room by asking everyone to retell the story. Read the timeline first. Ask what decision is blocked. Remove obstacles. Do not become a second incident commander by seniority.
Psychological safety has operational value. Engineers report contradictory evidence faster when they are not defending earlier hypotheses. A clean correction is more valuable than consistency with a guess made twenty minutes ago.
After recovery, review the management system: when ownership became clear, which signal changed the decision, which action increased ambiguity, and whether support could state customer impact.
Incident response is a distributed system made of people. A clear owner, explicit roles, shared timeline, and reversible actions create the control plane the team needs when the technical control plane is already failing.
All Field Notes · Incident command is a distributed system · Control planes must fail quietly