Incident Command Is a Distributed System

Aug 29

By

reliabilityincident responsetechnical leadershipobservability

During a serious outage, the organization itself becomes a distributed system. People have partial observations, messages arrive out of order, ownership changes, actions race, and the shared model of reality becomes stale.

Adding more engineers often reduces throughput because communication grows faster than useful work. Incident command exists to impose a small, reliable coordination protocol on that chaos.

The governing invariant is:

At any moment, one person owns incident priority and one shared record represents the latest accepted state.

This does not mean one person diagnoses everything. It means authority and information have explicit convergence points.

Separate command from investigation

The incident commander owns the process: severity, priorities, workstreams, escalation, and the decision to move between mitigation and recovery. Technical leads own investigation within bounded workstreams. A communications lead owns updates to customers and internal stakeholders.

incident commander
├── mitigation lead
├── database investigation
├── dependency investigation
└── communications lead

Without this split, the strongest technical investigator becomes a meeting router and status writer. With it, specialists can maintain focus while command sees the complete risk picture.

Create one state log

Chat is a transport, not a source of truth. Maintain an incident log containing:

  • start time, severity, and customer effect;
  • current hypothesis and confidence;
  • actions underway, owner, and deadline;
  • completed changes and observed result;
  • decisions explicitly rejected;
  • next update time;
  • links to dashboards, traces, deploys, and tickets.

Use timestamps in UTC. Record observations separately from interpretations:

14:07 observation: checkout 5xx rose from 0.2% to 18% in eu-west
14:09 hypothesis: connection exhaustion after release 8f31; medium confidence
14:11 action: halt rollout; owner Priya; result expected by 14:16

This structure makes handoff possible and reduces repeated investigation.

Mitigate before explaining

Incident response is an optimization problem under uncertainty. The first objective is reducing customer harm, not finding the intellectually complete root cause.

A good mitigation is:

  • reversible;
  • narrow in blast radius;
  • observable within minutes;
  • executable with current authority;
  • independent of the suspected failing control plane where possible.

Rollback, disable, shed, isolate, or serve a known-good snapshot before attempting a complex repair. Every action must have a predicted signal. “Restart it” is not an experiment unless you know what observation will confirm or reject the hypothesis.

Bound coordination

Use short update cadences based on severity. Each workstream reports only:

observation → current hypothesis → action → expected evidence → next checkpoint

Create new workstreams only when they test independent hypotheses. Ten people querying the same database are not ten workstreams; they are contention.

The commander should stop risky or duplicated actions. During an outage, production write access is a scarce capability. Make one owner accountable for each mutation and record it before execution when time permits.

Communicate customer truth

External updates should state what users experience, which surfaces are affected, what mitigation is underway, and when the next update will arrive. Do not publish an unverified root cause.

Internally, distinguish three states:

  • mitigated: customer impact has stopped;
  • recovered: service indicators remain healthy for an agreed window;
  • resolved: temporary controls are removed or converted into durable follow-up.

Calling an incident resolved immediately after a graph falls creates repeat incidents during cleanup.

Learn from the coordination system

A useful review asks more than “what bug caused this?”

  • Why did detection take this long?
  • Which dependency made mitigation difficult?
  • Which access or runbook was missing?
  • Where did responders hold conflicting state?
  • Which action increased risk?
  • Could the system have failed statically instead?
  • What reduces recurrence, impact, or recovery time most economically?

Track actions to owners and evidence of completion. “Improve monitoring” is not an action. “Page when admitted database sessions exceed 80% for ten minutes; validate in game day” is.

Conclusion

Incident management is architecture for humans under partial failure. Explicit authority prevents conflicting writes. A shared log creates convergent state. Bounded workstreams reduce coordination load. Reversible mitigation controls blast radius.

Do not wait for the outage to invent this protocol. Practice it until the organization can execute it while its assumptions—and its dashboards—are failing.

References

>