A node misses three heartbeats, so the cluster declares it dead. That sentence hides the hardest part of failure detection: the observer cannot distinguish a dead process from a delayed process, a saturated network, a paused runtime, or its own broken connection.
The signal is silence. Failure is an interpretation.
This distinction matters because the action after suspicion may be destructive: reassigning a partition, promoting a replica, revoking a lease, or allowing another worker to produce the same side effect.
Start with the consequence #
Do not choose a timeout before defining what it authorizes.
| Observation | Safe interpretation | Possible action |
|---|---|---|
| one missed probe | transient delay | record latency |
| repeated missed probes | suspect | probe indirectly |
| quorum cannot reach member | member unavailable to quorum | stop routing new work |
| lease expired at authority | ownership ended | fence the old owner |
| process responds again | communication recovered | reconcile state |
The important boundary is that a detector may inform an ownership protocol, but it must not be the ownership protocol. A late worker can wake after its peers replaced it. Storage must reject that stale worker through an epoch, generation, or fencing token.
Detection is a latency–mistake trade-off #
Let heartbeat observations have inter-arrival times x. A fixed threshold says:
suspect when now - last_seen > TA lower T shortens failover but increases false suspicions. A higher T reduces false positives but lengthens the interval in which traffic still targets an unavailable member. There is no universally correct number because pause distributions, network paths, and remediation costs differ.
An adaptive detector can instead compare current silence with historical observations. The output should be suspicion—not an invented certainty—and operators should be able to inspect the evidence behind it.
Separate probing from dissemination #
All-to-all heartbeats make every member send to every other member. Message load grows poorly as membership grows. SWIM’s useful design move is to separate two jobs:
- Probe a small, changing set of peers to detect possible failure.
- Disseminate membership changes through gossip.
SWIM also introduces a suspect phase before failed, allowing indirect probes or contrary evidence to correct an initial observation. This is more than an optimization: it makes uncertainty explicit in the state machine.
alive -> suspect -> failed
^ |
+--------+ refutation with newer incarnationProduction policy #
For each detector, document:
- the observation source and its blind spots;
- expected and worst-case detection time;
- false-positive budget under load and deployment pauses;
- whether suspicion removes traffic or transfers authority;
- how a returning member proves its incarnation;
- which metric explains every transition.
Test the detector during CPU starvation, packet loss, asymmetric reachability, long garbage-collection pauses, and rolling deploys. A test that only kills a process validates the easiest case.
Trade-offs #
Fast detection improves recovery time but makes healthy nodes easier to condemn. Indirect probes and suspicion stages reduce false positives at the cost of more protocol state. Central detectors simplify policy but introduce shared bottlenecks and correlated blind spots. Distributed detectors scale better, while different observers can temporarily disagree.
The CTO-level decision is not “use a five-second heartbeat.” It is deciding how much uncertainty the system can tolerate before taking an action, and ensuring that the action remains safe when the detector is wrong.