A Raft Write Is Not Committed Just Because Most Replicas Have It

Raft is often summarized as: send a write to the leader, copy it to a majority, and call it committed. That sentence is useful for orientation and dangerous for implementation.

A real system must answer harder questions. What if the leader dies after replication but before replying? Can a new leader overwrite the entry? Why can a leader safely commit an older entry only after committing something from its own term? Can the leader serve a read without writing to the log? Which bytes must reach stable storage before a message leaves the process?

This article follows one command through those boundaries.

The central invariant is:

Once an entry is committed, every future leader contains that entry, and the state machine applies committed entries in log order.

Everything in Raft—terms, voting restrictions, log matching, quorum intersection, and the commit rule—exists to preserve that sentence.

The replicated state machine underneath #

Raft does not directly replicate a database. It replicates an ordered log of deterministic commands. Each node applies the same committed commands in the same order to its local state machine.

client command
      │
      ▼
┌─────────────┐     AppendEntries      ┌─────────────┐
│   leader    │ ─────────────────────► │  follower   │
│ log + state │                        │ log + state │
└─────────────┘ ◄───────────────────── └─────────────┘
      │              acknowledgement
      ▼
advance commit index → apply command → return result

Each log entry contains an index, a term, and a command:

index:   1    2    3    4    5
term:    1    1    2    2    4
command: A    B    C    D    E

The index establishes position. The term identifies the leadership epoch in which the entry was created. The pair (index, term) is the entry’s identity for consistency checks.

Terms are logical epochs, not wall-clock time #

Every server stores a monotonically increasing currentTerm. A follower that stops hearing from a leader waits for a randomized election timeout, increments its term, becomes a candidate, votes for itself, and requests votes.

A term can have at most one elected leader because winning requires a majority and a server grants at most one vote per term. Two different majorities in the same cluster must intersect at at least one voter.

The candidate does not win merely by having the largest term. A voter also checks whether the candidate’s log is at least as up to date as its own. Raft compares the term of the final log entry first, then its index. This voting restriction is what links leader election to committed history.

If any server receives a valid message with a higher term, it updates its term and steps down to follower. A leader is therefore not a permanent role. It is a claim scoped to one logical epoch.

AppendEntries is both replication and consistency repair #

The leader sends AppendEntries containing:

• its current term;
• the index and term immediately before the new entries;
• zero or more log entries;
• the leader’s commit index.

The follower accepts new entries only if it has the preceding (index, term) pair. If that check fails, the leader moves backward to find the last shared prefix. Conflicting suffix entries are removed and replaced by the leader’s suffix.

This produces the log-matching property:

If two logs contain an entry with the same index and term, they contain identical entries through that index.

Heartbeats are simply AppendEntries calls with no new log entries. They assert leadership, carry the commit index, and continue consistency checks.

Majority replication is necessary, not always sufficient #

For an entry created in the leader’s current term, the leader can advance commitIndex when that entry is stored on a majority. Because election majorities intersect replication majorities, some voter in every future election has that entry. The voting up-to-date rule prevents a candidate with an older log from replacing it.

The subtle case is an entry from an earlier term.

Imagine five nodes. In term 2, leader A replicates entry x to A and B, then crashes. In term 3, another leader can be elected and create a different suffix on C, D, and E. After another election, x may appear on a majority of the currently visible logs through a combination of old copies and partial repair, yet a future leader whose last-log term is newer can still be elected without x.

The Raft paper’s Figure 8 demonstrates this counterintuitive history. Counting copies of an old-term entry does not prove it is committed.

Raft’s rule is stricter:

commit N when:
  a majority has matchIndex >= N
  AND log[N].term == currentTerm

Once the leader commits one entry from its own term, every preceding entry in its log becomes committed indirectly. The current-term entry anchors the entire prefix to Leader Completeness: any future leader must contain it, and because of log matching, must also contain everything before it.

Many implementations append a no-op entry when a leader is elected. Committing that no-op establishes the current-term anchor even when there is no immediate client write.

Commitment and application are different positions #

Three indexes matter:

last log index ≥ commit index ≥ applied index

• last log index: the newest entry stored locally;
• commit index: the highest entry known to be committed;
• applied index: the highest entry already executed by the state machine.

A follower can possess an uncommitted entry. A node can know an entry is committed but not yet have applied it. Returning a read from state at appliedIndex = 90 after proving commitIndex = 95 is still stale until the application catches up through 95.

This distinction becomes critical for snapshots. A snapshot may compact the log only through state that has actually been applied. Its metadata must preserve the last included index and term so future consistency checks can bridge the compacted prefix.

Why a leader cannot simply read local memory #

A node may believe it is leader after it has been partitioned from the majority. Meanwhile, the majority can elect a new leader and commit newer writes. If the old leader serves a local read, the result violates linearizability even though its state machine is internally consistent.

A safe linearizable read needs two proofs:

1. this node is still the leader for the relevant term;
2. its state machine has applied every write committed before the read began.

One simple method is to place every read through the replicated log. That is safe but expensive. Raft’s ReadIndex approach confirms leadership with a quorum without appending a new log entry, obtains a safe read index, then waits until the local applied index reaches it.

etcd exposes the trade-off explicitly. Linearizable reads coordinate through consensus. Its “serializable” read mode can be served by one member for lower latency and higher throughput, but may be stale relative to the quorum. In this API, “serializable” does not mean the same thing as SQL serializable isolation; names must be interpreted from their documented contract.

Lease-based reads avoid a quorum exchange by relying on a leader lease and bounded clock behavior. The etcd Raft implementation warns that unbounded clock drift can make this unsafe. A lease is a timing assumption added to the protocol, not free consensus.

Persistence must happen before dependent messages #

Consensus safety can be broken by the order of disk and network operations even when the state machine logic is correct.

Suppose a follower grants a vote in term 7, sends the response, crashes before persisting currentTerm and votedFor, then restarts and grants another vote in term 7. The one-vote-per-term property has been violated.

Likewise, acknowledging an appended entry before stable persistence allows the leader to count a copy that disappears after restart.

The etcd Raft library deliberately leaves storage and transport to its caller, but specifies the ordering: persist entries, hard state, and snapshot appropriately before sending messages that depend on them. Deterministic protocol code does not remove the integration’s durability responsibility.

The production boundary is:

state transition
  → durable write of required term/vote/log state
  → network acknowledgement

Batching and parallel disk writes may optimize this path, but they must preserve the happens-before relationship required by the protocol.

A successful client response is later than commitment #

Consider one write:

client → leader: put(order-42, paid)
leader → quorum: replicate
quorum → leader: persisted
leader: commit and apply
leader → client: success

The leader may crash after commit but before the response reaches the client. The client sees a timeout, yet the command is durable and may already be visible.

Consensus decides the log. It does not make the client transport exactly once.

Clients need a stable command identifier. The state machine should store the outcome associated with that identifier and return the same result when a retry reaches the current leader:

if command_id already applied:
    return recorded_result
else:
    apply command
    record command_id → result

Deduplication state is part of the replicated state machine and needs a retention policy aligned with the maximum retry horizon. Expire it too early and an old retry can create a second business effect.

What a partition actually does #

In a five-voter cluster, the majority side with three connected voters can elect a leader and continue. The minority side cannot commit new entries. An isolated old leader may accept client requests into its local log, but cannot commit them without a quorum; those entries may later be overwritten.

This is the availability boundary of quorum consensus. A five-node cluster does not “survive any two failures” if the remaining three cannot communicate with one another. Failure count is shorthand; quorum connectivity and latency are the real conditions.

Adding voters can reduce availability if it moves the quorum across unreliable or high-latency links. Learners and non-voting replicas may improve read locality, backup, or replacement workflows without changing the commit quorum, but their semantics must be explicit.

The operational metrics that explain Raft #

CPU and request rate will not tell you why consensus is slow. Track the protocol positions and the costs between them:

• leader changes and election duration;
• current term and leader identity;
• per-follower match-index lag;
• commit index minus applied index;
• proposal-to-commit and commit-to-apply latency;
• fsync latency and batching size;
• rejected proposals while leaderless;
• snapshot creation, transfer, and application time;
• ReadIndex latency and quorum failures;
• uncommitted log growth during quorum loss.

Alert on sustained gaps, not momentary differences. A brief apply lag during a batch may be normal; an increasing commit-to-apply gap means the state machine cannot keep up with consensus.

Common implementation mistakes #

The CTO decision #

Do not choose Raft because “we need high availability.” Choose a replicated system whose quorum geography, durability path, read semantics, recovery procedures, and operational ownership fit the business invariant.

For most teams, implementing consensus is the wrong product investment. Operating an established implementation still requires understanding its contract. You need to know whether a read is linearizable, when a write response can be ambiguous, what storage acknowledgements mean, how membership changes are performed, and what happens when the quorum spans regions.

The memorable rule is not “majority means committed.” It is this:

A current-term quorum commits an ordered prefix that every future leader must preserve; safe reads and client effects require additional proofs around that prefix.

That is the mechanism underneath the abstraction.

References #

• Ongaro and Ousterhout: In Search of an Understandable Consensus Algorithm
• etcd-io/raft: implementation contract and features
• etcd API guarantees
• etcd-io/raft read-safety configuration
• Papershelf: Raft and other systems papers
• Related: External consistency has a latency budget
• Related: Durable workflows do not remove idempotency