Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

Distributed Consensus Raft vs Paxos: etcd vs Spanner Deep Dive

By: Md Tarek · 18 Oct 2026

Distributed Consensus Raft vs Paxos: quorum-based leader election and log replication across a five-node cluster
Distributed Consensus Raft vs Paxos: quorum-based leader election and log replication across a five-node cluster

Executive Summary & Key Takeaways

  • Raft is a Paxos-family protocol with an engineer-friendly decomposition. It splits consensus into leader election, log replication, and safety — the reason etcd, Consul, CockroachDB, TiKV, and Kafka KRaft ship Raft instead of classic Paxos.
  • Durability dominates latency, not the protocol. A 5-node etcd cluster with NVMe group-commit sustains 10k–40k writes/sec at p99 ≈ 4–12 ms; the same cluster on EBS gp3 collapses to 1.5k–3k writes/sec at p99 ≈ 80–200 ms, because every commit pays a majority fsync.
  • Spanner runs tens of thousands of Paxos groups, not one. Each group owns a disjoint tablet range; Multi-Paxos with a 10-second leader lease removes the prepare phase from the steady state, leaving one Accept round trip plus commit-wait (2ε).
  • Split-brain is prevented arithmetically. Any two quorums of size ⌊n/2⌋+1 intersect in at least one node, and that node refuses a second vote in the same term — two leaders in one term can never both commit.
  • TrueTime is Spanner's structural advantage. Bounding clock uncertainty to ε < 7 ms (often < 1 ms with dense time masters) replaces a read quorum with commit-wait, enabling externally consistent lock-free snapshot reads at any timestamp.

Raft in etcd: Leader Election, Log Replication, and the Figure 8 Trap

The safety proof is one inequality

Raft replicates a deterministic state machine. Every node durably stores (currentTerm, votedFor, log[]). Safety rests entirely on quorum intersection: for two quorums Q1 and Q2 drawn from a cluster of n nodes,

|Q1| + |Q2| >= (n/2 + 1) + (n/2 + 1) > n

Because the two sets together exceed the cluster size, they must share at least one node. That node accepted the first quorum's entry, and it will refuse the second quorum's conflicting entry because its log already contains a higher term or index at that slot. Split-brain is therefore a *contradiction*, not a timing race.

| Cluster size n | Quorum ⌊n/2⌋+1 | Failures tolerated | Production reality | | --- | --- | --- | --- | | 1 | 1 | 0 | Local dev only. Zero HA. | | 2 | 2 | 0 | Strictly worse than 1 node: double cost, no availability gain. | | 3 | 2 | 1 | Minimum viable production. Survives one AZ loss. | | 4 | 3 | 1 | Same tolerance as 3, higher quorum latency. Never deploy this. | | 5 | 3 | 2 | The standard. Survives one AZ loss plus one rolling restart. | | 7 | 4 | 3 | Cross-region. p99 is bounded by the slowest quorum member. |

Election: randomized timeouts beat determinism

A follower that hears nothing for election-timeout increments its term, becomes a candidate, votes for itself, and sends RequestVote RPCs. etcd's defaults are --heartbeat-interval=100 ms and --election-timeout=1000 ms — a 10× ratio. The timeout is *randomized* across [T, 2T) to make simultaneous candidacies statistically rare: with 5 nodes the collision probability per round is roughly 1/5, so an election converges in ~1.25 rounds on average.

Log replication and the commit rule

The leader appends a LogEntry{Index, Term, Data} locally and ships AppendEntries{Term, PrevLogIndex, PrevLogTerm, Entries, LeaderCommit}. Followers enforce the Log Matching Property: if two logs contain an entry with the same index and term, the logs are identical up to that index. A follower rejects on PrevLogTerm mismatch, the leader decrements nextIndex, and retries until the logs converge.

The subtle rule — and the one that bites implementers — is the commit condition. A leader may advance commitIndex to N only if:

1. a majority of matchIndex[i] >= N, and 2. log[N].term == currentTerm.

Condition 2 is the Figure 8 fix. In the Raft paper's Figure 8, a leader replicates an entry from term 2 to a majority but crashes before committing it; a new leader from term 4 with a longer log can be elected, and if the term-2 leader had committed by replica count alone, that entry would be silently overwritten. Requiring the current term means old entries are only committed *indirectly*, when a current-term entry above them commits.

                 term = 7, n = 5, quorum = floor(5/2)+1 = 3

   +----------+  AppendEntries(idx=43, term=7, prevIdx=42, prevTerm=7)
   |  Leader  |----------+--------------+--------------+-------------+
   |  term 7  |          |              |              |             |
   +----+-----+          v              v              v             v
        |           +--------+    +--------+    +--------+    +--------+
        |           | F1 (7) |    | F2 (7) |    | F3 (7) |    | F4 (7) |
        |           +---+----+    +---+----+    +---+----+    +---+----+
        |               | ack         | ack         | ack         | partitioned
        |<--------------+             |             |             X
        |<----------------------------+             |
        |  acks = 1 (self) + 2 followers = 3 >= quorum
        v
   commitIndex = 43  ->  apply to state machine  ->  reply to client

War story: an election storm caused by a WAL device

A 5-node etcd cluster in a Kubernetes control plane began logging etcdserver: request timed out every ~2 seconds. The metric that explained it was wal_fsync_duration_seconds: p99 had crept from 3 ms to 41 ms because the WAL shared an EBS volume with the container image puller. With a 1000 ms election timeout and a 100 ms heartbeat, a follower that misses a few heartbeats during an fsync stall starts an election; the incumbent leader, also stalled, loses quorum, and the cycle repeats.

The fix was not to raise the timeout (that only masks it — we did bump it to 5000 ms as a stopgap). The fix was to move the WAL to a dedicated local NVMe device via --wal-dir, which brought fsync p99 back under 1 ms and reduced etcd_server_leader_changes_seen_total from ~30/day to 0.

Rule of thumb: election-timeout must exceed max(RTT) + p99(fsync) + scheduling jitter by at least 3×. If your fsync p99 is above ~30 ms, no timeout value will save you.

War story: the stale read that was not a consensus bug

A team reported that a value written to etcd was immediately invisible to a different pod. Raft was working perfectly. The client was using clientv3.WithSerializable(), which serves reads from the local node's state machine without a ReadIndex round trip — a legitimate optimization for caches, but not for read-your-writes. etcd's default (linearizable) read path issues a ReadIndex: the leader confirms quorum leadership, then waits for its applied index to reach the read index before answering. Serializable reads skip both steps and can return data that is arbitrarily stale.

Multi-Paxos in Spanner: TrueTime, Per-Shard Groups, and Commit-Wait

The hierarchy is the scaling trick

Raft's weakness at Google scale is not correctness, it is topology: one Raft group is one log, one leader, and one region-bound write path. Spanner sidesteps this by making consensus *per shard* rather than per database.

Universe (global)
  |
  +-- Zone (a failure domain / region, e.g. us-central1)
        |
        +-- Spanserver (Borg-allocated, ~2-4 vCPU, thousands of tablets)
              |
              +-- Tablet (100-1000 per spanserver)
                    |
                    +-- Paxos group (one replica per zone, 3-5 replicas)
                          |
                          +-- Directory = bucket of contiguous keys
                              (the unit of placement and data movement)

A 10,000-node Spanner universe runs tens of thousands of independent Paxos groups concurrently. Each group commits independently, so aggregate write throughput scales roughly linearly with the number of groups, not with the speed of any single leader.

Multi-Paxos: pay for prepare once per leader epoch

Classic Paxos requires two round trips per value: Prepare (promise) then Accept. Multi-Paxos elects a stable leader so the prepare phase happens once per leadership epoch. Ballot numbers are encoded as a monotonic pair — roughly (epoch << 32) | leader_id — so a new leader always has a strictly greater ballot and every acceptor can reject stale ballots with a single integer comparison.

Steady state is therefore one round trip: Accept(ballot, value) to the other replicas, a quorum of acks, commit. This is structurally the same as Raft's AppendEntries path; the difference is that Paxos leaves leader election and log management *out* of the protocol, which is exactly why Raft was invented.

Leader leases and why they need a clock

Spanner's Paxos leaders hold a lease, historically 10 seconds. A leader may serve reads without a quorum while the lease is valid, which is what makes single-region reads cheap. The lease is safe only because Spanner can *prove* the lease has not expired using TrueTime:

TT.now()      -> {earliest, latest}      where latest - earliest = 2*epsilon
TT.after(t)   -> true iff t has definitely passed (t < TT.now().earliest)
TT.before(t)  -> true iff t has definitely not arrived

ε is typically under 7 ms and, with dense time masters, often under 1 ms. A leader's lease expiry is computed from the *earliest* possible time of the last heartbeat ack, never from a local wall clock read.

Commit-wait: trading 2ε of latency for external consistency

When a transaction commits, the coordinator picks a commit timestamp s = TT.now().latest and then sleeps until `TT.after(s)` before releasing locks and responding. This is commit-wait, and it costs roughly 2ε (≈ 2–14 ms) per commit.

What it buys is enormous: any transaction that starts after s has definitely passed will receive a timestamp greater than s. That makes Spanner externally consistent and allows lock-free snapshot reads at any timestamp without a read quorum. Raft has no equivalent primitive — which is why etcd must pay a ReadIndex round trip for every linearizable read.

Cross-group transactions: 2PC over Paxos

A transaction spanning multiple Paxos groups runs two-phase commit where the coordinator is the leader of one participant group:

  Client
    |  1. begin, read/write rows across dir A (group GA) and dir B (group GB)
    v
  GA leader (coordinator)
    |-- 2. prepare --> GB leader (writes a PREPARE record via its own Paxos group)
    |<-- 3. prepared --
    |-- 4. commit record written INTO GA's Paxos log (replicated, durable decision)
    |-- 5. commit --> GB leader
    v
  Client receives commit timestamp s (after commit-wait)

Because the commit decision itself lives in a Paxos log, coordinator process failure is survivable: a new GA leader recovers the decision and drives the participants to completion. This is the key difference from naive 2PC, where a coordinator crash blocks participants indefinitely.

War story: when ε doubled, write p99 doubled with it

During a time-master maintenance window, Spanner's measured ε in one region drifted from ~3 ms to ~9 ms. Commit-wait is a direct function of ε, so the write p99 in that region rose by roughly 12 ms — a change large enough to trip an SLO that had nothing to do with consensus. The lesson generalizes: any consensus protocol that derives safety from bounded clock error inherits that clock's failure modes as latency. Google smears leap seconds across a 20-hour window precisely so that no single instant produces a discontinuous clock jump.

Split-Brain Mitigation: Fencing Tokens, Pre-Vote, and CheckQuorum

   Partition A (majority, 3 nodes)          Partition B (minority, 2 nodes)
   +---------------------------+   XXXXX   +-------------------------------+
   |  L(term 7)  F1   F2       |   XXXXX   |  F3 (was leader, term 6)      |
   |  quorum = 3 -> commits OK |   XXXXX   |  quorum = 3 -> CANNOT commit  |
   |  elects L'(term 8) if L   |   XXXXX   |  steps down after             |
   |  fails: 2 of 3 still form |   XXXXX   |  election-timeout; never      |
   |  a quorum                 |   XXXXX   |  serves a write again         |
   +---------------------------+   XXXXX   +-------------------------------+

1. Term monotonicity is a fencing token — use it downstream

Raft's term and Paxos's ballot are monotonically increasing fencing tokens. A stale leader can still *send* messages after it is deposed; what stops it is that every other node rejects any request with a term below its own. If your consensus layer writes to an external resource (object storage, a lock service, a database), you must pass the term through and have that resource reject tokens lower than the highest it has seen. Otherwise you have built a distributed lock with no fence — the classic bug where a paused-then-resumed leader corrupts a shared file.

2. Pre-Vote stops a partitioned node from disrupting a healthy leader

Without Pre-Vote, a node that is partitioned, restarted, or paused increments its term on every election timeout. When the partition heals, its inflated term forces the healthy leader to step down and triggers a real election — a self-inflicted outage. Pre-Vote (Raft thesis §9.6, enabled in etcd with PreVote: true) makes the candidate first ask whether it *would* win, without incrementing its term. If a majority says no, nothing changes.

3. CheckQuorum makes a leader abdicate instead of lying

A leader that cannot reach a majority must not keep answering reads. With CheckQuorum: true, a leader that fails to receive responses from a quorum within one election timeout transitions to follower. This closes the window where a partitioned leader serves reads from its own state machine while a new leader is already committing writes elsewhere.

4. Lease reads are a clock bet — most systems should not take it

Serving reads from a leader lease is safe only if lease_duration > max_clock_drift + max_RTT. If your NTP-synced clocks can drift 250 ms during a VM migration, a 1-second lease is a 4× safety margin — thin. etcd's default linearizable read uses ReadIndex rather than lease reads for exactly this reason. If you enable lease reads for latency, you must run a clock discipline layer with a hard bound, as Spanner does with TrueTime.

5. Membership changes are where production clusters actually break

Joint consensus (C_old,new) is the textbook answer, but etcd 3.4+ deliberately moved away from it to one voter at a time plus non-voting learners. The operational rules:

  • Never change more than one voter per operation.
  • Add the node as a learner first, let it catch up, then promote it.
  • Never remove enough voters to make the remaining set smaller than a quorum in flight.
  • Snapshot before large membership changes so the new node does not need the full log.

6. Witnesses, and why even-sized clusters are a trap

A 4-node cluster tolerates 1 failure but requires 3 acks — strictly worse than 3 nodes. If you need cross-AZ redundancy without 5 full replicas, use 2 voters + 1 witness (a node that votes but holds no data). It tolerates 1 failure at lower storage cost, at the price of the witness being unable to become leader.

Practical Implementation

Go: a linearizable replicated KV on etcd Raft

The following is the full production loop: WAL-before-RPC persistence, Ready handling, snapshot compaction, ReadIndex linearizable reads, and idempotent apply.

// Package raftkv is a linearizable key-value store built on the etcd Raft
// library (go.etcd.io/raft/v3). It implements the complete production loop:
// WAL-before-RPC persistence, Ready handling, snapshot compaction, ReadIndex
// linearizable reads, and idempotent apply.
package raftkv

import (
	"context"
	"encoding/binary"
	"encoding/json"
	"errors"
	"fmt"
	"log/slog"
	"os"
	"sync"
	"time"

	"go.etcd.io/raft/v3"
	"go.etcd.io/raft/v3/raftpb"
)

var ErrTimeout = errors.New(`raftkv: proposal did not commit before deadline`)

// WAL must fsync HardState (term, vote, commit) before any RPC leaves the
// process. A node that forgets its vote across a crash can vote twice in the
// same term, which breaks the quorum-intersection safety proof.
type WAL interface {
	Save(hs raftpb.HardState, entries []raftpb.Entry, snap raftpb.Snapshot) error
	SaveSnapshot(snap raftpb.Snapshot) error
	Replay() (raftpb.HardState, []raftpb.Entry, raftpb.Snapshot, error)
}

// Transport ships Raft messages to peers. It MUST be asynchronous and bounded:
// a blocking transport turns one slow follower into cluster-wide backpressure.
type Transport interface {
	Send(msgs []raftpb.Message)
}

// StateMachine is the actual data store (bbolt, Badger, RocksDB, in-memory).
type StateMachine interface {
	Apply(cmd Command) error
	Seen(reqID uint64) bool
	Get(key string) string
	AppliedIndex() uint64
	WaitApplied(index uint64)
	Snapshot() []byte
	Restore(data []byte)
}

// Command is the replicated unit of work. ReqID makes apply idempotent: Raft
// delivers entries to the state machine at least once, so the state machine
// must dedupe to obtain exactly-once semantics.
type Command struct {
	Op    string `json:"op"` // put | delete
	Key   string `json:"key"`
	Value string `json:"value,omitempty"`
	ReqID uint64 `json:"req_id"`
}

type Config struct {
	NodeID         uint64
	Peers          []raft.Peer
	TickInterval   time.Duration // 100ms => 10 ticks/s
	ElectionTicks  int           // 10  => 1000ms election timeout
	HeartbeatTicks int           // 1   => 100ms heartbeat
	SnapshotEvery  uint64        // compact the log every N applied entries
	MaxSizePerMsg  uint64
}

type pending struct{ done chan error }

type readReq struct {
	key  string
	rctx []byte
	ch   chan readResult
}

type readResult struct {
	value string
	err   error
}

type Store struct {
	cfg       Config
	node      raft.Node
	storage   *raft.MemoryStorage
	wal       WAL
	transport Transport
	sm        StateMachine

	readC chan readReq
	stopc chan struct{}
	done  chan struct{}

	mu        sync.Mutex
	pending   map[uint64]*pending
	readWait  map[uint64]readReq
	confState raftpb.ConfState
	reqSeq    uint64
	lastSnap  uint64
}

func NewStore(cfg Config, wal WAL, transport Transport, sm StateMachine) (*Store, error) {
	hs, entries, snap, err := wal.Replay()
	if err != nil {
		return nil, fmt.Errorf(`replay wal: %w`, err)
	}

	storage := raft.NewMemoryStorage()
	var confState raftpb.ConfState
	if !raft.IsEmptySnap(snap) {
		if err := storage.ApplySnapshot(snap); err != nil {
			return nil, fmt.Errorf(`apply snapshot to storage: %w`, err)
		}
		sm.Restore(snap.Data) // rebuild the state machine at the snapshot index
		if snap.Metadata.ConfState != nil {
			confState = *snap.Metadata.ConfState
		}
	}
	if err := storage.SetHardState(hs); err != nil {
		return nil, fmt.Errorf(`set hard state: %w`, err)
	}
	if err := storage.Append(entries); err != nil {
		return nil, fmt.Errorf(`append wal entries: %w`, err)
	}

	s := &Store{
		cfg:       cfg,
		storage:   storage,
		wal:       wal,
		transport: transport,
		sm:        sm,
		readC:     make(chan readReq, 128),
		stopc:     make(chan struct{}),
		done:      make(chan struct{}),
		pending:   make(map[uint64]*pending),
		readWait:  make(map[uint64]readReq),
		confState: confState,
	}

	s.node = raft.StartNode(&raft.Config{
		ID:                        cfg.NodeID,
		ElectionTick:              cfg.ElectionTicks,
		HeartbeatTick:             cfg.HeartbeatTicks,
		Storage:                   storage,
		MaxSizePerMsg:             cfg.MaxSizePerMsg,
		MaxInflightMsgs:           256,
		MaxUncommittedEntriesSize: 1 << 20,
		// PreVote stops a partitioned node from bumping its term and
		// disrupting a healthy leader. CheckQuorum makes a leader that cannot
		// reach a majority step down rather than serve stale reads.
		PreVote:     true,
		CheckQuorum: true,
	}, cfg.Peers)

	go s.run()
	return s, nil
}

func (s *Store) run() {
	ticker := time.NewTicker(s.cfg.TickInterval)
	defer ticker.Stop()

	for {
		select {
		case <-ticker.C:
			s.node.Tick()

		case rr := <-s.readC:
			// Register the waiter BEFORE issuing ReadIndex, otherwise the
			// ReadState can arrive before we know who to wake.
			s.mu.Lock()
			s.readWait[binary.BigEndian.Uint64(rr.rctx)] = rr
			s.mu.Unlock()
			s.node.ReadIndex(context.Background(), rr.rctx)

		case rd := <-s.node.Ready():
			// STEP 1 - persist before anything leaves the process.
			if err := s.wal.Save(rd.HardState, rd.Entries, rd.Snapshot); err != nil {
				slog.Error(`wal save failed; halting to avoid unsafe state`, `err`, err)
				os.Exit(1) // never continue with un-fsynced Raft state
			}

			// STEP 2 - ship messages. rd.MustSync is false when the library
			// knows the fsync is redundant (group commit), which is where most
			// of the NVMe throughput advantage comes from.
			s.transport.Send(rd.Messages)

			// STEP 3 - apply committed entries to the state machine.
			if !raft.IsEmptySnap(rd.Snapshot) {
				s.sm.Restore(rd.Snapshot.Data)
				if rd.Snapshot.Metadata.ConfState != nil {
					s.mu.Lock()
					s.confState = *rd.Snapshot.Metadata.ConfState
					s.mu.Unlock()
				}
			}
			for _, e := range rd.CommittedEntries {
				s.apply(e)
			}

			// STEP 4 - resolve linearizable reads whose read index is now
			// satisfiable. Never block the Ready loop doing this.
			for _, rs := range rd.ReadStates {
				s.resolveRead(rs)
			}

			// STEP 5 - advance in-memory state, then maybe compact.
			s.storage.Advance(rd)
			s.maybeCompact()
			s.node.Advance()

		case <-s.stopc:
			s.node.Stop()
			close(s.done)
			return
		}
	}
}

func (s *Store) apply(e raftpb.Entry) {
	switch e.Type {
	case raftpb.EntryNormal:
		if len(e.Data) == 0 {
			return // empty entry a fresh leader commits to fence old terms
		}
		var cmd Command
		if err := json.Unmarshal(e.Data, &cmd); err != nil {
			slog.Error(`corrupt log entry`, `index`, e.Index, `err`, err)
			return
		}
		if s.sm.Seen(cmd.ReqID) {
			s.notify(cmd.ReqID, nil) // duplicate after a snapshot restore
			return
		}
		if err := s.sm.Apply(cmd); err != nil {
			slog.Error(`state machine apply failed`, `index`, e.Index, `err`, err)
			s.notify(cmd.ReqID, err)
			return
		}
		s.notify(cmd.ReqID, nil)

	case raftpb.EntryConfChange:
		var cc raftpb.ConfChange
		if err := cc.Unmarshal(e.Data); err != nil {
			slog.Error(`corrupt conf change`, `index`, e.Index, `err`, err)
			return
		}
		cs := s.node.ApplyConfChange(cc)
		if cs != nil {
			s.mu.Lock()
			s.confState = *cs
			s.mu.Unlock()
		}
	}
}

func (s *Store) notify(reqID uint64, err error) {
	s.mu.Lock()
	p := s.pending[reqID]
	delete(s.pending, reqID)
	s.mu.Unlock()
	if p != nil {
		p.done <- err
		close(p.done)
	}
}

// Put is linearizable: it returns only after a quorum has committed the entry
// AND the local state machine has applied it.
func (s *Store) Put(ctx context.Context, key, value string) error {
	s.mu.Lock()
	s.reqSeq++
	reqID := s.reqSeq
	s.mu.Unlock()

	data, err := json.Marshal(Command{Op: `put`, Key: key, Value: value, ReqID: reqID})
	if err != nil {
		return fmt.Errorf(`marshal command: %w`, err)
	}

	p := &pending{done: make(chan error, 1)}
	s.mu.Lock()
	s.pending[reqID] = p
	s.mu.Unlock()

	if err := s.node.Propose(ctx, data); err != nil {
		s.mu.Lock()
		delete(s.pending, reqID)
		s.mu.Unlock()
		return fmt.Errorf(`raft propose: %w`, err)
	}

	select {
	case err := <-p.done:
		return err
	case <-ctx.Done():
		// The write may still commit. Callers must treat a timeout as an
		// UNKNOWN outcome and retry with the SAME ReqID, never as a failure.
		return fmt.Errorf(`%w: %v`, ErrTimeout, ctx.Err())
	}
}

// LinearizableGet issues a ReadIndex request. The leader first confirms with a
// quorum that it is still leader, then waits for its applied index to reach the
// read index. A stale leader never serves the read; the request times out.
func (s *Store) LinearizableGet(ctx context.Context, key string) (string, error) {
	s.mu.Lock()
	s.reqSeq++
	reqID := s.reqSeq
	s.mu.Unlock()

	rctx := make([]byte, 8)
	binary.BigEndian.PutUint64(rctx, reqID)
	ch := make(chan readResult, 1)

	select {
	case s.readC <- readReq{key: key, rctx: rctx, ch: ch}:
	case <-ctx.Done():
		return ``, ctx.Err()
	}

	select {
	case res := <-ch:
		return res.value, res.err
	case <-ctx.Done():
		return ``, fmt.Errorf(`readindex timed out: %w`, ctx.Err())
	}
}

func (s *Store) resolveRead(rs raft.ReadState) {
	id := binary.BigEndian.Uint64(rs.RequestCtx)
	s.mu.Lock()
	rr, ok := s.readWait[id]
	delete(s.readWait, id)
	s.mu.Unlock()
	if !ok {
		return
	}
	// Run off the Ready loop: WaitApplied can block on state-machine catch-up,
	// and blocking Ready stalls heartbeats, which causes elections.
	go func() {
		s.sm.WaitApplied(rs.Index)
		rr.ch <- readResult{value: s.sm.Get(rr.key)}
	}()
}

func (s *Store) maybeCompact() {
	applied := s.sm.AppliedIndex()
	if applied < s.lastSnap+s.cfg.SnapshotEvery {
		return
	}
	s.mu.Lock()
	cs := s.confState
	s.mu.Unlock()

	snap, err := s.storage.CreateSnapshot(applied, &cs, s.sm.Snapshot())
	if err != nil {
		slog.Error(`create snapshot`, `index`, applied, `err`, err)
		return
	}
	if err := s.wal.SaveSnapshot(snap); err != nil {
		slog.Error(`persist snapshot`, `index`, applied, `err`, err)
		return
	}
	if err := s.storage.Compact(applied); err != nil {
		slog.Error(`compact log`, `index`, applied, `err`, err)
		return
	}
	s.lastSnap = applied
}

func (s *Store) Close() {
	close(s.stopc)
	<-s.done
}

Python: the Paxos half, for contrast

This is a durable Multi-Paxos acceptor. Note that the *only* structural difference from the Raft code above is that there is no leader election and no log — the acceptor is stateless with respect to ordering, which is exactly the complexity Raft absorbed.

import json
import os
from dataclasses import asdict, dataclass


@dataclass(frozen=True)
class Promise:
    ballot: int           # highest ballot this acceptor has promised
    accepted_ballot: int  # ballot of the last accepted proposal (0 if none)
    accepted_value: str | None


class Acceptor:
    '''Durable Multi-Paxos acceptor.

    Invariants enforced here:
      P1: never accept a proposal numbered lower than the highest ballot
          already promised.
      P2: persist (ballot, accepted_ballot, accepted_value) BEFORE replying,
          so a crash cannot forget a promise and resurrect an older one.
    '''

    def __init__(self, node_id: int, state_path: str):
        self.node_id = node_id
        self._fd = os.open(state_path, os.O_RDWR | os.O_CREAT, 0o600)
        self.promise = self._load()

    # ---------- durable state ----------
    def _load(self) -> Promise:
        raw = os.pread(self._fd, 4096, 0)
        if not raw:
            return Promise(ballot=0, accepted_ballot=0, accepted_value=None)
        return Promise(**json.loads(raw.decode('utf-8')))

    def _persist(self, p: Promise) -> None:
        blob = json.dumps(asdict(p)).encode('utf-8')
        os.pwrite(self._fd, blob, 0)
        os.ftruncate(self._fd, len(blob))
        os.fsync(self._fd)  # <-- the entire cost of consensus lives on this line

    # ---------- phase 1: prepare (once per leadership epoch) ----------
    def prepare(self, ballot: int) -> tuple[int, int, str | None] | None:
        if ballot <= self.promise.ballot:
            return None  # reject stale ballot; a newer leader already exists
        self.promise = Promise(ballot, self.promise.accepted_ballot,
                               self.promise.accepted_value)
        self._persist(self.promise)
        return (ballot, self.promise.accepted_ballot, self.promise.accepted_value)

    # ---------- phase 2: accept (per write, steady state) ----------
    def accept(self, ballot: int, value: str) -> bool:
        if ballot < self.promise.ballot:
            return False  # P1 guard: a higher ballot has already been promised
        self.promise = Promise(ballot, ballot, value)
        self._persist(self.promise)
        return True

Two operational notes on the Python acceptor. First, os.pread/os.pwrite are POSIX-only; on Windows use a single-writer append-only log. Second, fsync per accept caps a single acceptor at roughly 200–2000 ops/sec on a typical cloud SSD — production deployments amortize it with group commit, exactly as etcd does.

Performance Benchmarks & Trade-Offs

Numbers below are representative steady-state figures for 1 KB values on a 5-replica configuration with a single-AZ quorum, unless noted. Cross-region figures assume three regions with ~35 ms inter-region RTT.

| Protocol / Deployment | Round trips per commit (steady state) | Throughput (writes/sec) | p99 commit latency (same AZ) | p99 commit latency (cross-region) | Leader failover | Failures tolerated | Clock dependency | | --- | --- | --- | --- | --- | --- | --- | --- | | etcd Raft, 5 voters, NVMe, batch=64 | 1 (+ majority fsync) | 10k–40k | 4–12 ms | 120–260 ms | 150–300 ms | 2 | None (ReadIndex) | | etcd Raft, 5 voters, EBS gp3 | 1 (+ slow majority fsync) | 1.5k–3k | 80–200 ms | 180–400 ms | 1–3 s | 2 | None | | Raft, 3 voters + 2 learners | 1 | 6k–20k | 5–15 ms | 130–280 ms | 200–500 ms | 1 | None | | Raft with lease reads enabled | 1 read RTT saved | — | reads: 0.2–0.8 ms | reads: local | Same as Raft | 2 | Strong (bounded drift) | | Spanner-style Multi-Paxos + 10s lease | 1 (+ commit-wait 2ε) | 15k–50k per group | 3–10 ms + 2ε | 60–160 ms + 2ε | 200–500 ms | 2 | Very strong (TrueTime) | | Classic Paxos, no stable leader | 2 (+ majority fsync) | 3k–8k | 15–40 ms | 200–400 ms | n/a (no leader) | 2 | None | | EPaxos / leaderless | 1–2 (depends on conflict rate) | 20k–60k under low conflict | 6–20 ms | 140–300 ms | n/a | 2 | None |

Reading the table: the protocol column matters far less than the fsync column. Raft vs Multi-Paxos differs by one round trip in the steady state; NVMe vs network-attached storage differs by 20–50× on p99. Optimize durability hardware before optimizing consensus algorithm.

| Property | Raft (etcd) | Multi-Paxos (Spanner) | | --- | --- | --- | | Leader election | In-protocol (randomized timeouts) | Out-of-protocol, ballot numbers | | Log structure | Explicit, contiguous, with index/term | Implicit; log per replica per instance | | Steady-state commits | 1 RTT | 1 RTT | | Reads | ReadIndex (1 RTT) or lease read | Lease read (no RTT) + commit-wait | | Global timestamps | None | TrueTime, externally consistent | | Horizontal write scaling | One group = one bottleneck | Thousands of independent groups | | Implementation complexity | Moderate, spec-complete | High; leader election and recovery unspecified |

Pro-Tips & Common Pitfalls

  • Never run a 2-node cluster. Quorum is 2, so any single failure halts writes. It is strictly worse than 1 node.
  • Never run an even number of voters. 4 nodes require 3 acks and tolerate 1 failure — worse than 3 nodes on every axis.
  • Do not disable fsync for the Raft WAL. --unsafe-no-fsync buys throughput and destroys the safety proof. A crash can produce two leaders in the same term.
  • Do not skip the `MustSync` optimization either. Always persist, but let the library tell you when the fsync is redundant.
  • Never block the `Ready` loop. Snapshotting, state-machine catch-up, and network sends all belong off the main loop, or heartbeats stall and you get spurious elections.
  • Treat a propose timeout as an unknown outcome, not a failure. Retry with the same idempotency key; the write may already be committed.
  • Make apply idempotent. Snapshot restore replays entries; without a dedup table you will double-apply.
  • Implement the Figure 8 rule. Only commit entries from the current term directly. This is the single most common Raft implementation bug.
  • Randomize election timeouts. Fixed timeouts produce permanent split votes under synchronized load.
  • Enable Pre-Vote and CheckQuorum. Without them, a flapping node causes repeated leadership churn.
  • Change one voter at a time. Joint consensus is the theory; single-voter changes plus learners are the production practice.
  • Do not compact the log to the applied index blindly. Snapshot first, persist the snapshot, then compact. Compact-then-snapshot loses data on crash.
  • Bound your message queues. An unbounded peer queue lets one slow follower exhaust leader memory.
  • Alert on `wal_fsync_duration_seconds` p99, not on leader change count. Leader churn is the symptom; disk latency is the disease.
  • Fence external resources with the term. A consensus leader that writes to S3 or a shared file without a fencing token is a split-brain waiting to happen.
  • Do not use serializable reads for read-your-writes. WithSerializable() and equivalent follower reads return stale data by design.
  • Budget commit-wait if you adopt TrueTime-style clocks. Externally consistent reads cost 2ε per commit; if ε doubles, your write p99 doubles with it.

Frequently Asked Questions (FAQ)

Is Raft just Paxos with a different name?

No, though they are in the same family. Both solve the same problem — getting a quorum of nodes to agree on a sequence of values — and both rely on quorum intersection for safety. The difference is scope. Paxos specifies only how a single value is chosen, and leaves leader election, log management, and membership change to the implementer. Raft specifies all of it: leader election with randomized timeouts, a contiguous log with index/term matching, and the Figure 8 commit rule. That is why etcd, Consul, CockroachDB, and TiKV implement Raft — the specification is complete enough that independent implementations behave identically.

How does etcd prevent split-brain during a network partition?

Three mechanisms combine. First, a leader needs an absolute majority (⌊n/2⌋+1) of votes to be elected, and every commit needs a quorum of acks, so a minority partition can never commit — it simply blocks. Second, terms are monotonic: a stale leader's AppendEntries messages carry an old term and are rejected by every node in the majority partition, which has already moved to a higher term. Third, CheckQuorum makes a leader that cannot reach a majority step down within one election timeout rather than continuing to answer requests. PreVote adds a fourth layer by preventing a partitioned node from inflating its term and disrupting the healthy leader when the partition heals.

Why does Google Spanner use Paxos instead of Raft?

Two reasons. First, historical: Spanner was designed in 2009–2012, when Raft (2014) did not yet exist in published form, and Multi-Paxos with a stable leader was the well-understood production protocol inside Google. Second, and more importantly, Spanner's scaling model is thousands of independent consensus groups rather than one big one. Multi-Paxos's separation of leader election from the core agreement protocol lets Spanner pair Paxos with TrueTime-based leases and a global timestamp oracle — the lease logic and the commit-wait logic are layered on top of Paxos rather than baked into it. A Raft group would work technically, but the lease and timestamp machinery would have to be grafted on anyway, and Multi-Paxos's looser structure accommodates that more naturally.