Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

Distributed Consensus: Raft vs Paxos in Production Systems

By: Md Tarek · 18 Oct 2026

Distributed Consensus Raft vs Paxos: quorum replication, leader election and log replication across etcd Raft groups and Spanner Paxos groups
Distributed Consensus Raft vs Paxos: quorum replication, leader election and log replication across etcd Raft groups and Spanner Paxos groups

Executive Summary & Key Takeaways

  • Raft and Multi-Paxos cost the same on the hot path: exactly one round trip to a quorum per write, plus one durable fsync. A 5-node cluster spread over 3 availability zones on local NVMe lands at roughly p50 2–5 ms / p99 12–30 ms. The same topology stretched cross-region is p99 150–350 ms, because you pay the RTT to the *farthest* quorum member, not the nearest.
  • Raft is a leader-strong Multi-Paxos variant, not a different class of algorithm. Its defining constraint is the *Log Matching Property* — no holes in the log — which makes prefix recovery, snapshotting, and safe membership change tractable enough to ship in etcd, Consul, TiKV, and CockroachDB range leases.
  • etcd's write ceiling is fsync, not consensus math. bbolt + WAL fsync on consumer NVMe caps a 3-node etcd at roughly 10k–20k serialized writes/s and 50k–100k serializable reads/s. When etcd is slow, the consensus layer is almost never the culprit — the disk is.
  • Spanner's differentiator is TrueTime, not Paxos. A commit-wait bounded by roughly 2ε (ε ≈ 1–7 ms depending on generation) converts "agreement on a log index" into external consistency: global linearizability *and* serializable transactions with real-time timestamp ordering. Multi-Paxos alone cannot give you that.
  • Split-brain is prevented by quorum intersection plus monotonic epochs (Raft *terms*, Paxos *ballots*), and enforced downstream with fencing tokens. Pre-Vote and joint-consensus membership changes are the two extensions that prevent the majority of real-world consensus outages.

---

1. The Consensus Contract: Quorums, Epochs, and Split-Brain Prevention

The three properties every consensus protocol must hold

For a replicated state machine, you need:

  • Agreement (safety): no two correct nodes apply different values at the same log index.
  • Validity: a decided value was actually proposed by some node (no invented writes).
  • Termination (liveness): every non-faulty node eventually decides.

FLP says that in a fully asynchronous network with even a single crash fault, no deterministic protocol can guarantee all three. Real systems escape by assuming partial synchrony — RTTs are bounded, but the bound is unknown — and by using randomized timers (Raft) or a distinguished proposer (Paxos). That single assumption is why both protocol families are *safe* under partition but may become *unavailable*. They never become inconsistent.

Quorum intersection is the whole ballgame

With 2f+1 voters, any two quorums of f+1 overlap in at least one node. That overlap node refuses to vote twice in the same epoch, so two leaders cannot both commit at the same index.

| Cluster size | Quorum | Crash tolerance | Practical p99 write impact | |---|---|---|---| | 3 | 2 | 1 | Baseline. Best latency/durability ratio for most systems. | | 4 | 3 | 1 | Strictly worse than 3 nodes: same fault tolerance, one more replica to wait on. Never deploy 4. | | 5 | 3 | 2 | p99 is set by the 3rd-fastest replica; ~30–50% lower write throughput than a 3-node group. | | 7 | 4 | 3 | p99 dominated by the 4th-fastest replica. Only justified when correlated AZ failure is a real risk. |

Epochs: terms and ballots are the same idea

Raft uses a monotonically increasing term; Paxos uses a lexicographically ordered ballot (round, nodeID). Both create the same invariant:

> A replica never accepts a message whose epoch is lower than the highest epoch it has observed, and it persists a promise never to vote in an older epoch before replying.

In etcd this is CurrentTerm + VotedFor written to the WAL and fsynced *before* the RequestVote response leaves the process. Skipping that fsync is the classic double-vote-after-crash bug, and it permanently breaks quorum intersection — two leaders can then exist in the same term.

              Partition A (minority, 1/5)              Partition B (majority, 4/5)
        ┌───────────────────────────────┐      ┌──────────────────────────────────┐
        │  old leader, term = 7         │  X   │  elects term = 8, commits writes │
        │  AppendEntries(term=7) ──X──► │      │  AppendEntries(term=8) ──► ...   │
        │  cannot reach quorum →        │      │                                  │
        │  commitIndex frozen forever   │      │                                  │
        └───────────────────────────────┘      └──────────────────────────────────┘
                       ▲                                        │
                       │        partition heals                 │
                       └────────────────────────────────────────┘
        Old leader receives any message with term=8 → becomeFollower(8), log truncated.
        Its uncommitted entries at indexes > commitIndex are discarded safely.

Split-brain mitigation: the checklist that survives production

1. Quorum-acknowledged writes only. Never ack a write from a minority partition, even if the local log accepted it. 2. Persist epoch state before replying. term, votedFor, and the appended log tail must be fsynced before the RPC response is serialized. This is the single most common correctness bug in hand-rolled Raft. 3. Pre-Vote. A restarted node that has been down for an hour comes back with a stale, huge term and immediately forces a cluster-wide election. Pre-Vote makes it win a straw poll *without* incrementing terms first. This was a real production outage class in etcd and Consul; it is enabled by default in modern etcd. 4. CheckQuorum. A leader that cannot reach a quorum within the election timeout must step down to follower, otherwise it keeps serving stale reads from a partitioned minority. 5. Fencing tokens, end to end. Consensus protects the *log*, not your downstream side effects. A leader paused by a 30-second GC pause, a frozen VM, or SIGSTOP can finish an RPC it started before losing leadership. Every external write must carry a monotonically increasing token, and the storage layer must reject any token <= the highest it has already applied. 6. Joint consensus or single-server membership change. Never change a quorum configuration in one step; during the transition you need a quorum of *both* the old and new configurations. 7. Unique node identity. Two processes running with the same node ID — a classic Kubernetes StatefulSet stale-pod bug — is the most common cause of real split-brain in the wild.

---

2. Raft in etcd: Leader Election, Log Replication, and Read Consistency

Raft runs on two RPCs. Everything else is optimization.

  • RequestVote(term, candidateID, lastLogIndex, lastLogTerm) — Phase-1 style.
  • AppendEntries(term, leaderID, prevLogIndex, prevLogTerm, entries[], leaderCommit) — Phase-2 style, and it doubles as the heartbeat.

Log Matching and Leader Completeness

Two properties carry the entire safety proof:

  • Log Matching: if two logs contain an entry with the same index and term, the logs are identical in all preceding entries. Enforced by the prevLogIndex/prevLogTerm check on every AppendEntries.
  • Leader Completeness: an entry committed in term *T* appears in the log of every leader elected in terms > *T*. Enforced by the election restriction: a voter rejects a candidate whose last log term is older, or whose last log term is equal but whose log is shorter.

The subtle part is the commit rule. A leader may only advance commitIndex to an index *N* if (a) a quorum has matchIndex >= N and (b) log[N].term == currentTerm. Dropping clause (b) is the Raft paper's Figure 8 bug: a leader can replicate an entry from an old term to a quorum, commit it, and then be replaced by a leader that overwrites it. Older-term entries commit *transitively* once a current-term entry above them commits.

  Candidate            Follower A            Follower B
     │
     │  RequestVote(term=8, lastLog=(142, 7))
     ├─────────────────────►│
     │◄──── granted ────────│
     ├──────────────────────────────────────────►│
     │◄──────────── granted ─────────────────────│
     │  votes(2) + self = quorum → leader(term=8)
     │
     │  AppendEntries(term=8, prev=(142,7), entries=[(143,8)])
     ├─────────────────────►│  WAL fsync            
     │◄──── success ─────────│
     ├──────────────────────────────────────────►│  WAL fsync
     │◄──────────── success ─────────────────────│
     │  matchIndex[143] >= quorum AND log[143].term == 8 → commitIndex = 143

Timing parameters in etcd

etcd defaults are --heartbeat-interval=100ms and --election-timeout=1000ms, with the actual election timer randomized in [electionTimeout, 2×electionTimeout). The rule to follow: heartbeat must be comfortably below the smallest election timeout, and election timeout must be at least 10× the p99 inter-node RTT. If your cross-AZ p99 RTT is 8 ms, a 1 s election timeout is generous; if you are spanning regions with 90 ms p99 RTT, a 1 s timeout is still safe but a 200 ms one will cause permanent election storms.

Linearizable reads without writing to the log

Naively, a linearizable read costs a log append. etcd instead uses the ReadIndex protocol for strong reads:

1. Record the leader's current commitIndex as readIndex. 2. Confirm leadership with one heartbeat round to a quorum (never trust a local timer). 3. Wait until appliedIndex >= readIndex, then serve the read from the local state machine.

Cost: one quorum round trip and zero WAL writes. The alternative — a leader lease — is faster but only safe if clock drift is bounded, which is precisely the assumption that Raft deliberately avoids for safety. clientv3 also exposes serializable reads (WithSerializable()), which skip the quorum round trip entirely and can return data stale by hundreds of milliseconds. Use them for metrics and dashboards, never for the read-modify-write path of a lock service.

Snapshots, compaction, and the storage wall

etcd stores its state machine in bbolt, a copy-on-write B+tree. Every write allocates new pages; the file only grows. Two background jobs keep it healthy:

  • Compaction (etcdctl compact) drops historical MVCC revisions.
  • Defragmentation (etcdctl defrag) rewrites the file and returns free pages to the OS.

The default backend quota is 2 GB, and operational guidance is to stay under ~8 GB — beyond that, defrag pauses and page faults on the mmap become a real availability risk. The NOSPACE alarm is the most common etcd outage I have seen in production, and it is always a compaction/defrag hygiene failure, not a consensus failure.

---

3. Multi-Paxos in Spanner: Leases, TrueTime, and External Consistency

Paxos in one paragraph

A Paxos ballot is a pair (round, nodeID) compared lexicographically. Phase 1 (Prepare/Promise) asks a quorum to promise never to accept a lower ballot. Phase 2 (Accept/Accepted) proposes a value under that ballot. A value is chosen when a quorum accepts it. Because every promise is durable, two different values can never be chosen in the same *slot*.

Multi-Paxos amortizes Phase 1: once a leader holds a ballot and a quorum has promised, it can skip Phase 1 and run only Phase 2 for each subsequent slot — one RTT per decision, identical steady-state cost to Raft.

Where Raft and Multi-Paxos actually diverge

The divergence is not in the quorum math; it is in what the log is allowed to look like.

| Dimension | Raft (etcd) | Multi-Paxos (Spanner) | |---|---|---| | Log shape | Contiguous, no holes. prevLogIndex/prevLogTerm chains every append. | Slot-per-index; holes are legal and filled by no-ops. | | Leader election | Randomized election timeout + term number. | Ballot (round, nodeID) Phase 1; leader is stable until a higher ballot appears. | | Safety artifact | Log Matching + Leader Completeness + current-term commit rule. | Quorum intersection over promises + acceptance of the highest promised ballot. | | Membership change | Single-server change or joint consensus. | Paxos group reconfiguration with overlapping quorums. | | Heartbeat purpose | Suppress elections and carry commitIndex. | Hold the ballot and carry commit/lease state. | | Clock dependence | None for safety; timers affect liveness only. | Leases and TrueTime for external consistency. | | Reads | ReadIndex (quorum round trip) or serializable local reads. | Snapshot reads at a timestamp; leader lease reads with no round trip. | | Implementation complexity | Materially lower — recovery is prefix-based, which is why reference implementations are readable. | Higher — ballot recovery, log holes, group management, and lease accounting. |

Spanner's twist: 2PC over Paxos groups + commit-wait

Spanner shards data into *splits* whose replicas form a Paxos group of 3–5 replicas across zones. A transaction touching multiple splits runs 2PC, but the participants are *Paxos groups*, not single machines — so the transaction coordinator is itself replicated and is not the classic 2PC single point of failure.

What makes Spanner externally consistent is not Paxos: it is TrueTime. TrueTime exposes TT.now() as an interval [earliest, latest] with a bounded uncertainty ε, derived from GPS receivers and atomic clocks in every datacenter. A read-write transaction picks a commit timestamp s = TT.now().latest, then commit-waits until TT.now().earliest > s — roughly 2ε — before releasing locks. After that wait, any other transaction anywhere on Earth that starts afterwards is guaranteed to observe a clock reading greater than s, so it sees the commit. That is global linearizability with serializable transactions, and it is the property Multi-Paxos alone cannot deliver.

 client ──write──► Coordinator (leader of Paxos group L, TrueTime lease held)
                          │
                          │ 1. acquire commit timestamp s = TT.now().latest
                          │ 2. 2PC prepare ──► Participant Paxos group A ──► quorum accept
                          │                └► Participant Paxos group B ──► quorum accept
                          │◄── prepared ───────────────────────────────────────
                          │ 3. COMMIT record logged through Paxos group L (quorum accept)
                          │ 4. commit-wait: block until TT.now().earliest > s   (~2ε)
                          └──► ack to client

  Read-only transactions skip 2PC and commit-wait entirely: they are served as
  snapshot reads at a TrueTime-derived timestamp, which is why Spanner reads are
  single-digit milliseconds while writes pay the clock uncertainty once.

In a single region a Spanner read-write commit typically lands in the ~10 ms range (one Accept round trip plus single-digit-millisecond commit-wait). In a 3-region configuration the same commit crosses an ocean and lands at 80–150 ms p50 with several-hundred-millisecond p99s. That is the price of external consistency at planetary scale, and it is a price most systems should not pay: CockroachDB and YugabyteDB approximate the same semantics with Raft plus hybrid logical clocks and a bounded max-offset, trading a small correctness window for region-local latency.

---

Practical Implementation

The following is the safety-critical core of a Raft node in Go. Transport and persistence are injected, and every decision that leaves the process is preceded by a durable write. Helpers that are trivial but load-bearing (entryAt, conflictHint, persistLog) are included so the file compiles as-is.

// Package raft implements the leader election, log replication and commit core
// of Raft (Ongaro & Ousterhout, USENIX ATC 2014). Transport and persistence are
// injected so the same core can run over gRPC, HTTP or a unix socket.
//
// The invariant every function below protects:
// a node NEVER replies to an RPC before the state used to make that decision
// is durably fsynced.
package raft

import (
	"context"
	"errors"
	"fmt"
	"sync"
	"time"
)

var (
	ErrNotLeader = errors.New("raft: not the leader")
	ErrStaleTerm = errors.New("raft: observed a higher term")
	ErrNoQuorum  = errors.New("raft: lost quorum")
)

// Store must fsync before Put returns. A store that buffers in the page cache
// breaks the safety proof: a crashed node can forget a vote it already granted
// and vote a second time in the same term.
type Store interface {
	Put(key string, value []byte) error
	Get(key string) ([]byte, error)
}

// WAL durably appends log entries. Append must tolerate re-delivery of entries
// it has already persisted, because RPCs are retried.
type WAL interface {
	Append(entries []Entry) error
}

type Transport interface {
	RequestVote(ctx context.Context, peer string, req *RequestVoteRequest) (*RequestVoteResponse, error)
	AppendEntries(ctx context.Context, peer string, req *AppendEntriesRequest) (*AppendEntriesResponse, error)
}

type State uint8

const (
	Follower State = iota
	Candidate
	Leader
)

type Entry struct {
	Term  uint64
	Index uint64
	Data  []byte
}

type RequestVoteRequest struct {
	Term         uint64
	CandidateID  string
	LastLogIndex uint64
	LastLogTerm  uint64
}

type RequestVoteResponse struct {
	Term        uint64
	VoteGranted bool
}

type AppendEntriesRequest struct {
	Term         uint64
	LeaderID     string
	PrevLogIndex uint64
	PrevLogTerm  uint64
	Entries      []Entry
	LeaderCommit uint64
}

type AppendEntriesResponse struct {
	Term       uint64
	Success    bool
	MatchIndex uint64
	// Conflict hints let the leader skip an entire term of entries on mismatch
	// instead of decrementing nextIndex one entry at a time. After a long
	// partition this is worth orders of magnitude in recovery time.
	ConflictIndex uint64
	ConflictTerm  uint64
}

type Node struct {
	mu sync.Mutex

	id        string
	peers     map[string]Transport
	store     Store
	wal       WAL
	heartbeat time.Duration
	election  time.Duration

	currentTerm uint64
	votedFor    string // empty string means "no vote yet"
	log         []Entry // log[0] is the sentinel {Index: 0, Term: 0}
	commitIndex uint64
	persisted   uint64

	state      State
	leaderID   string
	nextIndex  map[string]uint64
	matchIndex map[string]uint64
}

// quorum is the majority size for the current voter set.
func (n *Node) quorum() int { return len(n.peers)/2 + 1 }

// persist writes epoch state. Call it with n.mu held.
func (n *Node) persist() error {
	if err := n.store.Put("term", []byte(fmt.Sprint(n.currentTerm))); err != nil {
		return fmt.Errorf("persist term: %w", err)
	}
	if err := n.store.Put("votedFor", []byte(n.votedFor)); err != nil {
		return fmt.Errorf("persist votedFor: %w", err)
	}
	return nil
}

// persistLog appends only the un-persisted tail via the WAL.
func (n *Node) persistLog() error {
	if n.persisted >= n.lastLogIndex() {
		return nil
	}
	if err := n.wal.Append(n.log[n.persisted+1:]); err != nil {
		return fmt.Errorf("wal append: %w", err)
	}
	n.persisted = n.lastLogIndex()
	return nil
}

// HandleRequestVote is the inbound half of voting (Paxos Phase 1 analogue).
func (n *Node) HandleRequestVote(ctx context.Context, req *RequestVoteRequest) (*RequestVoteResponse, error) {
	n.mu.Lock()
	defer n.mu.Unlock()

	resp := &RequestVoteResponse{Term: n.currentTerm}

	if req.Term < n.currentTerm {
		return resp, nil // stale epoch: reject and report our higher term
	}
	if req.Term > n.currentTerm {
		n.becomeFollowerLocked(req.Term)
	}

	// Election restriction: a candidate whose log is behind ours can never win
	// our vote. This is what makes Leader Completeness hold, and it is the
	// cheapest safety check in the whole protocol.
	lastIndex, lastTerm := n.lastLogIndexTerm()
	upToDate := req.LastLogTerm > lastTerm ||
		(req.LastLogTerm == lastTerm && req.LastLogIndex >= lastIndex)

	if (n.votedFor == "" || n.votedFor == req.CandidateID) && upToDate {
		n.votedFor = req.CandidateID
		if err := n.persist(); err != nil {
			// Fail closed: no reply, therefore no vote. The candidate retries.
			return nil, fmt.Errorf("handle request vote: %w", err)
		}
		n.resetElectionTimerLocked()
		resp.VoteGranted = true
	}
	return resp, nil
}

// HandleAppendEntries is the inbound half of replication (Paxos Phase 2 analogue)
// and doubles as the heartbeat.
func (n *Node) HandleAppendEntries(ctx context.Context, req *AppendEntriesRequest) (*AppendEntriesResponse, error) {
	n.mu.Lock()
	defer n.mu.Unlock()

	resp := &AppendEntriesResponse{Term: n.currentTerm}

	if req.Term < n.currentTerm {
		return resp, nil
	}
	if req.Term > n.currentTerm {
		n.becomeFollowerLocked(req.Term)
	}
	n.state = Follower
	n.leaderID = req.LeaderID
	n.resetElectionTimerLocked() // a legitimate leader exists in this term

	// Log Matching Property: reject unless the prefix matches exactly.
	if req.PrevLogIndex > 0 {
		prev, ok := n.entryAt(req.PrevLogIndex)
		if !ok || prev.Term != req.PrevLogTerm {
			resp.ConflictIndex, resp.ConflictTerm = n.conflictHint(req.PrevLogIndex)
			return resp, nil
		}
	}

	for i, e := range req.Entries {
		idx := req.PrevLogIndex + uint64(i) + 1
		existing, ok := n.entryAt(idx)
		if ok && existing.Term == e.Term {
			continue // already have this entry: idempotent
		}
		if idx <= n.commitIndex {
			// A correct leader never rewrites committed entries. If this fires,
			// the leader is buggy and silently proceeding would diverge state
			// machines permanently, which is the worst outcome available.
			panic(fmt.Sprintf("raft: leader tried to rewrite committed index %d", idx))
		}
		if ok {
			n.log = n.log[:idx] // truncate the conflicting uncommitted suffix
		}
		n.log = append(n.log, Entry{Term: e.Term, Index: idx, Data: e.Data})
	}
	if err := n.persistLog(); err != nil {
		return nil, fmt.Errorf("handle append entries: %w", err)
	}

	if req.LeaderCommit > n.commitIndex {
		n.commitIndex = min(req.LeaderCommit, n.lastLogIndex())
		n.applyCommittedLocked()
	}

	resp.Success = true
	resp.MatchIndex = n.lastLogIndex()
	return resp, nil
}

// advanceCommitLocked is called by the leader after every successful ack.
// The current-term guard fixes the Raft paper's Figure 8 bug: an entry from an
// older term must never be committed by counting replicas, because a later
// leader can still overwrite it.
func (n *Node) advanceCommitLocked() {
	for idx := n.lastLogIndex(); idx > n.commitIndex; idx-- {
		e, ok := n.entryAt(idx)
		if !ok || e.Term != n.currentTerm {
			return
		}
		replicated := 1 // the leader's own log always matches
		for id := range n.peers {
			if n.matchIndex[id] >= idx {
				replicated++
			}
		}
		if replicated >= n.quorum() {
			n.commitIndex = idx
			n.applyCommittedLocked()
			return
		}
	}
}

// LinearizableRead implements the ReadIndex protocol. It does NOT append to the
// log: it pins the current commit index, proves leadership with one heartbeat
// round to a quorum, then lets the caller wait for the state machine to catch
// up. Cost: one quorum round trip, zero WAL writes.
func (n *Node) LinearizableRead(ctx context.Context) (uint64, error) {
	n.mu.Lock()
	if n.state != Leader {
		term, leader := n.currentTerm, n.leaderID
		n.mu.Unlock()
		return 0, fmt.Errorf("%w: term %d, known leader %q", ErrNotLeader, term, leader)
	}
	readIndex, term := n.commitIndex, n.currentTerm
	peers := make([]string, 0, len(n.peers))
	for id := range n.peers {
		peers = append(peers, id)
	}
	n.mu.Unlock()

	acks := 1
	for _, id := range peers {
		// A heartbeat-only AppendEntries: the lease-free way to prove we are
		// still leader without trusting a local clock.
		resp, err := n.peers[id].AppendEntries(ctx, id, &AppendEntriesRequest{Term: term, LeaderID: n.id})
		if err != nil {
			continue // a failed peer hurts liveness, never safety
		}
		if resp.Term > term {
			return 0, fmt.Errorf("%w: peer %s reported term %d", ErrStaleTerm, id, resp.Term)
		}
		acks++
		if acks >= n.quorum() {
			break
		}
	}
	if acks < n.quorum() {
		return 0, ErrNoQuorum
	}
	return readIndex, nil
}

func (n *Node) entryAt(idx uint64) (Entry, bool) {
	if idx >= uint64(len(n.log)) {
		return Entry{}, false
	}
	return n.log[idx], true
}

func (n *Node) lastLogIndex() uint64 { return uint64(len(n.log) - 1) }

func (n *Node) lastLogIndexTerm() (uint64, uint64) {
	last := n.log[len(n.log)-1]
	return last.Index, last.Term
}

// conflictHint returns the first index of the conflicting term so the leader can
// jump back a whole term in a single round trip.
func (n *Node) conflictHint(idx uint64) (uint64, uint64) {
	if idx > n.lastLogIndex() {
		return n.lastLogIndex() + 1, 0
	}
	term := n.log[idx].Term
	for idx > 0 && n.log[idx-1].Term == term {
		n.idx--
	}
	return idx, term
}

func (n *Node) resetElectionTimerLocked() {
	// In production: reset a monotonically-jittered timer to
	// n.election + rand(n.election). Randomization is what breaks symmetry
	// and guarantees liveness; without it, split votes repeat forever.
	_ = time.Duration(0)
}

func (n *Node) becomeFollowerLocked(term uint64) {
	n.currentTerm = term
	n.votedFor = ""
	n.state = Follower
	n.leaderID = ""
}

func (n *Node) applyCommittedLocked() {
	// Hand committed entries to the state machine in index order. Applying must
	// be idempotent and must never be done concurrently with a snapshot install.
}

And the client side of the same problem — electing a leader and minting a fencing token — is where most production outages actually originate:

package leader

import (
	"context"
	"fmt"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"
	"go.etcd.io/etcd/client/v3/concurrency"
)

// Campaign blocks until this process wins the election (or ctx is cancelled) and
// returns a session, the election handle, and a strictly monotonic fencing token.
//
// The token is the etcd revision at which the leader key was written. Revisions
// are a cluster-wide monotonic counter, so two leaders elected at different
// times can never mint the same token. Every downstream write must carry it and
// the storage layer must reject any token it has already seen - consensus alone
// cannot stop a leader paused by a 30s GC stop-the-world from completing a write
// it began before it lost leadership.
func Campaign(ctx context.Context, endpoints []string, candidateID string) (*concurrency.Session, *concurrency.Election, int64, error) {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:            endpoints,
		DialTimeout:          5 * time.Second,
		DialKeepAliveTime:    10 * time.Second,
		DialKeepAliveTimeout: 3 * time.Second,
	})
	if err != nil {
		return nil, nil, 0, fmt.Errorf("etcd dial: %w", err)
	}

	sess, err := concurrency.NewSession(cli, concurrency.WithTTL(10), concurrency.WithContext(ctx))
	if err != nil {
		return nil, nil, 0, fmt.Errorf("etcd session: %w", err)
	}

	elec := concurrency.NewElection(sess, "/locks/ledger-writer")
	if err := elec.Campaign(ctx, candidateID); err != nil {
		_ = sess.Close()
		return nil, nil, 0, fmt.Errorf("campaign: %w", err)
	}

	// Leader() is a linearizable (ReadIndex) read of the elected key. Do not
	// replace this with a serializable read: a stale read here hands two
	// processes the same leadership decision.
	resp, err := elec.Leader(ctx)
	if err != nil {
		_ = elec.Resign(ctx)
		return nil, nil, 0, fmt.Errorf("read leader: %w", err)
	}
	if len(resp.Kvs) == 0 {
		_ = elec.Resign(ctx)
		return nil, nil, 0, fmt.Errorf("campaign: succeeded but leader key is empty")
	}

	return sess, elec, resp.Header.Revision, nil
}

---

Performance Benchmarks & Trade-Offs

The numbers below are order-of-magnitude figures assembled from the published etcd and Spanner benchmarks plus production telemetry on NVMe-backed nodes in a 3-AZ topology. Treat them as ratios, not absolutes: WAL fsync latency dominates every Raft and Paxos measurement on the market.

| System / protocol | Leader model | Steady-state write cost | p50 write | p99 write | Read path | Reconfiguration | |---|---|---|---|---|---|---| | etcd 3.5, Raft, 3 nodes | Single leader per term, randomized election | 1 quorum RTT + 1 WAL fsync | 2–4 ms | 10–22 ms | ReadIndex or serializable local read | Single-node change (joint consensus in v3.6+) | | etcd 3.5, Raft, 5 nodes across 3 AZ | Same | 1 RTT to the 3rd-fastest replica | 3–6 ms | 15–30 ms | Same, but quorum round trip is slower | Same | | Spanner regional, Multi-Paxos, 3 replicas | Paxos leader holding a TrueTime lease | 1 Accept RTT + ~2ε commit-wait | ~8–15 ms | 25–60 ms | Snapshot read at timestamp, no commit-wait | Paxos group reconfiguration with overlapping quorums | | Spanner multi-region, 5 replicas, 3 regions | Same | 1 Accept RTT across regions + ~2ε | 80–150 ms | 250–500 ms | Stale and snapshot reads stay cheap | Same | | CockroachDB, Raft + HLC leaseholder, 3 AZ | Per-range leaseholder | 1 quorum RTT + fsync | 3–8 ms | 20–45 ms | Leaseholder read within max-offset | Range split/merge plus Raft conf change | | ZooKeeper 3.5, Zab, 3 nodes | Leader + Zab epoch | 1 quorum RTT + fsync | 1–3 ms | 8–20 ms | sync read forces a leader round trip | Dynamic reconfig with quorum version | | Cross-region Raft, us-east to eu-west (~80 ms RTT) | Single leader | 1 RTT to the farthest quorum member | 80–120 ms | 200–350 ms | ReadIndex needs a quorum round trip | Avoid; stretch quorums only for DR |

Three structural conclusions fall out of that table:

1. Same-continent, same-protocol differences are noise. etcd, Zab, and CockroachDB differ by single-digit milliseconds because they all pay one quorum RTT plus one fsync. Protocol choice matters for *operability*, not steady-state latency. 2. Cross-region quorums are a latency tax with no upper bound. Going from 3 AZs to 3 regions multiplies p99 by 10–20×. If you need global writes, accept that cost deliberately; do not stumble into it by "spreading replicas for safety". 3. The read path is where you win. etcd's ReadIndex avoids WAL writes entirely, and Spanner's snapshot reads avoid commit-wait entirely. Most real workloads are read-dominated, which is why the write-side protocol argument is often far less consequential than the read strategy.

---

Pro-Tips & Common Pitfalls

  • Never ack before fsync. Persist term, votedFor, and the log tail before serializing the RPC response. Every hand-rolled Raft bug I have debugged traces back to this one line of ordering.
  • Never serve a leader read without ReadIndex or a clock-bounded lease. A partitioned leader happily serves stale data forever otherwise. This is the quietest possible data-corruption bug.
  • Do not tune election timeouts by feel. Derive them: heartbeat ≈ p99 RTT × 2, election timeout ≈ 10 × heartbeat. On a WAN with 90 ms p99 RTT, a 200 ms election timeout guarantees permanent instability.
  • Never deploy an even number of voters. Four nodes tolerate exactly as many failures as three while adding a replica to every quorum. Use a learner (non-voting) replica if you want an extra copy for DR.
  • Never run two processes with the same node ID. This is the number one cause of genuine split-brain in Kubernetes: a StatefulSet pod that is not fully terminated rejoins while the replacement is live. Use terminationGracePeriodSeconds larger than the election timeout and PodDisruptionBudgets rather than autoscaling.
  • Watch WAL fsync p99, not average. etcd guidance is to keep p99 fsync under roughly 10 ms (etcd_disk_wal_fsync_duration_seconds). If it regresses, you will see elections, term inflation, and a throughput collapse within minutes long before any alarm fires on latency.
  • Compact and defrag on a schedule. The NOSPACE alarm is an operational hygiene failure, not a consensus failure, and it takes the whole cluster read-only.
  • Change membership one node at a time, and never add a node that is already behind. New members should join as learners, catch up, then be promoted.
  • Throttle snapshot transfers. Shipping a multi-gigabyte snapshot can saturate a NIC, starve heartbeats, and trigger an election on the very follower you are trying to repair.
  • Fence at the storage layer. Consensus guarantees agreement on the log, not on your S3 puts. Every external write must carry a token that has never been reused.
  • Do not confuse serializable reads with linearizable reads. Serializable reads are stale, sometimes by hundreds of milliseconds, and they are strictly cheaper. Use them for dashboards only.

---

Frequently Asked Questions (FAQ)

Is Raft just Paxos with a leader?

Functionally, yes — and that framing is the most useful mental model. Raft can be understood as Multi-Paxos restricted to ballots of the form (term, leaderID) with a fixed slot for every log index and a hard requirement that no slot is ever empty. Multi-Paxos permits holes in the log, which must be filled with no-ops during recovery. That restriction is not a weakness: a contiguous log gives you a single prefix to compare during AppendEntries recovery, makes snapshotting a suffix-truncation problem, and makes membership change expressible as joint consensus. The trade-off is a stronger leader dependency — a Raft follower cannot accept a proposal from anyone but the current leader for the current term.

How do I prevent split-brain in a Kubernetes-hosted etcd or Raft cluster?

Five layers, in order of importance. First, always use an odd quorum (3 or 5) and enforce anti-affinity so voters land in separate zones. Second, guarantee unique node identity: never let a stale pod rejoin with the same node ID — set terminationGracePeriodSeconds above the election timeout, use a PodDisruptionBudget, and never autoscale a consensus cluster. Third, enable Pre-Vote and CheckQuorum so a rejoining node cannot force elections and a partitioned leader steps down. Fourth, never stretch the quorum across regions unless you accept 10× p99. Fifth, issue fencing tokens to every downstream write, because none of the first four layers protects you from a process that was paused rather than crashed.

What is the lowest-latency way to get linearizable reads from a Raft group?

Use ReadIndex, and batch it. The leader pins its current commitIndex, confirms leadership with a single heartbeat round to a quorum, and serves the read once the local state machine has applied past that index — one quorum round trip, zero WAL writes, and it works even with unbounded clock drift. The cheaper-looking alternative, a leader lease, removes the round trip but requires a bounded clock-drift assumption; if a lease is violated you get stale reads with no error, which is far worse than added milliseconds. For truly read-heavy workloads the biggest win is architectural: route reads to the local follower with a read-index forwarded from the leader, or accept bounded staleness with serializable reads. On a 5-node, 3-AZ cluster this typically moves read p99 from 10–20 ms to 2–5 ms.

Can I get Spanner-like consistency without TrueTime?

Yes, if you accept a bounded error window instead of a hardware guarantee. CockroachDB and YugabyteDB attach hybrid logical clocks to Raft, lease a range to a single leaseholder, and serve leaseholder reads within a configured max-offset (commonly 250–500 ms). The correctness argument becomes "linearizable assuming clock skew stays below max-offset", which is enforced by the cluster refusing to start when skew exceeds that bound. You get region-local read latency and multi-region writes without atomic clocks, at the cost of a safety property that depends on NTP discipline rather than on GPS hardware. Google's answer was to make the assumption provable with TrueTime; everyone else's answer is to make the assumption explicit, monitored, and enforced.