Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

Distributed Consensus Raft vs Paxos: etcd vs Google Spanner

By: Md Tarek · 19 Oct 2026

Distributed consensus Raft vs Paxos architecture: leader election, quorum voting and log replication across a rack of servers
Distributed consensus Raft vs Paxos architecture: leader election, quorum voting and log replication across a rack of servers

Executive Summary & Key Takeaways

  • Raft is Multi-Paxos plus three restrictions: a single strong leader per term, a contiguous log with no holes, and a total order over terms. etcd 3.5/3.6 ships exactly that with a 100 ms heartbeat and a 1,000–2,000 ms randomized election timeout, so a leader failover costs roughly 1–2 s of write unavailability.
  • Spanner runs one Multi-Paxos group per tablet (3 or 5 replicas) with 10-second leader leases baked into the Paxos log itself. Steady-state writes cost one Accept round trip (~2 fsyncs); linearizable reads are served locally under the lease with zero network round trips.
  • Neither protocol can split-brain when quorums intersect. A 2f+1 group tolerates f failures and two majorities are mathematically disjoint. Real production split-brain incidents trace to misconfiguration (duplicate --initial-cluster bootstrap, stale data dirs) or missing fencing on external side effects — not to the consensus algorithm.
  • Measured order of magnitude: etcd p50 commit ≈ 2–4 ms in-region with p99 ≈ 8–25 ms under contention; Spanner regional writes p50 ≈ 4–8 ms, multi-region writes p50 ≈ 60–120 ms, dominated by WAN RTT plus commit-wait of ≈2ε (ε measured 1–7 ms, mean ≈4 ms in the 2012 paper; sub-millisecond in modern deployments).
  • Correctness is a property of the whole system: a leader that loses its lease but still holds a socket to S3/Postgres is a zombie. Always carry a monotonic fencing token (etcd lease revision, Paxos ballot number) into every downstream write.

Raft vs Multi-Paxos: The Real Differences

Both algorithms solve the same problem: getting 2f+1 replicas to agree on an ordered sequence of values while tolerating f crash-stop failures and arbitrary message loss, duplication, and reordering. Both rely on the same invariant — quorum intersection. Any two majorities of {A,B,C,D,E} share at least one member, so a value accepted by one quorum cannot be invisible to another.

The differences are engineering constraints, not theory:

| Concern | Multi-Paxos | Raft | |---|---|---| | Leader election | A separate Paxos instance (or ballot-based lease) decides the distinguished proposer | First-class, built into the protocol via terms + RequestVote | | Log shape | Per-slot consensus; holes are legal, chosen values may be at index 7 while 3–6 are still undecided | No holes; the log is a strict prefix-consistent sequence | | Commit rule | A value is chosen when a quorum accepts it for a ballot | Leader commits index i once a majority stores i; but a leader may not commit a prior-term entry by replica count alone — it must first commit an entry from its own term (the Raft Figure 8 trap) | | Membership change | Implemented ad hoc; famously the hardest part to get right in Paxos deployments | Joint consensus (ConfChangeV2) specified in the paper | | Implementation burden | High: "Paxos Made Live" (Chandra et al., 2007) is an 11-page catalogue of the gap between the paper and a shipped system — snapshots, disk corruption, group membership, and disk-failure-induced amnesia | Lower: the paper is explicit about every state transition; etcd, Consul, Nomad, CockroachDB, TiKV, Kafka KRaft, and RethinkDB all converged on it |

A useful mental model: Raft trades theoretical flexibility for implementability. Multi-Paxos (and its Flexible Paxos generalization, where Q1 + Q2 > N allows asymmetric phase quorums) can shave a round trip or reduce WAN traffic — Spanner exploits this with a stable lease-holding leader that skips Phase 1 entirely for every write in the lease window.

Multi-Paxos steady state (Spanner, leader L under lease):
  client -> L            : write(k, v)
  L      -> A, B         : Accept(ballot=b, slot=i, value)
  A, B   -> L            : Accepted(b, i)      [fsync to Paxos log]
  L                      : chosen(i) when quorum accepts; apply; commit-wait

Raft steady state (etcd, leader L in term t):
  client -> L            : Put(k, v)
  L      -> F2, F3       : MsgApp{term=t, prevIndex, entries=[i]}
  F2, F3 -> L            : MsgAppResp{term=t, index=i}   [fsync to WAL]
  L                      : advance commitIndex = i when majority matchIndex >= i

Leader Election and Log Replication Mechanics

etcd: Raft with PreVote and CheckQuorum

etcd's etcd-raft library (raft.go, node.go) is a textbook implementation with two production hardening features enabled by default since 3.4: PreVote and CheckQuorum. Without them, a node returning from a 30-second GC pause or a network partition wakes up, increments its term, and deposes a perfectly healthy leader — the classic "disruptive rejoin" that causes election storms.

t=0ms     F1 (leader, term 7) --MsgHeartbeat--> F2, F3
          [ F1 is partitioned away from F2,F3 ]
t=1100ms  F2 election timeout fires (uniform rand in [1000,2000)ms)
          F2 -> F3 : MsgPreVote{term 8}   (does NOT increment term yet)
t=1101ms  F3 grants PreVote (F3 has not heard a leader within lease)
          F2 becomes candidate: term=8, MsgVote to F3
t=1103ms  F3 grants vote -> F2 wins with 2/3 quorum -> leader
          F1's CheckQuorum fails (no heartbeat replies within election timeout)
          -> F1 steps down to follower. Without CheckQuorum, F1 would keep
             answering lease-based reads with stale data until its lease expired.

The replication core is a pipelined, batched state machine: the leader keeps up to MaxInflightMsgs = 256 in-flight MsgApps, tracks matchIndex/nextIndex per follower, and dribbles uncommitted entries under MaxUncommittedEntriesSize. Every Ready batch is persisted to the WAL with a single fsync before it may be sent to peers — fsync latency is the throughput ceiling, which is why etcd benchmarks scale almost linearly with write batching size.

Spanner: Multi-Paxos with Leases and TrueTime

Spanner decouples liveness from safety: the leader of a Paxos group holds a 10-second lease written into the log at a specific ballot. Until that lease expires, no other replica may propose at a higher ballot, so Phase 1 (Prepare/Promise) is skipped for every write in the window. The lease is safe under clock rate drift (a replica can only start an election after its own copy of the lease has provably expired), not absolute clock synchronization.

Reads split into two paths:

  • Strong reads are served by the leader under its lease — single-digit milliseconds, one hop, no Paxos round trip.
  • Snapshot reads at a past timestamp are served by any replica whose applied state has reached that timestamp.

The global ordering guarantee comes from TrueTime: every commit timestamp is assigned as TT.now().latest + 1, and the transaction is held in *commit-wait* until TT.now().earliest > commit timestamp. Committing therefore costs an extra ≈2ε of wall clock. When a TrueTime reference degrades, Spanner's behavior is unambiguous: it waits, and if ε grows unbounded, it stops serving writes. Availability is sacrificed, never correctness.

Log replication war stories

  • etcd, 3.4 regression: clusters with low --snapshot-count combined with a stalled follower produced repeated MsgSnap transfers that saturated the network and wedged the leader. Defaults moved from 10,000 to 100,000 entries precisely to make snapshots rare rather than per-second.
  • etcd, the backend quota trap: writes keep succeeding until the bbolt backend crosses --quota-backend-bytes (2 GiB default), at which point the cluster raises a NOSPACE alarm and rejects all writes including the ones you need to fix it. You must etcdctl defrag and alarm disarm manually. Compaction is a separate operation from defragmentation; forgetting either is the single most common etcd outage.
  • Spanner, dueling proposers: the pre-lease era of Multi-Paxos suffered from dueling leaders after a partition healed. Leases plus a stable leader turn a latency-expensive Phase 1 negotiation into a scheduled, single-owner renewal that is easy to reason about and to alert on.

Split-Brain Mitigation: Quorums, Leases, and Fencing

Split-brain is not one failure mode; it is three, and each needs a different defense.

1. Two leaders in the same group (algorithmic). Impossible under both protocols if acceptors persist their promises. In Raft, a candidate needs votes from a majority and each node votes at most once per term. In Paxos, an acceptor that answers Promise(ballot n) has durably rejected every ballot < n. The subtlety: the promise must be fsynced before the response is sent. An acceptor that promises in RAM and crashes can accept two conflicting values at different ballots — a genuine safety violation. This is why Spanner writes each Paxos accept to a replicated, fsynced log and why etcd treats WAL write failure as fatal (panic after retries).

2. A deposed leader serving stale data (temporal). The leader is alive, reachable by clients, and simply hasn't learned it was deposed. Two defenses:

  • CheckQuorum (etcd): the leader steps down if it cannot reach a quorum within one election timeout. Cost: a slow-but-alive leader can voluntarily abdicate under transient network jitter.
  • ReadIndex (the safe alternative): before serving a linearizable read, the leader confirms with a quorum that it is still leader, then waits until its applied index ≥ the read's index. Costs one round trip; assumes nothing about clocks. etcd's default ReadOnlyLeaseBased skips that round trip and instead relies on a bounded clock-drift assumption — faster, but only safe if your NTP/chrony discipline actually holds.

3. Zombie processes touching external systems (fencing). No consensus protocol can stop a paused process from writing to Postgres. This is the domain of fencing tokens: every leadership acquisition yields a monotonically increasing number (etcd: the revision of the leader key; Spanner: the Paxos ballot), and every downstream mutation asserts that number is greater than the last one seen. Systems that skip this — including several widely used distributed schedulers in their early versions — have produced double-executed jobs.

Partition: [ A(leader, term 7) ]  x  [ B, C ]

  A's view : still leader, CheckQuorum fails -> step down. Writes to A fail.
  B/C view : elect a new leader in term 8 after 1-2s. Writes to B succeed.
  Clients  : must route via leader-discovery (etcd /leader key watch), and
             must carry a fencing token for every external side effect.

Practical Implementation

Production-grade leadership is not if isLeader { doWork() }. You need a lease, a monotonic token, and a downstream guard that makes a zombie's write fail closed. This Go example uses etcd's Raft cluster as the consensus substrate and demonstrates all three.

package leadership

import (
	"context"
	"errors"
	"fmt"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"
	"go.etcd.io/etcd/client/v3/concurrency"
)

// ErrFenced means the downstream store rejected our token: another process has
// already won leadership. The write MUST be discarded, not retried blindly.
var ErrFenced = errors.New("leadership lost: fencing token rejected")

// Leader is a single-writer guard over an etcd (Raft) cluster.
type Leader struct {
	cli      *clientv3.Client
	session  *concurrency.Session
	election *concurrency.Election
	token    int64 // monotonic fencing token = etcd revision of our leader key
}

// Campaign blocks until this process wins the election on lockKey.
// ttl must be >= 3x cross-AZ p99 RTT (failover speed) and <= your max tolerated
// pause, otherwise a long GC stop will silently lose the lease mid-write.
func Campaign(ctx context.Context, cli *clientv3.Client, lockKey, id string, ttl time.Duration) (*Leader, error) {
	sess, err := concurrency.NewSession(cli,
		concurrency.WithTTL(int(ttl/time.Second)),
		concurrency.WithContext(ctx),
	)
	if err != nil {
		return nil, fmt.Errorf("open session: %w", err)
	}
	e := concurrency.NewElection(sess, lockKey)
	if err := e.Campaign(ctx, id); err != nil { // blocks until quorum grants the lease
		_ = sess.Close()
		return nil, fmt.Errorf("campaign %q: %w", lockKey, err)
	}
	// Linearizable read: no WithSerializable() => etcd performs a ReadIndex
	// barrier, so this revision is authoritative, not a stale local read.
	resp, err := cli.Get(ctx, e.Key())
	if err != nil || len(resp.Kvs) == 0 {
		_ = sess.Close()
		return nil, fmt.Errorf("read own leader key: %w", err)
	}
	return &Leader{cli: cli, session: sess, election: e, token: resp.Header.Revision}, nil
}

// Token is the value downstream systems must persist to detect a zombie writer.
func (l *Leader) Token() int64 { return l.token }

// Execute runs fn only while we still hold the lease. fn receives the token and
// must pass it to the downstream guard; the guard, not this check, is the
// real safety net (a lease can expire between the check and the write).
func (l *Leader) Execute(ctx context.Context, fn func(ctx context.Context, token int64) error) error {
	if _, err := l.cli.Get(ctx, l.election.Key(), clientv3.WithCountOnly()); err != nil {
		return fmt.Errorf("lease liveness probe: %w", err)
	}
	if err := fn(ctx, l.token); err != nil {
		if errors.Is(err, ErrFenced) {
			l.signalLoss()
		}
		return err
	}
	return nil
}

// Done fires when the lease is revoked (failover, partition, or TTL expiry).
// Wire this to context cancellation for every long-running side-effect worker.
func (l *Leader) Done() <-chan struct{} { return l.session.Done() }

func (l *Leader) signalLoss() {
	// Stop accepting new work immediately; in-flight writes will fail on the
	// downstream guard and must be surfaced as errors, not swallowed.
	_ = l.session.Close()
}

// Resign releases the key so a warm follower takes over in one RTT instead of
// waiting out the full TTL. Always call this on graceful shutdown.
func (l *Leader) Resign(ctx context.Context) error {
	if err := l.election.Resign(ctx); err != nil {
		return fmt.Errorf("resign: %w", err)
	}
	return l.session.Close()
}

The downstream guard is a single SQL predicate — cheap, atomic, and correct even if the leader was paused for 5 minutes between the probe and the update:

-- last_token is the fencing token of the last accepted writer.
-- RowsAffected == 0 means we were deposed: abort, do not retry on this node.
UPDATE inventory
   SET reserved   = reserved + $1,
       last_token = $2
 WHERE sku        = $3
   AND last_token < $2
RETURNING last_token;

Two operational notes that matter more than the code: watch session.Done() in a goroutine and cancel every worker context on it, and prefer ReadIndex-style linearizable reads (clientv3.Get without WithSerializable()) for anything that gates a write — stale reads that feed writes are how lost updates sneak into "strongly consistent" systems.

Performance Benchmarks & Trade-Offs

Representative order-of-magnitude figures, not vendor SLAs: 3-node clusters, NVMe, 0.5–1 ms cross-AZ RTT, 1 KiB payloads with batching enabled. Your fsync and WAN RTT dominate everything else.

| Dimension | Raft (etcd 3.5/3.6, 3 nodes) | Multi-Paxos (Cloud Spanner, regional 3) | Zab (ZooKeeper 3.8, 5 nodes) | |---|---|---|---| | Leader election trigger | Randomized timeout (1,000–2,000 ms), PreVote-gated | Ballot + 10 s lease expiry, no dueling | FLE over epochs + zxid ordering | | Steady-state write cost | 1 leader round trip + quorum fsync | 1 Accept round + quorum fsync (Phase 1 skipped under lease) | 1 proposal + quorum fsync | | p50 commit latency (in-region) | 2–4 ms | 4–8 ms | 1–3 ms | | p99 commit latency | 8–25 ms (contention, snapshots) | 15–40 ms | 10–30 ms | | p50 commit latency (multi-region) | 30–60 ms (2 AZ + 1 region) | 60–120 ms (WAN + commit-wait ≈2ε) | 40–80 ms | | Write throughput (small ops) | 5k–15k txn/s with batching | High; server-side batching + 2PC coordinator | 10k+ sync writes/s | | Clock assumptions | Lease reads need a bounded drift rate; ReadIndex needs none | TrueTime bounded uncertainty ε on every commit | None for correctness (sessions use client timeouts) | | Membership change | Joint consensus in the library; server applies one member at a time | Out-of-band Paxos group reconfiguration | Reconfig via Zab proposal | | Failure tolerance | f of 2f+1 | f of 2f+1, per tablet | f of 2f+1 | | Best fit | Cluster metadata, leader election, config, service discovery | Globally consistent OLTP across regions | Coordination, locks, watch-based config |

Choosing between them is rarely choosing a protocol; it is choosing an operational contract. Raft gives you a small, easy-to-reason-about state machine with a 1–2 s failover window you must design for. Multi-Paxos with leases gives you a stable writer and sub-round-trip reads, at the cost of needing a trustworthy time source and a much more complex implementation (2PC, timestamp oracle, commit-wait). If you do not need cross-region external consistency, TrueTime-style infrastructure buys you nothing and costs you availability.

Pro-Tips & Common Pitfalls

  • Even-sized clusters are a trap. 4 nodes tolerate the same 1 failure as 3, add a hop, and make quorum a coin flip. Use 3 or 5 plus non-voting learners for read scale.
  • Tune timeouts against p99, not p50. Rule of thumb: heartbeat ≈ election_timeout / 10, and election_timeout ≥ 10 × p99 heartbeat RTT and ≥ 2 × max GC pause. A 1 s election timeout on a JVM/Go service with 600 ms GC pauses produces permanent leader churn.
  • Never bootstrap two clusters from the same `--initial-cluster` spec. This is the most common real-world split-brain: stale data directories on two hosts form two independent quorums that both believe they are authoritative. Enforce --initial-cluster-state=existing on joins and pin a unique cluster token.
  • An acceptor that acknowledges before fsync has no safety. If your Paxos/Raft storage layer buffers promises in the page cache, a power loss can produce two accepted values at the same slot. Battery-backed or NVMe-with-FUA, or don't claim consensus.
  • Don't use stale (serializable) reads to drive writes. etcd's WithSerializable() and Spanner's stale reads are for analytics; piping them into a read-modify-write is a lost-update bug waiting for a partition.
  • Fence every external side effect. Leases expire; processes freeze. If the write target cannot enforce a monotonic token, route it through one that can (a queue partition owned by the leader, or a table with a last_token column).
  • Monitor the things that precede outages: etcd_server_leader_changes_seen_total, etcd_disk_wal_fsync_duration_seconds p99, etcd_mvcc_db_total_size_in_bytes vs quota, and applied-vs-committed index lag per follower.
  • Snapshot and compact on a schedule. Unbounded WAL growth is how etcd clusters hit NOSPACE and stop accepting the very writes needed to recover.

Frequently Asked Questions (FAQ)

Is Raft just Multi-Paxos with a leader? Effectively yes, with three deliberate restrictions that make it easier to implement and prove: one leader per term elected by a first-class vote, a contiguous log with no holes, and a total order on terms. Multi-Paxos permits per-slot consensus with gaps and lets implementations use asymmetric quorums (Flexible Paxos) or a stable lease-holding leader to skip Phase 1. Raft gives up that flexibility in exchange for a protocol where every state transition is specified — which is why the ecosystem converged on it for infrastructure metadata.

How do etcd and Spanner actually prevent split-brain during a network partition? Two layers. First, quorum intersection: any two majorities of a 2f+1 group share a member, so two leaders cannot both commit at the same log position, and an acceptor's durably fsynced promise/ballot rejection prevents a second value from being chosen. Second, liveness discipline: etcd's CheckQuorum makes an isolated leader step down within one election timeout, and Spanner's 10-second lease means no other replica may even propose at a higher ballot until the lease provably expires. Neither protects against a zombie process writing to an external system — that requires fencing tokens.

Why does Spanner need TrueTime when etcd does not? etcd only needs *a* total order, and it gets one for free from the Raft log at the leader — commit timestamps never leave the cluster, so no global clock is required. Spanner must assign commit timestamps that are meaningful across independent Paxos groups and consistent with real time (external consistency), so it needs a bounded-uncertainty clock plus commit-wait to guarantee that no transaction can commit before an earlier one is visible anywhere on Earth. That guarantee costs ≈2ε per write and makes availability contingent on TrueTime health — a trade etcd never has to make.