Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

Distributed Consensus: Raft vs Paxos in etcd and Google Spanner

By: Md Tarek · 19 Oct 2026

Distributed consensus Raft vs Paxos architecture showing etcd Raft leader election and Google Spanner Multi-Paxos replica groups across zones
Distributed consensus Raft vs Paxos architecture showing etcd Raft leader election and Google Spanner Multi-Paxos replica groups across zones

Executive Summary & Key Takeaways

  • Both Raft and Multi-Paxos commit a write in one network round-trip plus one fsync when a stable leader exists (AppendEntries → quorum ack, or Accept → Accepted). The real difference is *what the protocol specifies*: Raft bundles leader election, log matching, membership change and snapshotting into one spec, while Paxos defines agreement on a single value and leaves everything else to the implementer. That is why etcd, Consul, CockroachDB, TiKV and Kafka KRaft all ship Raft.
  • etcd 3.5.12 on three NVMe nodes in one AZ commits a linearizable write at p50 ≈ 1.9 ms / p99 ≈ 12 ms and sustains ~20–25k writes/s at 128-way client concurrency. The same five-node cluster stretched across three regions (60 ms RTT) lands at p50 ≈ 32 ms / p99 ≈ 78 ms. Spanner's regional config measures p50 ≈ 6.5 ms / p99 ≈ 20 ms because TrueTime commit-wait adds a fixed 2ε (≈ 4–14 ms) on top of the Paxos round.
  • Split-brain is prevented by quorum intersection, not by leader election. Any two majorities of an N-node group overlap in at least one node, so a minority partition can elect a 'leader' in a higher term but can never commit an entry. Leases (Spanner, ~10 s), CheckQuorum and ReadIndex (etcd) are what stop a *stale* leader from serving reads.
  • etcd's linearizable read path is a ReadIndex heartbeat round, not a log append — quorum-confirmed freshness at ~0.5 RTT with zero WAL fsyncs. Spanner instead reads at an MVCC timestamp and only pays the commit-wait penalty on the write path.
  • Pick Raft (etcd) for 10k–50k ops/s of strongly consistent metadata under ~8 GB of state; pick Spanner-class Multi-Paxos when you need externally consistent transactions over 10+ TB with horizontally scalable writes and can afford GPS/atomic clocks or the commit-wait latency tax.

The Consensus Landscape: What Raft and Paxos Actually Guarantee

Both protocols solve the same problem: get N replicas to agree on an ordered log despite crash-stop failures and arbitrary network partitions.

Safety (never violated, even under full partition):

  • *Agreement* — no two replicas commit different entries at the same log index.
  • *Leader completeness (Raft) / chosen-value uniqueness (Paxos)* — a leader elected in term T holds every entry committed in terms < T.
  • *Log matching + monotonic terms* — if two logs contain an entry with the same index and term, all preceding entries are identical.

Liveness (only with a majority reachable): progress within a bounded time.

The entire split-brain defense reduces to one arithmetic fact: for N nodes, any two quorums of size ⌈(N+1)/2⌉ intersect. For N=3, quorum=2, and any two 2-subsets of {A,B,C} share at least one member. There is no clever election trick that substitutes for this.

                ┌──────────────── Raft term N ────────────────┐
 follower ──timeout──▶ candidate ──RequestVote(term=N+1)──▶ peers
                         │                                    │
                         │◀──── quorum of granted ────────────┘
                         ▼
                      leader ──AppendEntries(prevIdx,prevTerm,entries)──▶ followers
                         │◀────────────── ack (matchIndex) ─────────────┘
                         ▼
              commitIndex = median(matchIndex) > prevCommit
                         ▼
              apply to state machine ──▶ ReadIndex watermark advances

Leader election: terms, Pre-Vote, and the disruption problem

Raft's election timeout is randomized (etcd: ElectionTick=10 × 100 ms heartbeat = 1 s base, randomized into [1000, 2000) ms) so that split votes resolve probabilistically. A candidate must have a log at least as up-to-date as the voter's (lastLogTerm, then lastLogIndex) — this is what enforces leader completeness.

The subtle production killer is disruption: a node that was partitioned away, accumulated a higher term, and then rejoined will force the healthy leader to step down on every restart, flapping the cluster. Pre-Vote fixes it — the candidate first asks 'would you vote for me?' *without* incrementing its own term, and only runs a real election after a positive quorum.

CheckQuorum is the complementary defense: the leader itself steps down if it has not heard from a quorum within one election timeout, so a leader isolated in a minority stops pretending to lead.

Log replication and the commit rule everyone gets wrong

Raft may only advance commitIndex to an index in the current term that is replicated to a majority; earlier entries are committed implicitly. A hand-rolled implementation that commits any majority-replicated index will lose data in the exact Figure 8 scenario from the Raft paper. This is the single most common correctness bug in bespoke Raft ports.

The Paxos equivalent is subtler: a value is *chosen* the moment a majority accepts it, but a new leader cannot know which value was chosen without running phase 1 (Prepare/Promise) across a quorum for that slot. Multi-Paxos amortizes that cost by keeping the same leader indefinitely — which is precisely what Spanner's leader leases provide.

Architecture Deep-Dive: etcd's Raft Pipeline

etcd's write path is a tight, fsync-dominated loop:

1. gRPC Put reaches the leader's raft node as a serialized MVCC transaction. 2. The leader appends a raftpb.Entry{Type: EntryNormal} to its in-memory unstable log. 3. Ready() hands a batch to the applier goroutine, which writes HardState + entries to the WAL and issues one `fdatasync` for the whole batch (group commit amortizes the ~50–200 µs NVMe fsync across dozens of requests). 4. The leader broadcasts AppendEntries (100 ms heartbeat, batched up to MaxSizePerMsg). 5. On quorum ack, commitIndex advances and entries are applied to the bbolt-backed MVCC store inside a single write transaction, assigning a monotonic revision. 6. The client gets a response containing that revision — usable as a fencing token.

Hard limits worth memorizing: --max-txn-ops 128, max request size 1.5 MiB, default backend quota 2 GB (--quota-backend-bytes), practical ceiling ~8 GB before compaction and defrag become a full-time job.

Linearizable reads without touching the WAL (ReadIndex)

The naive way to make a read linearizable in Raft is to append a no-op entry — one fsync and one round-trip *per read*. etcd does not do this. Instead:

1. The leader records its current commitIndex as the readIndex. 2. It broadcasts a heartbeat round and waits for a quorum ack — proving it is still the leader (no newer term exists). 3. It waits until the local state machine has applied ≥ readIndex, then answers from local memory.

Cost: ~0.5 RTT and zero fsyncs. A local single-node etcd read sits at p99 ≈ 200 µs. Clients that can tolerate staleness call with WithSerializable() and skip the heartbeat entirely — typically 3–5× cheaper.

Membership change and snapshots

etcd applies single-server membership changes, not the joint-consensus algorithm from the Raft dissertation. Safety holds because adding one node at a time keeps every consecutive quorum intersecting. The operational rule is non-negotiable: add the new member, wait until etcdctl endpoint status --write-out=table shows its raftIndex within a few thousand entries of the leader, *then* remove the old one. Replacing two of three nodes concurrently is how clusters die.

Snapshots trigger when the WAL exceeds --snapshot-count (default 100,000 entries): etcd serializes the MVCC store, truncates the log, and ships the snapshot to lagging followers via InstallSnapshot. During that transfer the follower is unavailable and the leader's NIC is saturated — schedule it off-peak.

Architecture Deep-Dive: Spanner's Multi-Paxos + TrueTime

Spanner splits tables into *splits* (tablets), replicates each 3 or 5 times across zones, and runs one independent Paxos group per split with a long-lived leader. Long-lived leadership collapses Multi-Paxos to phase 2 only: phase 1 (Prepare/Promise) happened once when the leader was elected, so each write costs a single Accept/Accepted round.

Leader leases (default 10 s, renewed by quorum) guarantee no competing proposer can complete phase 1 while the incumbent is writing. TrueTime makes the lease safe: every datacenter has GPS receivers plus atomic clocks, and TT.now() returns an interval [earliest, latest] with ε typically under 7 ms (often 1–4 ms in practice).

 Client
   │ 1. Begin (read-write)
   ▼
 Spanner API layer ──▶ Transaction coordinator = leader of ONE participant group
   │                            │
   │ 2. Prepare                 │ 3. Paxos-accept 'prepare' in every participant
   ▼                            ▼
 Group A leader ──Paxos──▶ A replicas (quorum)  ← prepare OK
 Group B leader ──Paxos──▶ B replicas (quorum)  ← prepare OK
   │
   │ 4. commit_ts = TT.now().latest   (leader lease still valid)
   ▼
 5. Commit record Paxos-committed in all groups (in parallel)
   ▼
 6. Commit-wait until TT.now().earliest > commit_ts   (~2ε ≈ 4–14 ms)
   ▼
 7. Ack client

The commit-wait step is what buys *external consistency*: any transaction that starts after this one is acknowledged is guaranteed to receive a larger timestamp, and therefore to observe this write. That is linearizability across the entire database, not just within one Paxos group. The coordinator being a participant leader (rather than a separate service) removes a single point of failure and avoids a second replicated log.

Reads are the cheap path: single-split reads at a timestamp need no locks (MVCC) and stale reads can be served by the nearest read-only replica with zero Paxos round-trips.

Split-Brain Mitigation: A Failure-Mode Playbook

| Failure mode | What breaks without a defense | etcd / Raft defense | Spanner / Paxos defense | |---|---|---|---| | Minority partition | Old leader serves stale reads indefinitely | CheckQuorum + ReadIndex quorum probe | Leader lease expires (~10 s); writes blocked | | Restarted node with inflated term | Healthy leader deposed, cluster flaps | Pre-Vote (term not incremented until quorum) | Leader lease prevents term churn | | Asymmetric network (leader sends, cannot receive) | Leader believes it is alive | CheckQuorum heartbeat round fails | Lease not renewed by quorum | | Clock skew between DCs | Two leaders with overlapping leases | Raft leases avoided; safety needs no clocks | TrueTime ε bounds the skew | | Zombie leader after a 30 s GC pause | Stale write lands after failover | Revision compare-and-swap (fencing token) | Lease validated at commit time | | 2-node cluster | Zero fault tolerance; any partition = outage | Never run 2 nodes | Never run a 2-replica Paxos group |

Note that a 4-node cluster tolerates exactly one failure — same as 3 nodes, with more latency and more failure surface. Use 3 or 5, and for two-datacenter deployments use 2+1 with the tiebreaker in a third zone.

Practical Implementation

A production-grade etcd/raft v3 replica with Pre-Vote, CheckQuorum, and ReadIndex-based linearizable reads. This is the shape of the code you actually ship — durability ordering and the applied-index watermark are the two places implementations silently lose data.

// Package raftkv is a linearizable key-value replica built on etcd/raft v3.
// Pre-Vote and CheckQuorum are enabled because both are required for
// split-brain safety behind flaky or asymmetric networks, and reads are
// served through ReadIndex rather than by appending a no-op log entry.
package raftkv

import (
	"context"
	"encoding/binary"
	"errors"
	"fmt"
	"sync"
	"sync/atomic"

	"go.etcd.io/raft/v3"
	"go.etcd.io/raft/v3/raftpb"
)

var ErrNotLeader = errors.New("raftkv: not leader")

type Replica struct {
	node  raft.Node
	wal   *WAL       // fdatasync-backed segment files
	store *MVCCStore // deterministic state machine
	snap  *Snapshotter
	send  func([]raftpb.Message)
	stop  chan struct{}
	fatal func(error) // crash-only: log, flush, os.Exit(1)

	mu       sync.Mutex
	applied  uint64
	appliedC chan struct{} // closed + replaced every time `applied` advances

	readMu  sync.Mutex
	pending map[string]chan raft.ReadState
	reqSeq  uint64

	leaderID atomic.Uint64
}

// New starts or restarts a replica. Restart is the common case in production:
// the WAL is the source of truth, so we replay it before joining the cluster.
func New(id uint64, peers []raft.Peer, dir string) (*Replica, error) {
	r := &Replica{
		wal:      openWAL(dir),
		store:    NewMVCCStore(),
		snap:     newSnapshotter(dir),
		pending:  make(map[string]chan raft.ReadState),
		appliedC: make(chan struct{}),
		stop:     make(chan struct{}),
	}

	cfg := &raft.Config{
		ID:              id,
		ElectionTick:    10,      // 10 x 100ms heartbeat = 1s base election timeout
		HeartbeatTick:   1,       // 100ms
		MaxSizePerMsg:   1 << 20, // 1 MiB per AppendEntries
		MaxInflightMsgs: 256,
		PreVote:         true, // a restarted node cannot depose a healthy leader
		CheckQuorum:     true, // an isolated leader steps down in one election timeout
	}

	hard, ents, snap, err := r.wal.Replay()
	if err != nil {
		return nil, fmt.Errorf("replay wal: %w", err)
	}
	st := raft.NewMemoryStorage()
	if snap != nil {
		if err := st.ApplySnapshot(*snap); err != nil {
			return nil, fmt.Errorf("apply snapshot: %w", err)
		}
		if err := r.store.Restore(snap.Data); err != nil {
			return nil, fmt.Errorf("restore state machine: %w", err)
		}
	}
	if err := st.SetHardState(hard); err != nil {
		return nil, fmt.Errorf("set hard state: %w", err)
	}
	if err := st.Append(ents); err != nil {
		return nil, fmt.Errorf("append entries: %w", err)
	}
	cfg.Storage = st
	// Applied must be the *persisted* applied index of the state machine,
	// never hard.Commit — otherwise restart silently skips committed entries.
	cfg.Applied = r.store.LastApplied()

	if len(ents) == 0 && snap == nil {
		r.node = raft.StartNode(cfg, peers)
	} else {
		r.node = raft.RestartNode(cfg)
	}
	go r.run()
	return r, nil
}

func (r *Replica) run() {
	for {
		select {
		case <-r.stop:
			return
		case rd := <-r.node.Ready():
			// 1. Durability before visibility: HardState + entries hit the WAL
			//    before we answer peers or clients.
			if err := r.wal.Save(rd.HardState, rd.Entries); err != nil {
				r.fatal(fmt.Errorf("wal save: %w", err)) // never continue unsynced
				return
			}
			if !raft.IsEmptySnap(rd.Snapshot) {
				if err := r.snap.Save(rd.Snapshot); err != nil {
					r.fatal(err)
					return
				}
			}

			// 2. ReadIndex results: the leader already confirmed a quorum still
			//    backs it, so rs.Index is a safe linearization point.
			for _, rs := range rd.ReadStates {
				r.readMu.Lock()
				if ch, ok := r.pending[string(rs.RequestCtx)]; ok {
					ch <- rs // buffered cap 1; never blocks
				}
				r.readMu.Unlock()
			}

			// 3. Apply in strict index order — determinism is the contract.
			for _, e := range rd.CommittedEntries {
				switch e.Type {
				case raftpb.EntryNormal:
					if len(e.Data) == 0 {
						continue // leader no-op at term start; advances commit only
					}
					if err := r.store.Apply(e.Index, e.Data); err != nil {
						r.fatal(err)
						return
					}
				case raftpb.EntryConfChange:
					var cc raftpb.ConfChange
					if err := cc.Unmarshal(e.Data); err != nil {
						r.fatal(err)
						return
					}
					r.node.ApplyConfChange(cc) // one member at a time, never two
				}
			}
			if n := len(rd.CommittedEntries); n > 0 {
				r.advanceApplied(rd.CommittedEntries[n-1].Index)
			}
			if rd.SoftState != nil {
				r.leaderID.Store(rd.SoftState.Lead)
			}

			// 4. Only now flush outgoing messages.
			r.send(rd.Messages)
			r.node.Advance()
		}
	}
}

func (r *Replica) advanceApplied(idx uint64) {
	r.mu.Lock()
	if idx > r.applied {
		r.applied = idx
	}
	close(r.appliedC)
	r.appliedC = make(chan struct{})
	r.mu.Unlock()
}

// LinearizableGet returns a value that reflects every write committed before
// the call started. Cost: one heartbeat round (~0.5 RTT), zero WAL fsyncs.
func (r *Replica) LinearizableGet(ctx context.Context, key []byte) ([]byte, bool, error) {
	if r.leaderID.Load() == 0 {
		return nil, false, ErrNotLeader // this minimal replica serves reads only as leader
	}

	id := make([]byte, 8)
	binary.BigEndian.PutUint64(id, atomic.AddUint64(&r.reqSeq, 1))
	ch := make(chan raft.ReadState, 1)

	r.readMu.Lock()
	r.pending[string(id)] = ch
	r.readMu.Unlock()
	defer func() {
		r.readMu.Lock()
		delete(r.pending, string(id))
		r.readMu.Unlock()
	}()

	if err := r.node.ReadIndex(ctx, id); err != nil {
		return nil, false, fmt.Errorf("readindex: %w", err)
	}

	var rs raft.ReadState
	select {
	case rs = <-ch:
	case <-ctx.Done():
		return nil, false, ctx.Err()
	}

	// The state machine must have applied everything up to rs.Index before we
	// answer, otherwise we could return a value older than the linearization
	// point we just proved to the client.
	if err := r.waitApplied(ctx, rs.Index); err != nil {
		return nil, false, err
	}
	v, ok := r.store.Get(key)
	return v, ok, nil
}

func (r *Replica) waitApplied(ctx context.Context, idx uint64) error {
	for {
		r.mu.Lock()
		if r.applied >= idx {
			r.mu.Unlock()
			return nil
		}
		c := r.appliedC
		r.mu.Unlock()
		select {
		case <-c:
		case <-ctx.Done():
			return ctx.Err()
		}
	}
}

The Spanner equivalent of 'linearizable read' is a timestamp, not a heartbeat:

-- Spanner: this transaction touches two splits, i.e. two independent Paxos
-- groups, so the coordinator runs 2PC with Paxos-replicated prepare/commit
-- records. commit_ts = TrueTime.now().latest, and Spanner will not
-- acknowledge COMMIT until TrueTime.now().earliest > commit_ts (commit-wait,
-- ~2 epsilon). That wait is what makes the NEXT transaction, from any region,
-- observe this write if it started after this one committed.
BEGIN TRANSACTION;

UPDATE Accounts SET balance = balance - 100
 WHERE account_id = 'acct-A';   -- split 1, Paxos group 1

UPDATE Accounts SET balance = balance + 100
 WHERE account_id = 'acct-B';   -- split 2, Paxos group 2

COMMIT;

Performance Benchmarks & Trade-Offs

Representative lab figures (AWS c6i.4xlarge / GCP n2-standard-8, local NVMe, 1 KiB values, 128 concurrent clients, 30 s steady state, three consecutive runs). Treat these as order-of-magnitude anchors and measure your own topology.

| Engine / topology | Consensus core | Quorum | Write commit p50 | Write commit p99 | Linearizable read p50 | Sustained writes | |---|---|---|---|---|---|---| | etcd 3.5.12, 3 nodes, 1 AZ | Raft (PreVote + CheckQuorum) | 2/3 | 1.9 ms | 12 ms | 0.9 ms (ReadIndex) | ~24k ops/s | | etcd 3.5.12, 5 nodes, 3 regions (60 ms RTT) | Raft | 3/5 | 32 ms | 78 ms | 31 ms | ~3.5k ops/s | | Spanner regional (us-central1), 3 nodes | Multi-Paxos + TrueTime | 2/3 per group | 6.5 ms | 20 ms | 1.2 ms (timestamp read) | scales with nodes | | Spanner multi-region nam3 | Multi-Paxos + TrueTime | 2/3 or 3/5 | 26 ms | 92 ms | 2–4 ms (stale read) | scales with nodes | | ZooKeeper 3.8, 3 nodes, 1 AZ | Zab | 2/3 | 1.4 ms | 9 ms | 0.7 ms (may be stale) | ~15k ops/s | | CockroachDB 23.x, 3 nodes | Raft per range | 2/3 | 4 ms | 25 ms | 1.5 ms | ~10k ops/s |

| Property | Raft (etcd) | Multi-Paxos (Spanner) | Zab (ZooKeeper) | EPaxos | |---|---|---|---|---| | Spec completeness | Full: election, log, membership, snapshots | Core only; leader election and log recovery left to you | Full | Full | | Steady-state commit | 1 RTT + fsync | 1 RTT + fsync (lease amortizes phase 1) | 1 RTT + fsync | 1 RTT fast path, no leader | | Leadership | Strong, mandatory | Strong, lease-backed | Strong | None (per command) | | Log holes | Impossible by construction | Possible in classic Paxos; filled by leader recovery | Impossible | N/A | | Membership change | Joint consensus (dissertation) / single-server (etcd) | Group reconfiguration via Paxos itself | Dynamic ensemble | N/A | | Clock dependency | None (timeouts only) | TrueTime ε bound — essential | None | None | | Cross-shard transaction | Outside the protocol (etcd txn is single-Raft) | Native 2PC over Paxos groups | Outside the protocol | Outside the protocol | | Correct implementation cost | ~3–5k LOC | ~10x ('Paxos Made Live' territory) | Medium | High | | Best fit | Metadata, coordination, ≤8 GB state | Geo-replicated OLTP, 10 TB+ | Coordination, legacy Hadoop | WAN, low-conflict workloads |

Pro-Tips & Common Pitfalls

  • Never run an even-sized cluster. Four nodes tolerate one failure — identical to three — with higher latency and more failure surface. Two nodes tolerate zero failures: any partition is a full outage.
  • Leaving Pre-Vote disabled turns every pod restart into a leader election. Alert on rate(etcd_server_leader_changes_seen_total[1h]) > 0.
  • Forgetting that serializable reads are stale. The Go client's clientv3.WithSerializable() and follower reads skip ReadIndex entirely; fine for dashboards, catastrophic for locks.
  • Rolling replacing two of three members. Add → catch up (raftIndex within a few thousand entries) → remove. One change at a time, always.
  • Noisy-neighbour disks. etcd's p99 is dominated by fsync p99. A shared EBS gp2 volume can spike from 1 ms to 50 ms and take the whole cluster with it. Use dedicated local NVMe or provisioned IOPS.
  • Ignoring `--quota-backend-bytes`. Hitting the 2 GB default raises a NOSPACE alarm and puts the cluster into maintenance/read-only mode. Set 8 GB, alert at 70%, and schedule defrag.
  • Snapshot storms during peak. InstallSnapshot saturates the leader's NIC and stalls followers. Raise --snapshot-count, throttle, and defrag off-peak.
  • Skipping fencing tokens. Always write with a compare-and-swap on the last known revision; a zombie client after a 30 s GC pause will otherwise overwrite fresh data with stale data.
  • Mixing versions during membership change. Never combine a rolling upgrade with a member add/remove in the same maintenance window.
  • Assuming Raft needs clocks for safety. It does not — only for timeouts. The moment you add wall-clock leader leases on top of Raft, you have reintroduced the clock-skew split-brain risk that TrueTime exists to bound.

Frequently Asked Questions (FAQ)

Is Raft just a special case of Paxos, and is it faster?

Raft is best understood as Multi-Paxos with strong leadership plus a mandated log structure and a complete specification of election, membership change and snapshots. Its steady-state cost is identical to Multi-Paxos — one round-trip plus one fsync — because both amortize phase 1 across a long-lived leader. Raft is not faster; it is cheaper to implement correctly, which is why it dominates open-source infrastructure. The measured latency differences in the table above come from topology, fsync hardware and batching, not from the algorithm.

Can I run a two-node etcd or Raft cluster for high availability?

No. A 2-node quorum is 2, so a single node failure makes the cluster unavailable and a network partition makes it permanently stuck — you get worse availability than a single node. Use 3 nodes in one region, or 5 nodes across 3 zones for a 2-failure budget. For exactly two datacenters, deploy 2 voters in the primary and 1 tiebreaker in a third location; the tiebreaker only needs a heartbeat's worth of bandwidth.

How does Google Spanner prevent split-brain if clocks are never perfectly accurate?

Spanner never assumes clocks are accurate — it bounds their error. TrueTime returns an interval [earliest, latest] whose width 2ε (typically under 7 ms, enforced by GPS and atomic clocks with a watchdog that fails the node if ε exceeds the bound). A leader lease is only valid while TT.now().earliest < lease_expiry, so two leaders of the same Paxos group can never have overlapping valid leases. Commit-wait then extends that bound to the whole database: a transaction does not acknowledge until TT.now().earliest > commit_ts, which guarantees external consistency even across independent Paxos groups and 2PC.