Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

Distributed Consensus Raft vs Paxos: etcd vs Spanner Deep Dive

By: Md Tarek · 21 Oct 2026

Distributed Consensus Raft vs Paxos architecture comparison
Distributed Consensus Raft vs Paxos architecture comparison

Executive Summary & Key Takeaways

  • Raft (etcd) uses a strong leader and randomized election timeouts; etcd defaults to a 100ms heartbeat and 1000ms election timeout, completing leader elections in 1–2s, while Multi-Paxos (Spanner) uses leader leases (10s default) and TrueTime to achieve external consistency with ~7ms single-region write latency.
  • Raft enforces the Log Matching Property and commits after a majority quorum fsyncs to disk; etcd p99 write latency is 2–10ms on local SSD and 50–150ms cross-region, whereas Spanner's Paxos groups achieve ~7ms single-region and ~100ms multi-region writes.
  • Split-brain is prevented in Raft by majority quorum, term numbers, and CheckQuorum (leader steps down without majority); in Spanner, Paxos leader leases plus TrueTime-bound lease expiration and fencing tokens ensure only one valid leader can commit.
  • Operational complexity: Raft is easier to implement and debug (etcdctl, metrics, learner nodes), while Paxos variants require careful lease management and clock synchronization; Spanner's TrueTime requires GPS/atomic clocks, adding infrastructure cost.
  • For most self-hosted systems, etcd's Raft is the pragmatic choice; for globally distributed, externally consistent databases, Spanner's Paxos+TrueTime model provides stronger guarantees at the cost of specialized hardware and operational overhead.

Leader Election: Raft's Randomized Timeouts vs Paxos Leases

Raft leader election is triggered by a follower missing heartbeats. The follower transitions to candidate, increments its term, votes for itself, and sends RequestVote RPCs to all peers. A candidate wins if it receives votes from a majority (N/2+1). To avoid split votes, Raft uses randomized election timeouts (etcd: 1000–2000ms). Once elected, the leader sends periodic AppendEntries (heartbeats) every 100ms in etcd. If a leader cannot reach a majority, CheckQuorum forces it to step down, preventing a stale leader from serving linearizable reads.

Paxos leader election is more flexible. In Multi-Paxos, a leader is established by running a Prepare phase with a higher proposal number; once a majority promises, the leader can propose values without re-running Prepare for each write. Spanner uses Paxos groups, each with a leader that holds a lease. The lease is granted for 10 seconds and renewed via Paxos; TrueTime is used to ensure that lease expiration is globally consistent. A replica can become leader only if it receives a majority of votes and its lease is valid. The leader lease allows reads to be served locally without a round trip, reducing read latency.

ASCII flow for Raft election:

Follower --(timeout)--> Candidate --(RequestVote majority)--> Leader
   ^                                                         |
   |------------------(AppendEntries heartbeat)--------------|

ASCII flow for Spanner Paxos lease:

Replica A (leader) --(lease 10s)--> TrueTime-validated lease
   |-- Accept (parallel) --> Replicas B,C,D,E
   |-- majority ack --> commit with timestamp

Log Replication: Raft's AppendEntries vs Paxos's Accept/Prepare

Raft log replication is leader-driven: the leader appends a new entry to its log, sends AppendEntries RPCs in parallel to followers, and waits for a majority to acknowledge. Followers persist the entry to their WAL (fsync) before acknowledging. Once a majority acknowledges, the leader commits the entry, applies it to the state machine, and responds to the client. etcd batches proposals and uses a single WAL fsync per batch; the backend (bbolt) commits every 100ms or when the batch fills, yielding high throughput. The Log Matching Property ensures that if two logs contain an entry with the same index and term, the logs are identical up to that index.

Paxos log replication in Spanner uses Multi-Paxos. The leader proposes a value with a proposal number. If the leader has a valid lease, it can skip the Prepare phase and send Accept messages directly to acceptors. A value is chosen when a majority accepts it. Spanner then assigns a commit timestamp using TrueTime and applies the write to the tablet. Because the leader lease is valid, Spanner avoids repeated Prepare rounds for each write, achieving high throughput. The Paxos group typically has 5 replicas across zones; a write is committed when 3 replicas accept.

ASCII flow for Raft replication:

Client -> Leader -> AppendEntries (parallel) -> Followers (WAL fsync)
Leader <- Ack majority <- Followers
Leader -> Commit -> Apply -> Response

ASCII flow for Spanner Paxos replication:

Client -> Spanner API -> Paxos Leader -> Accept (parallel) -> Acceptors
Leader <- Majority Ack <- Acceptors
Leader -> Commit with TrueTime timestamp -> Apply -> Response

Split-Brain Mitigation: Quorums, Leases, and Fencing

Raft prevents split-brain through majority quorum and term numbers. Only a candidate with a majority of votes can become leader in a given term. A partitioned leader cannot commit new entries because it cannot get a majority of acknowledgments. etcd enables CheckQuorum by default: if a leader cannot contact a majority within the election timeout, it steps down to follower. For reads, etcd provides linearizable reads via ReadIndex, which confirms leadership with a quorum before serving the read. This prevents a stale leader from returning stale data. However, serializable reads (stale reads) can be served from a partitioned leader if explicitly requested.

Spanner mitigates split-brain using Paxos leader leases, TrueTime, and fencing tokens. A leader must hold a valid lease to serve writes. The lease is renewed via Paxos and is bounded by TrueTime: the leader waits out the TrueTime uncertainty (epsilon) before assuming the lease is valid. If a leader loses its lease, it cannot commit new writes. Fencing tokens (Paxos proposal numbers) ensure that any write from an old leader with a lower proposal number is rejected by acceptors. This prevents a paused or partitioned leader from corrupting data. Spanner's external consistency relies on TrueTime commit-wait: after assigning a commit timestamp, Spanner waits until TrueTime guarantees that the timestamp is in the past before returning to the client.

Practical Implementation

The following Go code demonstrates production-grade leader election with etcd's Raft-based concurrency package. It creates a session with a 10-second TTL, campaigns for leadership, and handles errors and context cancellation. Use this pattern for distributed locks, singleton workers, or controller elections.

package main

import (
    "context"
    "fmt"
    "log"
    "time"

    clientv3 "go.etcd.io/etcd/client/v3"
    "go.etcd.io/etcd/client/v3/concurrency"
)

func main() {
    // Configure etcd client with timeouts.
    cli, err := clientv3.New(clientv3.Config{
        Endpoints:   []string{"https://etcd1:2379", "https://etcd2:2379", "https://etcd3:2379"},
        DialTimeout: 5 * time.Second,
        // TLS config omitted for brevity; use TLS in production.
    })
    if err != nil {
        log.Fatalf("failed to connect to etcd: %v", err)
    }
    defer cli.Close()

    // Create a session with a 10-second TTL. The session keeps the lease alive.
    session, err := concurrency.NewSession(cli, concurrency.WithTTL(10))
    if err != nil {
        log.Fatalf("failed to create session: %v", err)
    }
    defer session.Close()

    // Create an election on the "/myapp/leader" key.
    election := concurrency.NewElection(session, "/myapp/leader")

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    // Campaign for leadership. This blocks until elected or ctx canceled.
    if err := election.Campaign(ctx, "node-1"); err != nil {
        log.Fatalf("campaign failed: %v", err)
    }

    // Once elected, perform leader duties.
    log.Println("became leader")
    for {
        select {
        case <-ctx.Done():
            log.Println("context canceled, resigning leadership")
            // Resign to allow a new leader to be elected quickly.
            if err := election.Resign(context.Background()); err != nil {
                log.Printf("failed to resign: %v", err)
            }
            return
        case <-session.Done():
            log.Println("session expired, lost leadership")
            return
        case <-time.After(5 * time.Second):
            // Simulate leader work.
            fmt.Println("performing leader-only task")
        }
    }
}

For Spanner, a write would use a read-write transaction with a TrueTime commit timestamp; however, that requires Google Cloud Spanner client libraries and IAM credentials. The etcd example above is self-contained and demonstrates Raft leader election in production.

Performance Benchmarks & Trade-Offs

| Metric | Raft (etcd v3.5) | Multi-Paxos (Spanner) | |--------|------------------|------------------------| | Leader election time | 1–2s (default 1000ms election timeout, randomized) | ~10s lease duration; election via Paxos with majority | | Write latency (single-region, p99) | 2–10ms (local SSD, 3-node cluster) | ~7ms (single-region, 3 replicas) | | Write latency (cross-region, p99) | 50–150ms (depends on RTT) | ~100ms (multi-region, 5 replicas) | | Linearizable read latency | 1–5ms (ReadIndex or lease read) | 1–5ms (leader lease, local read) | | Quorum size | N/2+1 | N/2+1 | | Network messages per write | 2*(N-1) (AppendEntries + Ack) | 2*(N-1) (Accept + Ack; Prepare skipped with lease) | | Operational complexity | Low–medium (etcdctl, metrics, learner nodes) | High (TrueTime, lease management, global deployment) | | Clock requirements | None (randomized timeouts only) | TrueTime API: GPS/atomic clocks, epsilon bounded | | Split-brain mitigation | Majority quorum + term numbers + CheckQuorum | Leader leases + TrueTime + fencing tokens | | Typical use case | Kubernetes, service discovery, config management | Global financial ledgers, inventory, ads |

Pro-Tips & Common Pitfalls

  • Use odd cluster sizes (3, 5, 7). Even sizes increase split-vote probability and do not improve fault tolerance; a 4-node Raft cluster tolerates only 1 failure, same as 3 nodes, but with more overhead.
  • Tune election timeouts for your network. Too short causes election storms under packet loss; too long increases failover time. etcd recommends 100ms heartbeat and 1000ms election timeout for LAN, but WAN may need 2000–5000ms.
  • Never rely on leader election alone as a distributed lock. A paused leader (e.g., GC pause) can still hold the lock after its lease expires. Use fencing tokens (e.g., etcd revision or Spanner proposal number) to invalidate stale leaders.
  • Enable CheckQuorum and ReadIndex for linearizable reads. Without CheckQuorum, a partitioned leader may continue to serve stale reads. etcd's --experimental-enable-lease-check and ReadIndex ensure correctness.
  • Monitor Raft term changes and election counts. Frequent elections indicate network partitions, disk latency, or CPU starvation. Export etcd metrics (etcd_server_leader_changes_seen_total) and alert on spikes.
  • Fsync is non-negotiable. etcd's WAL fsync must complete before acknowledging writes. Using non-fsync disks or network storage with high latency can cause leader instability and data loss.
  • In Spanner, respect TrueTime epsilon. Never assume commit timestamps are immediately valid; Spanner waits out uncertainty. Client-side caching of timestamps can violate external consistency.
  • Test split-brain with network partitions. Use tc netem or chaos engineering to simulate partitions and verify that only one leader can commit. Verify that stale leaders step down and that clients retry with idempotency keys.
  • Avoid cross-region consensus for latency-sensitive workloads. Raft/Paxos quorum writes require at least one cross-region round trip. Use local quorums or regional leaders with async replication if strict consistency is not required.

Frequently Asked Questions (FAQ)

Q: What is the main difference between Raft and Paxos? A: Raft is a consensus algorithm designed for understandability and implementability, with a strong leader, log replication, and randomized leader election. Paxos is a family of protocols; Multi-Paxos is used in Google Spanner with leader leases and TrueTime. Raft enforces a single leader per term and log matching, while Multi-Paxos allows a stable leader to skip the Prepare phase after a lease is established. Raft is easier to implement; Paxos is more flexible but harder to reason about.

Q: How does etcd prevent split-brain? A: etcd uses Raft's majority quorum and term numbers. A candidate needs a majority of votes to become leader, and a leader cannot commit without a majority of acknowledgments. etcd enables CheckQuorum by default, so a leader that cannot reach a majority steps down after the election timeout. For linearizable reads, etcd uses ReadIndex to confirm leadership with a quorum before serving the read, preventing stale reads from a partitioned leader.

Q: Why does Google Spanner use Paxos instead of Raft? A: Spanner was designed before Raft was published and uses Paxos as a mature, flexible foundation for Multi-Paxos with leader leases. Spanner's TrueTime API provides globally consistent timestamps, which complements Paxos leader leases and enables external consistency. While Raft could be adapted for similar guarantees, Spanner's architecture leverages Paxos groups, leases, and TrueTime to achieve high throughput and global consistency. Raft's strong leader model is simpler, but Paxos's flexibility suited Spanner's multi-group replication design.