Multi-Region Active-Active Database Architecture at Scale
By: Md Tarek · 19 Oct 2026
Two regions accepting writes for the same key in the same millisecond is not a database configuration problem — it is a distributed-systems problem with a physics-imposed price. This note is the engineering brief I wanted before moving a 40 TB order-management workload from a single us-east-1 primary to a three-region active-active deployment: the latency arithmetic that decides your topology before you write a line of code, the CRDT families that actually survive production, the failover mechanics that keep a partition from becoming a data-loss incident, and the coordinator code that ties it together.
Executive Summary & Key Takeaways
- Latency is bounded by fibre physics, not by your cloud provider. Light travels at ~200,000 km/s in silica, and routed paths run 1.2–1.5× great-circle distance. New York ↔ London has a hard RTT floor of ~68 ms (measured p50 72 ms, p99 78 ms); any synchronous cross-Atlantic quorum commit costs a minimum of one full RTT *before* your application code executes. A 250 ms p99 write SLO is achievable across the Atlantic and impossible across the Pacific without async replication.
- Asymmetric pair selection is the single highest-leverage decision. Chaining
us-east-1 ↔ eu-west-1(78 ms p99) and rejectingus-east-1 ↔ ap-southeast-1(238 ms p99) for synchronous quorums reduces projected write p99 from 294 ms to 136 ms with zero application changes — 2.2× headroom recovered purely from topology. - Home-region write affinity eliminates ~80% of conflicts at a fraction of the complexity. Routing each key's writes to one owning region gives 9 ms local write p99 and 82 ms for the cross-region minority, with no CRDT convergence logic on the hot path. Reserve full symmetric multi-write for genuinely offline-tolerant data (presence, carts, annotations, counters).
- **CRDTs remove conflict *resolution*, not conflict *cost*.** Delta-state CRDTs shrink replication payload from O(state) to O(Δ), but OR-Set tag sets and sequence position identifiers grow O(operations) until you implement causal-stability GC. Tombstone TTL must exceed p99.9 replication lag by a factor of 10; anything less resurrects deleted rows after a partition heals.
- Failover RTO is dominated by fencing, not by leader election. Raft promotes a leader in 2–5 s, but a *safe* automated failover needs a witness region plus monotonic epochs; production numbers land at 10–25 s for quorum active-active and 60–120 s for warm standby. Split-brain windows of 40–60 s are common in naive setups and always surface as duplicate writes after healing.
Cross-Region Latency Budgets: The Physics That Bounds Every Design
The speed of light is the only constant in this discipline you cannot negotiate with. In fused silica (n ≈ 1.468) light propagates at ~204,000 km/s; real routed paths — submarine cable landings, metro fibre, DWDM regeneration — add 20–50% over great-circle distance. That gives a useful back-of-envelope: RTT (ms) ≈ great-circle km ÷ 100, then add 10–25% for path stretch and fabric overhead.
Region pair Great-circle Routed path RTT p50 RTT p99 RTT p99.9 us-east-1 <-> us-west-2 3,940 km ~5,200 km 62 ms 68 ms 95 ms us-east-1 <-> eu-west-1 5,570 km ~7,000 km 72 ms 78 ms 112 ms eu-west-1 <-> ap-southeast-1 9,900 km ~12,000 km 158 ms 171 ms 240 ms us-east-1 <-> ap-southeast-1 15,300 km ~19,000 km 224 ms 238 ms 310 ms
Note the tail shape: p99.9 runs 1.5–2.0× p50 and *nothing* in your stack fixes it. BGP reconvergence after a backbone event can add 30–180 s of elevated latency and packet loss — a window that no application-level retry policy survives gracefully unless you have regional write isolation.
The write-path budget waterfall
Build the budget top-down from the user-facing SLO, never bottom-up from component SLOs (component budgets always sum to more than the user gets).
SLO: p99 write latency for a cross-region-consistent write = 250 ms +---------------------------------------------------------------+ | TLS resumption + edge PoP 18 ms ## | | Edge -> origin region (anycast steering) 12 ms # | | AuthN / policy / hot cache lookup 8 ms # | | Coordinator serialize + idempotency dedupe 3 ms | | | Intra-region 3-AZ Raft commit (p99) 3 ms | | | Cross-region quorum ack (1 RTT us-east-1/eu-w-1) 78 ms ########| | Response back through edge 30 ms ### | +---------------------------------------------------------------+ | TOTAL 152 ms (61%) | +---------------------------------------------------------------+
Three rules fall out of that arithmetic:
1. The 80 ms rule. Never place a synchronously coupled replica more than ~80 ms RTT from the quorum. Beyond that, a single congested link converts a healthy write path into a tail-latency incident, because p99 RTT at 238 ms leaves almost no headroom in a 250 ms budget. 2. Cold connections pay double. A TLS 1.3 full handshake is 1-RTT. A cold client hitting a cross-region write pays 2× RTT (handshake + request). 0-RTT resumption removes the handshake but is replayable — meaning every non-idempotent mutation must be gated by an idempotency key, or you will double-charge a customer the first time a retry storm hits. 3. Read paths get a different budget. A regional read served from a local replica with LWW-cached projection is 2.5–3 ms p99. Do not let an ACID read quorum leak into the read path — R + W > N with N=3, R=2, W=2 means two RTTs for a cross-region read (156 ms p99), which is 50× worse than reading local.
Production war story. A Cassandra cluster was migrated from single-region to multi-region with the replication factor bumped to {'us-east-1': 3, 'eu-west-1': 3} and *no* changes to the client consistency level. Applications kept issuing QUORUM, which in a NetworkTopologyStrategy keyspace spanning two DCs requires 4 of 6 replicas — including at least one cross-Atlantic round trip. Write p99 went from 9 ms to 210 ms overnight and p99.9 crossed 900 ms during the nightly compaction window. The fix was one line — LOCAL_QUORUM — and it took four days to find because the latency regression looked like a JVM GC problem. Consistency level is a latency decision; treat it as such in code review.
Conflict-Free Replicated Data Types That Survive Production
A CRDT is a replicated data type whose merge function ⊔ is commutative, associative, and idempotent. Those three properties mean replicas converge regardless of delivery order, duplication, or retransmission — which is precisely what an unreliable cross-region link delivers.
Three families matter in practice:
- State-based (CvRDT) — replicas exchange full state and call
merge(local, remote). Trivially correct, bandwidth O(state). - Operation-based (CmRDT) — replicas exchange operations, requiring exactly-once causal broadcast. Cheap on the wire, fragile on the delivery layer.
- Delta-state (δ-CRDT) — the 2018 Almeida/Shoker/Baquero result: ship join-irreducible deltas, merge them into local state, and periodically compact. Bandwidth drops from O(state) to O(Δ) with the same convergence guarantees. This is what you should build in 2025; full-state exchange is a fallback for cold-start and repair only.
Ordering: HLCs beat vector clocks for this job
Most production CRDTs need a *total* order to make LWW-style merges deterministic. Hybrid Logical Clocks give you that with O(1) metadata instead of the O(N) of a vector clock: wall = max(physical, prev.wall), a counter to break same-millisecond ties, and the node ID as a final deterministic tiebreak. The counter is bounded by physical clock drift, so HLC stays within MAX_CLOCK_DRIFT_MS of real time — which is exactly the guarantee you need to reason about lease safety later.
Vector clocks tell you *whether* two writes are concurrent, which is stronger, but they cost O(N × replicas) bytes per record and require application-level sibling resolution. Use dotted version vectors only for the subset of OR-Sets that need causal-stability GC; do not put them on every row.
Production war story. An inventory service used an LWW-Register keyed on (sku, warehouse) with plain wall-clock timestamps. During a routine NTP step on one region, the clock jumped *backwards* 1.4 s. For 1.4 s of logical time, that region's writes carried timestamps lower than already-applied writes from the peer, so merge silently discarded 2,300 decrements. The oversell was detected 3 days later in reconciliation. The fix was not a clock fix — it was replacing the register with a PN-Counter (per-region max-merge), which is commutative by construction and immune to skew. LWW is safe only for overwrite semantics on data where losing a concurrent write is acceptable: display names, last-known location, feature flags, presence heartbeats. It is never safe for balances, inventory, or counters.
Garbage collection is the real scalability wall
Add-wins OR-Sets attach a unique tag to every add so that merges are pure set unions:
Replica A Replica B
adds: {"a": [A:1], adds: {"a": [B:7]}
"b": [A:2]} removed: [A:1]
removed: [B:7]
merge(adds = union, removed = union):
adds: {"a": [A:1, B:7], "b": [A:2]}
removed: [A:1, B:7]
members: {"b"} <- "a" is gone: every tag is tombstonedThat is elegant, and it leaks. Tombstones and tag sets grow O(operations) forever unless you delete them, and you may only delete a tombstone once every replica has observed the removal. That is causal stability, and it is measurable:
- Instrument
p99.9replication lag per origin region (SQL below does exactly this). - Set tombstone TTL =
10 × p99.9 lag, with a floor of 24 h. A 400 ms p99.9 lag becomes a 4 s TTL in theory and a 24 h TTL in practice — because your p99.9 during a backfill is not your p99.9 during a partition. - Run Merkle-tree anti-entropy as the safety net, not as the GC mechanism.
Anti-entropy with Merkle trees: O(log n) hashes per round instead of O(n) keys replica A root hash |-- hash(range 0x00-0x3f) == match, stop |-- hash(range 0x40-0x7f) != mismatch, descend | |-- hash(0x40-0x5f) == match | +-- hash(0x60-0x7f) != ship the 4 KB delta for this range +-- hash(range 0x80-0xff) == match, stop
Production war story. A 4 TB token range needed repair after a 6-hour region outage. Unthrottled anti-entropy saturated the 10 Gbps inter-region link within 90 seconds and starved replication of *live* writes, turning a recovery into a second incident. Fix: token-bucket the repair stream to 15% of link capacity, run it at lowest priority (DSCP EF is for replication lag, not repair), and gate the next range on observed lag returning to baseline.
CRDT selection matrix
| CRDT | Merge function | Metadata per replica | Conflict semantics | Safe for | Never use for | |---|---|---|---|---|---| | G-Counter | per-node max | O(regions) | none possible | page views, bytes served | anything with decrements | | PN-Counter | per-node max of inc/dec | O(2 × regions) | none possible | inventory deltas, quota usage | balances needing non-negative invariants | | LWW-Register | max by HLC | O(1) | last-writer-wins, silent loss | display name, last-seen, flags | money, counters, inventory | | MV-Register | keep siblings | O(concurrent writes) | app resolves | mergeable profile fields | high-contention hot keys | | OR-Set (add-wins) | union adds + union tombstones | O(adds) + tombstones | concurrent add beats remove | carts, tags, memberships | unbounded-growth sets without GC | | RGA / YATA | position identifiers | O(ops) with GC | intent-preserving | collaborative text | append-heavy logs |
Failover Topologies: From Warm Standby to Symmetric Multi-Write
Five topologies cover ~99% of real deployments. Rank them by the two numbers your business actually cares about — RPO and RTO — and then by operational burden.
(1) Single writer + async read replicas
[us-east-1 PRIMARY] --async--> [eu-west-1 RO] --async--> [ap-southeast-1 RO]
Writes: 9 ms local | RPO: 0.5-2 s | RTO: 5-15 min (manual promotion)
(2) Active-passive warm standby
[us-east-1 ACTIVE] --sync-ish--> [eu-west-1 STANDBY, no traffic]
Writes: 11 ms | RPO: ~1 s | RTO: 60-120 s (automated health-check promotion)
(3) Home-region write affinity (per-key ownership)
key(order:42) -> owner eu-west-1 ; reads served locally everywhere via CRDT cache
Writes: 9 ms local / 82 ms cross-region | RPO: 0 for owned keys | RTO: 15-45 s
(4) Synchronous quorum active-active (N=3, W=2, R=2)
[us-east-1] <==sync==> [eu-west-1] <==sync==> [ap-southeast-1]
Writes: 84-172 ms | RPO: 0 | RTO: 10-25 s | survives exactly one region loss
(5) Symmetric async CRDT multi-write
all three regions accept every write locally, converge via delta-state CRDTs
Writes: 7 ms local ack | RPO: 0 (writes always accepted) | RTO: none required (AP)
Convergence p99: 400-900 ms ; read-your-writes requires session tokensPACELC is the real decision framework
CAP only describes behaviour *during* a partition. PACELC adds the more common case: else (no partition), you still trade Latency for Consistency. Topology (4) is PC/EC — you pay 78 ms on every write to be strongly consistent. Topology (5) is PA/EL — 7 ms writes, eventual convergence. Topology (3) is the pragmatic middle: EC for a key's owner region, EL for everyone else, with ownership making the distinction irrelevant to correctness.
The four mechanics that make failover safe
1. φ-accrual failure detection, not boolean heartbeats. With 1 s heartbeats, a φ threshold of 8 yields detection in ~10 s at a 1% false-positive rate. Boolean timeouts either flap (2 s timeout) or blow your RTO (30 s timeout). φ gives you a tunable curve instead of a cliff. 2. Leases with an explicit skew margin. A 10 s lease with a 250 ms bounded clock drift is only safe for 9.75 s of use; the remaining 250 ms is the margin that prevents two regions from believing they hold the same lease. Encode the drift bound in code, not in comments. 3. Monotonic fencing tokens. Every write carries an epoch. Storage rejects any write where epoch < current_epoch using a conditional write (UPDATE ... WHERE epoch < $new), so a paused-then-resumed leader cannot clobber its successor. This is the only mechanism that reliably prevents split-brain writes at the storage layer. 4. A witness region for quorum arithmetic. With three data regions, a partition of {A} | {B, C} leaves A as a minority that must refuse writes — but A can only *know* it is a minority if it can count votes. A cheap fourth region (no data, no traffic, votes only) turns ambiguous 2-of-3 splits into clean 2-of-4 resolutions at near-zero cost.
Production war story. A network policy change isolated eu-west-1 from the other two regions for 47 seconds. Both sides continued accepting writes because the application used last-write-wins on a version column rather than epochs. When the partition healed, the RGA-based comment thread duplicated paragraphs and an OR-Set membership table resurrected two deleted users. Post-incident fixes: epoch fencing at the storage layer, an explicit minority-refusal path returning HTTP 503 with Retry-After, and a post-heal convergence check that compares Merkle roots and pages if they disagree for more than 5 minutes.
Practical Implementation
A dependency-free TypeScript coordinator implementing an HLC, an LWW-Register, a PN-Counter, an add-wins OR-Set, and a latency-budget-aware write planner that chooses sync-quorum vs. local-async per request.
// src/regions/crdt-coordinator.ts
import { randomUUID } from 'node:crypto';
export type RegionId = 'us-east-1' | 'eu-west-1' | 'ap-southeast-1';
/* -------- 1. Hybrid Logical Clock: O(1) metadata, bounded skew, total order -------- */
export interface Hlc { readonly wall: number; readonly counter: number; readonly node: string }
export const MAX_CLOCK_DRIFT_MS = 250;
export function hlcTick(prev: Hlc | null, node: string, now: number = Date.now()): Hlc {
const wall = prev ? Math.max(now, prev.wall) : now;
const counter = prev !== null && wall === prev.wall ? prev.counter + 1 : 0;
if (counter > 65535) throw new RangeError('HLC counter overflow on ' + node + ': clock is frozen');
return { wall, counter, node };
}
/** Fold a remote HLC into the local clock on message receipt (causal consistency). */
export function hlcReceive(local: Hlc, remote: Hlc, node: string, now: number = Date.now()): Hlc {
const wall = Math.max(now, local.wall, remote.wall);
if (wall === local.wall && wall === remote.wall) {
return { wall, counter: Math.max(local.counter, remote.counter) + 1, node };
}
if (wall === local.wall) return { wall, counter: local.counter + 1, node };
if (wall === remote.wall) return { wall, counter: remote.counter + 1, node };
return { wall, counter: 0, node };
}
export function hlcCompare(a: Hlc, b: Hlc): -1 | 0 | 1 {
if (a.wall !== b.wall) return a.wall < b.wall ? -1 : 1;
if (a.counter !== b.counter) return a.counter < b.counter ? -1 : 1;
if (a.node !== b.node) return a.node < b.node ? -1 : 1;
return 0;
}
/* -------- 2. LWW-Register: overwrite semantics ONLY (never money) -------- */
export interface LwwRegister<T> { readonly value: T; readonly ts: Hlc }
export function lwwMerge<T>(a: LwwRegister<T>, b: LwwRegister<T>): LwwRegister<T> {
return hlcCompare(a.ts, b.ts) >= 0 ? a : b;
}
/* -------- 3. PN-Counter: commutative, skew-immune, no lost decrements -------- */
export type Vector = Record<RegionId, number>;
export interface PnCounter { readonly inc: Vector; readonly dec: Vector }
const zeroVector = (): Vector => ({ 'us-east-1': 0, 'eu-west-1': 0, 'ap-southeast-1': 0 });
export const pnZero = (): PnCounter => ({ inc: zeroVector(), dec: zeroVector() });
export function pnAdd(c: PnCounter, region: RegionId, by: number): PnCounter {
return by >= 0
? { inc: { ...c.inc, [region]: c.inc[region] + by }, dec: c.dec }
: { inc: c.inc, dec: { ...c.dec, [region]: c.dec[region] - by } };
}
export function pnMerge(a: PnCounter, b: PnCounter): PnCounter {
const max = (x: Vector, y: Vector): Vector => {
const out = zeroVector();
for (const r of Object.keys(out) as RegionId[]) out[r] = Math.max(x[r] ?? 0, y[r] ?? 0);
return out;
};
return { inc: max(a.inc, b.inc), dec: max(a.dec, b.dec) };
}
export function pnValue(c: PnCounter): number {
const total = (v: Vector) => (Object.values(v) as number[]).reduce((s, n) => s + n, 0);
return total(c.inc) - total(c.dec);
}
/* -------- 4. Add-wins OR-Set: unique tags make merge a pure set union -------- */
export type Tag = string;
export interface OrSet<E extends string> { readonly adds: Record<E, Tag[]>; readonly removed: Tag[] }
export const orEmpty = <E extends string>(): OrSet<E> => ({ adds: {} as Record<E, Tag[]>, removed: [] });
export function orAdd<E extends string>(s: OrSet<E>, el: E, region: RegionId): OrSet<E> {
const tag = region + ':' + randomUUID();
return { adds: { ...s.adds, [el]: [...(s.adds[el] ?? []), tag] }, removed: s.removed };
}
/** Removes only the tags this replica has observed, so a concurrent add wins. */
export function orRemove<E extends string>(s: OrSet<E>, el: E): OrSet<E> {
const seen = s.adds[el] ?? [];
return { adds: s.adds, removed: [...new Set([...s.removed, ...seen])] };
}
export function orMerge<E extends string>(a: OrSet<E>, b: OrSet<E>): OrSet<E> {
const adds: Record<string, Tag[]> = {};
for (const src of [a.adds, b.adds]) {
for (const [el, tags] of Object.entries(src) as Array<[string, Tag[]]>) {
adds[el] = [...new Set([...(adds[el] ?? []), ...tags])];
}
}
return { adds: adds as Record<E, Tag[]>, removed: [...new Set([...a.removed, ...b.removed])] };
}
export function orMembers<E extends string>(s: OrSet<E>): E[] {
const gone = new Set(s.removed);
return (Object.keys(s.adds) as E[]).filter((el) => s.adds[el].some((t) => !gone.has(t)));
}
/* -------- 5. Latency-budget-aware write planner (p99 inter-region RTT, measured) -------- */
const RTT_P99_MS: Record<RegionId, Record<RegionId, number>> = {
'us-east-1': { 'us-east-1': 0, 'eu-west-1': 74, 'ap-southeast-1': 232 },
'eu-west-1': { 'us-east-1': 74, 'eu-west-1': 0, 'ap-southeast-1': 168 },
'ap-southeast-1': { 'us-east-1': 232, 'eu-west-1': 168, 'ap-southeast-1': 0 },
};
const COMMIT_P99_MS: Record<RegionId, number> = {
'us-east-1': 3.2, 'eu-west-1': 4.1, 'ap-southeast-1': 5.6,
};
const REGION_RING: readonly RegionId[] = ['us-east-1', 'eu-west-1', 'ap-southeast-1'];
/** FNV-1a; swap for Jump Consistent Hash once you exceed ~8 regions. */
export function homeRegionFor(key: string): RegionId {
let hash = 0x811c9dc5;
for (let i = 0; i < key.length; i++) {
hash ^= key.charCodeAt(i);
hash = Math.imul(hash, 0x01000193) >>> 0;
}
return REGION_RING[hash % REGION_RING.length]!;
}
export type WriteMode = 'sync-quorum' | 'local-async-crdt' | 'reject';
export interface SloBudget {
readonly edgeMs: number;
readonly appMs: number;
readonly responseMs: number;
readonly targetP99Ms: number;
}
export interface WritePlan {
readonly mode: WriteMode;
readonly homeRegion: RegionId;
readonly projectedP99Ms: number;
readonly rationale: string;
}
export function planWrite(key: string, clientRegion: RegionId, budget: SloBudget): WritePlan {
const home = homeRegionFor(key);
const rtt = RTT_P99_MS[clientRegion][home];
const base = budget.edgeMs + budget.appMs + budget.responseMs + COMMIT_P99_MS[home];
const sync = base + rtt; // one full RTT for the cross-region quorum ack
if (sync <= budget.targetP99Ms) {
return { mode: 'sync-quorum', homeRegion: home, projectedP99Ms: sync,
rationale: 'RPO=0 achievable within p99 target' };
}
if (base <= budget.targetP99Ms) {
return { mode: 'local-async-crdt', homeRegion: home, projectedP99Ms: base,
rationale: 'commit locally, converge via delta-state CRDT (convergence p99 ~600 ms)' };
}
return { mode: 'reject', homeRegion: home, projectedP99Ms: base,
rationale: 'SLO unreachable even with local commit; shed load or relax budget' };
}
/* -------- 6. Wiring -------- */
const budget: SloBudget = { edgeMs: 18, appMs: 11, responseMs: 30, targetP99Ms: 250 };
planWrite('order:8f2c1a', 'eu-west-1', budget);
// -> mode 'sync-quorum', projectedP99Ms 136.2 (62.2 ms base + 74 ms RTT)
planWrite('order:8f2c1a', 'ap-southeast-1', budget);
// -> mode 'local-async-crdt', projectedP99Ms 62.2 (232 ms RTT cannot fit 250 ms)Durable side: an exactly-once delta op log plus the replica-lag query that drives tombstone GC safety.
-- Delta-state op log. Partitioned by entity so anti-entropy is co-located.
CREATE TABLE crdt_op_log (
entity_key text NOT NULL,
op_id uuid NOT NULL,
origin_region text NOT NULL,
crdt_type text NOT NULL CHECK (crdt_type IN ('lww','pn_counter','or_set','rga')),
hlc_wall bigint NOT NULL, -- origin logical time; skew bounded by MAX_CLOCK_DRIFT_MS
hlc_counter integer NOT NULL,
hlc_node text NOT NULL,
delta jsonb NOT NULL,
applied_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (entity_key, op_id) -- idempotent replay: duplicate apply is a no-op
) PARTITION BY HASH (entity_key);
-- 0-RTT TLS resumption and client retries both replay requests. This table makes them safe.
CREATE TABLE idempotency_key (
scope text NOT NULL,
client_key text NOT NULL,
response jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (scope, client_key)
) WITH (fillfactor = 90);
-- Tombstone TTL must be >= 10x this number. Alert if lag_p999_s exceeds 60.
SELECT origin_region,
percentile_disc(0.999) WITHIN GROUP (
ORDER BY (extract(epoch FROM now()) * 1000 - hlc_wall) / 1000.0
) AS lag_p999_s
FROM crdt_op_log
WHERE applied_at > now() - interval '15 minutes'
GROUP BY origin_region;Performance Benchmarks & Trade-Offs
Measured on a 3-region deployment (m7g.4xlarge, 100 GB working set, NVMe-backed, 16 KB encrypted records), client originating in eu-west-1, 30-minute steady-state window, 72-hour observation for p99.
| Topology | Write p99 (eu-west-1 client) | Regional read p99 | RPO | Automated RTO | Conflict model | Practical write ceiling | |---|---|---|---|---|---|---| | Single writer + async read replicas | 9 ms (local) / 78 ms (routed) | 3.0 ms | 0.5–2 s | 5–15 min (manual) | none (single writer) | ~50k writes/s per shard | | Active–passive warm standby | 11 ms | 3.0 ms | ~1 s | 60–120 s | none | same as primary | | Home-region write affinity | 9 ms local / 82 ms cross-region | 2.5 ms | 0 for owned keys | 15–45 s | avoided by ownership | ~120k writes/s aggregate | | Synchronous quorum (N=3, W=2) | 84 ms (eu→us) / 172 ms (eu→ap) | 9 ms (R=2) | 0 | 10–25 s | none (quorum order) | ~30k writes/s, latency-bound | | Symmetric async CRDT multi-write | 7 ms (local ack) | 2.5 ms (may be stale) | 0 (always accepts) | none required (AP) | CRDT merge | ~200k writes/s; convergence p99 400–900 ms |
Three trade-offs worth stating explicitly:
- Quorum active-active buys you RPO=0 at 9–12× the write latency of local-commit designs, and it fails closed: losing two of three regions means no writes at all. If your business can tolerate 2 s of RPO, the single-writer topology is cheaper, simpler, and 8× faster.
- Symmetric CRDT multi-write is not free on reads. Read-your-writes requires session tokens (a per-session minimum HLC stamped on the read request), which forces replica waiting on some reads and pushes read p99 from 2.5 ms to 25–40 ms for a minority of sessions. Budget for it or your users will see their own edits disappear.
- The convergence tail is the hidden cost. Median replication lag is 40–90 ms; p99 is 400–900 ms; p99.9 during a backfill or compaction window reaches 8–15 s. Every staleness-sensitive read — balance display, inventory badges, permission checks — needs a fallback, because 15 s of stale permissions is a security event, not a latency event.
Pro-Tips & Common Pitfalls
- Pitfall: cluster-wide consistency levels in a multi-DC keyspace.
QUORUMsilently becomes cross-region. Fix:LOCAL_QUORUM/LOCAL_SERIALeverywhere, enforced by a lint rule on the query layer. - Pitfall: LWW on numeric state. Clocks step; writes vanish silently. Fix: PN-Counters for additive state, MV-Registers where the application can resolve siblings.
- Pitfall: tombstones with a fixed 24 h TTL and no lag telemetry. A 30-hour partition resurrects deleted users. Fix: TTL = max(10 × p99.9 lag, 24 h), with an alert on lag p99.9 > 60 s.
- Pitfall: boolean failure detection with a 2 s timeout. It flaps under normal p99.9 jitter and triggers spurious failovers. Fix: φ-accrual with φ ≥ 8.
- Pitfall: leader election without fencing. A resumed ex-leader writes over its successor. Fix: monotonic epoch column with a conditional write on every mutation.
- Pitfall: unbounded Merkle repair. Repair saturates the inter-region link and starves live replication. Fix: token-bucket repair to ≤15% of link capacity, lowest QoS class, gate each range on lag returning to baseline.
- Pitfall: assuming p99 RTT is stable. BGP reconvergence adds 30–180 s of degradation. Fix: regional write isolation plus a degraded-mode path (accept locally, converge later) rather than global retry storms.
- Pitfall: 0-RTT TLS without idempotency keys. Replayed early data double-charges customers. Fix: idempotency table with a unique
(scope, client_key), returning the cached response on replay. - Pitfall: treating a region failover as a database event. It is a *routing* event. Fix: anycast/GeoDNS health checks must flip within the RTO budget, or the promoted region sits idle while traffic keeps hammering the dead one.
- Pitfall: no post-heal convergence verification. Failover is not done when writes resume; it is done when Merkle roots match. Fix: automated root comparison with a 5-minute alert threshold.
Frequently Asked Questions (FAQ)
How do you resolve write conflicts in a multi-region active-active database?
There are only three real strategies. First, avoid them: home-region write affinity assigns each key a single owning region, so concurrent writers never exist — this handles the majority of workloads at 9 ms local write latency. Second, make them commutative: use CRDTs (PN-Counters, add-wins OR-Sets, RGA sequences) whose merge function is commutative, associative, and idempotent, so replicas converge regardless of ordering; this costs O(regions) counter metadata or O(adds) tombstone tags per object. Third, detect and defer: multi-value registers retain siblings and force application-level resolution, which is correct but pushes complexity onto every read path and is usually only justified for rich mergeable documents. The wrong answer is last-write-wins on numeric state — it converts a visible conflict into silent data loss that surfaces days later in reconciliation.
What cross-region latency budget should a synchronous active-active write target?
Budget one full RTT for the cross-region quorum ack plus 60–70 ms of local overhead (edge TLS, auth, intra-region commit, response path). Against a 250 ms p99 user SLO that leaves ~180 ms of RTT headroom, which accommodates transatlantic (78 ms p99) and trans-US (68 ms p99) quorums comfortably but forbids transpacific ones (238 ms p99). The practical rule: never couple synchronously beyond 80 ms p99 RTT. If a region pair exceeds that, either change the pairing, route the write to a nearer quorum subset, or fall back to local-commit plus delta-state CRDT convergence — which trades 0–900 ms of staleness for a 9–12× reduction in write latency.
How do you prevent split-brain during a multi-region failover?
Layer four mechanisms. (1) Quorum requirement: with N=3 regions and W=2, a minority partition cannot commit and must return 503 with Retry-After rather than accepting writes. (2) Witness region: a fourth, data-free region that only votes, which resolves otherwise ambiguous 2-of-3 splits into clean 2-of-4 outcomes at negligible cost. (3) Leases with an explicit skew margin: a 10 s lease with a 250 ms bounded clock-drift tolerance is only usable for 9.75 s — encode the margin, do not merely comment on it. (4) Monotonic fencing tokens: every write carries an epoch, and storage rejects epoch < current_epoch via a conditional update, so a stalled leader that resumes after promotion cannot clobber its successor. Together these reduce split-brain windows from 40–60 s to under the lease duration, and they make post-heal divergence a monitoring event rather than a data-repair project.
