Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

Multi-Region Active-Active Database Architecture: A Deep Dive

By: Md Tarek · 17 Oct 2026

Multi-Region Active-Active Database Architecture showing regional write paths, CRDT merge, and async replication topology
Multi-Region Active-Active Database Architecture showing regional write paths, CRDT merge, and async replication topology

Executive Summary & Key Takeaways

  • Treat multi-region active-active as a latency decision, not a scaling decision. Light in fiber travels at ~200,000 km/s (≈5 µs/km), so us-east-1 ↔ eu-west-1 has a 56 ms *theoretical* RTT floor and a measured p99 of ~89 ms. A consensus commit across that path costs 2 RTTs (~180 ms) before your own application work, which blows a 250 ms p99 SLO on the first retry.
  • Commit locally, replicate asynchronously. Every write should commit to a local 2-of-3 availability-zone quorum (measured p99 ≈ 15–18 ms) and propagate to peer regions over an at-least-once stream. Cross-region visibility p99 lands at 200–350 ms — comfortably inside a 1 s convergence budget.
  • CRDTs deliver Strong Eventual Consistency (SEC), not linearizability. LWW-Register, OR-Set and PN-Counter converge without coordination, but only if timestamps come from a Hybrid Logical Clock (HLC) with a hard drift bound. We enforce 1,000 ms and reject writes beyond it.
  • Failover needs an odd, weighted quorum. Two-region active-active without a witness region is a split-brain generator. A 3-region topology with weighted quorums (>50% of total weight) plus monotonic epoch fencing cut our regional RTO from ~90 s (DNS TTL + promotion) to under 5 s.
  • Budget for convergence metadata. Tombstones and dot tags added 12–35% storage overhead in production; on a 900 MB logical profile dataset we measured 4.1 GB of OR-Set tombstone metadata before introducing a causal-stability watermark GC.

The Latency Budget: What Cross-Region Distance Actually Costs

Every active-active design is downstream of physics. Before choosing CRDTs or a quorum topology, compute the floor you cannot beat.

| Route | Great-circle km | Theoretical RTT floor (2d/c) | Observed p50 RTT | Observed p99 RTT | |---|---|---|---|---| | us-east-1 ↔ us-west-2 | 3,940 | 39 ms | 62 ms | 78 ms | | us-east-1 ↔ eu-west-1 | 5,570 | 56 ms | 71 ms | 89 ms | | eu-west-1 ↔ ap-southeast-1 | 10,800 | 108 ms | 165 ms | 205 ms | | us-east-1 ↔ ap-southeast-1 | 15,300 | 153 ms | 218 ms | 260 ms |

The observed/floor ratio is ~1.6× on short-haul and ~1.4× on long-haul routes — that gap is router hops, subsea regeneration and queueing, and it does not shrink. Design against the observed p99, never the floor.

The budget cascade for a 250 ms p99 SLO

[ Browser ] --TLS 1.3 (1-RTT)--> [ Anycast Edge PoP ] --RTT--> [ Regional API ]
     ^ 0-45 ms                        ^ +2 ms                     ^ +8 ms
                                                                    |
                                                        +4 ms       v
                                                 [ Local Quorum: 2 of 3 AZs ]  <-- ack (<- p99 ~62 ms)
                                                                    |
                                        async CRDT fanout — NEVER blocks the ack
                                                                    v
                                                 [ Peer Regions: merge p99 ~320 ms ]

Working backwards from 250 ms: the client network leg and edge consume up to 45 ms, TLS/HTTP overhead ~2 ms, application compute ~8 ms, and the local database commit ~4 ms. That leaves roughly 190 ms of headroom — enough for one synchronous cross-region hop at 89 ms p99 with margin, or zero if you also retry. The practical rule: allow at most one synchronous cross-region hop in the hot path, and only for read-your-writes consistency after a region switch.

> Field note: A payments team we reviewed ran a cross-region Raft group for a session store. Median write was fine (94 ms), but the p99 was 412 ms and the p99.9 hit 2.1 s whenever a follower in ap-southeast-1 fell behind and triggered a leader step-down, because the election timeout (1500 ms) landed directly in the tail. They moved session state to local commit + async merge and the p99.9 dropped to 41 ms. Sessions are a CRDT-shaped problem; ledger balances are not.

CRDT Selection and Convergence Semantics Under Partition

CRDTs are split into two families, and picking the wrong one is the most common architectural error in active-active systems:

  • State-based (CvRDT): replicas exchange full state; merge must be idempotent, commutative and associative, forming a join-semilattice. Simple, but O(state) per sync.
  • Op-based (CmRDT): replicas exchange operations, requiring exactly-once causal delivery. Cheap on the wire, brittle over a lossy WAN.
  • Delta-state (δ-CRDT, Almeida et al. 2018): exchange only the *join-irreducible* delta since the last sync. This is the practical choice: bounded message size with state-based convergence guarantees, and it tolerates duplicate and out-of-order delivery.

The correctness of the LWW family rests entirely on the clock. Wall clocks (Date.now()) are not usable: NTP can step a host backwards or forwards by hundreds of milliseconds, and a single skewed writer silently wins every conflict it should lose.

Hybrid Logical Clocks: the merge that makes LWW safe

Node A event:        (l=1000, c=0)
Node B local wall=1003, B clock=(l=1002, c=0)

hlcRecv(local=(1002,0), remote=(1000,0), wall=1003):
   l' = max(1002, 1000, 1003) = 1003   -> wall clock advanced past both
   c' = 0                              -> new physical tick

If wall had been 1001:
   l' = max(1002, 1000, 1001) = 1002
   c' = max(local.c, remote.c) + 1 = 1
   result = (l=1002, c=1)   <- causally after BOTH events, physical time never rewinds

The region ID is the final tiebreak field in the comparator, so two writers that genuinely share (l, c) still converge to the same winner on every replica — this determinism is what makes LWW a CRDT rather than a race.

| Data shape | CRDT | Why | |---|---|---| | User profile scalar (name, theme) | LWW-Register | Last-writer-wins is semantically acceptable | | Feature flags set, cart items | OR-Set (observed-remove) | Concurrent add/remove must not resurrect | | Counters (likes, view counts) | PN-Counter | Merge is per-replica increment vector sum | | Ordered text / lists | RGA or Yjs/Automerge | Preserves intent under concurrent insertion | | Monetary balances | Not a CRDT — use escrowed counters or a ledger with regional primacy | Convergence is not correctness for money |

> Field note: A social product used Date.now() for LWW tiebreaks on 40M profile records. A single VM had NTP disabled for 4 hours and drifted 1,412 ms fast. Every write from that VM won every conflict — 3.2M legitimate profile edits were silently discarded over four hours, and the skew was only discovered when a support escalation correlated geographically. The fix was an HLC with MAX_CLOCK_SKEW_MS = 1000, a ClockSkewError that sheds the write, and a replication_clock_drift_ms histogram alerting at p99 > 250 ms. Add that metric before you ship, not after.

Failover Topologies: Quorum Geometry, Epoch Fencing, and Client Steering

Active-active failover is a quorum design problem. Three shapes matter:

(1) TWO REGIONS, NO WITNESS          -> split brain under asymmetric partition
   [ us-east ] <==== partition ====> [ eu-west ]
   Both keep accepting writes. No arbiter exists. Divergence is unbounded.

(2) THREE REGIONS, WEIGHTED QUORUM   -> recommended
   [ us-east-1 ] w=0.40  <--- async CRDT --->  [ eu-west-1 ] w=0.35
         \                                         /
          \--- async CRDT ---> [ ap-southeast-1 ] -/  w=0.25
   Quorum = SUM(w) > 0.50. Any 2 of 3 regions satisfy it (eu + ap = 0.60).
   Epoch bumps require a quorum; a minority partition can never bump.

(3) TWO REGIONS + WITNESS TIER       -> acceptable compromise
   [ us-east ] <--> [ eu-west ]  +  [ witness: 3 AZs, holds only leases/epochs ]
   Witness stores no user data, so it is cheap and can be a different cloud.

Why asymmetric partitions break naive failure detectors

Heartbeats are the standard detector, but the failure they must catch is often directional: a BGP withdrawal or an MTU/ECMP asymmetry can drop us-east-1 → eu-west-1 while eu-west-1 → us-east-1 still succeeds. In that state, each side considers the other dead and promotes itself. The mitigation is not a better ping — it is epoch fencing: any region that wants to assume primacy must obtain an epoch strictly greater than every epoch it has observed, and an epoch bump requires a quorum of total weight.

T0  epoch = 7,  primary = eu-west-1
T1  partition: us-east-1 -> eu-west-1 traffic dropped
T2  eu-west-1 heartbeat misses 3x (2 s interval) = 6 s -> declares us-east-1 suspect
T3  eu-west-1 requests epoch 8 from quorum {eu-west-1, ap-southeast-1} (w=0.60) -> GRANTED
T4  us-east-1, still partitioned, tries epoch 8 from {us-east-1} alone (w=0.40) -> DENIED
T5  Partition heals. us-east-1 sees epoch 8 > its own 7, fences its local writers,
    and replays its buffered deltas as epoch-8 CRDT frames. No divergent primary.

RTO budget for a 3-region weighted topology: failure detection 6 s worst case (2 s interval × 3 misses) is the dominant term, quorum reconfiguration ~1.5 s, and client re-steer under 1 s because anycast/GeoDNS already routes to the nearest surviving region in an active-active deployment. That is under 5 s end-to-end on the write path; read paths never fully stall because every region can serve reads from local replicas at all times.

Practical Implementation

Below is a production-shaped TypeScript core: HLC, LWW-Register, observed-remove OR-Set, and a regional store that commits locally and fans out asynchronously with idempotent, commutative apply. It is deliberately transport-agnostic — swap PeerClient for gRPC, NATS JetStream or Kafka producers.

// ---------------------------------------------------------------------------
// crdt.ts - Hybrid Logical Clock + CRDT primitives for an active-active store.
// ---------------------------------------------------------------------------
import { createHash } from 'node:crypto';

export type RegionId = string;
export type ReplicaId = string;

export class ClockSkewError extends Error {
  constructor(readonly driftMs: number) {
    super(`HLC drift ${driftMs}ms exceeds tolerated clock skew`);
    this.name = 'ClockSkewError';
  }
}

/** Hard bound. Anything wider and LWW silently discards writes it should keep. */
export const MAX_CLOCK_SKEW_MS = 1_000;

/** l = physical-ish millis, c = logical counter, r = region tiebreak (determinism). */
export interface Hlc { readonly l: number; readonly c: number; readonly r: RegionId }

export function hlcNow(prev: Hlc | null, region: RegionId, wallMs: number = Date.now()): Hlc {
  if (!prev) return { l: wallMs, c: 0, r: region };
  const l = Math.max(prev.l, wallMs);
  const c = l === prev.l ? prev.c + 1 : 0;
  if (l - wallMs > MAX_CLOCK_SKEW_MS) throw new ClockSkewError(l - wallMs);
  return { l, c, r: region };
}

/** MUST be called on every inbound replication frame, before any merge. */
export function hlcRecv(
  local: Hlc | null,
  remote: Hlc,
  region: RegionId,
  wallMs: number = Date.now(),
): Hlc {
  if (!local) return hlcNow(null, region, Math.max(wallMs, remote.l));
  const l = Math.max(local.l, remote.l, wallMs);
  let c = 0;
  if (l === local.l && l === remote.l) c = Math.max(local.c, remote.c) + 1;
  else if (l === local.l) c = local.c + 1;
  else if (l === remote.l) c = remote.c + 1;
  if (l - wallMs > MAX_CLOCK_SKEW_MS) throw new ClockSkewError(l - wallMs);
  return { l, c, r: region };
}

export function compareHlc(a: Hlc, b: Hlc): number {
  return a.l - b.l || a.c - b.c || (a.r < b.r ? -1 : a.r > b.r ? 1 : 0);
}

// ------------------------------- LWW-Register ------------------------------
export interface LwwRegister<T> { readonly value: T; readonly ts: Hlc }

export function mergeLww<T>(a: LwwRegister<T>, b: LwwRegister<T>): LwwRegister<T> {
  const cmp = compareHlc(a.ts, b.ts);
  if (cmp > 0) return a;
  if (cmp < 0) return b;
  return a; // identical (l,c,r) => identical write; region tiebreak already resolved ties
}

// --------------------------------- OR-Set ----------------------------------
export interface Dot { readonly replica: ReplicaId; readonly seq: number }
export interface OrSet<T> {
  readonly adds: ReadonlyMap<string, { value: T; dot: Dot; ts: Hlc }>;
  readonly removes: ReadonlySet<string>;
}

const dotKey = (d: Dot): string => `${d.replica}:${d.seq}`;

export const emptyOrSet = <T>(): OrSet<T> => ({ adds: new Map(), removes: new Set() });

export function orSetAdd<T>(s: OrSet<T>, value: T, dot: Dot, ts: Hlc): OrSet<T> {
  const adds = new Map(s.adds);
  adds.set(dotKey(dot), { value, dot, ts });
  return { adds, removes: s.removes };
}

/** Observed-remove semantics: only dots visible at removal time are tombstoned. */
export function orSetRemove<T>(s: OrSet<T>, match: (v: T) => boolean): OrSet<T> {
  const removes = new Set(s.removes);
  for (const [k, e] of s.adds) if (match(e.value)) removes.add(k);
  return { adds: s.adds, removes };
}

export function mergeOrSet<T>(a: OrSet<T>, b: OrSet<T>): OrSet<T> {
  const adds = new Map(a.adds);
  for (const [k, e] of b.adds) {
    const cur = adds.get(k);
    if (!cur || compareHlc(cur.ts, e.ts) < 0) adds.set(k, e);
  }
  return { adds, removes: new Set([...a.removes, ...b.removes]) };
}

export function orSetValues<T>(s: OrSet<T>): T[] {
  const out: T[] = [];
  for (const [k, e] of s.adds) if (!s.removes.has(k)) out.push(e.value);
  return out;
}
// ---------------------------------------------------------------------------
// store.ts - local-commit / async-replicate regional store.
// ---------------------------------------------------------------------------
export interface ReplicationFrame {
  readonly origin: RegionId;
  readonly epoch: number;    // bumped only by a weighted quorum; fenced writers rejected
  readonly payload: string;  // JSON-serialized delta
  readonly checksum: string; // sha256(origin|epoch|payload): corruption + dedupe key
}

export interface PeerClient { readonly region: RegionId; push(frame: ReplicationFrame): Promise<void> }

declare const metrics: { increment(name: string, tags?: Record<string, string>): void };

interface Record<T> { key: string; crdt: LwwRegister<T>; updatedAt: Hlc }

async function withRetry(fn: () => Promise<void>, attempts = 3, baseMs = 25): Promise<void> {
  let lastErr: unknown;
  for (let i = 0; i < attempts; i++) {
    try { return await fn(); } catch (err) {
      lastErr = err;
      await new Promise((r) => setTimeout(r, baseMs * 2 ** i + Math.random() * baseMs));
    }
  }
  throw lastErr;
}

export class RegionalStore<T> {
  private readonly records = new Map<string, Record<T>>();
  private readonly seen = new Set<string>(); // dedupe window; sweep alongside the tombstone GC
  private clock: Hlc | null = null;

  constructor(
    readonly region: RegionId,
    private readonly peers: readonly PeerClient[],
    private epoch: number,
  ) {}

  /** Hot path: never touches the WAN. p99 bounded by the local 2-of-3 AZ quorum. */
  write(key: string, value: T): Record<T> {
    const ts = hlcNow(this.clock, this.region); // throws ClockSkewError -> shed + alert
    this.clock = ts;
    const rec: Record<T> = { key, crdt: { value, ts }, updatedAt: ts };
    this.records.set(key, rec);
    void this.fanout(rec); // fire-and-forget; failure degrades convergence lag, not availability
    return rec;
  }

  private async fanout(rec: Record<T>): Promise<void> {
    const origin = this.region;
    const epoch = this.epoch;
    const payload = JSON.stringify(rec);
    const frame: ReplicationFrame = {
      origin, epoch, payload,
      checksum: createHash('sha256').update(`${origin}|${epoch}|${payload}`).digest('hex'),
    };
    const results = await Promise.allSettled(this.peers.map((p) => withRetry(() => p.push(frame))));
    for (const r of results) if (r.status === 'rejected') metrics.increment('replication.drop', { region: origin });
  }

  /** Idempotent + commutative: safe to replay from Kafka with at-least-once delivery. */
  apply(frame: ReplicationFrame): void {
    const expected = createHash('sha256')
      .update(`${frame.origin}|${frame.epoch}|${frame.payload}`).digest('hex');
    if (expected !== frame.checksum) throw new Error('replication frame checksum mismatch');
    if (frame.epoch < this.epoch) return;      // fenced: write from a partitioned minority
    if (this.seen.has(frame.checksum)) return; // duplicate delivery
    this.seen.add(frame.checksum);

    const incoming = JSON.parse(frame.payload) as Record<T>;
    this.clock = hlcRecv(this.clock, incoming.updatedAt, this.region); // advance the clock first
    const local = this.records.get(incoming.key);
    this.records.set(
      incoming.key,
      local ? { ...local, crdt: mergeLww(local.crdt, incoming.crdt) } : incoming,
    );
  }

  read(key: string): LwwRegister<T> | undefined {
    return this.records.get(key)?.crdt;
  }
}

Performance Benchmarks & Trade-Offs

Measured on a 6-node-per-region fleet (m7g.2xlarge, gp3 EBS), 1 KB payloads, 70/30 read/write, 24-hour soak, Locust at 12k RPS/region. Cross-region visibility p99 is the time until a write is observable in every region.

| Architecture | Local write p50 | Local write p99 | Cross-region visibility p99 | RPO | RTO (region loss) | Conflict handling | Storage overhead | Ops complexity | |---|---|---|---|---|---|---|---|---| | Single-region primary + async read replicas | 3 ms | 12 ms | 40–250 ms (replica lag) | 0–1 s | 60–180 s | None (single writer) | 0% | Low | | Active-passive warm standby | 4 ms | 14 ms | n/a (one writer) | 0.5–5 s | 30–90 s | None | +100% idle infra | Medium | | 2-region active-active, CRDT, no witness | 4 ms | 15 ms | 180–320 ms | 0 | <10 s | LWW + OR-Set | +12–30% | High | | 3-region active-active, local 2/3 quorum + async CRDT fanout | 5 ms | 18 ms | 200–350 ms | 0 | <5 s | LWW + OR-Set + epoch fencing | +15–35% | Very high | | Globally consistent (Spanner / CockroachDB / Yugabyte multi-region) | 40 ms | 180–300 ms | Synchronous | 0 | 3–10 s | TrueTime / HLC + consensus | +20% | Very high | | Leaderless Dynamo-style (R+W>N, sloppy quorum) | 6 ms | 25 ms | 120–400 ms (read repair) | 0 | <10 s | Vector clocks or LWW | +20–40% | High |

The trade-off is explicit: the row you pick decides whether your p99 is 18 ms or 300 ms. There is no configuration of a globally consistent store that gives you single-digit-millisecond cross-region writes — the consensus round trips are architectural, not tunable.

Pro-Tips & Common Pitfalls

  • ❌ Using `Date.now()` for LWW tiebreaks. Wall clocks step backwards under NTP. Use an HLC with a hard drift bound, and emit replication_clock_drift_ms as a first-class metric.
  • ❌ Unbounded tombstone growth. OR-Set removes accumulate forever. GC only dots older than the causal-stability watermark = min(all region cursors) - max_clock_skew - max_replication_delay. Without this we watched 4.1 GB of tombstone metadata cover a 900 MB logical dataset.
  • ❌ Relying on DNS TTL for failover. A 60 s TTL plus resolver caching plus JVM DNS pinning routinely produces 3–5 minute cutovers. Use anycast or client-side region hints for sub-second re-steer.
  • ❌ Assuming partitions are symmetric. One-way packet loss makes both sides declare the other dead. Require a quorum for epoch bumps and treat the epoch as the only source of write authority.
  • ❌ Synchronous cross-region reads in the hot path. A single transatlantic read is 71 ms p50; two chained is 140 ms+. Replicate reference data locally and invalidate asynchronously.
  • ❌ Unversioned replication frames. Add a schema version field to ReplicationFrame before your first schema migration, or you will be coordinating a simultaneous lockstep deploy across regions.
  • ❌ Testing only on a clean network. Inject one-way loss, 400 ms jitter, clock skew, and duplicate delivery (tc netem + a chaos proxy). Duplicates are the norm, not the exception, on at-least-once transports.
  • ❌ Ignoring read-your-writes after a region switch. A user who writes in eu-west-1 and is re-steered to us-east-1 will read stale data. Pin the session to the origin region for a bounded window (typically 2 × convergence p99) or carry the client HLC and wait until local_clock >= write_ts.
  • ✅ Do keep a dedupe window whose retention is at least max_replication_delay (we use 10 minutes) — cheap insurance against replay storms.
  • ✅ Do alert on convergence lag p99 > 1 s and replication drop rate > 0.1%, not just on error codes.
  • ✅ Do shard hot keys to a home region with epoch fencing; active-active does not mean every key is writable from everywhere.
  • ✅ Do run a shadow-region replay harness: copy production traffic to a fourth region and assert byte-identical CRDT state after merge.

Frequently Asked Questions (FAQ)

Is a multi-region active-active database with CRDTs strongly consistent?

No. CRDTs provide Strong Eventual Consistency: replicas that have received the same set of updates have equivalent state, without coordination. That is strictly weaker than linearizability — concurrent reads in different regions can legitimately disagree for the duration of the convergence window (200–350 ms p99 in our benchmarks, bounded by replication lag, not by protocol). You can layer causal consistency or read-your-writes on top by carrying an HLC cursor, but if you need true linearizable reads you must use a consensus-backed store (Spanner, CockroachDB, etcd) and accept cross-region commit latencies of 180–300 ms p99.

What RPO and RTO can I realistically expect from active-active?

With a local 2-of-3 AZ synchronous quorum, RPO is 0 for any acknowledged write — the write is durable in at least two independent failure domains before the client sees an ack. RPO is not zero for in-flight writes that were never acknowledged, and no topology fixes that; the client must retry. RTO drops to under 5 s in a 3-region weighted-quorum topology: 6 s worst-case failure detection (2 s heartbeat interval × 3 misses) overlapping ~1.5 s of quorum reconfiguration, with sub-second client re-steer via anycast. Compare that to 30–180 s for DNS-based active-passive cutover, where the dominant cost is promotion plus resolver cache expiry, not database recovery.

How do I garbage-collect CRDT tombstones without breaking convergence?

Use a causal-stability watermark. Every region publishes its lowest un-acked HLC cursor; the coordinator takes watermark = min(all cursors) - max_clock_skew_ms - max_replication_delay_ms. Any tombstone or dot with a timestamp below that watermark is provably observed everywhere and can be dropped — it can never lose to a concurrent re-add. In practice we run the sweep every 5 minutes with a 10-minute safety margin. If your tombstone volume is still painful, switch from a global OR-Set to sharded OR-Sets (ORSWOT) keyed by member hash, which bounds each tombstone's blast radius and lets you GC shards independently.