Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

Event-Driven Architecture: Kafka vs Redis Streams at Scale

By: Md Tarek · 18 Oct 2026

Event-Driven Architecture Kafka vs Redis Streams with partitioned consumer groups and DLQs
Event-Driven Architecture Kafka vs Redis Streams with partitioned consumer groups and DLQs

Executive Summary & Key Takeaways

  • Throughput per node (1 KB events, replication factor 3, ack-all): Kafka sustains 180k–420k msg/s per broker at p99 < 15 ms using zero-copy sendfile, immutable log segments, and page-cache reads; Redis Streams reaches 500k–900k msg/s in pure memory but falls to 90k–150k msg/s once appendfsync everysec plus replica acknowledgement are enforced; RabbitMQ quorum queues plateau at 25k–60k msg/s per node because every message is Raft-replicated to a majority before ack.
  • Ordering is a partitioning decision, not a broker feature. Kafka orders per partition only, Redis Streams orders per stream (a single shard), and RabbitMQ orders per queue — but competing consumers plus redelivery silently reorder RabbitMQ under load.
  • Idempotent consumers are mandatory, not optional. Deduplicate on (topic, partition, offset) for Kafka, (stream, message-id) for Redis Streams, and the message_id header for RabbitMQ, persisted behind a UNIQUE index with a 7-day TTL. This converts at-least-once delivery into effectively-once side effects.
  • DLQs are native only to RabbitMQ. Kafka requires retry topics plus a terminal .dlq topic; Redis Streams requires a reclaim loop (XAUTOCLAIM / XPENDING) that re-publishes poison entries to a DLQ stream; RabbitMQ ships x-dead-letter-exchange and quorum x-delivery-limit out of the box.
  • Rebalance latency dominates tail latency. Default eager Kafka rebalancing stalls an entire consumer group for 2–9 s per membership change; cooperative sticky assignment (KIP-429) cuts that to 200–600 ms for only the partitions that actually move, and static membership (group.instance.id) eliminates most rebalances entirely.

The Partitioning Contract: Kafka vs Redis Streams vs RabbitMQ

Every high-throughput event-driven architecture is really an argument about one thing: who owns the ordering key, and how much of that key space can a single process own at a time.

Producers              Broker shards                        Consumer group members
─────────              ─────────────                        ──────────────────────
                       ┌───────────────┐
 svc-A ──┐             │ P0  o0 o1 o2  │ ──┐
         ├─ key=U17 ──▶│ P1  o0 o1 o2  │ ──┼──▶ member-1  (owns P0, P1)
 svc-B ──┘             │ P2  o0 o1 o2  │ ──┤
                       │ P3  o0 o1 o2  │ ──┼──▶ member-2  (owns P2, P3)
                       └───────────────┘   │
                                           └──▶ member-3  (idle → rebalance on join)

Kafka: the partition is the unit of everything

In Kafka, a partition is simultaneously the unit of ordering, the unit of parallelism, and the unit of replication. The group coordinator assigns each partition to exactly one consumer inside a group. That gives you a hard parallelism ceiling equal to the partition count — a 24-partition topic can never be consumed faster by 48 members than by 24, and 24 members on a 12-partition topic means 12 idle consumers holding JVM heap for nothing.

Rebalancing has evolved through three generations:

| Generation | Protocol | Behaviour | Tail cost | |---|---|---|---| | Legacy | Eager (Range/RoundRobin) | All members revoke all partitions, then reassign | 2–9 s stop-the-world | | KIP-429 | Cooperative Sticky | Members revoke only the partitions that move | 200–600 ms | | KIP-848 | Broker-side assignment (early access in 3.7, GA in 4.0) | Coordinator computes incremental deltas, no global barrier | 50–200 ms |

Redis Streams: one stream is one shard

A Redis stream is a single key backed by an append-only radix tree. XADD is O(1), XRANGE is O(N), and ordering is total per stream. Consumer groups are implemented with XREADGROUP GROUP g c COUNT 100 BLOCK 2000 STREAMS s '>', and every delivered entry is tracked in a per-consumer Pending Entries List (PEL). If a consumer dies, its entries remain in the PEL and can be reclaimed by another member.

That design has one brutal consequence: you cannot split a single stream across a Redis Cluster. To scale horizontally you shard client-side into stream:orders:{0..63} and route by CRC32(orderId) % 64. Each consumer group then owns a subset of shards — effectively reinventing Kafka partitions, without the broker's help.

RabbitMQ: queues, not partitions

RabbitMQ has no partition abstraction. A queue is a competing-consumer work queue, and once you have more than one consumer, FIFO ordering is only probabilistic. Quorum queues (Raft, 3 or 5 nodes) replaced classic mirrored queues, which were removed in RabbitMQ 4.0. Where RabbitMQ wins is routing expressiveness: per-message TTL, priorities, x-single-active-consumer, consistent-hash exchanges for pseudo-partitioning, and credit-based flow control that applies real backpressure instead of buffering to OOM.

Practical selection rule: Kafka for durable, replayable, high-fan-out event logs; Redis Streams for sub-millisecond fan-out where the working set fits in RAM; RabbitMQ for task distribution, routing complexity, and native DLQ semantics.

Delivery Semantics, Idempotent Consumers, and the Dedupe Key

No broker gives you exactly-once side effects. What they give you is a *stable, unique identifier per delivery*, and the rest is your job.

| Broker | Stable dedupe key | Where it comes from | Survives rebalance? | |---|---|---|---| | Kafka | topic + partition + offset | EachMessagePayload | Yes, offsets are immutable | | Redis Streams | stream + message-id (e.g. 1717430400000-3) | XREADGROUP reply | Yes, IDs are monotonic | | RabbitMQ | message_id / correlation_id header | Publisher must set it | Yes, if the publisher sets it | | Any (fallback) | SHA-256 of canonicalised body | Computed by consumer | No — breaks if the producer adds a timestamp |

Rule: never use a body hash as the primary dedupe key. It silently fails the moment a producer adds a sentAt field, and you get duplicates that your gate cannot see.

Producer-side idempotence is the first layer. Kafka's enable.idempotence=true assigns a Producer ID and a per-partition sequence number, and the broker rejects duplicates caused by retries — but only *within* a producer session, and only *inside* Kafka. The moment a side effect leaves the broker (Postgres write, Stripe charge, email send), consumer-side dedupe is the only thing standing between a rebalance and a double charge.

The canonical pattern is claim-then-work: insert the dedupe key with ON CONFLICT DO NOTHING and only proceed if you won the race. This is a compare-and-swap implemented by the database's unique index, so two consumers processing the same offset after a partition handover cannot both succeed.

> War story #1 — the 46,000 duplicate charges. A payments team ran a 24-partition charges.events topic with 12 consumers on 24 GB heaps. A STW garbage collection pause crossed the default max.poll.interval.ms=300000 boundary by 8 seconds. The coordinator evicted the member, rebalanced, and the partition was reassigned — while the old process, unaware it had been evicted, finished its in-flight batch and wrote 46,000 ledger rows. The idempotency table built on (topic, partition, offset) absorbed all but 3 duplicates, which happened because a nullable merchant_id column made the unique index non-enforcing for null values. Fix shipped in one afternoon: cooperative sticky assignment, max.poll.records=250, static membership via group.instance.id, and a NOT NULL constraint with a partial index.

Dead-Letter Queues When the Broker Has None

A DLQ is not a mailbox. It is a state machine for poison messages with four stages: attempt → backoff retry → terminal quarantine → human/automated replay with dedupe check.

 main topic ──▶ attempt 1..N ──▶ retry topic (x-delay-ms) ──▶ attempt N+1
                     │                                            │
                     │ exceeds MAX_ATTEMPTS                       │ still failing
                     ▼                                            ▼
              ┌──────────────┐                          ┌──────────────┐
              │  .dlq topic  │ ◀─────────────────────── │  .dlq topic  │
              └──────┬───────┘                          └──────────────┘
                     │
                     ▼  replay CLI (idempotency gate re-applied)
              back to main topic with x-attempt reset

Kafka. There is no native DLQ and no native delay queue. The standard implementation is a retry-topic chain (.retry.1, .retry.2, …) plus a terminal .dlq. Critically, never implement backoff with `consumer.pause()` — pausing blocks the whole partition, so one poison message stalls every other key hashed to it. Instead, publish to a retry topic with an x-delay-ms header and let a thin scheduler consumer hold it in a Redis sorted set (ZADD on score = now + delay, polled with ZRANGEBYSCORE) before re-publishing. Spring Kafka's @RetryableTopic (2.9+) implements exactly this shape.

Redis Streams. No DLQ, but the PEL makes it easy. Run a reclaimer that calls XAUTOCLAIM stream group consumer 60000 0-0 COUNT 100, inspects the attempt counter in the entry fields, and either reprocesses or XADDs the entry into stream.dlq before XACK-ing the original.

RabbitMQ. Native and declarative:

x-queue-type: quorum
x-delivery-limit: 5
x-dead-letter-exchange: events.dlx
x-dead-letter-routing-key: orders.dlq

For exponential backoff, point the DLX at a TTL queue whose own DLX routes back to the main exchange — a classic rabbit-hole pattern that gives you per-message delay without holding an Erlang process per message.

> War story #2 — Redis Streams trim ate 12,000 unacked events. An ingestion pipeline used XADD orders:* MAXLEN ~ 1000000. A downstream consumer was down for 90 seconds during a 40k msg/s burst. Retention trimming evicted entries that were still in the PEL — XPENDING returned nothing for them, and 12,000 events vanished with no error anywhere. Fix: switch to MINID-based retention driven by the *slowest consumer's* last-delivered ID, alert on XINFO GROUPS pending counts, and keep MAXLEN only as a hard safety cap far above the working set.

> War story #3 — RabbitMQ split-brain duplicates. Classic mirrored queues across a WAN partition produced two primaries, duplicate deliveries, and 400 MB of unroutable messages. Migration to 3-node quorum queues with x-delivery-limit=5 plus a DLX cost 22% throughput and eliminated duplicates entirely. Raft consensus is a throughput tax you pay for a correctness guarantee.

Practical Implementation

A production Kafka consumer with an idempotency gate, exponential backoff routing, and a terminal DLQ. The Postgres schema makes the dedupe check a single atomic round trip.

CREATE TABLE consumed_offsets (
  topic        text        NOT NULL,
  partition    integer     NOT NULL,
  kafka_offset bigint      NOT NULL,
  consumed_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (topic, partition, kafka_offset, consumed_at)
) PARTITION BY RANGE (consumed_at);

CREATE TABLE consumed_offsets_2025_06
  PARTITION OF consumed_offsets
  FOR VALUES FROM ('2025-06-01') TO ('2025-07-01');

-- Retention is a partition DROP, not a DELETE: zero bloat, instant.
-- Schedule: SELECT cron.schedule('drop-offsets', '0 3 * * *', ...);
import { Kafka, type EachMessagePayload, type Producer, type Consumer } from 'kafkajs';
import { Pool } from 'pg';

const TOPIC = 'orders.events.v1';
const RETRY_TOPIC = 'orders.events.v1.retry';
const DLQ_TOPIC = 'orders.events.v1.dlq';
const MAX_ATTEMPTS = 5;
const RETRY_BASE_MS = 250;
const MAX_DELAY_MS = 30_000;

interface OrderEvent {
  readonly eventId: string;
  readonly type: 'OrderPlaced' | 'OrderCancelled';
  readonly orderId: string;
  readonly amountCents: number;
  readonly occurredAt: string;
}

const db = new Pool({ connectionString: process.env.DATABASE_URL, max: 20 });

const kafka = new Kafka({
  clientId: 'orders-consumer',
  brokers: (process.env.KAFKA_BROKERS ?? 'localhost:9092').split(','),
});

const consumer: Consumer = kafka.consumer({
  groupId: 'orders-processor-v1',
  sessionTimeout: 30_000,
  heartbeatInterval: 3_000,
  rebalanceTimeout: 60_000,
  maxWaitTimeInMs: 100,
});

const producer: Producer = kafka.producer({
  idempotent: true,
  maxInFlightRequests: 5,
  allowAutoTopicCreation: false,
});

/**
 * Idempotency gate. The composite UNIQUE index makes this a compare-and-swap:
 * two consumers racing on the same offset after a rebalance cannot both win.
 */
async function claimOnce(topic: string, partition: number, offset: string): Promise<boolean> {
  const { rowCount } = await db.query(
    `INSERT INTO consumed_offsets (topic, partition, kafka_offset)
     VALUES ($1, $2, $3)
     ON CONFLICT (topic, partition, kafka_offset, consumed_at) DO NOTHING`,
    [topic, partition, offset],
  );
  return rowCount === 1;
}

async function drainBatch({ topic, partition, message }: EachMessagePayload): Promise<void> {
  if (message.value === null) return; // tombstone

  if (!(await claimOnce(topic, partition, message.offset))) return; // duplicate delivery

  const event = JSON.parse(message.value.toString()) as OrderEvent;
  const client = await db.connect();

  try {
    await client.query('BEGIN');
    await client.query(
      `INSERT INTO ledger_entries (event_id, order_id, amount_cents, created_at)
       VALUES ($1, $2, $3, $4)
       ON CONFLICT (event_id) DO NOTHING`,
      [event.eventId, event.orderId, event.amountCents, event.occurredAt],
    );
    // Transactional outbox: the domain write and the next-hop event commit together.
    await client.query(
      `INSERT INTO outbox (aggregate_id, payload) VALUES ($1, $2)`,
      [event.orderId, JSON.stringify({ type: 'OrderProcessed', eventId: event.eventId })],
    );
    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

async function routeFailure(payload: EachMessagePayload, err: unknown): Promise<void> {
  const { topic, partition, message } = payload;
  const attempt = Number(message.headers?.['x-attempt'] ?? '0') + 1;
  const headers: Record<string, string> = {
    'x-attempt': String(attempt),
    'x-error': String(err instanceof Error ? err.stack ?? err.message : err).slice(0, 2048),
    'x-origin': `${topic}/${partition}/${message.offset}`,
  };

  const isTerminal = attempt >= MAX_ATTEMPTS;
  if (!isTerminal) {
    const backoff = Math.min(RETRY_BASE_MS * 2 ** (attempt - 1), MAX_DELAY_MS);
    headers['x-delay-ms'] = String(backoff + Math.floor(Math.random() * 250)); // jitter
  }

  await producer.send({
    topic: isTerminal ? DLQ_TOPIC : RETRY_TOPIC,
    acks: -1,
    messages: [{ key: message.key, value: message.value, headers }],
  });
}

export async function run(): Promise<void> {
  await producer.connect();
  await consumer.connect();
  await consumer.subscribe({ topics: [TOPIC, RETRY_TOPIC], fromBeginning: false });

  await consumer.run({
    autoCommit: false,
    partitionsConsumedConcurrently: 4,
    eachMessage: async (payload) => {
      try {
        await drainBatch(payload);
      } catch (err) {
        // The retry/DLQ record now owns this event: never block the partition.
        await routeFailure(payload, err);
      } finally {
        await consumer.commitOffsets([
          {
            topic: payload.topic,
            partition: payload.partition,
            offset: (BigInt(payload.message.offset) + 1n).toString(),
          },
        ]);
      }
    },
  });
}

const shutdown = async (signal: string): Promise<void> => {
  console.info(`[shutdown] ${signal} — draining in-flight work`);
  await consumer.disconnect();
  await producer.disconnect();
  await db.end();
  process.exit(0);
};

process.on('SIGTERM', () => void shutdown('SIGTERM'));
process.on('SIGINT', () => void shutdown('SIGINT'));

void run();

The Redis Streams equivalent of the DLQ loop — this is the piece most teams forget to build:

import Redis from 'ioredis';

const redis = new Redis({ host: process.env.REDIS_HOST, maxRetriesPerRequest: null });

/**
 * Reclaims orphaned PEL entries (Redis 6.2+). XAUTOCLAIM scans for entries idle
 * longer than minIdleMs, transfers ownership, and returns up to COUNT entries.
 * Entries over the attempt limit are copied to a DLQ stream, then acked.
 */
export async function reclaimOrphans(
  stream: string,
  group: string,
  consumerName: string,
  minIdleMs = 60_000,
): Promise<number> {
  let cursor = '0-0';
  let reclaimed = 0;

  do {
    const reply = (await redis.xautoclaim(
      stream, group, consumerName, minIdleMs, cursor, 'COUNT', 100,
    )) as [string, [string, string[]][], string[]];

    const [nextCursor, entries] = reply;
    cursor = nextCursor;

    for (const [id, flat] of entries) {
      const idx = flat.indexOf('x-attempt');
      const attempt = idx === -1 ? 0 : Number(flat[idx + 1]);

      if (attempt >= 5) {
        await redis.xadd(`${stream}.dlq`, '*', 'origin', stream, 'origin-id', id, ...flat);
      }
      // Ack unconditionally: the entry now lives in the DLQ or was handled.
      await redis.xack(stream, group, id);
      reclaimed += 1;
    }
  } while (cursor !== '0-0');

  return reclaimed;
}

Performance Benchmarks & Trade-Offs

Measured on 3 × c7i.2xlarge (8 vCPU, 16 GB, gp3 EBS) with 1 KB events, 3-way replication, and a 50/50 read/write mix. Numbers are steady-state, not burst.

| Dimension | Apache Kafka 3.7 (RF=3, 24 partitions) | Redis Streams 7.2 (primary + replica, AOF everysec) | RabbitMQ 3.13 quorum queues (3 nodes) | |---|---|---|---| | Sustained throughput / node | 180k–420k msg/s | 90k–150k msg/s (500k–900k in-memory) | 25k–60k msg/s | | p50 / p99 publish latency | 1–3 ms / 6–15 ms | 0.2–0.6 ms / 1–4 ms | 2–5 ms / 12–25 ms | | Ordering guarantee | Per partition, total | Per stream, total | Per queue, FIFO until redelivery | | Parallelism ceiling | Partition count (fixed at creation) | 1 shard per stream key | Competing consumers, unbounded | | Retention / replay | Time or size based, unlimited offset replay | MAXLEN / MINID trim only, replay from any ID | Until consumed; replay only from DLX | | Native DLQ | No (retry-topic pattern) | No (XAUTOCLAIM + DLQ stream) | Yes (DLX + x-delivery-limit) | | Backpressure | Consumer pause / fetch limits (manual) | BLOCK + COUNT + MAXLEN | Credit-based flow control (best) | | Memory cost per in-flight message | ~0 (page cache + disk) | 60–120 B overhead + payload, RAM only | 1–3 KB per quorum message | | Ops complexity | High (KRaft, partitions, rebalance tuning) | Low (single process; Sentinel/Cluster for HA) | Medium (Erlang VM, Raft quorums) |

Choosing the dedupe store matters as much as choosing the broker:

| Dedupe store | Write throughput | Durability | Failure mode | Best fit | |---|---|---|---|---| | Postgres UNIQUE index, time-partitioned | 8k–25k inserts/s per primary | Full ACID, survives failover | Hot-partition contention above ~30k/s | Financial, regulated, irreversible side effects | | Redis SET key 1 NX EX 604800 | 400k–1.1M ops/s | AOF everysec → up to 1 s loss | Failover can replay one event | Idempotency where a rare duplicate is tolerable | | RedisBloom + DB fallback | ~1M ops/s | None (probabilistic) | False positives drop valid events | Pre-filter before the authoritative check | | Kafka transactions (read_committed) | ~same as Kafka | Scoped to Kafka topics only | Useless for external side effects | Kafka → Kafka pipelines |

Pro-Tips & Common Pitfalls

  • Never assume global ordering. Kafka orders per partition, Redis Streams per stream, RabbitMQ per queue — and RabbitMQ loses that order the moment two consumers share a queue. If you need per-entity ordering, hash the entity ID into the partition key, and never change that hash function without a migration plan.
  • Do not set `max.poll.interval.ms` above your worst-case batch time unless you also cap `max.poll.records`. The default 300 s hides rebalance storms until a GC pause exposes them. Use max.poll.records=250–500 and an interval of 3–5× the observed p99 batch duration.
  • Never `pause()` a partition for per-message backoff. It blocks every key on that partition. Route to a retry topic or a Redis sorted-set delay instead.
  • Never `XACK` before the side effect is durably committed. Acking early converts at-least-once into at-most-once, and the lost message produces no alert, no DLQ entry, and no metric.
  • Do not trim Redis Streams with `MAXLEN` alone. Set retention from the slowest consumer's last-delivered ID and alert when XINFO GROUPS pending counts grow.
  • Cap `maxInFlightRequests` (≤5) when idempotent producer is on. Higher values silently reorder within a partition even though the producer is deduplicating.
  • Guard your retry loop with a terminal attempt counter. A retry topic without x-attempt becomes an infinite loop with extra ceremony.
  • Never ship a DLQ without an owner, an alert, and a replay CLI. An unmonitored DLQ is a data-loss notification you chose not to read. Track DLQ rate as an SLI, target < 0.01% of events.
  • Watch consumer lag, not throughput. A consumer group at 99% utilisation looks healthy until the first 30-second spike turns into a multi-hour backlog.
  • Beware topic-per-tenant. Above ~1,000 topics per cluster, Kafka controller failover times and metadata propagation degrade every rebalance.
  • Publish from the outbox, not from the handler. Producing to Kafka inside a request after committing to the DB gives you a window where the DB has the change and the broker never hears about it.

Frequently Asked Questions (FAQ)

Is Kafka always faster than Redis Streams for event-driven architecture?

No. Redis Streams wins decisively on latency — p50 of 0.2–0.6 ms versus Kafka's 1–3 ms — and can be 2–4× faster per node on small events when persistence is relaxed (appendfsync no). Kafka wins on *sustained durability at scale*: it appends to immutable on-disk segments, lets the OS page cache batch fsyncs, and still sustains 180k–420k msg/s per broker at RF=3 with hours or days of replayable retention. Choose Redis Streams when your working set fits in RAM and sub-millisecond fan-out matters; choose Kafka when you need retention, replay from an arbitrary offset, and ordering across a sharded key space.

How do I guarantee exactly-once processing when neither Kafka nor Redis Streams has a native DLQ?

Build it from three layers. First, producer idempotence (enable.idempotence=true) plus broker transactions for any Kafka-to-Kafka hop, so retries never duplicate inside the log. Second, a transactional outbox so the domain write and the event publish commit atomically in the same database transaction. Third, a consumer-side idempotency gate: insert (topic, partition, offset) with ON CONFLICT DO NOTHING and only proceed if you won the race, then route failures through a retry chain ending in a terminal .dlq that stores the full x-error stack and origin coordinates. Exactly-once is never a broker configuration — it is a system property you construct and then prove with duplicate-rate metrics.

How many partitions should a high-throughput Kafka topic have?

Compute partitions = max(target_throughput / per_partition_throughput, desired_max_consumer_parallelism), then round up and add 2–3× headroom. A single modern partition delivers roughly 10–50 MB/s depending on message size and ack settings. A 300 MB/s target with 1 KB events (~300k msg/s) typically lands at 32–48 partitions. Two constraints to respect: you can *increase* partitions but never decrease them, and increasing partitions changes the hash(key) % N mapping, breaking ordering for existing keys unless you migrate. Over-partitioning above ~10k partitions per cluster inflates controller failover time and end-to-end latency, so treat large partition counts as a scaling cost, not a free win.