Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Tech

Event-Driven Architecture: Kafka vs Redis Streams vs RabbitMQ

By: Md Tarek · 17 Oct 2026

Event-driven architecture comparing Kafka, Redis Streams and RabbitMQ with partitioned consumer groups, idempotent consumers and dead-letter queues
Event-driven architecture comparing Kafka, Redis Streams and RabbitMQ with partitioned consumer groups, idempotent consumers and dead-letter queues

Picking an event backbone is not a benchmark contest. It is a decision about where ordering lives, where backpressure lives, and who owns the retry contract. This note compares Apache Kafka, Redis Streams and RabbitMQ at the level of partitioned consumer groups, idempotency keys and dead-letter ladders, with reference numbers from a 3-node cluster and code you can ship.

Executive Summary & Key Takeaways

  • Kafka owns sustained throughput and replay. A 3-broker cluster (8 vCPU / 32 GB / NVMe, acks=all, min.insync.replicas=2, ~1 KiB events, zstd) sustains 300k–800k msgs/s aggregate produce and 1M+ msgs/s consume across 6 consumers; p99 produce latency is 5–12 ms and end-to-end consumer p99 is 15–40 ms, dominated by fetch.min.bytes and fetch.max.wait.ms.
  • Redis Streams owns latency and operational simplicity. XADD to XREADGROUP p99 is 0.3–1.0 ms in-AZ at 60k–90k msgs/s per shard, but a stream is a single key inside one cluster slot: there is no broker-side partitioning, so scale-out means N streams plus a shard key in the stream name.
  • RabbitMQ owns routing and native dead-lettering. Topic/headers exchanges plus x-dead-letter-exchange and quorum-queue delivery-limit give you retries with almost no application code, at ~8k–25k msgs/s per queue with publisher confirms and manual acks (100k+ with RabbitMQ Streams).
  • Exactly-once is a system property, not a broker flag. Kafka transactions cover Kafka to Kafka only; crossing into Postgres or Redis requires a durable dedupe key written in the same transaction as the business effect, enforced by PRIMARY KEY (consumer_group, event_id).
  • A DLQ is the last rung of a retry ladder, not a graveyard. Three tiers (retry.5s, retry.1m, retry.30m) then a terminal DLT caps blast radius; retrying in place is the single most common cause of multi-million-record consumer lag incidents.

How Partitioned Consumer Groups Actually Work

Kafka: the partition is the only unit of parallelism

A producer maps a record to a partition with partition = toPositive(murmur2(keyBytes)) % numPartitions. Null-keyed records use the sticky partitioner (KIP-480, Kafka 2.4+) and batch into one partition per linger.ms window — good for throughput, hostile to ordering. Within a consumer group, a partition can be owned by at most one member, so parallelism is capped by partition count, and surplus consumers sit idle.

topic orders.events.v1  (6 partitions, replication.factor=3)

 P0 ─┐                     consumer group: order-projector-v3
 P1 ─┤──▶ consumer-A  owns P0,P1
 P2 ─┤──▶ consumer-B  owns P2,P3
 P3 ─┤──▶ consumer-C  owns P4,P5
 P4 ─┤──▶ consumer-D  idle (no partitions left to assign)
 P5 ─┘

The knobs that actually move the needle in production:

  • `partitionAssigners: [PartitionAssigners.cooperativeSticky]` — incremental cooperative rebalancing (KIP-429). Members keep the partitions they already own; a rolling deploy goes from N stop-the-world rebalances to roughly 1.
  • Static membership (`group.instance.id`) — a pod restart inside session.timeout.ms no longer triggers a rebalance at all.
  • `max.poll.interval.ms` (default 300000) — the silent killer. If one batch takes longer, the broker evicts the member mid-flight, its partitions move, and every in-progress record is reprocessed by someone else. Budget it from your p99.9 batch duration, not the average.
  • `fetch.min.bytes` / `fetch.max.wait.ms` — setting fetch.min.bytes=65536 buys ~2x throughput and costs ~15 ms of p50 latency. On a low-volume topic this is the difference between 4 ms and 40 ms end-to-end.

Redis Streams: consumer groups without partitions

Redis Streams have real consumer groups (XGROUP CREATE, XREADGROUP, XACK) and a genuine at-least-once contract via the PEL — the per-group Pending Entries List — but no partitioning. A stream is one key; in Redis Cluster one key lives in one slot on one node. You cannot add brokers to make a single stream faster.

  • XREADGROUP GROUP g c COUNT 100 BLOCK 2000 STREAMS s > — the > ID means new messages only; a numeric ID means replay of *this consumer's* PEL.
  • XPENDING s g - + 10 returns [id, consumer, idleMs, deliveryCount] — your only signal for poison detection.
  • XAUTOCLAIM s g c 60000 0-0 COUNT 50 (Redis 6.2+) steals entries idle longer than 60 s from dead consumers. Without it, a crashed pod leaks PEL entries forever and those messages are never retried.
stream orders.events (one key -> one cluster slot -> one node)
  group order-projector
    consumer-A  PEL: [1731000000000-1, 1731000000000-4]
    consumer-B  PEL: [1731000000000-2]
  XACK removes an entry from the PEL (delivery bookkeeping, not storage)
  XAUTOCLAIM MIN-IDLE-TIME steals entries from a dead consumer

Durability ceiling: AOF appendfsync everysec loses up to 1 s of acknowledged writes on failover. If that is unacceptable, WAIT 1 100 forces a synchronous replica ack at a cost of roughly 0.4 ms per write.

RabbitMQ: queues, prefetch and super streams

RabbitMQ is a smart broker / dumb consumer system: routing happens in the broker via exchanges (direct, topic, headers, fanout), and the backpressure knob is basic.qos prefetch, not partition count. Quorum queues are Raft-replicated and support x-delivery-limit (3.12+/4.x), which dead-letters a message automatically after N redeliveries — the closest thing to a built-in retry ladder in this comparison. Note that RabbitMQ 4.0 removed classic mirrored queues; HA now means quorum queues, and per-queue throughput drops accordingly.

Ordering is only guaranteed with a single active consumer per queue, which caps that queue at one core. To scale ordered processing you need either a consistent-hash exchange (partition by routing key) or super streams (RabbitMQ Streams 3.11+, partitioned and replayable), which is RabbitMQ's direct answer to Kafka partitions.

Idempotency, Retry Ladders and Dead-Letter Queues

Why at-least-once is the only honest default

Every crash window between handler() and commit() produces a duplicate. Producer-side idempotence (enable.idempotence=true, PID + per-partition sequence numbers) only dedupes producer retries within one producer session — it does nothing for a consumer that crashed after writing to Postgres but before committing its offset.

Kafka's exactly-once semantics close that gap *inside Kafka*: a transactional producer, isolation.level=read_committed on the consumer, and offsets committed through sendOffsetsToTransaction(). The moment your side effect is a Postgres row or an HTTP call, you are back to at-least-once and need your own dedupe.

Three idempotency strategies (ranked)

1. Natural idempotency via upsert on a business key. INSERT ... ON CONFLICT (order_id) DO UPDATE is free — use it whenever the event carries full desired state rather than a delta. Fails for accumulation (+=) semantics. 2. Dedupe table with a unique constraint. One extra write per event (~0.2 ms at 10k writes/s on NVMe), atomic with the business effect because it shares the transaction. This is the general-purpose answer. 3. Redis `SETNX` with a TTL. Sub-millisecond, but not atomic with your database write: a crash between the two leaves a permanently lost event or a permanently suppressed one. Only acceptable when double-processing is cheaper than data loss.

The retry ladder

orders.events.v1 ──▶ handler ──ok──▶ commit offset
                        │
                        ├─ transient #1 ──▶ retry.5s    ─┐
                        ├─ transient #2 ──▶ retry.1m    ─┤── same handler,
                        ├─ transient #3 ──▶ retry.30m   ─┘   x-attempt + 1
                        ├─ transient #4 ──▶ dlt ──▶ page on-call, replay tool
                        └─ schema error ──▶ dlt immediately (never burns the ladder)

Hard rules: preserve the message key on every hop so a retried event stays on the same partition as its siblings; stamp x-not-before so retry-topic consumers can pause() the partition instead of sleeping in the handler; carry x-origin-topic, x-origin-partition and x-origin-offset so the DLT is replayable; and page on DLT depth > 0, not on a threshold.

> Production war story. A payments projector retried a provider 429 in place. Partition 7 stalled for 41 minutes while the provider rate-limited us; group lag crossed 4.2M records and the on-call engineer's first instinct — scaling consumers 2x — changed nothing because the partition was already owned. After moving to the ladder above, worst-case partition stall dropped to a single 10 s handler timeout, and p99 lag stayed under 2 s through the next provider incident.

Practical Implementation

The DDL first — the unique constraint *is* the idempotency guarantee:

CREATE TABLE IF NOT EXISTS processed_events (
  consumer_group text        NOT NULL,
  event_id       uuid        NOT NULL,
  processed_at   timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer_group, event_id)   -- dedupe, enforced by the engine
);

CREATE TABLE IF NOT EXISTS order_projection (
  order_id     uuid PRIMARY KEY,
  status       text        NOT NULL,
  amount_cents bigint      NOT NULL CHECK (amount_cents >= 0),
  occurred_at  timestamptz NOT NULL
);

-- Reclaim space only beyond retention + max consumer lag; replays must still dedupe.
DELETE FROM processed_events WHERE processed_at < now() - interval '30 days';
/**
 * order-projector.ts — at-least-once Kafka consumer with Postgres-backed
 * idempotency, a 3-tier retry ladder, and a terminal dead-letter topic.
 *
 * Node 20+, kafkajs 2.2.x, pg 8.x, zod 3.x.
 * kafkajs is in maintenance mode; the drop-in successor is
 * @confluentinc/kafka-javascript (librdkafka). Semantics below are identical.
 */
import { Kafka, logLevel, PartitionAssigners, type EachMessagePayload } from 'kafkajs';
import { Pool, type PoolClient } from 'pg';
import { z } from 'zod';

const log = { info: console.info, warn: console.warn, error: console.error };

const GROUP_ID = 'order-projector-v3';
const SOURCE_TOPIC = 'orders.events.v1';
const DLT = 'orders.events.dlt';
const RETRY_TIERS = [
  { topic: 'orders.events.retry.5s', delayMs: 5_000 },
  { topic: 'orders.events.retry.1m', delayMs: 60_000 },
  { topic: 'orders.events.retry.30m', delayMs: 1_800_000 },
] as const;
const MAX_ATTEMPTS = RETRY_TIERS.length + 1; // 3 retries, then DLT
const HANDLER_TIMEOUT_MS = 10_000;

const OrderEvent = z.object({
  eventId: z.string().uuid(),
  type: z.enum(['order.created', 'order.paid', 'order.cancelled']),
  orderId: z.string().uuid(),
  amountCents: z.number().int().nonnegative(),
  occurredAt: z.string().datetime(),
});
type OrderEvent = z.infer<typeof OrderEvent>;

/** Non-retryable: malformed payloads must never burn the ladder. */
class PoisonMessageError extends Error {}

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 20,
  idleTimeoutMillis: 30_000,
  connectionTimeoutMillis: 2_000,
});

const kafka = new Kafka({
  clientId: 'order-projector',
  brokers: (process.env.KAFKA_BROKERS ?? '').split(',').filter(Boolean),
  logLevel: logLevel.WARN,
  retry: { initialRetryTime: 300, retries: 10 },
});

const consumer = kafka.consumer({
  groupId: GROUP_ID,
  // Cooperative-sticky (KIP-429): members keep partitions they already own,
  // so a rolling deploy stops being a stop-the-world rebalance.
  partitionAssigners: [PartitionAssigners.cooperativeSticky],
  sessionTimeout: 45_000,
  heartbeatInterval: 3_000,
  // Exceed this and the broker evicts us mid-batch; every in-flight record
  // is then reprocessed by another member — the classic duplicate storm.
  maxPollInterval: 300_000,
  maxBytesPerPartition: 1_048_576,
});

const producer = kafka.producer({
  idempotent: true,       // PID + sequence numbers kill producer-side dupes
  maxInFlightRequests: 1, // required for idempotence without transactions
  allowAutoTopicCreation: false,
});

function statusFor(type: OrderEvent['type']): string {
  return { 'order.created': 'CREATED', 'order.paid': 'PAID', 'order.cancelled': 'CANCELLED' }[type];
}

/** Apply the event exactly once against the projection. */
async function applyEvent(evt: OrderEvent): Promise<'applied' | 'duplicate'> {
  let client: PoolClient | undefined;
  try {
    client = await pool.connect();
    await client.query('BEGIN');
    const inserted = await client.query(
      `INSERT INTO processed_events (consumer_group, event_id)
       VALUES ($1, $2) ON CONFLICT DO NOTHING RETURNING event_id`,
      [GROUP_ID, evt.eventId],
    );
    if (inserted.rowCount === 0) {
      await client.query('ROLLBACK');
      return 'duplicate';
    }
    await client.query(
      `INSERT INTO order_projection (order_id, status, amount_cents, occurred_at)
       VALUES ($1, $2, $3, $4)
       ON CONFLICT (order_id) DO UPDATE
         SET status = EXCLUDED.status,
             amount_cents = EXCLUDED.amount_cents,
             occurred_at = EXCLUDED.occurred_at`,
      [evt.orderId, statusFor(evt.type), evt.amountCents, evt.occurredAt],
    );
    await client.query('COMMIT');
    return 'applied';
  } catch (err) {
    if (client) await client.query('ROLLBACK').catch(() => undefined);
    throw err;
  } finally {
    client?.release();
  }
}

async function withTimeout<T>(p: Promise<T>, ms: number): Promise<T> {
  let timer: NodeJS.Timeout | undefined;
  const timeout = new Promise<never>((_, reject) => {
    timer = setTimeout(() => reject(new Error(`handler timed out after ${ms}ms`)), ms);
  });
  try {
    return await Promise.race([p, timeout]);
  } finally {
    if (timer) clearTimeout(timer);
  }
}

/** Publish to the next rung of the ladder, preserving the partition key. */
async function routeFailure(
  payload: EachMessagePayload,
  evt: OrderEvent | undefined,
  attempt: number,
  reason: string,
): Promise<void> {
  const raw = payload.message.value?.toString('utf8') ?? '';
  const exhausted = attempt >= MAX_ATTEMPTS - 1;
  const tier = RETRY_TIERS[Math.min(attempt, RETRY_TIERS.length - 1)];
  await producer.send({
    topic: exhausted ? DLT : tier.topic,
    messages: [
      {
        key: payload.message.key, // keeps per-order ordering across retries
        value: raw,
        headers: {
          ...payload.message.headers,
          'x-event-id': evt?.eventId ?? 'unparsed',
          'x-attempt': String(attempt + 1),
          'x-not-before': String(Date.now() + tier.delayMs),
          'x-error': reason.slice(0, 900),
          'x-origin-topic': SOURCE_TOPIC,
          'x-origin-partition': String(payload.partition),
          'x-origin-offset': String(payload.message.offset),
        },
      },
    ],
  });
}

async function handle(payload: EachMessagePayload): Promise<void> {
  const { message, partition } = payload;
  const attempt = Number(message.headers?.['x-attempt'] ?? 0);
  const body = message.value?.toString('utf8') || '{}';

  // 1. Validate. A malformed payload is a producer bug, not a transient fault.
  const parsed = OrderEvent.safeParse(JSON.parse(body));
  if (!parsed.success) {
    await routeFailure(payload, undefined, MAX_ATTEMPTS - 1, `schema: ${parsed.error.message}`);
    return;
  }
  const evt = parsed.data;

  // 2. Apply idempotently.
  try {
    const outcome = await withTimeout(applyEvent(evt), HANDLER_TIMEOUT_MS);
    if (outcome === 'duplicate') {
      log.warn({ eventId: evt.eventId, partition }, 'duplicate suppressed');
    }
  } catch (err) {
    // 3. Poison skips the ladder; transient escalates it. Never block here.
    const terminal = err instanceof PoisonMessageError;
    await routeFailure(payload, evt, terminal ? MAX_ATTEMPTS - 1 : attempt, (err as Error).message);
  }
}

async function main(): Promise<void> {
  await producer.connect();
  await consumer.connect();
  await consumer.subscribe({
    topics: [SOURCE_TOPIC, ...RETRY_TIERS.map((t) => t.topic)],
    fromBeginning: false,
  });

  // Never let an exception escape eachMessage: kafkajs would re-deliver the
  // same record in a tight loop and stall the partition until max.poll evicts us.
  await consumer.run({ autoCommit: true, eachMessage: handle });

  const shutdown = async (signal: string): Promise<void> => {
    log.info({ signal }, 'draining');
    await consumer.disconnect(); // stops the fetch loop, commits last offsets
    await producer.disconnect();
    await pool.end();
    process.exit(0);
  };
  process.once('SIGTERM', () => void shutdown('SIGTERM'));
  process.once('SIGINT', () => void shutdown('SIGINT'));
}

main().catch((err) => {
  log.error({ err }, 'fatal startup failure');
  process.exit(1);
});

Retry-topic consumers honour x-not-before by calling consumer.pause([{ topic, partitions: [partition] }]) and scheduling a resume() at that timestamp — omitted above for brevity, but without it the ladder only counts attempts and never buys backoff time.

The Redis Streams equivalent of the same handler is shorter but you own the reclaim loop:

// ioredis 5.x — reclaim-safe consumer loop
const stream = 'orders.events';
const group = 'order-projector';
const me = `pod-${process.env.HOSTNAME}`;

await redis.xgroup('CREATE', stream, group, '$', 'MKSTREAM').catch((e: Error) => {
  if (!e.message.includes('BUSYGROUP')) throw e; // idempotent group creation
});

for (;;) {
  // Steal entries orphaned by dead consumers (idle > 60s) before reading new ones.
  const [, claimed] = (await redis.xautoclaim(stream, group, me, 60_000, '0-0', 'COUNT', 50)) as [
    string, [string, string[]][], string[]
  ];
  const res = await redis.xreadgroup('GROUP', group, me, 'COUNT', 100, 'BLOCK', 2000, 'STREAMS', stream, '>');
  const fresh = (res?.[0]?.[1] ?? []) as [string, string[]][];

  for (const [id, fields] of [...claimed, ...fresh]) {
    try {
      await applyEvent(parseFields(fields));
      await redis.xack(stream, group, id); // removes it from the PEL
    } catch (err) {
      const [, , , deliveries] = await redis.xpending(stream, group, id, id, 1).then((r: any) => r[0]);
      if (deliveries >= 4) {
        await redis.xadd('orders.events.dlt', '*', 'id', id, 'error', String(err));
        await redis.xack(stream, group, id); // stop infinite redelivery
      }
      // otherwise leave it pending: XAUTOCLAIM retries it on the next pass
    }
  }
}

Performance Benchmarks & Trade-Offs

Reference setup: 3 nodes, 8 vCPU / 32 GB / NVMe, 3 replicas, ~1 KiB JSON events, zstd on Kafka, 50/50 read/write, clients in the same AZ. Treat these as order-of-magnitude planning numbers and re-measure on your hardware.

| Dimension | Apache Kafka 4.x | Redis Streams 7.x | RabbitMQ 4.x (quorum queues) | |---|---|---|---| | Sustained produce throughput | 300k–800k msgs/s aggregate | 60k–140k msgs/s per shard | 8k–25k msgs/s per queue (100k+ with Streams) | | p99 end-to-end latency | 15–40 ms | 0.3–1.0 ms | 4–9 ms | | Ordering guarantee | per partition | per stream | per queue, single active consumer | | Parallelism unit | partition | consumer in group (no data partitioning) | queue / super-stream partition | | Backpressure model | pull (fetch.min.bytes, max.poll.records) | pull + BLOCK | push + basic.qos prefetch | | Replay / rewind | yes (time or size retention) | bounded by RAM (MAXLEN, MINID) | no, unless RabbitMQ Streams | | Native dead-lettering | no — build retry topics | no — PEL + XADD to a DLT stream | yes (x-dead-letter-exchange, x-delivery-limit) | | Durability floor | acks=all + min.insync.replicas=2 | AOF everysec (≤1 s loss) or WAIT | Raft majority in a quorum queue | | Operational burden | high (partitions, rebalances, capacity) | very low | medium | | Failure blast radius | partition-local | node-local (one slot) | queue-local |

| Decision axis | Kafka | Redis Streams | RabbitMQ | |---|---|---|---| | Need replay / event sourcing | best fit | weak (RAM-bound) | only with Streams | | Need sub-5 ms p99 at modest volume | overkill | best fit | good fit | | Need complex routing (fan-out, headers) | consumer-side | consumer-side | best fit (exchanges) | | Need 100k+ msg/s with ordering | best fit | shard manually | Streams only | | Team has no platform engineer | risky | best fit | good fit |

Pro-Tips & Common Pitfalls

  • Never retry in place. A handler that loops on a downstream 429 stalls a whole partition. Escalate to a retry topic and commit the original offset.
  • Always set an explicit message key. Null keys plus a sticky partitioner destroy per-entity ordering silently — no error, no metric, just out-of-order order.paid before order.created.
  • Derive `max.poll.interval.ms` from p99.9 batch duration. max.poll.records=500 at 30 ms/record is 15 s per batch; a 10 s default in old clients guaranteed rebalance storms.
  • Commit offsets after the business transaction, never before. This is the whole game: commit-then-process is at-most-once, process-then-commit is at-least-once plus a dedupe key.
  • Producer-generated UUIDv7 event IDs, not payload hashes. Two legitimate order.updated events with identical fields are distinct events, and hashing collapses them.
  • Store `x-origin-topic`, `x-origin-partition`, `x-origin-offset` on the DLT. Without them your replay tool cannot partition-safely re-inject, and you reintroduce the ordering bug you just fixed.
  • Alert on `estimated_time_lag` (seconds), not message count. 500k messages is healthy at 2M msg/s and fatal at 2k msg/s.
  • Redis Streams: always run `XAUTOCLAIM` in the loop. Without reclaim, a SIGKILLed pod leaves its PEL entries orphaned and those events are never delivered to anyone.
  • Redis Streams: cap memory explicitly. XADD s MAXLEN ~ 2000000 * or MINID; an uncapped stream is an OOM waiting for its moment.
  • RabbitMQ: prefetch 20–50, not 250. High prefetch plus a slow consumer equals a queue that looks empty while latency climbs.
  • RabbitMQ: set an `alternate-exchange`. Unroutable messages are silently dropped by default — the most expensive bug class in AMQP.
  • Never put the DLT inside the same consumer group as the source topic. You will build an infinite loop that looks like progress.

Frequently Asked Questions (FAQ)

Which is better for event-driven architecture: Kafka, RabbitMQ, or Redis Streams?

It depends on your ordering and replay requirements. Choose Kafka when you need sustained 100k+ msg/s, multi-day replay, or stream processing (Kafka Streams, Flink); it costs the most operational effort. Choose RabbitMQ when routing complexity dominates — topic/headers exchanges, per-message TTL, native dead-lettering — and per-queue throughput under ~25k msg/s is fine. Choose Redis Streams when you need sub-millisecond p99 at moderate volume (under ~100k msg/s per shard) and already run Redis, accepting RAM-bounded retention and manual client-side sharding. A common production hybrid is RabbitMQ for command/routing traffic and Kafka for the durable event log.

How do I make a Kafka consumer idempotent with at-least-once delivery?

Insert a dedupe row keyed by (consumer_group, event_id) in the same database transaction as the business effect, and let the unique constraint decide. In Postgres: INSERT INTO processed_events (consumer_group, event_id) VALUES ($1,$2) ON CONFLICT DO NOTHING RETURNING event_id — a rowCount of 0 means duplicate, so roll back and commit the offset. Generate event_id in the producer (UUIDv7) so retries of the same logical event carry the same ID. Kafka transactions with sendOffsetsToTransaction() give you exactly-once only for Kafka-to-Kafka pipelines; they cannot make a Postgres write atomic with an offset commit.

How many partitions should a Kafka topic have?

Size by peak consumer parallelism, not average throughput: partitions >= max(peakMsgPerSec / perConsumerMsgPerSec, peakBytesPerSec / perConsumerBytesPerSec), then round up for growth headroom. Partition count is a ceiling you can raise but never lower, and raising it remaps every keyed record — breaking per-key ordering during the transition — so over-provision by ~2x rather than resizing later. Each partition costs file handles, replication threads and controller metadata: keep per-cluster partition counts in the low hundreds of thousands, and remember that more partitions means longer leader-election and rebalance times, which shows up directly in your p99.