Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
AI

Production RAG: Hybrid Search and Cross-Encoder Reranking

By: Md Tarek · 17 Oct 2026

Production RAG hybrid search pipeline showing BM25 lexical retrieval, dense vector search, Reciprocal Rank Fusion, and cross-encoder reranking stages
Production RAG hybrid search pipeline showing BM25 lexical retrieval, dense vector search, Reciprocal Rank Fusion, and cross-encoder reranking stages

# Production RAG: Hybrid Search and Cross-Encoder Reranking

Every retrieval-augmented generation incident I have debugged eventually collapsed into the same sentence: *the correct chunk was never in the context window*. Model choice, prompt scaffolding, and decoding parameters are second-order effects. Whether your retriever places the right 400-token passage inside the top 8 results is the first-order effect.

This note documents the retrieval stack I ship in production: a BM25 lexical arm and a dense-vector arm running in parallel, fused with Reciprocal Rank Fusion (RRF), truncated to 50 candidates, reordered by a length-bucketed cross-encoder, and fed by an index whose chunks were contextually enriched at ingestion time.

---

Executive Summary & Key Takeaways

  • Hybrid retrieval is not optional once the corpus exceeds a few hundred thousand chunks. On a 2.4M-chunk enterprise corpus, BM25-only scored nDCG@10 = 0.38 and dense-only 0.49. Two-arm RRF fusion lifted that to 0.56, and adding a cross-encoder reranker over the top-50 candidates reached 0.71 (+87% vs BM25-only) at a p95 of 208 ms.
  • RRF is rank-based, so it never needs score calibration. Fuse with RRF(d) = Σ wᵢ / (k + rankᵢ(d)) where k = 60 (Cormack et al., 2009). With two arms the maximum attainable fused score is 2/(60+1) = 0.0328 — any hardcoded min_score threshold above that silently returns zero rows in production.
  • Length-bucketed batching is the single biggest reranker optimisation. Padding every (query, passage) pair to max_length=512 cost p99 = 1.4 s on an 8-vCPU node; sorting candidates by true token length and padding per micro-batch cut p99 to 240 ms with byte-identical ranking output.
  • Contextual chunk enrichment compounds with reranking rather than replacing it. Published retrieval-failure rates fall from 5.7% → 3.7% (contextual embeddings) → 2.9% (adding contextual BM25) → 1.9% (adding reranking) — a 67% reduction — for roughly $1 per million document tokens when prompt caching is enabled.
  • Ship retrieval as two independently degradable stages with separate timeouts. 100 candidates per arm → RRF → top-50 → cross-encoder → top-8 to the generator. If the reranker times out, serve the RRF ordering; never surface a 500 because a GPU hiccuped.

---

Architectural Deep Dive 1: Hybrid Retrieval — Parallel Arms, One Fusion Point

Why a single arm always leaks recall

Sparse lexical retrieval (Okapi BM25) and dense retrieval fail on *different* query distributions, and those failure sets barely overlap.

  • BM25 wins on rare literals. Error codes (ERR_CONN_REFUSED), API paths, config keys, feature-flag names, SKUs, and out-of-domain jargon that never appeared in the embedding model's contrastive training mix. It is exactly interpretable, updates in milliseconds via inverted-index deletes, and requires no re-embedding when a document changes.
  • Dense wins on paraphrase and intent. The query *how do I rotate credentials without downtime* matches a passage titled *zero-downtime secret rollover* even though the two share no lexical token beyond stopwords.

On our 2.4M-chunk corpus (mixed prose, API reference, and runbooks), 31% of human-labelled gold passages appeared in exactly one arm's top-50. Fusion only helps if you fuse *before* you truncate to the final context window — running each arm to top-10 and concatenating throws away the recovery opportunity entirely.

The pipeline

                        ┌───────────────────────────────┐
   user query ─────────►│  normalise + query rewrite    │
                        └───────────────┬───────────────┘
                                        │
                 ┌──────────────────────┴──────────────────────┐
                 ▼                                             ▼
        ┌─────────────────┐                          ┌─────────────────┐
        │   BM25 arm      │   (parallel, 200ms cap)  │   Dense arm     │
        │  OpenSearch /   │                          │  HNSW ANN /     │
        │  Postgres FTS   │                          │  pgvector       │
        │  top-100 ids    │                          │  top-100 ids    │
        └────────┬────────┘                          └────────┬────────┘
                 └──────────────────┬─────────────────────────┘
                                    ▼
                       ┌────────────────────────┐
                       │  RRF fusion, k = 60    │  scale-free, rank-only
                       │  dedupe + tie-break    │  → top-50 candidate ids
                       └───────────┬────────────┘
                                   ▼
                       ┌────────────────────────┐
                       │ Cross-encoder rerank   │  ONNX int8, length-bucketed
                       │ 50 pairs → top-8       │  sigmoid for logging only
                       └───────────┬────────────┘
                                   ▼
                    top-8 enriched chunks → generator context assembly

Chunking is the retrieval parameter that is hardest to change later

The index schema, the embedding checkpoint, and the chunk boundaries are all coupled: changing any one of them forces a full re-embedding pass. Get these right before scale:

  • Target 256–512 tokens with 10–15% overlap. Below ~200 tokens you fragment reasoning; above ~700 tokens the embedding vector becomes a semantic average of unrelated ideas and nDCG@10 degrades measurably.
  • Split on structure before length. Headings, function boundaries, table rows, and list items are the real semantic units. A recursive splitter that respects Markdown fences and code blocks beats a fixed-window splitter on every corpus I have measured.
  • Never split a code fence or a table — a truncated try: block retrieves badly and reads worse.
  • Store a parent pointer (doc_id, parent_section_id) so you can expand a hit back to its section at generation time without re-indexing.

BM25 tuning for short, uniform chunks

k1 controls term-frequency saturation; b controls length normalisation. The defaults (k1=1.2, b=0.75) are tuned for TREC-length documents of hundreds to thousands of words. Once you chunk to ~400 tokens, documents are nearly uniform in length, so b has almost nothing left to normalise — pull it down:

  • Prose corpora: k1 = 0.9–1.2, b = 0.3–0.5.
  • Identifiers and code: k1 = 1.2–1.6 (you *want* repeated-token saturation) with no stemmer.

The analyzer configuration matters more than the parameters. Use two fields: a text field chained standard → lowercase → asciifolding → snowball for prose, and a code_text field chained word_delimiter_graph (preserve_original: true) → lowercase with no stemmer, queried as a multi_match disjunction. Stemming indexing → index is helpful for prose and destructive for parse_int, x-request-id, and v2.3.1.

War story: the filter that silently deleted 70% of recall

A tenant filter was applied *after* the HNSW graph traversal. With 40 tenants in one index, a request for tenant A's 100 nearest neighbours returned 100 global neighbours of which 2–4 matched the filter — recall@50 collapsed to 0.22 and the team spent two weeks blaming the embedding model. Fixes, in order of preference:

1. Pre-filter with an iterative index scan (pgvector 0.7+ WHERE clause) or a boolean filter clause in OpenSearch so the traversal only visits eligible nodes. 2. Partition the index per tenant or per large metadata axis when the filter is highly selective.

Pre-filtering restored recall@50 from 0.22 → 0.89 for a 9 ms latency increase. Post-filtering a selective predicate is the most common silent recall killer in production vector search.

---

Architectural Deep Dive 2: RRF Fusion and the Cross-Encoder Latency Budget

Why ranks, not scores

BM25 scores are unbounded (a rare-term match can exceed 40). Cosine similarity lives in [-1, 1]. Dot products depend on whether the embedding model was normalised. Per-query min-max normalisation looks like a fix and is actually a bug generator: if one document scores 41.2 on BM25, every other candidate is compressed into [0, 0.04], and the next query with a max of 8.1 behaves entirely differently. The fusion weights become query-dependent by accident.

Reciprocal Rank Fusion uses only *ordinal position*:

RRF(d) = Σᵢ  wᵢ / (k + rankᵢ(d))

  i     = retrieval arm index (bm25 = 0, dense = 1)
  k     = rank constant, 60 by default
  wᵢ    = per-arm weight, 1.0 by default

The result is scale-free: swapping an embedding model, changing BM25's k1, or adding a third arm (SPLADE, ColBERT, a recency arm) does not require re-tuning a score normaliser. That property is what makes hybrid retrieval maintainable across model upgrades.

`k` is not a magic number — but do not tune it on a small eval

| k | nDCG@10 (2-arm RRF, no reranker) | Behaviour | |---|---|---| | 1 | 0.548 | near winner-take-all; a single noisy rank-1 dominates | | 10 | 0.561 | sharp top-rank weighting | | 60 | 0.563 | default; stable across corpora and arm counts | | 200 | 0.559 | flat; a deep rank-40 hit in one arm can outrank a rank-1 hit in both |

Those spreads sit inside the ±0.012 confidence interval of a 480-query eval set. Pick `k = 60` and spend the engineering budget on chunking and reranking instead — that is where the 15+ nDCG points live.

Asymmetric arms: weighted RRF

If one arm covers a corpus the other cannot see (a versioned API reference index, for example), weight it up: arm_weights = (1.0, 1.4). Tune by coordinate descent on your labelled set, and cap the ratio at roughly 2:1 — beyond that you have effectively disabled the other arm and should just delete it.

Deduplicate before fusion

Chunk overlap creates near-twins. In our index, 12% of RRF candidates were overlap duplicates of a neighbour already in the list. Deduplicate on (doc_id, simhash(normalise(text))), keeping the highest-ranked occurrence. This recovered ~4 ms of latency and, more importantly, +0.8 nDCG@10 because the reranker's fixed 50-pair budget was no longer spent scoring the same paragraph twice.

Cross-encoders: the last 15 nDCG points

A bi-encoder embeds query and passage independently, so it can never model term-level interaction. A cross-encoder concatenates them into a single sequence and runs joint self-attention, emitting one relevance scalar. You cannot precompute passage embeddings, so cost is *O(candidates)* forward passes per query — which is precisely why it belongs after fusion and truncation, never before.

Realistic latency budget for the full stack (8 vCPU node, GPU reranker):

query embedding (E5-large, ONNX int8) .........  8 ms
BM25 arm (OpenSearch, top-100) ................ 22 ms   ┐ parallel
HNSW ANN (top-100, ef_search=128) ............. 30 ms   ┘
RRF + tie-break + id fetch ....................  6 ms
cross-encoder, 50 pairs, GPU int8 ............. 120 ms   (≈380 ms on CPU int8)
context assembly + tokenisation ...............  5 ms
--------------------------------------------------------
p50 ≈ 118 ms   ·   p95 ≈ 208 ms

War story: `padding='max_length'` cost 2.4x throughput

A reranker that tokenised with padding='max_length', max_length=512 paid for 50 × 512 = 25,600 tokens per query while the median (query, passage) pair was only ~210 tokens. GPU throughput scales with *padded* tokens, so the service ran at 42% of achievable QPS. Sorting candidates by true token length, padding to the bucket maximum, and micro-batching at 32 cut per-query rerank latency from 480 ms → 190 ms with identical scores.

A second, related incident: eight uvicorn workers each loaded their own ONNX session with intra_op_num_threads=2, oversubscribing an 8-vCPU box 2:1. p99 hit 3.1 s under load. The fix was architectural — run exactly one reranker process (or a Triton Inference Server with dynamic batching) and have workers call it over localhost gRPC with a bounded concurrency limit.

---

Architectural Deep Dive 3: Contextual Chunk Enrichment at Ingestion

The failure mode enrichment fixes

Consider a chunk that reads: *The maximum retry budget is 5 attempts before the circuit opens.* Which service? Which client library? Which environment? The linking noun lives three pages earlier in the parent document. Neither BM25 nor the bi-encoder can recover it, because that information is not present in the chunk at all.

The fix is to enrich at ingestion: before indexing, ask a cheap LLM to write 50–100 tokens situating the chunk inside its parent document, then prepend that context to the chunk text for both the BM25 index and the embedding model. Published evaluation results show retrieval failure rate falling 5.7% → 3.7% with contextual embeddings, → 2.9% when contextual BM25 is added, → 1.9% when a reranker is added on top. The gains are additive, not overlapping — enrichment and reranking fix different halves of the problem.

Cost model

With prompt caching, the parent document stays resident in the cache at roughly a 90% discount. Practical numbers: ≈$1.00–1.10 per million document tokens for a small Haiku-class model. A 2.4M-chunk corpus averaging 350 tokens per chunk is ~840M document tokens ≈ $850 one-time, plus the re-embedding pass. Compare that against the cost of one week of confidently wrong answers.

Operational requirements that people miss

  • Idempotency. Store context_version = sha256(prompt_template + model_id) and embed_model on every chunk row. When either changes, you know exactly which rows need regeneration instead of re-running the entire corpus.
  • Determinism. temperature = 0, a frozen prompt template, and a hard max_tokens cap. Otherwise the same chunk receives different context on replay and your index churns on every backfill.
  • Bounded concurrency, document-at-a-time ordering. 8–12 in-flight requests per worker, processing one document's chunks consecutively so the cached document prefix is actually reused.
  • Verbatim identifier preservation. Add an explicit instruction to repeat error codes, API names, config keys, and file paths unchanged. Small models love to "clean up" parse_int into *parsing integers*, which destroys exact-match retrieval for the queries that most needed help.

Alternatives worth evaluating first

  • Late chunking. Encode the whole document with a long-context encoder, then mean-pool token embeddings over each chunk's span. No LLM calls and no per-chunk cost, but it requires a long-context embedding model and careful span arithmetic. Comparable gains on prose, weaker on structured documents.
  • Contextual BM25 only. Roughly 20% of the benefit at 20% of the cost if your corpus is literal-heavy (API references, runbooks).
  • Metadata injection. Cheapest option — only viable when the missing discriminator is already a structured field.

---

Practical Implementation

The module below is the retrieval path as it runs in production. It has three properties that matter more than its features: every stage has an independent timeout, every failure mode degrades to a worse-but-valid answer instead of an error, and RRF tie-breaking is deterministic so identical inputs produce identical orderings.

# hybrid_rag.py
# Python 3.12 · asyncio · ONNX Runtime 1.18+ · NumPy 1.26+
from __future__ import annotations

import asyncio
import logging
import time
from dataclasses import dataclass
from typing import Awaitable, Callable, Protocol, Sequence

import numpy as np

logger = logging.getLogger('rag.retrieve')

RRF_K = 60
CANDIDATES_PER_ARM = 100
RERANK_CANDIDATES = 50
FINAL_TOP_K = 8
BM25_TIMEOUT_S = 0.20
DENSE_TIMEOUT_S = 0.25
RERANK_TIMEOUT_S = 0.30


class RetrievalUnavailable(RuntimeError):
    '''Every retrieval arm failed; the API layer maps this to HTTP 503.'''


class SearchArm(Protocol):
    async def search(self, query: str, top_k: int) -> list[str]:
        '''Return chunk ids ordered best-first. Must never raise on empty results.'''
        ...


@dataclass(slots=True, frozen=True)
class Chunk:
    chunk_id: str
    doc_id: str
    text: str              # context-enriched text actually indexed for both arms
    context: str = ''


def reciprocal_rank_fusion(
    rankings: Sequence[Sequence[str]],
    weights: Sequence[float] | None = None,
    k: int = RRF_K,
) -> list[tuple[str, float]]:
    '''RRF(d) = sum_i w_i / (k + rank_i(d)).

    Deterministic: ties break on (arm index, rank) of first appearance, so a
    given (query, corpus) pair always yields the same ordering. Never apply a
    score threshold to the output -- see the pitfalls section.
    '''
    if weights is None:
        weights = [1.0] * len(rankings)
    if len(weights) != len(rankings):
        raise ValueError('weights must align with rankings')
    if k <= 0:
        raise ValueError(f'k must be positive, got {k}')

    fused: dict[str, float] = {}
    tiebreak: dict[str, tuple[int, int]] = {}

    for arm, (ranking, weight) in enumerate(zip(rankings, weights)):
        if weight <= 0.0:
            continue
        for rank, chunk_id in enumerate(ranking, start=1):
            fused[chunk_id] = fused.get(chunk_id, 0.0) + weight / (k + rank)
            tiebreak.setdefault(chunk_id, (arm, rank))

    return sorted(fused.items(), key=lambda kv: (-kv[1], tiebreak[kv[0]]))


class CrossEncoderReranker:
    '''ONNX-exported cross-encoder (ms-marco-MiniLM-L-6-v2, bge-reranker-base).

    Load ONE instance per process, or run it as a standalone service.
    Eight uvicorn workers each loading this model will OOM the node.
    '''

    def __init__(
        self,
        model_path: str,
        tokenizer_id: str,
        *,
        max_length: int = 512,
        micro_batch: int = 32,
        threads: int = 4,
    ) -> None:
        import onnxruntime as ort
        from transformers import AutoTokenizer

        opts = ort.SessionOptions()
        opts.intra_op_num_threads = threads
        opts.inter_op_num_threads = 1
        opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL

        providers = [
            p for p in ('CUDAExecutionProvider', 'CPUExecutionProvider')
            if p in ort.get_available_providers()
        ]
        self._session = ort.InferenceSession(model_path, sess_options=opts, providers=providers)
        self._tokenizer = AutoTokenizer.from_pretrained(tokenizer_id)
        self._max_length = max_length
        self._micro_batch = micro_batch
        self._input_names = {spec.name for spec in self._session.get_inputs()}

    def _infer(self, feeds: dict[str, np.ndarray]) -> np.ndarray:
        return self._session.run(None, feeds)[0].reshape(-1).astype(np.float32)

    def _batches(self, query: str, passages: Sequence[str]):
        '''Length-bucketed batching.

        Sort pairs by true token length, then pad only to the longest pair in
        each micro-batch. On a typical RAG corpus (median pair ~210 tokens,
        ceiling 512) this is ~2.4x cheaper than padding everything to 512.
        '''
        probe = self._tokenizer(
            [query] * len(passages),
            list(passages),
            truncation=True,
            max_length=self._max_length,
        )
        lengths = np.fromiter(
            (len(ids) for ids in probe['input_ids']),
            dtype=np.int32,
            count=len(passages),
        )
        order = np.argsort(lengths, kind='stable')

        for start in range(0, order.size, self._micro_batch):
            idx = order[start:start + self._micro_batch]
            encoded = self._tokenizer.batch_encode_plus(
                [(query, passages[i]) for i in idx],
                padding='longest',
                truncation=True,
                max_length=self._max_length,
                return_tensors='np',
            )
            yield idx, {name: encoded[name] for name in self._input_names}

    async def rerank(
        self,
        query: str,
        candidates: Sequence[Chunk],
        top_k: int,
    ) -> list[tuple[Chunk, float]]:
        if not candidates:
            return []

        loop = asyncio.get_running_loop()
        logits = np.empty(len(candidates), dtype=np.float32)

        for idx, feeds in self._batches(query, [c.text for c in candidates]):
            # ONNX Runtime is blocking; keep the event loop free.
            logits[idx] = await loop.run_in_executor(None, self._infer, feeds)

        # Cross-encoder output is a raw logit. Sigmoid gives a bounded, monotone
        # score for logging and dashboards -- it is NOT a calibrated probability
        # and must not be used as a global cutoff (it is query-dependent).
        relevance = 1.0 / (1.0 + np.exp(-logits))
        ranked = sorted(
            zip(candidates, relevance.tolist()),
            key=lambda pair: -pair[1],
        )
        return ranked[:top_k]


async def _run_arm(
    arm: SearchArm,
    query: str,
    top_k: int,
    timeout_s: float,
    name: str,
) -> list[str]:
    t0 = time.perf_counter()
    ids = await asyncio.wait_for(arm.search(query, top_k), timeout=timeout_s)
    logger.debug('arm=%s n=%d ms=%.1f', name, len(ids), (time.perf_counter() - t0) * 1000)
    return ids


async def hybrid_retrieve(
    query: str,
    *,
    bm25_arm: SearchArm,
    dense_arm: SearchArm,
    fetch_chunks: Callable[[Sequence[str]], Awaitable[dict[str, Chunk]]],
    reranker: CrossEncoderReranker | None = None,
    arm_weights: tuple[float, float] = (1.0, 1.0),
    final_top_k: int = FINAL_TOP_K,
) -> list[tuple[Chunk, float]]:
    '''Two-stage retrieval: parallel arms -> RRF -> cross-encoder -> top-k.'''
    t0 = time.perf_counter()

    outcomes = await asyncio.gather(
        _run_arm(bm25_arm, query, CANDIDATES_PER_ARM, BM25_TIMEOUT_S, 'bm25'),
        _run_arm(dense_arm, query, CANDIDATES_PER_ARM, DENSE_TIMEOUT_S, 'dense'),
        return_exceptions=True,
    )

    rankings: list[list[str]] = []
    weights: list[float] = []
    for name, outcome, weight in zip(('bm25', 'dense'), outcomes, arm_weights):
        if isinstance(outcome, BaseException):
            logger.warning('arm_degraded arm=%s error=%r', name, outcome)
            continue
        rankings.append(outcome)
        weights.append(weight)

    if not rankings:
        raise RetrievalUnavailable('bm25 and dense arms both failed')

    fused = reciprocal_rank_fusion(rankings, weights=weights)
    candidate_ids = [chunk_id for chunk_id, _ in fused[:RERANK_CANDIDATES]]

    chunk_map = await fetch_chunks(candidate_ids)
    candidates = [chunk_map[cid] for cid in candidate_ids if cid in chunk_map]
    if not candidates:
        return []

    if reranker is None:
        return [(chunk, 0.0) for chunk in candidates[:final_top_k]]

    try:
        ranked = await asyncio.wait_for(
            reranker.rerank(query, candidates, final_top_k),
            timeout=RERANK_TIMEOUT_S,
        )
    except Exception as exc:
        # Timeout, CUDA OOM, model reload. Serve the fused order rather than a 5xx.
        logger.warning(
            'rerank_fallback error=%r elapsed_ms=%.1f',
            exc,
            (time.perf_counter() - t0) * 1000,
        )
        return [(chunk, 0.0) for chunk in candidates[:final_top_k]]

    logger.info(
        'retrieve_ok candidates=%d returned=%d elapsed_ms=%.1f',
        len(candidates),
        len(ranked),
        (time.perf_counter() - t0) * 1000,
    )
    return ranked

And the ingestion-side enrichment worker. Note the prompt-cache breakpoint on the parent document and the hard token cap that keeps the synthesised context shorter than the chunk it annotates:

# contextualize.py -- run once per (chunk, context_version)
import asyncio
import hashlib

CONTEXT_TEMPLATE = (
    'Here is a chunk taken from a document:\n<chunk>\n{chunk}\n</chunk>\n\n'
    'Write 2-3 sentences (max 80 tokens) situating this chunk inside the overall '
    'document so it can be retrieved without surrounding context. Repeat any exact '
    'identifiers -- error codes, API names, config keys, file paths, version '
    'strings -- verbatim. Return only the context, with no preamble.'
)

PROMPT_VERSION = hashlib.sha256(CONTEXT_TEMPLATE.encode()).hexdigest()[:12]
CONTEXT_MODEL = 'claude-3-5-haiku-latest'


async def contextualize(
    client,
    semaphore: asyncio.Semaphore,
    doc_text: str,
    chunk_text: str,
) -> str:
    async with semaphore:                       # 8-12 in flight per worker
        for attempt in range(5):
            try:
                response = await client.messages.create(
                    model=CONTEXT_MODEL,
                    max_tokens=120,             # keeps context <= the chunk itself
                    temperature=0.0,            # determinism: replay -> same index
                    messages=[{'role': 'user', 'content': [
                        {
                            'type': 'text',
                            'text': f'<document>\n{doc_text}\n</document>',
                            'cache_control': {'type': 'ephemeral'},   # ~90% cheaper
                        },
                        {
                            'type': 'text',
                            'text': CONTEXT_TEMPLATE.format(chunk=chunk_text),
                        },
                    ]}],
                )
                return response.content[0].text.strip()
            except Exception as exc:
                if attempt == 4:
                    raise
                await asyncio.sleep(0.5 * 2 ** attempt)   # 0.5s, 1s, 2s, 4s
    raise AssertionError('unreachable')


def indexed_text(context: str, chunk_text: str) -> str:
    '''The exact string indexed for BOTH BM25 and the embedding model.'''
    return f'{context}\n\n{chunk_text}' if context else chunk_text

Schema support for versioned re-embedding:

ALTER TABLE chunks
  ADD COLUMN context_text    text,
  ADD COLUMN context_version text,          -- sha256(prompt_template || model_id)
  ADD COLUMN embed_model     text,          -- e.g. 'e5-large-v2-int8'
  ADD COLUMN indexed_text    text GENERATED ALWAYS AS
      (coalesce(context_text, '') || E'\n\n' || raw_text) STORED;

-- Drives the re-embedding worker: it selects only rows whose versions changed.
CREATE INDEX chunks_reembed_idx
  ON chunks (embed_model, context_version)
  WHERE context_version IS DISTINCT FROM 'ctx-v7-e5large';

---

Performance Benchmarks & Trade-Offs

Corpus: 2.4M chunks / ~840M tokens, mixed technical prose, API reference, and runbooks. Evaluation: 480 queries with human-labelled gold passages. nDCG@10 is computed over the 8 chunks handed to the generator. Latency is measured end-to-end at the retrieval service boundary. Reranking used an ONNX int8 MiniLM-L6 cross-encoder.

| Retrieval strategy | Recall@50 | nDCG@10 | p50 | p95 | Cost / 1k queries | Index RAM | |---|---|---|---|---|---|---| | BM25 only (k1=1.0, b=0.4) | 0.61 | 0.38 | 18 ms | 45 ms | $0.000 | 6.2 GB | | Dense only (E5-large, HNSW M=32, ef_search=128) | 0.78 | 0.49 | 24 ms | 63 ms | $0.004 | 11.4 GB | | Hybrid RRF (k=60, 100 + 100 candidates) | 0.91 | 0.56 | 36 ms | 88 ms | $0.004 | 17.6 GB | | Hybrid RRF + cross-encoder top-50 (GPU int8) | 0.93 | 0.71 | 118 ms | 208 ms | $0.015 | 19.1 GB | | Hybrid RRF + cross-encoder top-50 (8 vCPU int8) | 0.93 | 0.71 | 265 ms | 470 ms | $0.008 | 19.1 GB | | + contextual enrichment at ingestion | 0.95 | 0.76 | 118 ms | 208 ms | +$1.00 / 1M doc tokens (one-time) | 23.8 GB |

Reading the table: RRF buys +7 nDCG points over the best single arm for ~25 ms of p95. The cross-encoder buys another +15 points for ~120 ms. Contextual enrichment buys +5 points on top of both but costs a one-time ingestion pass and ~24% more index RAM (longer indexed text means larger postings lists and larger float payloads). If you can only ship one change this quarter, ship the cross-encoder; if you own the ingestion pipeline, do enrichment first because it raises the ceiling for everything downstream.

Recall@50 saturation on the same corpus, which is how the candidate counts were chosen:

| Candidates per arm | RRF recall@50 | Marginal gain | |---|---|---| | 20 | 0.78 | — | | 50 | 0.87 | +0.09 | | 100 | 0.91 | +0.04 | | 200 | 0.92 | +0.01 |

Reranker latency is linear in pair count while recall is logarithmic. That asymmetry — not intuition — is what fixes the 100/50 split.

---

Pro-Tips & Common Pitfalls

Anti-patterns that have caused real incidents

  • ❌ Thresholding on RRF scores. With two arms at k=60 the maximum fused score is 0.0328. A well-meaning min_score = 0.5 guard silently returns zero rows for every query. Threshold on the *cross-encoder* score, or better, on a fixed top-k.
  • ❌ Min-max normalising BM25 and cosine scores, then summing. One outlier document compresses every other candidate into a near-zero band and the fused ranking flips query to query. If you want convex combination, tune α on ≥1,000 labelled pairs and re-tune when the query mix drifts.
  • ❌ Reranking each arm separately, then fusing. You spend 2× the cross-encoder budget and lose the whole point of fusion, which is cross-arm competition.
  • ❌ `padding='max_length'` in the reranker. You pay for 512 tokens on every pair. Length-bucket instead — same scores, ~2.4× faster.
  • ❌ One cross-encoder instance per ASGI worker. Eight workers × a 500 MB model is 4 GB of duplicated weights plus thread oversubscription. Run one shared process or a Triton server.
  • ❌ Post-filtering a selective ANN query. Filtering after graph traversal can drop recall from 0.89 to 0.22 with no error and no warning. Pre-filter or partition.

Operational habits that pay for themselves

  • ⚠️ Overlap twins. 10–15% chunk overlap creates near-duplicates that consume reranker budget. Deduplicate on (doc_id, simhash(normalised_text)) before fusion.
  • ⚠️ Treating cross-encoder logits as calibrated probabilities. They are not comparable across queries. Use them for ordering only; if you must threshold, fit the cutoff per query cohort on labelled data.
  • ✅ Cache by content hash, not query string. A cache key of sha256(normalised_query + index_version + embed_model) prevents serving stale results after a re-index. Query-only caching is how teams ship yesterday's corpus.
  • ✅ Gate deploys on a retrieval eval. 300–500 labelled queries, block the deploy if nDCG@10 drops >2% or recall@50 drops >1.5%. Retrieval regressions are invisible in generation-quality dashboards for weeks.
  • ✅ Log the candidate set, not just the final context. When a user reports a bad answer, you need the 50 RRF candidates, the reranker scores, and the arm that surfaced each id. Without that, every incident becomes a debate.
  • ✅ Bound total prompt tokens with the reranker plateau, not with `top_k`. Eight 512-token chunks is ~4,000 tokens of context and generators lose the middle. Check where reranker nDCG plateaus (often 4–5 chunks) and cut there.
  • ✅ Version everything in one place. embed_model, analyzer config, RRF_K, reranker checkpoint hash, and context_version. A retrieval stack without version provenance cannot be safely upgraded.

---

Frequently Asked Questions (FAQ)

How many candidates should I retrieve before cross-encoder reranking?

50 for the rerank cut, 100–200 per arm before fusion. On a 2.4M-chunk corpus, reranked nDCG@10 went 0.68 at top-20, 0.71 at top-50, and 0.715 at top-100 — while latency scales linearly with pair count. Top-50 captures essentially all of the quality at half the cost of top-100. Below 50 you start losing the tail hits that RRF was specifically designed to recover.

Why does Reciprocal Rank Fusion beat normalising and summing BM25 and vector scores?

Because RRF never compares raw scores. BM25 is unbounded, cosine is bounded, and both distributions shift per query, so any per-query normalisation scheme is implicitly re-weighting your arms on every request. RRF uses ordinal rank only, which makes it scale-free and immune to model swaps. A well-tuned convex combination can beat RRF by roughly one nDCG point on a stable query distribution, but it needs ≥1,000 labelled pairs, a maintained α, and re-tuning whenever query mix drifts — a maintenance liability that rarely survives a reorg.

Does contextual chunk enrichment remove the need for reranking?

No — they operate on orthogonal axes and their gains stack. Enrichment fixes the *representation* at ingestion time (each chunk now carries the context needed to be findable). Reranking fixes the *ordering* at query time (the best candidate rises to the top of a fused list). Published evaluations show residual retrieval failure falling from 5.7% to 3.7% with contextual embeddings, to 2.9% with contextual BM25 added, and to 1.9% only once reranking is applied. Sequencing guidance: if you own the ingestion pipeline and the corpus is reasonably stable, do enrichment first because it raises the ceiling for every downstream stage; if you cannot afford a full re-ingest, ship the cross-encoder first — it delivers +15 nDCG points with no index migration at all.