Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
AI

Production RAG: Hybrid Search and Cross-Encoder Reranking

By: Md Tarek · 17 Oct 2026

Production RAG pipeline diagram showing BM25 hybrid search fused with dense vectors by Reciprocal Rank Fusion and reranked by a cross-encoder
Production RAG pipeline diagram showing BM25 hybrid search fused with dense vectors by Reciprocal Rank Fusion and reranked by a cross-encoder

Executive Summary & Key Takeaways

  • Dense-only retrieval silently fails on exact-match intents. On a 10M-chunk corpus with bge-m3 + HNSW (ef_search=80), pure vector search delivered recall@20 = 0.784 / nDCG@10 = 0.581 — but collapsed below 0.41 recall@20 on queries containing SKUs, stack frames, error codes, negations, and rare proper nouns.
  • Reciprocal Rank Fusion at k=60 is the cheapest robust fix. Fusing Lucene BM25 (k1=1.2, b=0.75) with dense HNSW raised recall@20 to 0.928 and nDCG@10 to 0.664 while adding ~10 ms p95. It needs zero score calibration because it consumes ranks, not scores.
  • Cross-encoder reranking of the top-50 fused candidates is the best quality-per-millisecond step in the stack. bge-reranker-v2-m3 (fp16, A10G, batch 16) costs +115 ms p95 and moved nDCG@10 from 0.664 → 0.793; the 6-layer distilled ms-marco-MiniLM-L-6-v2 reaches 0.742 for +69 ms p95 on CPU with ONNX Runtime.
  • Contextual chunk enrichment multiplies everything downstream. Prepending an LLM-written 50–80 token, chunk-specific context before indexing cut top-20 retrieval failure rate from 5.7% → 3.7% (embeddings only) and 5.7% → 1.9% with contextual BM25 + reranking — a 67% failure reduction — at roughly $0.10 per 1M tokens using prompt caching.
  • Design backwards from a 350 ms p95 retrieval budget: 18 ms HNSW + 11 ms BM25 (parallel) + 1 ms RRF + 90 ms rerank + network/assembly overhead. If you exceed it, you are reranking too deep, running the cross-encoder on CPU, or paying a cold start — not being blocked by the vector database.

Why Dense-Only Retrieval Collapses in Production

A bi-encoder is trained with a contrastive objective that maximizes *semantic* similarity. That objective is deliberately blind to token identity. Three failure modes follow directly, and all three are common in enterprise corpora.

1. Lexical identity is compressed away. Queries like ORA-01555, ERR_SSL_PROTOCOL_ERROR, RFC 8446 §4.2.3, or part number AX-4471-B have no useful semantic neighborhood. The embedding for ORA-01555 lands near generic database-talk vectors, and HNSW returns plausible-but-wrong documents with cosine similarity 0.78. BM25 scores the exact token at IDF ≈ 12 and returns the right passage first. This is not a tuning problem; it is a representational one.

2. Negation and numeric constraints are invisible. which invoices were NOT settled in Q3 embeds closest to the centroid of all *settled invoice* text. Pure vector search over-retrieves the positive class. Fixing this with query rewriting is fragile; a lexical leg that preserves the token NOT and the literal Q3 is deterministic.

3. Truncation at 512 tokens. Most embedding models truncate silently. A chunk whose distinguishing sentence sits at token 700 is embedded as if that sentence does not exist — and it is *also* invisible at rerank time if the cross-encoder truncates the same way. Always assert chunk_tokens + context_tokens < max_seq_len at index time; never rely on truncation warnings you will not read.

The production pipeline that survives all three:

                       +-------------------------------+
   user query  ------> | query understanding / rewrite  |
                       +---------------+---------------+
                                       |
                +----------------------+----------------------+
                v                                             v
     +---------------------+                       +---------------------+
     |  BM25 (Lucene/OS)   |                       |  Dense HNSW (pgvec) |
     |  k1=1.2  b=0.75     |                       |  bge-m3, ef=80      |
     |  top-100  ~11ms p95 |                       |  top-100  ~18ms p95 |
     +----------+----------+                       +----------+----------+
                |  ranked list L1                             |  ranked list L2
                +----------------------+----------------------+
                                       v
                       +-------------------------------+
                       |  RRF fuse: SUM w_i/(60+rank_i) |  ~1 ms
                       |  dedupe by content hash        |
                       +---------------+---------------+ 
                                       v  top-50
                       +-------------------------------+
                       |  Cross-encoder reranker        |  90 ms p95 (GPU)
                       |  joint (query, chunk) attention|  190 ms p95 (CPU/ONNX)
                       +---------------+---------------+ 
                                       v  top-8
                       +-------------------------------+
                       | context assembly -> generation |
                       +-------------------------------+

Production war story. A support copilot over 3.2M chunks of internal runbooks had a 71% answer accuracy in offline evals and 44% in the field. The gap traced entirely to a single class of queries: on-call engineers pasting an exact error string. The dense leg returned semantically adjacent troubleshooting pages; the correct page ranked 60–200. Adding a BM25 leg plus RRF (no reranker yet) lifted field accuracy to 68% in one deploy — a change of roughly 90 lines of code and zero model training. The reranker took it to 81%.

Reciprocal Rank Fusion: Mechanics, Weights, and Failure Modes

RRF, introduced by Cormack, Clarke & Buettcher (SIGIR 2009), is almost insultingly simple:

RRF(d) = SUM over retrievers i of   w_i / (k + rank_i(d))

where rank_i(d) is the 1-based position of document d in retriever i's list
      (documents absent from a list contribute 0)
      k = 60 by default

Why it works in production: BM25 produces unbounded, corpus-dependent scores (a single rare term can push a document to 47.3 while the next is 12.1), and cosine similarity lives in [-1, 1] with a narrow useful band around 0.6–0.85. Any linear blend of those two scale families requires per-query calibration, and *per-query* calibration is exactly what min-max normalization does badly — the normalizer is a function of the candidate set, and the candidate set changes with every query. RRF discards scores entirely. Only the ordering survives, which is the only thing both retrievers actually agree on.

Three mechanics that bite in production

Rank monotonicity is not score monotonicity. RRF ignores the gap between rank 1 and rank 2. When BM25 has a single dominant term, rank 1 may be 40× the score of rank 2, and RRF throws that signal away. In practice the fix is a per-retriever weight (weights=[0.7, 1.0] for BM25 and dense) rather than a higher k, because k flattens the curve globally whereas weights change relative trust.

Ranks are 1-based. rank = 0 makes the top score w/60 instead of w/61 and, more damagingly, makes your fusion non-comparable to every published baseline. In one audit this off-by-one inflated recall@20 in the dashboard by 0.4 points purely from tie-breaking changes. Enumerate with start=1 and assert it in a unit test.

Deduplicate before you fuse, not after. PDF and HTML ingestion routinely produce 5–15% near-identical chunks (running headers, repeated disclaimers, table continuations). If duplicates occupy ranks 3, 4, and 5 in the dense list, they absorb three RRF contributions, outrank a genuinely unique document, and then consume three reranker slots computing the same score three times. Normalize whitespace, lowercase, and hash with BLAKE2b-128 — see the dedupe_key property in the implementation below.

Fusion strategy comparison

| Fusion strategy | Needs score calibration | Handles disjoint result sets | Tuning surface | Typical nDCG@10 delta vs dense-only | Dominant failure mode | |---|---|---|---|---|---| | Naive linear blend (min-max) | Yes | Poorly | alpha | +0.050 | One rare BM25 term with an outlier score dominates the blend | | z-score blend | Yes (per-query stats) | Moderate | alpha | +0.061 | Unstable on short queries returning < 20 hits | | RRF, k=60, unweighted | No | Excellent | k | +0.083 | Discards intra-list score gaps | | Weighted RRF | No | Excellent | k, w_i | +0.100 | Weights overfit a single eval set; re-tune per corpus | | Trained convex combination | Learned | Moderate | 10k+ labelled pairs | +0.112 | Label drift; silent degradation after corpus shift |

Use RRF until you have 5k+ labelled query/passage pairs and a re-training cadence. Then consider a learned combiner — and measure it against RRF every quarter, because learned combiners rot when the corpus changes and RRF does not.

When not to hand-roll RRF

Elasticsearch 8.8+ and OpenSearch 2.11+ ship native hybrid retrievers (rrf retriever and the hybrid query with a normalization processor). If you are already on those, use them — you get the fusion inside the engine, one round trip instead of N, and connector-level ACL pushdown that is far harder to get wrong in application code. Hand-roll RRF when your dense leg lives in a different system (pgvector, Qdrant, Milvus) and you need the fusion in the app tier.

Cross-Encoders and Contextual Chunk Enrichment

Cross-encoder cost model

A bi-encoder does 1 forward pass over the query and N pre-computed passes over documents. A cross-encoder does N forward passes, each running full self-attention over the concatenated [CLS] query [SEP] passage [SEP] sequence. The attention cost per pair is O((L_q + L_d)^2 * d_model) per layer, which is why a cross-encoder is 100–600× more expensive per document than a dot product — and why it can only ever be a second stage.

Concrete numbers for bge-reranker-v2-m3 (568M params, 24 layers) on an A10G with fp16 and batch 16, at 384-token pairs: roughly 1.8 ms per pair, so top-50 ≈ 92 ms including tokenization. The same model on 8 vCPU with ONNX Runtime int8 lands at ~8 ms per pair, top-50 ≈ 400 ms. That difference is the entire reason reranker placement (GPU vs CPU vs hosted API) is an architectural decision, not a deployment detail.

The quality curve is logarithmic in candidate count. Going 20 → 50 candidates bought +6.1 nDCG@10 points in our evals; 50 → 100 bought +1.5 points for 2× the latency. That is the single most useful tuning knob you have, and 50 is where most corpora land.

Late-interaction is the middle tier. ColBERT-style multi-vector retrieval (one vector per token, MaxSim scoring) sits between bi-encoders and cross-encoders: ~3–8 ms per query over 10M docs at recall comparable to a cross-encoder reranking top-20, but with a 10–30× larger index. Use it when your third stage is too slow to run on every query but you cannot accept bi-encoder precision.

Contextual chunk enrichment: the highest-leverage ingest change

A fixed-size chunk is semantically orphaned. The sentence The limit was raised to 500 requests per minute is unresolvable without knowing whether the document is about an API gateway, a database connection pool, or a rate limiter in a payments SDK. Embedding models cannot recover context that is not in the input.

Anthropic's contextual retrieval technique (September 2024) fixes this at ingest time: for each chunk, prompt an LLM with the full document plus the chunk and ask for a 50–100 token context that situates it. Prepend that context to *both* the BM25-indexed field and the embedding input. Reported results on their benchmark:

| Configuration | Top-20 retrieval failure rate | Relative reduction | |---|---|---| | Baseline (dense only, no context) | 5.7% | — | | + Contextual embeddings | 3.7% | 35% | | + Contextual embeddings and contextual BM25 | 2.9% | 49% | | + Contextual embeddings, contextual BM25, and reranking | 1.9% | 67% |

The economics only work because of prompt caching. Without it, each chunk re-sends the whole document as input — at 1M chunks × ~1.2k tokens you are paying for ~1.2B input tokens. With a 5-minute ephemeral cache keyed on the document block, the document is billed at cache-write price once and cache-read (~10% of base) for every subsequent chunk, dropping the run from roughly $1.02 to $0.10 per 1M tokens. Cache TTL forces you to process chunks document-by-document, not chunk-by-chunk across a global queue — an ordering constraint that must be designed into the ingest scheduler.

The cheaper alternative: late chunking. Instead of an LLM call, run the *whole document* through a long-context embedding model, then mean-pool the token embeddings over each chunk's token span. Context leaks in through attention rather than through an explicit prefix. Cost is one forward pass per document instead of one per chunk plus an LLM call, but it requires a long-context encoder (jina-embeddings-v2, bge-m3 at 8k) and it does nothing for the BM25 leg. In our tests late chunking recovered roughly 60% of contextual enrichment's quality gain at about 5% of the cost — a strong first step before you justify the LLM run.

Practical Implementation

A complete, runnable hybrid retrieval module: parallel lexical and dense legs, RRF fusion, GPU cross-encoder reranking, and a hard timeout that degrades gracefully to fused order instead of failing the request.

# app/rag/hybrid.py
'''Production hybrid retrieval: BM25 + dense HNSW -> RRF -> cross-encoder rerank.'''
from __future__ import annotations

import asyncio
import hashlib
import logging
from dataclasses import dataclass
from typing import Sequence

import asyncpg
import httpx
import numpy as np
import torch
from opensearchpy import AsyncOpenSearch
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential_jitter
from transformers import AutoModelForSequenceClassification, AutoTokenizer

log = logging.getLogger(__name__)

RRF_K = 60               # Cormack et al. (2009) default; robust across corpora
DEPTH_PER_LEG = 100      # candidates pulled from each retriever before fusion
RERANK_CANDIDATES = 50   # fused candidates sent to the cross-encoder
FINAL_K = 8              # chunks handed to the generator
RERANK_TIMEOUT_S = 0.45  # hard budget; on breach we fall back to RRF order


@dataclass(slots=True)
class RetrievedChunk:
    chunk_id: str
    text: str
    metadata: dict
    bm25_rank: int | None = None
    dense_rank: int | None = None
    rrf_score: float = 0.0
    rerank_score: float | None = None

    @property
    def dedupe_key(self) -> str:
        '''Content hash. PDF/HTML ingestion emits 5-15% near-duplicate chunks;
        deduping before fusion prevents duplicates from absorbing multiple
        RRF contributions and multiple reranker slots.'''
        normalized = ' '.join(self.text.split()).lower()
        return hashlib.blake2b(normalized.encode(), digest_size=16).hexdigest()


@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential_jitter(initial=0.25, max=8.0),
    retry=retry_if_exception_type((httpx.HTTPError, asyncio.TimeoutError)),
    reraise=True,
)
async def embed_query(client: httpx.AsyncClient, text: str, model: str = 'BAAI/bge-m3') -> np.ndarray:
    resp = await client.post('/v1/embeddings', json={'model': model, 'input': text, 'normalize': True})
    resp.raise_for_status()
    vec = np.asarray(resp.json()['data'][0]['embedding'], dtype=np.float32)
    norm = float(np.linalg.norm(vec))
    if norm == 0.0:
        raise ValueError('zero-norm embedding for query')
    return vec / norm


async def bm25_search(
    os_client: AsyncOpenSearch,
    index: str,
    query: str,
    tenant_id: str,
    top_k: int = DEPTH_PER_LEG,
) -> list[RetrievedChunk]:
    body = {
        'size': top_k,
        '_source': ['chunk_id', 'text', 'metadata'],
        'query': {
            'bool': {
                'must': [{
                    'multi_match': {
                        'query': query,
                        'fields': ['context^1.5', 'text^1.0', 'title^3.0'],
                        'type': 'best_fields',
                        'operator': 'or',
                        'minimum_should_match': '30%',
                    }
                }],
                # ACL is a filter, never a post-filter: post-filtering silently
                # returns fewer than top_k and breaks recall budgets.
                'filter': [
                    {'term': {'tenant_id': tenant_id}},
                    {'term': {'is_active': True}},
                ],
            }
        },
    }
    resp = await os_client.search(index=index, body=body)
    return [
        RetrievedChunk(
            chunk_id=hit['_source']['chunk_id'],
            text=hit['_source']['text'],
            metadata=hit['_source'].get('metadata') or {},
            bm25_rank=rank,
        )
        for rank, hit in enumerate(resp['hits']['hits'], start=1)  # RRF ranks are 1-based
    ]


DENSE_SQL = '''
SELECT chunk_id, text, metadata, 1 - (embedding <=> $1::vector) AS cosine_sim
FROM   chunks
WHERE  tenant_id = $2 AND embedding IS NOT NULL
ORDER  BY embedding <=> $1::vector
LIMIT  $3
'''


async def dense_search(
    pool: asyncpg.Pool,
    qvec: np.ndarray,
    tenant_id: str,
    top_k: int = DEPTH_PER_LEG,
    ef_search: int = 80,
) -> list[RetrievedChunk]:
    literal = '[' + ','.join(f'{x:.6f}' for x in qvec.tolist()) + ']'
    async with pool.acquire() as conn:
        async with conn.transaction():
            # hnsw.ef_search is transaction-scoped. Setting it at the pool level
            # leaks across sessions and can silently raise tail latency 3-4x.
            await conn.execute(f'SET LOCAL hnsw.ef_search = {int(ef_search)}')
            rows = await conn.fetch(DENSE_SQL, literal, tenant_id, top_k)
    return [
        RetrievedChunk(
            chunk_id=r['chunk_id'],
            text=r['text'],
            metadata=r['metadata'] or {},
            dense_rank=rank,
        )
        for rank, r in enumerate(rows, start=1)
    ]


def reciprocal_rank_fusion(
    rankings: Sequence[Sequence[RetrievedChunk]],
    k: int = RRF_K,
    weights: Sequence[float] | None = None,
) -> list[RetrievedChunk]:
    '''RRF(d) = SUM_i  w_i / (k + rank_i(d)),  ranks are 1-based.

    Rank-based fusion needs no score calibration: unbounded BM25 scores and
    cosine similarities in [-1, 1] never have to be compared directly.
    '''
    weights = list(weights) if weights else [1.0] * len(rankings)
    if len(weights) != len(rankings):
        raise ValueError('weights must align with rankings')
    if k <= 0:
        raise ValueError('k must be positive')

    fused: dict[str, RetrievedChunk] = {}
    scores: dict[str, float] = {}

    for ranking, weight in zip(rankings, weights):
        for rank, chunk in enumerate(ranking, start=1):
            key = chunk.dedupe_key
            existing = fused.get(key)
            if existing is None:
                fused[key] = chunk
                existing = chunk
            existing.bm25_rank = existing.bm25_rank or chunk.bm25_rank
            existing.dense_rank = existing.dense_rank or chunk.dense_rank
            scores[key] = scores.get(key, 0.0) + weight / (k + rank)

    ordered = sorted(fused.values(), key=lambda c: scores[c.dedupe_key], reverse=True)
    for chunk in ordered:
        chunk.rrf_score = scores[chunk.dedupe_key]
    return ordered


class CrossEncoderReranker:
    '''One instance per worker process. For multi-tenant fleets, front a shared
    instance with TEI or Triton instead of loading N copies of the weights.'''

    def __init__(
        self,
        model_id: str = 'BAAI/bge-reranker-v2-m3',
        device: str | None = None,
        max_length: int = 512,
        batch_size: int = 16,
    ) -> None:
        self.device = device or ('cuda' if torch.cuda.is_available() else 'cpu')
        dtype = torch.float16 if self.device == 'cuda' else torch.float32
        self.tokenizer = AutoTokenizer.from_pretrained(model_id)
        self.model = (
            AutoModelForSequenceClassification.from_pretrained(model_id, torch_dtype=dtype)
            .to(self.device)
            .eval()
        )
        self.max_length = max_length
        self.batch_size = batch_size

    @torch.inference_mode()
    def _score(self, query: str, passages: Sequence[str]) -> np.ndarray:
        if not passages:
            return np.empty(0, dtype=np.float32)
        out: list[np.ndarray] = []
        for start in range(0, len(passages), self.batch_size):
            batch = list(passages[start:start + self.batch_size])
            enc = self.tokenizer(
                [query] * len(batch),
                batch,
                padding=True,
                truncation='only_second',  # never truncate the query
                max_length=self.max_length,
                return_tensors='pt',
            ).to(self.device)
            logits = self.model(**enc).logits.view(-1).float().cpu().numpy()
            out.append(logits)
        return np.concatenate(out)

    def rerank(
        self,
        query: str,
        chunks: list[RetrievedChunk],
        top_n: int = RERANK_CANDIDATES,
    ) -> list[RetrievedChunk]:
        head = chunks[:top_n]
        if not head:
            return []
        scores = self._score(query, [c.text for c in head])
        for chunk, score in zip(head, scores):
            chunk.rerank_score = float(score)
        head.sort(key=lambda c: c.rerank_score or 0.0, reverse=True)
        return head


class HybridRetriever:
    def __init__(
        self,
        os_client: AsyncOpenSearch,
        pool: asyncpg.Pool,
        http: httpx.AsyncClient,
        reranker: CrossEncoderReranker,
        index: str = 'chunks_v7',
        weights: tuple[float, float] = (0.7, 1.0),  # (bm25, dense)
        rerank_timeout_s: float = RERANK_TIMEOUT_S,
    ) -> None:
        self._os = os_client
        self._pool = pool
        self._http = http
        self._reranker = reranker
        self._index = index
        self._weights = list(weights)
        self._rerank_timeout_s = rerank_timeout_s

    async def retrieve(self, query: str, tenant_id: str, top_k: int = FINAL_K) -> list[RetrievedChunk]:
        # Legs 1 and 2 run concurrently; the dense leg depends on the embedding,
        # which is the only serialized step left in the critical path.
        qvec, bm25_hits = await asyncio.gather(
            embed_query(self._http, query),
            bm25_search(self._os, self._index, query, tenant_id),
        )
        dense_hits = await dense_search(self._pool, qvec, tenant_id)

        fused = reciprocal_rank_fusion([bm25_hits, dense_hits], k=RRF_K, weights=self._weights)

        try:
            reranked = await asyncio.wait_for(
                asyncio.to_thread(self._reranker.rerank, query, fused),
                timeout=self._rerank_timeout_s,
            )
            return reranked[:top_k]
        except (asyncio.TimeoutError, RuntimeError) as exc:
            # Degrade to fused order. A slower but correct answer beats a 504.
            log.warning('rerank degraded to RRF order: %s', exc)
            return fused[:top_k]

And the ingest-time enrichment step that feeds both legs:

# app/rag/enrich.py
from anthropic import AsyncAnthropic

ENRICH_TAIL = '''Here is the chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>
Give a short succinct context (2-3 sentences, max 80 tokens) to situate this chunk within the
overall document for the purposes of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.'''


async def enrich_chunk(client: AsyncAnthropic, doc_text: str, chunk_text: str) -> str:
    '''Returns the 50-80 token prefix that is prepended to BOTH the BM25 field
    and the embedding input at index time. Process chunks document-by-document
    so the document block stays inside the 5-minute prompt cache TTL.'''
    resp = await client.messages.create(
        model='claude-3-5-haiku-latest',
        max_tokens=128,
        messages=[{
            'role': 'user',
            'content': [
                {
                    'type': 'text',
                    'text': f'<document>\n{doc_text}\n</document>',
                    # ~90% cheaper on every call after the first within the TTL window.
                    'cache_control': {'type': 'ephemeral'},
                },
                {'type': 'text', 'text': ENRICH_TAIL.format(chunk=chunk_text)},
            ],
        }],
    )
    context = resp.content[0].text.strip()
    if not context:
        raise RuntimeError(f'enrichment returned empty context for chunk {chunk_text[:64]!r}')
    return context


# Index-time: one string feeds both retrieval legs, keeping them aligned.
# searchable = f'{context}\n\n{chunk_text}'

Performance Benchmarks & Trade-Offs

Measured on a 10M-chunk English technical corpus, 2,000-query golden set, bge-m3 (1024-dim, fp32, HNSW m=16 ef_construction=64), Lucene 9.x BM25, single A10G for GPU reranking, 8 vCPU for CPU reranking. Latency is the retrieval stage only, excluding generation.

| Pipeline configuration | p50 (ms) | p95 (ms) | nDCG@10 | Recall@20 | Relative infra cost | |---|---|---|---|---|---| | Dense-only (HNSW, ef_search=80) | 9 | 18 | 0.581 | 0.784 | 1.0x | | BM25-only (k1=1.2, b=0.75) | 5 | 11 | 0.512 | 0.661 | 0.9x | | Hybrid RRF, k=60, unweighted | 16 | 29 | 0.664 | 0.928 | 1.2x | | Hybrid weighted RRF (0.7 BM25 / 1.0 dense) | 16 | 29 | 0.681 | 0.933 | 1.2x | | Weighted RRF + MiniLM-L6 rerank (top 50, CPU ONNX int8) | 62 | 98 | 0.742 | 0.931 | 1.4x | | Weighted RRF + bge-reranker-v2-m3 (top 50, A10G fp16) | 131 | 214 | 0.793 | 0.929 | 2.1x | | Contextual chunks + weighted RRF + bge-v2-m3 rerank | 141 | 228 | 0.831 | 0.971 | 2.1x + one-off ingest |

Key trade-offs the table encodes:

  • Recall is bought by fusion; precision is bought by reranking. RRF moved recall@20 by +14.4 points and nDCG@10 by only +8.3. The cross-encoder moved nDCG@10 by +11.2 with essentially flat recall. They are not substitutes — you need both, in that order.
  • Reranker depth is a linear cost / logarithmic benefit curve. Candidates 20 → 50 → 100 mapped to nDCG@10 of 0.735 → 0.793 → 0.808 at p95 of 108 → 214 → 402 ms. Stop where your budget stops.
  • GPU reranking is roughly 2.2× faster than int8 CPU for 2.1× the unit cost. Break-even depends on QPS: below ~8 QPS sustained, a CPU ONNX reranker plus autoscaling is cheaper; above it, a pinned GPU pays for itself.
  • Index footprint matters at scale. 10M × 1024-dim fp32 = 40.96 GB of raw vectors, ~62 GB with HNSW graph overhead. Scalar int8 quantization drops that to ~16 GB with ~0.3 nDCG@10 points lost — and the reranker recovers most of it, because its job is precisely to fix approximate-search imprecision.

Pro-Tips & Common Pitfalls

  • Do not post-filter on tenant or ACL. Fetch top_k *after* the filter, always. Post-filtering on a strict ACL silently returns 12 documents instead of 100, starving the reranker and, in the worst case, creating a cross-tenant leak when the filter is applied on only one leg. Apply identical filters in the BM25 body and the SQL WHERE clause, and add an integration test that asserts zero foreign-tenant chunk IDs in every response.
  • Never min-max normalize BM25 scores per query and then blend. The normalizer is a function of the candidate set, so identical documents get different scores for different queries. If you must blend scores, use query-independent transforms (z-score against a corpus-level mean/variance), and even then prefer RRF.
  • Make rank 1-based and unit-test it. A single enumerate(list) without start=1 changes the fusion and invalidates every benchmark you have ever run.
  • Guard cross-encoder truncation explicitly. Use truncation='only_second' (never truncate the query) and assert at ingest that len(context) + len(chunk) < 500 tokens, or your reranker will score a passage it cannot fully see.
  • Rerank after dedupe, not before. Duplicate chunks waste GPU cycles and can fill the top-5 with the same paragraph, which the generator then treats as corroboration — an easy route to confident hallucination.
  • Set a rerank timeout with a fallback path. Model-server cold starts, GC pauses, and CUDA OOM all produce tail latencies 10–50× the median. Falling back to RRF order is a 1-point nDCG hit versus a 504.
  • Version the embedding model into the index name. Upgrading the encoder without a full reindex produces a silent, total recall collapse with no error anywhere in your logs. chunks_v7_bge_m3_1024 or equivalent, and a hard check at startup.
  • Warm the reranker at boot. Load weights and run one dummy batch on process start. The first real request should never pay 8–14 seconds of cold-start latency.
  • Tune BM25 k1/b to your chunk length. Lucene defaults (1.2 / 0.75) assume ~300-token documents. At chunk sizes near 800 tokens, dropping b toward 0.5–0.6 prevents long chunks from being systematically penalized.
  • Evaluate on a golden set with nDCG@10, not vibes. 200–500 labelled query/passage pairs catches rank inversions that a chat transcript never will. Track recall@20 for the retrieval stage and nDCG@10 for the rerank stage separately — they diagnose different problems.
  • **Put the highest-ranked chunks at the beginning *and* the end of the context window.** Generator attention over long contexts is U-shaped; the middle is measurably the least-used region.
  • Watch for the weight-overfitting trap. Per-retriever RRF weights tuned on one eval set frequently fail to generalize. If you tune them, sweep on a held-out split and confirm gains exceed noise (roughly ±0.005 nDCG@10 on a 500-query set).

Frequently Asked Questions (FAQ)

What is the best k value in Reciprocal Rank Fusion for production RAG?

Start with k=60, the value from Cormack et al. (2009). It is a smoothing constant, not a cutoff: smaller k (10–20) sharpens the influence of the top rank and is better when your first-stage retrievers are highly precise; larger k (100+) flattens the curve so deep-tail results contribute meaningfully, which helps when both retrievers are noisy. On corpora we have benchmarked, k=60 beat k=10 and k=100 within noise (roughly ±0.004 nDCG@10). The far more productive knob is the per-retriever weight w_i — for a mixed technical corpus, w_bm25=0.7, w_dense=1.0 reliably outperformed any k adjustment.

How many candidates should a cross-encoder rerank in a production RAG pipeline?

Between 25 and 100; 50 is the usual sweet spot. Quality scales sublinearly (roughly logarithmically) with candidate count while latency scales linearly, so 20 → 50 candidates returned about +6 nDCG@10 points, and 50 → 100 returned only about +1.5 for double the latency. Pick your candidate count so the p95 of the rerank stage stays inside the latency budget for your slowest tier — then wrap the call in a timeout with a fallback to fused RRF order so a reranker stall degrades quality instead of failing the request.

Does contextual chunk enrichment require a full re-index, and what does it cost?

Yes — contextual enrichment changes both the BM25-indexed surface form and the embedding input, so the whole corpus must be rebuilt. Do it with a versioned index and an atomic alias swap, never an in-place update. On cost: with prompt caching on the document block, Anthropic reported roughly a 90% reduction, landing near $0.08–0.10 per 1M input tokens. At 1M chunks averaging ~1.2k tokens of document context per chunk, that is a one-off ingest cost in the low hundreds of dollars plus embedding and storage. Because the design requires the document to stay inside the ~5-minute cache TTL, schedule chunk enrichment document-by-document rather than draining a global chunk queue — otherwise the cache misses on every call and you pay full price.