Production RAG: Hybrid Search, RRF, and Cross-Encoder Reranking
By: Md Tarek · 18 Oct 2026
# Production RAG Hybrid Search and Cross-Encoders: A Reference Architecture
Retrieval is the ceiling on everything a RAG system can do. If the gold passage never enters the candidate set, no amount of prompt engineering, long-context stuffing, or 70B-parameter generation recovers it. Most teams that report "the LLM hallucinates" are actually running a retrieval stack with 0.71 Recall@20 and calling it a model problem. This note documents the three-stage architecture we run across ~40M indexed chunks in production: BM25 + dense vectors fused with Reciprocal Rank Fusion (RRF), a cross-encoder reranker with a hard deadline, and offline contextual chunk enrichment.
Executive Summary & Key Takeaways
- Hybrid retrieval with RRF beats either retriever alone, and it is cheap. On a frozen 12,400-query internal eval set (support tickets, docs, and SEC filings), BM25 (k1=1.2, b=0.75) fused with 1024-d dense vectors at k=60 lifted Recall@20 from 0.712 (dense-only) and 0.640 (BM25-only) to 0.884, at a p95 retrieval latency of 62 ms across both indexes.
- Cross-encoder reranking is the highest-ROI millisecond in the pipeline. Reranking the top-50 fused candidates with a 568M-parameter cross-encoder moved nDCG@10 from 0.612 to 0.741 (+21%) and cut "no answer found" escalations 38%, for +170 ms p95 on a single L4 GPU.
- Contextual chunk enrichment cut top-20 retrieval failures by 35% (49% when combined with reranking) — a 1–2 sentence LLM-generated context prefix per chunk, written before both the embedding and the BM25 indexing step, costs ~$1.02 per million document tokens when the parent document is prompt-cached.
- RRF is scale-free, not information-free — but it does discard magnitude. A document found only by BM25 at rank 40 scores 1/100 while a dense rank-1 hit scores 1/61; add weighted RRF or a confidence gate before the cross-encoder when your query distribution is bimodal (navigational vs. exploratory).
- Budget the tail, not the mean. Model end-to-end as
p95 ≈ max(BM25, ANN) + fusion + rerank_batch + LLM_TTFT, and enforce per-stageasyncio.timeoutdeadlines that degrade to the fused list instead of returning a 500 under GPU saturation.
Architectural Foundations: Why Three Retrievers Beat One
A bi-encoder and a BM25 index fail in complementary ways. That complementarity is the entire argument.
BM25 (Okapi) is a sparse, term-weighted scorer with document-length normalization:
score(q, D) = Σ_{t∈q} IDF(t) · (f(t,D) · (k1 + 1)) / (f(t,D) + k1 · (1 − b + b · |D| / avgdl))
IDF(t) = ln(1 + (N − n(t) + 0.5) / (n(t) + 0.5))It is unimpeachable on rare, high-IDF tokens — SKUs, error codes, statute numbers, proper nouns — and it costs 6–9 ms p50 in OpenSearch at 10M chunks. It is useless on paraphrase: "cancellation" does not match "terminate my subscription" unless your stemmer happens to save you.
Dense bi-encoders map query and passage into a shared space and search it with HNSW or IVF-PQ. They handle synonymy, cross-lingual matches, and abstract intent. They degrade badly on tokens that fragment under the tokenizer: ERR_SSL_VERSION_OR_CIPHER_MISMATCH becomes 8–11 subword pieces, and mean-pooling smears that signal across the vector. We measured 0.31 Recall@20 for dense-only on part-number queries versus 0.87 for BM25 on the same slice. The reverse holds for conceptual questions.
Query traffic is bimodal — navigational ("refund policy Q3 2024") versus conceptual ("why do webhooks retry") — so any single retriever is a compromise. Fuse them.
┌──────────────── query (raw) ────────────────┐
│ │
┌───────▼────────┐ ┌─────────▼────────┐
│ BM25 (lexical)│ │ Dense bi-encoder │
│ OpenSearch │ │ HNSW, 1024-d │
│ top-50 @ 9ms │ │ top-50 @ 24ms │
└───────┬────────┘ └─────────┬────────┘
└──────────────┬──────────────────────────────┘
┌───────▼────────┐
│ RRF (k=60) │ rank-space fusion + dedup by chunk_id
│ top-50 @ 1ms │
└───────┬────────┘
┌───────▼──────────────┐
│ Cross-encoder rerank │ 50 pairs, batch=32, 512 tok
│ top-8 @ 170ms p95 │
└───────┬──────────────┘
┌───────▼────────┐
│ LLM generation │ ~600 ms TTFT
└────────────────┘Fusion Mechanics: RRF, Deduplication, and Weighting
RRF (Cormack, Clarke & Büttcher, SIGIR 2009) scores a document by the sum of inverse ranks across retrievers:
RRF(d) = Σ_{r ∈ R} w_r / (k + rank_r(d))with k = 60 as the canonical default. Why ranks instead of scores? Because BM25 scores are unbounded and query-dependent (a 40-term query produces systematically larger magnitudes), while cosine similarity on L2-normalized embeddings is compressed into a narrow band near 0.85–0.95 — min-max normalizing the two and averaging produces a fusion weight that drifts with every query. RRF removes that entire class of bug. The cost is that a near-perfect lexical match and a marginal one are indistinguishable beyond their rank position.
Two production details matter more than k:
1. Deduplicate on `chunk_id`, and cap chunks per parent document. Without a max_per_doc guard, one 90-page PDF with 200 near-identical chunks will occupy 8 of your 8 context slots. Capping at 3 chunks per doc_id after fusion was worth +0.041 nDCG@10 in our eval — larger than the gain from swapping rerankers. 2. Handle missing documents explicitly. A chunk absent from the dense list simply contributes 0 from that retriever. That is correct RRF behavior, but it means short documents that BM25 drops on length normalization get systematically under-ranked. Weighted RRF (w_bm25 = 0.8, w_dense = 1.2) is the cheap knob; tune it on recall@50 *before* reranking, because a strong reranker will mask a broken fusion stage.
Cross-Encoders: The Second-Stage Precision Layer
A cross-encoder concatenates [CLS] query [SEP] passage and runs full self-attention across both, emitting a single logit. Every query token attends to every passage token, so it sees negation, numeric constraints, and entity alignment that a dot product structurally cannot. The cost model is O(n_candidates · L²) attention versus O(n · d) for the bi-encoder — which is why you rerank ~50 candidates instead of searching with it.
Three operational realities we learned the hard way:
- The 512-token truncation cliff is silent. After enabling contextual enrichment our chunks averaged 380 tokens and 34% exceeded 512. The reranker happily scored truncated tails and nDCG *regressed 0.04*. Instrument your chunk-token p50/p95 and either shrink chunks or move to a long-context reranker (bge-reranker-v2-m3 handles 8192 tokens at ~3× latency).
- Cross-encoder logits are not calibrated probabilities. Abstention thresholds ("if max score < 0.3, answer 'I don't know'") must come from Platt scaling or isotonic regression fitted on a labelled slice. Raw logits drift with model version and domain.
- GPU concurrency must be bounded. Under a traffic spike, unbounded rerank calls queued to p99 = 1.8 s. Adding
asyncio.Semaphore(2)plus a 350 ms deadline that degrades to the fused ranking brought p99 to 620 ms for a measured 0.008 nDCG@10 quality loss. That is the correct trade.
Contextual Chunk Enrichment: Making Chunks Self-Contained
Fixed-size chunking severs the referents. A 300-token slice reading "Revenue grew 3% over the prior quarter" is unretrievable by the query "Acme Q3 2024 filings revenue" because neither the company nor the quarter survives the cut. Section headers, table captions, and document titles live in the tokens you dropped.
Contextual retrieval (Anthropic, Sept 2024) fixes this at ingestion: for each chunk, prompt an LLM with the entire parent document plus the chunk, and ask for 50–100 tokens of situating context. Prepend it, then embed *and* BM25-index the enriched text. Because the document is sent once per document and served from prompt cache for every subsequent chunk, the marginal cost collapses to roughly $1.02 per million document tokens (versus ~$9–12 uncached).
raw chunk: "Revenue grew 3% over the prior quarter."
context: "From Acme Corp's Q3 2024 10-Q, segment 'Cloud Infrastructure',
covering the quarter ended 2024-09-30."
index_text: context + "\n\n" + raw chunk → embedded + BM25-indexed
display_text: raw chunk → shown to user, used for citation spansAlternatives worth benchmarking before you commit: late chunking (embed the full document with a long-context model, then mean-pool token embeddings per chunk — no LLM calls, but it cannot help BM25 and requires a long-context embedder); doc2query expansion (cheaper, but hallucination-prone); parent-document retrieval (return the parent to the LLM, index the child — strong complement, weak standalone); and metadata-header prefixing (title + section path, which captures ~60% of the gain for ~0% of the cost).
War story: our first enrichment rollout regressed latency because we passed index_text (context + chunk) to the cross-encoder, inflating reranker input by 38% for a statistically insignificant +0.012 nDCG@10. We now rerank on display_text and keep enrichment strictly at the index layer.
Practical Implementation
Drop-in module covering all three stages: parallel lexical/dense retrieval with per-source deadlines, RRF with document diversity caps, a bounded cross-encoder reranker with graceful degradation, and prompt-cached contextual enrichment. Python 3.11+ (requires asyncio.timeout).
# prod_rag/retrieval.py
# pip install sentence-transformers tenacity numpy anthropic opensearch-py asyncpg
from __future__ import annotations
import asyncio
import hashlib
import logging
import time
from dataclasses import dataclass, replace
from typing import Protocol, Sequence
import numpy as np
from tenacity import (
retry,
retry_if_exception_type,
stop_after_attempt,
wait_exponential_jitter,
)
logger = logging.getLogger('rag.retrieval')
@dataclass(slots=True, frozen=True)
class Hit:
chunk_id: str
doc_id: str
text: str
score: float
source: str
metadata: dict[str, str]
class LexicalIndex(Protocol):
async def search(self, query: str, k: int, filters: dict[str, str]) -> list[Hit]: ...
class DenseIndex(Protocol):
async def search(self, vector: np.ndarray, k: int, filters: dict[str, str]) -> list[Hit]: ...
class Embedder(Protocol):
async def embed_query(self, text: str) -> np.ndarray: ...
# --------------------------------------------------------------------------- #
# Stage 2a: Reciprocal Rank Fusion (Cormack et al., 2009)
# --------------------------------------------------------------------------- #
def reciprocal_rank_fusion(
rankings: Sequence[Sequence[Hit]],
*,
k: int = 60,
weights: Sequence[float] | None = None,
max_per_doc: int = 3,
) -> list[Hit]:
"""score(d) = sum_r w_r / (k + rank_r(d)).
Rank-space fusion is used deliberately: BM25 scores are unbounded and
query-dependent, cosine similarities are bounded and compressed near 1.0.
Min-max normalising them against each other makes fusion weights drift.
"""
weights = list(weights) if weights else [1.0] * len(rankings)
if len(weights) != len(rankings):
raise ValueError('weights must align with rankings')
fused: dict[str, float] = {}
canonical: dict[str, Hit] = {}
for ranking, weight in zip(rankings, weights):
for rank, hit in enumerate(ranking, start=1):
fused[hit.chunk_id] = fused.get(hit.chunk_id, 0.0) + weight / (k + rank)
best = canonical.get(hit.chunk_id)
if best is None or hit.score > best.score:
canonical[hit.chunk_id] = hit
ordered = sorted(fused.items(), key=lambda kv: kv[1], reverse=True)
# Diversity guard: one long PDF must not monopolise the context window.
per_doc: dict[str, int] = {}
out: list[Hit] = []
for chunk_id, score in ordered:
hit = canonical[chunk_id]
if per_doc.get(hit.doc_id, 0) >= max_per_doc:
continue
per_doc[hit.doc_id] = per_doc.get(hit.doc_id, 0) + 1
out.append(replace(hit, score=score, source='rrf'))
return out
# --------------------------------------------------------------------------- #
# Stage 3: cross-encoder reranking with hard deadline + graceful degradation
# --------------------------------------------------------------------------- #
class CrossEncoderReranker:
def __init__(
self,
model_name: str = 'BAAI/bge-reranker-v2-m3',
*,
device: str = 'cuda',
batch_size: int = 32,
max_length: int = 512,
max_concurrency: int = 2,
) -> None:
from sentence_transformers import CrossEncoder # heavy import, keep local
self._model = CrossEncoder(model_name, max_length=max_length, device=device)
self._batch_size = batch_size
self._slots = asyncio.Semaphore(max_concurrency) # bounds GPU queue depth
def _predict(self, query: str, passages: list[str]) -> np.ndarray:
pairs = [(query, p) for p in passages]
return np.asarray(
self._model.predict(
pairs,
batch_size=self._batch_size,
convert_to_numpy=True,
show_progress_bar=False,
),
dtype=np.float32,
)
async def rerank(
self,
query: str,
hits: list[Hit],
*,
top_k: int = 8,
deadline_s: float = 0.35,
) -> list[Hit]:
if not hits:
return []
started = time.perf_counter()
try:
async with asyncio.timeout(deadline_s): # Python 3.11+
async with self._slots:
scores = await asyncio.to_thread(
self._predict, query, [h.text for h in hits]
)
except Exception as exc: # TimeoutError, CUDA OOM, tokenizer fault, ...
logger.warning(
'rerank_degraded', extra={'err': repr(exc), 'n_candidates': len(hits)}
)
return hits[:top_k] # fall back to fused ranking, never fail the request
ranked = sorted(
(replace(h, score=float(s), source='cross_encoder') for h, s in zip(hits, scores)),
key=lambda h: h.score,
reverse=True,
)
logger.info(
'rerank_ok',
extra={'ms': round((time.perf_counter() - started) * 1000, 1), 'n': len(hits)},
)
return ranked[:top_k]
# --------------------------------------------------------------------------- #
# Orchestration: parallel retrieval -> RRF -> rerank
# --------------------------------------------------------------------------- #
class HybridRetriever:
def __init__(
self,
lexical: LexicalIndex,
dense: DenseIndex,
embedder: Embedder,
reranker: CrossEncoderReranker,
*,
rrf_k: int = 60,
candidates: int = 50,
weights: tuple[float, float] = (1.0, 1.0),
source_deadline_s: float = 0.12,
) -> None:
self._lexical = lexical
self._dense = dense
self._embedder = embedder
self._reranker = reranker
self._rrf_k = rrf_k
self._candidates = candidates
self._weights = weights
self._deadline = source_deadline_s
async def _safe(self, coro, name: str) -> list[Hit]:
try:
async with asyncio.timeout(self._deadline):
return await coro
except Exception as exc:
logger.warning('retriever_failed', extra={'stage': name, 'err': repr(exc)})
return [] # degrade to single-retriever recall rather than 5xx
async def retrieve(
self,
query: str,
*,
filters: dict[str, str] | None = None,
top_k: int = 8,
) -> list[Hit]:
filters = filters or {}
vector = await self._embedder.embed_query(query)
lexical_hits, dense_hits = await asyncio.gather(
self._safe(self._lexical.search(query, self._candidates, filters), 'bm25'),
self._safe(self._dense.search(vector, self._candidates, filters), 'ann'),
)
if not lexical_hits and not dense_hits:
return [] # caller renders an explicit 'no sources' state
fused = reciprocal_rank_fusion(
[lexical_hits, dense_hits], k=self._rrf_k, weights=self._weights
)
return await self._reranker.rerank(query, fused[: self._candidates], top_k=top_k)
# --------------------------------------------------------------------------- #
# Stage 1 (offline): contextual chunk enrichment, prompt-cached per document
# --------------------------------------------------------------------------- #
CONTEXT_PROMPT = (
'Here is a chunk from the document above:\n<chunk>\n{chunk}\n</chunk>\n\n'
'Write 1-2 sentences that situate this chunk inside the document: name the '
'entities, the section it belongs to, and any time period it refers to. '
'Answer with the context only, no preamble.'
)
class AnthropicContextualizer:
"""The full document is sent once per document and served from prompt cache
for every chunk, which is what makes this ~$1 per million document tokens."""
def __init__(self, model: str = 'claude-3-5-haiku-latest') -> None:
from anthropic import AsyncAnthropic
self._client = AsyncAnthropic()
self._model = model
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential_jitter(initial=0.5, max=8.0),
retry=retry_if_exception_type(Exception),
reraise=True,
)
async def contextualize(self, document: str, chunk: str) -> str:
msg = await self._client.messages.create(
model=self._model,
max_tokens=128,
temperature=0.0,
messages=[
{
'role': 'user',
'content': [
{
'type': 'text',
'text': f'<document>\n{document}\n</document>',
'cache_control': {'type': 'ephemeral'},
},
{'type': 'text', 'text': CONTEXT_PROMPT.format(chunk=chunk)},
],
}
],
)
block = msg.content[0]
return getattr(block, 'text', '').strip()
@dataclass(slots=True)
class EnrichedChunk:
chunk_id: str
doc_id: str
index_text: str # context + chunk -> embedded AND BM25-indexed
display_text: str # raw chunk -> shown to user, used for citation spans
async def enrich_chunks(
doc_id: str,
document: str,
chunks: list[str],
contextualizer: AnthropicContextualizer,
*,
concurrency: int = 16,
max_doc_chars: int = 24_000,
) -> list[EnrichedChunk]:
semaphore = asyncio.Semaphore(concurrency)
body = document[:max_doc_chars] # keep the cached prefix inside the context window
async def one(chunk: str) -> EnrichedChunk:
digest = hashlib.sha256(f'{doc_id}:{chunk}'.encode()).hexdigest()[:16]
async with semaphore:
try:
context = await contextualizer.contextualize(body, chunk)
except Exception as exc: # never block ingestion on enrichment
logger.warning(
'contextualize_failed', extra={'doc_id': doc_id, 'err': repr(exc)}
)
context = ''
index_text = f'{context}\n\n{chunk}' if context else chunk
return EnrichedChunk(
chunk_id=digest, doc_id=doc_id, index_text=index_text, display_text=chunk
)
return list(await asyncio.gather(*(one(c) for c in chunks)))Pair this with HNSW parameters tuned for your recall target, and pin ef_search at query time — the default is often too low for filtered ANN:
CREATE INDEX chunk_embedding_hnsw ON chunks USING hnsw (embedding vector_cosine_ops) WITH (m = 32, ef_construction = 256); SET hnsw.ef_search = 96; -- 0.97 recall@50 on our corpus; 64 dropped to 0.91
Performance Benchmarks & Trade-Offs
Eval set: 12,400 labelled queries / 3.9M chunks. Hardware: 1× NVIDIA L4 (24 GB) for the cross-encoder, 3-node OpenSearch 2.15 (c6g.xlarge), pgvector 0.7 on r6g.2xlarge. Latency is measured server-side, retrieval-only; $ includes embedding + rerank compute amortized at 500 QPS.
| Pipeline | Recall@20 | nDCG@10 | MRR@10 | p50 | p95 | Index (GB) | $/1k queries | |---|---|---|---|---|---|---|---| | BM25 only (OpenSearch) | 0.640 | 0.481 | 0.442 | 9 ms | 21 ms | 4.1 | $0.02 | | Dense only (HNSW, 1024-d) | 0.712 | 0.548 | 0.501 | 24 ms | 58 ms | 11.8 | $0.06 | | Hybrid + RRF (k=60) | 0.884 | 0.612 | 0.567 | 27 ms | 62 ms | 15.9 | $0.08 | | Hybrid + RRF + cross-encoder (top-50) | 0.884 | 0.741 | 0.703 | 132 ms | 241 ms | 15.9 | $0.19 | | Contextual + Hybrid + cross-encoder | 0.917 | 0.796 | 0.761 | 137 ms | 252 ms | 18.4 | $0.21 | | Hybrid + LLM reranker (GPT-4o-mini) | 0.884 | 0.752 | 0.714 | 2.1 s | 4.8 s | 15.9 | $3.40 |
Reranker selection, same 50-candidate fused input (L4, batch 32, 512-token truncation):
| Reranker | Params | Pairs/s | Δ nDCG@10 vs. RRF | p95 for 50 pairs | |---|---|---|---|---| | ms-marco-MiniLM-L-6-v2 | 22M | ~1,450 | +0.081 | 36 ms | | bge-reranker-base | 278M | ~520 | +0.104 | 96 ms | | bge-reranker-v2-m3 | 568M | ~310 | +0.129 | 162 ms | | Cohere Rerank 3.5 (API) | n/a | n/a | +0.122 | 210 ms (incl. 40 ms RTT) |
Read the first table carefully: hybrid retrieval buys recall, the cross-encoder buys ranking. RRF moved Recall@20 from 0.712 to 0.884 but nDCG@10 only from 0.548 to 0.612 — you now have the right documents in the candidate set, ranked badly. The reranker is what converts that recall into precision at the top. Skipping it wastes the recall you just paid for; skipping hybrid caps the reranker's ceiling.
Pro-Tips & Common Pitfalls
- Rerank after deduplication, never before. Reranking 50 candidates that resolve to 12 unique documents burns 4× the GPU for identical output.
- Never assume cross-encoder logits are probabilities. Fit Platt scaling on a labelled slice before wiring an abstention threshold; otherwise "I don't know" fires on domain shift, not on genuine absence.
- Instrument the 512-token truncation cliff. Log chunk-token p50/p95 at ingest. Contextual enrichment adds 50–100 tokens per chunk and silently pushes you over the limit.
- Do not min-max normalize BM25 and cosine then average. The normalization is query-relative and unstable. Use RRF, or a learned convex combination trained on labelled pairs.
- Cap chunks per parent document (`max_per_doc ≤ 3`). Worth +0.041 nDCG@10 on its own in our eval.
- Store `index_text` and `display_text` separately. Citations must quote the raw chunk; embedding the context prefix is fine, showing it to the user is not.
- Unbounded GPU concurrency is an outage waiting for traffic. Semaphore + deadline + degrade to fused ranking. A 0.008 nDCG loss beats a p99 of 1.8 s.
- Pin and version your BM25 analyzer. Stemmer, stopword list, lowercase filter, ICU folding for CJK — a silent analyzer change destroys recall with zero error signal.
- Evaluate retrieval separately from generation. Track Recall@k, nDCG@10, and MRR@10 on a frozen query set. End-to-end answer quality is too noisy to attribute.
- Shadow-index on embedding-model upgrades. Dual-write to
chunks_v1andchunks_v2, A/B the retrieval metrics, then cut over. Re-embedding 40M chunks is a week-long, expensive, irreversible operation if you skip the shadow. - Always ship a no-answer path. Empty fused list, all candidates below the calibrated threshold, or reranker timeout — all three should render "no sources found" with a fallback, not a hallucinated answer.
Frequently Asked Questions (FAQ)
What is the best k for Reciprocal Rank Fusion in production RAG?
k = 60 from Cormack et al. (2009) is the correct default and is what most teams should ship. The parameter controls how sharply top ranks dominate: lower k (20–40) concentrates weight on rank-1 results and is better when your retrievers are individually high-precision; higher k (80–100) flattens the curve and helps when a document appears at moderate ranks across both retrievers. Tune it on Recall@50 of the fused list before reranking, because a strong cross-encoder will mask fusion defects entirely. Critically, RRF is rank-based, so k is independent of corpus size — a value tuned on 100k chunks transfers to 40M.
Do I still need BM25 if I use a large, modern embedding model?
Yes, for three reasons. First, tokenizer fragmentation: identifiers like SKU-4471-B, ERR_SSL_VERSION, and statute numbers split into subword pieces whose mean-pooled vector loses exact-match signal — our dense-only Recall@20 on part-number queries was 0.31 versus 0.87 for BM25. Second, out-of-domain robustness: lexical models have no training distribution to fall out of, which is why BM25 remains competitive with dense retrievers on BEIR-style zero-shot benchmarks while costing a fraction of the latency. Third, cost and latency: BM25 runs at 9 ms p50 and effectively free, making it the cheapest insurance policy in the stack.
How many candidates should I rerank in production?
The practical range is 25–100, with 50 as the default. Reranker recall@k plateaus around k=50 on typical corpora — going to 200 costs 4× the GPU time for roughly +0.005 nDCG@10. The binding constraint is the latency budget: on a single L4, bge-reranker-v2-m3 processes ~310 pairs/s at 512 tokens, so 50 pairs is ~162 ms p95. If your p95 budget is 150 ms, drop to 25 candidates or step down to bge-reranker-base (~520 pairs/s). Always set a hard deadline (300–400 ms) that degrades to the fused ranking rather than queuing — tail latency under saturation is what users actually notice.
