Production RAG Hybrid Search and Cross-Encoders in Practice
By: Md Tarek · 19 Oct 2026
Executive Summary & Key Takeaways
- Hybrid retrieval is not optional at scale. On a 1.24M-chunk reference corpus (38k docs, 312 graded queries), BM25-only hit recall@50 = 0.71 and dense-only 0.78; weighted Reciprocal Rank Fusion (k=60) reached 0.89 for +11 ms p50 over dense-only.
- RRF is rank-based, so it is immune to score drift between a BM25 engine (unbounded log-odds) and cosine similarity (bounded, model-specific). Tune
kand per-retriever weights — min-max normalising raw scores across engines is the #1 hybrid-search bug found in production audits. - Cross-encoder reranking is the highest-ROI second stage. Reranking the fused top-100 → top-8 with
ms-marco-MiniLM-L-6-v2(ONNX INT8, batch 32) moved nDCG@10 from 0.57 → 0.71 and LLM-judge faithfulness from 82% → 89%, for ~180 ms of p95 CPU time. - Contextual chunk enrichment cut retrieval failures 49%, and 67% combined with reranking in Anthropic's published Contextual Retrieval evaluation — at roughly $1.02 per 1M document tokens once prompt caching is enabled.
- Design to a latency budget, not a vibe: embed 25 ms + hybrid retrieve 45 ms + fuse <1 ms + rerank 190 ms + assemble 15 ms ≈ 275 ms p50. Enforce a hard rerank timeout (2 s) that degrades gracefully to fused order rather than returning a 504.
Architecture: Why Single-Stage Dense Retrieval Breaks in Production
Dense bi-encoders compress a chunk into a single 1024-dim vector. That is a lossy summarisation, and it fails predictably on four query classes we see constantly in enterprise corpora:
1. Identifiers and codes — ORA-01555, AX-4419-B, CVE-2024-3094. Cosine similarity smears digits; BM25 nails exact tokens. 2. Negation and constraints — *"which plans are NOT HIPAA-eligible"*. Embeddings are notoriously poor at scope of negation. 3. Rare domain jargon — the token appeared 4 times in 38k documents, so the embedding model never learned a meaningful neighbourhood for it. 4. Freshness skew — a new document's vector exists but its lexical signals (product name, error string) are what users search for during incidents.
Conversely BM25 fails on vocabulary mismatch (*"cancel my subscription"* vs. *"terminate plan"*) and on long conversational queries where the salient phrase is diluted by 40 tokens of preamble. The two retrievers have anti-correlated failure modes, which is precisely why fusion works.
┌──────────────── user query ─────────────────┐
│ │
┌─────▼──────┐ ┌───────▼─────────┐
│ BM25 index │ │ embedder (TEI) │
│ (ParadeDB) │ │ bge-m3 · 1024d │
└─────┬──────┘ └───────┬─────────┘
│ top-100 │ top-100
│ │
┌─────▼─────────────────────────────────────────────▼────────┐
│ Reciprocal Rank Fusion: score(d) = Σ w_r / (k + rank_r) │
│ dedupe by content hash · keep top-50 · deterministic ties │
└─────────────────────────┬──────────────────────────────────┘
│
┌─────────────▼─────────────┐
│ Cross-encoder reranker │ 100 pairs → sigmoid
│ (q,d) joint cross-attn │ timeout 2s → fallback
└─────────────┬─────────────┘
│ top-8, score ≥ 0.02
┌─────────────▼─────────────┐
│ Assembly: token budget, │
│ parent-doc expansion │
└───────────────────────────┘Operational war story. A support-RAG deployment ran dense-only for six months at a comfortable 94 ms p95. During an incident, engineers searched the runbook corpus for ERRSRV-5031 and got five semantically-similar-but-useless chunks; the single chunk containing the literal error code ranked #117 and never made the top-8. Adding BM25 + RRF with zero re-embedding moved that chunk to #3. The lesson is not "dense is bad" — it is that exact-match recall is a hard requirement that ANN cannot guarantee, and fusion is a 11 ms insurance premium.
The latency budget you actually have
| Stage | p50 | p95 | Notes | |---|---|---|---| | Query embed (TEI, bge-m3) | 25 ms | 60 ms | Batch-of-1 penalty; co-locate with app or accept 20 ms RTT | | BM25 top-100 (ParadeDB) | 12 ms | 30 ms | pdb.score + GIN/BM25 index; sub-5 ms under 200k chunks | | Dense top-100 (pgvector HNSW, ef_search=160) | 33 ms | 90 ms | ef_search 160 ≈ 0.96 recall@100 on our data | | RRF fusion (100+100 candidates) | 0.4 ms | 0.9 ms | Pure Python, ~5 µs/chunk | | Cross-encoder top-100 (INT8, 8 vCPU) | 185 ms | 420 ms | Dominates; scales linearly in candidate depth | | Assembly + dedupe | 15 ms | 40 ms | Token budget trimming |
If your SLO is 500 ms p95, the reranker is the only stage that can blow it. Everything else must be async and concurrent.
Reciprocal Rank Fusion: Math, Weights, and Failure Modes
Cormack et al. (2009) introduced RRF as a parameter-light way to combine ranked lists from systems whose scores are not comparable:
score(d) = Σ_{r ∈ R} w_r / (k + rank_r(d))with rank_r(d) being the 1-based position of document *d* in retriever *r*'s list, and k a damping constant (canonical value 60).
Why `k` matters. With k=60, a document at rank 1 scores 1/61 ≈ 0.01639 and at rank 2 scores 1/62 ≈ 0.01613 — a 1.6% relative difference. That compression is the feature: it makes fusion stable when a retriever's ordering is noisy near the top. Lowering k to 10 amplifies top-rank dominance (1/11 = 0.0909 vs 1/12 = 0.0833, an 8.3% gap), which helps when one retriever is decisively more trustworthy on specific query classes. We grid-searched k ∈ {10, 20, 40, 60, 100} and found 60 best for general traffic, 20 better for identifier-heavy traffic (where BM25 rank-1 should win outright).
Weighting. Production deployments almost always need asymmetry. Our tuned defaults are w_sparse = 1.0, w_dense = 1.3 for conversational product queries, and 1.6 / 1.0 for log-and-error-code corpora. Route weights by query classifier (regex for code-like tokens, \b[A-Z]{2,}-\d style patterns) rather than globally.
Failure modes to design against
- Correlated retrievers double-count. If you fuse BM25 with SPLADE and both derive from the same lexical signal, a mediocre lexical match at rank 1 in both lists beats a genuinely relevant semantic match at rank 5. Either decorrelate the retrievers or down-weight one.
- Fusing too shallow. Fusing only top-10 per retriever destroys the recall benefit; the union of two top-10 lists frequently misses the document sitting at 20 in both. Use depth 100, fuse, then truncate to 50 before reranking.
- Overlapping chunks. With 512-token windows and 25% stride, the same sentence appears 2–3 times and can occupy 6 of 8 final slots. Dedupe by normalised text hash and by parent
doc_idbefore scoring. - Non-deterministic ties. RRF produces exact ties constantly (any doc appearing at the same rank in both lists). Sort by
(-rrf_score, dense_rank, chunk_id)so your offline eval is reproducible; otherwise a 3% nDCG swing between runs is pure noise. - Score-fusion temptation. Convex combination (
α·dense + (1-α)·bm25) requires calibrated scores. Cosine ∈ [-1,1] and BM25 is unbounded, so you end up z-scoring per query — which is unstable when the candidate set is small (a single outlier flattens the distribution). RRF avoids this entirely.
Cross-Encoder Reranking Within a Latency Budget
A bi-encoder computes f(q) and g(d) independently — O(1) document cost at index time, dot product at query time. A cross-encoder computes h(q, d) jointly with full cross-attention over the concatenated pair, so every candidate requires a fresh transformer forward pass at query time. That is the entire cost/relevance trade-off, in one sentence.
Empirically, cross-encoders recover most of the relevance gap between retrieval and human judgment: our nDCG@10 went 0.57 → 0.71 on the reranked top-10. But cost scales linearly with candidate depth, so depth is the only knob that matters.
| Reranker | Params | Throughput, 8 vCPU INT8 | p95 @ 100 candidates | Throughput, 1×A10G | |---|---|---|---|---| | cross-encoder/ms-marco-MiniLM-L-6-v2 | 22.7M | ~700 pairs/s | ~150 ms | ~4,800 pairs/s | | BAAI/bge-reranker-base | 278M | ~150 pairs/s | ~700 ms | ~1,600 pairs/s | | BAAI/bge-reranker-v2-m3 | 568M | ~45 pairs/s | ~2.3 s | ~1,100 pairs/s | | Cohere Rerank 3.5 (managed API) | n/a | n/a | ~250 ms (network) | n/a |
Rules of thumb we apply:
1. Never rerank more than 150 candidates unless you have GPU headroom; the nDCG gain from 100→200 is <0.01 on every corpus we have measured, but latency doubles. 2. Truncate pair length at 512 tokens (max_length=512). Long chunks cost quadratically-ish in attention and dilute the relevance signal. 3. Use sigmoid on raw logits — cross-encoder output is not a probability. Threshold at ~0.02 for MiniLM, ~0.15 for bge-reranker-base, calibrated on your own dev set; the threshold also guards against confidently reranking garbage when retrieval returns nothing relevant. 4. Degrade, don't fail. Wrap the rerank in asyncio.wait_for(..., timeout=2.0) and return fused order on timeout. A 2% relevance hit beats a 3% error-rate blip. 5. One process, one model. Semaphore(2) around the reranker; loading a transformer per request is a self-inflicted outage.
An important nuance: reranking does not fix recall. If the relevant chunk is absent from the fused top-100, no reranker can recover it. Optimise recall at the retrieval/fusion layer (depth, chunking, enrichment) and precision at the rerank layer. Measure them separately — that's why the benchmark table below reports recall@50 *and* nDCG@10.
Contextual Chunk Enrichment
Chunking destroys context. A 400-token chunk reading *"The limit was raised to 40 in Q3 after the incident review"* is semantically meaningless in isolation — *which* limit, *which* system, in *which* document? Embedding it produces a vector that retrieves poorly for any query that doesn't also lack the subject.
Contextual Retrieval (Anthropic, Sept 2024) solves this by having an LLM prepend a 50–100 token situating preamble to each chunk *before* both embedding and BM25 indexing:
Original chunk: "The limit was raised to 40 in Q3 after the incident review." Contextualised chunk (indexed text): "This chunk is from Acme's 2024 Rate-Limiting Runbook, section 'Per-tenant quotas', describing the Q3 increase of the per-tenant request ceiling from 20 rps to 40 rps following INC-2291. The limit was raised to 40 in Q3 after the incident review."
Anthropic's published results: 35% reduction in retrieval failure with contextual embeddings, 49% with contextual embeddings + contextual BM25, and 67% when combined with reranking. Those numbers are workload-specific but the direction is reproducible.
The cost problem and its fix. Naïvely, you send the full document once per chunk — a 20k-token doc with 50 chunks means 1M input tokens. Prompt caching makes this tractable: cache the document block, pay ~$1.02 per 1M document tokens (Anthropic's own figure for a cheap model + caching), and process chunks of the same document sequentially so the cache actually hits.
CONTEXT_PROMPT = '''<document>
{doc}
</document>
Here is the chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>
Give a short, succinct context (50-100 tokens) that situates this chunk within
in the overall document to improve search retrieval of the chunk. Answer only
with the succinct context and nothing else.'''
def build_request(doc: str, chunk: str) -> dict:
return {
'model': 'claude-haiku',
'max_tokens': 160,
'system': [
{'type': 'text',
'text': f'''<document>{doc}</document>''',
'cache_control': {'type': 'ephemeral'}}, # cache the doc block
],
'messages': [
{'role': 'user', 'content': CONTEXT_PROMPT.split('<document>')[1].replace('{doc}', '').strip()
.replace('{chunk}', chunk)},
],
}Cheaper alternatives, ranked by ROI per dollar:
| Approach | Failure-rate reduction | Ingest cost | Notes | |---|---|---|---| | Per-chunk LLM context (cached) | ~49% | $1.02/M doc tokens | Best quality; needs cheap model + caching | | Document-summary prepend (1 per doc) | ~30–35% | $0.05/M doc tokens | 80% of the benefit at ~5% of cost; great default | | Structural header prepend (title > h2 > h3) | ~12–18% | $0 | Free; do this always | | Late chunking (embed full doc, pool per span) | ~20% | GPU time only | No LLM calls; requires a long-context embedder |
In practice we ship structural header + document summary on day one (free), then A/B per-chunk contextual enrichment on the corpora where retrieval failure rate is still >20%.
One critical detail: index the contextualised text, but return the original chunk (plus the context) to the generator. Indexed preambles cost you 1.3× index size and pollute the answer context with boilerplate.
Practical Implementation
A complete, runnable hybrid retrieval module: async pgvector + ParadeDB BM25, deterministic weighted RRF, batched cross-encoder reranking with timeout fallback, and type-safe error handling.
# rag/retrieval.py
from __future__ import annotations
import asyncio
import hashlib
import logging
import time
from dataclasses import dataclass, field
from typing import Sequence
import asyncpg
import httpx
import numpy as np
from pydantic_settings import BaseSettings
from sentence_transformers import CrossEncoder
log = logging.getLogger('rag.retrieval')
class Settings(BaseSettings):
pg_dsn: str = 'postgresql://rag:rag@postgres:5432/rag'
embed_url: str = 'http://tei:8080/embed'
embed_dim: int = 1024
sparse_depth: int = 100
dense_depth: int = 100
fused_depth: int = 50
final_k: int = 8
rrf_k: int = 60
w_sparse: float = 1.0
w_dense: float = 1.3
ef_search: int = 160
rerank_model: str = 'cross-encoder/ms-marco-MiniLM-L-6-v2'
rerank_batch: int = 32
rerank_timeout_s: float = 2.0
min_rerank_score: float = 0.02
@dataclass(slots=True)
class Chunk:
chunk_id: str
doc_id: str
text: str
metadata: dict = field(default_factory=dict)
@property
def dedupe_key(self) -> str:
normalised = ' '.join(self.text.lower().split())
return hashlib.blake2b(normalised.encode(), digest_size=8).hexdigest()
@dataclass(slots=True)
class ScoredChunk:
chunk: Chunk
rrf_score: float = 0.0
rerank_score: float = 0.0
sources: tuple[str, ...] = ()
def reciprocal_rank_fusion(
rankings: dict[str, Sequence[Chunk]],
weights: dict[str, float],
k: int = 60,
) -> list[ScoredChunk]:
'''Weighted RRF: score(d) = sum_r w_r / (k + rank_r(d)).
Rank-based fusion is scale-free: a BM25 score of 41.2 and a cosine
similarity of 0.83 combine without per-query score calibration.
'''
if k <= 0:
raise ValueError('rrf k must be positive')
scores: dict[str, float] = {}
hits: dict[str, dict] = {}
for retriever, chunks in rankings.items():
weight = weights.get(retriever, 1.0)
seen: set[str] = set()
for rank, chunk in enumerate(chunks, start=1):
key = chunk.dedupe_key
if key in seen: # overlapping windows must not double-count
continue
seen.add(key)
scores[key] = scores.get(key, 0.0) + weight / (k + rank)
entry = hits.setdefault(key, {'chunk': chunk, 'sources': set()})
entry['sources'].add(retriever)
fused: list[ScoredChunk] = []
for key, score in scores.items():
entry = hits[key]
fused.append(
ScoredChunk(
chunk=entry['chunk'],
rrf_score=score,
sources=tuple(sorted(entry['sources'])),
)
)
# Deterministic ordering: exact RRF ties are common and break eval reproducibility.
fused.sort(key=lambda c: (-c.rrf_score, c.chunk.chunk_id))
return fused
class HybridRetriever:
SPARSE_SQL = '''
SELECT id::text AS id, doc_id::text AS doc_id, text, metadata
FROM chunks
WHERE text @@@ $1
ORDER BY pdb.score(id) DESC
LIMIT $2
'''
DENSE_SQL = '''
SELECT id::text AS id, doc_id::text AS doc_id, text, metadata
FROM chunks
ORDER BY embedding <=> $1::vector
LIMIT $2
'''
def __init__(self, settings: Settings) -> None:
self.s = settings
self._pool: asyncpg.Pool | None = None
self._http = httpx.AsyncClient(
timeout=httpx.Timeout(1.5, connect=0.5),
limits=httpx.Limits(max_connections=128, max_keepalive_connections=32),
)
self._reranker = CrossEncoder(
settings.rerank_model, max_length=512, device='cpu'
)
self._rerank_guard = asyncio.Semaphore(2) # CPU-bound: cap concurrency
async def start(self) -> None:
self._pool = await asyncpg.create_pool(
self.s.pg_dsn, min_size=4, max_size=32, command_timeout=5.0
)
log.info('retriever ready model=%s dim=%d', self.s.rerank_model, self.s.embed_dim)
async def close(self) -> None:
await self._http.aclose()
if self._pool is not None:
await self._pool.close()
# ------------------------------------------------------------------ embed
async def embed(self, texts: Sequence[str]) -> np.ndarray:
if not texts:
return np.empty((0, self.s.embed_dim), dtype=np.float32)
resp = await self._http.post(
self.s.embed_url,
json={'inputs': list(texts), 'normalize': True, 'truncate': True},
)
resp.raise_for_status()
vectors = np.asarray(resp.json(), dtype=np.float32)
expected = (len(texts), self.s.embed_dim)
if vectors.shape != expected:
raise ValueError(f'embedding shape {vectors.shape} != {expected}; '
'embedding model and index are out of sync')
return vectors
# ------------------------------------------------------------- retrievers
async def _sparse(self, conn: asyncpg.Connection, query: str, limit: int) -> list[Chunk]:
rows = await conn.fetch(self.SPARSE_SQL, query, limit)
return [Chunk(r['id'], r['doc_id'], r['text'], dict(r['metadata'] or {})) for r in rows]
async def _dense(self, conn: asyncpg.Connection, vector: np.ndarray, limit: int) -> list[Chunk]:
literal = '[' + ','.join(f'{x:.7f}' for x in vector) + ']'
async with conn.transaction():
# SET LOCAL is transaction-scoped; per-query recall tuning, no session leak.
await conn.execute(f'SET LOCAL hnsw.ef_search = {int(self.s.ef_search)}')
rows = await conn.fetch(self.DENSE_SQL, literal, limit)
return [Chunk(r['id'], r['doc_id'], r['text'], dict(r['metadata'] or {})) for r in rows]
# ---------------------------------------------------------------- reranker
def _rerank_sync(self, query: str, candidates: Sequence[ScoredChunk]) -> list[float]:
pairs = [(query, c.chunk.text) for c in candidates]
logits = self._reranker.predict(
pairs,
batch_size=self.s.rerank_batch,
show_progress_bar=False,
convert_to_numpy=True,
activation_fn=None,
)
# Cross-encoder logits are unnormalised; sigmoid gives a usable threshold.
return (1.0 / (1.0 + np.exp(-np.asarray(logits, dtype=np.float32)))).tolist()
async def _rerank(self, query: str, candidates: list[ScoredChunk]) -> list[ScoredChunk]:
if not candidates:
return []
try:
async with self._rerank_guard:
scores = await asyncio.wait_for(
asyncio.to_thread(self._rerank_sync, query, candidates),
timeout=self.s.rerank_timeout_s,
)
except (asyncio.TimeoutError, RuntimeError) as exc:
log.warning('rerank degraded to RRF order: %r', exc)
return candidates
reranked = [
ScoredChunk(
chunk=c.chunk,
rrf_score=c.rrf_score,
rerank_score=float(score),
sources=c.sources,
)
for c, score in zip(candidates, scores, strict=True)
]
reranked.sort(key=lambda c: -c.rerank_score)
return reranked
# ------------------------------------------------------------------ public
async def search(self, query: str, *, final_k: int | None = None) -> list[ScoredChunk]:
if self._pool is None:
raise RuntimeError('call start() before search()')
if not query or not query.strip():
return []
started = time.perf_counter()
vector = (await self.embed([query]))[0]
# asyncpg connections are not multiplexed: one connection per concurrent query.
async def run_sparse() -> list[Chunk]:
async with self._pool.acquire() as conn: # type: ignore[union-attr]
return await self._sparse(conn, query, self.s.sparse_depth)
async def run_dense() -> list[Chunk]:
async with self._pool.acquire() as conn: # type: ignore[union-attr]
return await self._dense(conn, vector, self.s.dense_depth)
sparse, dense = await asyncio.gather(run_sparse(), run_dense())
fused = reciprocal_rank_fusion(
{'sparse': sparse, 'dense': dense},
{'sparse': self.s.w_sparse, 'dense': self.s.w_dense},
k=self.s.rrf_k,
)[: self.s.fused_depth]
reranked = await self._rerank(query, fused)
k = final_k or self.s.final_k
final = [c for c in reranked if c.rerank_score >= self.s.min_rerank_score][:k]
log.debug(
'search sparse=%d dense=%d fused=%d final=%d latency_ms=%.1f',
len(sparse), len(dense), len(fused), len(final),
(time.perf_counter() - started) * 1000,
)
return final or reranked[:k]
async def _smoke() -> None:
retriever = HybridRetriever(Settings())
await retriever.start()
try:
hits = await retriever.search('ORA-01555 undo tablespace retention tuning')
for hit in hits:
print(f'ce={hit.rerank_score:.3f} rrf={hit.rrf_score:.5f} '
f'doc={hit.chunk.doc_id} via={hit.sources}')
finally:
await retriever.close()
if __name__ == '__main__':
logging.basicConfig(level=logging.DEBUG)
asyncio.run(_smoke())Index-side companion (run once, then on ingest):
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_search; -- ParadeDB: real BM25, not ts_rank
CREATE TABLE chunks (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
doc_id uuid NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
chunk_idx int NOT NULL,
text text NOT NULL, -- contextualised text lives here
raw_text text NOT NULL, -- original text returned to the LLM
metadata jsonb NOT NULL DEFAULT '{}',
embedding vector(1024) NOT NULL,
UNIQUE (doc_id, chunk_idx)
);
CREATE INDEX chunks_bm25 ON chunks USING bm25 (id, text)
WITH (key_field = 'id', text_fields = '{"text": {"tokenizer": {"type": "default"}}}');
CREATE INDEX chunks_hnsw ON chunks USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 200);Performance Benchmarks & Trade-Offs
Reference corpus: 38k documents → 1.24M chunks (512 tokens, 25% stride), 312 graded queries (0–3 relevance), single node 8 vCPU / 32 GB, TEI on 1×A10G, ParadeDB + pgvector co-located. Costs are fully loaded (compute + LLM) per 1k queries at steady state.
| Configuration | Recall@50 | nDCG@10 | Faithfulness (LLM-judge) | p50 | p95 | Cost / 1k queries | |---|---|---|---|---|---|---| | BM25 only (k1=1.2, b=0.75) | 0.71 | 0.42 | 71% | 22 ms | 48 ms | $0.04 | | Dense only (bge-m3, HNSW m=16, ef=160) | 0.78 | 0.49 | 76% | 34 ms | 90 ms | $0.09 | | Hybrid RRF (k=60, w=1.0/1.3) | 0.89 | 0.57 | 82% | 45 ms | 120 ms | $0.12 | | + Cross-encoder (MiniLM-L6 INT8, 100→8) | 0.91 | 0.71 | 89% | 185 ms | 420 ms | $0.21 | | + Contextual enrichment (per-chunk, cached) | 0.93 | 0.78 | 93% | 205 ms | 445 ms | $0.34 |
Reading the table:
- The big recall jump is retrieval-stage (0.78 → 0.89); the big precision jump is the rerank stage (nDCG 0.57 → 0.71). They are not substitutes.
- Recall@50 gains plateau after fusion — adding a third retriever (SPLADE) bought +0.01 recall for +8 ms. Not worth it unless your corpus is legal/patent-heavy.
- Contextual enrichment's benefit shows up mostly in downstream faithfulness (+4 pts) and in long-tail recall on documents with weak titles.
- The 0.21 → 0.34 cost delta is ingest-amortised enrichment at $1.02/M doc tokens; at 10× query volume the delta shrinks further.
Pro-Tips & Common Pitfalls
- ❌ Min-max or z-score normalising scores across BM25 and cosine. Per-query calibration is unstable with small candidate sets. Use RRF; it exists precisely to avoid this.
- ❌ Fusing top-10 lists. Use depth 100, fuse, truncate to 50. Fusing shallow lists is the most common silent recall killer.
- ❌ Reranking 500 candidates "for safety". Latency scales linearly; nDCG gains saturate around 100. Measure your own curve before assuming otherwise.
- ❌ No dedupe before fusion. Overlapping windows will fill your context with the same paragraph. Hash on whitespace-normalised text.
- ❌ Post-filtering after ANN. Filtering
WHERE tenant_id = ?after the vector scan collapses recall to near-zero for small tenants. Use pgvector partial indexes or partition by tenant. - ❌ Forgetting `hnsw.ef_search`. The default (40) gives ~0.90 recall@10 on our data; 160 gives 0.96. Set it per query with
SET LOCAL. - ❌ Indexing bare chunks without title/section headers. Free 12–18% retrieval-failure reduction you didn't take.
- ❌ Mixing embedding models in one index. Version the model in the table (
embedding_model,embedding_version) and rebuild, don't patch. Mixed vector spaces produce silently terrible rankings. - ❌ Blocking the event loop with a cross-encoder. Always
asyncio.to_thread+ a semaphore; a transformer on the loop will stall every coroutine in the process. - ❌ Dual-write drift between BM25 and vector store. If they're separate engines, write both in one outbox transaction; otherwise you rerank against documents that no longer exist.
- ✅ Log the retrieval provenance (
sources=('sparse','dense')) per hit. It makes fusion-weight tuning trivial and debuggable. - ✅ Evaluate recall and precision separately (
recall@50pre-rerank,nDCG@10post-rerank). Conflating them means you tune the wrong stage for weeks. - ✅ Build a 300-query golden set with graded relevance before touching
kor weights. Fifty queries tuned on will overfit you into a 5-point regression. - ✅ Cap the rerank with a timeout and fall back to fused order. Availability beats a marginal nDCG gain.
Frequently Asked Questions (FAQ)
Why use Reciprocal Rank Fusion instead of score normalisation for hybrid search?
Because BM25 scores and vector similarities live on incomparable, query-dependent scales. A BM25 score of 41.2 means nothing in absolute terms — it depends on document length, term IDF, and k1/b. Cosine similarity is bounded but its *distribution* shifts per query: a query with one obvious match clusters around 0.85, a fuzzy query around 0.45. Any normalisation scheme (min-max, z-score) therefore has to be recomputed per query, and with only 100 candidates a single outlier compresses everything else into a narrow band. RRF uses only ordinal ranks, so it is scale-free and needs no calibration; score(d) = Σ w_r / (k + rank_r(d)) with k=60 is stable across heterogeneous retrievers. In our benchmarks RRF matched a carefully tuned convex combination within 0.005 nDCG while requiring 15× less tuning effort.
Does cross-encoder reranking justify its latency in production RAG?
Yes, when you rerank a bounded candidate set with a small model. On our corpus, reranking the fused top-100 with ms-marco-MiniLM-L-6-v2 (INT8, batch 32, 8 vCPU) cost ~185 ms p50 / 420 ms p95 and bought nDCG@10 +0.14 and LLM-judge faithfulness +7 points — by far the best relevance-per-millisecond of any pipeline stage. The economics break when you (a) rerank more than ~150 candidates, (b) use a 500M+ parameter reranker on CPU, or (c) fail to paginate/batch. Rules: keep candidate depth ≤100, use INT8 ONNX, cap pair length at 512 tokens, run it off the event loop with a semaphore, and wrap it in a 2-second timeout that falls back to fused order. If your p95 budget is under 200 ms total, use a hosted rerank API in a sidecar region or skip reranking and invest in contextual enrichment instead.
Is contextual chunk enrichment still worth it now that long-context models exist?
Yes — because it changes *retrieval*, not generation. Long context doesn't help if the relevant chunk never enters the prompt. Prepending an LLM-generated situating preamble to each chunk (before both embedding and BM25 indexing) reduced retrieval failures 49% in Anthropic's published evaluation, and 67% combined with reranking. The economics depend entirely on prompt caching: without it, re-sending a 20k-token document for each of its 50 chunks costs ~$1.00 per 1M document tokens × 20; with the document block cached, it drops to about $1.02 per 1M document tokens total. If you can't justify that, take the free wins first — prepend title and heading structure to every chunk (~12–18% failure reduction, zero cost) — then a single document summary per doc (~30–35% at ~5% of the cost), and only then A/B full per-chunk contextual enrichment on the corpora where retrieval failure is still above 20%.
