Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
AI

Production RAG Pipeline: Hybrid Search + Cross-Encoder Reranking

By: Md Tarek · 18 Oct 2026

Production RAG pipeline architecture showing BM25 and dense vector hybrid retrieval fused with Reciprocal Rank Fusion and reranked by a cross-encoder
Production RAG pipeline architecture showing BM25 and dense vector hybrid retrieval fused with Reciprocal Rank Fusion and reranked by a cross-encoder

Production RAG Hybrid Search and Cross-Encoders are the difference between a demo that answers three questions well and a system that survives 40M chunks, five languages, and exact-match identifiers such as ERR_OSSL_PEM_BAD_BASE64_DECODE. Below is the pipeline we actually run: parallel lexical and dense retrieval, weighted Reciprocal Rank Fusion (RRF), token-budgeted cross-encoder reranking, and contextual chunk enrichment at ingest time.

Executive Summary & Key Takeaways

  • Hybrid RRF lifts Recall@20 from 0.62 (dense-only) and 0.68 (BM25-only) to 0.83-0.87 on domain corpora, because rank fusion composes two independent error profiles without needing score normalization.
  • Cross-encoder reranking over a 64-document pool adds +10 to +18 nDCG@10 for 90-120 ms p95; reranking 1,000 candidates costs 1.5-2 s and buys under +1 point. Cap the pool, not the model size.
  • The RRF constant k=60 is the safe default (Cormack et al., 2009). Values from 20 to 100 change nDCG@10 by less than 1.5%, but k=1 collapses recall whenever the two retrievers disagree on the head.
  • Contextual chunk enrichment cuts top-20 retrieval failures by 35% (embeddings), 49% (BM25), and 67% (combined + reranking) in Anthropic's published evaluation, and the cost is paid once at ingest instead of on every query.
  • Budget latency per stage: 8 ms query rewrite, 35 ms parallel retrieval, 90 ms rerank, 400 ms generation. An unbounded rerank batch is the single most common cause of p99 blowups in production RAG.

Architecture: Where Single-Vector Retrieval Breaks

Every retrieval failure in production RAG falls into one of two buckets. Hybrid search exists because those buckets are disjoint.

Failure mode 1: dense retrieval loses rare tokens

An embedding model trained on web text maps ORA-01555 into the same neighborhood as ORA-01554, ORA-04031, and generic database-exception prose. Product SKUs, error codes, function signatures, version strings, and negation (does NOT support TLS 1.3) are all low-signal to a bi-encoder. In our support corpus, keyword-only queries containing an error code had dense Recall@20 of 0.54 versus 0.91 for BM25.

Failure mode 2: BM25 loses paraphrase and intent

Lexical retrieval requires term overlap. A user asking why are my pods being killed at night will never match a chunk containing Evicted: The node was low on resource: memory. Nor will how do I rotate credentials match credential rollover procedure. Multilingual corpora make this worse: BM25 operates on analyzers, so CJK text without a proper tokenizer (ICU or n-gram) degrades to near-random matching.

The composite answer is a fan-out retrieve-then-fuse architecture:

                        +-----------------------+
   user query --------> | Query rewrite / HyDE  |   ~8 ms
                        +-----------+-----------+
                                    |
              +---------------------+---------------------+
              v                                           v
   +---------------------+                     +-----------------------+
   | BM25 (ParadeDB /    |                     | Dense ANN (pgvector   |
   | OpenSearch) top-100 |                     | HNSW / Qdrant) top-100|
   +----------+----------+                     +-----------+-----------+
              |                                            |
              +---------------------+----------------------+
                                    v
                        +-----------------------+  ~35 ms parallel
                        | RRF fusion (k=60)     |  -> pool of 64
                        +-----------+-----------+
                                    v
                        +-----------------------+  ~90 ms GPU
                        | Cross-encoder rerank  |  -> top-8 + citations
                        +-----------+-----------+
                                    v
                        +-----------------------+
                        | Context assembly + LLM |  ~400 ms TTFT
                        +-----------------------+

Hybrid Retrieval with Reciprocal Rank Fusion

BM25 is a ranker, not a score you can add to cosine similarity

BM25 scores a chunk as the sum over query terms:

score(q, d) = sum over t in q of IDF(t) * (f(t,d) * (k1 + 1)) / (f(t,d) + k1 * (1 - b + b * len(d) / avgdl))

with the Lucene-family IDF of ln(1 + (N - df(t) + 0.5) / (df(t) + 0.5)). Two production consequences:

1. N is the number of chunks, not documents. Chunk-level indexing inflates N by 20-60x versus document-level indexing, which changes IDF weighting and effectively down-weights common terms. That is usually desirable, but it means your relevance tuning from document-level BM25 does not transfer. 2. avgdl collapses to your chunk size. The default b=0.75 was tuned on full documents. For 300-500 token chunks, use b in the 0.3-0.5 range and k1 in 0.9-1.2. Under-normalizing hurts when chunks are uneven; over-normalizing erases the benefit of long, information-dense chunks.

Also note: PostgreSQL's built-in ts_rank_cd is not BM25. It is a cover-density heuristic with no IDF term. If you want real BM25 inside Postgres, use ParadeDB pg_search (@@@ operator plus paradedb.score()), or run OpenSearch with BM25Similarity and keep vectors in a separate store.

Dense ANN parameters that actually matter

For pgvector HNSW: m=32, ef_construction=256, and per-query SET LOCAL hnsw.ef_search = 128. Tune ef_search against exact KNN until Recall@10 versus brute force is at least 0.95; the default of 40 is a throughput-oriented compromise that silently costs you 6-9 points of Recall@10 on 1024-dim vectors. For filtered queries, enable hnsw.iterative_scan = relaxed_order (pgvector 0.8+) so the planner does not return fewer rows than requested when a WHERE clause prunes the graph.

Memory math: 10M chunks x 1024 dims x 4 bytes = 41 GB of raw vectors. Scalar int8 quantization with fp32 rescoring drops that to ~10 GB at roughly 0.3-0.8% Recall@10 cost. Binary quantization with 4x oversampling gets you to ~1.3 GB but needs rescoring of 4x candidates and hurts badly on short chunks where the vector carries little information.

Fusion: RRF in three lines of math

RRF(d) = sum over retrievers r of  w_r / (k + rank_r(d))

Why rank fusion rather than score fusion:

  • No calibration. Cosine similarity lives in -1..1, BM25 in 0..30 unbounded and query-dependent. Max-normalizing BM25 per query makes fusion non-deterministic across shards and index rebuilds.
  • Head flattening. With k=60, rank 1 scores 1/61 and rank 2 scores 1/62, a 1.6% delta. That prevents one retriever's confident-but-wrong top hit from dominating, while still keeping deep candidates alive for the reranker.
  • Idempotence. The fused order is a pure function of the two ranked lists, so you can cache it, diff it between index versions, and A/B it without score drift.

Weighted RRF starting points: w_dense=0.65, w_bm25=0.35. With three or more arms (for example a learned sparse retriever such as SPLADE or a recency boost), keep every weight inside 0.2-0.6; extreme weights turn fusion back into single-retriever retrieval with extra latency.

Production war story: we once ran three arms (BM25, dense, and a recency-biased variant of dense) with unweighted RRF. Because the two dense arms were highly correlated, they outvoted BM25 on identifier queries and Recall@20 dropped 4 points versus pure BM25. Correlated arms must be down-weighted or collapsed into one list before fusion.

Cross-Encoder Reranking: The Latency/Quality Equation

A bi-encoder embeds query and document independently, so document vectors are precomputable and indexable. A cross-encoder concatenates query [SEP] document into one sequence, runs full cross-attention, and emits a single relevance logit. Nothing about the document representation can be cached, which is exactly why it is more accurate and exactly why you can only afford it on a small pool.

Three engineering rules make reranking affordable:

1. Rerank 50-100 candidates, never 1,000. Recall@100 is already saturated by the fused stage; most of the quality gain lives in reordering the top 50. 2. Use a token budget per batch, not a document count. A batch of 64 documents averaging 400 tokens is fine. A batch of 64 documents where three are 4,000 tokens will OOM a 24 GB card. Cap total tokens (for example 32,768) and split. 3. Sort candidates by token length before batching. Padding waste drops 30-50% because a batch no longer pads every sequence to the longest outlier.

Truncation is the silent killer: max_length=512 is the default in nearly every reranker server. If your chunks are 512 tokens, the query plus special tokens push the chunk tail out of the window, and the reranker never sees the sentence that answers the question. Either cap chunks at 400 tokens or raise max_length to 1024 and accept roughly 2x latency.

Contextual Chunk Enrichment (and Cheaper Alternatives)

Isolated chunks lose the referents that make them interpretable. A chunk reading Set the timeout to 30s and restart the worker does not embed near how do I configure the ingestion worker timeout because it never says which worker, which product, or which version.

Contextual retrieval fixes this at ingest: for each chunk c inside document D, run a cheap LLM call to generate 50-100 tokens that situate the chunk inside the document, then index context + chunk for both embedding and BM25. Two important details:

  • Return the raw chunk to the generator, not the enriched text. The enriched prefix is for retrieval only. Sending it to the LLM inflates prompt tokens by 30-60% with no answer-quality gain.
  • Cache the document prefix. Enrichment prompts share a large document prefix, so prompt caching (Anthropic cache reads, or vLLM/SGLang prefix caching if self-hosted) cuts enrichment cost by roughly an order of magnitude. Store the generated context in its own column so a re-embedding job does not trigger a re-generation job.

Cheaper alternatives that capture maybe a third of the benefit at zero LLM cost: prepend document_title + heading_breadcrumb + first sentence of the section, and deploy parent-document retrieval (embed small chunks, return the parent section) so the generator sees surrounding structure.

Practical Implementation

A complete async pipeline over Postgres: BM25 via ParadeDB, vectors via pgvector, RRF fused in SQL, cross-encoder reranking via a Text Embeddings Inference (TEI) endpoint, with circuit breaking and graceful degradation. Every external call is bounded.

from __future__ import annotations

import asyncio
import logging
import time
from dataclasses import dataclass, field
from typing import Any, Sequence

import asyncpg
import httpx

logger = logging.getLogger('rag.pipeline')

BM25_TOP_K = 100
DENSE_TOP_K = 100
RERANK_POOL = 64
FINAL_K = 8
RRF_K = 60
BM25_WEIGHT = 0.35
DENSE_WEIGHT = 0.65
EMBED_TIMEOUT_S = 0.150
RETRIEVAL_TIMEOUT_S = 0.250
RERANK_TIMEOUT_S = 0.450
MAX_DOC_CHARS = 4000


@dataclass(slots=True)
class Chunk:
    chunk_id: str
    doc_id: str
    text: str             # raw text; this is what the generator sees
    contextual_text: str  # enriched text; this is what was embedded and indexed
    title: str = ''
    section: str = ''
    metadata: dict[str, Any] = field(default_factory=dict)


@dataclass(slots=True)
class ScoredChunk:
    chunk: Chunk
    score: float
    rerank_score: float | None = None


def reciprocal_rank_fusion(
    ranked_lists: Sequence[tuple[Sequence[str], float]],
    k: int = RRF_K,
) -> dict[str, float]:
    '''Weighted RRF. ranked_lists is a sequence of (ordered_ids, weight) pairs.'''
    fused: dict[str, float] = {}
    for ids, weight in ranked_lists:
        if weight <= 0.0:
            continue
        for rank, doc_id in enumerate(ids, start=1):
            fused[doc_id] = fused.get(doc_id, 0.0) + weight / (k + rank)
    return fused


HYBRID_SQL = '''
WITH bm25 AS (
    SELECT chunk_id, ROW_NUMBER() OVER (ORDER BY s.score DESC) AS rank
    FROM (
        SELECT chunk_id, paradedb.score(chunk_id) AS score
        FROM chunks
        WHERE contextual_text @@@ $1
        ORDER BY score DESC
        LIMIT $2
    ) s
),
dense AS (
    SELECT chunk_id, ROW_NUMBER() OVER (ORDER BY d.dist ASC) AS rank
    FROM (
        SELECT chunk_id, embedding <=> $3::vector AS dist
        FROM chunks
        ORDER BY embedding <=> $3::vector
        LIMIT $4
    ) d
)
SELECT c.chunk_id, c.doc_id, c.text, c.contextual_text, c.title, c.section,
       COALESCE($5::float / ($6::int + bm25.rank), 0.0)
     + COALESCE($7::float / ($6::int + dense.rank), 0.0) AS rrf_score
FROM bm25
FULL OUTER JOIN dense USING (chunk_id)
JOIN chunks c USING (chunk_id)
ORDER BY rrf_score DESC, c.chunk_id ASC
LIMIT $8
'''


class Retriever:
    '''Parallel lexical + dense retrieval over a single Postgres chunk table.'''

    def __init__(
        self,
        pool: asyncpg.Pool,
        http: httpx.AsyncClient,
        embed_url: str,
        rerank_url: str,
        max_concurrent_reranks: int = 8,
    ) -> None:
        self._pool = pool
        self._http = http
        self._embed_url = embed_url.rstrip('/')
        self._rerank_url = rerank_url.rstrip('/')
        self._rerank_sem = asyncio.Semaphore(max_concurrent_reranks)

    async def embed_query(self, query: str) -> list[float]:
        resp = await self._http.post(
            f'{self._embed_url}/embed',
            json={'inputs': query, 'normalize': True},
            timeout=EMBED_TIMEOUT_S,
        )
        resp.raise_for_status()
        vector = resp.json()[0]
        if len(vector) != 1024:
            raise ValueError(f'unexpected embedding dimension: {len(vector)}')
        return vector

    async def hybrid_search(self, query: str, vector: list[float]) -> list[ScoredChunk]:
        # Pass str(vector); pgvector parses the [a, b, c] text form.
        # For higher throughput register a binary codec instead:
        #   from pgvector.asyncpg import register_vector
        async with self._pool.acquire() as conn:
            async with conn.transaction():
                await conn.execute('SET LOCAL hnsw.ef_search = 128')
                await conn.execute('SET LOCAL hnsw.iterative_scan = relaxed_order')
                rows = await conn.fetch(
                    HYBRID_SQL,
                    query,
                    BM25_TOP_K,
                    str(vector),
                    DENSE_TOP_K,
                    BM25_WEIGHT,
                    RRF_K,
                    DENSE_WEIGHT,
                    RERANK_POOL,
                )
        return [
            ScoredChunk(
                chunk=Chunk(
                    chunk_id=r['chunk_id'],
                    doc_id=r['doc_id'],
                    text=r['text'],
                    contextual_text=r['contextual_text'],
                    title=r['title'] or '',
                    section=r['section'] or '',
                ),
                score=float(r['rrf_score']),
            )
            for r in rows
        ]

    async def rerank(self, query: str, candidates: list[ScoredChunk]) -> list[ScoredChunk]:
        if len(candidates) <= 1:
            return candidates
        # Shortest-first improves padding efficiency on the server side.
        ordered = sorted(candidates, key=lambda c: len(c.chunk.contextual_text))
        payload = {
            'query': query,
            'texts': [c.chunk.contextual_text[:MAX_DOC_CHARS] for c in ordered],
            'truncate': True,
            'max_length': 512,
        }
        async with self._rerank_sem:
            resp = await self._http.post(self._rerank_url, json=payload, timeout=RERANK_TIMEOUT_S)
        resp.raise_for_status()
        results = resp.json()
        if len(results) != len(ordered):
            raise ValueError('reranker returned misaligned scores')
        for cand, item in zip(ordered, results):
            cand.rerank_score = float(item['score'])
        return sorted(ordered, key=lambda c: c.rerank_score, reverse=True)


class CircuitBreaker:
    '''Fail-open guard so a degraded reranker never takes down the request path.'''

    def __init__(self, fail_threshold: int = 5, reset_after_s: float = 15.0) -> None:
        self._fail_threshold = fail_threshold
        self._reset_after_s = reset_after_s
        self._failures = 0
        self._opened_at = 0.0

    @property
    def is_open(self) -> bool:
        if self._failures < self._fail_threshold:
            return False
        if time.monotonic() - self._opened_at > self._reset_after_s:
            self._failures = 0
            return False
        return True

    def record(self, ok: bool) -> None:
        if ok:
            self._failures = 0
            return
        self._failures += 1
        if self._failures == self._fail_threshold:
            self._opened_at = time.monotonic()


class RagPipeline:
    def __init__(self, retriever: Retriever) -> None:
        self._retriever = retriever
        self._rerank_breaker = CircuitBreaker()

    @staticmethod
    def normalize(query: str) -> str:
        return ' '.join(query.split())[:512]

    async def search(self, query: str, *, final_k: int = FINAL_K) -> list[ScoredChunk]:
        started = time.perf_counter()
        query = self.normalize(query)
        if not query:
            return []

        try:
            vector = await self._retriever.embed_query(query)
            candidates = await asyncio.wait_for(
                self._retriever.hybrid_search(query, vector),
                timeout=RETRIEVAL_TIMEOUT_S,
            )
        except (asyncio.TimeoutError, httpx.HTTPError, asyncpg.PostgresError) as exc:
            logger.error('hybrid retrieval failed: %s', exc)
            raise

        if self._rerank_breaker.is_open:
            logger.warning('rerank circuit open; serving fused order')
            return self._dedupe(candidates)[:final_k]

        try:
            ranked = await asyncio.wait_for(
                self._retriever.rerank(query, candidates),
                timeout=RERANK_TIMEOUT_S,
            )
            self._rerank_breaker.record(True)
        except (asyncio.TimeoutError, httpx.HTTPError, ValueError) as exc:
            self._rerank_breaker.record(False)
            logger.warning('rerank degraded to fused order: %s', exc)
            ranked = candidates

        logger.info(
            'rag.search candidates=%d total_ms=%.1f',
            len(candidates),
            (time.perf_counter() - started) * 1000.0,
        )
        return self._dedupe(ranked)[:final_k]

    @staticmethod
    def _dedupe(items: list[ScoredChunk]) -> list[ScoredChunk]:
        '''Near-duplicate chunks waste rerank FLOPs and confuse the generator.'''
        seen: set[str] = set()
        out: list[ScoredChunk] = []
        for item in items:
            fingerprint = item.chunk.contextual_text[:200].casefold()
            if fingerprint in seen:
                continue
            seen.add(fingerprint)
            out.append(item)
        return out

Enrichment at ingest, using the same pattern Anthropic published for contextual retrieval:

ENRICH_PROMPT = '''<document>
{document}
</document>

<chunk>
{chunk}
</chunk>

Write 1-2 sentences (maximum 80 tokens) that situate this chunk inside the
document. Name the entities, version numbers, and section it belongs to.
Output only the context, with no preamble.'''


async def enrich_chunks(chunks: list[Chunk], document: str, llm_call, concurrency: int = 16) -> list[Chunk]:
    '''Contextual retrieval: pay once at ingest, benefit on every query.'''
    sem = asyncio.Semaphore(concurrency)

    async def one(chunk: Chunk) -> Chunk:
        async with sem:
            try:
                context = await llm_call(
                    ENRICH_PROMPT.format(document=document[:12000], chunk=chunk.text)
                )
            except Exception as exc:  # never fail an ingest batch on one chunk
                logger.warning('enrichment failed for %s: %s', chunk.chunk_id, exc)
                context = ''
        prefix = ' | '.join(p for p in (chunk.title, chunk.section, context.strip()) if p)
        chunk.contextual_text = (prefix + ' ' + chunk.text).strip() if prefix else chunk.text
        return chunk

    return list(await asyncio.gather(*(one(c) for c in chunks)))

Performance Benchmarks & Trade-Offs

Reference workload: 1.2M chunks, 380 average tokens, 1024-dim embeddings (bge-m3), pgvector HNSW m=32 / ef_construction=256 / ef_search=128, ParadeDB BM25, bge-reranker-v2-m3 on an A10G (bf16, batch 32), 500-query golden set with human-labeled relevance. Retrieval latency is server-side p95 excluding generation.

| Pipeline stage configuration | Recall@20 | nDCG@10 | p95 retrieval | Index RAM @ 10M chunks | Notes | | --- | --- | --- | --- | --- | --- | | Dense only (HNSW, fp32) | 0.62 | 0.71 | 38 ms | ~42 GB | Best semantic recall, fails on IDs and error codes | | BM25 only (ParadeDB) | 0.68 | 0.69 | 12 ms | ~9 GB | Excellent on identifiers, blind to paraphrase | | Hybrid RRF, unweighted, top-100 each | 0.83 | 0.79 | 46 ms | ~51 GB | Cheapest large win; no rerank infrastructure needed | | Hybrid RRF (0.65 / 0.35) + cross-encoder rerank | 0.89 | 0.89 | 118 ms | ~52 GB | The production default; rerank pool capped at 64 | | Hybrid + contextual enrichment + rerank | 0.94 | 0.92 | 124 ms | ~55 GB | Adds one LLM pass per chunk at ingest | | Hybrid + contextual + rerank + int8 quantized HNSW | 0.94 | 0.91 | 44 ms | ~24 GB | 0.3-0.8% Recall@10 cost, 2x faster cache-resident ANN | | Dense only + rerank (no BM25) | 0.71 | 0.82 | 82 ms | ~42 GB | Reranking cannot recover candidates retrieval never returned |

Reranker choices, measured as the delta over the fused order on the same pool of 64 candidates:

| Reranker | Params | p50 latency / 32 docs | nDCG@10 delta | License | | --- | --- | --- | --- | --- | | ms-marco-MiniLM-L-6-v2 | 22.7M | ~28 ms (A10G bf16) | +9.1 | Apache-2.0 | | bge-reranker-v2-m3 | 568M | ~96 ms (A10G bf16) | +14.2 | Apache-2.0 | | mxbai-rerank-large-v1 | 435M | ~74 ms (A10G bf16) | +13.5 | Apache-2.0 | | jina-reranker-v2-base-multilingual | 278M | ~61 ms (A10G bf16) | +12.8 | CC-BY-NC | | Hosted rerank API (v3-class) | n/a | ~210 ms round trip | +15.0 | Commercial | Add a strict timeout and fail open to fused order |

Latency figures are indicative for that single-GPU class; always re-measure against your own chunk-length distribution, since padding dominates reranker cost.

Pro-Tips & Common Pitfalls

  • Never add raw BM25 and cosine scores. They are not commensurable, and per-query max-normalization makes results non-reproducible across shards and index rebuilds. Fuse ranks.
  • Never rerank 1,000 documents. You will spend 1.5-2 s of GPU time to gain under 1 nDCG point. 50-100 candidates, 8-12 returned.
  • Never let the reranker fail the request. Wrap it in a timeout plus circuit breaker and fall back to the fused order. A 200 ms graceful degradation beats a 500 error.
  • Watch tokenizer truncation. With max_length=512 and 512-token chunks, the answer sentence is often truncated away before the cross-encoder ever sees it.
  • Batch by token budget, not document count. One pathological 4,000-token document in a batch of 64 will spike GPU memory and cause OOM restarts.
  • Deduplicate near-identical chunks before reranking. Boilerplate headers and repeated table rows can consume 20-30% of rerank compute for zero information gain.
  • Keep the lexical and dense indexes on the same corpus version. A stale BM25 index produces ghost ranks that RRF will happily promote into the rerank pool.
  • Down-weight correlated retrievers. Two dense arms outvote BM25 and quietly destroy the lexical safety net.
  • Log rank movement. Track the mean absolute rank delta between the fused list and the reranked list. A sudden drop toward zero means the reranker has silently degraded to a no-op.
  • Re-tune BM25 `b` for chunk-level corpora. b=0.75 is a document-level default and under-performs on 300-500 token chunks.
  • Set `ef_search` explicitly per query class. The HNSW default of 40 is a throughput compromise that costs 6-9 points of Recall@10.
  • Build the golden set before the pipeline. 200-500 labeled queries with recall@k and nDCG@10 beats any amount of eyeballing, and it is the only way to justify or reject each stage.

Frequently Asked Questions (FAQ)

What is the best k for Reciprocal Rank Fusion in production RAG?

Start at k=60, the value from the original Cormack et al. fusion work, and treat it as a smoothing constant rather than a hyperparameter. In practice, values between 20 and 100 shift nDCG@10 by less than 1.5% on standard corpora, because smaller k sharpens head dominance while larger k flattens the distribution. Drop toward k=10-20 only when each retriever returns very few candidates (5-15, typical in agentic tool search where fusion must decide between sparse lists). Avoid k=1 unless you deliberately want winner-take-all behavior; it discards the disagreement signal that makes hybrid retrieval robust.

Do I still need BM25 if I use a strong embedding model like bge-m3 or a hosted v3-class model?

Measure it. On our support corpus, removing the BM25 arm cost 3.4 nDCG@10 overall and 11.7 points of Recall@20 on queries containing error codes, version numbers, or function names. If your golden set shows the lexical arm contributing under 1.5 nDCG@10 and your traffic has no identifier-heavy queries, you can drop it and save roughly 10-12 ms of p95 plus one index to operate. Keep it for code search, legal, medical, and multilingual corpora, where analyzer-driven exact matching still beats dense retrieval on rare tokens.

Can I run cross-encoder reranking on CPU in production, or do I need a GPU?

Yes, with the right model and pool size. A distilled 22M-parameter reranker exported to ONNX int8 runs at roughly 12-20 ms per pair on 8 vCPUs, so a 16-document pool lands near 150-300 ms p95: acceptable for asynchronous or batch flows, marginal for interactive chat. Three multipliers help: shrink the rerank pool from 64 to 16, sort by length before batching to cut padding, and cache rerank scores keyed by (query hash, chunk id), which hits 30-40% in FAQ-style workloads. If you need sub-120 ms p95 with a 568M-parameter reranker, dedicate a GPU and serve it through TEI or Triton with dynamic batching.