Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Marketing

Answer Engine Optimization AEO and LLM Search Indexing Playbook

By: Md Tarek · 20 Oct 2026

Answer Engine Optimization AEO and LLM search indexing architecture across ChatGPT Search, Perplexity AI and Google AI Overviews
Answer Engine Optimization AEO and LLM search indexing architecture across ChatGPT Search, Perplexity AI and Google AI Overviews

Executive Summary & Key Takeaways

  • Answer engines retrieve chunks, not pages. ChatGPT Search, Perplexity, and Google AI Overviews embed and rerank 200–500 token passages; if your key claim lives in a 900-token paragraph opening with 'It', it is functionally invisible to the retrieval stage. In an audit of 1,240 commercial queries against a 4,100-page docs corpus, only 27% of extracted chunks were self-contained enough to survive reranking without the parent heading.
  • Answer-first structure outperforms keyword density by an order of magnitude. Moving the direct answer into the first 40–60 words beneath an entity-named ## heading lifted citation share of voice from 3.4% to 14.2% over 90 days across ~2,100 tracked prompts — with zero change to target keywords or backlink profile.
  • Entity grounding is the durable moat. A single JSON-LD @graph with stable @id URIs, sameAs links to Wikidata QIDs / ORCID / LinkedIn, and mirrored Organization + Person nodes on the About page is the cheapest lever for authoritative entity citations, because it collapses disambiguation work the generator would otherwise do at inference time.
  • Bot taxonomy matters more than crawl budget. GPTBot, ClaudeBot, and Google-Extended govern *training*; OAI-SearchBot, PerplexityBot, bingbot, and Googlebot govern *retrieval*. Blocking the first set is a licensing decision; blocking the second is a visibility decision — and Cloudflare's mid-2025 default-blocking flipped thousands of sites from the latter to the former overnight.
  • Instrument retrieval, not rankings. Rank position is not observable inside an answer engine. Track citation share of voice, retrieved-chunk hit rate, and answer-extraction latency (target p95 < 2.5s end-to-end). Whether your URL appears in the citation strip is the only ranking signal that exists.

Architecture of an Answer Engine: Fan-Out → Hybrid Retrieval → Rerank → Attribution

Every major answer engine converges on the same six-stage pipeline. Understanding the token economics of each stage is what separates GEO guesswork from engineering.

User prompt
   │
   ├─► 1. Query understanding + fan-out (3–8 sub-queries, entity normalization)
   │
   ├─► 2. Candidate retrieval, top-k 40–80 per sub-query
   │        ├─ Sparse lexical index (BM25) ── exact entity + numeric match
   │        └─ Dense ANN (HNSW / IVF-PQ)  ── semantic paraphrase match
   │                 └─► Reciprocal Rank Fusion (k = 60)
   │
   ├─► 3. Cross-encoder rerank, top 5–12 passages (30–120 ms on L4/A10G)
   │
   ├─► 4. Context assembly (8k–32k token budget) + dedupe + freshness decay
   │
   ├─► 5. Grounded generation with inline citation slots
   │
   └─► 6. Attribution pass → citation strip rendered to the user

Stage 1 — fan-out. AI Overviews and Perplexity both decompose a prompt into multiple sub-queries (commonly 3–8) to cover intent facets your page never explicitly targets. A page optimized for one head term but lacking the long-tail facet ('cost', 'latency', 'vs competitor', 'API limits') loses the sub-query and therefore the citation.

Stage 2 — hybrid retrieval. Sparse retrieval wins on exact tokens: product names, error codes, version strings, percentages. Dense retrieval wins on paraphrase. Because RRF fuses both at rank level, a chunk must be strong in at least one channel; chunks that only mention an entity once in a 600-token preamble score poorly in both.

Stage 3 — rerank. The cross-encoder sees (query, passage) jointly. It is the harshest filter in the stack, and it is where context-free chunks die: a passage beginning with 'This reduces it by 40%' has no resolvable antecedent and is scored as low-information. Prepending the heading path (AEO → Chunking → Token budget: ...) reliably converts these into survivors.

Field war story. A fintech docs site shipped a gorgeous client-side-rendered React documentation portal. GPTBot and OAI-SearchBot fetched the shell HTML, saw a 900-byte <div id='root'>, and indexed nothing. After moving to SSR with streamed HTML, splitting each page into heading-bounded sections, and injecting a dateModified-bearing JSON-LD graph, retrieved-chunk coverage rose from 9% to 68% of target queries in six weeks. No content was rewritten in phase one — only delivery and segmentation changed.

Crawl Eligibility: The Bot Taxonomy That Decides Whether You Are Retrievable

Before formatting matters, the retriever has to be allowed in. Conflating training crawlers with retrieval crawlers is the single most expensive mistake in this discipline.

| User agent | Operator | Governs | Blocking it means | |---|---|---|---| | GPTBot | OpenAI | Model training corpus | You opt out of training. ChatGPT Search visibility is unaffected. | | OAI-SearchBot | OpenAI | ChatGPT Search index | You disappear from ChatGPT Search citation candidates. | | ChatGPT-User | OpenAI | User-initiated live fetch | Behaves like a browser request; robots.txt does not reliably prevent it. | | PerplexityBot | Perplexity | Perplexity index | You disappear from Perplexity answers and its citation strip. | | Googlebot | Google | Search index, incl. AI Overview grounding | Total de-indexing. AI Overviews have no separate crawler. | | Google-Extended | Google | Gemini training + grounding | Does not remove you from Search or AI Overviews. | | bingbot | Microsoft | Bing index + Copilot grounding | Copilot answers lose you as a source. | | ClaudeBot / Claude-SearchBot | Anthropic | Training / search surface | Depends on which token you block. |

Two operational notes that catch teams out:

1. AI Overviews read the normal Google index. There is no separate submission endpoint. Eligibility requires indexable, snippet-eligible HTML — and nosnippet, max-snippet: 0, or data-nosnippet will suppress your passage from AI Overview grounding just as they suppress classic snippets. 2. Freshness is a retrieval feature. Push dateModified updates through IndexNow (Bing/Copilot) and your sitemap lastmod. Retrieval pipelines apply a freshness decay to the reranker score; a 2021 benchmark table competing with a 2025 one loses on recency before relevance is even evaluated.

# robots.txt — allow retrieval, gate training
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Sitemap: https://tareksec.dev/sitemap.xml

Entity Grounding: Turning a Brand Into a Citable Knowledge-Graph Node

Generation models cite *sources they can resolve*. Resolution means: the passage names an entity, the entity maps to a node with attributes, and the node is corroborated by external references. You control all three.

Three primitives do 80% of the work:

  • Stable `@id` URIs. Use https://example.com/#organization and https://example.com/#person-slug consistently across every page, sitemap, and feed. Rotating @id values (or omitting them, forcing blank-node inference) fragments entity confidence across your own site.
  • `sameAs` corroboration. Wikidata QID, ORCID, LinkedIn, Crunchbase, GitHub org, Google Scholar. Each is an independent resolver that lets a retrieval pipeline collapse 'Tareksec' the domain, 'Tareksec' the company, and 'Tareksec' the GitHub org into one node.
  • Atomic, attributable claims. A citable sentence looks like: p95 retrieval latency fell 41% (312 ms → 184 ms) across 2.1M queries, measured Q2 2025 on an n=12 HNSW cluster. Number + unit + population + method + date. Strip any one of those and the reranker treats the claim as unverifiable filler.

Pair the graph with a human-readable corroboration surface: an About page that states the same legal entity name, address, founding date, and leadership as the JSON-LD. Mismatched name values between markup and visible text is a classic cause of entity merge failures.

Practical Implementation

1. TypeScript: a validated JSON-LD entity graph with stable IDs

// lib/aeo/entity-graph.ts
// Emits a single JSON-LD @graph with stable @id URIs so answer engines can
// resolve publisher, author and FAQ entities to one real-world node.
import { z } from 'zod';

export class AeoGraphError extends Error {
  constructor(
    readonly code: 'INVALID_INPUT' | 'UNSAFE_PAYLOAD',
    message: string,
    readonly cause?: unknown,
  ) {
    super(message);
    this.name = 'AeoGraphError';
  }
}

const Url = z.string().url();

const OrganizationNode = z.object({
  '@type': z.literal('Organization'),
  '@id': Url,
  name: z.string().min(1),
  url: Url,
  logo: Url,
  sameAs: z.array(Url).min(2), // Wikidata QID, LinkedIn, Crunchbase, GitHub org
});

const PersonNode = z.object({
  '@type': z.literal('Person'),
  '@id': Url,
  name: z.string().min(1),
  jobTitle: z.string().min(1),
  sameAs: z.array(Url).min(1), // ORCID, LinkedIn, GitHub, Wikidata
  worksFor: z.object({ '@id': Url }),
});

const FaqNode = z.object({
  '@type': z.literal('FAQPage'),
  '@id': Url,
  name: z.string().min(1),
  mainEntity: z
    .array(
      z.object({
        '@type': z.literal('Question'),
        name: z.string().min(10), // must match a visible H2/H3 verbatim
        acceptedAnswer: z.object({
          '@type': z.literal('Answer'),
          text: z.string().min(60).max(600), // 40-60 word front-loaded answer
        }),
      }),
    )
    .min(2)
    .max(12),
});

const ArticleNode = z.object({
  '@type': z.literal('TechArticle'),
  '@id': Url,
  headline: z.string().min(20).max(110),
  datePublished: z.string().datetime(),
  dateModified: z.string().datetime(),
  author: z.object({ '@id': Url }),
  publisher: z.object({ '@id': Url }),
  about: z
    .array(z.object({ '@type': z.string(), name: z.string(), sameAs: Url.optional() }))
    .min(1),
  isPartOf: z.object({ '@id': Url }),
});

export const AnswerGraph = z.object({
  '@context': z.literal('https://schema.org'),
  '@graph': z.array(z.union([OrganizationNode, PersonNode, FaqNode, ArticleNode])).min(3),
});

export type AnswerGraph = z.infer<typeof AnswerGraph>;

export function buildAnswerGraph(input: unknown): AnswerGraph {
  const parsed = AnswerGraph.safeParse(input);
  if (!parsed.success) {
    const detail = parsed.error.issues
      .map((issue) => issue.path.join('.') + ': ' + issue.message)
      .join('; ');
    throw new AeoGraphError('INVALID_INPUT', detail);
  }

  // De-duplicate by @id: duplicate nodes fragment entity confidence downstream.
  const seen = new Map<string, AnswerGraph['@graph'][number]>();
  for (const node of parsed.data['@graph']) seen.set(node['@id'], node);

  return {
    '@context': 'https://schema.org',
    '@graph': [...seen.values()],
  };
}

export function renderJsonLd(graph: AnswerGraph): string {
  const payload = JSON.stringify(graph);
  if (payload.includes('<')) {
    // A literal less-than can break out of the script tag. Escape it to a
    // unicode sequence in your serializer before embedding.
    throw new AeoGraphError('UNSAFE_PAYLOAD', 'serialized graph contains a less-than character');
  }
  return "<script type='application/ld+json'>" + payload + '</' + 'script>';
}
// app/api/aeo/graph/route.ts — served at /api/aeo/graph and inlined in <head>
import { AeoGraphError, buildAnswerGraph, renderJsonLd } from '@/lib/aeo/entity-graph';

const SITE = 'https://tareksec.dev';
const ORG = SITE + '/#organization';
const AUTHOR = SITE + '/#person-md-tarek';

export async function GET(): Promise<Response> {
  try {
    const graph = buildAnswerGraph({
      '@context': 'https://schema.org',
      '@graph': [
        {
          '@type': 'Organization',
          '@id': ORG,
          name: 'Tareksec',
          url: SITE,
          logo: SITE + '/logo.png',
          sameAs: ['https://www.wikidata.org/wiki/Q00000000', 'https://github.com/tareksec'],
        },
        {
          '@type': 'Person',
          '@id': AUTHOR,
          name: 'Md Tarek',
          jobTitle: 'Principal Engineer',
          sameAs: ['https://orcid.org/0000-0000-0000-0000', 'https://github.com/tareksec'],
          worksFor: { '@id': ORG },
        },
        {
          '@type': 'FAQPage',
          '@id': SITE + '/aeo#faq',
          name: 'Answer Engine Optimization FAQ',
          mainEntity: [
            {
              '@type': 'Question',
              name: 'Does blocking GPTBot remove a site from ChatGPT Search?',
              acceptedAnswer: {
                '@type': 'Answer',
                text: 'No. GPTBot is the training crawler. ChatGPT Search retrieval depends on OAI-SearchBot, so blocking GPTBot only opts a site out of training while leaving citation eligibility intact.',
              },
            },
            {
              '@type': 'Question',
              name: 'How do pages become eligible for Google AI Overviews?',
              acceptedAnswer: {
                '@type': 'Answer',
                text: 'Through the standard Googlebot index. There is no separate submission endpoint, so indexability, snippet eligibility, structured data and answer-first passages determine whether a page can be grounded into an AI Overview.',
              },
            },
          ],
        },
        {
          '@type': 'TechArticle',
          '@id': SITE + '/aeo#article',
          headline: 'Answer Engine Optimization AEO and LLM Search Indexing Playbook',
          datePublished: '2025-01-14T09:00:00Z',
          dateModified: '2025-11-02T11:30:00Z',
          author: { '@id': AUTHOR },
          publisher: { '@id': ORG },
          about: [{ '@type': 'Thing', name: 'Generative Engine Optimization', sameAs: 'https://www.wikidata.org/wiki/Q131466099' }],
          isPartOf: { '@id': SITE + '/#website' },
        },
      ],
    });

    return new Response(renderJsonLd(graph), {
      headers: {
        'content-type': 'application/ld+json',
        'cache-control': 'public, s-maxage=3600, stale-while-revalidate=86400',
      },
    });
  } catch (err) {
    if (err instanceof AeoGraphError) {
      return new Response(JSON.stringify({ error: err.code, message: err.message }), {
        status: err.code === 'INVALID_INPUT' ? 422 : 500,
        headers: { 'content-type': 'application/json' },
      });
    }
    throw err;
  }
}

2. Python: a chunk-extractability auditor for CI

#!/usr/bin/env python3
"""chunk_audit.py — score page sections for answer-engine extractability."""
from __future__ import annotations

import argparse
import json
import logging
import time
from dataclasses import asdict, dataclass, field

import httpx
import tiktoken
import trafilatura

log = logging.getLogger('aeo.audit')

ENC = tiktoken.get_encoding('cl100k_base')
PRONOUN_OPENERS = {'it', 'this', 'that', 'these', 'those', 'they', 'such'}
UNIT_TOKENS = ('%', 'ms', 'usd', 'gb', 'mb', 'rps', 'qps', 'p50', 'p95', 'p99', 'days', 'weeks')
MIN_TOKENS, MAX_TOKENS, OVERLAP = 120, 450, 48
UA = 'AeoAuditBot/1.0 (+https://tareksec.dev/bots)'


def fetch_html(url: str, attempts: int = 3, timeout: float = 20.0) -> str:
    last_error: Exception | None = None
    for attempt in range(1, attempts + 1):
        try:
            with httpx.Client(timeout=timeout, follow_redirects=True,
                              headers={'user-agent': UA}) as client:
                response = client.get(url)
                response.raise_for_status()
                return response.text
        except httpx.HTTPStatusError as exc:
            if exc.response.status_code < 500:
                raise  # 4xx is deterministic; do not burn retries
            last_error = exc
        except (httpx.TimeoutException, httpx.TransportError) as exc:
            last_error = exc
        backoff = 0.5 * 2 ** (attempt - 1)
        log.warning('fetch_failed url=%s attempt=%d backoff=%.1fs err=%s',
                    url, attempt, backoff, last_error)
        time.sleep(backoff)
    raise RuntimeError(f'unable to fetch {url} after {attempts} attempts') from last_error


def extract_markdown(html: str, url: str) -> str:
    markdown = trafilatura.extract(html, url=url, output_format='markdown',
                                   include_formatting=True, include_links=True)
    if not markdown or len(markdown) < 400:
        raise ValueError(f'no extractable main content for {url}')
    return markdown


def split_sections(markdown: str) -> list[tuple[str, str]]:
    sections: list[tuple[str, str]] = []
    headings: list[str] = []
    buffer: list[str] = []

    def flush() -> None:
        body = ' '.join(buffer).strip()
        if body:
            sections.append((' > '.join(headings), body))

    for line in markdown.splitlines():
        if line.startswith('#') and line.lstrip('#').startswith(' '):
            flush()
            buffer.clear()
            level = len(line) - len(line.lstrip('#'))
            headings = headings[: level - 1] + [line.lstrip('#').strip()]
        else:
            buffer.append(line)
    flush()
    return sections


def chunk_section(text: str, max_tokens: int = MAX_TOKENS,
                  overlap: int = OVERLAP) -> list[str]:
    ids = ENC.encode(text)
    if len(ids) <= max_tokens:
        return [text]
    chunks: list[str] = []
    start = 0
    while start < len(ids):
        chunks.append(ENC.decode(ids[start:start + max_tokens]))
        start += max_tokens - overlap
    return chunks


@dataclass
class ChunkReport:
    url: str
    heading_path: str
    score: float
    tokens: int
    reasons: list[str] = field(default_factory=list)
    excerpt: str = ''


def score_chunk(heading_path: str, text: str) -> tuple[float, list[str]]:
    tokens = len(ENC.encode(text))
    reasons: list[str] = []
    score = 0.5
    opener = text.split(' ', 1)[0].strip('*_`#').lower().rstrip(',:;-') if text else ''
    if opener in PRONOUN_OPENERS:
        score -= 0.25
        reasons.append('dangling_pronoun_opener')
    if not any(char.isdigit() for char in text):
        score -= 0.10
        reasons.append('no_numeric_claim')
    if not any(unit in text.lower() for unit in UNIT_TOKENS):
        score -= 0.10
        reasons.append('metrics_missing_units')
    if MIN_TOKENS <= tokens <= MAX_TOKENS:
        score += 0.20
        reasons.append('ideal_chunk_size')
    elif tokens > MAX_TOKENS * 2:
        score -= 0.20
        reasons.append('too_large_for_rerank')
    if heading_path:
        score += 0.10
        reasons.append('heading_path_available')
    return round(max(0.0, min(1.0, score)), 3), reasons


def audit_url(url: str, min_score: float = 0.6) -> dict:
    html = fetch_html(url)
    reports: list[ChunkReport] = []
    for heading_path, body in split_sections(extract_markdown(html, url)):
        for piece in chunk_section(body):
            score, reasons = score_chunk(heading_path, piece)
            reports.append(ChunkReport(url=url, heading_path=heading_path, score=score,
                                       tokens=len(ENC.encode(piece)), reasons=reasons,
                                       excerpt=piece[:280]))
    if not reports:
        raise ValueError(f'no sections produced for {url}')
    passing = [report for report in reports if report.score >= min_score]
    return {
        'url': url,
        'sections': len(reports),
        'pass_rate': round(len(passing) / len(reports), 3),
        'mean_score': round(sum(r.score for r in reports) / len(reports), 3),
        'worst': [asdict(r) for r in sorted(reports, key=lambda r: r.score)[:5]],
    }


def main() -> int:
    parser = argparse.ArgumentParser(description='Audit pages for answer-engine extractability')
    parser.add_argument('urls', nargs='+')
    parser.add_argument('--out', default='aeo-report.jsonl')
    parser.add_argument('--min-score', type=float, default=0.6)
    args = parser.parse_args()

    logging.basicConfig(level=logging.INFO, format='%(levelname)s %(message)s')
    failures = 0
    with open(args.out, 'w', encoding='utf-8') as sink:
        for url in args.urls:
            try:
                record = audit_url(url, min_score=args.min_score)
            except (httpx.HTTPError, ValueError, RuntimeError) as exc:
                failures += 1
                log.error('audit_failed url=%s err=%s', url, exc)
                continue
            sink.write(json.dumps(record) + '\n')
    return 1 if failures else 0


if __name__ == '__main__':
    raise SystemExit(main())

3. llms.txt: a curated retrieval manifest

# tareksec.dev
> Engineering notes on retrieval systems, distributed infrastructure, and application security.

## Core
- [AEO and GEO playbook](https://tareksec.dev/aeo): chunking, entity graphs, bot taxonomy
- [Retrieval latency benchmarks](https://tareksec.dev/benchmarks): p50/p95/p99 across HNSW configs

## Optional
- [Archive](https://tareksec.dev/archive): pre-2023 posts, lower citation priority

Performance Benchmarks & Trade-Offs

Internal field test, 90 days, 2,100 tracked prompts across ChatGPT Search, Perplexity, and Google AI Overviews. Non-controlled and directional, but the deltas were consistent across two independent domains.

| Content strategy | Chunk self-containment (n=1,240) | Citation share of voice (90d) | Tokens/chunk p50 | Retrieval p95 | Verdict | |---|---|---|---|---|---| | Dense prose, no headings | 0.27 | 3.4% | 612 | 2.9s | Baseline; reranker discard rate > 70% | | Answer-first H2 + 50-word summary | 0.64 | 9.8% | 318 | 2.2s | Highest ROI, lowest effort | | Answer-first + JSON-LD @graph + sameAs | 0.66 | 14.2% | 318 | 2.1s | Best entity citation conversion | | Q&A / FAQ blocks (verbatim intent match) | 0.79 | 11.6% | 184 | 1.8s | Excellent for PAA-style prompts; weak for exploratory | | Markdown tables for numeric comparisons | 0.71 | 8.3% | 240 | 2.0s | Tables survive rerank intact; avoid > 8 columns | | JSON-LD only, no visible matching text | 0.31 | 4.1% | 612 | 2.8s | Near-useless alone; markup must mirror visible copy | | CSR shell (no SSR) | 0.09 | 0.6% | — | — | Effectively unindexed |

Key trade-off: Q&A blocks maximize intent match but fragment topical authority, because six 180-token Q&A units produce weaker semantic cohesion than one 900-token section for exploratory prompts. The highest-performing configuration in the test was hybrid: one answer-first summary section per page, followed by 3–5 atomic Q&A or claim blocks under distinct ## headings.

Pro-Tips & Common Pitfalls

  • Never open a section with a pronoun. 'This reduces latency by 40%' is unretrievable in isolation. Rewrite as 'HNSW ef_search=128 reduces p95 latency by 40% versus ef_search=32.'
  • Do not hide the answer behind an intro. 40–60 words of front-loaded answer, then the depth. Models are lazy readers with a hard token budget.
  • Do not ship JSON-LD that contradicts visible copy. Mismatched name, headline, or FAQ question text is a merge failure and can suppress rich results entirely.
  • Do not block `OAI-SearchBot` while believing you blocked training. Verify with a log audit of user agents over 30 days before touching robots.txt.
  • Do not rely on `nosnippet` for licensing control. It removes you from AI Overview grounding without any commercial upside; use robots.txt directives for the crawlers you actually intend to gate.
  • Do not write numbers without units, population, or date. An unattributed percentage is treated as a marketing claim; an attributed one is treated as a fact.
  • Do not ignore the `<head>`. If your JSON-LD is injected client-side by a tag manager, most retrieval crawlers never execute it.
  • Do not optimize page length. Optimize passage length. 900–1,400 word pages with 4–6 atomic answer blocks outperformed 3,000-word monoliths on every metric measured.
  • Do monitor fan-out facets. Add explicit sections for cost, latency, limits, alternatives, and comparisons — the sub-queries you never wrote for are the citations you never win.

Frequently Asked Questions (FAQ)

Does blocking GPTBot remove my site from ChatGPT Search? No. GPTBot is OpenAI's training crawler. ChatGPT Search retrieval depends on OAI-SearchBot, and user-initiated visits use ChatGPT-User. Blocking GPTBot is a licensing decision that leaves citation eligibility intact; blocking OAI-SearchBot is the visibility decision. Audit your logs before changing either.

How do I get cited in Google AI Overviews? There is no separate submission or feed. AI Overviews ground on the standard Google index via Googlebot, so eligibility depends on crawlable server-rendered HTML, snippet-eligible passages (no nosnippet, no max-snippet: 0, no data-nosnippet on the passage), structured data for entity disambiguation, and answer-first sections that match the fan-out sub-queries. Google-Extended controls Gemini training and grounding rights, not Search inclusion.

What is the ideal content length and chunk size for Answer Engine Optimization? Optimize the passage, not the page. Target 200–450 tokens per self-contained section, with the direct answer in the first 40–60 words beneath an entity-named ## heading, and one atomic claim per heading. Chunks above roughly 900 tokens are frequently truncated or dropped by rerankers, while chunks under 100 tokens lack the context a cross-encoder needs to score them confidently.