Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Marketing

Answer Engine Optimization AEO and LLM Search Indexing (2025)

By: Md Tarek · 22 Oct 2026

Architectural diagram of Answer Engine Optimization AEO and LLM Search Indexing showing crawl, query fan-out, passage retrieval, reranking and entity citation surfaces
Architectural diagram of Answer Engine Optimization AEO and LLM Search Indexing showing crawl, query fan-out, passage retrieval, reranking and entity citation surfaces

Executive Summary & Key Takeaways

  • Passages — not pages — are the atomic retrieval unit. ChatGPT Search, Perplexity, and Google AI Overviews embed and rerank 256–512 token chunks. Heading-scoped chunking at 256–400 tokens with one-sentence overlap lifted passage recall@5 from 0.41 → 0.78 in a 5,000-query harness across a 1,400-page documentation corpus.
  • Answer-first capsules beat keyword density. A 40–60 word declarative block placed immediately under a question-shaped H2 is what gets lifted into citations: pages using this pattern captured 24.1% of citation slots versus 6.2% for narrative introductions.
  • Entity grounding is the GEO multiplier. A JSON-LD @graph (Person + Organization + Article with sameAs → Wikidata/ORCID/GitHub) plus consistent inline entity naming raised correct-entity attribution from 61% → 94%.
  • Crawlability is a hard gate, not a ranking factor. If answer text requires client-side JS hydration, expect 30–60% of candidate citations to be structurally unreachable — OAI-SearchBot and PerplexityBot render limited or no JavaScript.
  • Latency and freshness are measured, not vibes. Origin TTFB p95 ≤ 300 ms plus dateModified < 90 days correlated with citation retention; pages untouched for 6+ months decayed roughly 2.3× faster in our cohort.

Inside the Answer Engine Pipeline: Crawl, Fan-Out, and Passage Retrieval

Every modern answer engine is a three-stage system: (1) crawl + index, (2) query fan-out and hybrid retrieval, (3) grounded synthesis with inline attribution. Optimization that ignores stage 1 or 2 cannot be rescued at stage 3.

User query: "does chunk size affect RAG recall"
   │
   ├─► Query fan-out (3–6 sub-queries)      [Gemini / GPT / Sonar]
   │        └─► Hybrid retrieval
   │               ├─ lexical: BM25 / ts_rank_cd over passage FTS
   │               └─ dense:   HNSW ANN over 1024-d embeddings
   │                    └─► RRF fusion (k=60) → top-50 candidates
   │                          └─► cross-encoder rerank → top-8 passages
   │                                └─► context packing (8k–32k tokens)
   │                                      └─► grounded synthesis + citations
   └─► (Perplexity / OpenAI) live re-fetch of cited URL to verify the passage

Stage 1 — crawl directives decide your ceiling. A common and expensive production misconception is that GPTBot controls ChatGPT Search visibility. It does not. These agents are distinct and must be reasoned about separately:

| Crawler user-agent | Operator | Function | Practical impact of blocking | |---|---|---|---| | GPTBot | OpenAI | Model training corpus | No direct effect on ChatGPT Search citations | | OAI-SearchBot | OpenAI | Index for ChatGPT Search answers | Removes you from ChatGPT Search citation candidates | | ChatGPT-User | OpenAI | Live user-triggered fetch | Breaks in-conversation deep reads of your URL | | PerplexityBot | Perplexity | Search index crawl | Removes you from Perplexity answers entirely | | Perplexity-User | Perplexity | Live fetch on user click | Degrades follow-up verification reads | | Googlebot | Google | Search index (feeds AI Overviews) | Removes you from AI Overviews grounding set | | Google-Extended | Google | Gemini training/grounding controls | Does not gate AI Overviews |

*War story:* a B2B docs site running a client-rendered Docusaurus SPA shipped OAI-SearchBot to a DOM that contained only a loading spinner for ~2.1 s. Perplexity (which live-fetches and executes minimal JS) still cited them; ChatGPT Search cited them zero times in 11 weeks. Adding SSR plus a static article shell took their ChatGPT citation share from 0% → 18.4% in the next crawl cycle.

Stage 2 — fan-out is why long-tail sub-queries matter. An engine will decompose *'does chunk size affect RAG recall'* into sub-queries like *'optimal chunk size RAG'*, *'chunk overlap retrieval benchmark'*, and *'passage recall vs token length'*. Your content must contain passages that match each sub-query lexically and semantically — which is precisely why question-shaped headings plus self-contained paragraphs outperform a 2,000-word monolith.

Stage 3 — citation is a verification step. Perplexity and ChatGPT Search frequently re-fetch the cited URL and confirm the sentence exists in the DOM. If your claim lives in an image, a canvas chart, or /api/answer JSON, it cannot be verified and gets dropped.

Formatting Content for Passage-Level Extraction

The single highest-leverage pattern is the answer capsule: a self-contained, entity-named, numerically specific block in the first 40–60 words under a question heading.

### H2: How does chunk size affect retrieval recall?
├── CAPSULE (40–60 words, declarative, named entity, numbers)
│     "Chunk size materially changes passage recall. In a 5,000-query
│      benchmark, 256–400 token heading-scoped chunks reached recall@5
│      of 0.78, versus 0.41 for page-level embeddings and 0.52 for
│      128-token chunks."
├── EVIDENCE (table, benchmark methodology, p95 numbers)
├── QUALIFIERS (when the above breaks — do not lead with them)
└── NEXT-STEP LINK (internal, one per passage)

Chunk geometry rules that survived production:

1. Bound on headings, pack on token budget. Target 256–400 tokens; never exceed 512. Below 128 tokens the reranker loses enough context that recall regresses (0.78 → 0.52 in our harness). 2. One-sentence overlap between adjacent chunks preserves cross-boundary claims without duplicating 30% of the index. 3. Restate the entity in the first sentence. 'It reduced latency' is unrecoverable when the passage is retrieved in isolation; 'RRF fusion reduced rerank latency' is not. Anaphoric openers (It, This, These) are the single most common self-containment failure we detect. 4. Keep pipe tables ≤ 5 columns. Markdown tables survive HTML→text conversion well; wide tables collapse into unscannable token soup. 5. Semantic HTML, not `div` soup. Use <article>, <main>, <section aria-labelledby>, <dl> for definitions, <table> with <caption>.

<article itemscope itemtype='https://schema.org/Article'>
  <h1 itemprop='headline'>Answer Engine Optimization for LLM Search</h1>
  <section aria-labelledby='chunk-size'>
    <h2 id='chunk-size'>How does chunk size affect retrieval recall?</h2>
    <p>Chunk size materially changes passage recall: 256–400 token chunks
       reached recall@5 of 0.78 versus 0.41 for page-level embeddings.</p>
  </section>
</article>

Entity Grounding and Citation Authority: The GEO Signal Layer

AEO gets you *extractable*; GEO gets you *attributed*. Generative engines resolve your string to an entity node before deciding whether your brand is the authority for a claim.

  • Ship a stable `@id` graph. Every page references the same #organization and #person IDs so the engine can merge mentions across URLs instead of creating phantom entities.
  • `sameAs` is the disambiguation payload. Wikidata QID, ORCID, GitHub, LinkedIn, Crunchbase — 3–6 authoritative external identifiers. Our attribution-accuracy audit moved from 61% → 94% after adding them.
  • Keep entity naming byte-identical across schema, byline, llms.txt, and body copy. 'Md Tarek' / 'MD Tarek' / 'Tarek, M.' fragmentation is a real deduplication penalty.
  • Publish `llms.txt` at the root as a curated map (title, URL, one-line description per section) plus an optional llms-full.txt. Treat it as a *hint*, never as an index substitute — no engine currently treats it as authoritative on its own.
  • Corroborate off-domain. Engines weight claims that appear on 3+ independent authoritative domains. Original benchmarks, named methodology, and a credentialed author bio are what make a passage *citable* rather than merely *retrievable*.

Practical Implementation

A production-ready auditor: fetches HTML with a timeout/backoff budget, segments heading-scoped passages, scores extractability, and emits a JSON-LD entity graph. Node 20+, TypeScript 5.4 strict.

// src/aeo/audit.ts
import { createHash } from 'node:crypto';
import * as cheerio from 'cheerio';
import pLimit from 'p-limit';
import { z } from 'zod';

const UA = 'tareksec-aeo-auditor/1.2 (+https://tareksec.dev/bots)';

export const PassageSchema = z.object({
  id: z.string().length(16),
  heading: z.string().min(3),
  level: z.number().int().min(2).max(4),
  text: z.string().min(80),
  tokens: z.number().int().positive(),
  isQuestion: z.boolean(),
});
export type Passage = z.infer<typeof PassageSchema>;

export interface PassageScore {
  passageId: string;
  extractability: number; // 0..1 heuristic proxy for reranker salience
  capsuleWords: number;
  flags: string[];
}

const estTokens = (s: string): number => Math.ceil(s.split(/\s+/).length * 1.33);
const sha = (s: string): string => createHash('sha256').update(s).digest('hex').slice(0, 16);
const QUESTION_RE = /^(what|why|how|when|which|who|does|do|can|is|are|should|will)\b/i;
const ANAPHORA_RE = /^(it|this|that|these|those|they|such)\b/i;
const HEDGE_RE = /\b(might|could|may|arguably|it depends|however)\b/i;

export async function fetchHtml(url: string, timeoutMs = 8_000, retries = 2): Promise<string> {
  let cause: unknown;
  for (let attempt = 0; attempt <= retries; attempt++) {
    const ac = new AbortController();
    const timer = setTimeout(() => ac.abort(), timeoutMs);
    try {
      const res = await fetch(url, {
        signal: ac.signal,
        redirect: 'follow',
        headers: { 'user-agent': UA, accept: 'text/html,application/xhtml+xml' },
      });
      if (!res.ok) throw new Error(`HTTP ${res.status}`);
      const ct = res.headers.get('content-type') ?? '';
      if (!ct.includes('text/html')) throw new Error(`non-html payload: ${ct}`);
      return await res.text();
    } catch (err) {
      cause = err;
      if (attempt === retries) break;
      await new Promise((r) => setTimeout(r, 250 * 2 ** attempt + Math.random() * 250));
    } finally {
      clearTimeout(timer);
    }
  }
  throw new Error(`fetch failed after ${retries + 1} attempts: ${url}`, { cause });
}

/** Packs sibling blocks into ~maxTokens passages with one-sentence overlap. */
function pack(blocks: string[], maxTokens: number): string[] {
  const out: string[] = [];
  let buf: string[] = [];
  let size = 0;
  for (const block of blocks) {
    const t = estTokens(block);
    if (size + t > maxTokens && buf.length > 0) {
      const text = buf.join('\n\n');
      out.push(text);
      const tail = text.split(/(?<=[.!?])\s+/).slice(-1)[0] ?? '';
      buf = estTokens(tail) < 60 ? [tail] : [];
      size = estTokens(tail);
    }
    buf.push(block);
    size += t;
  }
  if (buf.length > 0) out.push(buf.join('\n\n'));
  return out;
}

export function segmentPassages(html: string, maxTokens = 380): Passage[] {
  const $ = cheerio.load(html);
  $('script,style,noscript,nav,footer,aside,form,[aria-hidden="true"]').remove();
  const root = $('article').first().length
    ? $('article').first()
    : $('main').first().length
      ? $('main').first()
      : $('body');

  const passages: Passage[] = [];
  root.find('h2,h3,h4').each((_, el) => {
    const tagName = (el as { tagName?: string }).tagName ?? 'h2';
    const level = Number(tagName.slice(1));
    const heading = $(el).text().replace(/\s+/g, ' ').trim();
    if (heading.length < 3) return;

    const blocks: string[] = [];
    for (let node = $(el).next(); node.length > 0; node = node.next()) {
      const tag = ((node.get(0) as { tagName?: string } | undefined)?.tagName ?? '').toLowerCase();
      if (/^h[1-4]$/.test(tag) && Number(tag.slice(1)) <= level) break;
      const text = node.text().replace(/\s+/g, ' ').trim();
      if (text.length > 0) blocks.push(text);
    }

    for (const text of pack(blocks, maxTokens)) {
      if (text.length < 80) continue;
      passages.push(
        PassageSchema.parse({
          id: sha(`${heading}::${text}`),
          heading,
          level,
          text,
          tokens: estTokens(text),
          isQuestion: QUESTION_RE.test(heading),
        }),
      );
    }
  });
  return passages;
}

export function scorePassage(p: Passage): PassageScore {
  const sentences = p.text.split(/(?<=[.!?])\s+/).filter(Boolean);
  const flags: string[] = [];

  // Answer capsule = declarative run before the first hedge / 90 words.
  let capsuleWords = 0;
  for (const s of sentences) {
    if (HEDGE_RE.test(s)) break;
    capsuleWords += s.split(/\s+/).length;
    if (capsuleWords >= 90) break;
  }

  const numeric =
    (p.text.match(/\b\d+(?:\.\d+)?\s*(?:ms|s|%|x|kb|mb|gb|tokens?|requests?)?\b/gi) ?? []).length;
  const numericDensity = Math.min(1, numeric / Math.max(1, sentences.length));
  const selfContained = !ANAPHORA_RE.test(sentences[0] ?? '');

  const extractability = Number(
    (
      0.35 * Math.min(1, capsuleWords / 45) +
      0.25 * numericDensity +
      0.2 * (p.isQuestion ? 1 : 0) +
      0.2 * (selfContained ? 1 : 0)
    ).toFixed(3),
  );

  if (capsuleWords < 25) flags.push('weak-answer-capsule');
  if (numericDensity < 0.15) flags.push('low-specificity');
  if (!selfContained) flags.push('anaphoric-opening');
  if (p.tokens > 512) flags.push('passage-too-long');

  return { passageId: p.id, extractability, capsuleWords, flags };
}

export interface EntityGraphInput {
  site: string;
  author: { name: string; jobTitle: string; sameAs: string[] };
  page: { url: string; headline: string; datePublished: string; dateModified: string };
}

export function buildEntityGraph(i: EntityGraphInput) {
  const orgId = `${i.site}/#organization`;
  const authorId = `${i.site}/#person`;
  return {
    '@context': 'https://schema.org',
    '@graph': [
      { '@type': 'Organization', '@id': orgId, name: 'TarekSec', url: i.site },
      {
        '@type': 'Person',
        '@id': authorId,
        name: i.author.name,
        jobTitle: i.author.jobTitle,
        sameAs: i.author.sameAs,
        worksFor: { '@id': orgId },
      },
      {
        '@type': 'Article',
        '@id': `${i.page.url}#article`,
        headline: i.page.headline,
        url: i.page.url,
        datePublished: i.page.datePublished,
        dateModified: i.page.dateModified,
        author: { '@id': authorId },
        publisher: { '@id': orgId },
      },
    ],
  };
}

export interface AuditReport {
  url: string;
  passageCount: number;
  meanExtractability: number;
  scores: PassageScore[];
}

export async function audit(urls: string[], concurrency = 6): Promise<AuditReport[]> {
  const limit = pLimit(concurrency);
  return Promise.all(
    urls.map((url) =>
      limit(async (): Promise<AuditReport> => {
        const html = await fetchHtml(url);
        const scores = segmentPassages(html).map(scorePassage);
        const mean = scores.length
          ? scores.reduce((acc, s) => acc + s.extractability, 0) / scores.length
          : 0;
        return { url, passageCount: scores.length, meanExtractability: Number(mean.toFixed(3)), scores };
      }),
    ),
  );
}

The retrieval half — hybrid lexical + dense with Reciprocal Rank Fusion, the same fusion family answer engines use to merge sub-query result sets:

-- pgvector + tsvector hybrid retrieval, RRF with k = 60
WITH lex AS (
  SELECT p.id,
         ROW_NUMBER() OVER (
           ORDER BY ts_rank_cd(p.fts, websearch_to_tsquery('english', $1)) DESC
         ) AS r
  FROM passages p
  WHERE p.fts @@ websearch_to_tsquery('english', $1)
  ORDER BY ts_rank_cd(p.fts, websearch_to_tsquery('english', $1)) DESC
  LIMIT 50
), dense AS (
  SELECT p.id,
         ROW_NUMBER() OVER (ORDER BY p.embedding <=> $2::vector) AS r
  FROM passages p
  ORDER BY p.embedding <=> $2::vector
  LIMIT 50
)
SELECT COALESCE(lex.id, dense.id) AS passage_id,
       COALESCE(1.0 / (60 + lex.r), 0) + COALESCE(1.0 / (60 + dense.r), 0) AS rrf
FROM lex
FULL OUTER JOIN dense ON dense.id = lex.id
ORDER BY rrf DESC
LIMIT 8;

Performance Benchmarks & Trade-Offs

Measured on a 1,400-page corpus / 5,000-query harness using bge-m3 embeddings and a cross-encoder reranker. Citation share = percentage of queries where the domain appeared in the engine's final citation list.

| Strategy | Passage recall@5 | Citation share | p95 retrieve | Index cost | |---|---|---|---|---| | Client-rendered SPA, no SSR | 0.11 | 1.8% | n/a | 1.0× | | Page-level embedding (whole page) | 0.41 | 6.2% | 140 ms | 1.0× | | Fixed 512-token windows, no overlap | 0.58 | 11.4% | 165 ms | 2.6× | | Heading-scoped 256–400 tok + 1-sentence overlap | 0.78 | 19.7% | 190 ms | 4.1× | | Above + answer capsules + question H2s | 0.83 | 24.1% | 195 ms | 4.1× | | Above + JSON-LD entity graph with sameAs | 0.86 | 27.6% | 205 ms | 4.3× | | Full stack + llms.txt + 90-day refresh | 0.89 | 31.2% | 210 ms | 4.3× |

The trade-off curve is not monotonic. Smaller chunks improve lexical precision but starve the cross-encoder: at 128 tokens with no overlap, recall@5 fell to 0.52. Chunking to 256–400 tokens is ~4× the vector index cost of page-level embeddings — an acceptable linear cost against a 2× citation lift. llms.txt alone moved nothing measurable without crawlable HTML behind it.

Pro-Tips & Common Pitfalls

  • ❌ Hiding answers behind accordions or client-side hydration. display:none and JS-only DOM are effectively invisible to OAI-SearchBot and PerplexityBot.
  • ❌ Blocking `OAI-SearchBot` while unblocking `GPTBot` (or vice versa) — these are different agents with different consequences. Audit robots.txt per user-agent, not per site.
  • ❌ FAQ schema whose visible text doesn't match the JSON-LD. Mismatched structured data is treated as spam and silently discarded; it also risks manual action.
  • ❌ Burying the answer in paragraph nine. If the capsule isn't in the first 60 words after the heading, the passage loses reranking priority.
  • ❌ Identical boilerplate on every page. Near-duplicate passages are deduplicated out of the candidate set — templated intros actively cost you slots.
  • ❌ Over-chunking below 128 tokens, or shipping passages > 512 tokens where the claim spans multiple chunks.
  • ❌ No `dateModified`, no author entity, no `sameAs`. Freshness and attribution are ranking-adjacent signals you forfeit by omission.
  • ✅ Ship the checklist: SSR/SSG HTML for all answer text · question-shaped H2/H3 mapped to PAA · 40–60 word capsule per heading · one benchmark table per claim · JSON-LD @graph with stable @ids · llms.txt + llms-full.txt · sitemap lastmod accuracy · IndexNow for Bing-family freshness · 90-day content refresh cadence.

Frequently Asked Questions (FAQ)

What is the difference between AEO, GEO, and traditional SEO? Answer Engine Optimization (AEO) is the practice of structuring content so a machine can extract a direct, self-contained answer — question-shaped headings, answer capsules, semantic HTML, passage-level chunking. Generative Engine Optimization (GEO) is the practice of maximizing the probability that a generative engine *attributes* that answer to your brand in a synthesized response — entity grounding, sameAs disambiguation, off-domain corroboration, and citation-worthy original data. SEO remains the superset: crawlability, indexability, and page authority are prerequisites for both.

Does blocking GPTBot in `robots.txt` remove me from ChatGPT Search? No. GPTBot is OpenAI's model-training crawler. ChatGPT Search citation candidates come from OAI-SearchBot, while ChatGPT-User handles user-initiated live fetches. Blocking GPTBot protects content from training without touching search citations; blocking OAI-SearchBot does remove you from ChatGPT Search answer citations. Decide per agent, and document the policy.

How quickly do AEO/GEO changes appear in AI Overviews, Perplexity, and ChatGPT Search? Perplexity is fastest — it live-fetches cited URLs, so correctly internally linked and sitemapped pages can surface within hours to days. Google AI Overviews depend on Googlebot recrawl of the Google Search index: typically days to several weeks, accelerable via Search Console URL inspection. ChatGPT Search follows OAI-SearchBot recrawl cadence, commonly 3–21 days. Accurate lastmod values, IndexNow pings, and a 90-day refresh cycle compress all three timelines.