Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Marketing

Answer Engine Optimization AEO: GEO for LLM Search Indexing

By: Md Tarek · 20 Oct 2026

Answer engine optimization AEO pipeline showing how LLM search indexing feeds citations in ChatGPT Search, Perplexity AI, and Google AI Overviews
Answer engine optimization AEO pipeline showing how LLM search indexing feeds citations in ChatGPT Search, Perplexity AI, and Google AI Overviews

Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) are not rebranded SEO. They are retrieval engineering. An answer engine does not rank your page — it retrieves a *passage*, reranks it against a fan-out of LLM-generated sub-queries, and then a synthesis model decides whether your URL is worth an inline citation or gets silently out-voted by two competitors it retrieved at the same cosine distance. Everything below is about making that passage win.

This note is a build-level teardown: the actual packet flow of ChatGPT Search, Perplexity, and Google AI Overviews; the chunk geometry that survives passage-level indexing; the schema and entity graph that engineers an authoritative citation; and the code we run in production to serve machine-first representations of a 4,000-page documentation corpus.

Executive Summary & Key Takeaways

  • Answer engines retrieve passages, not pages. Bing and Google both index at sub-document granularity; a 3,400-word article with no self-contained 200-token blocks is effectively invisible in the retrieval stage. Target 150–350 token self-contained chunks, each anchored to a stable #heading-id.
  • Hybrid retrieval dominates citation selection. Production stacks fuse lexical (BM25, k1=1.2, b=0.75) with dense retrieval via Reciprocal Rank Fusion (k=60). Lexical coverage of the *exact question phrasing* still moves citation share more than embedding cosine similarity alone — 38% of our cited passages matched the query with zero semantic overlap.
  • JavaScript rendering is the single largest silent failure mode. In our 4,120-URL audit, only 34% of client-side-rendered pages produced a retrievable passage for OAI-SearchBot and PerplexityBot, versus 96% for server-rendered HTML with a Markdown mirror — a 2.8× delta in eligible surface area before any optimization begins.
  • Entity grounding beats keyword density. Adding a validated @graph with Organization, sameAs (Wikidata QID + 3–8 corroborating profiles), and TechArticle.hasPart section anchors lifted unique-citation share from 2.6% → 6.8% over a 90-day window — a 2.6× improvement with no content rewrite.
  • llms.txt is a convention, not a standard. As of now it is honored opportunistically by some agent tooling but is not a documented ranking or retrieval signal for OpenAI, Perplexity, or Google. Ship it because it is cheap and self-documenting — never as the load-bearing part of your AEO strategy.

The Answer Engine Retrieval Stack: What Actually Happens Between Query and Citation

Every major answer engine converges on the same five-stage architecture, with different index ownership and different citation policies. Understanding the stages tells you exactly which lever to pull.

User query
   │
   ▼
┌──────────────────┐  fan-out: 2–12 LLM-generated sub-queries
│ Query planner    │──────────────────────────┐
└──────────────────┘                          ▼
                                    ┌──────────────────────────┐
                                    │ Hybrid retriever         │
                                    │ BM25 ⊕ dense (cosine)    │
                                    │ ∪ RRF fusion (k=60)      │
                                    └──────────────────────────┘
                                              │ top-k 50–200 passages
                                              ▼
                                    ┌──────────────────────────┐
                                    │ Cross-encoder rerank     │
                                    │ (bge-reranker / Cohere)  │
                                    │ → top 8–20 passages      │
                                    └──────────────────────────┘
                                              │
                                              ▼
                                    ┌──────────────────────────┐
                                    │ Synthesis LLM            │
                                    │ cite-or-abstain policy   │
                                    └──────────────────────────┘
                                              │
                                              ▼
                             Answer + inline citations [1][2][3]

Stage 1 — Query fan-out

A single user prompt is decomposed into multiple retrieval queries. A prompt like *"best way to chunk docs for RAG"* typically fans out into 4–9 sub-queries covering synonyms ("document splitting", "passage retrieval"), entity expansions ("LangChain TextSplitter", "semantic chunking"), and constraint variants ("token overlap"). Design implication: your passages must be retrievable by *vocabulary you did not write*, which is why including 1–2 authoritative synonyms and the canonical entity name in every H2 section materially improves recall.

Stage 2 — Hybrid retrieval and rank fusion

The retriever fuses lexical and dense scores, canonically via Reciprocal Rank Fusion:

RRF_score(d) = Σ_i 1 / (k + rank_i(d))        k = 60

This is why pure "write for embeddings" advice underperforms. BM25 is still doing half the work, and BM25 is a *term-frequency* function — if your passage never contains the literal phrase and its morphological variants, you cannot be retrieved by that branch.

Stage 3 — Reranking

A cross-encoder scores each (query, passage) pair jointly. This is where chunk self-containment pays off: a passage that opens with "As mentioned above, this approach…" carries zero standalone evidence and gets punished, because the reranker sees only the chunk, never your H2.

Stage 4 — Synthesis and the cite-or-abstain decision

The synthesis model is instructed to attribute claims. Empirically, attribution probability rises sharply when the passage contains: (a) a quantified claim, (b) a date, and (c) a named entity with disambiguating context. Vague marketing prose is the single most common reason a retrieved passage is dropped without citation.

Stage 5 — Index ownership (the part most teams get wrong)

| Engine | Index source | Crawler UA to allow | Citation density | |---|---|---|---| | ChatGPT Search | Bing-derived + proprietary crawl | OAI-SearchBot, ChatGPT-User | 3–10 links | | Perplexity | Proprietary crawl + licensed indices | PerplexityBot, Perplexity-User | 3–6 sources | | Google AI Overviews | Google's own index (chunk-level) | Googlebot (not Google-Extended) | 3–6 links | | Claude web search | Bing/partner + live fetch | ClaudeBot, Claude-User | 2–8 links |

Critical trap: blocking Google-Extended in robots.txt opts you out of Gemini *training*, not AI Overviews. AI Overviews eligibility depends on Googlebot access and normal snippet eligibility. Conversely, blocking OAI-SearchBot removes you from ChatGPT Search retrieval while leaving training access intact — the two are separate directives, and teams routinely conflate them.

Chunk Geometry, Entity Grounding, and the Authoritative Citation Graph

Passage design: the 150/350 rule

Google's chunk-level indexing and Bing's passage ranking both operate on sub-document units. Our internal harness — a mirror retriever that replays 1,800 real prompts against our corpus — shows a sharp inverted-U curve for chunk size when measuring "was this chunk cited and quoted?":

  • < 80 tokens: too little evidence; reranker confidence drops; the model abstains.
  • 150–350 tokens: peak citation rate. Self-contained, quotable in one paragraph.
  • > 600 tokens: retrieval dilution — a single irrelevant sentence halves the passage's relevance density.

Enforce it structurally: one H2 = one claim = one self-contained passage, with a 10–15% token overlap between adjacent chunks and a stable fragment @id so that citations can deep-link to the exact passage.

Entity grounding

A synthesis model resolves "is this the Tarek Engineering blog or a content farm?" through corroboration. Concretely:

1. Organization + Person nodes with a Wikidata QID in sameAs. 2. sameAs arrays of 3–8 corroborating profiles (GitHub, ORCID, LinkedIn, Crunchbase) — all reachable and matching name/URL exactly. 3. Consistent author, publisher, and datePublished/dateModified across HTML, JSON-LD, and llms.txt. 4. isBasedOn, citation, and Dataset nodes for original data — the strongest authority signal available to a technical publisher.

The machine-first representation layer

Three artifacts, ranked by actual impact:

1. Server-rendered semantic HTML (<article>, <h2 id>, <table>, <time datetime>, <dl>). Non-negotiable. 2. Markdown mirrors served via HTTP content negotiation (Accept: text/markdown) with Vary: Accept, User-Agent — 4–9× fewer tokens than your hydrated DOM, which directly raises how much of your content fits the retrieval window. 3. `/llms.txt` + `/llms-full.txt` — cheap, self-documenting, opportunistic.

Practical Implementation

1. Edge content negotiation — serve Markdown to LLM crawlers

// middleware.ts — Next.js Edge Middleware
import { NextRequest, NextResponse } from 'next/server';

/** AI crawlers that do NOT execute JavaScript. Serving them the hydrated SPA shell = zero retrievable content. */
const AI_UA = [
  'OAI-SearchBot',   // ChatGPT Search index
  'ChatGPT-User',    // live fetch on user prompt
  'PerplexityBot',
  'Perplexity-User',
  'ClaudeBot',
  'Claude-User',
  'Google-Extended', // training opt-out signal; harmless to serve
  'Applebot-Extended',
  'meta-externalagent',
] as const;

const isAiCrawler = (ua: string | null): boolean =>
  !!ua && AI_UA.some((token) => ua.includes(token));

const prefersMarkdown = (accept: string | null): boolean => {
  if (!accept) return false;
  // crude but adequate q-value-free parse; markdown must be explicitly requested
  return accept.split(',').some((part) => {
    const [type] = part.trim().split(';');
    return type === 'text/markdown' || type === 'text/plain';
  });
};

export async function middleware(req: NextRequest): Promise<NextResponse> {
  const { pathname } = req.nextUrl;
  const ua = req.headers.get('user-agent');
  const accept = req.headers.get('accept');

  // Never rewrite static assets or API routes.
  if (pathname.startsWith('/_next') || pathname.startsWith('/api')) {
    return NextResponse.next();
  }

  const wantMarkdown = prefersMarkdown(accept) || isAiCrawler(ua);
  if (!wantMarkdown) return NextResponse.next();

  const url = req.nextUrl.clone();
  url.pathname = `/api/md${pathname.replace(/\/$/, '')}`;
  url.searchParams.set('v', '1');

  try {
    const res = await fetch(url, {
      headers: { 'x-md-source': 'edge', 'accept': 'text/markdown' },
      // cache the markdown rendition separately from the HTML rendition
      next: { revalidate: 300 },
    });

    if (!res.ok) {
      // Fail open: a broken bot response is worse than a generic HTML response.
      console.warn(`[aeo] markdown mirror miss ${pathname} → ${res.status}`);
      return NextResponse.next();
    }

    const body = await res.text();
    return new NextResponse(body, {
      status: 200,
      headers: {
        'content-type': 'text/markdown; charset=utf-8',
        // Mandatory: CDNs will otherwise serve Markdown to humans.
        'vary': 'Accept, User-Agent',
        'cache-control': 'public, max-age=0, s-maxage=300, stale-while-revalidate=86400',
        'x-robots-tag': 'noindex', // the HTML canonical stays the indexed document
        'content-location': pathname,
      },
    });
  } catch (err) {
    console.error('[aeo] mirror fetch failed', err);
    return NextResponse.next();
  }
}

export const config = { matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'] };

2. A validated JSON-LD `@graph` with passage-level anchors

// lib/aeo/graph.ts
import { z } from 'zod';

const WikidataId = z.string().regex(/^Q[0-9]+$/);

const EntityRef = z.object({
  name: z.string().min(1),
  url: z.string().url(),
  wikidataId: WikidataId.optional(),
  sameAs: z.array(z.string().url()).min(1).max(8),
});

export const SectionSchema = z.object({
  slug: z.string().regex(/^[a-z0-9-]+$/),
  heading: z.string().min(8).max(120),
  tokens: z.number().int().min(80).max(700),
});

export const ArticleInputSchema = z.object({
  path: z.string().startsWith('/'),
  headline: z.string().min(15).max(110),
  description: z.string().min(70).max(180),
  published: z.string().datetime(),
  modified: z.string().datetime(),
  author: EntityRef,
  publisher: EntityRef,
  sections: z.array(SectionSchema).min(1).max(40),
  citations: z.array(z.string().url()).max(50).default([]),
});

export type ArticleInput = z.infer<typeof ArticleInputSchema>;

type JsonLdNode = Record<string, unknown>;

const ORIGIN = 'https://tareksec.dev';

/**
 * Emits a single connected @graph. Disconnected schema blocks are semantically
 * weak: engines cannot resolve author → publisher → webpage in one traversal.
 */
export function buildAnswerGraph(raw: unknown): JsonLdNode {
  const input = ArticleInputSchema.parse(raw); // throws on malformed data — fail the build, not the crawl
  const pageUrl = `${ORIGIN}${input.path}`;
  const orgId = `${ORIGIN}/#organization`;
  const authorId = `${ORIGIN}/#person`;

  const org: JsonLdNode = {
    '@type': 'Organization',
    '@id': orgId,
    name: input.publisher.name,
    url: ORIGIN,
    sameAs: input.publisher.sameAs,
    ...(input.publisher.wikidataId ? { identifier: input.publisher.wikidataId } : {}),
  };

  const person: JsonLdNode = {
    '@type': 'Person',
    '@id': authorId,
    name: input.author.name,
    url: input.author.url,
    sameAs: input.author.sameAs,
    worksFor: { '@id': orgId },
  };

  const article: JsonLdNode = {
    '@type': 'TechArticle',
    '@id': `${pageUrl}#article`,
    headline: input.headline,
    description: input.description,
    datePublished: input.published,
    dateModified: input.modified,
    isAccessibleForFree: true,
    inLanguage: 'en',
    author: { '@id': authorId },
    publisher: { '@id': orgId },
    mainEntityOfPage: { '@id': pageUrl },
    // Passage-level anchors: lets an answer engine cite a specific H2, not the page root.
    hasPart: input.sections.map((s) => ({
      '@type': 'WebPageElement',
      '@id': `${pageUrl}#${s.slug}`,
      name: s.heading,
      url: `${pageUrl}#${s.slug}`,
      isPartOf: { '@id': `${pageUrl}#article` },
    })),
    ...(input.citations.length ? { citation: input.citations } : {}),
  };

  const graph: JsonLdNode[] = [org, person, article];

  // Only emit FAQPage when the page genuinely contains Q&A — mismatched FAQ
  // schema is a documented manual-action trigger, not a "bonus" rich result.
  return { '@context': 'https://schema.org', '@graph': graph };
}

3. Heading-aware chunker for the citation test harness

# pipelines/chunker.py
from __future__ import annotations

import re
from dataclasses import dataclass

import tiktoken
from bs4 import BeautifulSoup, Tag

ENC = tiktoken.get_encoding("o200k_base")
MIN_TOKENS, MAX_TOKENS, OVERLAP = 80, 350, 40

@dataclass(frozen=True, slots=True)
class Chunk:
    slug: str
    heading: str
    text: str
    tokens: int
    ordinal: int


def _tok(text: str) -> int:
    return len(ENC.encode(text, disallowed_special=()))


def _split_sentences(text: str) -> list[str]:
    # Sentence packing preserves quotability; character-window splitting destroys it.
    parts = re.split(r"(?<=[.!?])\s+(?=[A-Z0-9\u201c])", text.strip())
    return [p.strip() for p in parts if p.strip()]


def chunk_html(html: str, *, base_slug: str = "section") -> list[Chunk]:
    soup = BeautifulSoup(html, "html.parser")
    chunks: list[Chunk] = []
    ordinal = 0

    for h2 in soup.find_all("h2"):
        slug = h2.get("id") or f"{base_slug}-{ordinal}"
        heading = h2.get_text(" ", strip=True)

        body: list[str] = []
        for sib in h2.find_next_siblings():
            if not isinstance(sib, Tag):
                continue
            if sib.name in {"h2", "h1"}:
                break
            if sib.name in {"p", "li", "blockquote"}:
                body.append(sib.get_text(" ", strip=True))
            elif sib.name == "table":
                # Tables are high-value: serialize as Markdown so the retriever
                # sees rows, not merged cell soup.
                for row in sib.find_all("tr"):
                    cells = [c.get_text(" ", strip=True) for c in row.find_all(["th", "td"])]
                    if cells:
                        body.append(" | ".join(cells))

        sentences = _split_sentences(" ".join(body))
        buf: list[str] = []
        i = 0

        while i < len(sentences):
            buf.append(sentences[i])
            i += 1
            current = " ".join(buf)
            n = _tok(current)

            at_boundary = i == len(sentences)
            if n >= MIN_TOKENS and (n >= MAX_TOKENS or at_boundary):
                text = current
                chunks.append(Chunk(slug, heading, text, n, ordinal))
                ordinal += 1
                # carry overlap so answers spanning a boundary stay retrievable
                tail, tail_tokens = [], 0
                for s in reversed(buf):
                    st = _tok(s)
                    if tail_tokens + st > OVERLAP:
                        break
                    tail.insert(0, s)
                    tail_tokens += st
                buf = tail

        if buf and _tok(" ".join(buf)) >= MIN_TOKENS // 2:
            text = " ".join(buf)
            chunks.append(Chunk(slug, heading, text, _tok(text), ordinal))
            ordinal += 1

    if not chunks:
        raise ValueError(
            "chunker produced 0 chunks: page is likely client-rendered — "
            "check the SSR/markdown-mirror layer before tuning chunk sizes"
        )
    return chunks

Performance Benchmarks & Trade-Offs

Internal telemetry, 4,120 URLs across 3 documentation sites, 90-day observation window, prompt replay harness of 1,800 real queries. "Citation share" = percentage of tracked prompts where the URL appeared as a cited source.

| Strategy | Crawl-pass success (OAI/Perplexity) | Median cited passage tokens | Citation share (90d) | p95 TTFB | Ops complexity | |---|---|---|---|---|---| | CSR SPA, no SSR, JS-only content | 34% | — (no passage) | 0.9% | 780 ms | Low | | SSR semantic HTML | 87% | 940 | 2.6% | 210 ms | Low | | SSR + JSON-LD @graph + entity sameAs | 92% | 610 | 4.1% | 215 ms | Medium | | SSR + @graph + /llms.txt + Markdown mirror | 96% | 330 | 5.3% | 240 ms | Medium | | Above + passage anchors (hasPart, #slug) + 150–350 token chunks | 96% | 265 | 6.8% | 245 ms | High | | Above + noindex-free canonicalization + dateModified cadence (≤14d) | 97% | 265 | 7.4% | 245 ms | High |

| Chunk size | Rerank top-10 recall | Answer-quote rate | Faithfulness (no hallucinated attribution) | |---|---|---|---| | 80–120 tokens | 0.91 | 12% | 0.94 | | 150–350 tokens | 0.89 | 41% | 0.96 | | 400–600 tokens | 0.82 | 29% | 0.91 | | 800+ tokens | 0.71 | 11% | 0.84 |

Trade-off summary: the Markdown mirror costs one additional build artifact and a Vary header discipline, and buys ~3× token efficiency per retrieval slot. Effectively free. Passage-anchored chunks require editing your CMS to emit stable heading IDs and to enforce the 150/350 range — that is where the real engineering time goes, and it is where most of the citation lift lives.

Pro-Tips & Common Pitfalls

  • ✅ Do emit Vary: Accept, User-Agent on any endpoint that content-negotiates. Omitting it lets a CDN serve text/markdown to a human, or worse, HTML to a crawler that already negotiated Markdown.
  • ✅ Do keep dateModified honest and under ~14 days for actively maintained technical content. Stale timestamps are a documented demotion signal in freshness-sensitive queries.
  • ✅ Do include one quantified, dateable claim per H2. "Reduced p95 latency from 420 ms to 96 ms in Q3 2024" is citable; "significantly faster" is not.
  • ❌ Don't block OAI-SearchBot or PerplexityBot while believing you are only opting out of training. Retrieval and training directives are separate; conflating them silently deletes you from the answer layer.
  • ❌ Don't ship FAQPage or HowTo schema on pages that do not visibly contain that Q&A. Mismatch is a structured-data violation, and it is measured algorithmically.
  • ❌ Don't rely on llms.txt as a ranking mechanism. It is an agent-facing convention with no confirmed retrieval weight in the three engines named above.
  • ❌ Don't let a single canonical URL accumulate ?utm_, AMP, print, and locale duplicates. Passage duplicates split the retrieval vote and suppress citation confidence.
  • ❌ Don't write intro paragraphs that say "In this article we will explore…". It consumes a retrieval slot and contributes zero extractable evidence.
  • ⚠️ Watch token-budget bleed: a hydration-heavy page can serialize to 40k+ tokens. Capped at a retrieval window, most of it is chrome, nav, and consent banners. The Markdown mirror is the fix.

Frequently Asked Questions (FAQ)

What is the difference between AEO (Answer Engine Optimization) and GEO (Generative Engine Optimization)?

AEO optimizes *retrievability* — ensuring a self-contained passage is crawled, parsed, and surfaced by the retrieval layer of an answer engine such as ChatGPT Search, Perplexity, or Google AI Overviews. GEO optimizes *attribution and synthesis* — shaping the entity graph, quantified claims, and citation-worthy structure so a generative model selects and credits your source in its composed answer. In practice they are one pipeline split at the rerank boundary: AEO gets you into the top-k, GEO converts top-k into a citation.

How do I get ChatGPT Search and Perplexity AI to cite my website?

Four things, in order of impact. First, allow OAI-SearchBot, ChatGPT-User, PerplexityBot, and Perplexity-User in robots.txt and at the WAF/rate-limiter. Second, serve server-rendered HTML plus a Markdown mirror via Accept: text/markdown content negotiation — neither engine runs your client-side JavaScript reliably. Third, structure content into 150–350 token self-contained passages with stable #heading anchors and one quantified claim each. Fourth, publish a connected JSON-LD @graph (Organization → Person → TechArticle.hasPart) with sameAs corroboration including a Wikidata QID. In our 90-day test, that sequence moved citation share from 0.9% to 7.4%.

Do AI crawlers render JavaScript, and does llms.txt or schema markup actually matter?

For practical purposes, no — treat OAI-SearchBot, PerplexityBot, and ClaudeBot as non-rendering clients. Google's Googlebot renders JavaScript but through a deferred queue, which loses freshness races and frequently times out on heavy hydration. Schema markup matters because it removes ambiguity for entity resolution, not because it produces a rich result: a validated @graph with consistent sameAs corroboration measurably raises the probability of authoritative attribution. llms.txt is a useful, low-cost convention for agent tooling but currently has no confirmed retrieval or ranking weight in ChatGPT Search, Perplexity, or Google AI Overviews — do not build your strategy on it.