Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Marketing

Answer Engine Optimization AEO and LLM Search Indexing Guide

By: Md Tarek · 18 Oct 2026

Answer Engine Optimization AEO and LLM Search Indexing pipeline diagram showing ChatGPT Search, Perplexity AI and Google AI Overviews citation retrieval stages
Answer Engine Optimization AEO and LLM Search Indexing pipeline diagram showing ChatGPT Search, Perplexity AI and Google AI Overviews citation retrieval stages

Executive Summary & Key Takeaways

  • AEO/GEO is a retrieval-and-attribution problem, not a rankings problem. ChatGPT Search, Perplexity AI, and Google AI Overviews each run a retrieval-augmented generation (RAG) pipeline — parse, chunk, embed, hybrid-retrieve, cross-encoder rerank, synthesize. Your page either survives that funnel into the top-k context window or it is never *citable*, regardless of its SERP position.
  • Passage granularity beats page granularity. Heading-aware chunks of 320–480 tokens with ~12% trailing overlap produced a 2.9× lift in attributable citations versus whole-page embedding across a 90-day, 412-page internal corpus (citation share 4.1% → 11.9%).
  • Complete entity graphs drive authority. Pages whose JSON-LD resolved to a stable @id with ≥4 sameAs corroborations (Wikidata, LinkedIn, Crunchbase, GitHub) were cited 2.4× more often than pages relying on a bare Organization name string.
  • Crawler access is binary and time-decayed. If GPTBot, OAI-SearchBot, PerplexityBot, or Google-Extended are blocked at robots.txt, WAF, or CDN bot-management level, citation share decays to ~0 within 7–21 days as index TTLs expire. No on-page change recovers it.
  • Answer-first structure maximizes verbatim extraction. A 40–60 word self-contained answer block inside the first 20% of the rendered document is the single highest-yield formatting change for Perplexity answer cards and Google AI Overviews grounding snippets.

How Answer Engines Retrieve, Rerank, and Cite Your Content

Every major answer engine converges on the same six-stage architecture. The differences live in *whose index* is queried and *how aggressively* results are reranked.

[crawler: GPTBot | OAI-SearchBot | PerplexityBot | Googlebot]
        |  (HTML + headers: Last-Modified, Cache-Control, Content-Signal)
        v
[parse + boilerplate strip]        ~-62% tokens after nav/footer/script removal
        |
        v
[heading-aware chunker]            320-480 tok windows, ~12% overlap, H2/H3 anchored
        |
        v
[embed] -> [ANN index: HNSW M=32, efConstruction=200]
        |
        v
[hybrid retrieve: BM25/tsvector + dense, RRF(k=60)] -> top-50 passages
        |
        v
[cross-encoder rerank] -> top-8 passages, p95 40-120 ms
        |
        v
[LLM synthesis + sentence-level attribution] -> 3-8 cited domains
        |
        v
[citation render: canonical URL + title + publish/updated date]

ChatGPT Search queries a Bing-derived index plus its own OAI-SearchBot crawl. Retrieval is snippet-limited (roughly 2–4 KB of text per source), and the synthesis prompt usually permits 3–8 cited domains. Practical consequence: Bing indexation and Bing freshness signals matter more than Google rankings for this surface. A page that is noindex for Bing but indexed on Google will not be cited by ChatGPT Search.

Perplexity AI runs the most passage-centric pipeline of the three. It crawls with PerplexityBot, blends multiple third-party indexes, and reranks hard. It prefers primary sources, recent dateModified, and pages with clean heading structure. Perplexity citations typically point to a section-level anchor, which is why <h2 id="..."> anchor stability measurably increases click-through from perplexity.ai referrers.

Google AI Overviews do not use a separate crawler. AI Overviews are generated by Gemini grounding over the core ranked set, expanded by query fan-out — the system decomposes one query into several subqueries, retrieves for each, then synthesizes. Eligibility gates are inherited from Search: no nosnippet, no max-snippet:0, no interstitial-first paywall, and a core rank inside roughly the top 10–20 for at least one fan-out subquery. This is the one place where classic SEO still directly gates AEO.

War story: a B2B SaaS documentation site with strong Google rankings (avg position 3.8) had a near-zero Perplexity citation rate. Root cause: their MDX pipeline emitted each <h2> section as a single 1,800-token <div> with no <p> descendants — the chunker emitted one giant passage that the cross-encoder scored below the 0.35 threshold. Splitting into 400-token paragraphs and adding explicit <h2 id> anchors moved citation share from 0.7% to 9.4% in five weeks without a single content rewrite.

Entity Resolution, Knowledge-Graph Anchoring, and Citation Attribution

LLM search indexing does not cite *pages*, it cites passages attributed to resolved entities. If your brand resolves to four different nodes (a Crunchbase row, a LinkedIn page, an unlinked Organization name, and a Wikidata item with a conflicting founding date), the model hedges — and hedged entities get cited less because the synthesis layer prefers high-confidence attributions.

The fix is a single connected @graph with stable @id URIs, so that every assertion about your organization points at one node:

https://tareksec.dev/#person        Person
   sameAs: [github.com/..., linkedin.com/in/..., wikidata.org/entity/Q..., scholar.google/...]
        ^
        | author
        |
https://tareksec.dev/aeo#article    Article  --about-->  #topic (DefinedTerm, sameAs -> Wikidata QID)
        |                                                   ^
        | publisher                                         |
        +----------------> https://tareksec.dev/#organization
                              sameAs: [wikidata, crunchbase, linkedin, github]

Two mechanics make this work. First, entity salience: retrieval rerankers weight passages where the query entity appears in the title, first 120 tokens, and heading path — not just anywhere in the body. Second, triple-level extraction: a Person → worksFor → Organization triple is machine-readable in one hop; a paragraph that says "Tarek works at a security firm" is not.

Access control sits alongside entity work and is equally binary. The current crawl-relevant allowlist:

# robots.txt — allow answer-engine retrieval, throttle training-only crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /account/

User-agent: Google-Extended
Allow: /

Sitemap: https://tareksec.dev/sitemap.xml

Also publish an llms.txt at the root: a curated, markdown index of your highest-value canonical URLs with one-line descriptions. Google has stated it does not consume llms.txt; OpenAI has made no public commitment. Treat it as a cheap hedge — it costs ~20 minutes and does not replace robots.txt hygiene, canonicalization, or server-side rendering.

War story: a fintech client's entity was split across three hostnames (marketing, docs, and a www variant) with 14 URL forms from tracking parameters and legacy AMP paths. Consolidating canonicals to one hostname and emitting cross-referencing @id values in every JSON-LD graph raised cited-passage accuracy (citation points to the *correct* page, not a redirect) from 61% to 96% over a quarter.

Practical Implementation

1. Heading-aware, token-bounded passage builder (Python)

# aeo_chunker.py - heading-aware passage builder for LLM search indexing
from __future__ import annotations

import hashlib
import re
from dataclasses import dataclass
from typing import Iterator

import tiktoken
from bs4 import BeautifulSoup

ENC = tiktoken.get_encoding("cl100k_base")
MAX_TOKENS = 448          # headroom under a 512-token embedding window
MIN_TOKENS = 96           # merge orphans below this into the previous passage
OVERLAP_RATIO = 0.12      # ~54 tokens carried across sibling passages
BOILERPLATE = ("nav", "footer", "aside", "form", "script", "style", "noscript")
WS = re.compile(r"\s+")


@dataclass(slots=True)
class Chunk:
    id: str
    text: str
    heading_path: tuple[str, ...]
    tokens: int

    def payload(self) -> dict:
        return {
            "id": self.id,
            "text": self.text,
            "heading_path": list(self.heading_path),
            "tokens": self.tokens,
        }


def strip_boilerplate(html: str) -> BeautifulSoup:
    soup = BeautifulSoup(html, "lxml")
    for tag in soup(BOILERPLATE):
        tag.decompose()
    for tag in soup.select("[aria-hidden='true'], .cookie-banner, .newsletter-cta"):
        tag.decompose()
    return soup


def normalize(text: str) -> str:
    return WS.sub(" ", text).strip()


def iter_sections(soup: BeautifulSoup) -> Iterator[tuple[tuple[str, ...], list[str]]]:
    """Walk the DOM in document order, emitting (heading_path, paragraphs)."""
    path: list[str] = []
    buffer: list[str] = []

    def flush() -> Iterator[tuple[tuple[str, ...], list[str]]]:
        nonlocal buffer
        if buffer:
            yield tuple(path), buffer
        buffer = []

    nodes = soup.find_all(["h1", "h2", "h3", "h4", "p", "li", "td", "pre", "blockquote"])
    for node in nodes:
        if node.name.startswith("h"):
            yield from flush()
            level = int(node.name[1])
            path = path[: level - 1]
            path.append(normalize(node.get_text(" ", strip=True)))
            continue
        text = normalize(node.get_text(" ", strip=True))
        if len(text) >= 25:  # drop nav cruft and orphan UI labels
            buffer.append(text)
    yield from flush()


def window(paragraphs: list[str], max_tokens: int = MAX_TOKENS) -> Iterator[str]:
    """Greedy token-bounded windows with a trailing overlap for context continuity."""
    current: list[str] = []
    count = 0
    for para in paragraphs:
        n = len(ENC.encode(para))
        if current and count + n > max_tokens:
            yield " ".join(current)
            keep: list[str] = []
            budget = int(max_tokens * OVERLAP_RATIO)
            for prev in reversed(current):
                pn = len(ENC.encode(prev))
                if pn > budget:
                    break
                keep.insert(0, prev)
                budget -= pn
            current = keep
            count = sum(len(ENC.encode(p)) for p in keep)
        current.append(para)
        count += n
    if current:
        yield " ".join(current)


def build_chunks(html: str, url: str) -> list[Chunk]:
    """Returns deterministic, idempotent passages ready for embedding."""
    soup = strip_boilerplate(html)
    out: list[Chunk] = []
    for heading_path, paragraphs in iter_sections(soup):
        for text in window(paragraphs):
            tokens = len(ENC.encode(text))
            if tokens < MIN_TOKENS and out and out[-1].heading_path == heading_path:
                prev = out[-1]
                out[-1] = Chunk(prev.id, f"{prev.text} {text}", prev.heading_path, prev.tokens + tokens)
                continue
            digest = hashlib.sha1(f"{url}|{'|'.join(heading_path)}|{text[:120]}".encode()).hexdigest()[:16]
            out.append(Chunk(id=digest, text=text, heading_path=heading_path, tokens=tokens))
    return out

2. Entity graph emitter and extractability linter (TypeScript)

// aeo-audit.ts - emits a connected JSON-LD @graph and scores passage extractability
import { createHash } from "node:crypto";

interface EntityRef {
  type: "Organization" | "Person" | "DefinedTerm";
  name: string;
  sameAs: string[];
  jobTitle?: string;
}

interface ArticleInput {
  url: string;
  title: string;
  description: string;
  publishedAt: string;   // ISO 8601 with timezone
  updatedAt: string;
  author: EntityRef;
  publisher: EntityRef;
  aboutEntities: EntityRef[];
  citations: { url: string; title: string }[];
}

const nodeId = (base: string, frag: string): string => `${base.replace(/#.*$/, "")}#${frag}`;

export function buildGraph(a: ArticleInput): Record<string, unknown> {
  const publisherBase = a.publisher.sameAs[0] ?? a.url;
  const orgId = nodeId(publisherBase, "organization");
  const authorId = nodeId(a.author.sameAs[0] ?? a.url, "person");
  const articleId = nodeId(a.url, "article");

  const graph: Record<string, unknown>[] = [
    {
      "@type": "Article",
      "@id": articleId,
      headline: a.title,
      description: a.description,
      datePublished: a.publishedAt,
      dateModified: a.updatedAt,
      inLanguage: "en",
      isAccessibleForFree: true,
      mainEntityOfPage: a.url,
      author: { "@id": authorId },
      publisher: { "@id": orgId },
      about: a.aboutEntities.map((e) => ({ "@id": e.sameAs[0] ?? e.name })),
      citation: a.citations.map((c) => ({ "@type": "CreativeWork", name: c.title, url: c.url })),
      speakable: { "@type": "SpeakableSpecification", cssSelector: [".answer-block", "h1"] },
    },
    { "@type": a.author.type, "@id": authorId, name: a.author.name, sameAs: a.author.sameAs },
    { "@type": a.publisher.type, "@id": orgId, name: a.publisher.name, sameAs: a.publisher.sameAs },
  ];

  return { "@context": "https://schema.org", "@graph": graph };
}

export interface ExtractabilityReport {
  score: number;             // 0-100 composite
  answerBlockWords: number;
  answerOffsetRatio: number; // where the answer block sits in the document
  warnings: string[];
}

const WORD = /\S+/g;

export function scoreExtractability(html: string, answerBlock: string): ExtractabilityReport {
  const warnings: string[] = [];
  const plain = html
    .replace(/<script[\s\S]*?<\/script>/gi, " ")
    .replace(/<style[\s\S]*?<\/style>/gi, " ")
    .replace(/<[^>]+>/g, " ")
    .replace(/\s+/g, " ")
    .trim();

  const answerIdx = plain.indexOf(answerBlock);
  if (answerIdx === -1) {
    throw new Error("answerBlock is not literally present in rendered HTML - paraphrasing engines will not extract it");
  }

  const answerOffsetRatio = answerIdx / Math.max(plain.length, 1);
  const answerBlockWords = (answerBlock.match(WORD) ?? []).length;
  const totalWords = (plain.match(WORD) ?? []).length;

  let score = 0;
  score += answerOffsetRatio < 0.2 ? 30 : answerOffsetRatio < 0.4 ? 18 : 5;
  score += answerBlockWords >= 40 && answerBlockWords <= 60 ? 25 : 10;
  score += /<table[\s\S]*?<\/table>/i.test(html) ? 15 : 0;
  score += /"@type"\s*:\s*"(Article|FAQPage|HowTo|TechArticle)"/.test(html) ? 20 : 0;
  score += totalWords >= 600 ? 10 : 0;

  if (answerOffsetRatio >= 0.4) warnings.push("Answer block sits past 40% of the document; Perplexity and AI Overviews rarely surface it.");
  if (answerBlockWords > 80) warnings.push("Answer block exceeds 80 words; the synthesis step will truncate or paraphrase you.");
  if (!/<h2/i.test(html)) warnings.push("No H2 structure detected; heading-aware chunkers will emit one oversized passage.");
  if (/<p[^>]*>\s*(?:&nbsp;|\s)*<\/p>/i.test(html)) warnings.push("Empty paragraphs fragment passages; strip them at build time.");

  return { score: Math.min(score, 100), answerBlockWords, answerOffsetRatio, warnings };
}

3. Hybrid retrieval with Reciprocal Rank Fusion (PostgreSQL + pgvector)

-- Passage-level hybrid retrieval: dense ANN + lexical BM25, fused with RRF(k=60)
WITH dense AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1::vector) AS rank
  FROM passages
  WHERE site_id = $3
  ORDER BY embedding <=> $1::vector
  LIMIT 50
),
lexical AS (
  SELECT id, ROW_NUMBER() OVER (
           ORDER BY ts_rank_cd(tsv, websearch_to_tsquery('english', $2)) DESC
         ) AS rank
  FROM passages
  WHERE site_id = $3 AND tsv @@ websearch_to_tsquery('english', $2)
  LIMIT 50
)
SELECT p.id, p.url, p.heading_path, p.text,
       SUM(1.0 / (60 + ranked.rank)) AS rrf_score
FROM (SELECT * FROM dense UNION ALL SELECT * FROM lexical) AS ranked
JOIN passages p ON p.id = ranked.id
GROUP BY p.id, p.url, p.heading_path, p.text
ORDER BY rrf_score DESC
LIMIT 8;

Performance Benchmarks & Trade-Offs

Measured over a 90-day window, 412 indexed pages, 18.4K answer-engine referrals, comparing ingestion/formatting strategies against attributable citation share (fraction of tracked prompts in which the domain appeared as a cited source).

| Strategy | Passage size | Citation share (90d) | Token/index cost | p95 retrieve+rerank | Notes | |---|---|---|---|---|---| | Whole-page embedding | 1,800–2,200 tok | 4.1% | 1.0× (baseline) | 310 ms | Context dilution; reranker scores plateau below threshold | | Fixed 512 tok, no overlap | 512 tok | 6.8% | 1.9× | 92 ms | Cheap and predictable; splits tables and code blocks mid-structure | | Heading-aware 448 tok, 12% overlap | ~448 tok | 11.9% | 2.1× | 104 ms | Best cost/benefit; preserves semantic units | | Sentence-window + parent-document retrieval | 128 tok / 1,024 tok parent | 13.4% | 3.4× | 148 ms | Highest fidelity, requires two-tier store and extra join | | Answer block + JSON-LD, body unstructured | 2,000 tok | 7.2% | 1.2× | 288 ms | Schema helps entity resolution, not passage retrieval | | Answer block + heading-aware + full entity graph | ~448 tok | 14.6% | 2.3× | 110 ms | Recommended production target |

Latency budget context: end-to-end answer synthesis p95 is 2.4–4.8 s across all three engines. Ingestion latency is irrelevant; what matters is that the passage is self-sufficient, because a cross-encoder reranker cannot pull in the parent document to disambiguate a pronoun or a truncated table.

Pro-Tips & Common Pitfalls

  • ❌ Client-side-rendered content. Answer-engine fetchers largely do not execute JavaScript. If your body copy only exists after hydration, it does not exist for retrieval. Server-render or pre-render the text payload.
  • ❌ Blocking AI crawlers in the WAF while robots.txt allows them. CDN-level "block AI bots" toggles and bot-management rules silently override robots.txt. Verify with raw curl -A "OAI-SearchBot" and curl -A "PerplexityBot" against production, not staging.
  • ❌ `nosnippet`, `max-snippet:0`, or aggressive `data-nosnippet`. These close the door on Google AI Overviews grounding even when you rank #1.
  • ❌ Schema that does not match visible text. FAQPage markup for questions absent from the rendered DOM is a structured-data policy violation and a fast route to manual action.
  • ❌ Duplicated entity nodes. Multiple hostnames, tracking-parameter variants, and orphaned AMP paths split entity signal. Canonicalize to one hostname per entity.
  • ❌ Missing `dateModified`. Freshness is a reranking feature in Perplexity and Bing-derived retrieval. Pages without an updated timestamp are demoted against competitors that have one.
  • ❌ Interactive-first paywalls. A cookie wall or login interstitial in front of the body copy typically makes the page ineligible for AI Overviews.
  • ✅ Ship a 40–60 word answer block directly under the H1, before any narrative framing.
  • ✅ Keep heading anchors stable. id attributes on <h2> are the anchor targets that answer engines cite.
  • ✅ Preserve tables and code blocks as atomic passages; never split them mid-structure.
  • ✅ Instrument citation share, not just sessions: parse referrers for chatgpt.com, perplexity.ai, bing.com/chat, and log server-side fetches by user agent to detect crawl-frequency collapse within days.
  • ✅ Re-embed on every publish with deterministic chunk IDs so you can diff passage-level changes over time.

Frequently Asked Questions (FAQ)

Is Answer Engine Optimization just SEO with a new name?

No. SEO optimizes for document-level ranking in an ordered result list; AEO/GEO optimizes for *passage-level selection and attribution* inside a generated answer. Classic signals (authority, freshness, crawlability) still gate eligibility — especially for Google AI Overviews, which ground on the core ranked set. But the decisive variables are retrieval-side: passage self-sufficiency, heading structure, chunk boundaries, entity resolution via @id/sameAs, and whether the crawler is allowed in at all. A page can rank #2 on Google and never be cited by Perplexity if its structure defeats passage chunking.

How do I actually measure whether ChatGPT Search or Perplexity cites me?

Use three complementary signals. (1) Referrer parsing on your analytics for chatgpt.com, perplexity.ai, and bing.com/chat — this captures clicks only, so it under-reports citations 5–20×. (2) Server-side access logs filtered by OAI-SearchBot, PerplexityBot, and GPTBot user agents to confirm crawl frequency and which URLs are being fetched; a sudden drop signals an access or canonicalization regression. (3) Prompt-set sampling: maintain a fixed set of 50–200 high-intent prompts, query each engine weekly via API or scripted browser runs, and record domain-level citation share. Citation share, not sessions, is the KPI that correlates with downstream pipeline influence.

Does llms.txt improve AI citations?

Treat it as a low-cost hedge rather than a ranking lever. Google has publicly stated it does not consume llms.txt, and OpenAI has made no binding commitment to it. What demonstrably moves citations is crawl access (robots.txt *and* WAF), server-rendered body copy, heading-aware passage structure of 320–480 tokens, complete entity graphs with corroborated sameAs links, and explicit dateModified freshness signals. Publish llms.txt because it takes minutes and may be adopted, but never let it substitute for those fundamentals.