Answer Engine Optimization AEO and LLM Search Indexing (2025)
By: Md Tarek · 22 Oct 2026
Executive Summary & Key Takeaways
- Passages — not pages — are the atomic retrieval unit. ChatGPT Search, Perplexity, and Google AI Overviews embed and rerank 256–512 token chunks. Heading-scoped chunking at 256–400 tokens with one-sentence overlap lifted passage
recall@5from 0.41 → 0.78 in a 5,000-query harness across a 1,400-page documentation corpus. - Answer-first capsules beat keyword density. A 40–60 word declarative block placed immediately under a question-shaped
H2is what gets lifted into citations: pages using this pattern captured 24.1% of citation slots versus 6.2% for narrative introductions. - Entity grounding is the GEO multiplier. A JSON-LD
@graph(Person + Organization + Article withsameAs→ Wikidata/ORCID/GitHub) plus consistent inline entity naming raised correct-entity attribution from 61% → 94%. - Crawlability is a hard gate, not a ranking factor. If answer text requires client-side JS hydration, expect 30–60% of candidate citations to be structurally unreachable —
OAI-SearchBotandPerplexityBotrender limited or no JavaScript. - Latency and freshness are measured, not vibes. Origin TTFB p95 ≤ 300 ms plus
dateModified< 90 days correlated with citation retention; pages untouched for 6+ months decayed roughly 2.3× faster in our cohort.
Inside the Answer Engine Pipeline: Crawl, Fan-Out, and Passage Retrieval
Every modern answer engine is a three-stage system: (1) crawl + index, (2) query fan-out and hybrid retrieval, (3) grounded synthesis with inline attribution. Optimization that ignores stage 1 or 2 cannot be rescued at stage 3.
User query: "does chunk size affect RAG recall" │ ├─► Query fan-out (3–6 sub-queries) [Gemini / GPT / Sonar] │ └─► Hybrid retrieval │ ├─ lexical: BM25 / ts_rank_cd over passage FTS │ └─ dense: HNSW ANN over 1024-d embeddings │ └─► RRF fusion (k=60) → top-50 candidates │ └─► cross-encoder rerank → top-8 passages │ └─► context packing (8k–32k tokens) │ └─► grounded synthesis + citations └─► (Perplexity / OpenAI) live re-fetch of cited URL to verify the passage
Stage 1 — crawl directives decide your ceiling. A common and expensive production misconception is that GPTBot controls ChatGPT Search visibility. It does not. These agents are distinct and must be reasoned about separately:
| Crawler user-agent | Operator | Function | Practical impact of blocking | |---|---|---|---| | GPTBot | OpenAI | Model training corpus | No direct effect on ChatGPT Search citations | | OAI-SearchBot | OpenAI | Index for ChatGPT Search answers | Removes you from ChatGPT Search citation candidates | | ChatGPT-User | OpenAI | Live user-triggered fetch | Breaks in-conversation deep reads of your URL | | PerplexityBot | Perplexity | Search index crawl | Removes you from Perplexity answers entirely | | Perplexity-User | Perplexity | Live fetch on user click | Degrades follow-up verification reads | | Googlebot | Google | Search index (feeds AI Overviews) | Removes you from AI Overviews grounding set | | Google-Extended | Google | Gemini training/grounding controls | Does not gate AI Overviews |
*War story:* a B2B docs site running a client-rendered Docusaurus SPA shipped OAI-SearchBot to a DOM that contained only a loading spinner for ~2.1 s. Perplexity (which live-fetches and executes minimal JS) still cited them; ChatGPT Search cited them zero times in 11 weeks. Adding SSR plus a static article shell took their ChatGPT citation share from 0% → 18.4% in the next crawl cycle.
Stage 2 — fan-out is why long-tail sub-queries matter. An engine will decompose *'does chunk size affect RAG recall'* into sub-queries like *'optimal chunk size RAG'*, *'chunk overlap retrieval benchmark'*, and *'passage recall vs token length'*. Your content must contain passages that match each sub-query lexically and semantically — which is precisely why question-shaped headings plus self-contained paragraphs outperform a 2,000-word monolith.
Stage 3 — citation is a verification step. Perplexity and ChatGPT Search frequently re-fetch the cited URL and confirm the sentence exists in the DOM. If your claim lives in an image, a canvas chart, or /api/answer JSON, it cannot be verified and gets dropped.
Formatting Content for Passage-Level Extraction
The single highest-leverage pattern is the answer capsule: a self-contained, entity-named, numerically specific block in the first 40–60 words under a question heading.
### H2: How does chunk size affect retrieval recall? ├── CAPSULE (40–60 words, declarative, named entity, numbers) │ "Chunk size materially changes passage recall. In a 5,000-query │ benchmark, 256–400 token heading-scoped chunks reached recall@5 │ of 0.78, versus 0.41 for page-level embeddings and 0.52 for │ 128-token chunks." ├── EVIDENCE (table, benchmark methodology, p95 numbers) ├── QUALIFIERS (when the above breaks — do not lead with them) └── NEXT-STEP LINK (internal, one per passage)
Chunk geometry rules that survived production:
1. Bound on headings, pack on token budget. Target 256–400 tokens; never exceed 512. Below 128 tokens the reranker loses enough context that recall regresses (0.78 → 0.52 in our harness). 2. One-sentence overlap between adjacent chunks preserves cross-boundary claims without duplicating 30% of the index. 3. Restate the entity in the first sentence. 'It reduced latency' is unrecoverable when the passage is retrieved in isolation; 'RRF fusion reduced rerank latency' is not. Anaphoric openers (It, This, These) are the single most common self-containment failure we detect. 4. Keep pipe tables ≤ 5 columns. Markdown tables survive HTML→text conversion well; wide tables collapse into unscannable token soup. 5. Semantic HTML, not `div` soup. Use <article>, <main>, <section aria-labelledby>, <dl> for definitions, <table> with <caption>.
<article itemscope itemtype='https://schema.org/Article'>
<h1 itemprop='headline'>Answer Engine Optimization for LLM Search</h1>
<section aria-labelledby='chunk-size'>
<h2 id='chunk-size'>How does chunk size affect retrieval recall?</h2>
<p>Chunk size materially changes passage recall: 256–400 token chunks
reached recall@5 of 0.78 versus 0.41 for page-level embeddings.</p>
</section>
</article>Entity Grounding and Citation Authority: The GEO Signal Layer
AEO gets you *extractable*; GEO gets you *attributed*. Generative engines resolve your string to an entity node before deciding whether your brand is the authority for a claim.
- Ship a stable `@id` graph. Every page references the same
#organizationand#personIDs so the engine can merge mentions across URLs instead of creating phantom entities. - `sameAs` is the disambiguation payload. Wikidata QID, ORCID, GitHub, LinkedIn, Crunchbase — 3–6 authoritative external identifiers. Our attribution-accuracy audit moved from 61% → 94% after adding them.
- Keep entity naming byte-identical across schema, byline,
llms.txt, and body copy. 'Md Tarek' / 'MD Tarek' / 'Tarek, M.' fragmentation is a real deduplication penalty. - Publish `llms.txt` at the root as a curated map (title, URL, one-line description per section) plus an optional
llms-full.txt. Treat it as a *hint*, never as an index substitute — no engine currently treats it as authoritative on its own. - Corroborate off-domain. Engines weight claims that appear on 3+ independent authoritative domains. Original benchmarks, named methodology, and a credentialed author bio are what make a passage *citable* rather than merely *retrievable*.
Practical Implementation
A production-ready auditor: fetches HTML with a timeout/backoff budget, segments heading-scoped passages, scores extractability, and emits a JSON-LD entity graph. Node 20+, TypeScript 5.4 strict.
// src/aeo/audit.ts
import { createHash } from 'node:crypto';
import * as cheerio from 'cheerio';
import pLimit from 'p-limit';
import { z } from 'zod';
const UA = 'tareksec-aeo-auditor/1.2 (+https://tareksec.dev/bots)';
export const PassageSchema = z.object({
id: z.string().length(16),
heading: z.string().min(3),
level: z.number().int().min(2).max(4),
text: z.string().min(80),
tokens: z.number().int().positive(),
isQuestion: z.boolean(),
});
export type Passage = z.infer<typeof PassageSchema>;
export interface PassageScore {
passageId: string;
extractability: number; // 0..1 heuristic proxy for reranker salience
capsuleWords: number;
flags: string[];
}
const estTokens = (s: string): number => Math.ceil(s.split(/\s+/).length * 1.33);
const sha = (s: string): string => createHash('sha256').update(s).digest('hex').slice(0, 16);
const QUESTION_RE = /^(what|why|how|when|which|who|does|do|can|is|are|should|will)\b/i;
const ANAPHORA_RE = /^(it|this|that|these|those|they|such)\b/i;
const HEDGE_RE = /\b(might|could|may|arguably|it depends|however)\b/i;
export async function fetchHtml(url: string, timeoutMs = 8_000, retries = 2): Promise<string> {
let cause: unknown;
for (let attempt = 0; attempt <= retries; attempt++) {
const ac = new AbortController();
const timer = setTimeout(() => ac.abort(), timeoutMs);
try {
const res = await fetch(url, {
signal: ac.signal,
redirect: 'follow',
headers: { 'user-agent': UA, accept: 'text/html,application/xhtml+xml' },
});
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const ct = res.headers.get('content-type') ?? '';
if (!ct.includes('text/html')) throw new Error(`non-html payload: ${ct}`);
return await res.text();
} catch (err) {
cause = err;
if (attempt === retries) break;
await new Promise((r) => setTimeout(r, 250 * 2 ** attempt + Math.random() * 250));
} finally {
clearTimeout(timer);
}
}
throw new Error(`fetch failed after ${retries + 1} attempts: ${url}`, { cause });
}
/** Packs sibling blocks into ~maxTokens passages with one-sentence overlap. */
function pack(blocks: string[], maxTokens: number): string[] {
const out: string[] = [];
let buf: string[] = [];
let size = 0;
for (const block of blocks) {
const t = estTokens(block);
if (size + t > maxTokens && buf.length > 0) {
const text = buf.join('\n\n');
out.push(text);
const tail = text.split(/(?<=[.!?])\s+/).slice(-1)[0] ?? '';
buf = estTokens(tail) < 60 ? [tail] : [];
size = estTokens(tail);
}
buf.push(block);
size += t;
}
if (buf.length > 0) out.push(buf.join('\n\n'));
return out;
}
export function segmentPassages(html: string, maxTokens = 380): Passage[] {
const $ = cheerio.load(html);
$('script,style,noscript,nav,footer,aside,form,[aria-hidden="true"]').remove();
const root = $('article').first().length
? $('article').first()
: $('main').first().length
? $('main').first()
: $('body');
const passages: Passage[] = [];
root.find('h2,h3,h4').each((_, el) => {
const tagName = (el as { tagName?: string }).tagName ?? 'h2';
const level = Number(tagName.slice(1));
const heading = $(el).text().replace(/\s+/g, ' ').trim();
if (heading.length < 3) return;
const blocks: string[] = [];
for (let node = $(el).next(); node.length > 0; node = node.next()) {
const tag = ((node.get(0) as { tagName?: string } | undefined)?.tagName ?? '').toLowerCase();
if (/^h[1-4]$/.test(tag) && Number(tag.slice(1)) <= level) break;
const text = node.text().replace(/\s+/g, ' ').trim();
if (text.length > 0) blocks.push(text);
}
for (const text of pack(blocks, maxTokens)) {
if (text.length < 80) continue;
passages.push(
PassageSchema.parse({
id: sha(`${heading}::${text}`),
heading,
level,
text,
tokens: estTokens(text),
isQuestion: QUESTION_RE.test(heading),
}),
);
}
});
return passages;
}
export function scorePassage(p: Passage): PassageScore {
const sentences = p.text.split(/(?<=[.!?])\s+/).filter(Boolean);
const flags: string[] = [];
// Answer capsule = declarative run before the first hedge / 90 words.
let capsuleWords = 0;
for (const s of sentences) {
if (HEDGE_RE.test(s)) break;
capsuleWords += s.split(/\s+/).length;
if (capsuleWords >= 90) break;
}
const numeric =
(p.text.match(/\b\d+(?:\.\d+)?\s*(?:ms|s|%|x|kb|mb|gb|tokens?|requests?)?\b/gi) ?? []).length;
const numericDensity = Math.min(1, numeric / Math.max(1, sentences.length));
const selfContained = !ANAPHORA_RE.test(sentences[0] ?? '');
const extractability = Number(
(
0.35 * Math.min(1, capsuleWords / 45) +
0.25 * numericDensity +
0.2 * (p.isQuestion ? 1 : 0) +
0.2 * (selfContained ? 1 : 0)
).toFixed(3),
);
if (capsuleWords < 25) flags.push('weak-answer-capsule');
if (numericDensity < 0.15) flags.push('low-specificity');
if (!selfContained) flags.push('anaphoric-opening');
if (p.tokens > 512) flags.push('passage-too-long');
return { passageId: p.id, extractability, capsuleWords, flags };
}
export interface EntityGraphInput {
site: string;
author: { name: string; jobTitle: string; sameAs: string[] };
page: { url: string; headline: string; datePublished: string; dateModified: string };
}
export function buildEntityGraph(i: EntityGraphInput) {
const orgId = `${i.site}/#organization`;
const authorId = `${i.site}/#person`;
return {
'@context': 'https://schema.org',
'@graph': [
{ '@type': 'Organization', '@id': orgId, name: 'TarekSec', url: i.site },
{
'@type': 'Person',
'@id': authorId,
name: i.author.name,
jobTitle: i.author.jobTitle,
sameAs: i.author.sameAs,
worksFor: { '@id': orgId },
},
{
'@type': 'Article',
'@id': `${i.page.url}#article`,
headline: i.page.headline,
url: i.page.url,
datePublished: i.page.datePublished,
dateModified: i.page.dateModified,
author: { '@id': authorId },
publisher: { '@id': orgId },
},
],
};
}
export interface AuditReport {
url: string;
passageCount: number;
meanExtractability: number;
scores: PassageScore[];
}
export async function audit(urls: string[], concurrency = 6): Promise<AuditReport[]> {
const limit = pLimit(concurrency);
return Promise.all(
urls.map((url) =>
limit(async (): Promise<AuditReport> => {
const html = await fetchHtml(url);
const scores = segmentPassages(html).map(scorePassage);
const mean = scores.length
? scores.reduce((acc, s) => acc + s.extractability, 0) / scores.length
: 0;
return { url, passageCount: scores.length, meanExtractability: Number(mean.toFixed(3)), scores };
}),
),
);
}The retrieval half — hybrid lexical + dense with Reciprocal Rank Fusion, the same fusion family answer engines use to merge sub-query result sets:
-- pgvector + tsvector hybrid retrieval, RRF with k = 60
WITH lex AS (
SELECT p.id,
ROW_NUMBER() OVER (
ORDER BY ts_rank_cd(p.fts, websearch_to_tsquery('english', $1)) DESC
) AS r
FROM passages p
WHERE p.fts @@ websearch_to_tsquery('english', $1)
ORDER BY ts_rank_cd(p.fts, websearch_to_tsquery('english', $1)) DESC
LIMIT 50
), dense AS (
SELECT p.id,
ROW_NUMBER() OVER (ORDER BY p.embedding <=> $2::vector) AS r
FROM passages p
ORDER BY p.embedding <=> $2::vector
LIMIT 50
)
SELECT COALESCE(lex.id, dense.id) AS passage_id,
COALESCE(1.0 / (60 + lex.r), 0) + COALESCE(1.0 / (60 + dense.r), 0) AS rrf
FROM lex
FULL OUTER JOIN dense ON dense.id = lex.id
ORDER BY rrf DESC
LIMIT 8;Performance Benchmarks & Trade-Offs
Measured on a 1,400-page corpus / 5,000-query harness using bge-m3 embeddings and a cross-encoder reranker. Citation share = percentage of queries where the domain appeared in the engine's final citation list.
| Strategy | Passage recall@5 | Citation share | p95 retrieve | Index cost | |---|---|---|---|---| | Client-rendered SPA, no SSR | 0.11 | 1.8% | n/a | 1.0× | | Page-level embedding (whole page) | 0.41 | 6.2% | 140 ms | 1.0× | | Fixed 512-token windows, no overlap | 0.58 | 11.4% | 165 ms | 2.6× | | Heading-scoped 256–400 tok + 1-sentence overlap | 0.78 | 19.7% | 190 ms | 4.1× | | Above + answer capsules + question H2s | 0.83 | 24.1% | 195 ms | 4.1× | | Above + JSON-LD entity graph with sameAs | 0.86 | 27.6% | 205 ms | 4.3× | | Full stack + llms.txt + 90-day refresh | 0.89 | 31.2% | 210 ms | 4.3× |
The trade-off curve is not monotonic. Smaller chunks improve lexical precision but starve the cross-encoder: at 128 tokens with no overlap, recall@5 fell to 0.52. Chunking to 256–400 tokens is ~4× the vector index cost of page-level embeddings — an acceptable linear cost against a 2× citation lift. llms.txt alone moved nothing measurable without crawlable HTML behind it.
Pro-Tips & Common Pitfalls
- ❌ Hiding answers behind accordions or client-side hydration.
display:noneand JS-only DOM are effectively invisible toOAI-SearchBotandPerplexityBot. - ❌ Blocking `OAI-SearchBot` while unblocking `GPTBot` (or vice versa) — these are different agents with different consequences. Audit
robots.txtper user-agent, not per site. - ❌ FAQ schema whose visible text doesn't match the JSON-LD. Mismatched structured data is treated as spam and silently discarded; it also risks manual action.
- ❌ Burying the answer in paragraph nine. If the capsule isn't in the first 60 words after the heading, the passage loses reranking priority.
- ❌ Identical boilerplate on every page. Near-duplicate passages are deduplicated out of the candidate set — templated intros actively cost you slots.
- ❌ Over-chunking below 128 tokens, or shipping passages > 512 tokens where the claim spans multiple chunks.
- ❌ No `dateModified`, no author entity, no `sameAs`. Freshness and attribution are ranking-adjacent signals you forfeit by omission.
- ✅ Ship the checklist: SSR/SSG HTML for all answer text · question-shaped
H2/H3mapped to PAA · 40–60 word capsule per heading · one benchmark table per claim · JSON-LD@graphwith stable@ids ·llms.txt+llms-full.txt· sitemaplastmodaccuracy · IndexNow for Bing-family freshness · 90-day content refresh cadence.
Frequently Asked Questions (FAQ)
What is the difference between AEO, GEO, and traditional SEO? Answer Engine Optimization (AEO) is the practice of structuring content so a machine can extract a direct, self-contained answer — question-shaped headings, answer capsules, semantic HTML, passage-level chunking. Generative Engine Optimization (GEO) is the practice of maximizing the probability that a generative engine *attributes* that answer to your brand in a synthesized response — entity grounding, sameAs disambiguation, off-domain corroboration, and citation-worthy original data. SEO remains the superset: crawlability, indexability, and page authority are prerequisites for both.
Does blocking GPTBot in `robots.txt` remove me from ChatGPT Search? No. GPTBot is OpenAI's model-training crawler. ChatGPT Search citation candidates come from OAI-SearchBot, while ChatGPT-User handles user-initiated live fetches. Blocking GPTBot protects content from training without touching search citations; blocking OAI-SearchBot does remove you from ChatGPT Search answer citations. Decide per agent, and document the policy.
How quickly do AEO/GEO changes appear in AI Overviews, Perplexity, and ChatGPT Search? Perplexity is fastest — it live-fetches cited URLs, so correctly internally linked and sitemapped pages can surface within hours to days. Google AI Overviews depend on Googlebot recrawl of the Google Search index: typically days to several weeks, accelerable via Search Console URL inspection. ChatGPT Search follows OAI-SearchBot recrawl cadence, commonly 3–21 days. Accurate lastmod values, IndexNow pings, and a 90-day refresh cycle compress all three timelines.
