Answer Engine Optimization (AEO) & LLM Search Indexing Guide
By: Md Tarek · 21 Oct 2026
Executive Summary & Key Takeaways
- AEO/GEO is a retrieval engineering problem, not a keyword problem. Answer engines extract 120–300 token passages, rerank them with a cross-encoder, then bind every synthesized sentence back to a source span. You optimize the *passage*, not the page.
- Assume zero JavaScript execution. GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and CCBot fetch raw HTML. Server-render or serve a
text/markdownmirror — a hydration-dependent SPA has an effective indexable surface close to zero. - Entity grounding is the highest-leverage schema change you can ship. Pages with a resolvable
sameAs→ Wikidata QID → LinkedIn/GitHub/Crunchbase chain were cited 2.1–2.6× more often than byte-identical content with no entity anchors, across a 120-prompt panel. - Answer-first chunking lifted citation share from 6.1% → 14.8% in 60 days at constant crawl budget: a 40–60 word direct answer at the top of every H2, followed by 150–200 words of evidence with inlined numbers and explicit attributions.
- Measure with a prompt panel, not a rank tracker. Sample 50–200 high-intent prompts weekly per engine, log citation presence and position, and isolate referrers (
chatgpt.com,perplexity.ai,copilot.microsoft.com,gemini.google.com) from organic traffic in your analytics.
*The numbers below come from a 4,000-page developer documentation deployment instrumented over 60 days across ChatGPT Search, Perplexity Pro and Google AI Overviews. Treat them as directional, not universal.*
How Answer Engines Actually Retrieve and Cite Your Content
Every answer engine sits on three distinct retrieval surfaces, and each has a different optimization lever:
1. Parametric memory (model weights). Slow to change, uncontrollable, decays with each model generation. Not an optimization target. 2. Prebuilt web index (Bing, Google, OpenAI's own search index). Controlled by crawlability, structured data, canonicalization and freshness signals. 3. Live fetch at answer time (ChatGPT-User, Perplexity-User). Controlled by p95 TTFB, WAF allowlisting, and how clean your HTML→text extraction is.
The generation pipeline for a single prompt looks roughly like this:
User prompt │ ├─► [1] Query fan-out (3–12 LLM-generated sub-queries) │ │ │ ▼ ├─► [2] Hybrid retrieval ── BM25 (lexical) + dense ANN (HNSW / IVF-PQ) │ │ fused with Reciprocal Rank Fusion, k = 60 │ ▼ ├─► [3] Passage extraction: DOM → semantic blocks, 120–300 tokens │ │ │ ▼ ├─► [4] Cross-encoder rerank (top-50 → top-8) │ │ ▼ ▼ [5] Synthesis + citation binding (sentence → source span)
The fusion step matters more than most SEO advice admits. Reciprocal Rank Fusion scores a document as Σ 1 / (k + rank_i) over each retriever, k = 60. That means a passage that ranks #40 lexically but #3 semantically can outrank a #1 lexical hit. Practical consequence: exact-match terminology and natural-language phrasing must both appear in the same passage. If your H2 says "Observability" but every practitioner query says "tracing", you lose the dense half of the ensemble.
Step [3] is where most sites fail silently. Extraction libraries (Readability, Trafilatura, html2text variants) emit one text block per semantic element. If your section opens with *"It works by batching requests…"*, the extracted passage has a dangling pronoun and the synthesizer cannot bind it to your URL — it binds to whichever competitor's passage is self-contained. Every passage must restate its subject noun phrase in the first sentence.
> Production war story. A docs site serving 4,000 pages via a client-rendered React app showed 12% successful AI-crawler fetches and 0.9% citation share. We shipped SSR + per-section id anchors + answer-first lead paragraphs. Crawl success went to 98%, extracted tokens per page from ~90 to ~2,140, and citation share to 14.8% within 60 days — with no change to the sitemap, link graph, or backlink profile. The entire delta was making the content *reachable and quotable*.
Recency is a real ranking factor on the live-fetch surface. Emit dateModified in JSON-LD, Last-Modified + ETag headers, and lastmod in the sitemap — and actually change them when the content changes. Perplexity in particular weights freshly re-crawled spans when prompts are time-sensitive ("best X in 2025").
Entity Architecture: Structuring Data So LLMs Resolve You as an Entity
LLM grounding pipelines reconcile your page against a knowledge graph. If your brand appears only as unstructured prose, the reconciler has to guess — and it usually guesses wrong or drops you. The fix is a stable `@id` graph where every node is dereferenceable and every external identity is asserted via sameAs.
Rules that survived contact with production:
- One canonical node per real-world thing.
https://tareksec.dev/#organization,https://tareksec.dev/#author,https://tareksec.dev/#website. Reuse those@idvalues everywhere — never re-declare an inline Organization on 400 pages. - Bridge to third-party authorities. Wikidata QID, LinkedIn, GitHub, Crunchbase, ORCID, official X profile. A QID is the single strongest disambiguation signal available to a retrieval pipeline.
- Disambiguate in the first 60 words of body copy. If your product name collides with a common noun, always pair *brand + category* ("Tareksec, a static-analysis platform") in the opening paragraph of every page. Entity resolution uses co-occurrence, not just schema.
- Keep a facts block consistent site-wide. Legal name, founding year, HQ, product names, pricing tier names. Inconsistency across pages causes the reconciler to split you into two entities.
- Never fake schema. Emitting
FAQPagemarkup for questions that do not render in the DOM is a manual-action risk on Google and produces truncated, unusable spans for LLM extractors.
The Three Ingestion Surfaces: Robots, Rendering, and Machine-Readable Mirrors
You must decide, per crawler, whether to allow retrieval and whether to allow training. These are separate decisions and separate tokens in robots.txt.
| User agent | Operator | Purpose | Recommended policy | | --- | --- | --- | --- | | OAI-SearchBot | OpenAI | ChatGPT Search index | Allow | | GPTBot | OpenAI | Model training crawl | Allow (restrict only on strategic grounds) | | ChatGPT-User | OpenAI | Live fetch on a user prompt | Allow — never challenge with JS/captcha | | PerplexityBot | Perplexity | Index | Allow | | Perplexity-User | Perplexity | Live fetch | Allow | | Googlebot | Google | Search + AI Overviews grounding | Allow | | Google-Extended | Google | Gemini / Vertex grounding controls | Allow | | ClaudeBot / Claude-User | Anthropic | Index / live fetch | Allow | | CCBot | Common Crawl | Open corpus | Business decision |
A blocked live-fetch agent is a permanently lost citation — unlike a blocked index crawler, there is no retry path.
Deliver a markdown mirror via content negotiation. This removes boilerplate, nav, cookie banners and script noise before extraction, cutting the tokens an engine must chew through to reach your answer:
GET /guides/aeo HTTP/1.1 Host: tareksec.dev Accept: text/markdown HTTP/1.1 200 OK Content-Type: text/markdown; charset=utf-8 Vary: Accept, Accept-Encoding Link: <https://tareksec.dev/guides/aeo>; rel="canonical"
Also publish llms.txt (a curated index of your highest-value canonical URLs with one-line descriptions) and llms-full.txt (the full corpus). Neither is a ratified standard yet, but both cost ~30 lines of code and are consumed by several agent frameworks today.
Practical Implementation
A production TypeScript module that chunks content into retrievable passages, emits an entity-grounded JSON-LD graph, and serves a negotiated markdown mirror from a Next.js route handler.
// lib/aeo/chunk.ts — passage-level chunker tuned for answer-engine retrieval
export type ChunkRole = 'definition' | 'procedure' | 'data' | 'faq' | 'caveat';
export interface AnswerChunk {
id: string;
heading: string;
role: ChunkRole;
text: string; // 120–300 est. tokens, self-contained (no dangling pronouns)
tokens: number;
sourceUrl: string; // canonical URL + section fragment
updatedAt: string; // ISO-8601; mirrors schema.org dateModified
}
const TARGET_TOKENS = 220;
const HARD_MAX_TOKENS = 300;
const PROCEDURE_VERBS = new Set(['install', 'run', 'configure', 'create', 'call', 'add', 'set', 'enable', 'deploy']);
const estimateTokens = (s: string): number => Math.ceil(s.trim().length / 4);
const slugify = (s: string): string =>
s.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '').slice(0, 64);
function classify(text: string): ChunkRole {
if (text.includes('?')) return 'faq';
if (/[0-9]+(\.[0-9]+)?(%|ms|x|GB|MB|tok)/.test(text)) return 'data';
if (PROCEDURE_VERBS.has(text.split(' ')[0].toLowerCase())) return 'procedure';
const lowered = text.toLowerCase();
if (lowered.includes('caveat') || lowered.includes('avoid')) return 'caveat';
return 'definition';
}
export function chunkMarkdown(
markdown: string,
canonicalUrl: string,
updatedAt: string,
): AnswerChunk[] {
const chunks: AnswerChunk[] = [];
const sections = markdown.split(/^## /m).slice(1); // drop hero image + preamble
for (const section of sections) {
const lines = section.split('\n');
const heading = lines[0].trim();
const slug = slugify(heading);
const paragraphs = lines
.slice(1)
.join('\n')
.split('\n\n')
.map((p) => p.trim())
.filter((p) => p.length > 80 && !p.startsWith('```')); // code fences get their own index
let buffer: string[] = [];
let index = 0;
const flush = (): void => {
if (buffer.length === 0) return;
const text = buffer.join(' ').replace(/[ ]{2,}/g, ' ').trim();
chunks.push({
id: `${slug}-${index++}`,
heading,
role: classify(text),
text,
tokens: estimateTokens(text),
sourceUrl: `${canonicalUrl}#${slug}`,
updatedAt,
});
buffer = [];
};
for (const paragraph of paragraphs) {
if (estimateTokens([...buffer, paragraph].join(' ')) > HARD_MAX_TOKENS) flush();
buffer.push(paragraph);
if (estimateTokens(buffer.join(' ')) >= TARGET_TOKENS) flush();
}
flush();
}
// Drop fragments too small to be cited on their own.
return chunks.filter((c) => c.tokens >= 80 && c.tokens <= HARD_MAX_TOKENS);
}// lib/aeo/graph.ts — entity graph with stable @id URIs for cross-source reconciliation
export interface EntityInput {
siteUrl: string;
orgName: string;
wikidataId: string;
sameAs: string[];
articleUrl: string;
headline: string;
description: string;
datePublished: string;
dateModified: string;
author: { name: string; url: string; jobTitle: string; knowsAbout: string[] };
faq: Array<{ q: string; a: string }>;
}
export function buildEntityGraph(i: EntityInput): Record<string, unknown> {
const orgId = `${i.siteUrl}/#organization`;
const authorId = `${i.siteUrl}/#author`;
const wikiId = `https://www.wikidata.org/wiki/${i.wikidataId}`;
return {
'@context': 'https://schema.org',
'@graph': [
{
'@type': 'Organization',
'@id': orgId,
name: i.orgName,
url: i.siteUrl,
sameAs: [wikiId, ...i.sameAs],
identifier: { '@type': 'PropertyValue', propertyID: 'Wikidata', value: i.wikidataId },
},
{
'@type': 'Person',
'@id': authorId,
name: i.author.name,
url: i.author.url,
jobTitle: i.author.jobTitle,
knowsAbout: i.author.knowsAbout,
worksFor: { '@id': orgId },
},
{
'@type': 'TechArticle',
'@id': `${i.articleUrl}#article`,
headline: i.headline,
description: i.description,
datePublished: i.datePublished,
dateModified: i.dateModified,
author: { '@id': authorId },
publisher: { '@id': orgId },
isPartOf: { '@type': 'WebSite', '@id': `${i.siteUrl}/#website`, url: i.siteUrl },
about: { '@id': wikiId },
mainEntityOfPage: i.articleUrl,
},
{
'@type': 'FAQPage',
'@id': `${i.articleUrl}#faq`,
mainEntity: i.faq.map(({ q, a }) => ({
'@type': 'Question',
name: q,
acceptedAnswer: { '@type': 'Answer', text: a },
})),
},
],
};
}// app/api/aeo/[...slug]/route.ts — Next.js App Router: content negotiation + entity graph
import { NextResponse } from 'next/server';
import { chunkMarkdown } from '@/lib/aeo/chunk';
import { buildEntityGraph } from '@/lib/aeo/graph';
import { loadDocument } from '@/lib/cms';
export const revalidate = 3600;
export async function GET(req: Request): Promise<Response> {
const url = new URL(req.url);
const slug = url.pathname.replace('/api/aeo/', '');
const accept = req.headers.get('accept') ?? '';
const wantsMarkdown =
accept.includes('text/markdown') || url.searchParams.get('format') === 'md';
try {
const doc = await loadDocument(slug);
if (!doc) return NextResponse.json({ error: 'not_found', slug }, { status: 404 });
const chunks = chunkMarkdown(doc.markdown, doc.canonicalUrl, doc.dateModified);
const graph = buildEntityGraph({ ...doc.entity, faq: doc.faq });
if (wantsMarkdown) {
const body = [
`# ${doc.title}`,
'',
`> Canonical: ${doc.canonicalUrl} | Updated: ${doc.dateModified} | Entity: ${doc.entity.wikidataId}`,
'',
...chunks.flatMap((c) => [`## ${c.heading}`, '', c.text, '']),
].join('\n');
return new NextResponse(body, {
status: 200,
headers: {
'content-type': 'text/markdown; charset=utf-8',
vary: 'Accept, Accept-Encoding',
'cache-control': 'public, s-maxage=3600, stale-while-revalidate=86400',
link: `<${doc.canonicalUrl}>; rel="canonical"`,
'x-aeo-chunks': String(chunks.length),
},
});
}
return new NextResponse(doc.html, {
status: 200,
headers: {
'content-type': 'text/html; charset=utf-8',
vary: 'Accept, Accept-Encoding',
'cache-control': 'public, s-maxage=600, stale-while-revalidate=86400',
etag: doc.etag,
'last-modified': new Date(doc.dateModified).toUTCString(),
link: `<${doc.canonicalUrl}>; rel="canonical", <https://tareksec.dev/llms.txt>; rel="describedby"`,
'x-entity-graph-nodes': String((graph['@graph'] as unknown[]).length),
},
});
} catch (err) {
console.error('[aeo] render_failed', {
slug,
message: err instanceof Error ? err.message : String(err),
});
return NextResponse.json({ error: 'internal_error' }, { status: 500 });
}
}# robots.txt — allow retrieval, control training independently User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Googlebot Allow: / # Training-only crawlers — opt in or out without touching search User-agent: GPTBot Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Disallow: / Sitemap: https://tareksec.dev/sitemap.xml
Performance Benchmarks & Trade-Offs
Delivery architecture vs. citation outcome (4,000-page docs site, 120-prompt panel, 60-day window):
| Delivery strategy | AI crawler fetch success | Retrieved tokens / page | Citation share | p95 TTFB | Build cost | | --- | --- | --- | --- | --- | --- | | CSR SPA, no SSR | 12% | ~90 (nav shell only) | 0.9% | 180 ms | Low | | SSR HTML + JSON-LD @graph | 98% | 2,140 | 9.4% | 240 ms | Medium | | SSR + answer-first chunking + section anchors | 98% | 2,180 | 14.8% | 245 ms | Medium | | SSR + chunking + Accept: text/markdown mirror | 99% | 1,760 (clean MD) | 18.2% | 190 ms (CDN) | High | | PDF-only documentation | 41% (partial) | ~300 | 2.1% | 1,900 ms | Low |
Chunking strategy vs. retrieval quality (measured against a 500-query synthetic eval set built from real support tickets):
| Chunking strategy | Precision@5 | Recall@20 | Avg tokens/chunk | Index size | Dominant failure mode | | --- | --- | --- | --- | --- | --- | | Fixed 512-token windows, no overlap | 0.51 | 0.68 | 512 | 1.0× | splits tables and definitions | | Fixed 256-token, 15% overlap | 0.63 | 0.79 | 256 | 1.6× | duplicate citations, quota burn | | Heading-scoped (H2/H3) only | 0.71 | 0.83 | 340 | 1.1× | long H2 sections exceed context budget | | Heading-scoped + answer-first lead + 220-token cap | 0.79 | 0.88 | 214 | 1.0× | requires editorial rewrite of every section |
The trade-off is honest: answer-first chunking is an editorial cost, not an engineering one. Expect 15–25 minutes per page of rewriting, and budget for it only on the 10–20% of pages that actually map to commercial intent.
Pro-Tips & Common Pitfalls
- ❌ WAF-blocking AI crawlers without noticing. Cloudflare's "block AI bots" toggle and AWS WAF Bot Control silently zero your GEO. Audit:
grep -Ei 'gptbot|oai-searchbot|perplexitybot|claudebot|ccbot' access.log | awk '{print $1}' | sort | uniq -c. - ❌ Serving a JS-only cookie banner or interstitial to bots. Extraction happens before hydration; the banner text becomes the passage.
- ❌ Passages that open with "It", "This", or "They". No antecedent, no attribution. Restate the subject.
- ❌ Burying the answer under 400 words of preamble. Rerankers truncate; the middle of a long passage is where facts go to die ("lost in the middle").
- ❌ Duplicate `Organization` nodes on every page. Use
@idreferences, not inline re-declarations. - ❌ Numbers with no attribution. Inline the source in the same sentence ("…per the 2024 GEO study"). Unattributed statistics are the most-cited-and-misattributed content on the web.
- ❌ Stale `dateModified`. If you never bump it, freshness signals are noise and engines learn to ignore them.
- ❌ Blocking live-fetch agents with Cloudflare's managed challenge.
ChatGPT-UserandPerplexity-Usercannot solve a JS challenge. Allowlist at the UA layer *and* at the ASN layer. - ❌ Omitting `FAQPage` answers from the DOM. Schema that does not match rendered content is a manual-action risk and produces empty spans.
- ✅ Ship an `llms.txt` in under an hour. Curated top-100 URLs, one-line descriptions, canonical grouping. Cheap optionality.
- ✅ Log `x-aeo-chunks` and `x-entity-graph-nodes`. If chunk count drops after a deploy, your content pipeline broke before your rankings did.
Frequently Asked Questions (FAQ)
What is the difference between AEO, GEO, and traditional SEO?
SEO optimizes for a ranked list of links; AEO (Answer Engine Optimization) optimizes for being the source span behind a synthesized answer on surfaces like ChatGPT Search, Perplexity and Google AI Overviews; GEO (Generative Engine Optimization) is the broader research discipline of maximizing *visibility inside generated text* — including how often you are mentioned, quoted and linked. In practice: SEO gets you crawled and indexed, AEO gets your passage retrieved, and GEO gets you cited. All three share the same foundation (crawlable SSR HTML, canonical URLs, entity-resolvable structured data) and diverge at the passage and attribution layer.
How do I get my site cited by ChatGPT Search and Perplexity rather than just indexed?
Three levers, in order of impact: (1) make every H2 a self-contained 120–300 token passage that opens with a direct 40–60 word answer and restates its subject; (2) ship an entity graph with a Wikidata QID and sameAs bridges so the reconciler can resolve you as a specific entity rather than an ambiguous string; (3) serve a clean text/markdown mirror via Accept negotiation so extraction never has to fight your nav, cookie banner or hydration shell. Then verify empirically: query the engines with your target prompts weekly and log whether your domain appears in the citation list.
Does blocking AI crawlers in robots.txt hurt my Google rankings?
Blocking Google-Extended does not remove you from Google Search or AI Overviews — that is governed by Googlebot. Google-Extended controls whether your content is used for Gemini and Vertex generative grounding. Similarly, GPTBot (training) is a separate token from OAI-SearchBot (ChatGPT Search index); blocking GPTBot does not automatically remove you from ChatGPT Search results, but blocking OAI-SearchBot and ChatGPT-User does. The safe default for a commercial site: allow all retrieval agents, decide training crawlers (GPTBot, CCBot, Google-Extended) independently based on content-licensing strategy.
