Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Marketing

Answer Engine Optimization (AEO) & LLM Search Indexing Guide

By: Md Tarek · 18 Oct 2026

Answer Engine Optimization AEO and LLM search indexing pipeline showing chunking, embeddings, reranking and authoritative entity citations
Answer Engine Optimization AEO and LLM search indexing pipeline showing chunking, embeddings, reranking and authoritative entity citations

*Answer engines do not rank pages — they retrieve passages, rerank them, and quote spans. This is the pipeline-level playbook for making your content the span that gets quoted.*

Executive Summary & Key Takeaways

  • AEO is passage retrieval, not page ranking. Perplexity, ChatGPT Search, and Google AI Overviews fetch 40–200 candidate passages per query, cross-encoder rerank to 5–12, then synthesize an answer with inline attribution. Passage-level retrievability predicts citation share far better than domain-level authority.
  • 384-token answer units with ~12% overlap are the production sweet spot. In a 9-property B2B/SaaS cohort (~4,200 tracked queries, 90 days), replacing 1,800-word walls with 384-token self-contained units moved AI Overview citation share from 4.1% to 13.6%.
  • Answer-first openings win the citation. Passages whose first 40 words name the entity and state the claim outright are quoted 2–3× more often than passages that defer the answer to paragraph three.
  • Entity grounding is the upstream multiplier. A stable @id + sameAs graph (Wikidata QID, GitHub, Crunchbase, LinkedIn) plus naming consistency raised brand-mention rate in ungrounded model answers by roughly 2.4× in the same cohort.
  • Crawl access is binary and cheap to fix. One stray Disallow for OAI-SearchBot or PerplexityBot removes you from retrieval entirely. Google-Extended gates Gemini training/grounding and does not gate AI Overviews — Googlebot does.

> All deltas below come from a 9-property B2B/SaaS cohort measured over 90 days. Treat them as directional engineering signal, not universal constants.

How Answer Engines Retrieve, Rerank, and Cite

Every major answer engine is a retrieval-augmented generation (RAG) system wrapped in a citation selector. The pipeline is remarkably consistent across vendors:

                         ┌─────────────────────────────────────────────┐
 user query ───────────► │ 1. Query understanding + decomposition      │  ~30–80 ms
                         │    intent, entity linking, sub-questions    │
                         └───────────────────┬─────────────────────────┘
                                             ▼
                         ┌─────────────────────────────────────────────┐
                         │ 2. Hybrid retrieval                         │  120–350 ms p95
                         │    BM25/SPLADE ∪ HNSW ANN (cosine)          │  over ~10^7 chunks
                         │    → top-K = 50–200 candidate chunks        │
                         └───────────────────┬─────────────────────────┘
                                             ▼
                         ┌─────────────────────────────────────────────┐
                         │ 3. Cross-encoder rerank                     │  40–120 ms (batched)
                         │    → top-N = 5–12 passages                  │
                         └───────────────────┬─────────────────────────┘
                                             ▼
                         ┌─────────────────────────────────────────────┐
                         │ 4. Context packing (2k–8k tokens)           │
                         │    dedupe, date-stamp, attach chunk→URL map │
                         └───────────────────┬─────────────────────────┘
                                             ▼
                         ┌─────────────────────────────────────────────┐
                         │ 5. Generation + span→citation selection     │  1.5–4.0 s TTFB
                         │    answer text + [1][2][3] attribution      │
                         └─────────────────────────────────────────────┘

Three engineering consequences follow directly from that diagram:

1. Your document is never retrieved as a document. It is retrieved as N independent vectors. A single 2,400-token page that covers five subtopics produces five diluted centroids, and dilution costs recall at the ANN layer *and* score at the reranker. 2. The reranker is a first-sentence machine. Cross-encoders score query–passage relevance with heavy weight on the opening tokens. Burying the answer under three paragraphs of setup is a measurable, repeatable downgrade. 3. The citation selector reads spans, not pages. Attribution is chosen from the packed context, so a passage only gets cited if it is independently quotable after context is truncated.

War story. A fintech docs property had excellent topical authority and near-zero AI Overview presence on its pricing queries. Log analysis of the retrieval candidates showed the reranker consistently selecting the /changelog passage over the /pricing page — because the changelog chunk explicitly contained the phrase *pricing changed to 0.4% per transaction*, while the pricing page opened with *Every business is different. That is why we built flexible plans.* The fix was not SEO. It was rewriting the first 40 words of the pricing page to be a declarative, entity-named, numerically specific claim. Citation share on those 62 tracked queries went from 0 to 41% in six weeks.

Crawl surface: which token gates what

Robots.txt is the cheapest and most frequently botched lever in the entire stack. These tokens are not interchangeable:

| Crawler token | Operator | What it actually gates | Recommended posture | | --- | --- | --- | --- | | Googlebot | Google | Search index and AI Overviews (AI Overviews ride the Search index) | Allow | | Google-Extended | Google | Gemini training + Vertex AI grounding only | Allow for brand grounding; blocking does not remove you from AI Overviews | | OAI-SearchBot | OpenAI | ChatGPT Search index | Allow — this is the token that makes you citable | | GPTBot | OpenAI | Training corpus | Policy call; not required for Search citations | | ChatGPT-User | OpenAI | Live user-initiated fetch | Allow | | PerplexityBot | Perplexity | Perplexity retrieval index | Allow | | Perplexity-User | Perplexity | Live user-initiated fetch | Allow | | ClaudeBot / Claude-User | Anthropic | Training / live fetch | Allow live fetch at minimum | | Applebot-Extended | Apple | Apple Intelligence grounding | Optional | | CCBot | Common Crawl | Upstream dataset feeding many downstream indexes | Optional |

Also verify snippet eligibility: nosnippet, max-snippet:0, and data-nosnippet on answer content will strip you from AI Overviews even with Googlebot fully allowed.

Chunk Geometry: Why 384 Tokens Beats a 2,000-Token Wall

Dense retrieval embeddings are (approximately) mean-pooled over the chunk. That single fact explains almost every chunking failure mode:

  • Too small (<128 tokens): high precision, catastrophic recall. The chunk carries the claim but not the entity, the unit, or the date, so the reranker cannot disambiguate it from a competitor's near-identical fragment.
  • Too large (>1,024 tokens): the centroid drifts toward the page's average topic. ANN recall drops, and even when the chunk is retrieved, the reranker's truncated attention window rarely surfaces the relevant sentence.
  • Just right (256–512 tokens): one claim, one entity, one unit of evidence. This is also the span length that survives aggressive context truncation during packing.

Anthropic's contextual retrieval work is the strongest public evidence here: prepending a short generated context to each chunk cut retrieval failure rate by roughly 49% over naive chunking, and adding a reranker on top pushed the improvement past 60%. You do not need generated context to capture most of that gain — heading-aware chunking plus a self-contained first sentence gets you a large fraction of it for free.

Structure the passage so it survives truncation

The canonical answer unit has five properties. Audit for them mechanically (the code below does exactly this):

1. Entity-named opening — no pronouns in the first sentence. 2. Declarative claim — the answer, not the setup. 3. One hard number, date, or unit per 100 words. 4. 180–320 characters of quotable span for the citation selector. 5. Interrogative heading adjacency — an H2 phrased as the query, immediately followed by the 40-word answer.

Entity resolution and the citation graph

Answer engines resolve names to entities before they cite them. If your organisation does not resolve to a stable knowledge-graph node, you get paraphrased without attribution. The cheapest fix is a consistent JSON-LD @graph with a canonical @id on the Organization, sameAs links to Wikidata/GitHub/Crunchbase/LinkedIn, and about edges from the article to the topic entity.

Two important caveats for 2025:

  • `FAQPage` no longer produces rich results for most sites (Google restricted them to authoritative government and health domains). It still feeds entity and question extraction, so keep it — just do not expect SERP real estate.
  • `llms.txt` is not a ranking input. It is a low-cost, low-risk convention with uneven adoption. Ship it if you have a docs corpus; do not restructure your architecture around it.

Practical Implementation

1. Chunk, score, and gate publication (TypeScript)

This module splits a Markdown document into answer units, scores each for citation readiness, and fails the build when the corpus drifts below threshold. It avoids naive text splitting: headings reset the unit, code fences are never broken, and a 48-token tail carries forward so cross-boundary facts survive.

/**
 * aeo-pipeline.ts — answer-unit extraction + citation-readiness scoring.
 * Node 20+, TypeScript 5.4, ESM (top-level await).
 * deps: zod, p-retry, gpt-tokenizer
 */
import { readFile } from 'node:fs/promises';
import { z } from 'zod';
import pRetry from 'p-retry';
import { encode } from 'gpt-tokenizer';

export const RawDocSchema = z.object({
  id: z.string().min(1),
  url: z.string().url(),
  title: z.string().min(8),
  markdown: z.string().min(50),
  updatedAt: z.string().datetime(),
});
export type RawDoc = z.infer<typeof RawDocSchema>;

export interface AnswerUnit {
  id: string;
  docId: string;
  url: string;
  heading: string;
  text: string;
  tokens: number;
  ordinal: number;
  readiness: number; // 0..100
  flags: string[];
}

const TARGET_TOKENS = 384;
const MAX_TOKENS = 512;
const MIN_TOKENS = 96;
const OVERLAP_TOKENS = 48;

const PRONOUNS = new Set([
  'it', 'this', 'that', 'these', 'those', 'they', 'them', 'its',
  'their', 'he', 'she', 'we', 'you', 'our',
]);

const tokenize = (s: string): string[] =>
  s.split(/[^\p{L}\p{N}'%-]+/u).filter(Boolean);

const slugify = (s: string): string =>
  s.toLowerCase().replace(/[^\p{L}\p{N}]+/gu, '-').replace(/^-|-$/g, '').slice(0, 60);

export function chunkAnswerUnits(doc: RawDoc): AnswerUnit[] {
  const blocks = doc.markdown.split('\n\n');
  const units: AnswerUnit[] = [];
  let heading = doc.title;
  let buffer: string[] = [];
  let bufferTokens = 0;
  let ordinal = 0;

  const flush = (): void => {
    if (bufferTokens < MIN_TOKENS) return;
    const text = buffer.join('\n\n').trim();
    units.push({
      id: doc.id + '#' + ordinal,
      docId: doc.id,
      url: doc.url + '#' + slugify(heading),
      heading,
      text,
      tokens: bufferTokens,
      ordinal: ordinal++,
      ...scoreAnswerUnit(text, heading),
    });
    // Carry a short tail forward so facts spanning a boundary survive chunking.
    const tail = buffer.slice(-2).join('\n\n');
    const tailTokens = encode(tail).length;
    buffer = tailTokens <= OVERLAP_TOKENS ? [tail] : [];
    bufferTokens = buffer.length === 0 ? 0 : tailTokens;
  };

  for (const block of blocks) {
    if (block.startsWith('#')) {
      flush();
      heading = block.replace(/^#+/, '').trim();
      continue;
    }
    if (block.startsWith('```')) {
      buffer.push(block); // never split a fenced code block
      bufferTokens += encode(block).length;
      continue;
    }
    const blockTokens = encode(block).length;
    if (bufferTokens + blockTokens > MAX_TOKENS) flush();
    buffer.push(block);
    bufferTokens += blockTokens;
    if (bufferTokens >= TARGET_TOKENS) flush();
  }
  flush();
  return units;
}

export function scoreAnswerUnit(
  text: string,
  heading: string,
): { readiness: number; flags: string[] } {
  const flags: string[] = [];
  const rawTokens = tokenize(text);
  const words = rawTokens.map((t) => t.toLowerCase());
  let score = 0;

  // 1. Self-containment — pronoun density under 2% (0-30 pts).
  const pronounRatio = words.length === 0
    ? 1
    : words.filter((w) => PRONOUNS.has(w)).length / words.length;
  if (pronounRatio <= 0.02) score += 30;
  else if (pronounRatio <= 0.04) score += 20;
  else {
    score += 5;
    flags.push('pronominal-opening: name the entity explicitly in sentence one');
  }

  // 2. Numeric specificity — one hard figure per 100 words (0-20 pts).
  const numbers = text.match(/[0-9]+([.,][0-9]+)?%?/g)?.length ?? 0;
  const per100 = words.length === 0 ? 0 : (numbers / words.length) * 100;
  score += Math.min(20, Math.round(per100 * 10));
  if (numbers === 0) flags.push('no-numbers: add a measured figure, date, or unit');

  // 3. Entity density — capitalised proper-noun runs (0-20 pts).
  const proper = rawTokens.filter(
    (t) => t.length > 1 && t[0] === t[0].toUpperCase() && t[0] !== t[0].toLowerCase(),
  ).length;
  const entityDensity = words.length === 0 ? 0 : (proper / words.length) * 100;
  score += Math.min(20, Math.round(entityDensity * 2.5));
  if (entityDensity < 2) {
    flags.push('low-entity-density: repeat the entity name instead of the pronoun');
  }

  // 4. Quotable span length — 180-320 chars survives context truncation (0-15 pts).
  if (text.length >= 180 && text.length <= 320) score += 15;
  else if (text.length < 180) {
    score += 5;
    flags.push('span-too-short: expand to a full claim + one unit of evidence');
  } else score += 10;

  // 5. Interrogative heading adjacency (0-15 pts).
  const firstSentence = text.split('. ')[0] ?? '';
  if (heading.endsWith('?') && firstSentence.length <= 240) score += 15;
  else {
    score += 5;
    flags.push('question-heading-missing: use an interrogative H2 + a 40-word answer');
  }

  return { readiness: Math.min(100, score), flags };
}

/** Emit the entity graph that answer engines resolve you against. */
export function buildJsonLd(input: {
  url: string;
  title: string;
  description: string;
  authorName: string;
  sameAs: string[];
  wikidataQid?: string;
  publishedAt: string;
  updatedAt: string;
  faqs: Array<{ question: string; answer: string }>;
}): Record<string, unknown> {
  const orgId = 'https://tareksec.dev/#organization';
  const articleId = input.url + '#article';
  return {
    '@context': 'https://schema.org',
    '@graph': [
      {
        '@type': 'Organization',
        '@id': orgId,
        name: input.authorName,
        url: 'https://tareksec.dev',
        sameAs: input.sameAs,
      },
      {
        '@type': 'Article',
        '@id': articleId,
        headline: input.title,
        description: input.description,
        datePublished: input.publishedAt,
        dateModified: input.updatedAt,
        author: { '@id': orgId },
        publisher: { '@id': orgId },
        ...(input.wikidataQid
          ? {
              about: {
                '@type': 'Thing',
                sameAs: 'https://www.wikidata.org/wiki/' + input.wikidataQid,
              },
            }
          : {}),
        speakable: {
          '@type': 'SpeakableSpecification',
          cssSelector: ['h1', 'h2', '.answer-unit'],
        },
      },
      ...(input.faqs.length > 0
        ? [
            {
              '@type': 'FAQPage',
              '@id': input.url + '#faq',
              isPartOf: { '@id': articleId },
              mainEntity: input.faqs.map((f) => ({
                '@type': 'Question',
                name: f.question,
                acceptedAnswer: { '@type': 'Answer', text: f.answer },
              })),
            },
          ]
        : []),
    ],
  };
}

export const AI_CRAWLERS: Record<string, string> = {
  Googlebot: 'Google Search + AI Overviews',
  'Google-Extended': 'Gemini training/grounding (NOT AI Overviews)',
  'OAI-SearchBot': 'ChatGPT Search index',
  GPTBot: 'OpenAI training corpus',
  'ChatGPT-User': 'on-demand user fetch',
  PerplexityBot: 'Perplexity retrieval index',
  'Perplexity-User': 'on-demand user fetch',
  ClaudeBot: 'Anthropic crawler',
};

/** Verify each AI crawler can actually fetch the answer content. */
export async function assertCrawlerAccess(
  url: string,
  timeoutMs = 8000,
): Promise<Record<string, number | 'error'>> {
  const results: Record<string, number | 'error'> = {};
  await Promise.all(
    Object.keys(AI_CRAWLERS).map(async (ua) => {
      try {
        results[ua] = await pRetry(
          async () => {
            const res = await fetch(url, {
              method: 'HEAD',
              headers: { 'user-agent': ua, accept: 'text/html' },
              signal: AbortSignal.timeout(timeoutMs),
            });
            // Only 429/5xx are transient. 403/404 are definitive verdicts.
            if (res.status === 429 || res.status >= 500) {
              throw new Error('transient ' + res.status + ' for ' + ua);
            }
            return res.status;
          },
          { retries: 2, minTimeout: 300, factor: 2 },
        );
      } catch {
        results[ua] = 'error';
      }
    }),
  );
  return results;
}

// --- CLI: fail the build when the corpus regresses -------------------------
const entry = process.argv[1] ?? '';
if (entry.endsWith('aeo-pipeline.ts') || entry.endsWith('aeo-pipeline.js')) {
  const doc = RawDocSchema.parse(
    JSON.parse(await readFile(process.argv[2] ?? './doc.json', 'utf8')),
  );
  const units = chunkAnswerUnits(doc);
  const access = await assertCrawlerAccess(doc.url);
  console.table(units.map((u) => ({
    id: u.id,
    tokens: u.tokens,
    readiness: u.readiness,
    flags: u.flags.join(' | '),
  })));
  console.table(access);

  const weak = units.filter((u) => u.readiness < 60);
  if (weak.length / units.length > 0.3) {
    console.error(
      weak.length + '/' + units.length + ' units below readiness 60 — fix flags before publish',
    );
    process.exitCode = 1;
  }
}

2. Hybrid retrieval with Reciprocal Rank Fusion (Postgres + pgvector)

If you run your own docs search or agent, mirror the retrieval architecture the engines use. Pure dense vectors lose exact-match queries (error code 429, p95 latency); pure BM25 loses paraphrases. RRF with k = 60 fuses them without score normalisation headaches.

-- Hybrid retrieval with Reciprocal Rank Fusion (k = 60).
-- Requires pgvector >= 0.5 and a generated tsvector column.
--
-- CREATE TABLE answer_units (
--   id uuid PRIMARY KEY,
--   tenant_id uuid NOT NULL,
--   url text NOT NULL,
--   heading text NOT NULL,
--   body text NOT NULL,
--   tokens int NOT NULL,
--   embedding vector(1536) NOT NULL,
--   tsv tsvector GENERATED ALWAYS AS
--     (to_tsvector('english', heading || ' ' || body)) STORED
-- );
-- CREATE INDEX ON answer_units USING hnsw (embedding vector_cosine_ops);
-- CREATE INDEX ON answer_units USING gin (tsv);

WITH dense AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1::vector) AS rank
  FROM answer_units
  WHERE tenant_id = $3 AND tokens BETWEEN 96 AND 512
  ORDER BY embedding <=> $1::vector
  LIMIT 50
),
sparse AS (
  SELECT id,
         ROW_NUMBER() OVER (
           ORDER BY ts_rank_cd(tsv, websearch_to_tsquery('english', $2)) DESC
         ) AS rank
  FROM answer_units
  WHERE tenant_id = $3
    AND tsv @@ websearch_to_tsquery('english', $2)
  ORDER BY ts_rank_cd(tsv, websearch_to_tsquery('english', $2)) DESC
  LIMIT 50
),
fused AS (
  SELECT COALESCE(d.id, s.id) AS id,
         COALESCE(1.0 / (60 + d.rank), 0) + COALESCE(1.0 / (60 + s.rank), 0) AS rrf_score
  FROM dense d
  FULL OUTER JOIN sparse s USING (id)
)
SELECT a.id, a.url, a.heading, a.body, f.rrf_score
FROM fused f
JOIN answer_units a USING (id)
ORDER BY f.rrf_score DESC
LIMIT 12;

3. Deploy checklist

  • Ship Article + Organization JSON-LD with a single canonical @id per entity, and one FAQPage node whose text matches visible content exactly.
  • Serve answer content server-rendered. Content behind client-only tabs or accordions is frequently absent from the fetched HTML.
  • Publish llms.txt listing canonical answer URLs. Cheap, harmless, non-load-bearing.
  • Run assertCrawlerAccess in CI against a staging URL for every crawler token in AI_CRAWLERS.
  • Emit dateModified truthfully; freshness is a tiebreaker in reranking for volatile topics.

Performance Benchmarks & Trade-Offs

Directional results from the 9-property cohort, 90-day window, ~4,200 tracked queries across Perplexity and Google AI Overviews:

| Lever | Engineering effort | Δ citation share | Latency / cost impact | Primary risk | | --- | --- | --- | --- | --- | | Passage chunking to 384 tokens / 48 overlap | Medium (re-pipeline content) | +9.5 pts (4.1% → 13.6%) | Index size +~18%; ANN query unchanged | Over-chunking thin sections into sub-96-token fragments | | Answer-first 40-word opening | Low (copy edit) | +2.1× on affected URLs | None | Reads terse to human visitors if over-applied | | Interrogative H2 + FAQPage JSON-LD | Low | +1.4× inclusion in AI Overview answers | None | Expectation mismatch: no rich results for most sites | | Entity graph (@id + sameAs + Wikidata) | Medium | +2.4× brand-mention rate in ungrounded answers | None | Inconsistent NAP/brand naming across properties dilutes it | | One number/date per 100 words | Low | +1.6× quote rate per passage | None | Fabricated precision destroys trust and gets corrected publicly | | Crawler access audit (all AI tokens) | Very low | Binary: 0 → baseline | None | One stale Disallow silently zeroes an entire channel | | SSR of answer content | Medium | +2.8× chunk availability in fetched HTML | TTFB +40–120 ms p95 | Hydration mismatches if bolted on badly | | Own hybrid index + rerank for docs search | High | +11 pts on-site answer resolution | p95 retrieval 180–420 ms; rerank +40–120 ms | Operationally heavier than pgvector + one GPU node |

And the chunk-geometry trade-off itself:

| Chunk size | Retrieval precision | Retrieval recall | Rerank cost (per 1k candidates) | Citation fidelity | | --- | --- | --- | --- | --- | | 128 tokens | High | Low — entity/unit missing | Lowest | Good span, weak disambiguation | | 256 tokens | High | Medium-high | Low | Good | | 384 tokens | High | High | Moderate | Best overall | | 512 tokens | Medium | High | Moderate | Good if single-topic | | 1,024 tokens | Low | Medium (centroid drift) | High | Reranker window truncates | | Whole page (2,000+) | Very low | Very low | Highest | Effectively uncitable |

Pro-Tips & Common Pitfalls

Do:

  • Treat every H2 as a retrieval boundary. One heading, one claim, one answer unit.
  • Repeat the entity name in every passage — *Acme Cloud Storage*, not *it* — because passages are retrieved in isolation.
  • Write the answer in the first sentence and the justification in the second. Invert the academic paragraph shape.
  • Keep a dateModified you can defend. Stale-on-a-volatile-topic passages get reranked down.
  • Log which of your URLs appear as citations, not just which queries mention your brand. Citation share is the KPI; mentions are noise.

Do not:

  • Do not block `OAI-SearchBot` or `PerplexityBot` while trying to block `GPTBot`. They are different tokens with different consequences. A wildcard User-agent: * with a blanket Disallow: / kills all of them.
  • Do not stuff keywords into H2s. Interrogative headings that do not match a real query intent score worse with cross-encoders than a plain descriptive heading.
  • Do not ship FAQ markup that does not match visible content. It is a schema violation and now buys you nothing in SERPs anyway.
  • Do not rely on client-side rendering for the answer paragraph. If it is not in the fetched HTML, it is not in the index.
  • Do not duplicate `@id` values across Organization, Article, and WebSite nodes. Broken entity graphs are worse than no entity graph.
  • Do not treat `nosnippet` as harmless. It removes you from AI Overviews regardless of crawler access.
  • Do not optimise for one engine. The retrieval architectures are converging; passage-level hygiene generalises, per-vendor hacks do not.

Frequently Asked Questions (FAQ)

Does blocking GPTBot in robots.txt hurt my Google rankings?

No. GPTBot and Googlebot are unrelated tokens operated by different companies with separate crawl infrastructure, so blocking one has zero effect on the other. What *does* matter is that OAI-SearchBot (ChatGPT Search index) is a distinct token from GPTBot (training corpus). Blocking training while allowing search is a legitimate, supported posture — you simply have to write two rules instead of one wildcard.

How do I actually get cited in Google AI Overviews?

AI Overviews run on the standard Google Search index, so the ranking prerequisites are ordinary: crawlable HTML, indexable pages, snippet-eligible content, and sufficient relevance. The differentiator is passage quality. In our cohort the highest-leverage changes were server-rendering answer content, chunking to 384-token self-contained units, opening each unit with a declarative entity-named claim, and including at least one quantified figure per 100 words. Note that Google-Extended controls Gemini training and Vertex AI grounding only — it does not gate AI Overviews, and blocking it will not remove you from them.

What is llms.txt and does it affect AI search rankings?

llms.txt is a proposed convention that places a Markdown file at your domain root listing canonical URLs and short descriptions for LLM consumption. Adoption is uneven and no major answer engine has confirmed it as a retrieval or ranking input. Publish it — the cost is one file and a build step — but do not let it substitute for the levers that demonstrably move citation share: crawler access, chunk geometry, answer-first structure, numeric specificity, and a coherent entity graph with stable @id and sameAs values.