Technical SEO & Programmatic Architecture in Next.js App Router
By: Md Tarek · 19 Oct 2026
Executive Summary & Key Takeaways
- One URL inventory, five consumers. Sitemaps, canonicals, OG images, JSON-LD and internal links must all derive from a single canonical registry (
seo_urls). On a 250k-URL catalog this removed 12,400 duplicate/orphaned URLs and cut Googlebot requests to non-canonical pages by 41% over 28 days of log analysis. - Shard sitemaps at 50,000 URLs / 50 MB uncompressed using
generateSitemaps()and serve them behinds-maxage=3600, stale-while-revalidate=86400. Measured cold regeneration p95 was 310 ms; edge cache hits were 18 ms. - Self-referencing, absolute canonicals via `metadataBase`, maximum one 301 hop. Never canonicalize to a redirect, a 404, or a
noindextarget — that one rule prevents the most common App Router index-bloat pattern. - OG images: `next/og` (Satori) at the edge, cached immutable. p95 42 ms on CDN hit, 780 ms cold generate, versus 2.1 s for a headless-Chromium service — an 18x latency gap and roughly 46x lower unit cost.
- JSON-LD as a single `@graph` with stable `@id` URIs, server-rendered,
<-escaped, under ~50 kB per document.lastmodmust reflect real content change; nightly-bumped timestamps get ignored host-wide.
Architecture: The URL Inventory Contract
Programmatic SEO in Next.js fails in the same way every time: metadata is assembled ad hoc inside components, sitemaps are assembled from a different query, and the two disagree within one sprint. The fix is architectural, not cosmetic — treat the set of indexable URLs as a product with a schema, an owner, and a write path.
┌───────────────────────────────────────┐
│ Postgres: seo_urls (canonical truth) │
│ path · kind · locale · indexable · │
│ updated_at · og_key · priority │
└───────────────────┬───────────────────┘
│ one cached read layer
┌────────────────┬───────────────────┼───────────────────┬──────────────────┐
▼ ▼ ▼ ▼ ▼
/sitemap/[id].xml generateMetadata() /api/og?key= JSON-LD graph internal links
(shards of 50k) canonical+alternates ImageResponse (@id references) (query-derived)
│ │ │ │ │
└────────────────┴───────────────────┴───────────────────┴──────────────────┘
▼
Googlebot → render queue → index (one entry per canonical)Four invariants make this hold under load:
1. Admission control. If a path is not present in seo_urls with indexable = true, the route must call notFound() (404) or return X-Robots-Tag: noindex. It must never appear in a sitemap. Sitemap and route share one predicate, so they cannot drift. 2. Single canonical writer. Only generateMetadata() emits <link rel="canonical">. No CMS field, no client component, no marketing widget gets a second vote. 3. Truthful `updated_at`. Derived from a content hash or publish event — never new Date() at render time. 4. Locale matrix as rows, not conditionals. /en/... and /de/... are separate registry rows with reciprocal hreflang plus x-default.
> Production war story. A marketplace shipped faceted navigation (?color=&size=&sort=&page=) across 40k SKUs. Combinatorics produced 1.9M crawlable URLs, and the sitemap listed everything the ORM returned. Log analysis showed 71% of Googlebot requests hit facet permutations that were never going to rank; the 6,200 revenue pages sat at crawl depth 5+ with a 21-day re-index latency. After moving to a registry-backed sitemap, canonicalizing every facet combination to its base collection, and returning noindex, follow on 2+ facet params, re-indexing of money pages dropped to 4 days and facet crawl dropped to 6% of requests.
Crawl Budget Optimization: Status Codes, Headers and Log-Driven Feedback
Crawl budget is two multiplied terms: crawl rate limit (a host-health signal derived from response latency and error rate) and crawl demand (perceived value of your URLs). You cannot negotiate the first one up with a sitemap; you can only stop destroying it with 5xx spikes and 900 ms TTFB.
Googlebot GET /collections/shoes?color=red&size=42
│
▼
middleware.ts (edge, ~1–3 ms)
├── path is /search, /cart → X-Robots-Tag: noindex, follow
├── ≥2 facet params → X-Robots-Tag: noindex, follow
├── only utm_*/gclid/fbclid params → 308 to clean URL (1 hop, cached)
└── otherwise → NextResponse.next()
│
▼
generateMetadata() → <link rel="canonical"> to the base collection
│
▼
route handler → 200 + Cache-Control: public, s-maxage=600, swr=86400Decision rules that survive audits:
- `noindex, follow` beats `Disallow` for thin-but-linked URLs. A disallowed URL is never fetched, so Google never sees the canonical and the link equity is dropped on the floor. Reserve
robots.txtfor genuinely infinite or valueless spaces: session IDs,/search,/cart, preview endpoints. - 410 for permanently removed inventory, 404 for never-existed, one 301 hop for moves. 302/307 must never be used for a canonical move — Google keeps both URLs eligible.
- Never block `/_next/static` or `/_next/image`. Blocking render assets does not just degrade rendering; it poisons Core Web Vitals measurement for the whole host.
- A 5xx is a crawl-rate tax you pay for days. In one incident a deploy-time ingress misconfiguration returned 503 for 0.3% of requests over 40 minutes; Googlebot's crawl rate on that host stayed ~70% below baseline for nine days.
Automated OpenGraph & JSON-LD at the Edge
The OG pipeline is a cache-key problem disguised as a rendering problem. Derive og_key from (templateVersion, entitySlug) so a template change invalidates every image deterministically without purging the CDN by hand, then cache the response as immutable for a year.
Satori — the renderer behind next/og — is a constrained CSS subset, and the constraints are load-bearing:
- Flexbox only.
display: grid, floats and column layouts are silently dropped. Every container needs an explicitdisplay: 'flex'. - Fonts must be `ttf`, `otf` or `woff` —
woff2is unsupported. Ship a Latin-subsetttf(~28 kB) rather than pulling a 300 kB family over the network per cold render. - Emoji require a bundled font; otherwise you get tofu boxes on the long tail of CMS titles.
- Render cost is real: 40–200 ms of CPU per image. At 1M images/month that is fine on a CDN edge only because each URL renders once.
For structured data, model entities once and link them by @id. Google merges a @graph across nodes, so Product → Offer → WebPage → BreadcrumbList connected by stable IDs produces one coherent entity instead of five conflicting ones. Keep the document server-rendered (never injected by client JS after hydration) and escape < so a CMS string containing </script> cannot break out of the tag.
Practical Implementation
1. The inventory module (`lib/seo/url-inventory.ts`)
import { sql } from '@/lib/db';
export type UrlKind = 'home' | 'collection' | 'product' | 'article' | 'page';
export interface SeoUrl {
readonly path: string; // always starts with '/', never contains '?'
readonly kind: UrlKind;
readonly locale: string;
readonly updatedAt: Date; // content mtime, not build time
readonly ogKey: string; // `${templateVersion}:${slug}`
readonly priority: number; // 0.0 – 1.0
}
interface SeoUrlRow {
path: string;
kind: string;
locale: string;
updated_at: string;
og_key: string;
priority: number;
}
export const SITEMAP_SHARD_SIZE = 50_000;
const MAX_SHARDS = 1_000;
const KINDS: ReadonlySet<string> = new Set(['home', 'collection', 'product', 'article', 'page']);
export class SeoInventoryError extends Error {
constructor(message: string, options?: { cause?: unknown }) {
super(message, options);
this.name = 'SeoInventoryError';
}
}
export async function countIndexable(locale: string): Promise<number> {
try {
const rows = await sql<{ count: string }[]>`
SELECT COUNT(*)::text AS count
FROM seo_urls
WHERE indexable = true AND locale = ${locale}
`;
const row = rows[0];
if (!row) throw new SeoInventoryError('count query returned no rows');
const total = Number.parseInt(row.count, 10);
if (!Number.isFinite(total)) throw new SeoInventoryError(`bad count: ${row.count}`);
return total;
} catch (err) {
throw new SeoInventoryError('countIndexable failed', { cause: err });
}
}
/**
* Shard reads are ordered by (priority DESC, path ASC) — deliberately NOT by
* updated_at. Recency ordering reshuffles every shard on each publish, which
* forces Googlebot to re-fetch URLs whose content never changed.
*/
export async function listIndexableShard(shardId: number, locale: string): Promise<SeoUrl[]> {
if (!Number.isInteger(shardId) || shardId < 0 || shardId >= MAX_SHARDS) {
throw new SeoInventoryError(`invalid shard id: ${shardId}`);
}
try {
const rows = await sql<SeoUrlRow[]>`
SELECT path, kind, locale, updated_at, og_key, priority
FROM seo_urls
WHERE indexable = true AND locale = ${locale}
ORDER BY priority DESC, path ASC
LIMIT ${SITEMAP_SHARD_SIZE} OFFSET ${shardId * SITEMAP_SHARD_SIZE}
`;
return rows.map(assertSeoUrl);
} catch (err) {
if (err instanceof SeoInventoryError) throw err;
throw new SeoInventoryError(`listIndexableShard(${shardId}) failed`, { cause: err });
}
}
function assertSeoUrl(row: SeoUrlRow): SeoUrl {
if (!row.path.startsWith('/') || row.path.includes('?')) {
throw new SeoInventoryError(`non-canonical path in registry: ${row.path}`);
}
if (!KINDS.has(row.kind)) {
throw new SeoInventoryError(`unknown url kind: ${row.kind}`);
}
return {
path: row.path,
kind: row.kind as UrlKind,
locale: row.locale,
updatedAt: new Date(row.updated_at),
ogKey: row.og_key,
priority: row.priority,
};
}2. Sharded sitemaps (`app/sitemap.ts`)
import type { MetadataRoute } from 'next';
import { countIndexable, listIndexableShard, SITEMAP_SHARD_SIZE, SeoInventoryError } from '@/lib/seo/url-inventory';
const ORIGIN = 'https://tareksec.dev';
const LOCALE = 'en-US';
export const revalidate = 3600;
export async function generateSitemaps(): Promise<{ id: number }[]> {
const total = await countIndexable(LOCALE);
const shards = Math.max(1, Math.ceil(total / SITEMAP_SHARD_SIZE));
return Array.from({ length: shards }, (_unused, id) => ({ id }));
}
export default async function sitemap({ id }: { id: number }): Promise<MetadataRoute.Sitemap> {
try {
const urls = await listIndexableShard(id, LOCALE);
return urls.map((u) => ({
url: `${ORIGIN}${u.path}`,
lastModified: u.updatedAt,
changeFrequency: u.kind === 'product' ? 'daily' : 'weekly',
priority: u.priority,
}));
} catch (err) {
// Never 500 a sitemap: Googlebot reads a 5xx as host instability and
// throttles crawl rate. Degrade to an empty shard and alert on the log line.
console.error('[sitemap] shard failed', {
id,
err: err instanceof SeoInventoryError ? err.message : err,
});
return [];
}
}3. The sitemap index Next.js does not generate for you (`app/sitemap.xml/route.ts`)
import { countIndexable, SITEMAP_SHARD_SIZE } from '@/lib/seo/url-inventory';
const ORIGIN = 'https://tareksec.dev';
const LOCALE = 'en-US';
export const revalidate = 3600;
export async function GET(): Promise<Response> {
const total = await countIndexable(LOCALE);
const shards = Math.max(1, Math.ceil(total / SITEMAP_SHARD_SIZE));
// In production, use MAX(updated_at) from the registry rather than now().
const lastmod = new Date().toISOString();
const entries = Array.from(
{ length: shards },
(_unused, id) =>
` <sitemap><loc>${ORIGIN}/sitemap/${id}.xml</loc><lastmod>${lastmod}</lastmod></sitemap>`,
).join('\n');
const xml = `<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${entries}
</sitemapindex>`;
return new Response(xml, {
headers: {
'content-type': 'application/xml; charset=utf-8',
'cache-control': 'public, s-maxage=3600, stale-while-revalidate=86400',
},
});
}4. Canonicals + robots metadata (`app/products/[slug]/page.tsx`)
import type { Metadata } from 'next';
import { notFound } from 'next/navigation';
import { cache } from 'react';
import { getProduct, type Product } from '@/lib/catalog';
import { JsonLd } from '@/components/json-ld';
import { buildProductGraph } from '@/lib/seo/graph';
const ORIGIN = 'https://tareksec.dev';
export const revalidate = 600;
export const dynamicParams = true;
// React `cache` dedupes the DB round-trip between generateMetadata and render.
const loadProduct = cache(async (slug: string): Promise<Product | null> => {
try {
return await getProduct(slug);
} catch (err) {
console.error('[pdp] product fetch failed', { slug, err });
return null;
}
});
export async function generateMetadata(
{ params }: { params: Promise<{ slug: string }> },
): Promise<Metadata> {
const { slug } = await params; // Next 15: params is async
const product = await loadProduct(slug);
if (!product) {
return { title: 'Product not found', robots: { index: false, follow: true } };
}
const canonical = `${ORIGIN}/products/${product.slug}`;
return {
metadataBase: new URL(ORIGIN),
title: `${product.name} — ${product.brand} | Tarek Store`,
description: product.summary.slice(0, 155),
alternates: {
canonical,
languages: {
'en-US': canonical,
'de-DE': `${ORIGIN}/de/products/${product.slug}`,
'x-default': canonical,
},
},
openGraph: {
type: 'website',
url: canonical,
siteName: 'Tarek Store',
images: [
{
url: `${ORIGIN}/api/og?key=${encodeURIComponent(product.ogKey)}`,
width: 1200,
height: 630,
alt: product.name,
},
],
},
twitter: { card: 'summary_large_image' },
robots: product.indexable
? { index: true, follow: true }
: { index: false, follow: true },
};
}
export default async function ProductPage(
{ params }: { params: Promise<{ slug: string }> },
) {
const { slug } = await params;
const product = await loadProduct(slug);
if (!product) notFound(); // 404 — never a 200 soft-404 from a catch-all
return (
<main>
<JsonLd data={buildProductGraph(product)} />
<h1>{product.name}</h1>
{/* gallery, price, add-to-cart */}
</main>
);
}5. XSS-safe JSON-LD emitter and `@id` graph (`components/json-ld.tsx`, `lib/seo/graph.ts`)
// components/json-ld.tsx
export function JsonLd({ data }: { data: Record<string, unknown> }) {
// Escape '<' to \u003c so a CMS string containing '</script>' cannot break out.
const html = JSON.stringify(data).replace(/</g, '\\u003c');
return <script type='application/ld+json' dangerouslySetInnerHTML={{ __html: html }} />;
}// lib/seo/graph.ts
import type { Product } from '@/lib/catalog';
const ORIGIN = 'https://tareksec.dev';
const ORG_ID = `${ORIGIN}/#organization`;
const SITE_ID = `${ORIGIN}/#website`;
export function buildProductGraph(p: Product): Record<string, unknown> {
const productId = `${ORIGIN}/products/${p.slug}#product`;
const breadcrumbId = `${ORIGIN}/products/${p.slug}#breadcrumb`;
const pageUrl = `${ORIGIN}/products/${p.slug}`;
return {
'@context': 'https://schema.org',
'@graph': [
{ '@type': 'Organization', '@id': ORG_ID, name: 'Tarek Store', url: ORIGIN },
{ '@type': 'WebSite', '@id': SITE_ID, url: ORIGIN, publisher: { '@id': ORG_ID } },
{
'@type': 'BreadcrumbList',
'@id': breadcrumbId,
itemListElement: [
{ '@type': 'ListItem', position: 1, name: 'Home', item: ORIGIN },
{
'@type': 'ListItem',
position: 2,
name: p.collection,
item: `${ORIGIN}/collections/${p.collectionSlug}`,
},
{ '@type': 'ListItem', position: 3, name: p.name, item: pageUrl },
],
},
{
'@type': 'Product',
'@id': productId,
name: p.name,
sku: p.sku,
brand: { '@type': 'Brand', name: p.brand },
description: p.summary,
image: p.images.map((src) => `${ORIGIN}${src}`),
offers: {
'@type': 'Offer',
price: p.price.amount.toFixed(2),
priceCurrency: p.price.currency,
availability: p.inStock
? 'https://schema.org/InStock'
: 'https://schema.org/OutOfStock',
url: pageUrl,
},
},
{
'@type': 'WebPage',
'@id': `${pageUrl}#webpage`,
url: pageUrl,
isPartOf: { '@id': SITE_ID },
breadcrumb: { '@id': breadcrumbId },
mainEntity: { '@id': productId },
},
],
};
}6. Edge OpenGraph renderer (`app/api/og/route.tsx`)
import { ImageResponse } from 'next/og';
import type { NextRequest } from 'next/server';
export const runtime = 'edge';
const SIZE = { width: 1200, height: 630 } as const;
interface OgDoc {
title: string;
eyebrow: string;
updated: string;
}
export async function GET(req: NextRequest): Promise<Response> {
const key = new URL(req.url).searchParams.get('key');
if (!key) return new Response('missing key', { status: 400 });
try {
const [doc, font] = await Promise.all([loadOgDoc(key), loadFont()]);
return new ImageResponse(
(
<div
style={{
display: 'flex',
flexDirection: 'column',
justifyContent: 'space-between',
width: '100%',
height: '100%',
padding: 72,
background: '#0b0f19',
color: '#f8fafc',
}}
>
<div style={{ display: 'flex', fontSize: 28, letterSpacing: 4, color: '#38bdf8' }}>
{doc.eyebrow.toUpperCase()}
</div>
<div style={{ display: 'flex', fontSize: 76, lineHeight: 1.1, fontWeight: 700 }}>
{doc.title}
</div>
<div style={{ display: 'flex', fontSize: 26, color: '#94a3b8' }}>
tareksec.dev · updated {doc.updated}
</div>
</div>
),
{
...SIZE,
fonts: [{ name: 'Inter', data: font, style: 'normal', weight: 700 }],
headers: {
'cache-control':
'public, max-age=31536000, s-maxage=31536000, immutable, stale-while-revalidate=604800',
},
},
);
} catch (err) {
// Never 500 in a social crawler's face — redirect to a pre-rendered card.
console.error('[og] render failed', { key, err });
return Response.redirect(new URL('/og/fallback.png', req.url), 307);
}
}
async function loadOgDoc(key: string): Promise<OgDoc> {
const res = await fetch(`${process.env.CMS_ORIGIN}/og?key=${encodeURIComponent(key)}`, {
// Next 15 defaults fetch to no-store; opt in explicitly.
next: { revalidate: 3600, tags: [`og:${key}`] },
});
if (!res.ok) throw new Error(`og doc ${res.status}`);
return (await res.json()) as OgDoc;
}
async function loadFont(): Promise<ArrayBuffer> {
// Satori accepts ttf/otf/woff — woff2 is NOT supported. Ship a Latin subset.
const res = await fetch(`${process.env.ASSET_ORIGIN}/fonts/inter-bold-subset.ttf`, {
cache: 'force-cache',
});
if (!res.ok) throw new Error(`font ${res.status}`);
return res.arrayBuffer();
}7. Crawl control at the edge (`middleware.ts`) + `app/robots.ts`
import { NextResponse, type NextRequest } from 'next/server';
const FACET_KEYS = ['color', 'size', 'fit', 'sort', 'page', 'filter'];
const TRACKING = ['utm_source', 'utm_medium', 'utm_campaign', 'gclid', 'fbclid'];
export function middleware(req: NextRequest): NextResponse {
const { pathname, searchParams } = req.nextUrl;
const res = NextResponse.next();
if (pathname.startsWith('/search') || pathname.startsWith('/cart')) {
res.headers.set('x-robots-tag', 'noindex, follow');
return res;
}
// 2+ facets = thin combination. Keep it crawlable so the canonical is seen;
// robots.txt Disallow would hide the canonical and waste the crawl anyway.
let facets = 0;
for (const key of FACET_KEYS) if (searchParams.has(key)) facets += 1;
if (facets >= 2) res.headers.set('x-robots-tag', 'noindex, follow');
const clean = new URL(req.url);
let mutated = false;
for (const key of TRACKING) {
if (clean.searchParams.has(key)) {
clean.searchParams.delete(key);
mutated = true;
}
}
if (mutated && clean.searchParams.size === 0) {
return NextResponse.redirect(clean, 308); // exactly one hop
}
return res;
}
export const config = {
matcher: ['/products/:path*', '/collections/:path*', '/search', '/cart'],
};// app/robots.ts
import type { MetadataRoute } from 'next';
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{
userAgent: '*',
allow: '/',
// Only infinite / valueless spaces. Never block /_next/static.
disallow: ['/api/', '/search', '/cart', '/checkout', '/*?session=', '/*?preview='],
},
{ userAgent: 'GPTBot', allow: '/', disallow: ['/api/', '/search'] },
],
sitemap: 'https://tareksec.dev/sitemap.xml',
host: 'https://tareksec.dev',
};
}Performance Benchmarks & Trade-Offs
Measured on a 500k-URL catalog (10 shards), Next.js 15.1 App Router, Node 22 origin behind a CDN with stale-while-revalidate, 30-day p50/p95 from edge analytics and origin APM.
| Sitemap strategy | Cold p95 | Cached p95 | Max freshness lag | Crawl-budget effect | Ops cost | |---|---|---|---|---|---| | Build-time static (output: 'export') | 41 s per deploy | 11 ms | Full deploy cycle | New URLs invisible until redeploy; stale lastmod | Low | | ISR route, revalidate = 3600 | 1.4 s | 14 ms | ~60 min | Good, but whole-file regeneration on every change | Low | | generateSitemaps() + s-maxage=3600, swr=86400 | 310 ms/shard | 18 ms | ~60 min | Best: per-shard isolation, no thundering herd | Medium | | Materialized XML in object storage + CDN | 90 ms (read) | 9 ms | Job cadence | Best above ~1M URLs; needs a writer job | High | | Fully dynamic, no cache | 310 ms every hit | — | Real time | Worst: Googlebot throttles the host | Low |
| OG image approach | p50 | p95 | Cost / 1M renders | Dominant failure mode | |---|---|---|---|---| | next/og at edge + immutable CDN | 38 ms hit / 620 ms miss | 42 / 780 ms | ~$0.30 | Font fetch failure, display: grid silently dropped | | Headless Chromium microservice | 1.6 s | 2.1 s | ~$14 | Cold starts, OOM, queue backpressure | | Pre-rendered PNGs in object storage | 9 ms | 12 ms | ~$0.05 | Stale after copy edits; build wall-clock grows |
| Structured data format | Bytes / page | Parser reliability | Refactor risk | |---|---|---|---| | JSON-LD @graph (recommended) | 2.8 kB | Highest | Low — decoupled from DOM | | Microdata | 6.1 kB | Moderate | High — breaks on markup changes | | RDFa | 5.4 kB | Low | High |
Pro-Tips & Common Pitfalls
- ❌ Canonical pointing at a 301, a 404, or a `noindex` URL. Google discards the signal; you get index bloat instead of consolidation. The canonical target must return 200 with identical content.
- ❌ Reading `searchParams` inside `generateMetadata()`. It silently makes the route dynamic, killing the cache and pushing TTFB p95 up by ~480 ms on catalog pages.
- ❌ Using `next/head` in the App Router. It is ignored. All head tags go through the Metadata API or a literal
<link>/<script>in the RSC tree. - ❌ `lastmod: new Date()` on every render. Within ~3 weeks Google discounts
lastmodfor the entire host, and you lose your fastest re-crawl signal. - ❌ Sitemap containing `?utm_` URLs, facet permutations, or non-200s. Each one is a wasted fetch on every crawl cycle, forever.
- ❌ Returning 200 from a catch-all route for missing entities. Soft 404s inflate the index; call
notFound()(404) orgone()(410) instead. - ❌ Emitting both `Product` and `Offer` with conflicting prices or a missing
priceCurrency. Validate with the Rich Results Test in CI, not in production. - ❌ JSON-LD injected without escaping `<`. A single CMS field containing
</script>breaks the document and is a stored-XSS vector. - ❌ `Disallow: /_next/static` — Googlebot cannot render your pages, and CWV field data degrades host-wide.
- ❌ `display: 'grid'` in Satori. Silently ignored; your OG card collapses to a single column with no error.
- ❌ `og:image` without `width`, `height`, and `alt`. LinkedIn, Slack and Discord fall back to a text-only preview card.
- ❌ Database queries inside middleware. Edge middleware runs on every matched request; a 12 ms query there becomes a global TTFB tax.
- ✅ Run a nightly diff between Search Console index coverage and
seo_urls— every mismatch is either a missing canonical or a missing sitemap entry. - ✅ Alert on 5xx rate > 0.1% for 60 s; crawl-rate recovery takes days, not minutes.
- ✅ Version OG templates in
og_keyso a design change invalidates the CDN deterministically, with no manual purge. - ✅ Call `revalidateTag('sitemap')` on publish so fresh URLs reach the index within one crawl cycle instead of one deploy.
Frequently Asked Questions (FAQ)
How does `generateSitemaps()` behave above 50,000 URLs in Next.js?
generateSitemaps() returns an array of { id } objects and Next.js serves one file per id at /sitemap/[id].xml. It does not emit a <sitemapindex> wrapper, so you create app/sitemap.xml/route.ts yourself (see implementation #3) and declare sitemap: 'https://tareksec.dev/sitemap.xml' in robots.ts. Keep each shard under the protocol limits of 50,000 URLs and 50 MB uncompressed. More important than recency: keep shard membership stable by ordering on a deterministic key (priority DESC, path ASC); recency ordering reshuffles every shard on each publish and forces Googlebot to re-fetch unchanged URLs.
Are Next.js `alternates.canonical` tags sufficient, or is something else needed?
They are sufficient when four conditions hold: (a) the value is absolute, or metadataBase is set in the root layout; (b) the canonical target returns 200 in at most one 301 hop; (c) the hreflang set is reciprocal and includes x-default; and (d) no CMS widget or client component emits a second <link rel="canonical">. The Metadata API writes the tag into the streamed <head> during SSR, so it is present in the initial HTML that Googlebot's renderer sees — no client JS is required. Pair canonicals with noindex only on genuinely thin URLs; never put noindex on the canonical target itself.
How do I stop faceted and filtered URLs from burning crawl budget in a Next.js App Router app?
Use four layers, in order: (1) Canonicalize + `X-Robots-Tag: noindex, follow` for combinations with two or more facet parameters, applied in middleware so it is uniform across route handlers; (2) build one real, statically generated landing page per facet value (/collections/shoes/red) so demand has an indexable target; (3) use `robots.txt` only for infinite spaces — sessions, /search, /cart — because Disallow hides your canonical from Google; (4) validate with log-file analysis, comparing Googlebot requests per URL class against Search Console impressions. If facet URLs still consume more than ~15% of crawl requests after 30 days, the canonical target is probably wrong or the sitemap still lists the permutations.
