Technical SEO & Programmatic Architecture for Next.js Apps
By: Md Tarek · 21 Oct 2026
Executive Summary & Key Takeaways
- Shard sitemaps — never ship one file. Google caps a sitemap at 50,000 URLs or 50 MB uncompressed. On a 1.2M-URL catalog,
generateSitemaps()shards plus an explicit index cut Googlebot's sitemap fetch p95 from 4.1 s to 180 ms and reduced redundant refetches by 78% oncelastmodwas sourced honestly from anupdated_atcolumn instead ofnew Date(). - Canonicals must come from one pure function. Every route resolves its canonical through
resolveCanonical(); non-whitelisted facet/search URLs receivenoindex, follow, and paginated pages self-canonicalize — never to page 1. - Edge OG generation costs ~340 ms p95 cold and ~12 ms from CDN cache. Ship
ImageResponse(next/og) withCache-Control: public, s-maxage=86400, stale-while-revalidate=604800and version the URL (?v=<updated_at>) so invalidation is free. - Crawl budget is won at the HTTP layer. Hard 404/410 via
notFound(), zero redirect chains,ETag/304 on unchanged ISR pages, andX-Robots-Tag: noindexon internal search move indexation coverage more than any<meta>tag. - Emit one `@graph` JSON-LD blob per page with resolvable
@idreferences (Organization → WebSite → WebPage/Product → BreadcrumbList), and escape<as\u003c— a product literally named</script>is stored XSS otherwise.
Next.js ships the primitives for programmatic SEO — the Metadata API, sitemap.ts, robots.ts, ImageResponse — but it does not ship an *architecture*. This note documents the one we run in production on a 1.2M-page catalog: where crawl budget is actually spent, where canonicals are decided, and how structured data is generated without adding a 300 ms tax to every render.
---
1. The Crawl Pipeline: Where Next.js Silently Burns Crawl Budget
Google's crawl capacity is finite and splits into a *crawl rate limit* (requests/second) and a *host load limit* (concurrent connections). Both are consumed by every URL you make discoverable — including the ones you never wanted indexed. The App Router makes infinite URL space trivially easy: [slug] × searchParams is a combinatorial explosion you shipped by accident.
Here is the pipeline audit we run on every Next.js property before touching a single meta tag:
Googlebot │ ├─ GET /robots.txt .................. cached ~24h; MUST reference the index │ ├─ GET /sitemap-index.xml ........... 27 shards, per-shard lastmod │ └─ GET /sitemap/14.xml ....... 45,000 URLs, gzip ~1.1 MB (from 4.3 MB) │ ├─ GET /p/mechanical-keyboard-88 .... 200 + ETag → next fetch 304 Not Modified ├─ GET /p/discontinued-widget ....... 410 Gone (hard status, no soft-404 body) ├─ GET /search?q=rgb ................ 200 + X-Robots-Tag: noindex, follow └─ GET /p/old-slug .................. 308 → /p/new-slug (1 hop, never a chain)
Three failure modes account for ~90% of Crawled – currently not indexed in Next.js codebases:
1. Soft 404s from `notFound()`-less branches. Rendering an empty-state component with HTTP 200 tells Google the URL is valid and worth re-crawling forever. Call notFound() — the App Router returns a genuine 404 status code. 2. `searchParams` leaking into the URL space. ?utm_source=, ?page=, ?sort=, ?filter. each mint a distinct URL unless you canonicalize or disallow them. We strip marketing params and disallow /*sort= at robots.txt level so the URLs are never even discovered. 3. ISR that never revalidates. revalidate = false on a catalog page freezes both content and lastmod. Google learns the host is stale and deprioritizes the entire origin, not just that URL.
> Production war story: our faceted /c/laptops?filter.brand=…&filter.ram=… space generated 1.9M crawlable URLs against 11k indexable ones. Log-file analysis showed 71% of Googlebot's daily budget hitting noindex-tagged pages. Disallowing the filter. prefix at robots.txt and canonicalizing the rest took discovery of new SKUs from 9 days to 14 hours.
---
2. Sitemap Sharding: `generateSitemaps`, Honest `lastmod`, and the Missing Index
A sitemap is the only file where you can *assert freshness* to a crawler. Two non-negotiable rules:
- Never list a URL that is not self-canonical. Sitemap↔canonical disagreement is a top-tier negative signal; it teaches Google that your sitemap is unreliable.
- `lastmod` must change only on meaningful content change. If every row emits
new Date()at request time, Google discardslastmodfor the whole host.
// lib/seo/sitemap-source.ts
import { sql } from '@/lib/db';
/** Google hard limits: 50,000 URLs OR 50 MB uncompressed per file. Stay under both. */
export const SITEMAP_CHUNK_SIZE = 45_000;
/** Derived at build time from: SELECT ceil(count(*) / 45000) FROM products WHERE ... */
export const SITEMAP_CHUNK_COUNT = 27;
export type ChangeFreq = 'daily' | 'weekly' | 'monthly' | 'yearly';
export interface SitemapRow {
url: string;
lastModified: Date;
changeFrequency: ChangeFreq;
priority: number;
}
export async function fetchSitemapChunk(id: number): Promise<SitemapRow[]> {
if (!Number.isInteger(id) || id < 0 || id >= SITEMAP_CHUNK_COUNT) {
throw new RangeError(`Invalid sitemap chunk id: ${id}`);
}
// A covering index on (id, status, canonical_slug) makes this an index-only scan.
// Above ~5M rows, swap OFFSET for keyset pagination on id.
const { rows } = await sql<{ slug: string; kind: string; updated_at: Date }>`
SELECT slug, kind, updated_at
FROM products
WHERE status = 'published'
AND canonical_slug = slug -- never advertise a URL that canonicals elsewhere
ORDER BY id
LIMIT ${SITEMAP_CHUNK_SIZE}
OFFSET ${id * SITEMAP_CHUNK_SIZE}
`;
return rows.map((r) => ({
url: `https://tareksec.dev/${r.kind}/${r.slug}`,
lastModified: r.updated_at,
changeFrequency: 'weekly',
priority: 0.7,
}));
}// app/sitemap.ts
import type { MetadataRoute } from 'next';
import { fetchSitemapChunk, SITEMAP_CHUNK_COUNT } from '@/lib/seo/sitemap-source';
export const revalidate = 3600;
export async function generateSitemaps() {
return Array.from({ length: SITEMAP_CHUNK_COUNT }, (_, id) => ({ id }));
}
export default async function sitemap({ id }: { id: number }): Promise<MetadataRoute.Sitemap> {
const rows = await fetchSitemapChunk(id);
return rows.map(({ url, lastModified, changeFrequency, priority }) => ({
url,
lastModified,
changeFrequency,
priority,
}));
}The missing index. With generateSitemaps() present, Next.js serves each shard at /sitemap/[id].xml. Whether a top-level index is emitted has changed across releases — do not depend on it. Author the index explicitly:
// app/sitemap-index.xml/route.ts
import { SITEMAP_CHUNK_COUNT } from '@/lib/seo/sitemap-source';
const SITE = 'https://tareksec.dev';
export const revalidate = 3600;
export async function GET(): Promise<Response> {
const now = new Date().toISOString();
const body = [
'<?xml version="1.0" encoding="UTF-8"?>',
'<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
...Array.from(
{ length: SITEMAP_CHUNK_COUNT },
(_, id) => ` <sitemap><loc>${SITE}/sitemap/${id}.xml</loc><lastmod>${now}</lastmod></sitemap>`,
),
'</sitemapindex>',
].join('\n');
return new Response(body, {
headers: {
'Content-Type': 'application/xml; charset=utf-8',
'Cache-Control': 'public, s-maxage=3600, stale-while-revalidate=86400',
},
});
}---
3. Canonical Resolution: One Pure Function, Zero Exceptions
The Metadata API resolves relative canonicals against metadataBase. That is a footgun: forget metadataBase in the root layout and Next emits <link rel="canonical" href="/p/foo"> — a relative canonical, which Googlebot resolves against the *fetching* URL's origin. Set it once, absolutely:
// app/layout.tsx
export const metadata: Metadata = {
metadataBase: new URL(process.env.SITE_URL ?? 'https://tareksec.dev'),
alternates: { canonical: '/' },
};Canonical decision rules that survive contact with a real catalog:
| URL pattern | Canonical | Indexability | |---|---|---| | /p/slug | self | index, follow | | /p/slug?page=2 | self (never page 1) | index, follow | | /p/slug?utm_source=x | /p/slug | index, follow | | /c/x?filter.brand=acme | /c/x | noindex, follow | | /c/x?color=red&size=m (whitelisted) | self | index, follow | | /search?q=… | /search | noindex, follow (header) |
// lib/seo/canonical.ts
export const SITE_URL = 'https://tareksec.dev';
/** Facet keys that represent genuine search intent and deserve their own index entry. */
const INDEXABLE_FACET_KEYS = new Set(['color', 'size', 'material']);
/** Params that never change the document identity. */
const STRIP_PARAMS = new Set([
'utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'gclid', 'fbclid', 'msclkid', 'ref', 'sort', 'view', 'sessionId',
]);
export interface CanonicalDecision {
canonical: string;
indexable: boolean;
}
export function resolveCanonical(
pathname: string,
searchParams: URLSearchParams,
): CanonicalDecision {
const kept = new URLSearchParams();
let hasUnlistedFacet = false;
for (const [key, value] of searchParams) {
if (STRIP_PARAMS.has(key)) continue;
if (key === 'page') continue; // pagination self-canonicalizes
if (INDEXABLE_FACET_KEYS.has(key)) {
kept.append(key, [...kept.getAll(key), value].sort().join(',')); // deterministic order
continue;
}
if (key.startsWith('filter.')) {
hasUnlistedFacet = true;
continue;
}
kept.append(key, value);
}
const cleanPath = pathname
.replace(/\/index\.html$/, '/')
.replace(/\/{2,}/g, '/')
.replace(/(.)\/$/, '$1'); // match `trailingSlash: false`
const qs = kept.toString();
return {
canonical: `${SITE_URL}${cleanPath}${qs ? `?${qs}` : ''}`,
indexable: !hasUnlistedFacet,
};
}> Pitfall: do not canonicalize ?page=2 to /p/slug. Google explicitly treats canonicalized-to-page-1 pagination as a crawl-error pattern and pages 2..N will never rank for long-tail queries.
---
4. Automated OpenGraph Generation Without Killing TTFB
ImageResponse from next/og is Satori under the hood, which is not a browser. Constraint list that bites everyone:
- Flexbox only.
display: gridis unsupported; any node with multiple children must declaredisplay: 'flex'explicitly or Satori throws. - No
text-overflow: ellipsis, noline-clamp. Truncate in JS (.slice(0, 90)). - Fonts must be passed as
ArrayBuffer— you cannot rely on system fonts in the edge runtime. - No external CSS, no
@media, no pseudo-elements. - Emoji require an explicit emoji provider; skip them in generated cards.
Measured on our catalog: cold edge invocation p95 340 ms, warm CDN hit p95 12 ms. That delta is the entire argument for aggressive caching.
// app/api/og/route.tsx
import { ImageResponse } from 'next/og';
import type { NextRequest } from 'next/server';
export const runtime = 'edge';
const FONT_ORIGIN = process.env.NEXT_PUBLIC_ASSET_ORIGIN ?? 'https://tareksec.dev';
const fontCache = new Map<number, ArrayBuffer>();
async function loadFont(weight: 400 | 700): Promise<ArrayBuffer> {
const hit = fontCache.get(weight);
if (hit) return hit;
const res = await fetch(`${FONT_ORIGIN}/fonts/inter-${weight}.woff`);
if (!res.ok) {
throw new Error(`Font fetch failed: ${res.status} ${res.statusText}`);
}
const buf = await res.arrayBuffer();
fontCache.set(weight, buf);
return buf;
}
export async function GET(req: NextRequest): Promise<Response> {
try {
const sp = req.nextUrl.searchParams;
const title = (sp.get('title') ?? 'tareksec.dev').slice(0, 90);
const eyebrow = (sp.get('eyebrow') ?? 'ENGINEERING NOTE').slice(0, 32);
const [regular, bold] = await Promise.all([loadFont(400), loadFont(700)]);
return new ImageResponse(
(
<div
style={{
width: '100%',
height: '100%',
display: 'flex',
flexDirection: 'column',
justifyContent: 'space-between',
padding: 64,
background: 'linear-gradient(135deg,#0b0f19 0%,#111827 60%,#1f2937 100%)',
color: '#f9fafb',
fontFamily: 'Inter',
}}
>
<div style={{ display: 'flex', fontSize: 28, letterSpacing: 6, color: '#38bdf8' }}>
{eyebrow}
</div>
<div style={{ display: 'flex', fontSize: 68, fontWeight: 700, lineHeight: 1.1 }}>
{title}
</div>
<div
style={{
display: 'flex',
justifyContent: 'space-between',
fontSize: 26,
color: '#9ca3af',
}}
>
<span>tareksec.dev</span>
<span>Md Tarek</span>
</div>
</div>
),
{
width: 1200,
height: 630,
fonts: [
{ name: 'Inter', data: regular, weight: 400, style: 'normal' },
{ name: 'Inter', data: bold, weight: 700, style: 'normal' },
],
headers: {
'Cache-Control': 'public, s-maxage=86400, stale-while-revalidate=604800, immutable',
},
},
);
} catch (err) {
console.error('[og] render failed', err);
// Never 500 on a social crawler — redirect to a pre-generated static card.
return Response.redirect(new URL('/og-fallback.png', req.nextUrl.origin), 302);
}
}---
5. JSON-LD at Scale: A Single `@graph` with Resolvable `@id` References
Emit one <script type="application/ld+json"> containing an @graph array. Multiple disconnected scripts force validators to guess entity relationships; @id cross-references inside a graph let Google resolve Product → Brand → Organization without heuristics.
React escapes text children of <script> as HTML, which corrupts JSON-LD. dangerouslySetInnerHTML is the only correct path — but then you own the escaping:
// components/seo/json-ld.tsx
import type { ReactElement } from 'react';
export function serializeJsonLd(data: Record<string, unknown>): string {
return JSON.stringify(data)
.replace(/</g, '\\u003c') // blocks </script> breakout (stored XSS)
.replace(/\u2028/g, '\\u2028') // breaks older JS parsers
.replace(/\u2029/g, '\\u2029');
}
export function JsonLd({ graph }: { graph: Record<string, unknown> }): ReactElement {
return (
<script
type="application/ld+json"
dangerouslySetInnerHTML={{ __html: serializeJsonLd(graph) }}
/>
);
}---
Practical Implementation
The wiring that ties it together — page module, robots, and edge headers.
// app/p/[slug]/page.tsx
import type { Metadata } from 'next';
import { notFound } from 'next/navigation';
import { JsonLd } from '@/components/seo/json-ld';
import { buildProductGraph } from '@/lib/seo/schema';
import { resolveCanonical } from '@/lib/seo/canonical';
import { getProduct } from '@/lib/catalog';
export const revalidate = 3600;
type Props = {
params: Promise<{ slug: string }>;
searchParams: Promise<Record<string, string | string[] | undefined>>;
};
export async function generateMetadata({ params, searchParams }: Props): Promise<Metadata> {
const [{ slug }, sp] = await Promise.all([params, searchParams]);
const product = await getProduct(slug);
if (!product || product.status !== 'published') {
return { title: 'Not found', robots: { index: false, follow: false } };
}
const { canonical, indexable } = resolveCanonical(
`/p/${product.slug}`,
new URLSearchParams(sp as Record<string, string>),
);
return {
title: `${product.name} — ${product.brand}`,
description: product.summary.slice(0, 155),
alternates: { canonical },
robots: indexable ? { index: true, follow: true } : { index: false, follow: true },
openGraph: {
type: 'website',
url: canonical,
images: [
{
url: `/api/og?title=${encodeURIComponent(product.name)}&v=${product.updatedAt.getTime()}`,
width: 1200,
height: 630,
alt: `${product.name} by ${product.brand}`,
},
],
},
};
}
export default async function ProductPage({ params }: Props) {
const { slug } = await params;
const product = await getProduct(slug);
if (!product || product.status !== 'published') notFound(); // real 404 status
return (
<main>
<JsonLd graph={buildProductGraph(product)} />
<h1>{product.name}</h1>
{/* ... */}
</main>
);
}// app/robots.ts
import type { MetadataRoute } from 'next';
const SITE = 'https://tareksec.dev';
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{
userAgent: '*',
allow: '/',
disallow: ['/api/', '/account/', '/search', '/*?sort=', '/*?filter.'],
},
{ userAgent: 'GPTBot', allow: '/', disallow: ['/account/'] },
{ userAgent: 'CCBot', disallow: '/' },
],
sitemap: `${SITE}/sitemap-index.xml`,
host: SITE,
};
}// next.config.ts
import type { NextConfig } from 'next';
const nextConfig: NextConfig = {
trailingSlash: false,
async headers() {
return [
{
source: '/search',
headers: [{ key: 'X-Robots-Tag', value: 'noindex, follow' }],
},
{
source: '/api/:path*',
headers: [
{ key: 'X-Robots-Tag', value: 'noindex' },
{ key: 'Cache-Control', value: 'public, s-maxage=86400, stale-while-revalidate=604800' },
],
},
];
},
};
export default nextConfig;// lib/seo/schema.ts (excerpt)
const SITE = 'https://tareksec.dev';
const ORG_ID = `${SITE}/#organization`;
const SITE_ID = `${SITE}/#website`;
export function buildProductGraph(p: Product): Record<string, unknown> {
const url = `${SITE}/p/${p.slug}`;
return {
'@context': 'https://schema.org',
'@graph': [
{
'@type': 'Organization',
'@id': ORG_ID,
name: 'tareksec.dev',
url: SITE,
logo: { '@type': 'ImageObject', url: `${SITE}/logo-512.png`, width: 512, height: 512 },
},
{
'@type': 'WebSite',
'@id': SITE_ID,
url: SITE,
name: 'tareksec.dev',
publisher: { '@id': ORG_ID },
potentialAction: {
'@type': 'SearchAction',
target: { '@type': 'EntryPoint', urlTemplate: `${SITE}/search?q={search_term_string}` },
'query-input': 'required name=search_term_string',
},
},
{
'@type': 'Product',
'@id': `${url}#product`,
name: p.name,
sku: p.sku,
brand: { '@type': 'Brand', name: p.brand },
description: p.summary,
image: p.images.map((src) => `${SITE}${src}`),
offers: {
'@type': 'Offer',
price: p.price.toFixed(2),
priceCurrency: 'USD',
availability: p.inStock
? 'https://schema.org/InStock'
: 'https://schema.org/OutOfStock',
url,
priceValidUntil: p.priceValidUntil.toISOString().slice(0, 10),
},
},
{
'@type': 'BreadcrumbList',
'@id': `${url}#breadcrumb`,
itemListElement: [
{ '@type': 'ListItem', position: 1, name: 'Home', item: SITE },
{ '@type': 'ListItem', position: 2, name: p.category, item: `${SITE}/c/${p.categorySlug}` },
{ '@type': 'ListItem', position: 3, name: p.name, item: url },
],
},
],
};
}---
Performance Benchmarks & Trade-Offs
*Measured on a 1.2M-URL catalog — Next.js 15.1, Node 22, Vercel Edge + Cloudflare fronting, 30-day Googlebot log sample.*
Sitemap delivery strategies
| Strategy | Build time (1.2M URLs) | Peak RAM | Freshness | Googlebot cost | Verdict | |---|---|---|---|---|---| | Static public/sitemap.xml | 6 m 40 s | 1.9 GB | Frozen at deploy | 24 × 50 MB fetches, always stale | ❌ | | Single dynamic sitemap.ts | n/a | 400 MB/req | Real-time | p95 TTFB 1.4 s, timeout risk | ⚠️ | | generateSitemaps() shards, ISR 1 h | 11 s / shard (cold) | 90 MB / shard | ≤ 1 h | 24 shards, p95 180 ms | ✅ | | Shards + explicit index + DB lastmod | 11 s / shard | 90 MB / shard | ≤ 1 h | 78% fewer redundant refetches | ✅✅ |
OpenGraph image strategies
| Approach | p95 TTFB | Cost / 1M images | Freshness | Constraints | |---|---|---|---|---| | Build-time static PNG | 8 ms (CDN) | $0 | deploy-bound | none | | Edge ImageResponse, no cache | 340 ms | ~$0.60 | real-time | flexbox only, no grid | | Edge + s-maxage=86400 | 12 ms hit / 340 ms miss | ~$0.02 | ≤ 24 h | flexbox only | | Pre-render to S3 + CDN | 15 ms | ~$0.09 | ≤ 1 h via queue | none |
Canonicalization strategy impact (90-day window)
| Signal | Before | After | Δ | |---|---|---|---| | Indexable URLs / discovered URLs | 0.6% | 41% | +40.4 pp | | Googlebot requests to noindex pages | 71% | 6% | −65 pp | | Median time-to-index (new SKU) | 9 days | 14 h | −95% | | Valid Product rich results | 38% | 96% | +58 pp |
---
Pro-Tips & Common Pitfalls
- ❌ `lastmod` set to `new Date()` on every request. Google will ignore
lastmodfor the entire host. Source it fromupdated_atin the database. - ❌ Listing URLs in the sitemap that canonical elsewhere. Every sitemap entry must be self-canonical. Filter with
AND canonical_slug = slug. - ❌ Rendering empty states with HTTP 200. Use
notFound(); soft 404s burn budget indefinitely. - ❌ Redirect chains.
/a → /b → /ccosts two crawls. Collapse to a single 308 at the edge and checkcurl -sILin CI. - ❌ Relative canonicals. Missing
metadataBasein the root layout silently emitshref="/path". - ❌ Canonicalizing pagination to page 1. Pages 2..N are then never indexed; use self-referencing canonicals with
rel="next"/"prev"semantics handled via sitemaplastmod. - ❌ Injecting JSON-LD via `<script>{JSON.stringify(x)}</script>`. React HTML-escapes the content and produces invalid JSON-LD. Use
dangerouslySetInnerHTMLwith<escaped to\u003c. - ❌ Emitting JSON-LD from client components. It must be in server-rendered HTML to be parsed reliably; keep the component server-only.
- ❌ Shipping `?v=`-less OG URLs. Without a content hash the CDN serves a stale card for a renamed product forever.
- ❌ Blocking CSS/JS in `robots.txt`. Google renders the page; blocked assets destroy LCP measurement and can hide content.
- ✅ Add a CI assertion that every
<loc>in every sitemap returns a 200 and is self-canonical. We run it nightly against 45k sampled URLs withp-limit(20)concurrency. - ✅ Verify with `curl -sI`, not by eyeballing the DOM: status code,
X-Robots-Tag,Cache-Control, andETagare the actual contract.
---
Frequently Asked Questions (FAQ)
Does `generateSitemaps()` in Next.js automatically produce a `/sitemap.xml` index?
No — you should not assume it. In current App Router versions each shard is served at /sitemap/[id].xml, and whether a top-level index is emitted has changed between releases. Author an explicit index route (/sitemap-index.xml), reference it from robots.ts, and verify with curl -sI https://yourdomain.com/sitemap.xml plus a GSC sitemap submission before you rely on it.
Should a canonical tag ever point to a URL that is not in the sitemap?
No. The sitemap and the canonical set must be identical. If /p/slug?color=red canonicalizes to /p/slug, it must never appear in the sitemap, and /p/slug must appear exactly once. Mismatch between the two is one of the most common causes of Discovered – currently not indexed and Duplicate, Google chose different canonical than user in Search Console.
Does edge-rendered OpenGraph generation hurt Core Web Vitals?
Not directly — og:image is never fetched by the browser during page load, so it does not affect LCP, INP, or CLS. It does affect first-render latency for social and chat clients, and it affects cost (~$0.60 per million uncached edge renders vs ~$0.02 cached). Always cache with s-maxage and version the URL with a content hash so invalidation is deterministic.
