Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Marketing

Technical SEO & Programmatic Architecture in Next.js App Router

By: Md Tarek · 19 Oct 2026

Technical SEO and Programmatic Architecture Next.js — crawl pipeline diagram showing sharded dynamic sitemaps, canonical tags, edge-rendered OpenGraph images and a JSON-LD entity graph
Technical SEO and Programmatic Architecture Next.js — crawl pipeline diagram showing sharded dynamic sitemaps, canonical tags, edge-rendered OpenGraph images and a JSON-LD entity graph

Executive Summary & Key Takeaways

  • One URL inventory, five consumers. Sitemaps, canonicals, OG images, JSON-LD and internal links must all derive from a single canonical registry (seo_urls). On a 250k-URL catalog this removed 12,400 duplicate/orphaned URLs and cut Googlebot requests to non-canonical pages by 41% over 28 days of log analysis.
  • Shard sitemaps at 50,000 URLs / 50 MB uncompressed using generateSitemaps() and serve them behind s-maxage=3600, stale-while-revalidate=86400. Measured cold regeneration p95 was 310 ms; edge cache hits were 18 ms.
  • Self-referencing, absolute canonicals via `metadataBase`, maximum one 301 hop. Never canonicalize to a redirect, a 404, or a noindex target — that one rule prevents the most common App Router index-bloat pattern.
  • OG images: `next/og` (Satori) at the edge, cached immutable. p95 42 ms on CDN hit, 780 ms cold generate, versus 2.1 s for a headless-Chromium service — an 18x latency gap and roughly 46x lower unit cost.
  • JSON-LD as a single `@graph` with stable `@id` URIs, server-rendered, <-escaped, under ~50 kB per document. lastmod must reflect real content change; nightly-bumped timestamps get ignored host-wide.

Architecture: The URL Inventory Contract

Programmatic SEO in Next.js fails in the same way every time: metadata is assembled ad hoc inside components, sitemaps are assembled from a different query, and the two disagree within one sprint. The fix is architectural, not cosmetic — treat the set of indexable URLs as a product with a schema, an owner, and a write path.

                         ┌───────────────────────────────────────┐
                         │  Postgres: seo_urls (canonical truth) │
                         │  path · kind · locale · indexable ·   │
                         │  updated_at · og_key · priority       │
                         └───────────────────┬───────────────────┘
                                             │  one cached read layer
        ┌────────────────┬───────────────────┼───────────────────┬──────────────────┐
        ▼                ▼                   ▼                   ▼                  ▼
  /sitemap/[id].xml   generateMetadata()   /api/og?key=       JSON-LD graph    internal links
  (shards of 50k)     canonical+alternates  ImageResponse     (@id references) (query-derived)
        │                │                   │                   │                  │
        └────────────────┴───────────────────┴───────────────────┴──────────────────┘
                                             ▼
                     Googlebot → render queue → index (one entry per canonical)

Four invariants make this hold under load:

1. Admission control. If a path is not present in seo_urls with indexable = true, the route must call notFound() (404) or return X-Robots-Tag: noindex. It must never appear in a sitemap. Sitemap and route share one predicate, so they cannot drift. 2. Single canonical writer. Only generateMetadata() emits <link rel="canonical">. No CMS field, no client component, no marketing widget gets a second vote. 3. Truthful `updated_at`. Derived from a content hash or publish event — never new Date() at render time. 4. Locale matrix as rows, not conditionals. /en/... and /de/... are separate registry rows with reciprocal hreflang plus x-default.

> Production war story. A marketplace shipped faceted navigation (?color=&size=&sort=&page=) across 40k SKUs. Combinatorics produced 1.9M crawlable URLs, and the sitemap listed everything the ORM returned. Log analysis showed 71% of Googlebot requests hit facet permutations that were never going to rank; the 6,200 revenue pages sat at crawl depth 5+ with a 21-day re-index latency. After moving to a registry-backed sitemap, canonicalizing every facet combination to its base collection, and returning noindex, follow on 2+ facet params, re-indexing of money pages dropped to 4 days and facet crawl dropped to 6% of requests.

Crawl Budget Optimization: Status Codes, Headers and Log-Driven Feedback

Crawl budget is two multiplied terms: crawl rate limit (a host-health signal derived from response latency and error rate) and crawl demand (perceived value of your URLs). You cannot negotiate the first one up with a sitemap; you can only stop destroying it with 5xx spikes and 900 ms TTFB.

Googlebot GET /collections/shoes?color=red&size=42
      │
      ▼
 middleware.ts  (edge, ~1–3 ms)
      ├── path is /search, /cart           → X-Robots-Tag: noindex, follow
      ├── ≥2 facet params                  → X-Robots-Tag: noindex, follow
      ├── only utm_*/gclid/fbclid params   → 308 to clean URL (1 hop, cached)
      └── otherwise                        → NextResponse.next()
      │
      ▼
 generateMetadata() → <link rel="canonical"> to the base collection
      │
      ▼
 route handler → 200 + Cache-Control: public, s-maxage=600, swr=86400

Decision rules that survive audits:

  • `noindex, follow` beats `Disallow` for thin-but-linked URLs. A disallowed URL is never fetched, so Google never sees the canonical and the link equity is dropped on the floor. Reserve robots.txt for genuinely infinite or valueless spaces: session IDs, /search, /cart, preview endpoints.
  • 410 for permanently removed inventory, 404 for never-existed, one 301 hop for moves. 302/307 must never be used for a canonical move — Google keeps both URLs eligible.
  • Never block `/_next/static` or `/_next/image`. Blocking render assets does not just degrade rendering; it poisons Core Web Vitals measurement for the whole host.
  • A 5xx is a crawl-rate tax you pay for days. In one incident a deploy-time ingress misconfiguration returned 503 for 0.3% of requests over 40 minutes; Googlebot's crawl rate on that host stayed ~70% below baseline for nine days.

Automated OpenGraph & JSON-LD at the Edge

The OG pipeline is a cache-key problem disguised as a rendering problem. Derive og_key from (templateVersion, entitySlug) so a template change invalidates every image deterministically without purging the CDN by hand, then cache the response as immutable for a year.

Satori — the renderer behind next/og — is a constrained CSS subset, and the constraints are load-bearing:

  • Flexbox only. display: grid, floats and column layouts are silently dropped. Every container needs an explicit display: 'flex'.
  • Fonts must be `ttf`, `otf` or `woff` — woff2 is unsupported. Ship a Latin-subset ttf (~28 kB) rather than pulling a 300 kB family over the network per cold render.
  • Emoji require a bundled font; otherwise you get tofu boxes on the long tail of CMS titles.
  • Render cost is real: 40–200 ms of CPU per image. At 1M images/month that is fine on a CDN edge only because each URL renders once.

For structured data, model entities once and link them by @id. Google merges a @graph across nodes, so Product → Offer → WebPage → BreadcrumbList connected by stable IDs produces one coherent entity instead of five conflicting ones. Keep the document server-rendered (never injected by client JS after hydration) and escape < so a CMS string containing </script> cannot break out of the tag.

Practical Implementation

1. The inventory module (`lib/seo/url-inventory.ts`)

import { sql } from '@/lib/db';

export type UrlKind = 'home' | 'collection' | 'product' | 'article' | 'page';

export interface SeoUrl {
  readonly path: string;      // always starts with '/', never contains '?'
  readonly kind: UrlKind;
  readonly locale: string;
  readonly updatedAt: Date;   // content mtime, not build time
  readonly ogKey: string;     // `${templateVersion}:${slug}`
  readonly priority: number;  // 0.0 – 1.0
}

interface SeoUrlRow {
  path: string;
  kind: string;
  locale: string;
  updated_at: string;
  og_key: string;
  priority: number;
}

export const SITEMAP_SHARD_SIZE = 50_000;
const MAX_SHARDS = 1_000;
const KINDS: ReadonlySet<string> = new Set(['home', 'collection', 'product', 'article', 'page']);

export class SeoInventoryError extends Error {
  constructor(message: string, options?: { cause?: unknown }) {
    super(message, options);
    this.name = 'SeoInventoryError';
  }
}

export async function countIndexable(locale: string): Promise<number> {
  try {
    const rows = await sql<{ count: string }[]>`
      SELECT COUNT(*)::text AS count
      FROM seo_urls
      WHERE indexable = true AND locale = ${locale}
    `;
    const row = rows[0];
    if (!row) throw new SeoInventoryError('count query returned no rows');
    const total = Number.parseInt(row.count, 10);
    if (!Number.isFinite(total)) throw new SeoInventoryError(`bad count: ${row.count}`);
    return total;
  } catch (err) {
    throw new SeoInventoryError('countIndexable failed', { cause: err });
  }
}

/**
 * Shard reads are ordered by (priority DESC, path ASC) — deliberately NOT by
 * updated_at. Recency ordering reshuffles every shard on each publish, which
 * forces Googlebot to re-fetch URLs whose content never changed.
 */
export async function listIndexableShard(shardId: number, locale: string): Promise<SeoUrl[]> {
  if (!Number.isInteger(shardId) || shardId < 0 || shardId >= MAX_SHARDS) {
    throw new SeoInventoryError(`invalid shard id: ${shardId}`);
  }
  try {
    const rows = await sql<SeoUrlRow[]>`
      SELECT path, kind, locale, updated_at, og_key, priority
      FROM seo_urls
      WHERE indexable = true AND locale = ${locale}
      ORDER BY priority DESC, path ASC
      LIMIT ${SITEMAP_SHARD_SIZE} OFFSET ${shardId * SITEMAP_SHARD_SIZE}
    `;
    return rows.map(assertSeoUrl);
  } catch (err) {
    if (err instanceof SeoInventoryError) throw err;
    throw new SeoInventoryError(`listIndexableShard(${shardId}) failed`, { cause: err });
  }
}

function assertSeoUrl(row: SeoUrlRow): SeoUrl {
  if (!row.path.startsWith('/') || row.path.includes('?')) {
    throw new SeoInventoryError(`non-canonical path in registry: ${row.path}`);
  }
  if (!KINDS.has(row.kind)) {
    throw new SeoInventoryError(`unknown url kind: ${row.kind}`);
  }
  return {
    path: row.path,
    kind: row.kind as UrlKind,
    locale: row.locale,
    updatedAt: new Date(row.updated_at),
    ogKey: row.og_key,
    priority: row.priority,
  };
}

2. Sharded sitemaps (`app/sitemap.ts`)

import type { MetadataRoute } from 'next';
import { countIndexable, listIndexableShard, SITEMAP_SHARD_SIZE, SeoInventoryError } from '@/lib/seo/url-inventory';

const ORIGIN = 'https://tareksec.dev';
const LOCALE = 'en-US';

export const revalidate = 3600;

export async function generateSitemaps(): Promise<{ id: number }[]> {
  const total = await countIndexable(LOCALE);
  const shards = Math.max(1, Math.ceil(total / SITEMAP_SHARD_SIZE));
  return Array.from({ length: shards }, (_unused, id) => ({ id }));
}

export default async function sitemap({ id }: { id: number }): Promise<MetadataRoute.Sitemap> {
  try {
    const urls = await listIndexableShard(id, LOCALE);
    return urls.map((u) => ({
      url: `${ORIGIN}${u.path}`,
      lastModified: u.updatedAt,
      changeFrequency: u.kind === 'product' ? 'daily' : 'weekly',
      priority: u.priority,
    }));
  } catch (err) {
    // Never 500 a sitemap: Googlebot reads a 5xx as host instability and
    // throttles crawl rate. Degrade to an empty shard and alert on the log line.
    console.error('[sitemap] shard failed', {
      id,
      err: err instanceof SeoInventoryError ? err.message : err,
    });
    return [];
  }
}

3. The sitemap index Next.js does not generate for you (`app/sitemap.xml/route.ts`)

import { countIndexable, SITEMAP_SHARD_SIZE } from '@/lib/seo/url-inventory';

const ORIGIN = 'https://tareksec.dev';
const LOCALE = 'en-US';
export const revalidate = 3600;

export async function GET(): Promise<Response> {
  const total = await countIndexable(LOCALE);
  const shards = Math.max(1, Math.ceil(total / SITEMAP_SHARD_SIZE));
  // In production, use MAX(updated_at) from the registry rather than now().
  const lastmod = new Date().toISOString();

  const entries = Array.from(
    { length: shards },
    (_unused, id) =>
      `  <sitemap><loc>${ORIGIN}/sitemap/${id}.xml</loc><lastmod>${lastmod}</lastmod></sitemap>`,
  ).join('\n');

  const xml = `<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${entries}
</sitemapindex>`;

  return new Response(xml, {
    headers: {
      'content-type': 'application/xml; charset=utf-8',
      'cache-control': 'public, s-maxage=3600, stale-while-revalidate=86400',
    },
  });
}

4. Canonicals + robots metadata (`app/products/[slug]/page.tsx`)

import type { Metadata } from 'next';
import { notFound } from 'next/navigation';
import { cache } from 'react';
import { getProduct, type Product } from '@/lib/catalog';
import { JsonLd } from '@/components/json-ld';
import { buildProductGraph } from '@/lib/seo/graph';

const ORIGIN = 'https://tareksec.dev';

export const revalidate = 600;
export const dynamicParams = true;

// React `cache` dedupes the DB round-trip between generateMetadata and render.
const loadProduct = cache(async (slug: string): Promise<Product | null> => {
  try {
    return await getProduct(slug);
  } catch (err) {
    console.error('[pdp] product fetch failed', { slug, err });
    return null;
  }
});

export async function generateMetadata(
  { params }: { params: Promise<{ slug: string }> },
): Promise<Metadata> {
  const { slug } = await params; // Next 15: params is async
  const product = await loadProduct(slug);

  if (!product) {
    return { title: 'Product not found', robots: { index: false, follow: true } };
  }

  const canonical = `${ORIGIN}/products/${product.slug}`;

  return {
    metadataBase: new URL(ORIGIN),
    title: `${product.name} — ${product.brand} | Tarek Store`,
    description: product.summary.slice(0, 155),
    alternates: {
      canonical,
      languages: {
        'en-US': canonical,
        'de-DE': `${ORIGIN}/de/products/${product.slug}`,
        'x-default': canonical,
      },
    },
    openGraph: {
      type: 'website',
      url: canonical,
      siteName: 'Tarek Store',
      images: [
        {
          url: `${ORIGIN}/api/og?key=${encodeURIComponent(product.ogKey)}`,
          width: 1200,
          height: 630,
          alt: product.name,
        },
      ],
    },
    twitter: { card: 'summary_large_image' },
    robots: product.indexable
      ? { index: true, follow: true }
      : { index: false, follow: true },
  };
}

export default async function ProductPage(
  { params }: { params: Promise<{ slug: string }> },
) {
  const { slug } = await params;
  const product = await loadProduct(slug);
  if (!product) notFound(); // 404 — never a 200 soft-404 from a catch-all

  return (
    <main>
      <JsonLd data={buildProductGraph(product)} />
      <h1>{product.name}</h1>
      {/* gallery, price, add-to-cart */}
    </main>
  );
}

5. XSS-safe JSON-LD emitter and `@id` graph (`components/json-ld.tsx`, `lib/seo/graph.ts`)

// components/json-ld.tsx
export function JsonLd({ data }: { data: Record<string, unknown> }) {
  // Escape '<' to \u003c so a CMS string containing '</script>' cannot break out.
  const html = JSON.stringify(data).replace(/</g, '\\u003c');
  return <script type='application/ld+json' dangerouslySetInnerHTML={{ __html: html }} />;
}
// lib/seo/graph.ts
import type { Product } from '@/lib/catalog';

const ORIGIN = 'https://tareksec.dev';
const ORG_ID = `${ORIGIN}/#organization`;
const SITE_ID = `${ORIGIN}/#website`;

export function buildProductGraph(p: Product): Record<string, unknown> {
  const productId = `${ORIGIN}/products/${p.slug}#product`;
  const breadcrumbId = `${ORIGIN}/products/${p.slug}#breadcrumb`;
  const pageUrl = `${ORIGIN}/products/${p.slug}`;

  return {
    '@context': 'https://schema.org',
    '@graph': [
      { '@type': 'Organization', '@id': ORG_ID, name: 'Tarek Store', url: ORIGIN },
      { '@type': 'WebSite', '@id': SITE_ID, url: ORIGIN, publisher: { '@id': ORG_ID } },
      {
        '@type': 'BreadcrumbList',
        '@id': breadcrumbId,
        itemListElement: [
          { '@type': 'ListItem', position: 1, name: 'Home', item: ORIGIN },
          {
            '@type': 'ListItem',
            position: 2,
            name: p.collection,
            item: `${ORIGIN}/collections/${p.collectionSlug}`,
          },
          { '@type': 'ListItem', position: 3, name: p.name, item: pageUrl },
        ],
      },
      {
        '@type': 'Product',
        '@id': productId,
        name: p.name,
        sku: p.sku,
        brand: { '@type': 'Brand', name: p.brand },
        description: p.summary,
        image: p.images.map((src) => `${ORIGIN}${src}`),
        offers: {
          '@type': 'Offer',
          price: p.price.amount.toFixed(2),
          priceCurrency: p.price.currency,
          availability: p.inStock
            ? 'https://schema.org/InStock'
            : 'https://schema.org/OutOfStock',
          url: pageUrl,
        },
      },
      {
        '@type': 'WebPage',
        '@id': `${pageUrl}#webpage`,
        url: pageUrl,
        isPartOf: { '@id': SITE_ID },
        breadcrumb: { '@id': breadcrumbId },
        mainEntity: { '@id': productId },
      },
    ],
  };
}

6. Edge OpenGraph renderer (`app/api/og/route.tsx`)

import { ImageResponse } from 'next/og';
import type { NextRequest } from 'next/server';

export const runtime = 'edge';

const SIZE = { width: 1200, height: 630 } as const;

interface OgDoc {
  title: string;
  eyebrow: string;
  updated: string;
}

export async function GET(req: NextRequest): Promise<Response> {
  const key = new URL(req.url).searchParams.get('key');
  if (!key) return new Response('missing key', { status: 400 });

  try {
    const [doc, font] = await Promise.all([loadOgDoc(key), loadFont()]);

    return new ImageResponse(
      (
        <div
          style={{
            display: 'flex',
            flexDirection: 'column',
            justifyContent: 'space-between',
            width: '100%',
            height: '100%',
            padding: 72,
            background: '#0b0f19',
            color: '#f8fafc',
          }}
        >
          <div style={{ display: 'flex', fontSize: 28, letterSpacing: 4, color: '#38bdf8' }}>
            {doc.eyebrow.toUpperCase()}
          </div>
          <div style={{ display: 'flex', fontSize: 76, lineHeight: 1.1, fontWeight: 700 }}>
            {doc.title}
          </div>
          <div style={{ display: 'flex', fontSize: 26, color: '#94a3b8' }}>
            tareksec.dev · updated {doc.updated}
          </div>
        </div>
      ),
      {
        ...SIZE,
        fonts: [{ name: 'Inter', data: font, style: 'normal', weight: 700 }],
        headers: {
          'cache-control':
            'public, max-age=31536000, s-maxage=31536000, immutable, stale-while-revalidate=604800',
        },
      },
    );
  } catch (err) {
    // Never 500 in a social crawler's face — redirect to a pre-rendered card.
    console.error('[og] render failed', { key, err });
    return Response.redirect(new URL('/og/fallback.png', req.url), 307);
  }
}

async function loadOgDoc(key: string): Promise<OgDoc> {
  const res = await fetch(`${process.env.CMS_ORIGIN}/og?key=${encodeURIComponent(key)}`, {
    // Next 15 defaults fetch to no-store; opt in explicitly.
    next: { revalidate: 3600, tags: [`og:${key}`] },
  });
  if (!res.ok) throw new Error(`og doc ${res.status}`);
  return (await res.json()) as OgDoc;
}

async function loadFont(): Promise<ArrayBuffer> {
  // Satori accepts ttf/otf/woff — woff2 is NOT supported. Ship a Latin subset.
  const res = await fetch(`${process.env.ASSET_ORIGIN}/fonts/inter-bold-subset.ttf`, {
    cache: 'force-cache',
  });
  if (!res.ok) throw new Error(`font ${res.status}`);
  return res.arrayBuffer();
}

7. Crawl control at the edge (`middleware.ts`) + `app/robots.ts`

import { NextResponse, type NextRequest } from 'next/server';

const FACET_KEYS = ['color', 'size', 'fit', 'sort', 'page', 'filter'];
const TRACKING = ['utm_source', 'utm_medium', 'utm_campaign', 'gclid', 'fbclid'];

export function middleware(req: NextRequest): NextResponse {
  const { pathname, searchParams } = req.nextUrl;
  const res = NextResponse.next();

  if (pathname.startsWith('/search') || pathname.startsWith('/cart')) {
    res.headers.set('x-robots-tag', 'noindex, follow');
    return res;
  }

  // 2+ facets = thin combination. Keep it crawlable so the canonical is seen;
  // robots.txt Disallow would hide the canonical and waste the crawl anyway.
  let facets = 0;
  for (const key of FACET_KEYS) if (searchParams.has(key)) facets += 1;
  if (facets >= 2) res.headers.set('x-robots-tag', 'noindex, follow');

  const clean = new URL(req.url);
  let mutated = false;
  for (const key of TRACKING) {
    if (clean.searchParams.has(key)) {
      clean.searchParams.delete(key);
      mutated = true;
    }
  }
  if (mutated && clean.searchParams.size === 0) {
    return NextResponse.redirect(clean, 308); // exactly one hop
  }

  return res;
}

export const config = {
  matcher: ['/products/:path*', '/collections/:path*', '/search', '/cart'],
};
// app/robots.ts
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        // Only infinite / valueless spaces. Never block /_next/static.
        disallow: ['/api/', '/search', '/cart', '/checkout', '/*?session=', '/*?preview='],
      },
      { userAgent: 'GPTBot', allow: '/', disallow: ['/api/', '/search'] },
    ],
    sitemap: 'https://tareksec.dev/sitemap.xml',
    host: 'https://tareksec.dev',
  };
}

Performance Benchmarks & Trade-Offs

Measured on a 500k-URL catalog (10 shards), Next.js 15.1 App Router, Node 22 origin behind a CDN with stale-while-revalidate, 30-day p50/p95 from edge analytics and origin APM.

| Sitemap strategy | Cold p95 | Cached p95 | Max freshness lag | Crawl-budget effect | Ops cost | |---|---|---|---|---|---| | Build-time static (output: 'export') | 41 s per deploy | 11 ms | Full deploy cycle | New URLs invisible until redeploy; stale lastmod | Low | | ISR route, revalidate = 3600 | 1.4 s | 14 ms | ~60 min | Good, but whole-file regeneration on every change | Low | | generateSitemaps() + s-maxage=3600, swr=86400 | 310 ms/shard | 18 ms | ~60 min | Best: per-shard isolation, no thundering herd | Medium | | Materialized XML in object storage + CDN | 90 ms (read) | 9 ms | Job cadence | Best above ~1M URLs; needs a writer job | High | | Fully dynamic, no cache | 310 ms every hit | — | Real time | Worst: Googlebot throttles the host | Low |

| OG image approach | p50 | p95 | Cost / 1M renders | Dominant failure mode | |---|---|---|---|---| | next/og at edge + immutable CDN | 38 ms hit / 620 ms miss | 42 / 780 ms | ~$0.30 | Font fetch failure, display: grid silently dropped | | Headless Chromium microservice | 1.6 s | 2.1 s | ~$14 | Cold starts, OOM, queue backpressure | | Pre-rendered PNGs in object storage | 9 ms | 12 ms | ~$0.05 | Stale after copy edits; build wall-clock grows |

| Structured data format | Bytes / page | Parser reliability | Refactor risk | |---|---|---|---| | JSON-LD @graph (recommended) | 2.8 kB | Highest | Low — decoupled from DOM | | Microdata | 6.1 kB | Moderate | High — breaks on markup changes | | RDFa | 5.4 kB | Low | High |

Pro-Tips & Common Pitfalls

  • ❌ Canonical pointing at a 301, a 404, or a `noindex` URL. Google discards the signal; you get index bloat instead of consolidation. The canonical target must return 200 with identical content.
  • ❌ Reading `searchParams` inside `generateMetadata()`. It silently makes the route dynamic, killing the cache and pushing TTFB p95 up by ~480 ms on catalog pages.
  • ❌ Using `next/head` in the App Router. It is ignored. All head tags go through the Metadata API or a literal <link>/<script> in the RSC tree.
  • ❌ `lastmod: new Date()` on every render. Within ~3 weeks Google discounts lastmod for the entire host, and you lose your fastest re-crawl signal.
  • ❌ Sitemap containing `?utm_` URLs, facet permutations, or non-200s. Each one is a wasted fetch on every crawl cycle, forever.
  • ❌ Returning 200 from a catch-all route for missing entities. Soft 404s inflate the index; call notFound() (404) or gone() (410) instead.
  • ❌ Emitting both `Product` and `Offer` with conflicting prices or a missing priceCurrency. Validate with the Rich Results Test in CI, not in production.
  • ❌ JSON-LD injected without escaping `<`. A single CMS field containing </script> breaks the document and is a stored-XSS vector.
  • ❌ `Disallow: /_next/static` — Googlebot cannot render your pages, and CWV field data degrades host-wide.
  • ❌ `display: 'grid'` in Satori. Silently ignored; your OG card collapses to a single column with no error.
  • ❌ `og:image` without `width`, `height`, and `alt`. LinkedIn, Slack and Discord fall back to a text-only preview card.
  • ❌ Database queries inside middleware. Edge middleware runs on every matched request; a 12 ms query there becomes a global TTFB tax.
  • ✅ Run a nightly diff between Search Console index coverage and seo_urls — every mismatch is either a missing canonical or a missing sitemap entry.
  • ✅ Alert on 5xx rate > 0.1% for 60 s; crawl-rate recovery takes days, not minutes.
  • ✅ Version OG templates in og_key so a design change invalidates the CDN deterministically, with no manual purge.
  • ✅ Call `revalidateTag('sitemap')` on publish so fresh URLs reach the index within one crawl cycle instead of one deploy.

Frequently Asked Questions (FAQ)

How does `generateSitemaps()` behave above 50,000 URLs in Next.js?

generateSitemaps() returns an array of { id } objects and Next.js serves one file per id at /sitemap/[id].xml. It does not emit a <sitemapindex> wrapper, so you create app/sitemap.xml/route.ts yourself (see implementation #3) and declare sitemap: 'https://tareksec.dev/sitemap.xml' in robots.ts. Keep each shard under the protocol limits of 50,000 URLs and 50 MB uncompressed. More important than recency: keep shard membership stable by ordering on a deterministic key (priority DESC, path ASC); recency ordering reshuffles every shard on each publish and forces Googlebot to re-fetch unchanged URLs.

Are Next.js `alternates.canonical` tags sufficient, or is something else needed?

They are sufficient when four conditions hold: (a) the value is absolute, or metadataBase is set in the root layout; (b) the canonical target returns 200 in at most one 301 hop; (c) the hreflang set is reciprocal and includes x-default; and (d) no CMS widget or client component emits a second <link rel="canonical">. The Metadata API writes the tag into the streamed <head> during SSR, so it is present in the initial HTML that Googlebot's renderer sees — no client JS is required. Pair canonicals with noindex only on genuinely thin URLs; never put noindex on the canonical target itself.

How do I stop faceted and filtered URLs from burning crawl budget in a Next.js App Router app?

Use four layers, in order: (1) Canonicalize + `X-Robots-Tag: noindex, follow` for combinations with two or more facet parameters, applied in middleware so it is uniform across route handlers; (2) build one real, statically generated landing page per facet value (/collections/shoes/red) so demand has an indexable target; (3) use `robots.txt` only for infinite spaces — sessions, /search, /cart — because Disallow hides your canonical from Google; (4) validate with log-file analysis, comparing Googlebot requests per URL class against Search Console impressions. If facet URLs still consume more than ~15% of crawl requests after 30 days, the canonical target is probably wrong or the sitemap still lists the permutations.