Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
Marketing

Technical SEO and Programmatic Architecture in Next.js 2025

By: Md Tarek · 17 Oct 2026

Next.js programmatic SEO architecture showing dynamic sitemaps, canonical tags, automated OpenGraph generation and JSON-LD structured data
Next.js programmatic SEO architecture showing dynamic sitemaps, canonical tags, automated OpenGraph generation and JSON-LD structured data

Primary keyword: Technical SEO and Programmatic Architecture Next.js Reference stack for every number below: Next.js 15 App Router, React 19, Prisma + Postgres, Vercel Edge Runtime, 1.24M programmatic product/landing URLs, ~300k Googlebot requests/day.

Executive Summary & Key Takeaways

  • Shard sitemaps at 45,000 URLs, never at the 50,000 spec limit. At 1.24M URLs, moving from a single app/sitemap.ts function to generateSitemaps() shards behind ISR cut p95 sitemap generation from 4.2s cold to 180ms per cached shard, and eliminated every Couldn't fetch error in Search Console.
  • Canonical consolidation is the highest-ROI technical SEO change available to you. Annotating 341k faceted parameter permutations moved Googlebot's share of requests hitting canonical URLs from 41% to 93%, and indexed pages from 62% to 91% within six weeks.
  • Edge-rendered OpenGraph with `next/og` (Satori + resvg) costs 210ms p50 / 480ms p95 cold for a 1200x630 PNG at ~62KB. With revalidate = false plus immutable CDN caching, repeat fetches land under 15ms. Measured social CTR: +18% versus a generic fallback card.
  • One `@graph` JSON-LD payload beats five separate `<script>` tags. Cross-linking entities with stable @id values (Organization, WebSite, Product, BreadcrumbList) is the only pattern Google reliably deduplicates across templates. Keep total payload under 8KB gzipped and inline it in the server HTML.
  • Crawl budget is an engineering SLA, not a marketing KPI. Every 200 OK handed to a junk URL burns a crawl slot. Disallow parameter space in robots.ts, noindex genuinely thin pages, and return real 404/410 via notFound() instead of soft-404s that return 200.

The Five-Layer Programmatic SEO Stack

A programmatic site is a data pipeline with a rendering layer bolted on. Treat discovery, canonicalization, social rendering, and entity markup as four independent subsystems, each with its own cache lifetime and failure mode.

 Crawler (Googlebot / Bingbot / GPTBot)
        |
        v
 +----------------------+  edge cache HIT (~9ms TTFB)
 |  Edge / CDN (ISR)    |<-------------------------------+
 +----------+-----------+                                |
            | MISS                                       |
            v                                            |
 +----------------------+  /sitemap/[id].xml             |
 | app/sitemap.ts       |--> 45k-URL shards,             |
 | generateSitemaps()   |    revalidate = 3600           |
 +----------+-----------+                                |
            v                                            |
 +----------------------+  disallow ?sort= ?filter=      |
 | app/robots.ts        |  sitemap: index.xml            |
 +----------+-----------+                                |
            v                                            |
 +----------------------+  generateMetadata()            |
 | /products/[slug]     |--> title, canonical, hreflang  |
 | ISR: revalidate 3600 |--> JSON-LD @graph (inline)     |
 +----------+-----------+                                |
            v                                            |
 +----------------------+  ImageResponse (Satori)        |
 | /og/products/[slug]  |--> 1200x630 PNG, immutable ----+
 +----------------------+

Layer boundaries matter because cache invalidation correctness is the hardest part of programmatic SEO. A sitemap that lists URLs whose canonicals point elsewhere is worse than no sitemap at all.

Dynamic Sitemaps That Survive 1M+ URLs

The sitemap protocol caps a single file at 50,000 URLs or 50MB uncompressed, and a sitemap index at 50,000 child sitemaps. Next.js App Router encodes this natively: export generateSitemaps() and the framework emits both /sitemap/[id].xml children and a /sitemap.xml index automatically.

Three details separate a working sitemap from a broken one at scale:

1. Never use `OFFSET` pagination for shards. OFFSET 900000 in Postgres forces a sequential scan of every skipped row. Shard by a stable keyset (ID ranges or a shard_id column) so each shard is an index-only range scan. On our dataset, offset pagination cost 3.1s per shard; keyset ranges cost 140ms. 2. `lastmod` must be truthful. Google treats lastmod as a hint it only trusts while it stays consistently accurate. A new Date() default on every regeneration is the fastest way to get the signal ignored permanently. 3. Never let a sitemap return 5xx. A single failed fetch demotes the file to Couldn't fetch and Google backs off on re-crawling it for days. Catch, log, and return an empty or partial shard instead.

Note that Google ignores priority and changeFrequency entirely; Bing and Yandex read them inconsistently. Do not spend engineering hours tuning them.

Canonical Tags, Faceted URL Explosion, and Crawl Budget Math

This is where most programmatic sites lose the majority of their crawl budget.

War story. In Q1 we shipped faceted filtering as real, indexable routes because a product decision framed them as long-tail landing pages. Within 11 days Google had discovered 412k ?color= and ?sort= permutations, and 68% of the daily 300k crawl requests were going to them. Canonical PDPs were being recrawled every 11 days instead of every 2. Index coverage sat at 62%.

The fix was three coordinated changes, not one:

  • robots.ts disallows /*?sort= and /*?filter= so the parameter space is never crawled. This is the only place where blocking is correct, because those URLs genuinely have no standalone value.
  • For single-facet pages that do have search demand, the page renders rel=canonical to itself and gets a self-referencing entry. Never canonical a page to a URL that does not itself resolve 200.
  • Multi-facet combinations and zero-result pages emit robots: { index: false, follow: true } and never a cross-canonical. Mixing noindex with a canonical pointing elsewhere sends contradictory consolidation signals.
Crawl budget arithmetic (daily):

  300,000 requests/day
  x 0.68 on parameter URLs   = 204,000 wasted slots  (BEFORE)
  x 0.07 on parameter URLs   =  21,000 wasted slots  (AFTER)

  Effective budget on canonical PDPs: 96k/day -> 279k/day  (2.9x)
  Time to full coverage of 380k canonical URLs:
    380,000 / 96,000  = ~40 days   (BEFORE)
    380,000 / 279,000 = ~1.4 days  (AFTER)

One more lever most teams miss: crawl rate scales with server latency. Google reduces host load allocation when p95 response time degrades. Keeping PDP TTFB at 40-90ms via ISR is directly a crawl-budget optimization, not just a Core Web Vitals one.

Critical pitfall: if you Disallow a URL in robots.txt, Googlebot can never fetch it, which means it can never read the noindex meta tag or X-Robots-Tag header inside it. Blocking and noindexing the same URL means the URL stays in the index as a bare URL with no snippet. Choose one mechanism per URL class.

Layer 4: Automated OpenGraph Generation at the Edge

next/og ships Satori (JSX to SVG) plus a resvg WASM rasterizer. It runs in the Edge Runtime, which means no filesystem, no node-canvas, and a constrained CSS subset in which every `div` with more than one child must declare `display: flex` — grid is unsupported.

Production rules that matter:

  • Render at 1200x630. Below 600px wide, X/Twitter downgrades your card to summary instead of summary_large_image. Images are capped at 5MB and must be absolute URLs.
  • Load fonts as `ArrayBuffer`, not CSS. Satori accepts TTF and OTF; WOFF2 is not supported. Fetch once per cold start, and prefer a subsetted font to keep cold start under 150ms.
  • Set `revalidate = false` for immutable slugs. OG images keyed to an immutable slug should be generated once and cached forever. Use revalidatePath on the product path when the underlying data mutates.
  • Always set `og:image:alt` and dimensions. This is an accessibility requirement and improves image understanding for the social scrapers that consume it.

Layer 5: Structured JSON-LD as an Entity Graph

The difference between markup that produces rich results and markup that produces nothing is entity linking. Emit a single @graph with stable @id values so Google can merge the Product node on this page with the Organization node on the homepage without guessing.

Rules that survive schema updates:

  • Use absolute @id URLs (https://tareksec.dev/#organization) — stable fragments, not paths that change with routing.
  • BreadcrumbList items must be in position order and match the visible breadcrumb exactly.
  • Only emit aggregateRating when the rating is genuinely user-generated and visible on the page. Fabricated rating markup is a manual-action vector.
  • Google retired the Sitelinks Search Box rich result in 2024, but WebSite + SearchAction still helps Knowledge Graph entity resolution and is consumed by Bing. Keep it, but do not expect a search box.
  • Escape <, >, and & when serializing into dangerouslySetInnerHTML. A product name containing </script> is a stored XSS payload, not a markup bug.

Practical Implementation

Five files, copy-pasteable. Assume NEXT_PUBLIC_SITE_URL is set and Prisma exposes a product model with slug, updatedAt, published, noindex, price, sku, brand, stock, rating, reviewCount.

1. Root metadata base (required for canonical resolution)

// app/layout.tsx
import type { Metadata } from 'next';

export const metadata: Metadata = {
  // metadataBase makes every relative canonical and OG URL resolve to an absolute one.
  metadataBase: new URL(process.env.NEXT_PUBLIC_SITE_URL ?? 'https://tareksec.dev'),
};

2. Sharded, ISR-cached sitemaps

// app/sitemap.ts
import type { MetadataRoute } from 'next';
import { prisma } from '@/lib/prisma';

// Hourly ISR with stale-while-revalidate. Sitemaps must never be request-time rendered.
export const revalidate = 3600;
export const dynamic = 'force-static';

const SITE = process.env.NEXT_PUBLIC_SITE_URL ?? 'https://tareksec.dev';
// Spec limit is 50_000 URLs / 50MB uncompressed. Keep 10% headroom for growth mid-shard.
const CHUNK = 45_000;

export async function generateSitemaps(): Promise<{ id: number }[]> {
  const total = await prisma.product.count({ where: { published: true, noindex: false } });
  const shards = Math.max(1, Math.ceil(total / CHUNK));
  return Array.from({ length: shards }, (_, id) => ({ id }));
}

export default async function sitemap({ id }: { id: number }): Promise<MetadataRoute.Sitemap> {
  try {
    const rows = await prisma.product.findMany({
      where: { published: true, noindex: false },
      // Keyset ordering keeps shard boundaries stable across regenerations.
      orderBy: { id: 'asc' },
      skip: id * CHUNK,
      take: CHUNK,
      select: { slug: true, updatedAt: true, priority: true },
    });

    return rows.map((row) => ({
      url: `${SITE}/products/${row.slug}`,
      // Real mutation timestamps only. A synthetic new Date() makes Google ignore lastmod.
      lastModified: row.updatedAt.toISOString(),
      changeFrequency: 'weekly' as const,
      priority: row.priority ?? 0.7,
      images: [`${SITE}/og/products/${row.slug}.png`],
      alternates: {
        languages: {
          en: `${SITE}/products/${row.slug}`,
          bn: `${SITE}/bn/products/${row.slug}`,
        },
      },
    }));
  } catch (err) {
    // A 5xx here gets the whole file marked Couldn't fetch. Degrade to an empty shard.
    console.error('[sitemap] shard failed', { shard: id, err });
    return [];
  }
}

3. Crawl-budget guardrails

// app/robots.ts
import type { MetadataRoute } from 'next';

const SITE = process.env.NEXT_PUBLIC_SITE_URL ?? 'https://tareksec.dev';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: [
          '/api/',
          '/admin/',
          '/cart',
          '/search?',        // unbounded combinatorial internal search space
          '/*?sort=',        // parameter permutations of the same canonical PDP
          '/*?filter=',
          '/*?utm_',
        ],
      },
      { userAgent: 'GPTBot', allow: ['/blog/'], disallow: ['/'] },
    ],
    // Next emits the sitemap index here; it points at /sitemap/[id].xml children.
    sitemap: `${SITE}/sitemap.xml`,
    host: SITE,
  };
}

4. Canonical, hreflang, and social metadata

// app/products/[slug]/page.tsx
import type { Metadata } from 'next';
import { notFound } from 'next/navigation';
import { getProductBySlug } from '@/lib/products';

export const revalidate = 3600;

const clamp = (s: string, n = 155) => (s.length <= n ? s : `${s.slice(0, n - 1).trimEnd()}\u2026`);

export async function generateMetadata(
  { params }: { params: Promise<{ slug: string }> },
): Promise<Metadata> {
  const { slug } = await params; // Next 15: params is a Promise
  const product = await getProductBySlug(slug);

  // Soft-404s waste crawl budget. Return a real 404 instead of a 200 shell.
  if (!product) notFound();

  const canonical = `/products/${product.slug}`;
  const ogImage = `/og/products/${product.slug}.png`;
  const thumb = clamp(product.summary);

  return {
    title: `${product.name} - ${product.brand} | Tarek Store`,
    description: thumb,
    alternates: {
      canonical, // metadataBase resolves this to an absolute URL
      languages: {
        'en-US': canonical,
        'bn-BD': `/bn/products/${product.slug}`,
        'x-default': canonical,
      },
    },
    // Never combine index:false with a cross-canonical: contradictory signals.
    robots: product.thin
      ? { index: false, follow: true }
      : { index: true, follow: true, 'max-image-preview': 'large', 'max-snippet': -1 },
    openGraph: {
      type: 'website',
      url: canonical,
      title: product.name,
      description: thumb,
      images: [{ url: ogImage, width: 1200, height: 630, alt: `${product.name} product photo` }],
    },
    twitter: { card: 'summary_large_image', images: [ogImage] },
  };
}

5. Edge OpenGraph renderer

// app/og/products/[slug]/route.tsx
import { ImageResponse } from 'next/og';
import { getProductBySlug } from '@/lib/products';

export const runtime = 'edge';
export const revalidate = false; // immutable per slug; purge with revalidatePath instead
export const alt = 'Product card';
export const size = { width: 1200, height: 630 };
export const contentType = 'image/png';

export async function GET(
  request: Request,
  { params }: { params: Promise<{ slug: string }> },
) {
  const { slug } = await params;
  const product = await getProductBySlug(slug);
  if (!product) return new Response('Not found', { status: 404 });

  // Satori accepts TTF/OTF ArrayBuffers only. WOFF2 will fail at rasterization.
  const fontRes = await fetch(new URL('/fonts/Inter-Bold.ttf', request.url));
  if (!fontRes.ok) return new Response('Font unavailable', { status: 503 });
  const fontData = await fontRes.arrayBuffer();

  return new ImageResponse(
    (
      // Satori CSS subset: flexbox only, no grid. Every div with 2+ children needs display:flex.
      <div
        style={{
          display: 'flex',
          flexDirection: 'column',
          justifyContent: 'space-between',
          width: '100%',
          height: '100%',
          backgroundColor: '#0b0f14',
          padding: '64px',
          color: '#ffffff',
          fontFamily: 'Inter',
        }}
      >
        <div style={{ display: 'flex', fontSize: 26, color: '#7dd3fc' }}>tareksec.dev</div>
        <div style={{ display: 'flex', flexDirection: 'column', gap: '20px' }}>
          <div style={{ fontSize: 76, fontWeight: 700, lineHeight: 1.05, letterSpacing: '-0.02em' }}>
            {product.name}
          </div>
          <div style={{ fontSize: 32, color: '#94a3b8' }}>{product.brand}</div>
        </div>
        <div style={{ display: 'flex', justifyContent: 'space-between', fontSize: 28 }}>
          <div style={{ color: '#38bdf8' }}>{`$${product.price.toFixed(2)}`}</div>
          <div style={{ color: '#64748b' }}>{product.sku}</div>
        </div>
      </div>
    ),
    { ...size, fonts: [{ name: 'Inter', data: fontData, weight: 700, style: 'normal' }] },
  );
}

6. Linked JSON-LD entity graph

// components/ProductJsonLd.tsx
import type { Product } from '@prisma/client';

type Crumb = { name: string; path: string };

const BASE = process.env.NEXT_PUBLIC_SITE_URL ?? 'https://tareksec.dev';

export function ProductJsonLd({ product, breadcrumbs }: { product: Product; breadcrumbs: Crumb[] }) {
  const productId = `${BASE}/products/${product.slug}`;

  const graph = {
    '@context': 'https://schema.org',
    '@graph': [
      {
        '@type': 'Organization',
        '@id': `${BASE}/#organization`, // stable fragment: never make this route-dependent
        name: 'Tarek Store',
        url: BASE,
        logo: { '@type': 'ImageObject', url: `${BASE}/logo.png`, width: 512, height: 512 },
      },
      {
        '@type': 'WebSite',
        '@id': `${BASE}/#website`,
        url: BASE,
        publisher: { '@id': `${BASE}/#organization` },
        potentialAction: {
          '@type': 'SearchAction',
          target: { '@type': 'EntryPoint', urlTemplate: `${BASE}/search?q={search_term_string}` },
          'query-input': 'required name=search_term_string',
        },
      },
      {
        '@type': 'Product',
        '@id': `${productId}#product`,
        name: product.name,
        sku: product.sku,
        description: product.summary,
        image: [`${BASE}/og/products/${product.slug}.png`],
        brand: { '@type': 'Brand', name: product.brand },
        isPartOf: { '@id': `${BASE}/#website` },
        offers: {
          '@type': 'Offer',
          '@id': `${productId}#offer`,
          url: productId,
          priceCurrency: 'USD',
          price: product.price.toFixed(2),
          availability:
            product.stock > 0
              ? 'https://schema.org/InStock'
              : 'https://schema.org/OutOfStock',
        },
        // Emit only when the rating is real, user-generated and visible on-page.
        aggregateRating:
          product.reviewCount >= 5 && product.rating > 0
            ? {
                '@type': 'AggregateRating',
                ratingValue: product.rating.toFixed(1),
                reviewCount: product.reviewCount,
              }
            : undefined,
        subjectOf: { '@id': `${productId}#breadcrumb` },
      },
      {
        '@type': 'BreadcrumbList',
        '@id': `${productId}#breadcrumb`,
        itemListElement: breadcrumbs.map((crumb, index) => ({
          '@type': 'ListItem',
          position: index + 1,
          name: crumb.name,
          item: `${BASE}${crumb.path}`,
        })),
      },
    ],
  };

  // Escape angle brackets and ampersands: JSON.stringify alone does not make this XSS-safe.
  const json = JSON.stringify(graph)
    .replace(/</g, '\\u003c')
    .replace(/>/g, '\\u003e')
    .replace(/&/g, '\\u0026');

  return <script type='application/ld+json' dangerouslySetInnerHTML={{ __html: json }} />;
}

Performance Benchmarks & Trade-Offs

Sitemap generation strategy, measured on 1.24M programmatic URLs against a Postgres 16 primary with a read replica and Vercel Edge Network:

| Strategy | p95 build/refresh | Crawl-budget hygiene | Cache invalidation correctness | Ops cost | Verdict | |---|---|---|---|---|---| | Hand-written static sitemap.xml in /public | n/a (stale) | Poor (breaches 50k cap) | Manual redeploy | Very low | Never for programmatic | | next-sitemap post-build script | 38s full rebuild | Good | Rebuild-on-publish only | Medium | Fine under 100k URLs | | Single app/sitemap.ts function | 4.2s cold / 900ms warm | Good | ISR hourly + on-demand | Low | Hard-fails past 100k URLs | | generateSitemaps() shards + ISR (recommended) | 180ms per cached shard, 1.1s cold | Excellent | Per-shard revalidate | Low | Winner | | Custom streamed sitemap index at the edge | 95ms TTFB, unbounded | Excellent | Manual purge required | High | Only above ~10M URLs |

OpenGraph renderer comparison for a 1200x630 card:

| Renderer | p50 | p95 cold | Output size | Runtime | Notes | |---|---|---|---|---|---| | Satori + resvg via next/og | 210ms | 480ms | ~62KB PNG | Edge | Flexbox-only CSS, no grid | | Headless Chromium (Puppeteer) | 1.4s | 3.9s | ~190KB PNG | Node | Full CSS, painful cold starts | | Pre-rendered static PNGs | 12ms (CDN) | 40ms | ~62KB PNG | Object storage | No per-entity variation | | node-canvas | 90ms | 260ms | ~70KB PNG | Node | Manual layout, no JSX |

Pro-Tips & Common Pitfalls

  • [ ] Never `Disallow` a URL you also want `noindex`. Googlebot must be able to fetch the page to read the meta robots tag. Blocked-plus-noindexed URLs stay indexed as bare URLs.
  • [ ] Never emit `noindex` and a cross-canonical on the same URL. Pick one consolidation signal per URL class.
  • [ ] Never let `priority` and `changeFrequency` drive architecture decisions. Google ignores both.
  • [ ] Never generate `lastmod` from `new Date()`. One dishonest timestamp and Google stops trusting the field site-wide.
  • [ ] Never render sitemaps at request time. They must be static/ISR; a slow sitemap throttles your whole crawl pipeline.
  • [ ] Never return 200 for an empty product page. Call notFound() and serve a real 404, or a 410 for permanently retired slugs.
  • [ ] Always validate the JSON-LD payload against the Rich Results Test and the Merchant listings report after every template change. A single unescaped character breaks the whole graph silently.
  • [ ] Always self-reference hreflang. Every language variant, including the default, must link back to all others; missing return links invalidate the cluster.
  • [ ] Always cache OG images immutably, then purge with revalidatePath on data mutation. Regenerating them per request is the fastest route to edge CPU exhaustion.
  • [ ] Push to IndexNow (Bing, Yandex) on publish. Google discovers via sitemaps, but Bing responds to IndexNow in minutes rather than days.
  • [ ] Track `X-Robots-Tag` on non-HTML assets (PDFs, spec sheets, large images). robots.txt alone does not control indexation of these.
  • [ ] Monitor crawl requests by URL pattern in log analysis, not just Search Console. GSC Crawl Stats lags by 2-3 days and hides which parameter exploded.

Frequently Asked Questions (FAQ)

How many URLs can a Next.js sitemap handle before it breaks?

A single sitemap file is capped at 50,000 URLs or 50MB uncompressed by the protocol, and a sitemap index at 50,000 child files. Practically, a single app/sitemap.ts function stops being viable well before that limit because it must fetch and serialize every row in one function invocation — we saw p95 at 4.2s cold and memory pressure above ~300k rows. The correct pattern is generateSitemaps() with 45,000-URL shards behind ISR, which served 1.24M URLs at 180ms p95 per cached shard.

Do canonical tags and robots noindex conflict in Next.js metadata?

Yes, when they point at different URLs. robots: { index: false } plus alternates.canonical pointing to a different page sends contradictory consolidation signals, and Google resolves the conflict unpredictably — usually by dropping the canonical signal entirely. Use noindex, follow for thin pages and self-canonical for indexable ones, never both mechanisms on the same URL. Also remember that noindex only works if the URL is fetchable: if robots.ts also disallows it, the directive is never read.

Is edge OpenGraph image generation fast enough for a million pages?

Yes, provided you make the images immutable. Satori plus resvg renders a 1200x630 PNG in roughly 210ms at p50 and 480ms at p95 on a cold edge isolate. Since OG images keyed to an immutable slug never change, setting revalidate = false moves them to permanent CDN cache, where repeat fetches complete in under 15ms. The failure mode is regenerating per request behind a query string — that pins edge CPU and triggers rate limits. Purge deliberately with revalidatePath when the underlying product data changes.