SEO & Metadata

Comprehensive SEO optimization with rich metadata, Open Graph images, and search engine discoverability.

Overview

AI Web Feeds implements Next.js Metadata API for:

Dynamic OG Images Custom Open Graph images for every page

Rich Metadata Complete SEO tags for all content

Social Sharing Optimized for Twitter, LinkedIn, Slack

AI-Friendly Special rules for AI crawlers

Features

✨ What's Included

Dynamic OG Images - Unique images generated for each documentation page
Rich Metadata - Complete title, description, keywords, and author information
Twitter Cards - Summary cards with large images
Canonical URLs - Proper canonical link tags
Structured Data - JSON-LD for better search results
Sitemap - Auto-generated XML sitemap
Robots.txt - Search engine crawling rules with AI bot support
PWA Manifest - Progressive Web App configuration
RSS Discovery - Feed links in HTML head

Open Graph Images

Dynamic Page Images

Every documentation page gets a unique OG image:

lib/source.ts

export function getPageImage(page: InferPageType<typeof source>) {
  const segments = [...page.slugs, "image.png"];

  return {
    segments,
    url: `/og/docs/${segments.join("/")}`,
  };
}

Image Design

Custom-designed OG images with:

Dark theme - Modern dark background with gradient accents
Brand identity - Logo and site name
Page title - Large, readable typography
Description - Supporting text for context
Category badge - Visual categorization
Site URL - Domain attribution

Example URLs

/og/docs/image.png

Main documentation landing page OG image

/og/docs/features/pdf-export/image.png

PDF Export feature page OG image

/og/docs/guides/quick-reference/image.png

Quick Reference guide OG image

Image Specifications

Property	Value
Width	1200px
Height	630px
Format	PNG
Generation	Static at build time
Caching	Permanent (`revalidate = false`)

Metadata Structure

Root Layout

Site-wide metadata in app/layout.tsx:

export const metadata: Metadata = {
  metadataBase: new URL(baseUrl),
  title: {
    default: 'AI Web Feeds - RSS/Atom Feeds for AI Agents',
    template: '%s | AI Web Feeds',
  },
  description: 'Curated RSS/Atom feeds optimized for AI agents...',
  keywords: ['AI', 'RSS feeds', 'Atom feeds', ...],
  authors: [{ name: 'Wyatt Walsh', url: '...' }],
  openGraph: {
    type: 'website',
    locale: 'en_US',
    url: baseUrl,
    siteName: 'AI Web Feeds',
    images: [{ url: '/og-image.png', width: 1200, height: 630 }],
  },
  twitter: {
    card: 'summary_large_image',
    creator: '@wyattowalsh',
  },
  robots: {
    index: true,
    follow: true,
    googleBot: {
      'max-video-preview': -1,
      'max-image-preview': 'large',
      'max-snippet': -1,
    },
  },
};

Documentation Pages

Dynamic metadata for each page in app/docs/[[...slug]]/page.tsx:

export async function generateMetadata(
  props: PageProps
): Promise<Metadata> {
  const page = source.getPage(params.slug);

  return {
    title: page.data.title,
    description: page.data.description,
    keywords: ['documentation', 'AI', ...],
    openGraph: {
      type: 'article',
      title: page.data.title,
      url: pageUrl,
      images: [{ url: imageUrl, width: 1200, height: 630 }],
    },
    twitter: {
      card: 'summary_large_image',
      title: page.data.title,
      images: [imageUrl],
    },
    alternates: {
      canonical: pageUrl,
      types: {
        'application/rss+xml': '/docs/rss.xml',
        'application/atom+xml': '/docs/atom.xml',
      },
    },
  };
}

Sitemap

Auto-generated XML sitemap at /sitemap.xml:

app/sitemap.ts

export default function sitemap(): MetadataRoute.Sitemap {
  const pages = source.getPages();

  return pages.map((page) => ({
    url: `${baseUrl}${page.url}`,
    lastModified: new Date(),
    changeFrequency: "weekly",
    priority: 0.8,
  }));
}

Sitemap Features

✅ All documentation pages included
✅ Proper priority levels
✅ Change frequency hints
✅ Last modified dates
✅ Auto-updates on build

Access Sitemap

curl https://ai-web-feeds.vercel.app/sitemap.xml

Robots.txt

Custom robots.txt with AI crawler support:

Generated robots.txt

User-agent: *
Allow: /
Disallow: /api/
Disallow: /_next/
Disallow: /static/

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://ai-web-feeds.vercel.app/sitemap.xml

AI Crawler Support

Explicitly allows common AI crawlers:

GPTBot - OpenAI's web crawler
ChatGPT-User - ChatGPT browsing
Google-Extended - Google's AI training crawler
anthropic-ai - Anthropic's crawler
ClaudeBot - Claude's web crawler

PWA Manifest

Progressive Web App configuration:

Generated manifest.json

{
  "name": "AI Web Feeds - RSS/Atom Feeds for AI Agents",
  "short_name": "AI Web Feeds",
  "description": "Curated RSS/Atom feeds optimized for AI agents",
  "start_url": "/",
  "display": "standalone",
  "background_color": "#0a0a0a",
  "theme_color": "#667eea",
  "icons": [
    {
      "src": "/icon-192.png",
      "sizes": "192x192",
      "type": "image/png"
    },
    {
      "src": "/icon-512.png",
      "sizes": "512x512",
      "type": "image/png"
    }
  ]
}

How It Looks

When shared on social media, links display:

Twitter/X

Large image (1200x630)
Page title
Description
Site name
Creator handle

LinkedIn

Large image
Page title
Description
Site URL

Slack/Discord

Rich embed with image
Title and description
Site information

Use Twitter Card Validator:

Enter page URL
Click "Preview card"
Verify image and text

Use LinkedIn Post Inspector:

Enter page URL
Click "Inspect"
Review preview

Use Facebook Sharing Debugger:

Enter page URL
Click "Debug"
Scrape again if needed

Search Engine Optimization

Google Search Features

Optimized for:

Rich snippets - Enhanced search results
Knowledge graph - Structured data integration
Image preview - Large image thumbnails
Site links - Auto-generated navigation
Breadcrumbs - Clear page hierarchy

Verification

Add verification codes in app/layout.tsx:

verification: {
  google: 'your-google-verification-code',
  yandex: 'your-yandex-verification-code',
  bing: 'your-bing-verification-code',
}

Implementation

File Structure

app/
├── layout.tsx              # Root metadata
├── manifest.ts             # PWA manifest
├── robots.ts               # Robots.txt
├── sitemap.ts              # XML sitemap
├── og-image.png/
│   └── route.tsx          # Homepage OG image
├── (home)/
│   └── page.tsx           # Homepage metadata
├── docs/
│   └── [[...slug]]/
│       └── page.tsx       # Dynamic page metadata
└── og/
    └── docs/
        └── [...slug]/
            └── route.tsx  # Dynamic OG images

lib/
└── source.ts              # getPageImage helper

Key Functions

Get Page Image URL

const image = getPageImage(page);
// { segments: ['features', 'pdf-export', 'image.png'],
//   url: '/og/docs/features/pdf-export/image.png' }

Generate Metadata

export async function generateMetadata(props): Promise<Metadata> {
  const page = source.getPage(params.slug);
  return {
    title: page.data.title,
    openGraph: { images: getPageImage(page).url },
  };
}

Best Practices

1. Title Templates

Use templates for consistent branding:

title: {
  default: 'AI Web Feeds',
  template: '%s | AI Web Feeds',
}

Results in:

Homepage: "AI Web Feeds"
Docs page: "Getting Started | AI Web Feeds"

2. Description Length

Keep descriptions under 160 characters:

description: "Clear, concise description under 160 characters";

3. Image Optimization

Use 1200x630 for OG images (1.91:1 ratio)
Keep file sizes under 1MB
Use high-contrast text
Test on multiple platforms

4. Canonical URLs

Always set canonical URLs:

alternates: {
  canonical: pageUrl,
}

5. Keywords

Include relevant keywords:

keywords: ["specific", "relevant", "keywords"];

Troubleshooting

OG Images Not Showing

OG images are generated at build time. Rebuild after changes: bash pnpm build

If old images persist:

Clear platform cache using their debug tools
Add query parameter: ?v=2 to force refresh
Wait 24-48 hours for automatic cache expiry

Missing Metadata

Check browser dev tools:

# View page source
curl https://ai-web-feeds.vercel.app/docs | grep -i "og:"
curl https://ai-web-feeds.vercel.app/docs | grep -i "twitter:"

Expected tags:

<meta property="og:title" content="..." />
<meta property="og:description" content="..." />
<meta property="og:image" content="..." />
<meta name="twitter:card" content="summary_large_image" />

Testing

Verify Metadata

Check HTML Head

View page source and verify tags:

curl https://ai-web-feeds.vercel.app/docs | head -100

Test OG Images

Visit image URLs directly:

/og-image.png
/og/docs/image.png
/og/docs/features/pdf-export/image.png

Validate Sitemap

curl https://ai-web-feeds.vercel.app/sitemap.xml

Check Robots.txt

curl https://ai-web-feeds.vercel.app/robots.txt

SEO Audit Tools

Performance

Build-Time Generation

All OG images generated during build:

Development: Images generated on-demand
Production: All images pre-rendered
Caching: Permanent (revalidate = false)

Size Optimization

Asset	Size
OG Image (PNG)	~50-100KB
Sitemap XML	~5-10KB
Manifest JSON	~1KB
Robots.txt	~500B

RSS Feeds - Feed discovery and metadata
AI Integration - AI crawler support
Quick Reference - Metadata endpoints

SEO & Metadata

On this page