inkreach-official-website/apps/website/.agents/skills/nuxt-seo/references/crawlability.md

Crawlability: Robots & Sitemap

Robots.txt

Auto-generated at /robots.txt. Respects site.indexable setting.

Configuration

// nuxt.config.ts
export default defineNuxtConfig({
  robots: {
    // Block AI crawlers
    blockAiBots: true,
    // Block non-SEO bots (reduces server load)
    blockNonSeoBots: true,
    // Custom rules
    groups: [
      { userAgent: '*', disallow: ['/admin'] }
    ]
  }
})

Per-Page Control

// Disable indexing
useRobotsRule('noindex, nofollow')

// Object syntax with AI directives
useRobotsRule({
  noindex: true,
  nofollow: true,
  noai: true,           // Block AI training
  noimageai: true,      // Block AI image training
  'max-snippet': 150,   // Preview controls
  'max-image-preview': 'large'
})

Route rules:

export default defineNuxtConfig({
  routeRules: {
    '/admin/**': { robots: 'noindex, nofollow' },
    '/hidden': { robots: false }
  }
})

Nuxt Content Frontmatter

---
robots: noindex, nofollow
# Or structured:
robots:
  noindex: true
  nofollow: true
---

Sitemap.xml

Auto-generated at /sitemap.xml from app routes.

Configuration

// nuxt.config.ts
export default defineNuxtConfig({
  sitemap: {
    sources: ['/api/__sitemap__/urls'],
    exclude: ['/admin/**', '/secret'],
    // For static sites - no runtime generation
    zeroRuntime: true
  }
})

Dynamic URLs via API

// server/api/__sitemap__/urls.ts
import { defineSitemapEventHandler } from '#imports'
import type { SitemapUrlInput } from '#sitemap/types'

export default defineSitemapEventHandler(async () => {
  const posts = await $fetch('/api/posts')
  return posts.map(post => ({
    loc: post.path,
    lastmod: post.updatedAt,
    // Image sitemap
    images: [{ loc: post.image, title: post.title }],
    // Video sitemap
    videos: [{ content_loc: post.videoUrl, title: post.title }]
  } satisfies SitemapUrlInput))
})

Per-Page Control

Route rules:

export default defineNuxtConfig({
  routeRules: {
    '/blog/**': { sitemap: { changefreq: 'daily', priority: 0.9 } },
    '/hidden': { sitemap: false }
  }
})

Nuxt Content frontmatter:

---
sitemap:
  changefreq: weekly
  priority: 0.8
  lastmod: 2025-01-15
---

Multiple Sitemaps

For large sites:

export default defineNuxtConfig({
  sitemap: {
    sitemaps: {
      pages: { include: ['/**'], exclude: ['/blog/**'] },
      blog: { include: ['/blog/**'] }
    }
  }
})

Generates /pages-sitemap.xml, /blog-sitemap.xml, and /sitemap_index.xml.

i18n Sitemaps

With @nuxtjs/i18n, auto-generates per-locale sitemaps with hreflang alternates.

Debug

In development:

• Robots: Check /robots.txt directly
• Sitemap: Visit /__sitemap__/debug.json for raw data