Two files control how search engines discover and crawl the site, robots.ts decides what they’re allowed to look at, and sitemap.ts tells them what exists. Both are generated automatically by Next.js at /robots.txt and /sitemap.xml, no manual file maintenance required.
app/robots.ts blocks crawlers from indexing account areas, internal tools, and anything that should never show up in search results.
export default function robots(): MetadataRoute.Robots {
return {
rules: {
userAgent: "*"
0 / 2,000 characters
Every path under disallow is account-specific, internal, or unfinished content, areas with no value in search results and that may expose information that shouldn’t be public. Add a new path here any time you build a feature that lives behind a login or is meant to stay internal.
disallow is a request, not a lock. Well-behaved crawlers respect it, but it
does not prevent access to a URL. It only asks search engines not to index it.
Malicious bots and direct URL access are not affected.
app/sitemap.ts lists every public URL on the site along with how often it changes and how important it is relative to other pages. Search engines use this to prioritize crawling and to discover pages that might not be linked from anywhere else.
The file is marked force-dynamic, meaning it queries the database fresh on every request rather than being generated once at build time. This matters because blog posts, categories, and authors change after the site is deployed.
export const dynamic = "force-dynamic"Marketing and legal pages are listed by hand, each with a changeFrequency and priority.
{ url, lastModified: new Date(), changeFrequency: "yearly", priority: 1 },
{ url: `${url}/pricing`, lastModified: new Date(), changeFrequency: "monthly", priority: 0.8 },
{ url: `${url}/blog`, lastModified: new Date(), changeFrequency: "daily", priority: 0.9 },priority is a number from 0 to 1, relative to other pages on the same site, not an absolute ranking signal search engines guarantee to honor. The home page is given 1, legal pages like terms and privacy are given 0.5, reflecting how often each is likely to matter to a new visitor.
Blog posts, categories, and authors are pulled from the database and mapped into sitemap entries automatically. Only published posts are included, and only authors with at least one published post are listed.
const posts = await db.query.post.findMany({
where: eq(post.published, true),
columns: { slug: true, updatedAt: true },
})
...posts.map((p) => ({
url: `${url}/blog/${p.slug}`,
lastModified: p.updatedAt,
changeFrequency: "weekly" as const,
priority: 0.8,
})),A post’s lastModified is its actual updatedAt timestamp from the database, not the current date, so search engines can tell when content genuinely changed rather than re-crawling unchanged pages on every visit.
Every page registered with fumadocs is included automatically, no manual entry needed when a new doc page is added.
const docsPages = source.getPages()
...docsPages.map((page) => ({
url: `${url}${page.url}`,
lastModified: new Date(),
changeFrequency: "monthly" as const,
priority: 0.7,
})),Note that docs pages use new Date() as their lastModified value rather than a real file modification timestamp. This means search engines will see all docs pages as updated on every crawl. It is a known tradeoff of sourcing docs from the filesystem without tracking file modification times.
Confirm the page is genuinely public. If it requires login, add its path to
disallow in robots.ts instead, and skip the remaining steps.
Open app/sitemap.ts and add a new static entry to the returned array,
following the existing entries as a template.
Choose a changeFrequency that reflects reality. Use daily only for pages
that genuinely change daily, like the blog index, overstating this wastes
crawl budget without earning more frequent crawls.
Choose a priority relative to existing pages. A new marketing page
typically sits between 0.6 and 0.8, below the home page and above legal
pages.
Pages backed by the database (posts, categories, authors) or by fumadocs (docs) do not need a manual entry, both are already mapped automatically.