Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

15.2 docs: document missing htmlLimitedBots option #76616

Merged
merged 2 commits into from
Feb 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions docs/01-app/04-api-reference/04-functions/generate-metadata.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1197,31 +1197,38 @@ export const metadata = {
}
```

#### Streaming Metadata
## Streaming metadata

Starting from v15.2, metadata returned by `generateMetadata` will be streamed to the client. This allows Next.js to inject metadata into the HTML as soon as it's resolved.
Metadata returned by `generateMetadata` is streamed to the client. This allows Next.js to inject metadata into the HTML as soon as it's resolved.

Since page metadata often primarily targets bots & crawlers, Next.js will continue to block the render until the metadata is resolved for **HTML-limited bots**.

Some bots, like `Googlebot`, can execute JavaScript and are able to inspect the full page DOM, meaning they **don't** require blocking metadata. However, bots like `Twitterbot` **cannot** execute JavaScript while crawling a page—they fall into the **HTML-limited** category.
Since page metadata primarily targets bots and crawlers, Next.js will stream metadata for bots that can execute JavaScript and inspect the full page DOM (e.g. `Googlebot`). However, metadata will continue blocking the render of the page for **HTML-limited** bots (e.g. `Twitterbot`) as these cannot execute JavaScript while crawling.

Next.js automatically detects the user agent of incoming requests to determine whether to serve streaming metadata or fallback to blocking metadata.

If you need to customize this list, you can define them manually using the `htmlLimitedBots` option in `next.config.js`. Next.js will ensure user agents matching this regex receive blocking metadata when requesting your web page.
Specifying a `htmlLimitedBots` config will override the Next.js' default list, allowing you full control over what user agents should opt into this behavior. This is advanced behavior, and the default should be sufficient for most cases.

```js filename="next.config.js"
```ts filename="next.config.ts" switcher
import type { NextConfig } from 'next'

const config: NextConfig = {
htmlLimitedBots: 'MySpecialBot|MyAnotherSpecialBot|SimpleCrawler',
}

export default config
```

```js filename="next.config.js" switcher
module.exports = {
htmlLimitedBots: 'MySpecialBot|MyAnotherSpecialBot|SimpleCrawler',
}
```

> **Note:** Next.js includes [a default list of HTML limited bots](https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/router/utils/html-bots.ts)
Specifying a `htmlLimitedBots` config will override the Next.js' default list, allowing you full control over what user agents should opt into this behavior. This is advanced behavior, and the default should be sufficient for most cases.

## Version History

| Version | Changes |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `v15.2.0` | introduced streaming support to `generateMetadata`. |
| `v15.2.0` | Introduced streaming support to `generateMetadata`. |
| `v13.2.0` | `viewport`, `themeColor`, and `colorScheme` deprecated in favor of the [`viewport` configuration](/docs/app/api-reference/functions/generate-viewport). |
| `v13.2.0` | `metadata` and `generateMetadata` introduced. |
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
title: htmlLimitedBots
description: Specify a list of user agents that should receive blocking metadata.
---

The `htmlLimitedBots` config allows you to specify a list of user agents that should receive blocking metadata instead of [streaming metadata](/docs/app/api-reference/functions/generate-metadata#streaming-metadata).

```ts filename="next.config.ts" switcher
import type { NextConfig } from 'next'

const config: NextConfig = {
htmlLimitedBots: 'MySpecialBot|MyAnotherSpecialBot|SimpleCrawler',
}

export default config
```

```js filename="next.config.js" switcher
module.exports = {
htmlLimitedBots: 'MySpecialBot|MyAnotherSpecialBot|SimpleCrawler',
}
```

## Default list

Next.js includes [a default list of HTML limited bots](https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/router/utils/html-bots.ts).

Specifying a `htmlLimitedBots` config will override the Next.js' default list, allowing you full control over what user agents should opt into this behavior. However, this is advanced behavior, and the default should be sufficient for most cases.

## Version History

| Version | Changes |
| ------- | ------------------------------------ |
| 15.2.0 | `htmlLimitedBots` option introduced. |
Loading