LLM bot insights

Use LLM bot insights to understand when and how LLM bots crawl your site.

LLM bot insights tracks which LLM bots visit your site and which pages they request. AI bots are part of how your content reaches users of tools like ChatGPT, Claude, and Gemini, but they don't show up in standard visitor analytics. LLM bot insights gives you a dedicated view of that activity, so you can see which AI tools are reaching your content and which pages they're prioritizing. Use it alongside Prompt insights to connect crawler activity with how your site appears in AI-generated answers.

Before you get started

LLM bot insights requires a Team or Enterprise Platform plan with the Analyze add-on.

How LLM bot insights works

Webflow collects bot data from your site's server logs. When an LLM bot requests a page on your site, Webflow surfaces that activity in LLM bot insights — organized by bot name, provider, purpose, and page path.

Because this data comes directly from server logs, it covers all LLM bot activity on your site, regardless of whether visitor tracking is enabled. You don't need to publish any changes to your site to start collecting bot data.

Access LLM bot insights

You can view LLM bot insights for your entire site or for specific pages.

To view LLM bot insights for your entire site:

Open your site in Webflow
Click Insights in the top bar
Go to Site overview
Choose the AI discovery tab
Scroll down to LLM bots

To view LLM bot insights for a specific page:

Open your site in Webflow
Click Insights in the top bar
Open Pages
Choose the page you want to view insights for
Choose the AI discovery tab
Scroll down to Bot traffic

Bot categories

Each bot tracked in LLM bot insights is grouped into one of four categories based on its known or likely purpose. Understanding the category helps you interpret what a bot's visit means for your site.

Agentic bots (shown as Agent in filters) request live content from your site to support AI-assisted experiences, such as answering a user's question. These visits help you understand which pages AI assistants are actively fetching from your site. For citation and prompt-level visibility, use Prompt insights alongside LLM bot insights.
Training bots collect or review web content for AI model development and related data pipelines. Seeing training bot activity on a recently updated page can help you understand when AI crawlers are reaching your content after you publish changes.
Indexing bots (shown as Indexer in filters) crawl pages to discover, map, or refresh content that AI tools may use for retrieval. Indexing bot activity can help you understand whether AI-related crawlers are reaching important pages, particularly ones you've recently published or updated.
Bots with multiple purposes (shown as Multiple in filters) are associated with more than one type of activity, or can't be confidently attributed to a single purpose. They only appear when you filter by Multiple or view the unfiltered list — not when you filter by Agent, Indexer, or Training.

LLM bots we track

LLM bot insights tracks the following bots:

Agentic bots

Bot	Provider
ChatGPT-User	OpenAI
Claude-User	Anthropic
Perplexity-User	Perplexity
Google-Agent	Google
Meta-ExternalFetcher	Meta
MistralAI-User	Mistral AI
DuckAssistBot	DuckDuckGo

Training bots

Bot	Provider
GPTBot	OpenAI
ClaudeBot	Anthropic
Meta-ExternalAgent	Meta
AmazonBot	Amazon
AI2Bot	Allen Institute for AI
Bytespider	ByteDance
CCBot	Common Crawl

Indexing bots

Bot	Provider
OAI-SearchBot	OpenAI
Claude-SearchBot	Anthropic
PerplexityBot	Perplexity
Meta-WebIndexer	Meta
MistralAI-Index	Mistral AI
DuckDuckBot	DuckDuckGo

Bots with multiple purposes

Bot	Provider
BingBot	Microsoft
GoogleBot	Google
AppleBot	Apple

Read your bot data

The AI discovery tab includes the LLM bots card, which shows which bots have visited your site and which pages they focused on. The card appears in both views: on Site overview, it shows bot activity across your whole site; on a specific page, it shows activity for just that page.

You can use the data to:

Understand which pages AI bots prioritize — bots often focus on different pages than your human visitors do
Track crawl activity over time — spot increases or drops in bot visits after you publish changes to your pages
Find out if you're tracking the right prompts — compare the pages bots crawl most with the owned domain citations in Prompt insights
Check whether new content is being discovered — indexing bot activity on a recently published page can indicate that AI-related crawlers have found it

Note

LLM bot insights shows when tracked AI bots request pages on your site. It doesn’t show the user’s original query, or confirm whether a page was cited or if your brand was mentioned in an AI-generated answer. To understand prompt-level visibility and citations, use Prompt insights alongside LLM bot insights.

Bot data comes from server-side logs and doesn’t require Webflow Analyze visitor tracking to be enabled. AI-referred visitor data is separate and requires Analyze tracking to be turned on.

Filter and export data

You can adjust the time range of your data using the dropdown in the top right of the AI discovery tab. The LLM bots card has two additional dropdowns to help you narrow your data further:

All LLMs — filter by a specific provider
All categories — filter by bot category (Agent, Indexer, Multiple, or Training)

To export data as a CSV, hover over the LLM bots card (or the Human visitors by source card) and click the “download” icon. The exported CSV data reflects your applied filters.

Good to know

You may see /manifest.json appearing frequently in your bot data — sometimes as one of the most-visited paths on your site. This is expected behavior and isn’t related to your content. Learn more about the manifest.json file in Webflow.

AI credits

LLM bot insights doesn't consume Webflow AI credits.

Learn more about Webflow AI credits.