Prevent search engines from indexing pages, folders, your entire site, CMS items, or just your webflow.io subdomain.
You can control which pages search engines crawl on your site by writing a robots.txt file or by adding a noindex tag to certain pages. Then, you can prevent search engines from crawling and indexing specific pages, folders, your entire site, or your webflow.io subdomain. This is useful for hiding pages — like your site’s 404 page — from being indexed and listed in search results.
Important
Content from your site may still be indexed, even if it hasn’t been crawled. That happens when a search engine finds your content either because it was published previously, or there’s a link to that content from other content online. To ensure that a previously indexed page is not indexed, don’t add it in the robots.txt. Instead, use the Sitemap indexing toggle to remove that content from Google’s index. Additionally, you can use Google’s removals tool.
How to disable indexing of the Webflow subdomain
You can prevent Google and other search engines from indexing your site’s webflow.io subdomain by disabling indexing from your Site settings.
- Go to Site settings > SEO > Indexing section
- Set Staging indexing to off
- Click Save and publish your site
This will publish a unique robots.txt only on the subdomain that tells search engines to ignore this domain.
How to enable or disable indexing of site pages
There are two ways to disable indexing of site pages:
- By using the Sitemap indexing toggle in Page settings
- By generating a
robots.txt file
Note that if you disable indexing of a site page via a robots.txt file, the page will still be included in your site’s auto-generated sitemap (if you’ve enabled the sitemap). Additionally, if you’ve previously added a noindex tag to a site page via custom code, the page will still be included in your site’s auto-generated sitemap (unless you toggle Sitemap indexing to off).
How to disable indexing of site pages with the Sitemap indexing toggle
If you disable indexing of a static site page with the Sitemap indexing toggle, that page will no longer be indexed by search engines and will no longer be included in your site’s sitemap. You can only disable indexing with the toggle if you’ve enabled your site’s auto-generated sitemap.
Note
The Sitemap indexing toggle adds <meta content="noindex" name="robots"> to your site page. This prevents the page from being crawled and indexed by search engines.
To prevent search engines from indexing certain site pages:
- Go to the page you want to prevent search engines from indexing
- Go to Page settings > SEO settings
- Toggle Sitemap indexing to off
-
Publish your site
How to re-enable indexing of site pages with the Sitemap indexing toggle
To allow search engines to index certain site pages:
- Go to the page you want to allow search engines to index
- Go to Page settings > SEO settings
- Toggle Sitemap indexing to on
-
Publish your site
How to generate a robots.txt file
You can also generate a robots.txt file which instructs bots (also known as robots, spiders, or web crawlers) on how they should interact with your site. You can add rules to manage bot access to specific pages, folders, or your entire site. It's typically used to list pages or folders on your site that you don't want search engines to crawl or index.
How to bulk control bot traffic
You can also use the Traffic control toggles in Site settings > SEO > Indexing to auto-generate robots.txt rules for a group of known bots in each category.
Note
Blocking search engine or AI bots may reduce your site’s bandwidth consumption, but may also reduce your site’s visibility in traditional and AI-powered search experiences. Make sure to carefully consider your SEO and content discovery goals when you use traffic controls.
You can allow or disallow traffic from the following bots:
- Search engine crawlers (including
YandexBot, Googlebot, Bingbot, Baiduspider, OAI-SearchBot, Sogou, DuckDuckBot, MojeekBot, SeznamBot, Qwantify, Yeti, Slurp, and 360Spider)
- AI bots (including
meta-externalagent, ChatGPT-User, ClaudeBot, PerplexityBot, GPTBot, GoogleOther, Timpibot, SBIntuitionsBot, Bytespider, AI2Bot, Amazon Q, Claude-User, CCBot, Diffbot, YandexGPT, MistralBot, cohere-ai, youBot, and NeevaAI)
Traffic from search engine crawlers and AI bots is allowed by default. When you toggle these settings off, Webflow will request that bots not crawl or index your site. Note that you must publish your site for these settings to take effect.
Best practices for privacy
If you’d like to prevent the discovery of a particular page or URL on your site, don’t use the robots.txt to disallow the URL from being crawled. Instead, use either of the following options:
Important
Content from your site may still be indexed, even if it hasn’t been crawled. That happens when a search engine finds your content either because it was published previously, or there’s a link to that content from other content online. To ensure that a previously indexed page is not indexed, don’t add it in the robots.txt. Instead, use the Sitemap indexing toggle to remove that content from Google’s index. Additionally, you can use Google’s removals tool.
How to enable or disable indexing of a CMS item
You can also control whether specific CMS items are indexed by search engines. This is helpful if you want to exclude draft-like content or specialized landing pages from search results. Note that this feature is only available on paid Site plans.
To toggle search engine indexing for an individual CMS item:
- Go to your Collection in Webflow
- Open the CMS item you want to edit
- Go to Item details
- Toggle Sitemap Indexing on or off
- Save and publish your changes
When disabled, the CMS item will include a noindex meta tag, instructing search engines not to index it. This setting overrides the global site indexing setting for that specific item. The CMS item will also be excluded from your sitemap.
Note that changes to indexing settings may take time to reflect in search engine results Also, disabling indexing does not remove content already indexed — you can use Google's URL Removal Tool if necessary.
FAQ and troubleshooting tips
Can I use a robots.txt file to prevent my Webflow site assets from being indexed?
It’s not possible to use a robots.txt file to prevent Webflow site assets from being indexed because a robots.txt file must live on the same domain as the content it applies to (in this case, where the assets are served). Webflow serves assets from our global CDN, rather than from the custom domain where the robots.txt file lives. Learn more about asset and file privacy in Webflow.
I removed the robots.txt file from my Site settings, but it still shows up on my published site. How can I fix this?
Once the robots.txt has been created, it can’t be completely removed. However, you can replace it with new rules to allow the site to be crawled, e.g.:
User-agent: *
Disallow:
Make sure to save your changes and republish your site. If the issue persists and you still see the old robots.txt rules on your published site, please contact customer support.
How do I remove previously indexed pages from Google search?
You can use Google’s removals tool to remove previously indexed pages from Google search. Note that unless you also take steps to disable search engine indexing of these pages (e.g., using the Sitemap indexing toggle), they may be indexed by Google again.