If you’d like to block bots from accessing your site or site pages, you can generate a robots.txt file which limits the pages that search engines crawl. However, you make want to take a more exact approach by blocking bad bots from your site or site pages. This is useful especially when bad bots are causing unwanted increases in your site’s bandwidth usage.
Note that you’ll need a Site plan or paid Workspace plan to add a robots.txt file to your site.
A robots.txt file instructs bots (also known as robots, spiders, or web crawlers) on how they should interact with your site. You can add rules to manage bot access to specific pages, folders, or your entire site. It's typically used to list pages or folders on your site that you don't want search engines to crawl or index.
Just like a sitemap, the robots.txt file lives in the top-level directory of your domain, e.g., yourdomain.com/robots.txt.
To generate a robots.txt file:
Important
\nNot all bots follow the rules specified in your robots.txt file, especially malicious or poorly configured ones. As a result, these bots may still access your site, including restricted folders and pages. Search the list of good/verified bots.
For instance, if you’re trying to block a bad bot named “ThisBot” your robots.txt file would look something like this:
User-agent: ThisBot\nDisallow: /\n\nSitemap: https://www.mysite.com/sitemap.xmlLearn more about creating a robots.txt file.
","user_segment_ids":[]}}