If you’d like to block bots from accessing your site or site pages, you can generate a robots.txt file which limits the pages that search engines crawl. However, you make want to take a more exact approach by blocking bad bots from your site or site pages. This is useful especially when bad bots are causing unwanted increases in your site’s bandwidth usage.
Note that you’ll need a Site plan or paid Workspace plan to add a robots.txt file to your site.
How to generate a robots.txt file
A robots.txt file instructs bots (also known as robots, spiders, or web crawlers) on how they should interact with your site. You can add rules to manage bot access to specific pages, folders, or your entire site. It's typically used to list pages or folders on your site that you don't want search engines to crawl or index.
Just like a sitemap, the robots.txt file lives in the top-level directory of your domain, e.g., yourdomain.com/robots.txt.
To generate a robots.txt file:
- Go to Site settings > SEO > Indexing
- Add the robots.txt rule(s) you want
- Click Save and publish your site
Important
Not all bots follow the rules specified in your robots.txt file, especially malicious or poorly configured ones. As a result, these bots may still access your site, including restricted folders and pages. Search the list of good/verified bots.
How to block a bad bot
- Go to Site settings > SEO > Indexing section
- Add the robots.txt rule(s) you want
- Click Save changes and publish your site
For instance, if you’re trying to block a bad bot named “ThisBot” your robots.txt file would look something like this:
User-agent: ThisBot
Disallow: /ThisBot/
Sitemap: https://www.mysite.com/sitemap.xml
Learn more about creating a robots.txt file.