First, decide what you want
AI bots do three different jobs, and blocking is not all-or-nothing.
- Training crawls collect your pages to train models. Nothing comes back to you. Many sites choose to block these.
- Search crawls index your pages so an AI search product can find and link them later. Blocking them removes you from that product.
- Live reads happen when a person asks an assistant something and it reads your page to answer. These are the visits that turn into visitors sent to your site. Blocking them means the assistant cannot read your page for that person.
A common choice: block training crawls, allow search crawls and live reads. Forge AI Bot Log labels every bot by type so you can decide per bot.
Way 1: robots.txt
robots.txt is a public file on your site that asks bots to stay away. Well-behaved bots follow it, which includes GPTBot, ClaudeBot, OAI-SearchBot, Amazonbot and most others. In Forge AI Bot Log, flip the switch beside a bot on the Access Control screen and the plugin adds the rule to your robots.txt for you.
For Google and Apple, the training opt-out is a separate name. Blocking Google-Extended or Applebot-Extended stops training use without touching Google Search, Siri or Spotlight.
Way 2: a server-level block
Some bots ignore robots.txt. Bytespider is the best-known; Perplexity-User and Grok’s fetchers may also ignore it. For these, the only effective block is at the web server. Forge AI Bot Log writes an Apache .htaccess snippet for any blocked bot known to ignore robots.txt. Paste it at the top of your .htaccess file, or give it to your host. On Nginx, ask your host to add the equivalent rule.
What blocking cannot do
- It cannot remove pages already collected.
- It cannot stop a bot that uses a disguised browser name. Some of Grok’s fetching works this way.
- It cannot block Google AI Overviews without also blocking Google Search, because both use Googlebot.
Check it worked
After blocking, the bot’s visits should stop appearing in the activity log within a day or two. A well-behaved bot re-reads robots.txt regularly. If visits continue from a bot that should obey robots.txt, check that your robots.txt is actually served: open yoursite.com/robots.txt in a browser and look for the rule.