Company
Anthropic
Type
Training crawl
Obeys robots.txt
Yes
Announces itself as
ClaudeBot
Official documentation
https://support.anthropic.com/en/articles/8896518
What it does
Anthropic's crawler that collects public web data to train the Claude family of models. Obeys robots.txt.
How Forge AI Bot Log checks it
Visits are checked against the IP ranges Anthropic publishes, so a spoofed ClaudeBot is labelled as such.
Should you block it?
Block it if you do not want your pages used for training. Claude can still read your pages live for a user through Claude-User. Blocking ClaudeBot takes one switch in Forge AI Bot Log's Access Control screen. The plugin adds the rule to your robots.txt for you.