Every bot below is one that Forge AI Bot Log recognises and checks. Each page explains who runs the bot, what it does with your pages, whether it obeys robots.txt, and how to block it if you want to. Two entries, Google-Extended and Applebot-Extended, are not bots at all; they are names you can use in robots.txt to opt out of AI training, and they are listed so the difference is clear.
Live read
| Bot | Company | Type | Obeys robots.txt |
|---|---|---|---|
| ChatGPT-User | OpenAI | Live read | Yes |
| Claude-User | Anthropic | Live read | Yes |
| Perplexity-User | Perplexity | Live read | Not always |
| meta-externalfetcher | Meta | Live read | Yes |
| MistralAI-User | Mistral AI | Live read | Yes |
| DuckAssistBot | DuckDuckGo | Live read | Yes |
| Grok-DeepSearch | xAI | Live read | Not always |
Search crawl
| Bot | Company | Type | Obeys robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search crawl | Yes |
| Claude-SearchBot | Anthropic | Search crawl | Yes |
| PerplexityBot | Perplexity | Search crawl | Yes |
| Applebot | Apple | Search crawl | Yes |
| YouBot | You.com | Search crawl | Yes |
| xAI-Grok | xAI | Search crawl | Not always |
Training crawl
| Bot | Company | Type | Obeys robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Training crawl | Yes |
| ClaudeBot | Anthropic | Training crawl | Yes |
| GoogleOther | Training crawl | Yes | |
| CCBot | Common Crawl | Training crawl | Yes |
| Bytespider | ByteDance | Training crawl | Not always |
| Amazonbot | Amazon | Training crawl | Yes |
| meta-externalagent | Meta | Training crawl | Yes |
| cohere-ai | Cohere | Training crawl | Yes |
| Diffbot | Diffbot | Training crawl | Yes |
| Timpibot | Timpi | Training crawl | Yes |
| GrokBot | xAI | Training crawl | Yes |
Opt-out name
| Bot | Company | Type | Obeys robots.txt |
|---|---|---|---|
| Google-Extended | Opt-out name | Yes | |
| Applebot-Extended | Apple | Opt-out name | Yes |