Company
Common Crawl
Type
Training crawl
Obeys robots.txt
Yes
Announces itself as
CCBot
Official documentation
https://commoncrawl.org/ccbot
What it does
The Common Crawl crawler. Its open dataset is one of the most widely used sources for training AI models across the industry, so a CCBot visit can feed many downstream models.
How Forge AI Bot Log checks it
Common Crawl publishes no way to check this bot, so visits are recorded as No check rather than guessed at.
Should you block it?
Block it if you want to keep your pages out of the most widely shared training dataset. Blocking CCBot takes one switch in Forge AI Bot Log's Access Control screen. The plugin adds the rule to your robots.txt for you.