Company
Diffbot
Type
Training crawl
Obeys robots.txt
Yes
Announces itself as
Diffbot
Official documentation
https://www.diffbot.com/dev/docs/
What it does
A crawler that builds a structured 'knowledge graph' of the web, which is sold and used as training and grounding data for AI systems.
How Forge AI Bot Log checks it
Diffbot publishes no way to check this bot, so visits are recorded as No check rather than guessed at.
Should you block it?
Block it if you do not want your pages resold as AI data. Blocking Diffbot takes one switch in Forge AI Bot Log's Access Control screen. The plugin adds the rule to your robots.txt for you.