Company
ByteDance
Type
Training crawl
Obeys robots.txt
Not always
Announces itself as
Bytespider
Official documentation
https://www.bytedance.com
What it does
ByteDance's crawler, associated with training its AI products. Widely reported to ignore robots.txt and to crawl aggressively, so a hard block is often the only effective control.
How Forge AI Bot Log checks it
ByteDance publishes no way to check this bot, so visits are recorded as No check rather than guessed at.
Should you block it?
Most sites block it. Use the server-level block; robots.txt alone is unlikely to work. Blocking Bytespider takes one switch in Forge AI Bot Log's Access Control screen. The plugin adds the rule to your robots.txt for you. Bytespider may not honour robots.txt, so the plugin also gives you a server-level block to paste into .htaccess.