You open your server log, or your host sends a bandwidth warning, and one name is everywhere. ClaudeBot. Hundreds of lines, maybe thousands, and a quick search turns up stories of it hitting sites a million times in a day. Being alarmed is reasonable. Those stories were real.
This article explains what ClaudeBot is, shows what it actually did on two ordinary WordPress sites this month, minute by minute, and gives you three ways to deal with it in order of strength. It also covers the mistake most people make when they try to block it, which is blocking a name Anthropic said in 2024 it no longer uses.
The short answer
ClaudeBot is the bot Anthropic uses to collect web pages that may be used to train its Claude models. Anthropic says it obeys robots.txt. On the two sites we watched, it fetched the sitemap, read a batch of pages over a few minutes, and left.
If it is genuinely straining your server, ask it to slow down with a crawl-delay line first. If you do not want your pages in training data, block it with two lines in robots.txt. Either way, leave Claude-User and Claude-SearchBot alone, because those are the Anthropic bots that can send you readers.
If you only do one thing, check that the name in your robots.txt is ClaudeBot and not anthropic-ai or Claude-Web. Anthropic said in July 2024 that those two names were no longer in use, and a great many block lists still carry them.
What ClaudeBot did to two ordinary sites this month
The scary numbers come from 2024. In April that year the Linux Mint forums went down after ClaudeBot generated 20 times the traffic of the next worst bot, and the moderators blocked it at the firewall. In July, iFixit’s chief executive posted that ClaudeBot had hit the repair site about a million times in 24 hours.
Anthropic’s reply, given to 404 Media at the time, was that its crawling should not be disruptive and that it respects crawl-delay. iFixit added ClaudeBot to robots.txt and told The Register the crawling stopped, after which the bot came back only to re-read the robots.txt file.
Complaints have not vanished since. A September 2025 thread on r/webdev carried a headline much like this one.
Those were large sites with enormous numbers of pages. To see what ClaudeBot does to a site like yours, we looked at the activity logs on two WordPress sites we run, one a travel guide and one a news site. Both sit behind a full-page cache, and the news site had only nine days of data, so treat these as two worked examples, not a survey.
The travel site. Six minutes, then silence
In the 30 days to 24 September, ClaudeBot made 82 requests to the travel site. Almost all of them happened on one evening.
| When | What ClaudeBot fetched | Requests |
|---|---|---|
| 13 Sep, 17.31 | The sitemap index, twice | 2 |
| 13 Sep, 18.21 to 18.22 | The three sitemap parts | 3 |
| 13 Sep, 19.16 to 19.22 | Pages, one every four or five seconds | 75 |
| 16 Sep | The sitemap again, twice, three days later | 2 |
That is the whole month. It read the sitemap, then read 75 pages in six minutes at a steady pace, and did not come back for pages again. Every request came from an address on Anthropic’s published list. The site did not notice.
The news site. Short bursts, several times a day
The news site publishes many new pages a day, so ClaudeBot behaves differently there. In nine days it made 121 requests, arriving in bursts of four to eight pages within a minute or two, a few times a day. It fetched the sitemap on 18, 21 and 24 September, and robots.txt on 18 and 23 September. Again, every request was from an address on Anthropic’s list.
For comparison, in the same nine days the site recorded 8,852 AI bot visits in total, and 1,858 of those were Claude-User, the Anthropic bot that fetches a page when a real person asks Claude a question. ClaudeBot was one request in 73. Claude-User was one in five. Anthropic’s assistant sent five visitors to the news site in the period and none to the travel site.
These counts are a floor. Both sites serve most pages from a cache, and a request answered by the cache never reaches WordPress, so the plugin that recorded these numbers never saw it. The true totals are higher. Why caches hide bot visits, and two fixes.
What that means for you
On these two sites ClaudeBot was a modest, tidy visitor. It announced itself, read the sitemap, read pages at a pace of one every few seconds, and left. The load on a site with a few hundred pages is minutes, not days. A site with hundreds of thousands of pages will see the same pace for a lot longer, which is what happened to iFixit and the Linux Mint forums.
The bigger Anthropic presence on both sites was Claude-User, and that is people asking questions, not training. If your log is full of Anthropic, check which name it is before you decide anything.
Across the whole web the picture is similar. Cloudflare’s August 2025 analysis found that training accounted for nearly 80 percent of AI bot activity, and that for every visitor Anthropic sent to a site its bots had fetched about 38,000 pages in July 2025.
An analysis of Cloudflare’s public traffic data by TechnologyChecker put ClaudeBot at about 16 percent of AI bot traffic in July 2026, and Claude-User at about 8 percent of all bot traffic, the second busiest bot it measured.
What is ClaudeBot, and which two Anthropic bots get confused with it
Anthropic runs three bots, and its crawler support article describes each one. ClaudeBot collects web content that may contribute to training Claude models. Claude-User fetches a page when a person asks Claude something that needs it. Claude-SearchBot builds the index behind Claude’s search feature. Anthropic says all three obey robots.txt, and it lists what you lose by blocking each.
| Bot | Job | Block it and you lose |
|---|---|---|
| ClaudeBot | Training | Use of your pages in future training, per Anthropic |
| Claude-User | A person asked Claude | Visibility when people ask Claude |
| Claude-SearchBot | Search index | Presence in Claude’s search results |
ClaudeBot identifies itself with this user agent, the name a visitor sends with every request. This is how it appears in our logs.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)Claude-SearchBot is the newest of the three. TechnologyChecker’s reading of Cloudflare’s data first shows it in May 2026. If you have not seen it yet, you will.
ClaudeBot IP addresses, and how to check a visit is real
A name in a log proves nothing. Any script can call itself ClaudeBot, and in August 2026 a widely discussed Hacker News report described a campaign scanning sites for security holes while using the ClaudeBot and Googlebot names. On our news site a handful of requests calling themselves Claude-User came from New Zealand addresses that are not on Anthropic’s list at all. We checked them against the list by hand. Most likely someone’s own tool borrowing the name.
Anthropic publishes the addresses its bots use at claude.com/crawling/bots.json. On 24 September the list held 26 entries. One is a block of about a thousand addresses, everything from 216.73.216.0 to 216.73.219.255. Three are small blocks of 16 addresses each, and the rest are single addresses.
To check a visit by hand, open the list and search for the address. If it starts with 216.73.216, 216.73.217, 216.73.218 or 216.73.219 it is inside the block. Every ClaudeBot request on both our sites came from that block or from one of the single addresses.
Doing that for every line is tedious, which is why Forge AI Bot Log does it for you.
It fetches Anthropic’s list once a day and marks each Anthropic visit Verified when the address is on the list, Spoofed when the list was available and the address is not on it, or Unverified when the list could not be fetched, no recent copy is on hand, and no judgement is possible. It never calls a bot fake without evidence.
How to block ClaudeBot, or just slow it down
Three options, from gentlest to strongest. Start at the top.
- Ask it to slow down with a crawl-delay line.
- Block it with two lines in robots.txt.
- Refuse it at the server, only if the first two fail.
Slow it down with a crawl-delay
A crawl-delay line in robots.txt asks a bot to wait a number of seconds between requests. It is not part of the official robots.txt standard. Google ignores it, Bing honours it, and Anthropic says its bots respect it where appropriate and gives a two-line example in its documentation with a delay of one second. Here is a fuller version for WordPress, in a block that names ClaudeBot.
User-agent: ClaudeBot
Crawl-delay: 5
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpTwo things to know. First, once a bot has its own block in robots.txt it stops reading the general block, so repeat any rules you want it to keep, which is why the wp-admin lines are there. Second, on our sites ClaudeBot was already leaving four or five seconds between pages, so a delay of 1 changes nothing. The 5 in the example is a sensible starting point. Go higher only if your server is genuinely struggling.
WordPress generates robots.txt on the fly when your permalinks are set to anything other than Plain, so most sites have no file to edit. Add the lines through the robots.txt editor in your SEO plugin, or ask your developer to add a small piece of code called a filter. This one appends the block.
add_filter( 'robots_txt', function ( $output ) {
return $output . "\nUser-agent: ClaudeBot\nCrawl-delay: 5\nDisallow: /wp-admin/\nAllow: /wp-admin/admin-ajax.php\n";
} );Block ClaudeBot in robots.txt
If you do not want your pages used for training, this is the whole fix.
User-agent: ClaudeBot
Disallow: /iFixit reported that the crawling stopped once this was in place. Anthropic does not say how often ClaudeBot re-reads robots.txt. On our news site it read the file twice in nine days while allowed, and iFixit saw it every 30 minutes once blocked, so expect it to notice within a day or so. Clear your cache after the change, otherwise the bot may keep receiving the old file.
Blocking ClaudeBot does not block Claude-User or Claude-SearchBot. They have their own names and read your rules separately, so people asking Claude can still be sent to you. Our free plugin does the same thing with one switch on its Access Control screen, and it writes the lines into the robots.txt WordPress generates. If a real robots.txt file exists on your server, edit that instead, because a real file wins.
Block it at the server, only if robots.txt fails
If a bot calling itself ClaudeBot keeps coming after the robots.txt block, first check the addresses. If they are not on Anthropic’s list, it is not ClaudeBot, and the fix is a server rule that refuses the name. If they are on the list, email Anthropic at the address on its support page, from an address at your own domain, before doing anything drastic.
A server rule refuses the request before WordPress runs. On Apache hosts it goes in .htaccess, the settings file Apache reads before serving a page, above the line that reads “# BEGIN WordPress”. Back the file up first. The second line lets the bot keep reading robots.txt, so it can still learn that it is not welcome.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC]
RewriteRule .* - [F,L]
</IfModule>Hosts that use Nginx, a different web server, including Kinsta and Flywheel, ignore .htaccess. Ask the host to add the equivalent, or turn on Cloudflare’s own AI bot blocking if you use Cloudflare. The rule below also leaves robots.txt reachable.
set $deny_claudebot 0;
if ($http_user_agent ~* "ClaudeBot") { set $deny_claudebot 1; }
if ($request_uri = "/robots.txt") { set $deny_claudebot 0; }
if ($deny_claudebot) { return 403; }The 403 in both rules is the code a server sends to refuse a request.
Anthropic advises against blocking by address, and the reason is worth knowing. A bot that cannot reach your site cannot read your robots.txt either, so it never learns your rules. That is why the rules above block by name and leave robots.txt reachable.
The mistake most people make
In July 2024, 404 Media reported that large numbers of sites were blocking two Anthropic names, anthropic-ai and Claude-Web, while leaving ClaudeBot untouched. Anthropic told 404 Media and The Register that those two names were no longer in use, and that it had set ClaudeBot to respect any old rules written for them.
So old rules are honoured, according to Anthropic. But the name to use today is ClaudeBot, and if your robots.txt came from a copied list, open it and check. A list that names anthropic-ai and not ClaudeBot is relying on Anthropic’s stated handling of the old names rather than on a rule the bot reads directly.
Should you block ClaudeBot at all
That depends on how you feel about your pages being used to train models, because on the evidence above it costs you nothing you can measure today. Anthropic’s stated consequence of blocking ClaudeBot is exclusion from future training, and on our two sites the visitors Anthropic sent came through Claude-User, which is a different bot.
Our earlier piece on whether to block AI bots sets out the general rule. Block the training crawls, keep the search crawls and the live reads. ClaudeBot is a training crawl.
Check ClaudeBot has stopped
Open yoursite.com/robots.txt in a browser and make sure the ClaudeBot lines are there. Then watch the log for a day or two. Page requests from ClaudeBot should stop. Requests for robots.txt itself will continue, and that is the bot re-reading the file, not a failure. Claude-User and Claude-SearchBot will keep appearing, and if you followed the advice above that is exactly what you want.
If ClaudeBot page requests continue, look at the addresses. From Anthropic’s list, email them. From anywhere else, it was never ClaudeBot, and a server rule on the name will end it.
See every ClaudeBot visit, checked
The free plugin records each visit from ClaudeBot, Claude-User and Claude-SearchBot, checks the address against Anthropic’s list, labels each one by job, and gives you a switch per bot. Install it and you will know within a day whether ClaudeBot is a problem on your site or just a name in a log.