The AI tools that ask your site for markdown are the ones nobody told you about. The bots everyone tells you to serve it to have never asked. That is the finding from three sites that logged the request in 2026, and from a month of logs on a news site we run, and it changes what a site owner should do about the whole markdown question.
You have read that AI wants markdown, that Cloudflare will convert your pages on the fly, and that you should add a markdown copy of every page. Before you spend an afternoon on it, this article shows who actually supports content negotiation for markdown, what the big crawlers do instead, and how to tell in your own logs which kind of visitor you have.
The short answer
No. GPTBot, ChatGPT-User, ClaudeBot and PerplexityBot fetch HTML and do not send Accept text/markdown, the request that asks for markdown. Three sites that logged the header in 2026 agree. Coding tools do ask. Claude Code, Cursor, OpenCode and, by June 2026, GitHub Copilot Chat send it, and Claude Code alone made about a fifth of all Claude reads on our news site last month. The crawlers will fetch a markdown copy you link to. They do not ask for one.
| Who is asking | Asks for markdown? |
|---|---|
| GPTBot, ChatGPT-User, OAI-SearchBot | No. Fetch HTML. GPTBot follows a linked .md copy |
| ClaudeBot, Claude-User (the hosted assistant) | No. Fetch HTML |
| PerplexityBot, Perplexity-User | No. Fetch HTML |
| Claude Code, Cursor, OpenCode, OpenClaw | Yes, verified between November 2025 and May 2026 |
| Copilot Chat, Copilot CLI, Microsoft Copilot | Yes, verified June 2026 |
| OpenAI Codex CLI | Follows a linked .md copy, does not send the header |
| Gemini CLI, Windsurf, Devin, Cline, Aider | No |
What content negotiation for markdown is
Almost every request to your site carries a short note called the Accept header, in which the visitor says what kinds of file it can use. A browser says it wants HTML. A page can be offered in two forms at the same address, and the server picks by reading that note. That is content negotiation. It has been part of the web since the 1990s. What is new is a machine saying it would rather have markdown, the plain-text format that AI tools read cheaply.
What Cloudflare’s Markdown for Agents does
On 12 February 2026 Cloudflare launched Markdown for Agents. When a visitor sends Accept text/markdown, Cloudflare fetches your HTML, converts it at the edge and returns markdown, with a header estimating its size in tokens and a marker telling caches to keep the two versions apart. Its documentation says it is available on Pro, Business and Enterprise plans, must be switched on per site, only converts HTML, and only handles pages under 2 MB.
There is a second, older route. You publish a separate markdown copy of each page at its own address, usually the page address with .md on the end, and tell machines about it with a small tag in the HTML, a link marked as an alternate of type text/markdown. A crawler that reads the tag can fetch the copy. No negotiation is involved. Keep the two routes apart, because the crawlers treat them very differently.
Who sends Accept text/markdown
A week after Cloudflare’s launch, Checkly tested the question the simplest way possible. It told seven coding agents to fetch a page that echoes back the request headers, and read what each one sent. Claude Code asked for markdown first, then HTML, then anything. Cursor and OpenCode listed markdown first as well, with weightings behind it. OpenAI’s Codex and GitHub Copilot sent the ordinary browser-style request for HTML. Gemini CLI and Windsurf sent */*, which means anything. Three of seven asked for markdown.
A community status page, acceptmarkdown.com, keeps a dated list. As of its 22 June 2026 update it lists Claude Code (verified November 2025), Cursor and OpenCode (spring 2026), and Copilot Chat, Copilot CLI and Microsoft Copilot (verified 22 June 2026) as sending the header. Copilot did not send it in Checkly’s February test, and by June the status page recorded it doing so. Read that as a change in the product between those dates rather than a contradiction. It lists Codex CLI as following the alternate link tag instead, and twelve tools as fetching HTML only, ChatGPT, Claude.ai, Gemini on the web and in the terminal, Grok, Perplexity, Devin, Windsurf, Zed, Aider, Cline and v0.
We checked our own copy on 25 September 2026, having Claude Code version 2.1.280 fetch the same header-echo page. It sent the same Accept text/markdown request Checkly recorded in February on version 2.1.38, so the behaviour has held across roughly 240 releases and seven months. One difference is the name. In February the tool announced itself as axios, a generic library, so older builds may sit in your logs under that name. The two lines it sent are below. The second is how Claude Code shows up in your logs.
Accept: text/markdown, text/html, */*
User-Agent: Claude-User (claude-code/2.1.280; +https://support.anthropic.com/)Three sites that logged the header
Testing a tool in a lab tells you what it can send. Logging a real site tells you what arrives. Three people did that in 2026 and published the counts.
| Site and period | What the header showed |
|---|---|
| Dries Buytaert’s blog, a month to early March 2026 | No AI crawler sent it. Several fetched .md copies via the link tag |
| suganthan.com, 7 March to 19 April 2026 | 1,421 requests sent it, none from GPTBot, ClaudeBot or PerplexityBot |
| GoodBarber sites, 4 to 18 September 2026 | Claude Code sent it 99 times in 100. ClaudeBot and ChatGPT-User never did |
Dries Buytaert, who created Drupal, offered his site both ways in January 2026 and read a month of Cloudflare logs. His finding, in his words, was “No AI crawler uses content negotiation. Not one.”
Suganthan Mohanadasan switched on Cloudflare’s feature and counted 1,421 markdown requests in 44 days, with none of the big crawlers among the requesters. The 500 that arrived under a plain Claude name he attributed to Anthropic’s own infrastructure. Given the user agent above, some may have been Claude Code, though his post does not say.
On 24 September 2026 GoodBarber’s co-founder and head of engineering published two weeks of nginx logs across the company’s eleven language sites, which is the largest count so far and the clearest. Claude Code sent the header on 99.1 percent of its requests, ShapBot on 98.3 percent and ExaSearchBot on 93.5 percent. GPTBot and PerplexityBot sent it on 0.1 percent of theirs. Claude Code’s requests came from more than 500 addresses and 35 versions of the client. ClaudeBot and ChatGPT-User sent the header on none of roughly 34,000 requests between them. The two other heavy requesters, ShapBot and ExaSearchBot, are search services for AI tools, and their names are declared rather than verified, a caveat the author makes himself.
What the big crawlers do instead
The crawlers are not ignoring markdown. They are ignoring the negotiation. Give them a linked .md copy and several will take it. In Buytaert’s month, once his pages carried the alternate link tag, the crawlers took the copies.
- Amazonbot fetched 1,840 markdown files
- OAI-SearchBot 1,300 and GPTBot 1,177
- Meta’s crawler 485 and ClaudeBot 149
- The live fetchers ChatGPT-User and Claude-User, a handful each
On the GoodBarber sites GPTBot followed the alternate link to the .md copy about 5,800 times, almost always within minutes of fetching the HTML. The same crawlers that never send the header will follow a link to a file.
Whether that earns you anything is a separate question, and the one controlled test so far says probably not much. Profound, an AI visibility analytics company, split 381 pages into two groups from 19 January to 8 February 2026, serving 192 of them to AI crawlers as markdown and 189 as HTML, and counted visits from the OpenAI, Anthropic, Perplexity, Meta and DuckDuckGo bots. The markdown group averaged about 16 percent more visits, but the typical page gained one visit in three weeks and the lift came from pages that were already busy. Profound’s own conclusion was that it could not say with confidence that markdown drove more traffic, and that the models are already very good at reading HTML. Our own test of what ChatGPT-User sees on a page points the same way. It reads your raw HTML perfectly well.
Thirty days on a news site
What our logs can and cannot show
Our news site is not on Cloudflare and does not serve markdown, so we cannot count negotiation there. Its raw server logs also do not record the Accept header, which is true of most hosts’ default logs. What the logs do show is who came and what address they asked for, and that answers two useful questions. Did any AI tool that asks for markdown turn up? And did any crawler go looking for a markdown copy that does not exist?
We went through 2.48 million log lines from 26 August to 24 September 2026.
Claude Code, a tool that asks
The first thing the logs show is that a tool which asks for markdown is already here.
| Claude Code on a news site, 30 days | Count |
|---|---|
| Successful page fetches | 1,291 |
| Different pages fetched | 446 |
| Different addresses it came from | 744 |
| Different versions of the client | 65 |
| Fetches on the quietest and busiest days | 14 and 99 |
That is a developer tool reading a general news site more than forty times a day, from several hundred different machines or networks, and the daily count roughly tripled across the month. The 65 versions reflect a tool that releases almost daily and updates itself, not 65 separate installs, and the 744 addresses are not 744 people, since home connections change address and offices share one. The pages it fetched most were a fuel price tracker, a monthly house sales report and a couple of economic news pieces. Our guess, and it is a guess, is that people were asking their coding assistant to pull a figure into something they were building. What the log shows for certain is that a tool which asks for markdown identified itself about 1,300 times and got HTML every time. Each fetch was started by a person or by a task a person set up, though not necessarily read by a person, since the tool can fetch inside a longer automated job.
For scale, the hosted Claude assistant, the plain Claude-User fetcher, read about 4,350 pages in the same window, so Claude Code was about a fifth of all Claude reads on the site. ChatGPT-User read about 37,000. A few other self-declared agents appeared in small numbers, a tool called agent-browser with 156 requests, Manus with 19, the Qoder coding tool with 21 and Google’s Antigravity agent with 2.
Nobody went looking for a markdown copy
The second question has a shorter answer. In 30 days GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Amazonbot and Meta’s crawler made no request for any .md address or for llms.txt. None. The only visitor that tried was agent-browser, which asked for markdown copies of three articles 26 times and got 26 not-found errors, plus one curl. Nobody else guessed. That matches Buytaert’s finding that crawlers only find a markdown copy when the page links to it, and it matches our earlier month of watching for llms.txt requests, when the assistants made none.
Two notes on how a bot log shows this. Claude Code’s user agent begins with Claude-User, so a plugin that matches on that name, ours included, files its fetches under the hosted assistant as live reads. The full user agent string is kept, so the data is there, but our dashboard does not yet count Claude Code on its own. Second, these fetches come from the user’s own machine, not from Anthropic’s servers, so a check against Anthropic’s published addresses will not confirm them, and our dashboard labels them Spoofed. On the news site none of the 744 addresses was in Anthropic’s list, against nearly all of the hosted Claude-User fetches. A visit that fails that check is not necessarily a fake. It may be a real person’s tool.
So should you serve markdown?
Three rulings, by site type. If you publish documentation, developer tools or anything people read while coding, yes, because the tools that read you ask for it. If you run a news site, a shop, a service or a local business, no, because nothing that reads you asks for it. If you are not sure which you are, measure first, using the steps in the next section. The detail behind each ruling follows.
- Documentation, developer tools, APIs, anything people read while coding. Yes, and it is cheap. Claude Code, Cursor, OpenCode and Copilot all ask, they get a smaller and cleaner page, and the person on the other end gets a better answer. If you are on Cloudflare’s Pro plan or above, switch the feature on. Otherwise publish .md copies and add the alternate link tag.
- A news site, a shop, a local business, a blog read by ChatGPT-User and the crawlers. Probably not yet. The bots that matter to you fetch HTML and read it well. A linked .md copy will get crawled, and Profound’s test suggests that changes little. Put the afternoon into the page itself.
- If you serve two versions at one address, mind your cache. A full-page cache that ignores the Accept header can store the markdown version and hand it to the next human visitor. Cloudflare marks the two versions as different for this reason, but most WordPress page caches and many host caches key on the address alone and ignore that marker. Negotiate at the edge, in front of the cache, or use separate .md addresses and avoid the problem.
Doing it on WordPress
There are three routes. Cloudflare’s feature, if your site is behind Cloudflare on a paid plan, needs no code. A handful of WordPress plugins add a .md address for each post and the alternate link tag, which is the route the crawlers follow. And a developer can add negotiation by hand in a few lines. Whichever you pick, keep the markdown copy from being indexed as a duplicate page. A .md file cannot carry a canonical tag, so send the canonical link in the response headers instead. Check that your cache keeps the two versions apart before you switch anything on. One April report found Cloudflare’s conversion failed quietly on a host that sends pages in chunks without a length, so test your own host. And the alternate link tag is a convention that some clients follow, not a standard every crawler honours.
How to find out on your own site
- Read your bot log’s Claude-User count. A free bot log shows live reads by Claude-User, which includes Claude Code. Then search the stored user agents for claude-code to see how many were the coding tool. If the count is tiny, the question is settled for now.
- Search the raw access log. Download it from your hosting panel and search for claude-code and for .md. Not-found errors for addresses ending in .md tell you something is already hunting for a markdown copy, and who.
- Log the Accept header for bots. Default server logs leave it out. On WordPress a ten-line plugin can write the header and the user agent to a file for every request that reaches PHP, though on a cached site most bot requests never do. If you run your own server you can add the header to the Apache or Nginx log format instead. Shared hosts do not allow that.
- Test your own tools. Ask whatever coding assistant you use to fetch a header-echo page and read back what it sent. It takes a minute and settles any argument.
Common questions
Does GPTBot support content negotiation for markdown?
Not that anyone has observed. Across three sites that logged the header in 2026, GPTBot sent it on 0.1 percent of requests at most, and on our news site it never asked for a markdown address. It does fetch .md copies when a page links to them.
Does ChatGPT read markdown?
ChatGPT-User, the fetcher that reads a page when someone asks ChatGPT a question, requests HTML and sent the markdown header on none of about 10,000 requests in the GoodBarber logs. It reads HTML well, as our JavaScript test showed. Serving it markdown is not something it asks for.
Does Cloudflare’s Markdown for Agents help SEO?
It changes nothing for Google, which fetches HTML, and nothing for the AI crawlers, which do not send the header. It helps the coding tools that do, and it saves them tokens. Treat it as a service to developers who use your site, not as a ranking lever.
Is llms.txt the same thing?
No. llms.txt is a single index file at your site’s root. Markdown negotiation is per page, on request. Neither is fetched by the big assistants in our logs. Our llms.txt article has the 30-day count.
Should I block Claude Code?
Its fetches are made because a person asked, they are modest in volume, and it identifies itself honestly. Blocking it means that person’s tool cannot read your page. Our guide on whether to block AI bots sorts the three jobs bots do. Claude Code is doing the one that most resembles a reader.
What to do this week
- Search last month’s log for claude-code and for .md and write down the two counts.
- If the first count is meaningful and you publish documentation, switch on markdown, by Cloudflare or by .md copies with the alternate link tag.
- If it is not, leave markdown alone and put the time into the pages the crawlers already read.
- Either way, keep watching. Copilot changed its behaviour between February and June. The crawlers could too.
Do AI crawlers support content negotiation for markdown? As of September 2026 the coding tools do and the crawlers do not, and a month of logs on an ordinary site will tell you which of the two is reading yours.
See who is reading, and how
The free plugin records every AI bot visit that reaches your WordPress site, with the user agent kept in full so a search for claude-code takes a second, and imports your server log so a cache cannot hide the rest. Pro adds the people each assistant sent.