Your SEO plugin offered to make an llms.txt file. A checker gave you a red mark for not having one. Google says its search ignores it. To settle it, we pulled 30 days of raw server logs from a news site we run, 2.48 million requests, and counted every fetch of the file and who made it. The answer is clear, and the more useful finding is that most WordPress owners could not have run this check from inside WordPress.
What is llms.txt, and the short answer
llms.txt is a plain text file at the top folder of a website, written in a simple text format with headings and links, that gives AI systems a hand-picked reading list of the site’s pages with a one-line note on each. Jeremy Howard of Answer.AI proposed it on 3 September 2024 and revised it in August 2026.
It does not control access, which is robots.txt’s job, and it does not list every page, which is the sitemap’s job. It grants nothing and blocks nothing. It is an offer of help that an AI system may take or ignore.
As of 24 September 2026, they ignore it. Three facts cover the whole question.
- We found no document from OpenAI, Anthropic, Perplexity, Google or Microsoft that says its assistant reads llms.txt on other people’s sites. Google says in writing that Search ignores it.
- Every independent log study we could find since 2025, from one site to 137,210 domains, finds that no AI assistant goes looking for the file on its own. Most of the fetches that do arrive come from SEO tools, link previews and scanners.
- On our news site over 30 days, 152 requests asked for an llms.txt address, 78 of them misses under article paths. Not one came from a named AI bot, meaning a request whose browser label names one of the AI bots the plugin knows. The same bots, together with Googlebot and bingbot, fetched robots.txt more than 3,000 times.
The file is harmless, and if a plugin makes it for free there is no reason to remove it. It is not a way into ChatGPT. The one place it earns its keep is developer documentation read by coding agents, and our log shows exactly that kind of reader. And if you run WordPress, your dashboard probably cannot see the file at all, for a reason explained below.
What the AI companies say
| Company | Publishes one for its own docs | Says it reads yours |
|---|---|---|
| OpenAI | Yes | No |
| Anthropic | Yes | No |
| Perplexity | Yes | No |
| Not found at its main developer sites | No. Says Search ignores it | |
| Microsoft | Not found | No |
The first column is the source of most confusion. OpenAI, Anthropic and Perplexity all serve an llms.txt for their own developer documentation, and OpenAI’s and Perplexity’s crawler pages point coding tools at it.
Neither of those pages, nor Anthropic’s, says their assistants read the file on anyone else’s site. Publishing one and consuming one are different things.
Google is the only company that has said anything in writing, and it said it twice. Its guide to generative AI features, published 15 May 2026, lists creating AI text files among the tactics you can ignore. A clarification added in June, in the guide’s current wording, reads, “Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.”
A year earlier John Mueller of Google had written, “FWIW no AI system currently uses llms.txt.” In the same thread he gave the test this article runs. “if you find that an AI system which sends you significant traffic is checking for the file as often as robots.txt in your server logs, that seems like a good time to think about it. My logs don’t show that currently.”
One thing muddies this. On 7 May 2026 Google’s Chrome team shipped a Lighthouse audit that fetches llms.txt. Lighthouse is the page checking tool built into Chrome. By Google’s own documentation a missing file is marked not applicable and only a server error fails the audit, but its arrival put the file on every audit checklist. Eight days later Google Search published the guide saying it ignores the file.
Two Google teams published different guidance within eight days of each other, and third party SEO checkers now hand out red marks for not having a file Google Search does not read. That is where most of the pressure to add one comes from.
What every log study found
Since Mueller’s test was published, several people have run it, at every scale. The results agree, with two exceptions covered below.
- Ahrefs, May 2026 traffic, 137,210 domains. Published 15 June 2026. About 28 percent of the sites had a valid llms.txt, and 97 percent of those files got no requests at all in the month. Of the requests that did arrive, all AI categories together were under a fifth. “Slackbot alone fetched llms.txt files more often than PerplexityBot did.” Sites without a file got no AI probes for it. Ahrefs’ conclusion, “no AI system goes looking for one you haven’t published.” The study notes its sample skews technical, and Ahrefs sells log analytics.
- WeekERP, August 2026, two production sites. 68,759 verified AI bot requests. Fetches of llms.txt by AI bots, zero. Fetches of robots.txt, 1,938. The 61 non-AI fetches of the file came mostly from Lighthouse. Their verdict, “no value beyond satisfying a checkbox in certain SEO auditing tools.” Read the WeekERP study.
- Saaslinks, July to August 2026, one site, 14 days. AI crawlers fetched robots.txt 723 times and llms.txt never. Thirteen requests for the file in total, four from the site itself. Read the Saaslinks study.
- The GEO Lab, July 2026, one WordPress site, 78 days. 241 requests for the file. Twenty-five claimed to be a known bot, and 18 of those 25 failed address verification. Six came from OpenAI’s published addresses and one from Bing. Read the GEO Lab study.
- Evil Martians, July 2026, one site, two months. About 770 fetches of the file, 37 from named assistants, the rest search crawlers, SEO bots and scanners. Read the Evil Martians study.
Two exceptions, and why they are not adoption
Ray Martinez at Wix logged over 5,000 fetches of llms.txt by OAI-SearchBot across seven education sites in three months, after adding a link tag for the file to each home page. The peak came in the week his LinkedIn posts about the files gathered hundreds of interactions, which he credits, along with a coming ChatGPT release, for the surge.
Nathan Hall of Lefty Media found that once he announced the file through a robots.txt Sitemap line and page links, ClaudeBot fetched it 1,131 times across seven sites, while the unannounced copy at the site root drew two verified crawler fetches in 31 days and several hundred from scanners and SEO tools. His own conclusion is that this is discovery, not adoption. Tell a crawler where a file is and it fetches the file. Fetching is not use.
Most hits on llms.txt are not AI assistants
Put the studies together and a pattern appears. Fetches of llms.txt come from SEO audit tools, technology profilers, Lighthouse, link previews in Slack and Skype, directory sites that scan every domain for the file, and scripts pretending to be bots. Mueller added in a reply in the same thread in September 2026 that those directory sites now link to every llms.txt they find, so ordinary crawlers follow the links.
A hit on the file is not evidence of anything until you know who made it and check it against the company’s published internet addresses.
What 30 days of our own logs show
The news site has had an llms.txt for the whole period, a file of about 37 KB with 155 links under ten headings, most with a one-line note. It is not mentioned in robots.txt and not linked from the home page, so this is the unannounced case. We took the web server’s own log for the 30 days to 24 September 2026, every one of the 2,483,780 requests, and counted.
Who fetched llms.txt
| Who | Requests | What it is |
|---|---|---|
| agent-browser | 104 | An open source browsing tool for AI coding agents |
| SEOJuice-SearchBot | 23 | An SEO and AI visibility tool |
| Scripted scanner, browser names, rotating addresses | 12 | Unknown operator |
| Lighthouse and PageSpeed | 8 | Google’s page audit tool |
| Dataprovider, Piperic, TheWebReport, Semrush | 4 | Data aggregators and one site audit |
| Odd browser strings | 1 | A scanner claiming to be an old iPhone |
| Any named AI bot | 0 | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Googlebot, bingbot and 20 others |
Zero from any AI assistant or crawler, in a month when the AI bots in our list, Googlebot and bingbot aside, made more than 100,000 page requests to the site, and Google’s GoogleOther crawler another 134,000. ChatGPT-User alone read pages 37,061 times. It never once asked for the file that was written for it.
The largest fetcher is worth a closer look, because it is the exception the spec was written for. agent-browser is an open source tool from Vercel Labs built for coding agents such as Claude Code, Codex and Cursor to read web pages. Its read command asks for a page, then walks up the address looking for the nearest llms.txt, which is why our log shows it trying the file under the article, under the month, under the year and finally at the root, where it found it.
On four days in September someone’s coding agent read our articles this way, almost all of it from a cloud server in Tokyo, 26 successful fetches of the file and 78 misses on the way up. That looks like a person’s agent doing exactly what Jeremy Howard hoped for. It is not ChatGPT, Claude or Perplexity deciding to cite us.
How often the same bots fetch robots.txt
Mueller’s test needs something to compare against, and robots.txt is it. A file the standards say bots must read, sitting at the same root, in the same 30 days.
- YouBot 781, Claude-User 637, OAI-SearchBot 632, Googlebot 429, bingbot 335
- PerplexityBot 161, DuckAssistBot 155, Applebot 139, ClaudeBot 78, Amazonbot 21, CCBot 10
- llms.txt, from all of them together, 0
These are counts of requests whose bot name matched, not checked against published addresses, but since none of them touched llms.txt there is nothing to check. Sitemaps tell the same story. Bingbot fetched a sitemap file 863 times and Applebot 585 times. The bots read the files the web agreed on years ago. They do not read the new one.
Why your WordPress plugin probably cannot see the file
Here is the part that surprised us. Before we had the raw log we looked in our own plugin, which records every AI bot visit from inside WordPress, and it showed zero fetches of llms.txt. That zero was not a finding. It was blindness, and it applies to most WordPress sites.
When a real file named llms.txt sits in your site’s top folder, the one that holds wp-config.php, the web server hands it out itself. WordPress is never started, so nothing installed inside WordPress runs, not a plugin, not the theme.
The web server and anything in front of it still see the request, which is why the server’s own log can answer the question. The rules that make WordPress work only send a request into WordPress when no file with that name exists.
That is how robots.txt behaves differently on most sites. WordPress makes it up on request, so WordPress runs, and a plugin inside WordPress can count the visit when the cache lets the request through. The screenshot above is our plugin doing exactly that. The counts before it came from the raw log, which saw far more, because on a cached site everything a plugin counts is a minimum, never the full total.
Which kind of llms.txt you have depends on who made it. We read each plugin’s code to check. Yoast and All in One SEO write a real file into your root folder. Yoast rewrites it weekly and leaves a file it did not create alone. All in One SEO rewrites it daily and again a few seconds after each post you save, and overwrites whatever is there. Neither is doing anything wrong. A real file is the faster way to serve one.
Rank Math and SEOPress, in its paid version, do the opposite. They answer the request from inside WordPress the way WordPress answers robots.txt, and Rank Math adds a noindex instruction while it does so. On a Rank Math or SEOPress site a bot fetching llms.txt starts WordPress, and a logger inside WordPress sees the fetch unless a cache answers first. On a Yoast or All in One SEO site, or with a hand-made file like ours, nothing inside WordPress ever will.
Note. A zero for llms.txt in any WordPress plugin, including ours, is not evidence that nobody fetched it. If the file is a real file, the web server answers before WordPress starts and the plugin never sees the request. Only the web server’s own log can answer the question.
How to tell which kind you have
Ask for the file and look at the notes the server sends back with it. This one-line command shows them. If it looks unfamiliar, ask your host which of the two cases below applies and skip it.
curl -sL -D - -o /dev/null -A "Mozilla/5.0" https://yoursite.com/llms.txtLook at the last group of lines, since a site that redirects prints more than one. If one of them begins link: or Link: and contains rel="https://api.w.org/", WordPress answered and a plugin inside WordPress could have counted the fetch. If you see x-robots-tag: noindex, nofollow and no link line, that is probably Rank Math, also WordPress.
If you see neither, open your host’s file manager and look for llms.txt in the same folder as wp-config.php. If it is there, the web server hands it out and WordPress never runs.
If it is not there, something inside WordPress answered and removed the link line. A cache in front of the site repeats whatever notes your actual server sent, and some security plugins remove the link line, so a missing line proves less than a present one. On Windows use curl.exe in PowerShell, or read the response headers in your browser’s developer tools.
Where the real log lives
On a server you control it is usually at /var/log/nginx/access.log or /var/log/apache2/access.log. On shared hosting it varies, and our host was an education. SiteGround’s Statistics screen shows only the latest 300 requests, about five minutes of traffic on a busy site, but its File Manager has a folder called logs beside public_html with one compressed file per day for the last 30 days. That is where these numbers came from.
Our guide to importing a server log lists the usual places by host. The Pro version of our plugin reads a plain text log every hour where the server lets it open one, and the free plugin imports one when you click a button. Neither can yet read the compressed daily files SiteGround keeps, so for these numbers we downloaded the files and counted them ourselves. Everything a plugin counts on a cached site is a minimum, as our article on caches hiding bots explains.
What to do
- Keep it if a plugin makes it for free. It costs nothing to serve and Google says it does no harm.
- Tell Google not to list it as a page. Mueller suggested a noindex header in July 2025 because other sites link to these files and they end up in search results, which is odd for a reader. The header is a hidden note sent with the file. Rank Math sends one already. For a real file on an Apache host, three lines in
.htaccessadd it. On nginx or a managed host, ask support. - Do not pay for a generator, a checker or a score. The red mark comes from SEO checkers copying a Lighthouse audit, for a file Google Search itself ignores.
- If you publish developer documentation, take it seriously. Coding agents do read these files, our log shows one doing it, and OpenAI, Anthropic and Perplexity publish them for their own docs for that reason. If you run a bakery, a law firm or a news site, nothing reads it yet.
- Check again in a few months, in the server log. The day an assistant that sends you traffic fetches llms.txt as often as robots.txt is the day it starts to matter. That will show up in the web server’s log first and in your WordPress dashboard never, unless your file goes through WordPress.
Forge AI Bot Log shows you the other side of this, the bots that do visit, which files and pages they read, and whether each one came from its company’s published addresses. That is the robots.txt half of Mueller’s test, on your own site, today.
Common questions
Does ChatGPT read llms.txt?
Not that anyone can show. OpenAI’s crawler documentation says nothing about reading it on other sites, and in our 30 days ChatGPT-User read pages 37,061 times and the file zero times. OpenAI publishes an llms.txt for its own developer docs, which is a different thing.
Why does my llms.txt return a 404?
If Rank Math or SEOPress makes it, WordPress has not learned the address yet. Go to Settings, Permalinks and click Save. If Yoast or All in One SEO makes it, the plugin was unable to write the file, usually a folder permission problem, or on some managed hosts it wrote it to a folder the public cannot reach. If two plugins both try to make the file, the real file wins and you may be looking at one you did not choose.
Is llms.txt worth it?
Worth keeping if it is free, not worth paying for, and not worth expecting anything from unless you publish documentation that coding agents read. Nothing in any log, ours included, connects the file to being cited by an assistant.
Should I block the SEO tools that fetch it?
No need. A few hundred fetches of a small text file a month cost nothing. The SEO and data tools in our table say they respect a robots.txt rule with their own name. The scanners with fake browser names will not, and neither will a person’s agent.
See what actually reads your site
The free plugin logs every AI bot visit on your WordPress site, names each bot, checks it against its company’s published addresses, and shows the files and pages it read. Run Mueller’s test on your own robots.txt today.