Does llms.txt work? The honest answer, with receipts

Francesco MontesantoFrancesco Montesanto
13. Okt. 2026

Four studies, 500M+ AI bot visits, and Google on record all point the same way. Here's what llms.txt actually does, and the few places it still earns a spot.

Somewhere right now, a marketing team is adding llms.txt to its site because a LinkedIn post said AI search depends on it. I get the impulse. When AI answers start eating your clicks, doing something feels better than doing nothing.

The data is in, and it's close to unanimous. Four independent studies covering hundreds of thousands of domains and more than 500 million AI bot visits found that AI crawlers almost never request the file. Google has said in writing that its search products ignore it.

So here's the honest verdict, receipts attached. Plus the short list of sites where llms.txt still earns its spot.

  • llms.txt is a proposed Markdown file that summarizes your site for large language models.
  • Across four studies, AI search crawlers almost never fetch it. One test logged 84 requests out of more than 62,100 AI bot visits.
  • Google says its search products, including AI Overviews and AI Mode, don't use it.
  • It's worth shipping for docs-heavy and developer-facing sites, where coding agents do read it.
  • For AI visibility, crawl access, answer-first pages, and well-structured content still do the heavy lifting.

What is llms.txt?

llms.txt is a plain Markdown file that sits at the root of a website (yoursite.com/llms.txt) and gives large language models a short, curated map of the site: what it is, which pages matter, and a one-line description of each. Jeremy Howard proposed the format in September 2024. Some sites also publish llms-full.txt, which bundles the full text of key pages into a single file.

On paper, it makes sense. Modern web pages are heavy with navigation, scripts, and cookie banners. A clean summary should save an AI system time and context.

The catch? It's a community proposal. No standards body backs it, and no major AI provider has committed to reading it. Which brings us to the server logs.

What the data says

Four teams went looking for evidence that llms.txt changes how AI systems treat a site. Here's what they found.

SE Ranking checked roughly 300,000 domains and found 10.13% had an llms.txt file. It found no clear link between having the file and getting cited in AI answers. Among the 50 most AI-cited domains in the set, only one had the file at all.

Limy monitored more than 500 million AI bot visits over 90 days. Only 408 requested llms.txt directly.

Otterly ran a 90-day server-log test on a site with a correctly built file. Of more than 62,100 AI bot visits, 84 touched llms.txt. That's about 0.1%.

Ahrefs tracked 137,210 domains and found 97% of llms.txt files got zero requests in May 2026. Of the requests that did happen, less than a fifth came from named AI tools, led by GPTBot and Claude Code.

Different methods, same shape. Adoption is real but thin, and the bots that decide what gets cited in AI answers mostly skip the file and read your HTML.

StudySampleWhat it found
SE Ranking~300,000 domains10.13% adoption; no clear effect on AI citations
Limy500M+ AI bot visits over 90 days408 requests to llms.txt
Otterly62,100+ AI bot visits over 90 days84 requests to llms.txt (about 0.1%)
Ahrefs137,210 domains97% of files got zero requests in May 2026

What Google has said on the record

Google has been consistent on this for more than a year. Then it put its position in writing.

July 2025: Gary Illyes said at Search Central Live that Google doesn't support llms.txt and has no plans to.

The keywords meta tag comparison: John Mueller likened llms.txt to the old keywords meta tag, a site's description of itself, when a search engine would rather read the site.

May 15, 2026: Google's first AI optimization guide listed llms.txt among the tactics you don't need for AI Overviews and AI Mode.

June 15, 2026: Google added a note saying you don't need machine-readable AI files or Markdown to appear in Google Search, because Search doesn't use them. Keeping one won't hurt or help.

One wrinkle worth knowing: Chrome's Lighthouse team added an experimental audit that checks for llms.txt as a sign of agentic browsing readiness. Same company, different job. Google Search ignores the file, while some browser-based agents may go looking for it.

Why AI search skips the file

None of this is mysterious once you look at how AI search actually works.

AI crawlers already read HTML well. Stripping navigation and boilerplate is a solved problem.

Retrieval happens page by page. When an assistant answers a question, it pulls the page that answers it best. A site summary doesn't answer much.

Self-declared summaries are easy to game. That's exactly what killed the keywords meta tag.

No big player has signed up. Without a major provider committing to read it, the file is a letter with no recipient.

An llms.txt file tells AI what your site claims to be about. AI search would rather check.

When llms.txt is worth shipping

There's a real audience for this file. It's smaller and more technical than the hype suggests.

Documentation and API reference sites. Coding agents fetch llms.txt to find the right docs page fast. Claude Code was the second most frequent named fetcher in Ahrefs' data.

Developer tools and technical SaaS. When developers ask their coding assistant about your product, a clean map of your docs helps it answer.

Large knowledge bases. A curated index points agents to the canonical page instead of the outdated one.

Teams testing agentic browsing. Chrome's experimental Lighthouse audit looks for the file.

If you're in one of those camps, ship it. It's cheap. Just generate it from your CMS so it updates when your content does. A hand-edited file that lists moved pages or old pricing is worse than no file, because the few tools that read it will treat it as the truth.

The real risk is a second site that drifts

Here's the objection I hear most from developers, and it's a fair one. Every "agent-ready" tactic seems to ask teams to maintain a second version of their content: one for people, one for machines. A commenter in a Hacker News thread on WebMCP put it plainly:

If you maintain two versions of information, one for regular humans and one for agents, it's just a matter of time before they start to diverge from each other.

They're right. Parallel files drift. The fix is one well-modeled source of content that serves both readers. When your content lives in structured fields like titles, summaries, answers, and entities, you can render a page for people, an API response for agents, or an llms.txt file from the same source. That's what content modeling is for.

It's also the thinking behind newer proposals like WebMCP, which describe the actions an agent can take on your site. Build the source once, then let each format pull from it.

What to do instead

If the goal is showing up in AI answers, spend this quarter's hours here first.

1

Let AI crawlers in. Check robots.txt and your CDN's bot settings. Nothing else on this list matters if the crawlers behind AI answers can't reach your pages.

2

Put the answer first. Lead each page with a direct answer, then add depth. It's easier for a person to skim and for a model to quote. Here's more on how SEO and AEO differ.

3

Structure content in fields, not blobs. Clean headings, consistent summaries, and well-modeled content are easier to retrieve and cite.

4

Keep structured data honest. Use standard schema to describe what's really on the page. Skip anything invented purely for bots.

5

Measure citations, not file fetches. Track whether AI answers mention and cite you, and watch impressions alongside clicks.

The verdict

llms.txt is a reasonable idea that the systems most marketers care about have chosen to ignore. If you run a docs site or a developer product, publish one and generate it automatically. If you're trying to win AI search, it won't move the needle, and those hours belong to the basics that do.

Doing something about AI search feels good. Doing the right thing works better.

Want fixes that actually move AI visibility?

Here are three practical AEO moves your team can ship this quarter.