llms.txt in 2026: What It Is, Who Actually Reads It, and Whether You Need One

What is llms.txt?

llms.txt is a plain Markdown file at the root of a website (yoursite.com/llms.txt) that gives AI agents a short, curated map of the site: what it is, and which pages are worth reading first. Think of it as a table of contents written for language models, not for people or search engines.

Jeremy Howard proposed the format on September 3, 2024. The spec at llmstxt.org reached version 2 on August 10, 2026. Only one line is required, an H1 with the site's name. After that comes an optional one-line summary in a blockquote, any plain-text context, and H2 sections that list links with short notes. A section titled "Optional" holds links an agent can skip when it's short on context.

The spec also suggests serving a clean Markdown copy of each page at the same URL with .md added. Many sites publish a second file, llms-full.txt, with the full text of their key pages in one place. That one is a common convention rather than part of the spec.

llms.txt vs robots.txt: a guide, not a gate

The two files are easy to confuse because they sit side by side at your root. They do opposite jobs. robots.txt decides who may crawl your site. It's where you allow or block GPTBot, ClaudeBot, PerplexityBot and Google-Extended. llms.txt can't block anything. It only points an agent that's already allowed in toward your best pages.

It isn't a sitemap either. A sitemap lists every URL so search engines find them all. llms.txt is a short, hand-picked list with context, closer to the page you'd hand a new employee on day one. If you want to stop AI companies from training on your content, llms.txt won't do it; robots.txt and your firewall rules will.

Google says skip it. Lighthouse checks for it.

Google's guidance points two ways at once. Google Search's own documentation on AI features says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." It adds that there are no additional requirements for AI Overviews or AI Mode beyond being indexed and eligible for a snippet.

Google's newer AI optimization guide goes further and lists llms.txt among tactics to skip, Search Engine Journal reported on May 20, 2026. Google's John Mueller has compared the file to the old keywords meta tag.

Then, in May 2026, Chrome's Lighthouse 13.3 added an experimental Agentic Browsing category that includes an llms.txt audit. A missing file (a 404) is marked Not Applicable, not failed. What gets flagged is a server error when Lighthouse tries to fetch it. Chrome's docs call llms.txt "an emerging convention" and recommend placing it at the root. The audit is about browser agents finding their way around your site. It isn't a ranking signal.

Who actually reads llms.txt? The server-log data

The best public test so far comes from EZY.ai, which logged bot traffic on 83 websites with llms.txt live from April 27 to July 19, 2026. The major AI crawlers fetched robots.txt thousands of times and llms.txt almost never: OpenAI 7 times, Anthropic's ClaudeBot 9 times, PerplexityBot zero. Googlebot fetched it 67 times against 5,125 robots.txt requests. Meta's crawler was the one exception, fetching llms.txt 193 times, more often than robots.txt.

Two caveats. The panel leans toward small-business sites, and EZY sells an AI visibility tool. Still, the direction matches what Google says publicly: training and search crawlers aren't built around this file.

Where llms.txt does get used is on demand, by coding agents and developer tools. When you ask an agent to read a library's docs, a clean index saves it from crawling the whole site. OpenAI, Anthropic and Google all publish llms.txt files for their own developer docs, and Anthropic's Claude Code documentation tells agents to fetch its llms.txt index first. That's a user-triggered fetch, not a crawler, so it rarely shows up as a bot in your logs.

How to create an llms.txt file

If your site has documentation, an API, pricing, or anything an AI agent might be asked about, adding one costs about an hour. Here's the process we used for our own file.

1. 1.List the 10 to 30 pages that answer the questions people ask about you: pricing, product, docs, comparisons, policies.
2. 2.Write the H1 (your site or product name) and a one-sentence summary in a blockquote.
3. 3.Group the links under H2 headings, each with a short note on what the page answers.
4. 4.Put nice-to-have pages under an H2 called Optional.
5. 5.Serve it at /llms.txt as plain text with a 200 status, with no login wall, redirect or bot challenge in front of it.
6. 6.Generate it from your CMS or sitemap so it stays current. A stale file points agents at pages that no longer exist.

A real llms.txt example

Here's the llms.txt file AI Eating The World serves. The site generates it from our CMS, links a fuller llms-full.txt, and points agents to our public MCP server and to the fact that every page returns Markdown when requested with Accept: text/markdown. The developer page explains that setup.

For a model to copy, look at the llms.txt Anthropic serves for its Claude Code docs. It opens with the H1 and a one-line summary, then lists every doc page under section headings, each with a one-line note on what the page answers.

Generators and CMS plugins can build a first draft from your sitemap. Edit the result by hand: the notes on each link are the part that helps an agent choose, and generators tend to write vague ones. The llmstxt.org spec lists tools and directories if you want a head start.

What decides AI search visibility instead

If llms.txt isn't what gets you cited, what is? Generative engine optimization, GEO SEO, LLM SEO, AI search optimization: the names change, but the checks that decide how to get cited by ChatGPT or how to rank in ChatGPT's answers are mostly old-fashioned. Google says AI Overviews draw from pages that are indexed and eligible for a snippet. ChatGPT, Claude and Perplexity can only quote pages their crawlers are allowed to fetch. So the order of work is:

• Let the crawlers in. Check robots.txt and your CDN or firewall rules for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. Blocks often happen by accident.
• Serve real HTML. Pages that render only with JavaScript can come back empty to many AI crawlers.
• Write answers that are easy to quote: a direct answer near the top, specific numbers with sources, and clear headings that match the question.
• Keep facts consistent across your site, your docs and third-party profiles, so models don't learn two versions.
• Get mentioned elsewhere. AI answers lean on reviews, comparisons and directories, not only your own pages.
• Add llms.txt last, as cheap insurance for agents and developer tools.

Want someone to check your site for you?

That list is what our AI Search Readiness Audit checks. We test whether GPTBot, ClaudeBot, PerplexityBot and Google-Extended can reach your pages through robots.txt and at the server, review your llms.txt and structured data, check how easy your key pages are to quote, look for JavaScript-only content and redirect problems, and compare you with up to three competitors you name.

You get a written PDF with a prioritized fix list your developer can act on, within three business days, with no calls. It costs $149 at the intro price. It's a readiness audit: we don't track how often assistants mention you today, and we don't promise rankings or traffic. You can read a free sample audit first. We ran it on our own site.

llms.txt FAQ

Does ChatGPT use llms.txt? OpenAI hasn't said its crawlers use it, and in EZY.ai's 2026 logs OpenAI fetched llms.txt 7 times across 83 sites in 12 weeks. OpenAI's guidance for crawler control is robots.txt.

Does Google use llms.txt? Google Search says no special AI files are needed for AI Overviews or AI Mode. Chrome's Lighthouse checks for the file, but only as an agent-readiness audit, not a ranking signal.

Is llms.txt worth it? For documentation, API and SaaS sites, yes: it's cheap and coding agents use it. For a small content or local business site, fix crawler access and page content first.

What's the difference between llms.txt and llms-full.txt? llms.txt is a short index of links. llms-full.txt, a common companion file, holds the full text of key pages in one document.

Where do I put llms.txt? At your domain root, so it loads at yoursite.com/llms.txt with a 200 status.

Does llms.txt help SEO? Not for Google rankings. Its job is helping AI agents find the right pages once they're on your site.

What to remember

• llms.txt is a Markdown index for AI agents at your site's root. Only the H1 is required.
• Google Search says it's not needed for AI Overviews; Lighthouse checks it only for agent readiness.
• In 2026 logs, OpenAI, Anthropic and Perplexity crawlers almost never fetched it.
• Crawler access, real HTML and quotable answers decide AI visibility. Add llms.txt after those.

Sources