llms.txt — why your site needs one and how to build it

Short version, for the impatient: llms.txt is a curated Markdown map of a site for AI engines, it takes minutes, it doesn’t replace sitemap.xml or robots.txt — and the bigger risk isn’t the missing file but blocking the wrong bot.

Someone asked me again whether their site needs an llms.txt, and I noticed I’d been answering “yes, probably” for a while without ever spelling out why — so I went and read the spec properly. llms.txt is a file at your site root (/llms.txt) that gives AI engines a curated map of the content in Markdown: what matters on the site and where it lives. It isn’t robots.txt (access) or sitemap.xml (a full URL list for crawlers) — it’s the “cover sheet for AI.”

Why it matters

Language models are limited by context and reading time, and sitemap.xml lists every URL with no priority or context. llms.txt answers a different question: “if you have little time, read these, in this order.” Curation and human-readable Markdown are the whole difference.

How it differs from sitemap.xml and robots.txt

	sitemap.xml	robots.txt	llms.txt
Audience	search crawlers	bots (access)	AI engines / agents
Format	XML, all URLs	directives	Markdown, selected
Job	crawl every page	allow/disallow	what’s worth reading
Context	none	none	title + description

None of them compete — each has its own role, so llms.txt complements, it doesn’t replace. (Could the spec have picked a less confusing name? Probably, but here we are.)

What llms.txt looks like

# Site name

> One or two sentences: what the site is and who it's for.

## Main sections
- [Services](https://site.tld/services/): what and for whom
- [Blog](https://site.tld/blog/): takes on the topic

## Optional
- [About](https://site.tld/about/)
- [Contact](https://site.tld/contact/)

The structure from the llmstxt.org spec: an H1 title, a blockquote description, sections of “URL: short note” links, and secondary pages under ## Optional.

How to add one — step by step

1. Group pages into semantic clusters, each with a short description (one line per link).
2. Separate the secondary pages (legal, utility) into ## Optional.
3. Keep it compact: summary ≤ 80 words, not a hundred links — it’s a map, not a catalog.
4. Place /llms.txt at the root next to robots.txt (serve it as text/plain).
5. Optionally build /llms-full.txt with the full text of key articles in cluster order.

Five steps, and the hard one is the first — deciding what matters is editorial work.

Common mistakes

• A duplicate of sitemap.xml — dumping every URL with no descriptions. llms.txt is about priority and meaning, not completeness.
• Too long — a wall of hundreds of links eats the model’s context.
• No descriptions — without “what this page is about,” a link is useless to AI.
• Forgot to update — after a redesign/migration the links rot, just like a sitemap — and who checks?

Is it worth doing now

Here’s the uncomfortable question, then: does anyone actually read it yet? It isn’t a mandatory standard, and not every engine reads it. But it costs minutes, does no harm, and a growing list of tools already supports it, so my subjective take is that it’s cheap insurance: when an engine decides to read llms.txt, you’ll already have one — with the right priorities.

Who reads your site, and how

AI bots aren’t one “robot.” It helps to split them into three classes, and to block them deliberately:

• Training (safe to block without losing visibility): GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended, Applebot-Extended.
• Search / citation (block them and you vanish from answers): OAI-SearchBot (ChatGPT Search), PerplexityBot, Claude-SearchBot, bingbot (Bing/Copilot).
• On user demand: ChatGPT-User, Claude-User, Perplexity-User — they fetch a page when a person asks the AI to open it.

A common mistake, and an easy one to make: one Disallow under GPTBot that accidentally also catches OAI-SearchBot — you blocked training and lost citation with it. Which of the two did the site want to block? Usually training. Which did it block? Both!

Two technical facts on why machine-readability is critical:

• AI crawlers don’t execute JavaScript. GPTBot/ClaudeBot/PerplexityBot see only the first HTML response — client-rendered content is empty to them. You need server-side rendering.
• Structured data feeds the Knowledge Graph. Per the US v. Google materials, many features and entities come from markup and the Knowledge Graph, not from crawling the page. Schema.org + llms.txt give the machine structure explicitly, instead of relying on retrieval to “guess.”

In short

• llms.txt = a curated, Markdown site map for AI.
• It doesn’t replace sitemap.xml or robots.txt — it complements them.
• Bots are training / citation / on-demand — don’t block the citation ones by accident.
• AI doesn’t run JS → you need SSR; Schema.org feeds the Knowledge Graph.
• llms-full.txt = all key content in one file for a single fetch.

So, does your site need one? Probably yes. Just don’t celebrate the file while the robots.txt underneath it is quietly telling the search bots to go away!

Sources

• llmstxt.org — the llms.txt / llms-full.txt format spec.
• schema.org — structured-data types (Article, FAQPage, Product…).
• AI bot registry (GEO-HowTo 2026) — crawler classification by role (training / search / on-demand) and behavior (JS, robots.txt).
• US v. Google, case materials (2025) — the role of the Knowledge Graph and structured data in assembling answers.