llms.txt: The robots.txt of the AI Era

If your site already has robots.txt and a sitemap.xml, you've solved the search-engine problem of 2005. The new problem is different: large language models need to find, parse, and decide whether to cite your content. llms.txt is the answer the community converged on.

What llms.txt is, in one sentence

A plain-text Markdown file at /llms.txt that gives an LLM a structured tour of your site: what it is, what's important, and where to read more.

Why it matters

An AI model that lands on your homepage has a few seconds (and a few thousand tokens of context budget) to figure out whether you're worth citing. Search engines were trained on link graphs. LLMs are trained on text. llms.txt meets them on their own terms.

Three things happen when you ship one:

• Faster ingestion: ChatGPT, Perplexity, and Claude can index the parts you actually want them to know about, instead of guessing from your nav menu.
• Higher citation odds: When an LLM has to choose between two equivalent sources, the one with structured intent wins.
• You control the narrative: You decide what counts as "the important page" — not the model's heuristics.

A working template

Here's the entire llms.txt we ship at aiaeoscan.com/llms.txt. Copy it, change the names, ship it today:

# Your Product Name

> One-sentence description of what your product does and who it serves.

A 2-3 sentence paragraph explaining the value prop. Be concrete. Avoid marketing speak.

## Features
- Feature 1 with a measurable outcome
- Feature 2 with a measurable outcome
- Feature 3 with a measurable outcome

## Links
- Homepage: https://yourdomain.com
- Pricing: https://yourdomain.com/pricing
- Documentation: https://yourdomain.com/docs
- Blog: https://yourdomain.com/blog

Common mistakes to avoid

• Don't dump your sitemap into it. 200 URLs is noise. Pick 5-15 that actually represent the site.
• Don't write marketing copy. LLMs are trained to detect it and discount it. Write like a senior engineer would in a README.
• Don't forget to update it. Treat it like documentation. Stale = penalised.

Where this fits in the bigger AEO picture

llms.txt is necessary but not sufficient. To actually be cited by ChatGPT or Gemini, you also need:

1. AI bots not blocked in robots.txt (check yours — Cloudflare's defaults block GPTBot)
2. Real Schema.org markup (Product, Article, FAQ, HowTo)
3. Substantive content the model has reason to surface

Run a free scan at aiaeoscan.com to see where your site stands across all of those.