llms.txt vs robots.txt vs sitemap.xml
The three files are complementary, but each one has a different job.
llms.txt
Explains important content to AI assistants and language models.
robots.txt
Tells crawlers what they may or may not access.
sitemap.xml
Lists URLs for search engines to discover and index.
When to use each file
Use robots.txt for crawl rules, sitemap.xml for URL discovery, andllms.txt for curated AI-readable context. A well-prepared site can use all three.
Where each file lives
https://example.com/llms.txthttps://example.com/robots.txthttps://example.com/sitemap.xml
Create your llms.txt
Create a curated llms.txt file from your existing sitemap and publish it next to robots.txt.
Open the generatorFrequently asked questions
Does llms.txt replace robots.txt?
No. robots.txt controls crawler access. llms.txt provides a curated content guide for AI tools. They solve different problems.
Does llms.txt replace sitemap.xml?
No. sitemap.xml is a URL inventory for search engines. llms.txt is a short markdown guide to the most useful pages and context.