llms.txt
llms.txt is a proposed convention for publishing a Markdown file at a website's root, usually at /llms.txt. The file gives language-model consumers a concise description of the site and curated links to useful, preferably clean Markdown versions of important pages. It is intended to help a model or agent find relevant public material when processing a site. It is not a crawler-access rule, sitemap protocol, authentication mechanism, or guaranteed search-ranking signal.
Origin and context
Jeremy Howard published the proposal on 3 September 2024, motivated by limited context windows and the difficulty of extracting essential information from complex HTML pages. The proposal defines a simple Markdown structure and distinguishes the compact index from optional full-content files. Cloudflare later described an llms.txt implementation in its developer documentation, showing adoption beyond the originating project without turning the convention into a formal standard.
Why it matters
Documentation sites often contain navigation, scripts, repeated chrome, and many low-priority pages. A maintained llms.txt can identify the small set of pages an AI consumer should read first and can point to cleaner representations. That may reduce discovery effort for tools that deliberately request the file. The benefit depends on actual consumer support, accurate curation, and accessible linked content; publishing the file alone does not make every model use it.
Example
A framework documentation site can publish /llms.txt with a short description, links to installation and API pages, and a separate optional section for examples. Each link should lead to current, public documentation and use clear labels. An agent that recognizes the convention can start with this index instead of exploring the entire navigation tree. The site should still maintain normal HTML navigation, a sitemap where appropriate, and ordinary crawler controls.
How it differs
AGENTS.md
llms.txt is a website-root content guide for consumers of public web documentation. AGENTS.md gives coding agents operational instructions inside a software repository and may vary by directory. The two files can coexist for a project's website and source repository, but they have different audiences, locations, and authority boundaries.
Maturity and evidence
Maturity is rated 3. The proposal has a stable public specification, recognizable filename, and documented use by independent infrastructure and documentation providers. It remains an informal convention with uneven client demand. An Ahrefs study of available traffic found that 97% of observed llms.txt files received no requests during May 2026, so deployment does not demonstrate meaningful adoption by model providers.
Limits and open questions
A file can become stale, omit important material, or conflict with the linked pages. Consumers may ignore it, and a request does not prove that content affected an answer. Ahrefs' traffic study found very limited observed fetching and argued that llms.txt should not be treated as an SEO tactic. Because the file is public input rather than trusted policy, publishers should exclude secrets and credentials, keep links canonical, review changes, and continue using robots.txt, sitemaps, access controls, and normal documentation quality for their intended purposes.
Related terms
References
- The /llms.txt file, v2llms.txt Project · 2024-09-03 · class A
- An AI Index for all our customersCloudflare · 2025-09-26 · class A
- We Analyzed 137K Sites: 97% of llms.txt Files Never Get ReadAhrefs · 2026-06-15 · class B
- AGENTS.mdAGENTS.md Project · 2025-12-10 · class A
Last updated: 2026-09-04
This term is also covered in the Skills Atlas as information retrieval skill.