llms.txt: The Complete Guide for AI Crawlers (With a Free Generator)
llms.txt is a plain Markdown file at your domain root that tells AI assistants what your site contains and which pages matter. How to write one properly.
Every few years the web agrees on a new file that sits at the root of a domain. First robots.txt, then sitemap.xml, now llms.txt. The idea is simple: an LLM reading a compressed Markdown summary understands a site faster and more accurately than an LLM parsing megabytes of navigation markup.
What llms.txt actually is
It is a Markdown file served at https://yourdomain.com/llms.txt. There is no official standard body behind it, but the convention has stabilised: an H1 with the project name, a blockquote summary, then H2 sections containing Markdown link lists with one-line descriptions.
# Kashi Ahmer — Developer & Webmaster Hub
> Free AI visibility and webmaster tools.
## Tools
- [AI Visibility Checker](https://kashiahmer.github.io/#aivc): audits a page for AI readiness
- [llms.txt Generator](https://kashiahmer.github.io/tools.html#llms): builds this file
## Guides
- [What is AI visibility](https://kashiahmer.github.io/what-is-ai-visibility-seo-2026.html): definitions and metricsWhy bother if adoption is partial?
- It costs one file and no maintenance once generated.
- Agents and coding assistants that do read it get a dramatically better map of your content.
- It forces you to decide which pages are actually important — a useful editorial exercise.
- It is a clean signal of intent that costs nothing if a crawler ignores it.
The rules that matter
- Keep the summary to one or two sentences. It is a description, not a pitch.
- Link only canonical URLs. A file full of redirects teaches the model nothing.
- Describe every link in eight to fifteen words. The description is the payload.
- Group by intent — tools, guides, documentation, company, legal.
- Add a content policy. State whether quoting is allowed and under what attribution.
Publish a second file called llms-full.txt containing the actual text of your main pages. Some agents fetch the full version, and it removes any ambiguity about what your pages say.
Common mistakes
| Mistake | Why it hurts | Fix |
|---|---|---|
| Listing 200 URLs | Noise; the model cannot tell what matters | Cap at 30–40 links, best pages only |
| Marketing copy in the summary | Reduces factual density | Write a neutral one-line description |
| Relative links | Break when parsed outside a browser | Always use absolute https URLs |
| Stale file | Describes a site that no longer exists | Regenerate when the site map changes |
Serving it correctly
Serve llms.txt as text/plain; charset=utf-8 at the domain root, not inside a subfolder. On GitHub Pages, Jekyll and most static hosts a plain file in the repository root is served automatically. If you use a CDN, make sure it does not rewrite the extension or force a download.
You can generate a compliant file in ten seconds with the llms.txt Generator on this site. It writes the header, sections, contact block and content policy for you.
Does it replace robots.txt or a sitemap?
No. robots.txt is access control, sitemap.xml is an exhaustive index for crawlers, and llms.txt is a curated briefing for models. All three serve different readers, and all three are cheap to maintain. Publish all three, and keep the sitemap exhaustive while keeping llms.txt selective.
Frequently asked questions
Where exactly does llms.txt go?
At the domain root: https://yourdomain.com/llms.txt. It must be absolute, public and served as plain text.
Is llms.txt officially supported by OpenAI or Google?
No. It is a community convention. It is harmless and sometimes useful, and adoption is growing among agentic tools.
Should llms.txt be in English?
Match your site language. If your site is mostly English, write it in English; bilingual sites can add both, but keep one section per language.