What is llms.txt? A guide with an example | ProveFix
AI and SEO
What is llms.txt? A plain guide with an example file
llms.txt is a proposed Markdown file that gives AI assistants a short map of your site. Here is what it is, how the format works, an example file, and an honest look at what it can and can't do.
By Editorial team··6 min read·Updated
llms.txt is a plain Markdown file at the root of a website, at /llms.txt, that gives AI assistants a short, curated map of the site: what it is, and links to the pages that matter most, each with a one line description. It was proposed in 2024 as a way to help large language models find a site's best content without wading through menus, scripts and ads.
This guide explains the format, shows an example file, compares it with robots.txt, and is honest about what it can and can't do today.
What is llms.txt for?
Web pages are built for browsers. A large language model reading a page has to get past navigation, cookie banners, scripts and layout code to reach the content, and the useful pages may be several clicks deep. llms.txt solves a narrow problem: it gives a model one small file with a summary of the site and direct links to the pages worth reading.
It's most useful for sites with documentation, product details or reference content that people ask AI assistants about.
A blockquote with a short summary of what the site is.
Optional paragraphs with any details a reader needs to understand the rest.
H2 sections with lists of links, each link followed by a short description.
An optional section called "Optional" for links that can be skipped when space is short.
An example llms.txt file
# Example Tools
> Example Tools makes uptime and DNS checks for small websites. This file lists the pages that explain what each tool does.
## Tools
- [Uptime checks](https://example.com/uptime): Checks a page every minute and alerts after a confirmed failure.
- [DNS lookup](https://example.com/dns): Looks up A, MX and TXT records and checks SPF and DMARC.
## Guides
- [Fix a 503 error](https://example.com/guides/503): What a 503 means and how to find the cause.
## Optional
- [Company news](https://example.com/news): Release notes and announcements.
Keep descriptions factual and short. They should say what each page contains, not market it.
llms.txt vs robots.txt
The two files are often mentioned together, but they do different jobs:
robots.txt
llms.txt
Purpose
Says which paths crawlers may request
Points to the pages worth reading
Format
Plain text rules
Markdown
Standard
Widely followed for decades
A proposal, adopted by some sites and tools
Controls access
Yes, for crawlers that obey it
No
llms.txt doesn't block or allow anything. If you want to control whether AI crawlers such as GPTBot, ClaudeBot or PerplexityBot can fetch your pages, that still happens in robots.txt.
Does llms.txt help SEO or AI visibility?
Be realistic. llms.txt is a proposal, not a standard, and support varies. Google has said publicly that its Search doesn't use llms.txt, and adding one won't change your Google rankings. Some AI tools and coding assistants do read it, especially for documentation.
So treat it as cheap and harmless rather than as a ranking lever. What matters far more for being quoted by AI assistants is the same as for search: pages that crawlers are allowed to read, clear answers near the top, accurate facts and a site people trust.
Which AI crawlers should you allow?
llms.txt only helps if AI tools can read your pages, and that's set in robots.txt. AI crawlers fall into two groups:
Training crawlers, such as GPTBot, ClaudeBot, Google-Extended and CCBot, collect pages to train models.
Search and answer crawlers, such as OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-User, fetch pages to answer a person's question, often with a link back to you.
Many sites block training crawlers and allow the search and answer ones, because those are the ones that can send visitors. It's a business decision with no single right answer. Whatever you choose, check that robots.txt says what you intend, since one broad Disallow can block both groups at once.
What to include in llms.txt, and what to leave out
A good llms.txt is short. Include the pages someone would need to understand and use what you offer:
What you do: the home or about page, and the main product or service pages.
How it works: documentation, setup guides and pricing.
Answers people ask for: your best guides and FAQ pages.
Leave out what adds noise: tag and archive pages, login and account pages, thin or outdated posts, and anything you've blocked in robots.txt. Some sites also publish a longer llms-full.txt with the full text of their docs. That's a convention some documentation tools use, not part of the core proposal, so add it only if your docs are the main thing people ask about.
How to add llms.txt to your site
List your most useful pages: product, pricing, documentation, key guides.
Write the summary in one or two plain sentences.
Group the links under H2 headings, with a short description for each.
Save it as /llms.txt at the root of the domain, served as plain text.
Check robots.txt so the AI crawlers you want aren't blocked.
Update it when you add or remove important pages, so it never links to dead URLs.
Create one with the LLMs.txt Generator
The LLMs.txt Generator checks which AI crawlers your robots.txt allows or blocks, including GPTBot, ClaudeBot and PerplexityBot, and whether an llms.txt already exists and follows the format. It then reads a small number of pages from your sitemap, obeying robots.txt, and writes a ready llms.txt from your own titles and descriptions for you to copy.
To check the robots.txt rules themselves, use the Robots.txt Tester, which tests any address against search engine and AI crawlers. Our robots.txt guide explains how those rules work.
Tags
llms.txt
ai search
ai crawlers
robots.txt
Questions and answers
What is llms.txt?
llms.txt is a Markdown file at /llms.txt that gives AI assistants a short summary of a site and links to its most useful pages, each with a one line description.
Is llms.txt the same as robots.txt?
No. robots.txt controls which paths crawlers may request. llms.txt doesn't block or allow anything. It only points to the pages worth reading.
Does Google use llms.txt?
Google has said its Search doesn't use llms.txt, so it won't change Google rankings. Some AI tools and coding assistants do read it.
Where do I put the llms.txt file?
At the root of the domain, such as https://example.com/llms.txt, served as plain text, like robots.txt.
