What is llms.txt? A plain guide with an example file

llms.txt is a plain Markdown file at the root of a website, at /llms.txt, that gives AI assistants a short, curated map of the site: what it is, and links to the pages that matter most, each with a one line description. It was proposed in 2024 as a way to help large language models find a site's best content without wading through menus, scripts and ads.

This guide explains the format, shows an example file, compares it with robots.txt, and is honest about what it can and can't do today.

What is llms.txt for?

Web pages are built for browsers. A large language model reading a page has to get past navigation, cookie banners, scripts and layout code to reach the content, and the useful pages may be several clicks deep. llms.txt solves a narrow problem: it gives a model one small file with a summary of the site and direct links to the pages worth reading.

It's most useful for sites with documentation, product details or reference content that people ask AI assistants about.

How the llms.txt format works

The format, described in the llms.txt proposal at llmstxt.org, is ordinary Markdown in a set order:

1. An H1 with the site or project name. This is the only required part.
2. A blockquote with a short summary of what the site is.
3. Optional paragraphs with any details a reader needs to understand the rest.
4. H2 sections with lists of links, each link followed by a short description.
5. An optional section called "Optional" for links that can be skipped when space is short.

An example llms.txt file

# Example Tools

> Example Tools makes uptime and DNS checks for small websites. This file lists the pages that explain what each tool does.

## Tools

- [Uptime checks](https://example.com/uptime): Checks a page every minute and alerts after a confirmed failure.
- [DNS lookup](https://example.com/dns): Looks up A, MX and TXT records and checks SPF and DMARC.

## Guides

- [Fix a 503 error](https://example.com/guides/503): What a 503 means and how to find the cause.

## Optional

- [Company news](https://example.com/news): Release notes and announcements.

Keep descriptions factual and short. They should say what each page contains, not market it.

llms.txt vs robots.txt

The two files are often mentioned together, but they do different jobs:

llms.txt doesn't block or allow anything. If you want to control whether AI crawlers such as GPTBot, ClaudeBot or PerplexityBot can fetch your pages, that still happens in robots.txt.

Does llms.txt help SEO or AI visibility?

Be realistic. llms.txt is a proposal, not a standard, and support varies. Google has said publicly that its Search doesn't use llms.txt, and adding one won't change your Google rankings. Some AI tools and coding assistants do read it, especially for documentation.

So treat it as cheap and harmless rather than as a ranking lever. What matters far more for being quoted by AI assistants is the same as for search: pages that crawlers are allowed to read, clear answers near the top, accurate facts and a site people trust.

Which AI crawlers should you allow?

llms.txt only helps if AI tools can read your pages, and that's set in robots.txt. AI crawlers fall into two groups:

• Training crawlers, such as GPTBot, ClaudeBot, Google-Extended and CCBot, collect pages to train models.
• Search and answer crawlers, such as OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-User, fetch pages to answer a person's question, often with a link back to you.

Many sites block training crawlers and allow the search and answer ones, because those are the ones that can send visitors. It's a business decision with no single right answer. Whatever you choose, check that robots.txt says what you intend, since one broad Disallow can block both groups at once.

What to include in llms.txt, and what to leave out

A good llms.txt is short. Include the pages someone would need to understand and use what you offer:

• What you do: the home or about page, and the main product or service pages.
• How it works: documentation, setup guides and pricing.
• Answers people ask for: your best guides and FAQ pages.

Leave out what adds noise: tag and archive pages, login and account pages, thin or outdated posts, and anything you've blocked in robots.txt. Some sites also publish a longer llms-full.txt with the full text of their docs. That's a convention some documentation tools use, not part of the core proposal, so add it only if your docs are the main thing people ask about.

How to add llms.txt to your site

1. List your most useful pages: product, pricing, documentation, key guides.
2. Write the summary in one or two plain sentences.
3. Group the links under H2 headings, with a short description for each.
4. Save it as /llms.txt at the root of the domain, served as plain text.
5. Check robots.txt so the AI crawlers you want aren't blocked.
6. Update it when you add or remove important pages, so it never links to dead URLs.

Create one with the LLMs.txt Generator

The LLMs.txt Generator checks which AI crawlers your robots.txt allows or blocks, including GPTBot, ClaudeBot and PerplexityBot, and whether an llms.txt already exists and follows the format. It then reads a small number of pages from your sitemap, obeying robots.txt, and writes a ready llms.txt from your own titles and descriptions for you to copy.

To check the robots.txt rules themselves, use the Robots.txt Tester, which tests any address against search engine and AI crawlers. Our robots.txt guide explains how those rules work.