Robots.txt guide: rules, examples and mistakes to avoid

A robots.txt file tells search engine crawlers which parts of your site they may request. It lives at the root of your domain, it controls crawling rather than indexing, and one wrong line can hide your most important pages from Google.

This robots.txt guide shows you how the file works, how to write rules that do what you expect, and how to catch the mistakes that quietly cost traffic.

In this guide

• What a robots.txt file is and where it lives
• How robots.txt rules work
• Robots.txt examples you can adapt
• Robots.txt vs noindex: which one to use
• Common robots.txt mistakes
• How to test and monitor your robots.txt file

What is a robots.txt file?

A robots.txt file is a plain text file at the root of a host, such as https://example.com/robots.txt, that lists which paths crawlers may and may not request. Well-behaved crawlers like Googlebot and Bingbot read it before they crawl the site.

Three details matter more than most people expect:

• One file per host. www.example.com, example.com and shop.example.com each need their own file. The protocol and port count too, so http:// and https:// are treated separately.
• It must sit at the root. A file at /blog/robots.txt is ignored.
• It is public. Anyone can read it, so never use it to hide private areas. Use authentication for that.

How do robots.txt rules work?

Robots.txt rules work in groups: a User-agent line names the crawler, and the Disallow and Allow lines under it set what that crawler may request. The crawler follows the most specific group that matches its name.

A few rules of thumb keep the file predictable:

1. Paths are case sensitive. /Products/ and /products/ are different paths.
2. The longest matching rule wins for Google. If Disallow: /docs/ and Allow: /docs/public/ both match, the more specific Allow applies.
3. Wildcards help. Google supports * for any sequence of characters and $ for the end of a URL, so Disallow: /*.pdf$ blocks PDF files.
4. Google ignores Crawl-delay. If Googlebot crawls too fast, the server slowing down with 429 or 503 replies is the signal Google listens to.

Robots.txt examples you can adapt

These robots.txt examples cover the situations agencies meet most often. Always check them against your own URLs before you publish.

Allow everything, and point to the sitemap

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

An empty Disallow means nothing is blocked. This is the safest starting point for most sites.

Block internal search results and cart pages

User-agent: *
Disallow: /search
Disallow: /cart/
Disallow: /checkout/

Internal search pages create endless low-value URLs, so blocking them saves crawl effort for pages that matter.

Block one crawler only

User-agent: ExampleBot
Disallow: /

This blocks a single named crawler while everyone else keeps full access.

Robots.txt vs noindex: which should you use?

Use robots.txt to stop crawling, and use a noindex tag to stop a page from appearing in search results. They solve different problems, and mixing them up is one of the most common technical SEO errors.

A page blocked in robots.txt can still appear in Google if other sites link to it, because Google can index the URL without reading the page. Worse, if the page carries a noindex tag, Google never sees that tag, because it is not allowed to fetch the page.

Common robots.txt mistakes

These robots.txt mistakes appear again and again in audits, and each one can wipe out traffic on its own.

1. A leftover Disallow: / from staging. A development site ships with everything blocked and the rule goes live with the launch.
2. Blocking CSS and JavaScript. Google renders pages much like a browser. Blocking /assets/ or /_next/ can make pages look broken to it.
3. Using robots.txt to remove indexed pages. As explained above, a block stops Google from seeing noindex, so the pages can linger.
4. Forgetting other hosts. The www host gets a careful file while the bare domain serves a blocking one.
5. A file that is too large. Google reads up to 500 KiB of a robots.txt file and ignores anything after that.
6. Server errors on the file. If robots.txt returns a 5xx error, Google treats the whole site as temporarily blocked. A 4xx error, by contrast, is read as "no restrictions".

How to test and monitor your robots.txt file

Test every change before it goes live, then monitor the file, because it changes more often than anyone expects: a deploy, a plugin or a new hosting setup can rewrite it.

1. Fetch the live file at /robots.txt on every host you own and confirm it returns 200.
2. Check important URLs against it. Take your top pages and confirm none of them match a Disallow rule. The Robots.txt Tester tests any address against Googlebot, Bingbot and AI crawlers using Google's rule order, and checks your sitemaps too.
3. Crawl the site. A website crawler reads robots.txt first and shows exactly which URLs are blocked, so you see the real effect instead of guessing.
4. Watch for changes. The Change Monitor checks robots.txt every hour and raises one clear incident when a new block appears on an important path.
5. Turn findings into fixes. The SEO Issue Checker flags robots.txt server errors, sudden changes and new blocks on important paths, each with its proof.

Google's own robots.txt introduction is the reference for how its crawlers read the file.

The short version

Keep your robots.txt file small, boring and tested. Allow crawling by default, block only what wastes crawl effort, never use it for privacy or to remove pages, and watch it for changes. For the wider picture, work through the technical SEO checklist next.