Robots.txt guide: rules, examples and mistakes to avoid
A robots.txt file controls what crawlers may request, not what gets indexed. Learn how the rules work, see examples you can adapt, and avoid the mistakes that hide pages from Google.
By Editorial team··5 min read·Updated
A robots.txt file tells search engine crawlers which parts of your site they may request. It lives at the root of your domain, it controls crawling rather than indexing, and one wrong line can hide your most important pages from Google.
This robots.txt guide shows you how the file works, how to write rules that do what you expect, and how to catch the mistakes that quietly cost traffic.
In this guide
What a robots.txt file is and where it lives
How robots.txt rules work
Robots.txt examples you can adapt
Robots.txt vs noindex: which one to use
Common robots.txt mistakes
How to test and monitor your robots.txt file
What is a robots.txt file?
A robots.txt file is a plain text file at the root of a host, such as https://example.com/robots.txt, that lists which paths crawlers may and may not request. Well-behaved crawlers like Googlebot and Bingbot read it before they crawl the site.
Three details matter more than most people expect:
One file per host.www.example.com,
example.com
and
shop.example.com
each need their own file. The protocol and port count too, so
http://
and
https://
are treated separately.
It must sit at the root. A file at /blog/robots.txt is ignored.
It is public. Anyone can read it, so never use it to hide private areas. Use authentication for that.
How do robots.txt rules work?
Robots.txt rules work in groups: a User-agent line names the crawler, and the Disallow and Allow lines under it set what that crawler may request. The crawler follows the most specific group that matches its name.
Directive
What it does
Example
User-agent
Names the crawler a group applies to
User-agent: Googlebot
Disallow
Blocks a path from being crawled
Disallow: /cart/
Allow
Re-opens a path inside a blocked folder
Allow: /cart/help
Sitemap
Points crawlers to your XML sitemap
Sitemap: https://example.com/sitemap.xml
A few rules of thumb keep the file predictable:
Paths are case sensitive./Products/ and /products/ are different paths.
The longest matching rule wins for Google. If Disallow: /docs/ and Allow: /docs/public/ both match, the more specific Allow applies.
Wildcards help. Google supports * for any sequence of characters and $ for the end of a URL, so Disallow: /*.pdf$ blocks PDF files.
Google ignores Crawl-delay. If Googlebot crawls too fast, the server slowing down with 429 or 503 replies is the signal Google listens to.
Robots.txt examples you can adapt
These robots.txt examples cover the situations agencies meet most often. Always check them against your own URLs before you publish.
Internal search pages create endless low-value URLs, so blocking them saves crawl effort for pages that matter.
Block one crawler only
User-agent: ExampleBot
Disallow: /
This blocks a single named crawler while everyone else keeps full access.
Robots.txt vs noindex: which should you use?
Use robots.txt to stop crawling, and use a noindex tag to stop a page from appearing in search results. They solve different problems, and mixing them up is one of the most common technical SEO errors.
A page blocked in robots.txt can still appear in Google if other sites link to it, because Google can index the URL without reading the page. Worse, if the page carries a noindex tag, Google never sees that tag, because it is not allowed to fetch the page.
Goal
Right tool
Keep a page out of search results
noindex meta tag or X-Robots-Tag header, with crawling allowed
Save crawl effort on endless URLs
Disallow in robots.txt
Keep content private
Password protection or login
Remove a page for good
Return 404 or 410
Common robots.txt mistakes
These robots.txt mistakes appear again and again in audits, and each one can wipe out traffic on its own.
A leftover Disallow: / from staging. A development site ships with everything blocked and the rule goes live with the launch.
Blocking CSS and JavaScript. Google renders pages much like a browser. Blocking /assets/ or /_next/ can make pages look broken to it.
Using robots.txt to remove indexed pages. As explained above, a block stops Google from seeing noindex, so the pages can linger.
Forgetting other hosts. The www host gets a careful file while the bare domain serves a blocking one.
A file that is too large. Google reads up to 500 KiB of a robots.txt file and ignores anything after that.
Server errors on the file. If robots.txt returns a 5xx error, Google treats the whole site as temporarily blocked. A 4xx error, by contrast, is read as "no restrictions".
How to test and monitor your robots.txt file
Test every change before it goes live, then monitor the file, because it changes more often than anyone expects: a deploy, a plugin or a new hosting setup can rewrite it.
Fetch the live file at /robots.txt on every host you own and confirm it returns 200.
Check important URLs against it. Take your top pages and confirm none of them match a Disallow rule. The Robots.txt Tester tests any address against Googlebot, Bingbot and AI crawlers using Google's rule order, and checks your sitemaps too.
Crawl the site. A website crawler reads robots.txt first and shows exactly which URLs are blocked, so you see the real effect instead of guessing.
Watch for changes. The Change Monitor checks robots.txt every hour and raises one clear incident when a new block appears on an important path.
Turn findings into fixes. The SEO Issue Checker flags robots.txt server errors, sudden changes and new blocks on important paths, each with its proof.
Keep your robots.txt file small, boring and tested. Allow crawling by default, block only what wastes crawl effort, never use it for privacy or to remove pages, and watch it for changes. For the wider picture, work through the technical SEO checklist next.
Tags
robots.txt
crawling
technical seo
indexing
Questions and answers
Does robots.txt stop a page from being indexed?
No. Robots.txt stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. Use a noindex tag on a crawlable page to keep it out of search.
Where should the robots.txt file be placed?
At the root of each host, such as https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain and protocol needs its own file.
Does Google follow Crawl-delay?
No. Google ignores the Crawl-delay line. If Googlebot crawls too fast, slowing responses with 429 or 503 status codes is the signal it respects.
What happens if robots.txt returns an error?
A 4xx error is treated as no restrictions, so crawlers may fetch everything. A 5xx error makes Google treat the site as temporarily blocked, which can pause crawling.
Should I block CSS and JavaScript files?
No. Google renders pages much like a browser, so it needs your CSS and JavaScript to understand layout and content. Blocking them can make pages look broken to search engines.
