AI-powered
A robots.txt tester that reads a site's rules and sitemaps and tests whether search engines and AI crawlers may read an address.
AvailableReady to use.
Open the tool| Crawler | Address | Result |
|---|---|---|
| Googlebot | /blog/post | Allowed |
| Googlebot | /search?q=x | Blocked |
| GPTBot | /blog/post | Blocked |
AI explains each rule in plain words. Whether an address is allowed is decided by the same matching rules search engines use.
The file is fetched and its rules are grouped by crawler.
Sitemaps listed in robots.txt or at the usual address are read.
Your address is matched against each crawler's rules.
Which rule decided, and what to change.
robots.txt is a small text file at the root of a site that tells crawlers which parts they may request. A robots.txt tester reads that file and checks whether a given address is allowed for a given crawler, such as Googlebot or an AI crawler.
One wrong line can hide a whole section from search, so testing before and after a change is worth a minute.
It groups the rules by crawler, flags lines that are not valid, and finds rules that block CSS or scripts that pages need to display. You can test any address against search crawlers and AI crawlers like GPTBot, ClaudeBot and PerplexityBot.
It also reads the sitemaps the file lists, counts their addresses and spots addresses that fail or that robots.txt blocks.
robots.txt controls crawling, not indexing. A blocked page can still appear in search if other pages link to it. To keep a page out of search, let it be crawled and use a noindex tag.
Most robots.txt accidents happen in a release, when a test site's rules reach the live site. Change Monitor checks robots.txt and the sitemap and warns you when they change.
No. It stops crawling. A blocked page can still be indexed from links. Use noindex to keep it out.
It depends on whether you want AI assistants to read and quote your pages. The tester shows what your file says today.
For Google, the most specific rule wins, and Allow wins a tie. The tester shows which rule decided.