How to use a website crawler for an SEO audit

A website crawler is a program that starts at one page, follows every link it finds, and records what each URL returns, just as a search engine bot does. For an SEO audit you point it at the site, choose a polite speed and a URL limit, let it finish, then read the status codes, directives, canonicals, titles and links it collected.

This guide shows you how to set up a website crawler for an audit, what each part of the results tells you, and the mistakes that waste a crawl.

In this guide

• How a website crawler works
• Settings to choose before you crawl
• How to run an SEO audit crawl in 7 steps
• How to read the results
• Common crawling mistakes

How does a website crawler work?

A website crawler works by keeping a queue of URLs to visit. It takes the next URL, fetches it, saves what the server returned, pulls out the links, and adds any new internal links to the back of the queue. It repeats until the queue is empty or a limit is reached.

A careful crawler also reads robots.txt and the XML sitemaps first, because both shape what it should visit. Once the crawl is done, it works out three numbers for every page:

• Crawl depth: how many clicks the page sits from the start URL.
• Inlinks: how many internal links point to it.
• Orphan candidates: pages found in the sitemap that no crawled page links to.

These three numbers tell you more about site structure than any diagram.

Website crawler settings that matter for an SEO audit

The settings that matter most are speed, URL limit, scope and robots.txt handling. Get those four right and the rest can stay on defaults.

Two notes on robots.txt. If the file returns a server error, stop and report it, because Google treats that as the whole site being blocked for a while. If it returns a 404, crawling is allowed. The Google robots.txt guide explains what the file can and can't do.

How to run an SEO audit crawl in 7 steps

Follow these steps in order so the crawl answers the audit's questions instead of burying you in rows.

1. Check robots.txt and the sitemap by hand. Open both in a browser. Note any blocked folders and how many URLs the sitemap lists.
2. Pick the start points. Use the home page, and add the sitemap so pages with no links still get fetched.
3. Set speed, scope and limit. Start slow on a host you don't know.
4. Add exclude rules for carts, logout links, internal search results and calendar pages.
5. Run a short test crawl of about 100 URLs. Look for parameter loops or a sudden flood of near identical URLs, then fix the rules.
6. Run the full crawl and watch the found, done, failed and blocked counts as it runs.
7. Export and sort by the tabs below, then move findings into tickets.

How to read website crawler results

Read the results tab by tab, from the problems that block many pages to the ones that affect a single page. Each tab is a filter on the same URL table.

Status codes

Start here. Sort by 5xx, then 4xx, then 3xx. A 5xx on a key template is urgent. A 404 matters when internal links still point to it. A redirect matters when it is part of a chain, ends on an error, or is linked from your own menus.

Directives

Look for noindex in the meta robots tag and in the X-Robots-Tag header. Then ask one question for each: is this page meant to be out of search? An unexpected noindex on a product template is one of the most costly mistakes a site can make.

Canonicals

Filter for missing, multiple, pointing elsewhere and pointing to a bad URL. A canonical that points to a 404 or a redirected URL sends a confusing signal about which page to index.

Titles, descriptions and headings

Filter for missing and duplicate first. Treat "too long" and "too short" as guidance, since Google sets no fixed limit and rewrites snippets when it likes.

Links and sitemaps

Check broken internal links, pages with very high depth and pages with no inlinks. Then compare the crawl with the sitemap: pages in the sitemap but not found by crawling are orphan candidates, and pages found by crawling but missing from the sitemap may need adding.

Images

Missing alt text, large files and broken images live here. Fix the broken ones first.

A worked example

Here is an example of how one crawl reads. Say you crawl a 4,800 URL site and see 4,100 pages return 200, 520 return a 301 and 180 return a 404. That 301 count looks alarming, but sorting by inlinks shows most come from one menu link to an old category path. Updating that single menu link removes hundreds of internal redirects at once. The 404 list, sorted by inlinks, then shows which dead pages still get traffic from your own links. Those go first.

The lesson: sort every list by inlinks or by template before you count it. Counts alone mislead.

How often should you crawl a website?

Crawl a full site at least weekly while you are actively working on it, and after every large release or migration. Between crawls, check the high risk files, robots.txt and the sitemap, more often.

The reason is comparison. A single crawl tells you what is broken today. Two crawls a week apart tell you what changed: a section that lost its titles, a folder that started returning 404, a template that dropped its canonical. Keep each crawl, label it with the date and any deploy, and compare the counts per tab. A sudden jump in any filter is the first thing to investigate.

Common website crawler mistakes

Most crawl problems come from settings, not from the site.

• Crawling too fast. You slow the site for real visitors and may get blocked, which hides the data you came for.
• No URL limit. Faceted filters can create millions of URLs. Always cap the crawl and watch for repeating patterns.
• Ignoring JavaScript. If pages look blank, the site may build its content in the browser. Check one page with rendering before you report missing titles.
• Mixing subdomains. A blog on another subdomain is a separate audit with its own robots.txt.
• Pretending to be Googlebot. Some firewalls treat a fake Googlebot as an attack. Use a named crawler and ask the host to allow it.
• Crawling once. One crawl is a snapshot. Comparing crawls over time is what shows you a template change or a lost section.

Where the platform fits

the platform is building a website crawler that runs on its servers and shows results in the browser, so there is nothing to install. It is designed with the defaults above: 2 requests per second, slowing on 429 and 503, obeying robots.txt and a 100 URL cap for sites whose owner has not been verified. The results feed the SEO Issue Checker, which turns crawl data into a ranked list of problems with proof. To decide which of those problems to check first, use our technical SEO audit checklist.