Every guide in this section goes deep on one specific technical problem, but none of them replaces the prior question every real audit needs to answer first: of all the possible problems, which ones actually exist on this specific site, and how severe are they? This guide isn't about a specific technical problem, but about the methodological process for finding all of them systematically with a crawling tool, instead of checking at random or relying only on intuition.
Before crawling: configuring the crawler to reflect reality
A badly configured crawl produces misleading data before you even start analyzing it. Prior configuration should include: the correct user-agent (crawling as Googlebot, not the tool's default user-agent, because some sites serve different content depending on the detected user-agent); whether to respect robots.txt depending on what you're auditing (crawling while ignoring robots.txt temporarily lets you see what actually exists, even if blocked, useful for catching accidental blocks); and the crawl rate limit, tuned so as not to overload the production server during the audit, especially on shared hosting with limited resources.
The first report to check: HTTP response codes
Before analyzing anything else, it's worth getting an overview of how many URLs return each status code (200, 301, 404, 500), as explained in the redirects and HTTP codes guide. A high volume of 404s reveals broken internal links or content removed without a redirect; a high volume of chained 301s reveals poorly cleaned-up past migrations; any 500 indicates server errors that must be fixed with top priority, since they affect both real users and Google's crawling.
Typical summary of a full crawl:
200 OK 94.2%
301 Redirect 3.1%
404 Not Found 2.4%
5xx Server Error 0.3%
0.3% 5xx errors on a 10,000-URL site
means 30 pages actively failing: high priority.The second report: duplicate titles and meta descriptions
A crawler automatically groups pages that share the same <title> or the same meta description, a direct signal of duplicate or near-duplicate content (see the canonicalization guide) that in practice usually reveals poorly parameterized templates: product pages inheriting the category's generic title instead of a specific one, or a default meta description that was never overwritten when the content was created.
The third report: click depth and internal links
Cross-referencing each URL's click depth (how many clicks it takes from the homepage to reach it) with the number of internal inbound links it receives, as explained in the architecture and internal linking guide, quickly identifies high business-value pages buried too deep in the structure, top candidates for reinforced internal linking.
The fourth report: structured data and validation
A full crawl can automatically extract and validate the JSON-LD present on every page, cross-checking it against the properties required by each Schema.org type (see the structured data guide), to catch at scale pages with missing, malformed, or inconsistent markup, something unfeasible to check manually page by page on a site with thousands of URLs.
The fifth report: aggregate performance
Many crawling tools integrate a call to the PageSpeed Insights API for every crawled URL, letting you see at a glance which page templates (not just which individual pages) systematically have worse LCP, INP or CLS, as explained in the Core Web Vitals guide. This is more efficient than checking page by page because it reveals patterns: if every product page fails the same threshold, the problem is in the template, not in a specific product.
How to prioritize findings: impact by volume, not isolated severity
The prioritization criterion that works best in practice isn't "which problem sounds worse in the abstract" but "which problem affects the most pages of real value." A misconfigured canonical affecting 5,000 product pages is more urgent than an isolated 404 on a page with no traffic or links, even though the latter "sounds" like a more visible failure. Cross-referencing each technical finding against real traffic and business-value data (not just the volume of affected URLs) is what separates a useful audit from a generic list of technical problems with no context.
Documenting findings with evidence, not just a conclusion
A useful technical finding isn't "there's duplicate content," but the specific list of affected URLs, grouped and with representative examples, along with the identified root cause (a specific template, a specific configuration). This lets whoever implements the fix do so without repeating the diagnostic process from scratch, and serves as a reference to later verify the fix actually worked.
Frequently asked questions
How often should I repeat a full technical audit?
For most sites, a full technical audit once or twice a year is enough, complemented by ongoing monitoring of Search Console reports (coverage, Core Web Vitals, security issues) that flag new problems between full audits.
Should I always crawl ignoring robots.txt?
Not as a default setting: crawling while respecting robots.txt reflects what Google can actually see. Ignoring it temporarily is only useful as a one-off check to catch whether some robots.txt block is accidental and hiding content that should be crawlable.
Does a crawl also detect content problems, not just technical ones?
It can approximate some (word count, missing H1, titles too short or too long), but it doesn't replace an editorial content-quality review, which requires human judgment about relevance and real usefulness to the user, something a crawler can't evaluate.
How many URLs can I crawl without affecting production server performance?
It depends on the hosting's capacity, but as a cautious reference: limiting the crawl to 1-2 requests per second avoids overloading most shared hosting, at the cost of the audit taking longer to complete on large sites.
Do I need a paid tool to run a serious technical audit?
Not necessarily for small or medium sites: several crawling tools offer a free tier with a URL limit sufficient for sites up to a few thousand pages. The need for a paid version mainly shows up on very large sites or when recurring audits need to be automated.