A crawl trap is a linking pattern that technically generates an unlimited, or near-unlimited, number of distinct URLs out of a finite content space. The difference from generic crawl budget waste (loose parameters, combined filters) is one of scale and shape: a crawl trap doesn't just waste budget, it can catch a crawler in a loop that, in practice, never ends, generating tens of thousands of new URLs for every real URL that existed. This guide details the most common patterns, how to detect them before a crawler gets stuck in one for hours, and how to close each one for good depending on the specific pattern.
Calendars and date pickers with no ceiling
The most classic pattern: a calendar widget or date picker that generates a "next month" link with its own URL (/events/2032/03/, /events/2032/04/...) with no upper date limit whatsoever. A crawler that follows those links systematically can move forward month by month indefinitely, because every calendar page, even one with no real events, still generates a valid "next" link to the one after it. Without anyone noticing, the site has created a URL space that grows with no practical limit, and every one of those empty URLs competes for the same crawl budget as pages with real content.
Faceted navigation: combinatorial explosion
In a catalog with filters (color, size, price, brand), every combination of active filters technically generates a distinct URL if the filters are reflected in the URL through parameters or path segments. The problem isn't linear but combinatorial: with just 5 filters of 4 values each, the number of possible combinations is around 1,024, and with 8 filters of 5 values it exceeds 390,000 combinations, almost none with real users behind them and most with the same (or nearly the same) content as another combination. An unbounded crawler discovers those combinations by following filter-upon-filter-upon-filter links, generating a URL space that's practically infinite even though the real catalog only has a few hundred products.
/sneakers?color=blue
/sneakers?color=blue&size=42
/sneakers?color=blue&size=42&brand=x
/sneakers?color=blue&size=42&brand=x&price=50-100
/sneakers?color=blue&size=42&brand=x&price=50-100&sort=price
... every extra filter multiplies, not adds, the possible URLsSession parameters embedded in the URL
Some systems (mostly older applications without well-configured cookies) embed a unique session identifier directly in the URL instead of in a cookie: /product/chair?PHPSESSID=a3f9c8e1b2. Every visit, including every crawler visit, generates a different session identifier, so technically every URL with a session ID is unique even though the content is exactly the same. If the crawler follows internal links that carry that parameter along (common when the site's own templating engine propagates the session parameter into every generated link), every new page visited generates links with yet another different session ID, multiplying URLs with no real limit.
Poorly implemented infinite pagination
Results pagination with no clear upper bound, combined with sort or order parameters (?page=847&sort=price_desc), can generate empty results pages indefinitely if the system doesn't return a 404 or equivalent status once it exceeds the real number of results, but instead keeps serving a 200 with a blank page and, worse, keeps linking to a "next" page that goes even higher. The combination of pagination with no real ceiling and multiple sort criteria multiplies the problem: every sort criterion generates its own infinite sequence of pages.
The relative-link bug that nests paths without end
A lesser-known but devastating pattern when it shows up: a badly built link using a relative path instead of an absolute one, in a template that repeats across every page. If a page at /catalog/chairs/ has a link mistakenly written as href="chairs/" (relative) instead of href="/catalog/chairs/" (absolute), the browser and the crawler resolve it by concatenating with the current path, generating /catalog/chairs/chairs/. If that resulting page contains the same template with the same broken link, the pattern repeats indefinitely: /catalog/chairs/chairs/chairs/chairs/.... Every extra level is a technically new URL, and if the server doesn't return an error for abnormally deep paths, the crawler can keep generating them for thousands of levels before anything stops it.
How to detect these patterns with a crawler
The most reliable signal isn't a single tool but a pattern in the crawl data: a volume of discovered URLs growing far beyond what the site's real content could justify, or a path depth (number of segments separated by /) that keeps increasing without settling as the crawl progresses. In Screaming Frog and equivalent tools, sorting the report by URL depth and checking whether the deepest URLs follow the pattern of a repeated folder is the fastest diagnostic. A custom extraction (via XPath or a regular expression) aimed at spotting links pointing to the same path segment as the current page catches the relative-link bug even before the crawl balloons to thousands of URLs, letting you stop the test crawl manually the moment the volume starts growing out of control.
Typical warning sign in a crawl:
Level 1: 40 URLs
Level 2: 180 URLs
Level 3: 1,200 URLs
Level 4: 9,800 URLs
Level 5: 78,000 URLs (and climbing)
A site with a real catalog of a few hundred products
shouldn't generate this kind of exponential growth per level.Concrete fixes per pattern
Every pattern needs a different fix, and applying the wrong generic one doesn't solve it. For infinite calendars: cap the navigable date range at the root (for example, don't generate "next month" links beyond a 12-18 month horizon) and add noindex to months with no real events. For faceted navigation: declare in Search Console which parameters shouldn't be crawled, apply rel="canonical" from every filter combination back to the unfiltered view (or to the most relevant combination), and block the noisiest parameter patterns in robots.txt. For session IDs in the URL: the only robust fix is to stop propagating the session through the URL and use cookies instead, fixing the problem at the source rather than trying to canonicalize every variant. For uncapped pagination: return a real 404 past the last existing result and stop linking a "next" page when there's no further content. For the relative-link bug: it's a code defect, not an SEO configuration issue, so the fix belongs in the template (always use absolute paths for internal links), and while it's being fixed, block in robots.txt any path containing a consecutively repeated folder using a regex pattern supported by the server.
Frequently asked questions
Can a crawl trap affect small sites too?
Yes, and it's more common than it seems: the relative-link bug or a poorly bounded calendar can show up on a site with only a few dozen real pages just as easily as on a large one, because the problem doesn't depend on catalog size but on a specific defect in a template repeated across the whole site.
Is blocking the pattern in robots.txt enough to fix the problem?
It's a containment fix, not a root-cause one: it stops more URLs of the pattern from being crawled, but if the pattern is generated by a code bug (like broken relative links), the robots.txt block doesn't fix the cause and those infinite URLs will keep existing technically, just out of sight of crawlers that respect the block.
How do I know if a crawl trap has already affected my real crawl budget?
By reviewing server logs (see the crawling and indexing guide) filtered by Googlebot: if a disproportionate volume of real requests concentrates on the suspicious pattern, the crawl trap is already active and eating budget that should be going to valuable pages.
Do crawl traps also affect real users' experience, not just crawlers?
Usually not, because a real user doesn't navigate by systematically following every "next month" link or every filter combination the way an exhaustive crawler does; the impact is almost exclusively technical and SEO-related, though a badly built relative-link bug can sometimes also break visible navigation for a user who clicks the wrong path by mistake.