Before a page can rank it has to go through two distinct processes that get confused constantly: crawling, the visit of a bot to a URL to download its content, and indexing, the later decision to store that URL in the database Google uses to answer searches. A page can be crawled and never indexed, and on large sites this happens constantly without anyone noticing until traffic is missing.
This guide goes into the technical detail of both processes: what controls how much Google crawls your site, how to read robots.txt and sitemaps the way a crawler does, and how to diagnose with real data, not assumptions, why a specific URL isn't in the index.
What crawl budget is and why it isn't infinite
Crawl budget is the number of URLs Googlebot is willing to crawl on your domain within a given period. It doesn't affect every site equally: Google allocates more budget to domains it perceives as more important or that change more frequently, and it calculates it by combining two independent factors.
The first is the crawl rate limit: how many simultaneous requests, and with what spacing between them, Googlebot can make without degrading your server's performance. If your hosting starts responding more slowly or with 5xx errors, Google automatically lowers this limit, and takes a while to raise it again even after the problem is fixed. The second is crawl demand: the actual interest Google has in revisiting your URLs, which rises if content changes often, if the page receives links and traffic, or if Google detects that a section updates regularly.
On small sites (a few hundred URLs) crawl budget is almost never the bottleneck: Google can crawl the whole site several times a week without effort. The problem shows up on sites with thousands or tens of thousands of URLs, especially if a meaningful share of them are low value: combined-filter pages on an ecommerce site, infinite calendars, session or tracking parameters that generate distinct URLs for the same content. Every one of those low-value URLs competes for the same budget as the pages you actually want to rank.
robots.txt: the first gate, not the only one
The /robots.txt file is the first thing a crawler that respects the exclusion protocol (Googlebot, Bingbot and most legitimate bots) checks before crawling any other resource on the domain. It defines which paths each user-agent may visit, but it's important to understand its limits: it's a crawling directive, not an indexing one. Blocking a URL in robots.txt stops Googlebot from visiting it, but if that URL already has external links pointing to it, it can still show up in search results (usually with no description, just the URL), because Google indexes it from external signals without ever having crawled it.
The basic syntax combines User-agent blocks with Disallow/Allow directives, and supports the * wildcard and end-of-path anchoring with $:
User-agent: *
Disallow: /cart/
Disallow: /*?sessionid=
Disallow: /*.pdf$
Allow: /wp-content/uploads/
Sitemap: https://www.yourdomain.com/sitemap.xmlCommon mistakes seen in real audits: accidentally blocking resource folders the rendering itself needs (CSS or JS inside /wp-content/ or /assets/, which if blocked stop Google from seeing the page the way a user does); leaving a Disallow: / inherited from a staging environment when the site goes live, blocking the entire domain in one shot; and confusing robots.txt with a privacy tool, when in reality it doesn't stop access from anyone who decides to ignore it (it's not a security measure, it's a good-faith request).
XML sitemaps: the map that helps but doesn't force anything
An XML sitemap is a list of URLs you're telling Google you consider relevant, along with optional metadata like the last modification date. It doesn't guarantee indexing (Google can ignore sitemap URLs it considers low quality or duplicate) but it does speed up discovery, especially on new sites or sections that change often.
Practical rules that make a real difference on large sites: a sitemap can't exceed 50,000 URLs or 50 MB uncompressed, so past that volume you need a sitemap index pointing to several child sitemaps, usually split by content type (products, categories, blog); the <lastmod> field only adds value if it's accurate, updated automatically every time the actual content changes rather than stamped with the day the sitemap was regenerated on a schedule, because Google detects and discounts the confidence of sitemaps that systematically lie about this date; and the sitemap should only include canonical, indexable URLs that return 200, never redirected, noindexed or erroring URLs, because each of those inconsistencies gets reported as "excluded" in Search Console and erodes Google's trust in the whole file.
Meta robots and X-Robots-Tag: two ways of saying the same thing
Once Googlebot crawls a page, it decides whether to index it based on the directives it finds. The best known is the <meta name="robots"> tag in the HTML <head>:
<meta name="robots" content="noindex, follow">This combination is the right one when you want a page out of the index but still want Google to crawl the links it contains (for example, an intermediate pagination page). The most common mistake is defaulting to noindex, nofollow without thinking it through, which also cuts off crawl flow to whatever that page links to.
When the resource isn't HTML (a PDF, an image, a JSON response), the meta tag doesn't work because there's no <head> to put it in; that's where the equivalent HTTP header, X-Robots-Tag, comes in, configured at server level and accepting exactly the same values:
Header set X-Robots-Tag "noindex, nofollow"A contradiction that shows up often on sites with a long history: a URL blocked in robots.txt that also carries a noindex in the HTML. Since it's blocked, Google never gets to see that tag, so if the URL was already indexed through external signals, the noindex does nothing until the crawl is unblocked long enough for Google to revisit it and process the directive.
Server log analysis: seeing the real crawl, not the estimated one
Search Console shows a useful but limited aggregated sample of crawling. Server logs (the raw access records, typically in Apache or Nginx format) show every real request made by any bot, with its user-agent, the exact URL, the response code and the timestamp. It's the only source that lets you answer with certainty questions like "how much of my crawl budget is going to URLs I don't care about?"
The basic process of a log analysis: filter lines whose user-agent matches Googlebot (also verifying by reverse IP, since the user-agent can easily be spoofed), group by path or URL pattern, and cross-reference the request volume against the real value of each group of pages. It's common to discover that a huge share of Googlebot's visits (sometimes more than half on ecommerce sites with poorly managed filters) concentrates on parameter combinations that should never have been crawled, while new product pages take days to get their first visit.
The mistakes that waste the most crawl budget
In order of how often they show up in real audits: uncontrolled faceted navigation (color, size or price filters on ecommerce sites that generate a new URL for every combination, multiplying crawlable pages without multiplying unique content); redirect chains, where every extra hop consumes a crawl request without adding anything new; soft 404s, pages that return a 200 status but show "not found" content, which Google keeps re-crawling trying to make sense of; and session or tracking parameters (?sessionid=, ?utm_source=) that generate technically distinct URLs for identical content if they aren't correctly canonicalized or excluded.
How to prioritize crawling toward what matters
The combination that works best in practice isn't a single tool but three layers working together: robots.txt to block from the root paths that should never be crawled (internal search results pages, carts, user areas), canonical tags and parameter handling in Search Console for variants that should occasionally be crawled but consolidated into a single indexable URL, and an internal linking architecture that directs maximum weight toward the highest-value pages, because Google prioritizes crawling URLs that receive more internal links from pages that are already indexed and carry authority.
Frequently asked questions
How long does it take Google to index a new page?
There's no fixed timeframe: it can be a matter of hours on high-authority domains with frequent crawling, or weeks on new sites with little external linking. Manually submitting the URL through Search Console and linking to it from already-indexed pages speeds up the process, but doesn't guarantee a specific timeframe.
Does blocking a URL in robots.txt remove it from Google's index?
Not necessarily. If that URL already had external links, it can remain indexed (with no description, just the URL) because Google indexes it from external signals without needing to crawl it. To reliably remove a URL from the index you need to allow crawling and use noindex, or request temporary removal through Search Console while the noindex takes effect.
Does crawl budget affect small sites?
For the vast majority of sites under a thousand pages, not noticeably: Google has plenty of margin to crawl everything frequently. It becomes a real problem past several thousand URLs, especially if a large share are low value (filters, parameters, poorly managed pagination).
Is it worth submitting the sitemap manually if it's already declared in robots.txt?
Yes: declaring the sitemap in robots.txt makes it discoverable to any crawler that respects the protocol, but also submitting it manually in Search Console lets you directly see how many of the submitted URLs got indexed and catch problems sooner.
What's the difference between "noindex" and removing a page from the sitemap?
They're different, non-equivalent signals. Removing a URL from the sitemap only says "I'm not specifically highlighting this one," but Google can still crawl and index it through other paths (internal or external links). noindex is the only directive that explicitly tells Google not to store it in the index, regardless of how it was discovered.