The crawling and indexing guide mentions server log analysis as the only source that shows real crawling activity, with no sampling or prior aggregation. This guide stays exactly there: how that analysis is actually done, what data format you're working with, how to verify a request genuinely comes from Googlebot and not from something spoofing its user-agent, and which concrete patterns are worth looking for in a file that can hold millions of lines.
Log formats: what each line's fields actually mean
The most common format is Apache's combined log format, which adds the referer and user-agent to the basic fields:
# Apache combined format (LogFormat "combined")
66.249.66.1 - - [15/Jan/2032:10:23:41 +0100] "GET /products/nordic-chair HTTP/1.1" 200 15234 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"Each field has a fixed meaning: the source IP, the remote identity and authenticated user (almost always empty, shown as a dash), the timestamp, the full request line (method, path and protocol version), the HTTP status code of the response, the response size in bytes, the referer, and finally the user-agent. Nginx doesn't enforce a fixed format (it's declared with the log_format directive in the configuration) but by default usually mirrors a structure very similar to Apache's, and it's common to add an extra field with the server's own response time ($request_time), useful for cross-referencing log analysis directly with the TTFB diagnosis covered in the server speed guide.
Verifying a bot is genuinely Googlebot: reverse DNS and forward confirmation
Relying solely on the user-agent string is a common mistake: any script can declare "Googlebot" in its User-Agent header without being Google at all, and it's a common technique among aggressive scrapers trying to dodge blocks meant for unwanted bots. Reliable verification requires two chained steps, known as forward-confirmed reverse DNS.
The first step is a reverse lookup: from the IP that made the request, you get the hostname assigned to it. The second, essential step is a forward lookup of that same hostname: if the resulting hostname ends in googlebot.com or google.com, you resolve it forward again and check it returns exactly the same original IP. If it doesn't match, the request isn't from Googlebot no matter what its user-agent claims.
# 1. Reverse lookup: which hostname corresponds to this IP
host 66.249.66.1
# 66.249.66.1.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
# 2. Forward confirmation: does that hostname resolve back to the same IP?
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1For large traffic volumes, doing this double lookup line by line isn't practical; Google also publishes documented, downloadable IP ranges in JSON format for each of its crawlers, which serve as a fast first filter before reserving the full DNS verification for doubtful cases or higher-volume ones.
Practical parsing approaches: from the command line to dedicated tools
For one-off explorations, commands like grep and awk are enough and require installing nothing extra on the server itself:
# How many Googlebot requests fall into each status code
grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn
# The 20 paths Googlebot visits most
grep "Googlebot" access.log | awk -F'"' '{print $2}' | awk '{print $2}' | sort | uniq -c | sort -rn | head -20For volumes beyond what's manageable from the command line, with several months of history or millions of lines a day, there are tools built specifically for this analysis: Screaming Frog Log File Analyser automatically cross-references the URLs seen in the logs with a full site crawl, directly flagging orphan pages and wasted crawl budget; and for infrastructure with sustained volume, a centralized aggregation stack (Elasticsearch, Logstash and Kibana, or lighter alternatives like GoAccess for real-time dashboards) enables queries and filters that wouldn't be practical running manual commands every time.
Which concrete patterns to look for
Crawl frequency by URL pattern is the first thing to check: grouping requests by template type (product pages versus filter pages versus blog posts) immediately reveals where crawl budget is actually concentrated, something no aggregated estimate can show with the same precision.
The status code distribution of Googlebot's crawling is the second: a high proportion of 404s or 410s means Google keeps visiting dead URLs, almost always because broken internal links still point there or because the sitemap still lists them; a high proportion of 5xx errors signals server problems Google detects directly and that automatically lower the crawl rate limit, as explained in the crawling and indexing guide.
Orphan pages Google crawls with no internal links pointing to them are the third pattern, and one of the most revealing: cross-referencing the URLs seen in the logs against a full site crawl (spider mode, with a tool like Screaming Frog) surfaces URLs Google visits through external signals, old external links, sitemaps inherited from a previous version of the site, that the current internal architecture no longer links to from anywhere, a clear opportunity to decide whether those pages should be recovered with real internal linking or removed outright.
Wasted crawl budget is the fourth: URLs with session parameters, tracking parameters, or listing sort order that receive a disproportionate volume of requests relative to their real value. Here the log isn't a suspicion but direct evidence, with real numbers of how many requests are going to those combinations versus the pages that actually matter.
And the fifth is the delay between publication and first visit: comparing a new URL's publication date against its first appearance in Googlebot's logs gives a direct, verifiable metric of how long new content actually takes to enter the crawl flow, without relying on estimates.
Cross-referencing logs with Search Console and the sitemap
Search Console's crawl stats report gives a useful aggregated summary and a sample, good as an initial warning sign, but the logs are the only source with one hundred percent of the real requests and the exact paths, not grouped into generic categories the way Search Console does. The most productive combination in practice is using Search Console to spot an anomaly at the aggregate level (a sudden drop in the number of crawled pages, for example) and turning to the logs to pinpoint exactly which paths are causing it.
Frequently asked questions
How long should logs be kept for the analysis to be useful?
At least 30 days for a representative sample of normal crawling; ideally several months, to be able to spot seasonal patterns or compare behavior before and after a migration or a structural change to the site.
Do I need root access to the server to analyze logs?
Not always. Many hosting providers and control panels (cPanel, Plesk) expose raw access logs for direct download without needing root access, though the exact format and available retention can vary by provider configuration.
Is it worth analyzing logs if the site is small?
It adds less value because crawl budget is rarely a real bottleneck on sites with few pages, as explained in the crawling and indexing guide, but it's still useful for catching bots spoofing Googlebot's user-agent or one-off server failures that other tools don't surface with the same detail.
How do I tell Googlebot apart from other Google bots, like AdsBot or Google-InspectionTool?
Each has its own user-agent string and its own separately documented IP ranges and verification hostnames. It's best to treat them as independent sources in the analysis instead of lumping them all under the generic "Google" label, since each serves a different crawling purpose.
Do logs also show real user visits or only bots?
They show both by default, mixed in the same file. You need to explicitly filter by user-agent and, as a complement, by known bot IP ranges, to reliably separate crawl traffic from human traffic before drawing any conclusions.