The /robots.txt file looks so simple that it rarely gets reviewed with the same care as other technical parts of a site, and that's precisely why some of the costliest mistakes in an audit show up there: a badly written rule can accidentally block entire sections of a site, or let through content that should have stayed out of crawling. This guide goes beyond the basic User-agent, Disallow and Allow syntax already covered in the crawling and indexing guide, and gets into where the real failures happen: how crawlers actually interpret wildcards, which rule wins when several could match the same URL, and how to verify for real, not from memory, that the file does what you think it does.
Wildcards: how a crawler actually interprets * and $
The robots exclusion protocol, formalized as RFC 9309 in 2022 after decades of running as a de facto convention, supports two wildcards in Disallow and Allow paths. The asterisk * represents any sequence of characters, including an empty one, and can be used any number of times within the same rule. The $ symbol anchors the end of the path, meaning the match is only valid if the URL ends exactly there and doesn't continue with more characters.
The difference between blocking with and without that anchor looks cosmetic but isn't: without $, every Disallow rule carries an implicit wildcard at the end, so it matches any continuation of that string.
Disallow: /report.pdf
# also blocks /report.pdf?v=2 and /report.pdflegacy.html
Disallow: /report.pdf$
# only blocks the URL that ends exactly thereThis nuance is behind accidental blocks of URLs that were never meant to be excluded, simply because they shared a prefix with the path that actually needed blocking.
Specificity: which rule wins when several could apply
When a URL matches more than one rule within the same User-agent block, most crawlers that follow RFC 9309, Googlebot among them, don't apply the first rule that matches in reading order, but the most specific rule, understood as the one with the longest matching path in number of characters. This is a fundamental difference from what many people assume by intuition, that "the first rule wins" as in a classic firewall, and it's the cause of plenty of robots.txt files that don't do what their author expected.
User-agent: *
Disallow: /private/
Allow: /private/public/
# /private/public/item.html CAN be crawled:
# "/private/public/" (16 characters) is more specific
# than "/private/" (9 characters), regardless of orderWhen two rules match with exactly the same length, the specification breaks the tie in favor of Allow over Disallow, though not every less rigorous crawler honors that tiebreak identically, so in practice it's best to avoid relying on identical lengths and to be explicit with wildcards whenever there's real ambiguity.
Crawl-delay: why Google ignores it and other bots don't
Crawl-delay is a directive that specifies, in seconds, the minimum time a bot must wait between consecutive requests to the same site. It started as a non-standard extension, never part of the original protocol or the final RFC 9309, originally pushed by search engines like Yahoo and later adopted by Bing and Yandex, which do respect it.
Google is the notable exception: Googlebot completely ignores the Crawl-delay directive in robots.txt, regardless of the value set. The correct way, and the only one currently supported, to ask Google to slow down its crawl rate is the crawl rate limit setting in Search Console, a control that Google only acts on if it detects the crawling is causing real performance problems, not as an arbitrary preference from the site owner. A real and fairly widespread mistake is assuming a Crawl-delay: 10 in the User-agent: * block also throttles Googlebot: it doesn't, and the site owner is surprised when Google's crawling stays just as intense despite the directive.
User-agent: bingbot
Crawl-delay: 10
User-agent: Googlebot
# Crawl-delay here has no effect at all; it's ignoredMultiple User-agent blocks in the same file
A single robots.txt can declare different rules for different bots, and this is where the real behavior of crawlers causes the most confusion: each bot applies only the rules from the block that most specifically matches its user-agent name, never the sum of several blocks. If there's a block for Googlebot-Image and a generic one for *, Google's image crawler obeys only its own block, completely ignoring whatever the generic block says, even if that generic block contains rules that would seem to also apply to it.
User-agent: *
Disallow: /private/
User-agent: Googlebot-Image
Disallow: /internal-photos/
# Googlebot-Image can crawl /private/ without any issue:
# it inherits nothing from the generic block, only obeys its ownThis has an important practical consequence: if you add a specific block for a particular bot, for example to give different instructions to an AI training crawler, you have to explicitly repeat in that block any rule from the generic block that should also apply to it, because there's no inheritance between blocks.
The real mistakes that show up most in audits
The most frequent is accidentally blocking CSS and JavaScript: an overly broad Disallow: /wp-includes/ or /assets/ stops Googlebot from downloading resources its own rendering needs, which stops it from seeing the page the way a user does, as explained in the JavaScript and technical SEO guide. The second is a Disallow: / inherited from a staging environment that nobody removes when the site goes live, blocking the entire domain in one shot. The third is case sensitivity: robots.txt paths are case-sensitive, so a Disallow: /Discounts/ does not block /discounts/ even though to a person they look like the same folder. The fourth is a trailing-slash mismatch: Disallow: /folder (no slash) also blocks /folder-old or /folder2 through prefix matching, while Disallow: /folder/ (with a slash) only blocks what literally hangs off that folder. And a fifth, less known but real on sites with automatically generated rules: Google only processes the first 500 KiB of a robots.txt file; any rule beyond that limit is silently ignored with no visible warning.
How to actually verify robots.txt, not from memory
Google retired the classic robots.txt Tester from Search Console and replaced it with a report under Settings that shows the detected file, when it was last crawled, and whether there are syntax or availability errors, but it no longer lets you simulate a specific URL against the rules the way the old tool did. For that manual simulation, Google keeps open source (published code) the same robots.txt parsing library Googlebot uses internally, which can be installed and run locally against the real file and a list of test URLs: the most reliable way to reproduce the exact behavior without relying on a human interpretation of the rules.
A quick, if less rigorous, manual check is simply requesting the file with curl and looking at the response header before the content:
curl -I https://www.yourdomain.com/robots.txtA robots.txt that returns a server error (5xx) makes Google temporarily treat the entire site as fully blocked from crawling until it responds normally again, while a 404 is interpreted as "no restrictions," with implicit permission to crawl everything. That difference between a server error and a simply absent file is the one that surprises people most when they don't already know it.
Frequently asked questions
Is a Disallow in robots.txt a security measure?
No. It's a good-faith request that only crawlers choosing to respect it will honor: it doesn't stop a malicious bot from accessing that path anyway, and publishing a sensitive path in robots.txt to block it is counterproductive, because the file is public and announces that path's existence to anyone who reads it.
Can I use wildcards in a User-agent name?
No. User-agent names must match literally (or by exact prefix) the bot's actual identifier; wildcards are only allowed inside Disallow and Allow paths, never on the User-agent line.
Can I block an AI training bot without blocking Googlebot?
Yes, by declaring a specific User-agent block for that bot, for example GPTBot or CCBot, each with its own documented name. Remember there's no inheritance between blocks, so that specific block needs its own full set of rules, not just the exception you want to add.
Does robots.txt have to live at the domain root?
Yes, it's only honored if it lives exactly at https://yourdomain.com/robots.txt. One placed in a subfolder has no effect at all, and every subdomain needs its own independent file at its own root.