AI crawlers and robots.txt: GPTBot, ClaudeBot, Google-Extended and llms.txt, what to allow and what to expect

For years, the only robot you paid attention to in your logs was Googlebot. Today you also see names such as GPTBot, ClaudeBot or PerplexityBot, and each one does something different: some collect your content to train models, others build the index an assistant uses to answer questions, and others visit a page only because someone has just asked them to. Treating them all the same, whether by throwing the door wide open or by closing it on everyone, is the most common mistake.

Before we start, a warning we will repeat: crawler names, their policies and the way they behave change often. What follows is a map to get your bearings as of the time of writing, not a definitive reference. Before you touch your configuration, confirm every detail in each provider's official documentation.

Three types of visit and which user-agent belongs to each

To decide well, you first need to separate why each visitor comes. Almost all of them fit one of these three categories:

• Training: they collect pages to feed or improve language models. Your content may end up influencing what a model "knows", usually without a link back to you.
• AI search: they crawl and index so that, when someone asks an assistant with search, it can find sources and cite them, normally with a link. This is the closest thing to what Googlebot does.
• User-requested agents: the assistant visits a specific URL because a person asked it to summarise a page or pasted it into the chat. They do not crawl the web at scale: they act on someone's behalf at that moment.

Each provider usually identifies each function with a different user-agent, which is the name the crawler uses to introduce itself to your server. These are the main ones as of the time of writing:

There are also three pieces that deserve a separate explanation, because they are widely misunderstood:

• Google-Extended: it is not a crawler. It is a robots.txt token that tells Google whether content it has already crawled with its usual agents may be used for Gemini. It does not affect whether you appear in Google Search or how you rank, so blocking it does not remove you from the results. Do not assume either that it controls everything Google does with AI inside the search engine: that is governed by the usual Search controls, and the right place to confirm it is Google's documentation.
• Applebot-Extended: same approach at Apple. Applebot is the one that crawls; the Extended token indicates whether that content may be used to train the company's generative models.
• CCBot: the Common Crawl crawler. Common Crawl is an open archive of the web that many projects use as raw material for training models. Blocking it reduces the chance of your pages entering future dumps, but it does not remove what was already published.

The list is not exhaustive and may have changed. In addition, some providers state that agents acting at the user's request may not follow robots.txt, because they do not crawl on their own. Read what each one says.

robots.txt: what it is, what it is not and rules to copy

The robots.txt file is a text file that lives at the root of your domain (for example, https://www.example.com/robots.txt) and contains groups of rules. Each group starts with a User-agent line and continues with Allow or Disallow instructions. It is a convention, now written up as an internet standard, that well-behaved crawlers follow voluntarily. That nuance changes everything. Three ideas to avoid misunderstandings:

• It is not access control or a security measure. A crawler that does not want to obey simply does not obey. On top of that, the file is public: listing /private/ there is announcing that it exists. Anything confidential goes behind a password-protected login, not behind a Disallow.
• Blocking crawling is not the same as not indexing. A Disallow stops the robot from reading the page, but the URL can still show up in a search engine if other sites link to it. To keep a page out of the index you need the noindex directive, and for the robot to see it, it has to be allowed to crawl it.
• It is not retroactive. Changing the file affects future visits from whoever respects it. It does not delete what was already collected or force anyone to forget it.

Another technical detail: each robot looks for the group whose User-agent best matches its name and, if there is none, it uses the * group. That is why, as soon as you create a specific group for GPTBot, that robot stops reading the rules in the general group. Here are three real examples you can adapt. The first allows access to everything and declares the sitemap:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The second blocks only the training uses and leaves traditional search, AI search crawlers and on-request agents open:

# Training: blocked
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Everyone else, search included: allowed
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The third closes just one directory to several AI crawlers and leaves the rest of the site accessible. Notice that several consecutive User-agent lines share the same rules:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /client-area/
Disallow: /paid-reports/

A practical warning: a Disallow: / under User-agent: * inherited from a test environment is one of the most expensive mistakes there is, because it shuts the door on everyone at once. Check yours.

The business dilemma: allow, block or fine-tune

The technical part is simple. The hard part is deciding what you want, because each crawler category implies a different bet:

• AI search: if you want ChatGPT, Gemini or Perplexity to cite or recommend you, blocking their search crawling works against your goal. If they cannot read you, they are unlikely to mention you with up-to-date information. And the other way round: allowing it does not guarantee they will.
• Training: this is a different decision and a legitimate one either way. It weighs more if you live off your intellectual property, if you have paid content, courses or databases, or if you are a publisher that depends on visits or licences. For a service business with a commercial website, the cost of allowing it is usually lower, although there is no proven, simple relationship between allowing training and being mentioned more. If money or rights are at stake, talk to someone who can advise you on the legal side.
• On-request agents: blocking them means the person who pastes your URL into an assistant will get an error or a poor answer about your company. On a website that is looking for customers, that is rarely what you want.

As a starting point, a typical commercial website usually allows search and on-request visits, and makes a conscious decision about training. A publisher or a platform with paid content will probably want to close off training and review the rest case by case. What matters is that it is your decision, written down, and not the result of pasting a file you saw on a forum.

How to see who is really crawling you

The robots.txt file records what you ask for; the logs record what happens. They are two different things, and it is worth looking at both.

• Server logs: the Apache or Nginx access file, or the logs and statistics section of your hosting, shows every request with its user-agent. If you have terminal access, a simple search gives you a first idea of the volume.
• CDN or firewall analytics: many dashboards group bot traffic by category and let you filter by name.
• Web analytics: tools such as GA4 normally are no use for this, because these robots do not run your measurement code.

grep -iE 'gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|perplexity|ccbot' access.log | less

Look at three things: how often they come, which URLs they request and what response code they get. A 200 means they got in; a 403 or a 429 means someone (you, your hosting or your firewall) closed the door on them, even if your robots.txt says otherwise.

Beware of impostors. The user-agent is just a piece of text and anyone can type it, so some bots pretend to be GPTBot or Googlebot to slip in. If you need to know whether a visit is genuine, compare the IP with the information each provider publishes (some offer lists of IP ranges, others explain how to verify it with DNS lookups). Not all of them document it the same way, so check what each one offers.

Firewalls and CDNs. Many CDN services, web application firewalls, security plugins and hosting plans include bot management, and some come with switches to block AI crawlers that may be turned on without you remembering. We cannot take for granted how yours behaves, so check it: go into the dashboard, look for sections such as bots, AI crawlers or rate limiting, and compare with what you see in the logs. A quick test is to request your homepage pretending to be one of them:

curl -I -A 'GPTBot' https://www.example.com/

Interpret it with care: if the system also validates the source IP, your test is not the same as a real visit, but an obvious block is a valuable clue.

llms.txt: what it is and the uncomfortable truth

llms.txt is a community proposal, published in 2024, to offer language models a short, clean version of your site. It is a markdown file located at the root (/llms.txt), with the site's name, a brief summary and lists of links to key pages, each with a descriptive sentence. The idea is that a model should not have to untangle menus, banners and scripts to understand what your website is about. An example with a fictional business:

# Example Renovations

> Full-service renovation company in Madrid. Free, no-obligation quotes.

## Services

- [Bathroom renovations](https://www.example.com/bathroom-renovations/): what is included, timescales and how to request a quote
- [Kitchen renovations](https://www.example.com/kitchen-renovations/): project types and materials

## Company

- [Contact](https://www.example.com/contact/): address, phone and opening hours

Now, the uncomfortable truth: as of the time of writing there is no confirmation that the major assistants use it systematically. It is not an official standard, it does not replace robots.txt (it neither grants nor denies permissions) or your sitemap, and you should neither sell it nor believe it as a ranking lever. Check each provider's documentation to see whether it has announced any support, because this may change.

So, is it worth it? It is optional and low cost: you can write it in an hour. Its real risk is that it goes out of date, so generate it from the same sources as your sitemap or do not publish it. Do it if you have spare time, never instead of what comes next.

What actually moves the needle for GEO

If llms.txt is not the lever, what is? The answer is a little disappointing because it looks a lot like good old SEO, with a few nuances. None of these actions guarantees that an assistant will cite you, but they do increase the chances that it understands you and finds you:

• Crawlable content in HTML. What matters has to be in the HTML your server delivers, not just behind JavaScript, tabs that load on click or forms. Not every crawler runs JavaScript. Check it with "view source" or with curl: if your offer does not show up, they probably cannot see it either.
• Clarity and structure. Descriptive headings, paragraphs that answer a specific question, lists, simple tables and direct answers at the start of each section. What is easy for a person to read is usually easy for a machine too.
• Structured data. schema.org markup in JSON-LD (Organization, LocalBusiness, Article, Product, depending on the case) that matches what is visible on the page. It helps disambiguate who you are and what you offer.
• External mentions. Industry directories, specialist press, reviews and your own profiles. Assistants do not rely only on what you say about yourself.
• Consistent information. The same details everywhere, contact details included: the NAP (name, address and phone number) must match on your website, your Google Business Profile listing and the directories. A different opening time on each site erodes trust.
• Fast, stable pages. If the server is slow or returns errors, crawlers lose interest. An up-to-date sitemap, sensible internal links and correct canonicals remain the foundation.

Five-step action plan and checklist

1. Decide your stance for each category. AI search, training and on-request agents: what you allow, what you block and why. Write it down.
2. Review and write your robots.txt. Check that you are not carrying over a blanket block and apply the rules from the previous step with the correct syntax.
3. Look at your logs and your firewall. Find out who really comes, which responses they get and whom you are blocking by accident.
4. Prepare the content. Crawlable HTML, clear structure, structured data and a consistent NAP everywhere.
5. Optional: llms.txt and periodic review. Publish it if it is worth it to you and put a quarterly review of the providers' documentation in your calendar, because names and policies change.

And here is the short checklist to go through:

• I have written down what I allow for search, training and on-request visits.
• My robots.txt loads at the root, is well written and does not block everything by mistake.
• I have verified the user-agent names in each provider's official documentation.
• I know which crawlers visit me according to my logs or my CDN and what response code they get.
• I have checked that my firewall or CDN does not block the ones I want to allow.
• The key content is in the HTML and the contact information is consistent everywhere.

Frequently asked questions