Two technical questions come up in almost every AI visibility conversation. Should we block or allow AI crawlers in robots.txt? And should we publish an llms.txt file?
The first question has clear, documented answers. The second has a more modest answer than much of the online discussion suggests. This article covers both, using the providers’ own documentation.
Training crawlers vs search crawlers
The single most important distinction: most AI providers now separate crawling for model training from crawling or fetching for search and answers. Blocking one does not automatically block the other.
That gives you a real choice. You can decline to have your content used for training while still allowing your pages to be found and cited in AI search features.
The main user agents and what they do
Here is a summary based on each provider’s documentation. Check the linked pages before you change anything, since providers update them.
| User agent | Provider | Documented purpose |
|---|---|---|
| GPTBot | OpenAI | Crawling to make generative AI foundation models more useful and safe; disallowing it signals content should not be used for training |
| OAI-SearchBot | OpenAI | Surfacing websites in ChatGPT’s search features |
| ChatGPT-User | OpenAI | Visits triggered by user actions in ChatGPT or Custom GPTs; not used to decide what appears in search |
| ClaudeBot | Anthropic | Collecting content that may contribute to model training |
| Claude-SearchBot | Anthropic | Improving the quality of search results |
| Claude-User | Anthropic | Fetching pages when a Claude user asks a question |
| PerplexityBot | Perplexity | Surfacing and linking websites in Perplexity search results; not used for training foundation models |
| Perplexity-User | Perplexity | Fetching pages to answer a user’s question |
| Google-Extended | Control token for use of content in Gemini training and grounding |
Sources: OpenAI crawlers overview, Anthropic crawler documentation, Perplexity crawlers, Google common crawlers.
Details worth knowing
OpenAI. OpenAI states that each setting is independent of the others, so you can allow OAI-SearchBot while disallowing GPTBot. It also says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that robots.txt rules may not apply to ChatGPT-User because those visits are initiated by users.
Anthropic. Anthropic says its bots honor industry-standard robots.txt directives, and that the opt-out has to be set on each subdomain you want covered. It notes that blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude’s search and user-directed answers.
Perplexity. Perplexity recommends allowing PerplexityBot in robots.txt and states that Perplexity-User generally ignores robots.txt rules, because the fetch is requested by a user.
Google. Google-Extended is a product token, not a separate crawler: crawling happens with existing Google user agents. Google states that Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal. For AI Overviews and AI Mode, which are part of Search, Google points site owners to the standard Search controls such as nosnippet, data-nosnippet, max-snippet and noindex.
A sensible default robots.txt policy
There is no universal right answer. It depends on your business, your content and your legal position. But a common, coherent policy for brands that want AI visibility looks like this: allow search and user-fetch agents, and make an explicit, separate decision about training agents.
# Search and answer features: allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Training: an explicit business decision (example: opted out)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This is an example, not a recommendation for every site. If you want your content to inform future models, you would allow the training agents instead.
Two cautions. First, robots.txt is a request, not a lock: the standard itself, RFC 9309, says its rules “are not a form of access authorization”. Second, a broad rule such as User-agent: * with Disallow: /, sometimes left over from a staging site, blocks every compliant crawler, including the search ones.
Common robots.txt mistakes
- Blocking search agents while meaning to block training. The most frequent and most costly error for visibility.
- Forgetting subdomains. Each host has its own robots.txt.
- Copying a list from a blog post without checking. User agent names change. Use the provider documentation.
- Blocking at the firewall or CDN level without knowing it. Bot protection settings can block AI agents even when robots.txt allows them. Check your server logs.
- Assuming robots.txt changes are instant. OpenAI, for example, says search changes can take about 24 hours to take effect.
llms.txt: what it is
llms.txt is a proposal published at llmstxt.org by Jeremy Howard in September 2024. It suggests placing a Markdown file at /llms.txt that gives language models a concise, curated entry point to a site.
The format is simple:
- an H1 with the name of the site or project (the only required section),
- a short blockquote summary,
- optional paragraphs of detail,
- H2 sections containing lists of links, each with an optional note,
- by convention, a section titled “Optional” for secondary links that can be skipped.
llms.txt: where it really stands
Be precise about what is and is not known.
- It is a proposal, not a standard in the sense that robots.txt is (robots.txt is an IETF RFC).
- The proposal itself does not list major assistants as confirmed consumers of the file for answering general questions. Its author describes it as mainly useful at inference time, for example when an agent is pointed at a site.
- Google is explicit that you do not need new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode.
- We have not found primary documentation from OpenAI, Anthropic, Google or Perplexity stating that their consumer assistants use llms.txt to choose which brands to mention or cite.
So what should you do? If you have documentation, a developer audience, or a complex site, an llms.txt file is cheap to produce and may help agents and tools that read it. For most brands, it is a low-priority item that comes after crawler access, clear content and presence on the sources assistants cite.
Checklist
- robots.txt reviewed against each provider’s current documentation
- Search and user-fetch agents allowed if you want AI visibility
- Training agents handled as a separate, deliberate decision
- No leftover global
Disallow: / - Subdomains checked
- Firewall and CDN bot settings checked against server logs
- llms.txt considered as optional, after the basics
How SearchFirefly helps
SearchFirefly’s technical analysis groups issues by theme and severity, with the pages affected, and its tracked questions show whether ChatGPT, Gemini, Perplexity, Google AI Overviews and Google AI Mode actually mention your brand and cite your pages. Seeing both together helps you tell an access problem apart from a content or reputation problem, and each prioritized recommendation comes with the exact fix and a check after release.