Is your website blocking ChatGPT without you knowing? AI crawlers and robots.txt

The short answer. To appear in AI assistants’ answers, your website has to let their search crawlers in: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity), Googlebot (Google, including AI Overviews and AI Mode) and Bingbot (the Bing index, which Copilot queries). Training bots, such as GPTBot or the Google-Extended token, are a separate decision: according to OpenAI and Google, blocking them doesn’t remove you from ChatGPT search or Google Search.

A website can do well on Google and, without anyone realising, shut the door on AI. This guide helps you check. We verified every fact against each company’s documentation on 25 October 2026.

Four facts to start with:

• ~24 hours is how long ChatGPT search takes to apply a change to your robots.txt, according to OpenAI.
• 5xx = everything off-limits: if your robots.txt returns a server error, the standard requires the bot to assume it can’t access any page.
• No effect: blocking Google-Extended doesn’t change your inclusion in Google Search or your ranking, according to Google.
• 15 Sep 2026: since that day, Cloudflare’s “Block” setting against AI training also stops Googlebot, Bingbot and Applebot.

What kinds of AI bots visit your website?

Each AI company runs several crawlers, each with a different purpose.

Search bots decide whether you show up. Training bots collect content for future models. User-requested bots read a page because someone asked for it in the chat.

What happens if you block each bot?

A summary of each company’s official documentation:

What happens if you block each bot?

Bot	Company	What it’s for	If you block it
OAI-SearchBot	OpenAI	ChatGPT search	You don’t appear in ChatGPT search answers
GPTBot	OpenAI	Training	Your content isn’t used to train its models
ChatGPT-User	OpenAI	Visits a user asks for	OpenAI warns that robots.txt “may not apply”
Claude-SearchBot	Anthropic	Claude search	Your content isn’t indexed for its search
ClaudeBot	Anthropic	Training	Your future content is excluded from training
Claude-User	Anthropic	Visits a user asks for	Claude can’t read your page when someone asks it to
PerplexityBot	Perplexity	Search; doesn’t train models	Perplexity recommends allowing it
Perplexity-User	Perplexity	Visits a user asks for	Generally ignores robots.txt
Googlebot	Google	Google Search, including AI Overviews and AI Mode	It doesn’t crawl you for Search or its AI features
Google-Extended	Google	Token for Gemini training and grounding	Doesn’t affect Google Search or ranking
Bingbot	Microsoft	Bing index, which Copilot queries	Copilot loses that route to your pages (our inference)
Applebot / Applebot-Extended	Apple	Spotlight, Siri and Safari / training opt-out	Applebot removes you from those features; -Extended, only from training
Meta-WebIndexer	Meta	Meta AI search	Meta says allowing it helps it cite you
Meta-ExternalAgent	Meta	Training and product improvement	You leave training and possibly some indexing

Two useful details. OpenAI explains that if you allow both OAI-SearchBot and GPTBot, it may use a single crawl for both purposes. And Anthropic asks you to repeat the setup on each subdomain, because each one has its own robots.txt.

How does a bot read your robots.txt?

The rules are set out in RFC 9309 (IETF, September 2022). Four are worth knowing:

1. It isn’t a lock. The standard puts it this way: “These rules are not a form of access authorization”. It’s a request.
2. Each bot looks for its own group by name. Only if there’s no group with its name does it follow the User-agent: * group. If you create a group for GPTBot, the * rules no longer apply to GPTBot.
3. The most specific rule wins. If the same group has Disallow: / and Allow: /blog/, the blog stays open.
4. A server error closes everything. With a 5xx, the bot must assume everything is off-limits; with a 404, it can crawl everything.

A warning from Google: robots.txt doesn’t remove a page from search. A blocked URL can still appear if other pages link to it. To keep it out of the index, use noindex, and don’t block it at the same time, because the bot has to get in to read that tag.

Which robots.txt suits your business?

Replace yoursite.com with your domain. The first one works for most businesses.

1. Maximum visibility. Every bot can read everything.

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

2. Search yes, training no. You appear in AI search and ask that your content not be used for training.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

We’ve left Meta-ExternalAgent out on purpose: Meta says it also uses it to index content, so blocking it may go beyond training.

3. The mistake to avoid. These two lines block every bot, including OAI-SearchBot and Googlebot. They can slip into production from a staging site without anyone noticing.

User-agent: *
Disallow: /

Should you block GPTBot?

It’s a decision about training, not about your visibility in ChatGPT search, which depends on OAI-SearchBot. If your priority is being recommended, we’d leave it open.

If your content is your product (a publication, a course, a database), blocking it makes sense. What nobody knows is whether opting out of training changes what future models “know” about your business: there’s no public data that measures it.

What else can block a bot, apart from robots.txt?

A perfect robots.txt is no use if another layer stops the bot first.

• CDN and firewall. Cloudflare changed its settings on 15 September 2026. Since then, choosing “Block” in the training category also stops Googlebot, Bingbot and Applebot, “search included”, because they are mixed-use crawlers. To stop training only, Cloudflare offers the “Disallow AI Training” option.
• Fake bots. Anyone can call themselves “GPTBot”. OpenAI and Perplexity publish their crawlers’ IP addresses so your provider can check they’re genuine.
• JavaScript. In a Vercel and MERJ analysis from December 2024, none of the major AI crawlers executed JavaScript, except Gemini (using Googlebot’s infrastructure) and Applebot. That data is old, so the safe move is to have prices, services and contact details in the HTML your server delivers.

How do you check whether your website blocks AI?

1. Open yoursite.com/robots.txt and check that it loads without errors and says what you think it says.
2. Look for groups with Disallow: / and see which bots they affect.
3. Review the AI bot settings in your CDN or firewall.
4. If you can see your server logs, search for the bots’ names and their response codes. A repeated 403 for OAI-SearchBot points to the firewall or the CDN.

Frequently asked questions

What now?

If you see blocks you don’t understand, or you want to know how AI assistants describe you, we can help. The free diagnostic includes a technical review of AI bot access, and in services we explain how we fix what we find and keep it up to date every month. Nobody can guarantee a position in an AI answer; what you can control is that no bot is left at the door.