How to prevent AI from training on your content, but still show in AI search
All of my client conversations about AI crawlers start the same way. Someone asks if we can just block them, and I have to ask “which them?” Every major AI company runs several crawlers, all built for different purposes. Usually one for training, one for search, and sometimes a third for live user requests. If you treat them as a single "AI bot," you're going to see disappointing results.
That mix-up also costs money. If you get it backward, you might spend a month wondering why AI search traffic dried up when the fix was one line in the wrong place. I find that I explain this distinction on call after call, so I finally wrote it down. Most of the robots.txt files I audit are outdated or running default settings that allow every bot access to your content.
The most important question: can you keep AI from Training on your content, but still get cited in AI search?
This is the main question people have. How can we either protect our server requests, limit training on proprietary data, or exclude AI crawlers without blowing up our GEO strategy? And in some cases, this is in fact possible. Here is a full breakdown:
Platform | Can you separate training from AI-search visibility? | Practical approach | Where to check |
|---|
OpenAI / ChatGPT | Yes | Disallow GPTBot to opt out of training, then allow OAI-SearchBot separately so you don't lose ChatGPT Search visibility by accident. | OpenAI crawler docs |
Anthropic / Claude | Yes | Disallow ClaudeBot to opt out of training, then allow Claude-SearchBot separately so you stay visible in Claude's search results. | Anthropic crawler info |
Perplexity | Yes, largely by default | Perplexity says its search crawler exists to surface and link sites in results, separate from training foundation models. Allow it, and its published IP ranges, to stay visible in Perplexity search. | Perplexity crawler docs |
Amazon | Yes | Disallow Amazonbot to opt out of training. Amzn-SearchBot is built specifically to support Amazon's own search experiences while staying out of generative AI training, so leave it running if that split is what you want. | Amazonbot documentation |
Google Search / AI Overviews | Yes for training. No dedicated opt-out for AI Overviews specifically | Disallow Google-Extended to limit the Gemini and Vertex AI training and grounding uses Google has named. Keep Googlebot crawlable to stay in Google Search, since AI Overviews run on that same Search infrastructure and Google hasn't published a separate lever for them. | Google-Extended docs |
Some notable exceptions here are that Meta, Micosoft and Bytedance don’t offer a split between training bots and AI coverage.
- Meta has 2 crawlers that - one for genera crawling and one for user-triggered fetches.
- Bing folds everything into the same bingpot that handles all of it’s regular search crawling.
- Bytespider doesn’t make any promises or publish documentation to verify how it is used.
The part that confuses most people
When setting rules in your Robots.txt it will only apply to that exact user agent. This means if you block Claudebot but you haven’t specifically referenced Claude-searchbot, then Claude-searchbot can still crawl, even though Anthropic runs both.
But you don’t need to allow every user you’re fine with on it's own line. If you don’t have a disallow or allow rule for a bot, the default will be that it assumes it's allowed to crawl. I only add explicit allow rules when I want to make the rules obvious to any humans who are reading the file
Here is a great reference table for audits
I made this reference table which is a good source to refer back to when doing any acess level audits or answering questions
Company | User agent / control | Current purpose | robots.txt behavior | What blocking affects | Example rule | Official documentation |
|---|
OpenAI | GPTBot | Collects publicly accessible content that may improve OpenAI's foundation models | Independently controlled through robots.txt | GPTBot's training-related crawl only | User-agent: GPTBot
Disallow: / | OpenAI crawler docs |
OpenAI | OAI-SearchBot | Crawls for ChatGPT Search | OpenAI recommends allowing it (plus its published IP ranges) for discoverability | Automated crawling for ChatGPT Search; allowing it doesn't guarantee citation | User-agent: OAI-SearchBot
Disallow: / | OpenAI crawler docs |
OpenAI | ChatGPT-User | Fetches a page after someone asks ChatGPT or a Custom GPT to retrieve it | User-initiated; OpenAI says robots.txt rules may not apply | User-triggered retrieval only, with no role in ChatGPT Search inclusion | Don't rely on robots.txt as your only control here | OpenAI crawler docs |
Anthropic | ClaudeBot | Collects web content that could contribute to training Anthropic's models | Independent crawler, respects Disallow and Crawl-delay | ClaudeBot's training-related crawl only | User-agent: ClaudeBot
Disallow: / | Anthropic crawler info |
Anthropic | Claude-SearchBot | Indexes content to improve Claude's search responses | Documented separately from ClaudeBot | Search indexing for Claude; blocking can reduce your visibility in Claude's search results | User-agent: Claude-SearchBot
Disallow: / | Anthropic crawler info |
Anthropic | Claude-User | Retrieves pages when someone asks Claude a question that needs web access | Separate, user-directed crawler | User-requested retrieval, which affects visibility in user-directed Claude queries | User-agent: Claude-User
Disallow: / | Anthropic crawler info |
Perplexity | PerplexityBot | Surfaces and links sites in Perplexity search results. Perplexity limits it to search, keeping it separate from foundation-model training | Respects robots.txt; Perplexity actively recommends allowing it and its IP ranges | Perplexity's indexing of your page text for search results | User-agent: PerplexityBot
Disallow: / | Perplexity crawler docs |
Perplexity | Perplexity-User | Fetches a page in response to a user's specific query | Generally ignores robots.txt, since it's triggered by a live user request instead of automated crawling | User-triggered page retrieval; still subject to WAF and other access controls | Treat separately from PerplexityBot | Perplexity crawler docs |
Google | Google-Extended | A robots.txt product token for Gemini Apps and Vertex AI API for Gemini, covering both training future Gemini models and grounding | A control token you add to robots.txt directly, distinct from a standard HTTP user-agent | Google's stated Gemini/Vertex AI training and grounding use of your content | User-agent: Google-Extended
Disallow: / | Google-Extended docs |
Google | Googlebot | Builds Google's Search index | Standard, well-documented robots.txt handling | Google Search crawling, and by extension the Search surfaces (including AI Overviews) built on top of it | Only touch standard Googlebot rules if you intend to manage Search crawling | Google crawler overview |
Microsoft | bingbot | Crawls for Bing Search; Bing's index can feed Copilot experiences | Check Microsoft's current documentation before changing rules | Bing Search crawling, with potential downstream effects on Microsoft's AI surfaces | User-agent: bingbot
Disallow: / | Bing crawler documentation |
Meta | meta-externalagent | Public-web crawling tied to Meta AI and related indexing/training uses | Verify current documentation and traffic identity before you act on it | Meta's general automated AI crawl | User-agent: meta-externalagent
Disallow: / | Meta crawler documentation |
Meta | meta-externalfetcher | Retrieves content for a user-directed Meta AI request | Verify current documentation before using it as a control | User-triggered retrieval through Meta AI surfaces | User-agent: meta-externalfetcher
Disallow: / | Meta crawler documentation |
Amazon | Amazonbot | Crawls public content to improve Amazon's products, including training Amazon's AI models | Respects Disallow, Allow, noindex, noarchive, and rel=nofollow, but skips Crawl-delay | Amazonbot's automated crawl | User-agent: Amazonbot
Disallow: / | Amazonbot documentation |
Amazon | Amzn-SearchBot | Powers search experiences in Amazon services like Alexa. Amazon keeps it separate from generative AI training | Same robots.txt handling as Amazonbot | Amazon search-related crawler access | User-agent: Amzn-SearchBot
Disallow: / | Amazonbot documentation |
Amazon | Amzn-User | Handles real-time requests from Alexa queries, kept separate from AI training the same way Amzn-SearchBot is | May not always follow robots.txt, since it's responding to a live user request | User-requested retrieval through Amazon services | User-agent: Amzn-User
Disallow: / | Amazonbot documentation |
ByteDance | Bytespider | Web crawling for ByteDance products, including TikTok-related systems | No official, first-party documentation page as of this review. Verify purpose and behavior through third-party crawler monitoring before you rely on it | Automated ByteDance crawling. Don't assume it carries AI-search value the way OAI-SearchBot or PerplexityBot do | User-agent: Bytespider
Disallow: / | No official documentation published |
What about Amazon and Bytedance? What do they do?
Nobody really talks about these, so let’s dive in to their purpose to decide what rules you want to apply.
Amazon has crawlers that train their own AI models as well as support Amazon’s own search and shopping tools. Those tools connect directly to Rufus, Amazon’s AI shopping assistant in their app. Amazon says that Rufus pulls in outside sources like the NY Times, USA Today or Good Housekeeping when it answers questions about products and trends. This is information it layers on top of their own catalogue and reviews. This bot is most useful to you if you run an ecommerce brand, write product reviews or publish anything meant to surface when people are doing physical product research.
Bytedance on the other hand, is a whole different kind of problem. You can probably hear me sighing as I type this. Crawler monitoring reports describe instances of Bytedane repeatedly ignoring robots.txt outright. One site owner found that Bytedance alone made up 90% of all their AI bot traffic, and after blocking at the server level (because again, it doesn’t listen to robots.txt rules) their hosting costs reduced significantly. Bytedance trains Doubao, which is Bytedance’s own model and feeds search recommendations inside TikTok, which is important to consider given the fact that TikTok’s audience uses the app as a search engine, and 34% of the users are between 13 and 24. You should care about Bytespider if you are finding your server logs are showing unusual levels of bot traffic, if TikTok is a main discovery channel for your audience, or if you dont want to hand your content to it to train.
One pattern worth calling out is that the companies being the most transparent about their own crawlers are also the ones that have separate crawlers for training purposes. Thats OpenAI, Anthropic, and Perplexity. Perplexity takes it a step farther and tells siteowners they should actively allow Perplexitybot since it's only used for search results and not training.
What is Google-Extended bot, and is it important?
This bot trips a lot of people up because people assume it behaves like Google-bot, but the purpose is different and more defined. The point of Google-extended bot is to manage whether or not Google can train on your content for Gemini models, and whether it can be used for answers in Gemini apps and Vertex AI.
Being included in standard Google search is done entirely through Googlebot, which is a completely separate system from Google-Extended. AI Overviews lives inside of that infrastructure so if you don’t want to be listed inside AI Overviews, you will want to look at standard search controls like no-index. Google hasn’t published a full AI Overviews opt-out, and I honestly wouldn’t wait around for one.
How soon will changes to robots.txt take effect?
It depends on the bot. OpenAI says their bot OAI-SearchBot can take up to 24 hours to notice the changes. Amazon caches your robots.txt for 30 days, so you may still see crawlers during that entire period.
What should you block, and why?
When working with clients the first thing I want to understand is what lane they want to be in. Full training opt-out, AI-search visibility, or both because those outcomes can all be different setups.
Here’s a quick guide
If the business wants to… | Check first | Likely next step |
|---|
Limit use of public content for future model development | Training-related crawlers for the relevant company | Review and, if appropriate, add a Disallow rule for that specific crawler |
Remain available to AI-search products | Search-related crawlers for the relevant company | Confirm they aren't blocked in robots.txt, the CDN, the WAF, or bot-management settings |
Prevent a user-directed AI assistant from retrieving a page | User-triggered fetchers, access controls, and page availability | Check the platform's current documentation, since robots.txt alone may fall short here |
Remove content from Google Search or AI Overviews | Googlebot, noindex, and snippet controls | Use standard Google Search controls; Google-Extended doesn't manage Search inclusion |
Reduce bot traffic or infrastructure load | Server logs, CDN/WAF logs, and crawler verification | Verify the crawler's identity before blocking anything, and apply the least disruptive control that solves the actual problem |
Stop public access to sensitive content | Authentication, permissions, paywalls, or server rules | Don't rely on robots.txt for this. It only asks nicely. |
How AI can help audit your bot access
First export your live robots.txt file.
Then take a look at your server logs to find every instance of AI related user agents over the past 30 to 90 days. Usually your dev team can help you get access to server logs if needed.
Prompt:
You’re auditing a site for AI crawler coverage. I am adding the robots.txt and a full list of AI crawler agents seen in server logs over the past (90 days, etc). Compare these two sources against eachother. Flag any logged user-agent with no matching robots.txt rule, any rules targeting a crawler that has not showed up in our server logs, and any wildcard rules that might be blocking a crawler we want to allow. List each finding in detail with any information you have about that bot
Hope you found this useful! Tell me if you catch a documentation change I missed or a crawler doing something it's own docs say it doesnt. This reference only stays useful if it keeps getting checked against reality.
