Effect of AI bot blocking on AI overviews/search
I have a client with a small marketing site on Workers w/Static Assets and a custom domain that is now run through Cloudflare.
He wants to know if Cloudflare's AI bot detection will prevent or degrade his site's ability to appear in all of the different types of AI overviews and searches that are becoming common online now. Understandably, his priority is to be featured as widely as possible.
I've been looking at the Cloudflare documentation and also on Google, but haven't found anyone squarely addressing this.
I'm interested in the following:
If you block a site from training, does the AI know the site less well, and, therefore, feature it less often?
- Can someone explain what result Cloudflare's AI bot blocking features (for training or otherwise) has on consumers who are using AI to search for information?
- Should all of these features be turned off if maximum exposure is the goal? (And would that increase costs?)
At present, I believe the client has the following features turned on:
- Cloudflare is managing the robots.txt file by adding AI-specific provisions to it
- A Cloudflare-managed rule is active via a toggle to "Block AI Bots", apparently from training on site data
- Cloudflare has said publicly that it's blocking Perplexity, although, I see in AI Crawl Control that there's a toggle to allow Perplexity? As a result, I'm not really sure what the default position is on this one now. Have I missed an update or further explanation?
Any and all thoughts and/or pointers to docs, etc, are appreciated.
He wants to know if Cloudflare's AI bot detection will prevent or degrade his site's ability to appear in all of the different types of AI overviews and searches that are becoming common online now. Understandably, his priority is to be featured as widely as possible.
I've been looking at the Cloudflare documentation and also on Google, but haven't found anyone squarely addressing this.
I'm interested in the following:
If you block a site from training, does the AI know the site less well, and, therefore, feature it less often?
- Can someone explain what result Cloudflare's AI bot blocking features (for training or otherwise) has on consumers who are using AI to search for information?
- Should all of these features be turned off if maximum exposure is the goal? (And would that increase costs?)
At present, I believe the client has the following features turned on:
- Cloudflare is managing the robots.txt file by adding AI-specific provisions to it
- A Cloudflare-managed rule is active via a toggle to "Block AI Bots", apparently from training on site data
- Cloudflare has said publicly that it's blocking Perplexity, although, I see in AI Crawl Control that there's a toggle to allow Perplexity? As a result, I'm not really sure what the default position is on this one now. Have I missed an update or further explanation?
Any and all thoughts and/or pointers to docs, etc, are appreciated.
