AI Crawl Control blocks Googlebot crawls
Hello there, for some reason under my AI Crawl Control section, I see many unsuccessful crawls made by googlebot, even though I have proper robots.txt that allows everything, and it just quite scared me since it might damage the seo, I don’t have any ai bot blockers activated, only ddos protection and caching, but it still seems improper since out of 292 crawls, only 189 were allowed, and the rest marked as unsuccessful.
Could it be related perhaps to fake crawls? Like people imitating googlebot, and Cloudflare just blocks it, but still classifies it as “Googlebot”?,
Thank everyone who answers in advance!
Could it be related perhaps to fake crawls? Like people imitating googlebot, and Cloudflare just blocks it, but still classifies it as “Googlebot”?,
Thank everyone who answers in advance!
Welcome to the official Cloudflare Developers server. Here you can ask for help and stay updated with the latest news
87,353Members
Resources
Similar Threads
Was this page helpful?
