Problems with crawlbots
Hi Guys,
We have a problem with a website of ours. The website is super big 25,000 pages approximately but because of this we get super many crawl requests. Is there any way to mitigate this via cloudflare. We don't want to turn them off completely.
We have a problem with a website of ours. The website is super big 25,000 pages approximately but because of this we get super many crawl requests. Is there any way to mitigate this via cloudflare. We don't want to turn them off completely.
Welcome to the official Cloudflare Developers server. Here you can ask for help and stay updated with the latest news
87,353Members
Resources
Similar Threads
Was this page helpful?
