I'm building a webperf tester (e.g. say
I'm building a webperf tester (e.g. say similar to PageSpeed Insights) but on top of CF Puppeteer. I saw turnstile/bot challenges on some Cloudflare zones, but I'm surprised that Cloudflare's Radar itself also does get challenged.
I guess this is bot traffic so it makes sense. Would the expected behavior be that any domains who aggressively block bots should allow explicitly (via custom header or user-agent)? Or if you hit enough traffic (+1k requests/day) apply to be a verified bot?
5 Replies
interestingly enough I don't see https://webpagetest.org/ or https://pagespeed.web.dev/ getting challenged, but might just be that bot detection doesn't really detect them accidentally
hi this is for your own site?
this might help - either allow verified bots are create a custom rule
https://developers.cloudflare.com/browser-rendering/faq/#will-browser-rendering-bypass-cloudflares-bot-protection
Cloudflare Docs
FAQ
Below you will find answers to our most commonly asked questions. If you cannot find the answer you are looking for, refer to the Discord to explore additional resources.
hey Kathy! thanks for the response. In general users would test their own properties so they can definitely whitelist/add some custom rules. My question was more generic in case users tested an external property – I understand to apply individually for "verified bot" I need to get a lot of daily usage, or is that something Cloudflare would file for the Browser Rendering binding Worker itself?
browser rendering itself will not be filing as a verified bot bc there are many things a user could do with it. therefore will require each to individually apply for being a verified bot
yeah that makes total sense – thanks!