CThey will allowed by default. If you are making firewall rules to block them then you use the
verified bot option in WAF rules to ignore the known good bots
DAlready found i5
Dit still blocks the bingbot from crawling "(http.user_agent eq " Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/") or (http.user_agent eq "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) W.X.Y.Z Safari/537.36") or (http.user_agent eq "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)") or (http.user_agent eq "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)") or (http.user_agent eq "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36") or (http.user_agent eq "Mozilla/5.0 (compatible; Google-InspectionTool/1.0)") or (http.user_agent eq "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)") or (http.user_agent eq "Googlebot-Image/1.0") or (ip.src eq 20.191.45.212) or (ip.src eq 40.88.21.235) or (ip.src eq 40.76.173.151) or (ip.src eq 40.76.163.7) or (ip.src eq 20.185.79.47) or (ip.src eq 52.142.26.175) or (ip.src eq 20.185.79.15) or (ip.src eq 52.142.24.149) or (ip.src eq 40.76.162.208) or (ip.src eq 40.76.163.23) or (ip.src eq 40.76.162.191) or (ip.src eq 40.76.162.247)" its inside my waf rules if it isnt one of these it skips all the rules
CYou don't need to do all the user-agent checking. Use
cf.client.bot to get if a verified botDWhere
DIts that simple
DI still get a 400-499 from microsoft while the bot is allowed
D
MHi, I am getting still same bingbot crawl blocking issue. any help on that?
