CrowdSec•4mo ago

CrowdSec blocking googlebot SEO crawler

Hello, I made a topic on crowdsec forum, for faster answer I post here too. I apologize. I noticed that Crowdsec is repeatedly blocking legitimate traffic from Googlebot (from IP address 66.249.64.165), which is negatively affecting our SEO. This IP is confirmed to be part of Google’s crawler range. This is my crowdsec version and other details: version: v1.6.8-debian-pragmatic-amd64-f209766e Codename: alphaga BuildDate: 2025-03-25_14:48:27 GoVersion: 1.24.1 Platform: linux libre2: C++ User-Agent: crowdsec/v1.6.8-debian-pragmatic-amd64-f209766e-linux can you guys kindly help me? Is there a known issue with false positives related to Googlebot? Is this normal? what do I need to do? Kind regards

5 Replies

CrowdSec•4mo ago

Important Information

Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command /resolve or press the green resolve button below.

Log Files

If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.

Guide Followed (CrowdSec Official)

If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.

Screenshots

Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.

iiamloz•4mo ago

We mark it as a safe IP from the CTI side, so most likely its crawling and triggering a scenario. Do you have any details if you search your alerts cscli alerts list and find the IP? also we have this https://app.crowdsec.net/hub/author/crowdsecurity/collections/whitelist-good-actors which will help to prevent seo bots from being banned

whichlockeOP•4mo ago

################################################################################################

 - ID           : 41377
 - Date         : 2025-05-23T13:48:20Z
 - Machine      : 9ee08ec60eb74d0993e74357fc6e09easxTFmychSpqoTU3R
 - Simulation   : false
 - Remediation  : true
 - Reason       : crowdsecurity/http-probing
 - Events Count : 25
 - Scope:Value  : Ip:66.249.64.165
 - Country      : US
 - AS           : GOOGLE
 - Begin        : 2025-05-23 13:44:39.320760638 +0000 UTC
 - End          : 2025-05-23 13:48:19.752924162 +0000 UTC
 - UUID         : 12682da9-e3ce-4d01-add6-5202c9b18c39

################################################################################################

 - ID           : 41377
 - Date         : 2025-05-23T13:48:20Z
 - Machine      : 9ee08ec60eb74d0993e74357fc6e09easxTFmychSpqoTU3R
 - Simulation   : false
 - Remediation  : true
 - Reason       : crowdsecurity/http-probing
 - Events Count : 25
 - Scope:Value  : Ip:66.249.64.165
 - Country      : US
 - AS           : GOOGLE
 - Begin        : 2025-05-23 13:44:39.320760638 +0000 UTC
 - End          : 2025-05-23 13:48:19.752924162 +0000 UTC
 - UUID         : 12682da9-e3ce-4d01-add6-5202c9b18c39

Here is the detail.

iiamloz•4mo ago

Ye so it crawled very fast and triggered a scenario, you can do cscli decisions delete -i 66.249.64.165 to remove the decision and install the collection I linked and it will prevent it in future. just note once installled you need to restart crowdsec via systemctl restart crowdsec and it will reload the configuration

whichlockeOP•4mo ago

Ok will do now. Thank you

Gaming

Programming

CrowdSec blocking googlebot SEO crawler

Did you find this page helpful?