CrowdSec•3mo ago

Inconsistencies between web, cscli, and ipset

Hi there! Lately we've been getting a lot of complaints from customers that Microsoft tools can't reach their pages (bingbot). We've been hacking away at it for days now, and we've found the root of the issue (or at least we think so, nonetheless it is a problem). These are the IP ranges used by bingbot: https://www.bing.com/toolbox/bingbot.json I've been trying to find decisions on all these ranges with the cscli tool (with the --all flag to count for CAPI decisions), and found nothing. But then, I found that there were indeed bans in place on the machines using ipset. For example, this /24, which belongs to bingbot and it came from crowdsec-blacklists-1:

ipset list | grep "207\.46"
207.46.13.0/24 timeout 1775506

ipset list | grep "207\.46"
207.46.13.0/24 timeout 1775506

I tried searching for this range on the CrowdSec Threat Intelligence IP Search tool using the ip_range and ip_range_24 queries, but nothing came up. I also couldn't use cscli decision delete to get rid of this rule. (Another prefix on the ipset list belonging to bingbot: 52.167.144.0/24, but I haven't searched this one up in the other tools yet) Could you please help me find out where this inconsistency is coming from? I know we could get around it with a CAPI whitelist, but it is troubling that these tools don't all show the same thing. Thanks a lot!

8 Replies

CrowdSec•3mo ago

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

iiamloz•3mo ago

very odd, so to point something out firstly CAPI and Blocklist never provide range remediations, only two cases from console are: - User uses decisions and types range remediation - User uses SAPI custom blocklists and uploads a range remediation So can we have some details like: - Are you importing decisions at all? - Do you have any custom scenarios that are triggering on ranges? - Which mode is the firewall in? (as if you run in iptables mode using iptables -L CROWDSEC_CHAIN it will provide a comment which crowdsec-x is tied to which origin) as an example

Chain CROWDSEC_CHAIN (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-6 src /* CrowdSec: CAPI */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-5 src /* CrowdSec: lists:crowdsec_public_scanners */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-4 src /* CrowdSec: lists:tor-exit-nodes */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-3 src /* CrowdSec: lists:free_proxies */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-2 src /* CrowdSec: lists:crowdsec_mail */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-1 src /* CrowdSec: lists:threat_forecast_df9b68d7-0667-4d06-8a3c-b8616dae9003 */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-0 src /* CrowdSec: crowdsec */

Chain CROWDSEC_CHAIN (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-6 src /* CrowdSec: CAPI */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-5 src /* CrowdSec: lists:crowdsec_public_scanners */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-4 src /* CrowdSec: lists:tor-exit-nodes */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-3 src /* CrowdSec: lists:free_proxies */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-2 src /* CrowdSec: lists:crowdsec_mail */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-1 src /* CrowdSec: lists:threat_forecast_df9b68d7-0667-4d06-8a3c-b8616dae9003 */
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists-0 src /* CrowdSec: crowdsec */

alukasOP•3mo ago

And that's it, you've nailed it, it comes from the imports.

REJECT     all  --  anywhere             anywhere             match-set crowdsec-blacklists-1 src /* CrowdSec: cscli-import */ reject-with icmp-port-unreachable

REJECT     all  --  anywhere             anywhere             match-set crowdsec-blacklists-1 src /* CrowdSec: cscli-import */ reject-with icmp-port-unreachable

Give me a second and I'll check in with my colleague on when he last imported rules, because he told me he hasn't in a while. Yeah, apparently that is where the decision comes from. However, what is the reason that cscli decisions list doesn't show any of the imported decisions?

$ sudo cscli decisions list --origin cscli-import
No active decisions

$ sudo cscli decisions list --origin cscli-import
No active decisions

And we also can't seem to use cscli decisions delete to get rid of it.

iiamloz•3mo ago

Try restarting the crowdsec firewall service, to see if it flushes the ipsets

alukasOP•3mo ago

That fixed it, thank you very much! It got rid of whole match-set.

iiamloz•3mo ago

Just note if you use cscli allowlist feature and import ranges/ips it will now correctly check those imports so you can automate importing it to an allowlist:

curl -s https://www.bing.com/toolbox/bingbot.json | jq '.prefixes[].ipv4Prefix'

curl -s https://www.bing.com/toolbox/bingbot.json | jq '.prefixes[].ipv4Prefix'

then for loop over the items

alukasOP•3mo ago

Yeah, unfortunately our problem with allowlists still stands, and that is that we want to control the whitelists on a machine-level, instead of one LAPI-level whitelist. This is because we do shared hosting, and on customer's machines we specifically whitelist certain IPs that they request us to. But perhaps for use-cases like this we should consider setting up allowlists. We're also in the midst of setting up capi-blocklist whitelists. Thanks very much for the help!

CrowdSec•3mo ago

Resolving Inconsistencies between web, cscli, and ipset This has now been resolved. If you think this is a mistake please run /unresolve

Gaming

Programming

Inconsistencies between web, cscli, and ipset

Did you find this page helpful?