CrowdSec•9mo ago

LAPI stream mode doesn't return active decision when filtering origin "crowdsec"

https://github.com/crowdsecurity/crowdsec/issues/3373 I just made this bug report since it clearly looks like a bug to me, but does anyone have any idea how to fix it ?

GitHub

LAPI stream mode doesn't return active decision when filtering orig...

What happened? LAPI stram mode doesn't return active decision when filtering origin "crowdsec" but will when using querry mode What did you expect to happen? The active decision shoul...

26 Replies

CrowdSec•9mo ago

Important Information

This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve

blotus•9mo ago

Hello, Can you provide the output of cscli decisions list ? crowdsec origin means a local automated decision, so you won't get anything from CAPI in this case (origin can be CAPI , lists), from cscli (origin can be cscli or cscli-import or from the console for the paid plans (console origin) Seems to work fine for me:

curl -H "x-api-key: test" "localhost:8081/v1/decisions/stream?startup=true&origins=crowdsec&scopes=ip&scenarios_containing=http-&startup=true"
{"deleted":null,"new":[{"duration":"3h59m44s","id":67613973,"origin":"crowdsec","scenario":"crowdsecurity/http-path-traversal-probing","scope":"Ip","type":"ban","uuid":"a4a8aff0-d92a-4f5e-8022-d7add0fb23e7","value":"127.0.0.1"}]}

curl -H "x-api-key: test" "localhost:8081/v1/decisions/stream?startup=true&origins=crowdsec&scopes=ip&scenarios_containing=http-&startup=true"
{"deleted":null,"new":[{"duration":"3h59m44s","id":67613973,"origin":"crowdsec","scenario":"crowdsecurity/http-path-traversal-probing","scope":"Ip","type":"ban","uuid":"a4a8aff0-d92a-4f5e-8022-d7add0fb23e7","value":"127.0.0.1"}]}

AR2000OP•9mo ago

╭──────────┬──────────┬────────────────────┬───────────────────────────┬────────┬─────────┬───────────────────────────────────────────────────┬────────┬────────────┬──────────╮
│    ID    │  Source  │     Scope:Value    │           Reason          │ Action │ Country │                         AS                        │ Events │ expiration │ Alert ID │
├──────────┼──────────┼────────────────────┼───────────────────────────┼────────┼─────────┼───────────────────────────────────────────────────┼────────┼────────────┼──────────┤
│ 17252026 │ crowdsec │ Ip:103.216.116.126 │ crowdsecurity/ssh-slow-bf │ ban    │ VN      │ 135905 VIETNAM POSTS AND TELECOMMUNICATIONS GROUP │ 18     │ 3h0m24s    │ 8589     │
╰──────────┴──────────┴────────────────────┴───────────────────────────┴────────┴─────────┴───────────────────────────────────────────────────┴────────┴────────────┴──────────╯

╭──────────┬──────────┬────────────────────┬───────────────────────────┬────────┬─────────┬───────────────────────────────────────────────────┬────────┬────────────┬──────────╮
│    ID    │  Source  │     Scope:Value    │           Reason          │ Action │ Country │                         AS                        │ Events │ expiration │ Alert ID │
├──────────┼──────────┼────────────────────┼───────────────────────────┼────────┼─────────┼───────────────────────────────────────────────────┼────────┼────────────┼──────────┤
│ 17252026 │ crowdsec │ Ip:103.216.116.126 │ crowdsecurity/ssh-slow-bf │ ban    │ VN      │ 135905 VIETNAM POSTS AND TELECOMMUNICATIONS GROUP │ 18     │ 3h0m24s    │ 8589     │
╰──────────┴──────────┴────────────────────┴───────────────────────────┴────────┴─────────┴───────────────────────────────────────────────────┴────────┴────────────┴──────────╯

I do want only the local automated decision. I don't want to fill my firewall with the CAPI IPs In my issue I decribed using

curl --header "X-Api-Key: <REDACTED>" "http://127.0.0.1:8080/v1/decisions?origins=crowdsec&scenarios_containing=ssh&scopes=Ip&startup=true" -s

and having a valid result (duplicated decisions since the bouncer didn't work) However the stream api with the startup argument did not return anything (exact same parameters)

curl --header "X-Api-Key: <REDACTED>" "http://127.0.0.1:8080/v1/decisions/stream?origins=crowdsec&scenarios_containing=ssh&scopes=Ip&startup=true" -s

blotus•9mo ago

when you did the request for /stream, are you 100% sure you had specified startup=true ? (assuming you did the request after the non-stream one) that's really weird, as you can see from my example, it seems to work just fine (i'm filtering on http- instead of ssh, but this has no practical impact)

AR2000OP•9mo ago

Copy pasted the cmd from the terminal

blotus•9mo ago

i may have an idea can you run cscli decisions list --all | grep 103.216.116.126 ? this IP is part of the community blocklist so you may already have a decision for it coming from CAPI, and crowdsec might be a bit confused when it comes to choosing which decisions to take

AR2000OP•9mo ago

I think that's the issue. But I really don't want to fill my firewall I've seen it in the CAPI list when testing

│ 17262607 │ CAPI   │ Ip:103.216.116.126                        │ crowdsecurity/ssh-slow-bf                    │ ban    │         │    │ 0      │ 166h0m15s  │ 8590     │

│ 17262607 │ CAPI   │ Ip:103.216.116.126                        │ crowdsecurity/ssh-slow-bf                    │ ban    │         │    │ 0      │ 166h0m15s  │ 8590     │

But that shouldn't prevent a local decision from being returned. They are 2 different ones, not one with multiples origin (wich is not possible)

blotus•9mo ago

We only take the longest available decision if you have multiple decisions on the same IP. I need to check the code, but I guess the condition for the duration takes priority and we do not even consider the local one and what's the reason for not wanting to have every decisions in your firewall ? What kind of firewall is it ? I only know about very few firewalls that cannot deal with a few 10s of thousands of IPs in a blocklist but yes, that's indeed a bug, I'll answer to the issue with the details about the duplicated decision with CAPI

AR2000OP•9mo ago

It's a ASUSTOR Nas, with a old firewall that does not support ipset or nft

AR2000OP•9mo ago

I had to make a custom bouncer to work with it. https://gitea.ar2000.me/AR2000/crowdsec-legacy-firewall-bouncer

Gitea

crowdsec-legacy-firewall-bouncer

Crowdsec firewall bouncer for systems that does not support ipset. Base on crowdsec-custom-bouncer

blotus•9mo ago

I see

AR2000OP•9mo ago

The official firewall bouncer doesn't know how to handle such old firewall

blotus•9mo ago

yeah, we require ipset I don't think I ever remember seeing a machine where ipset was not supported 😄 something that could be important (although if you are ignoring decisions from CAPI, probably not): the custom bouncer supports feeding the decisions to your script standard input, it's much faster than the default behaviour of calling your script for each decision

AR2000OP•9mo ago

https://www.ldlc.com/fiche/PB00561760.html?offerId=AR202307060013 OS : ADM 4.3.3.RC92 uname -a : Linux jacinthe 5.13.x #1 SMP Mon Dec 9 00:11:50 CST 2024 x86_64 GNU/Linux

ASUSTOR Nimbustor 4 Gen 2 AS5404T - Serveur NAS - LDLC

Achat Serveur NAS ASUSTOR Nimbustor 4 Gen 2 AS5404T (AS5404T) sur LDLC, n°1 du high-tech. Barebone Serveur NAS 4 baies - Intel Celeron N5105 4 Go Dual LAN 2.5 GbE.

AR2000OP•9mo ago

Maybe I'll update it someday. But for now it work perfectly (in theory)

blotus•9mo ago

is it running a custom linux version or something (or custom kernbel that cannot load modules) ? as it seems to support docker, the kernel cannot be that old

AR2000OP•9mo ago

idk. I just know that ipset is not a thing, and iptable-nft will return a error

$ iptables-nft -L
iptables: Failed to initialize nft: Protocol not supported

$ iptables-nft -L
iptables: Failed to initialize nft: Protocol not supported

AR2000OP•9mo ago

https://github.com/crowdsecurity/crowdsec/issues/3373#issuecomment-2554198963 A easy fix would be to apply the origin filters first when looking for the decision

GitHub

LAPI stream mode doesn't return active decision when filtering orig...

What happened? LAPI stram mode doesn't return active decision when filtering origin "crowdsec" but will when using querry mode What did you expect to happen? The active decision shoul...

blotus•9mo ago

can you try to add dedup=false to your curl query ? (although I don't think it's supported in the custom bouncer 😦 ) yes that would be the idea, but it could be a little bit trickier than that: we had some performance issues with this exact query in the past when running on mysql

AR2000OP•9mo ago

It work fair reason to not do it like that

blotus•9mo ago

at least, that confirms the issue, when setting this parameter, we don't do the check for the duration I just checked, we don't even expose the parameter in our API client so you'd have to basically reimplement an entire bouncer to work around this (it would be very barebones, but still)

AR2000OP•9mo ago

and the custom bouncer doesn't support the querry api, only the stream one. It would have been a valid workaround since the querry api return the correct result Thx a lot for the info. It didn't help, but at least I know what is happening, and that it is taken seriously by the team

AR2000OP•9mo ago

btw ^^ :

AR2000OP•9mo ago

I don't know if I'll ever do it, but it's in the issue list

AR2000OP•9mo ago

https://gitea.ar2000.me/AR2000/crowdsec-legacy-firewall-bouncer/issues/1

Gitea

Support reading input from stdin for performance increase

something that could be important (although if you are ignoring decisions from CAPI, probably not): the custom bouncer supports feeding the decisions to your script standard input, it's much faster than the default behaviour of calling your script for each decision

From crowdsec team discord...

CrowdSec•9mo ago

Resolving LAPI stream mode doesn't return active decision when filtering origin "crowdsec" This has now been resolved. If you think this is a mistake please run /unresolve

Gaming

Programming

LAPI stream mode doesn't return active decision when filtering origin "crowdsec"

Did you find this page helpful?