CrowdSec•4w ago

Crowdsec FW bouncer with nftables configured but I think It’s not working

Hello everyone I have a caddy + Coraza + Crowdsec with docker compose working fine and reporting to the console. My OS is raspian which is based on Debian 12 (bookworm) I have decided to add the FW bouncer so that the IPs are blocked in the system FW, which is an nftables. I have installed the nftables version on the OS directly and connected it to crowdsec. When I force a block on Coraza, the Crowdsec integration parses the log, detects it and blocks it. I see that the blocked IP is added to the blacklist of the FW bouncer but I don’t see anything in the kern.log and that makes me suspect that something is not working well because I do see all the UFW entries. I've created some days ago a post in the Crowdsec forums where I put all the log, configurations, cscli commands output, etc https://discourse.crowdsec.net/t/crowdsec-fw-bouncer-with-nftables-configured-but-i-think-its-not-working/2678 Probably is the best place to see all the all the information organized and structured because I have not seen how to do it here but if you think it is better to put it back here, I will do it immediately. Thank you very much in advance

CrowdSec

Crowdsec FW bouncer with nftables configured but I think It's not w...

17 Replies

CrowdSec•4w ago

Important Information

Thank you for getting in touch with your support request. To expedite a swift resolution, could you kindly provide the following information? Rest assured, we will respond promptly, and we greatly appreciate your patience. While you wait, please check the links below to see if this issue has been previously addressed. If you have managed to resolve it, please use run the command /resolve or press the green resolve button below.

Log Files

If you possess any log files that you believe could be beneficial, please include them at this time. By default, CrowdSec logs to /var/log/, where you will discover a corresponding log file for each component.

Guide Followed (CrowdSec Official)

If you have diligently followed one of our guides and hit a roadblock, please share the guide with us. This will help us assess if any adjustments are necessary to assist you further.

Screenshots

Please forward any screenshots depicting errors you encounter. Your visuals will provide us with a clear view of the issues you are facing.

iiamloz•4w ago

From your discourse post to this you missed out this information

Note: I’m behind Cloudflare tunnel and I see in caddy.log the access from the same IP (172.18.0.1).

Note: I’m behind Cloudflare tunnel and I see in caddy.log the access from the same IP (172.18.0.1).

This is crucial as cloudflare tunnel basically bypasses your firewall, as that is the intended purpose of tunnels to bypass NAT or other obstatcles that may cause issues from getting from WAN to LAN. Cause remember from the eyes of the firewall, there is no ip state from the tunnel all it knows is the request is originating from 172.18.0.1 as it doesnt have the ability to see the forwarded headers at layer 3/4

IgnacioSeijoOP•4w ago

Hi, Thanks for the reply I understand you but I think I don't understand completely. Before installint FW bouncer, Caddy see the X-Forwarder for header with the original IP. After installing FW bouncer, Caddy only see the FW bouncer IP (172.18.0.1) Why? On the other hand, Its explains why 172.18.0.1 is the unique IP in the logs but not why there is no crowdsec entries in the kern.log, isn't it? It shoud be a 172.18.0.1 in the log as this IP is in the crowdsec black list, shouldn't it? And the same for the manual ban 1.2.3.4

iiamloz•4w ago

Before installing the FW bouncer, Caddy sees the original IP in the X-Forwarded-For header. After installing the FW bouncer, Caddy only sees 172.18.0.1. Why? This is very likely a Docker networking issue rather than a CrowdSec issue. Docker’s networking mode can make the firewall see only the Docker bridge IP, which then shows up in your app logs. There are known problems in some Docker setups that use nftables; see the discussion here: https://github.com/docker/for-linux/issues/1472 It explains why 172.18.0.1 is the only IP in the logs, but why are there no CrowdSec entries in kern.log? kern.log only gets a line when a firewall rule actually matches and takes action. If nothing matches, there is nothing to log. Shouldn’t kern.log show 172.18.0.1 since it is in the CrowdSec blacklist? Not necessarily. Private network ranges are allowlisted by default, so 172.18.0.1 is probably being ignored by CrowdSec and never reaches a “block” rule in the firewall. What about a manual ban like 1.2.3.4? If you use a tunnel (for example Cloudflare Tunnel), the firewall only sees traffic coming from the tunnel process or proxy on your machine. It cannot see the original client IP, so the firewall rules cannot match the original IP. In that setup, the firewall will not block 1.2.3.4 because, at the firewall level, the request looks like it comes from the tunnel endpoint, not directly from 1.2.3.4.

IgnacioSeijoOP•4w ago

Ok, let me see if I'm undertanding. Private network ranges are allowlisted by default Is there any way to modify or disable this list just to test? It cannot see the original client IP, so the firewall rules cannot match the original IP. You mean that in addition to being on the crowdsec blacklist, the FW has to see the connection of that IP and if not, it doesn't create the rule?

iiamloz•4w ago

You mean that in addition to being on the crowdsec blacklist, the FW has to see the connection of that IP and if not, it doesn't create the rule?

No it creates the rule, but its never matched because of the tunnel

Private network ranges are allowlisted by default

Yes but highly unadvised as it means it will ban cloudflared, meaning it would ban everyone at once cscli parsers remove crowdsecurity/whitelists then restart the container or systemd service.

IgnacioSeijoOP•4w ago

No it creates the rule, but its never matched because of the tunnel Ahhhh NOW I see the light 🙂 Yo the rule is created because the IP is in the list but there is no mach so no kern.log. Sorry for the stupid question.

iiamloz•4w ago

There is no stupid questions, maybe lacking answers 😄

IgnacioSeijoOP•4w ago

cscli parsers remove crowdsecurity/whitelists I did something like that a few says ago but reversible I renamed the whitelist to whitelists.disabled and commented all the lines. So, the IP should be banned?

iiamloz•4w ago

it depends if the IP did anything bad in the logs, this whitelist only stops crowdsec from coming up with the decision from logs. it doesnt stop you from running cscli decisions add -i 172.18.0.1 -d 2m

IgnacioSeijoOP•4w ago

I force a bad behaviour in my access and coarza blocks it. The, the coraza collection parse the log and the decission is created. I see it in the decission list and in the black list but nothing in kern.log

iiamloz•4w ago

Yes, as previously stated, tunnels bypass the firewall

IgnacioSeijoOP•4w ago

so, the FW don't see the 172.18.0.1 incomming connection. Sorry again for the stupid question 🙂 Which lead to this. You cannot firewall any connection from the tunnel, you hace tu trust in the tunnel. Right?

iiamloz•4w ago

it depends where the cloudflared is situated, most likely the packet going from cloudflared to docker is not going through the same chains as it originating from itself and docker will NAT the connection. Basically yes that is the downside to using a tunnel so you have to just remediate at the Caddy level

IgnacioSeijoOP•4w ago

Theorically, I've added the docker chain.

raspi@raspberrypi3:~ $ sudo more /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
mode: nftables
update_frequency: 10s
log_mode: file
log_dir: /var/log/
log_level: info
log_compression: true
log_max_size: 100
log_max_backups: 3
log_max_age: 30
api_url: http://localhost:8080/
api_key: XXX
## TLS Authentication
# cert_path: /etc/crowdsec/tls/cert.pem
# key_path: /etc/crowdsec/tls/key.pem
# ca_cert_path: /etc/crowdsec/tls/ca.crt
insecure_skip_verify: false
disable_ipv6: true
deny_action: DROP
deny_log: true
supported_decisions_types:
  - ban
#to change log prefix
#deny_log_prefix: "crowdsec: "
#to change the blacklists name
blacklists_ipv4: crowdsec-blacklists
blacklists_ipv6: crowdsec6-blacklists
#type of ipset to use
ipset_type: nethash
#if present, insert rule in those chains
iptables_chains:
  - INPUT
#  - FORWARD
  - DOCKER-USER
iptables_add_rule_comments: true

## nftables
nftables:
  ipv4:
    enabled: true
    set-only: false
    table: crowdsec
    chain: crowdsec-chain
    priority: -10
  ipv6:
    enabled: true
    set-only: false
    table: crowdsec6
    chain: crowdsec6-chain
    priority: -10

nftables_hooks:
  - input
#  - forward
  - docker-user

# packet filter
pf:
  # an empty string disables the anchor
  anchor_name: ""

prometheus:
  enabled: false
  listen_addr: 127.0.0.1
  listen_port: 60601

raspi@raspberrypi3:~ $ sudo more /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
mode: nftables
update_frequency: 10s
log_mode: file
log_dir: /var/log/
log_level: info
log_compression: true
log_max_size: 100
log_max_backups: 3
log_max_age: 30
api_url: http://localhost:8080/
api_key: XXX
## TLS Authentication
# cert_path: /etc/crowdsec/tls/cert.pem
# key_path: /etc/crowdsec/tls/key.pem
# ca_cert_path: /etc/crowdsec/tls/ca.crt
insecure_skip_verify: false
disable_ipv6: true
deny_action: DROP
deny_log: true
supported_decisions_types:
  - ban
#to change log prefix
#deny_log_prefix: "crowdsec: "
#to change the blacklists name
blacklists_ipv4: crowdsec-blacklists
blacklists_ipv6: crowdsec6-blacklists
#type of ipset to use
ipset_type: nethash
#if present, insert rule in those chains
iptables_chains:
  - INPUT
#  - FORWARD
  - DOCKER-USER
iptables_add_rule_comments: true

## nftables
nftables:
  ipv4:
    enabled: true
    set-only: false
    table: crowdsec
    chain: crowdsec-chain
    priority: -10
  ipv6:
    enabled: true
    set-only: false
    table: crowdsec6
    chain: crowdsec6-chain
    priority: -10

nftables_hooks:
  - input
#  - forward
  - docker-user

# packet filter
pf:
  # an empty string disables the anchor
  anchor_name: ""

prometheus:
  enabled: false
  listen_addr: 127.0.0.1
  listen_port: 60601

iiamloz•4w ago

Yes, but its NAT and within nftables docker doesnt use docker-user it uses forward AFAIK

IgnacioSeijoOP•4w ago

I will try to use iptables in debian 12 ASAP. So I will have to use FORWARD chain instead DOCKER-USER? On the other hand, if I'm not able to revert nftables to iptables, I'm not sure about the nftables configuration. Is this configuration correct? lowercase instead uppercase?

nftables_hooks:
  - input
  - forward
  - docker-user

nftables_hooks:
  - input
  - forward
  - docker-user

I'ts something that I haven't found documentatin but It was in the file after installing the FW bouncer Another question (sorry) I'm confused abot the bouncer version I should use.

raspi@raspberrypi3:~ $ sudo iptables -V
iptables v1.8.9 (nf_tables)

raspi@raspberrypi3:~ $ sudo iptables -V
iptables v1.8.9 (nf_tables)

I see nf_tables buy also iptables. I read that it means Debian uses nftables but has a iptables frontent so may be I'm using the worng bouncer version (nftables version) and I shoud use the iptables mode. This is the startup log when using iptables edit: I can't post all together due to a character restriction

level=info msg="Starting crowdsec-firewall-bouncer v0.0.34-debian-pragmatic-arm64-4144555453620958398aee64253dfd90bbc1f698"
 level=info msg="backend type: iptables"
level=info msg="IPV6 is disabled"
level=info msg="using 'DROP' as deny_action"
level=info msg="iptables for ipv4 initiated"
level=info msg="Deleting rule : /usr/sbin/iptables -D INPUT -j CROWDSEC_CHAIN"
level=error msg="error while removing rule : exit status 2 --> iptables v1.8.9 (nf_tables): Chain 'CROWDSEC_CHAIN' does not exist\nTry `iptables -h' or 'iptables --help' for more information.\n"
 level=info msg="Deleting rule : /usr/sbin/iptables -D FORWARD -j CROWDSEC_CHAIN"
 level=error msg="error while removing rule : exit status 2 --> iptables v1.8.9 (nf_tables): Chain 'CROWDSEC_CHAIN' does not exist\nTry `iptables -h' or 'iptables --help' for more information.\n"
level=info msg="Deleting rule : /usr/sbin/iptables -D DOCKER-USER -j CROWDSEC_CHAIN"
level=error msg="error while removing rule : exit status 2 --> iptables v1.8.9 (nf_tables): Chain 'CROWDSEC_CHAIN' does not exist\nTry `iptables -h' or 'iptables --help' for more information.\n"

level=info msg="Starting crowdsec-firewall-bouncer v0.0.34-debian-pragmatic-arm64-4144555453620958398aee64253dfd90bbc1f698"
 level=info msg="backend type: iptables"
level=info msg="IPV6 is disabled"
level=info msg="using 'DROP' as deny_action"
level=info msg="iptables for ipv4 initiated"
level=info msg="Deleting rule : /usr/sbin/iptables -D INPUT -j CROWDSEC_CHAIN"
level=error msg="error while removing rule : exit status 2 --> iptables v1.8.9 (nf_tables): Chain 'CROWDSEC_CHAIN' does not exist\nTry `iptables -h' or 'iptables --help' for more information.\n"
 level=info msg="Deleting rule : /usr/sbin/iptables -D FORWARD -j CROWDSEC_CHAIN"
 level=error msg="error while removing rule : exit status 2 --> iptables v1.8.9 (nf_tables): Chain 'CROWDSEC_CHAIN' does not exist\nTry `iptables -h' or 'iptables --help' for more information.\n"
level=info msg="Deleting rule : /usr/sbin/iptables -D DOCKER-USER -j CROWDSEC_CHAIN"
level=error msg="error while removing rule : exit status 2 --> iptables v1.8.9 (nf_tables): Chain 'CROWDSEC_CHAIN' does not exist\nTry `iptables -h' or 'iptables --help' for more information.\n"

level=info msg="Flushing chain : /usr/sbin/iptables -F CROWDSEC_CHAIN"
level=error msg="error while flushing chain : exit status 1 --> iptables: No chain/target/match by that name.\n"
ti level=info msg="Deleting chain : /usr/sbin/iptables -X CROWDSEC_CHAIN"
level=error msg="error while deleting chain : exit status 1 --> iptables: No chain/target/match by that name.\n"
level=info msg="Flushing logging chain : /usr/sbin/iptables -F CROWDSEC_LOG"
level=error msg="error while flushing logging chain : exit status 1 --> iptables: No chain/target/match by that name.\n"
level=info msg="Deleting logging chain : /usr/sbin/iptables -X CROWDSEC_LOG"
level=error msg="error while deleting logging chain : exit status 1 --> iptables: No chain/target/match by that name.\n"
level=info msg="Creating chain : /usr/sbin/iptables -N CROWDSEC_CHAIN -t filter"
level=info msg="Adding rule : /usr/sbin/iptables -I INPUT -j CROWDSEC_CHAIN"
level=info msg="Adding rule : /usr/sbin/iptables -I FORWARD -j CROWDSEC_CHAIN"
level=info msg="Adding rule : /usr/sbin/iptables -I DOCKER-USER -j CROWDSEC_CHAIN"
level=info msg="Creating logging chain : /usr/sbin/iptables -N CROWDSEC_LOG -t filter"
level=info msg="Adding logging rule : /usr/sbin/iptables -I CROWDSEC_LOG -j LOG --log-prefix crowdsec drop: "
level=info msg="Adding target rule to logging chain : /usr/sbin/iptables -A CROWDSEC_LOG -j DROP"
level=info msg="Using API key auth"
level=info msg="Processing new and deleted decisions . . ."
level=info msg="974 decisions deleted"
level=info msg="Using crowdsec-blacklists-0 as set for origin CAPI"
level=info msg="Creating rule : /usr/sbin/iptables -I CROWDSEC_CHAIN -m set --match-set crowdsec-blacklists-0 src -j CROWDSEC_LOG -m comment --comment CrowdSec: CAPI"
level=info msg="17317 decisions added"

level=info msg="Flushing chain : /usr/sbin/iptables -F CROWDSEC_CHAIN"
level=error msg="error while flushing chain : exit status 1 --> iptables: No chain/target/match by that name.\n"
ti level=info msg="Deleting chain : /usr/sbin/iptables -X CROWDSEC_CHAIN"
level=error msg="error while deleting chain : exit status 1 --> iptables: No chain/target/match by that name.\n"
level=info msg="Flushing logging chain : /usr/sbin/iptables -F CROWDSEC_LOG"
level=error msg="error while flushing logging chain : exit status 1 --> iptables: No chain/target/match by that name.\n"
level=info msg="Deleting logging chain : /usr/sbin/iptables -X CROWDSEC_LOG"
level=error msg="error while deleting logging chain : exit status 1 --> iptables: No chain/target/match by that name.\n"
level=info msg="Creating chain : /usr/sbin/iptables -N CROWDSEC_CHAIN -t filter"
level=info msg="Adding rule : /usr/sbin/iptables -I INPUT -j CROWDSEC_CHAIN"
level=info msg="Adding rule : /usr/sbin/iptables -I FORWARD -j CROWDSEC_CHAIN"
level=info msg="Adding rule : /usr/sbin/iptables -I DOCKER-USER -j CROWDSEC_CHAIN"
level=info msg="Creating logging chain : /usr/sbin/iptables -N CROWDSEC_LOG -t filter"
level=info msg="Adding logging rule : /usr/sbin/iptables -I CROWDSEC_LOG -j LOG --log-prefix crowdsec drop: "
level=info msg="Adding target rule to logging chain : /usr/sbin/iptables -A CROWDSEC_LOG -j DROP"
level=info msg="Using API key auth"
level=info msg="Processing new and deleted decisions . . ."
level=info msg="974 decisions deleted"
level=info msg="Using crowdsec-blacklists-0 as set for origin CAPI"
level=info msg="Creating rule : /usr/sbin/iptables -I CROWDSEC_CHAIN -m set --match-set crowdsec-blacklists-0 src -j CROWDSEC_LOG -m comment --comment CrowdSec: CAPI"
level=info msg="17317 decisions added"

I see the chains created and the blacklist with "sudo ipset list crowdsec-blacklists-0" I have more info. I've made a trick to test de FW bouncer and works. I've done it with nftables and iptables modes. 1º I access to my ssh (open just to test) ans see the IP in the auth.log 2º I add this IP manually to the decission list 3º I access to my ssh again and now I see the drop in the kern.log "crowdsec drop: IN=eth0 OUT= MAC=b8:27:eb:99:b9:69:94:83:c4:38:49:0d:08:00:45:00:00:3c:d0:9b:40:00:30:06:d8:7c SRC=176.83.40.150 DST=192.168.8.18" Finally I've understood what it's happening, THANKS 🙂 The question still applies. Do I have to use the nftables or the iptables bouncer version? Sincerly, seem that both works because Debian 12 uses nftables but has a trasnlator iptables<->nftables

Gaming

Programming

Crowdsec FW bouncer with nftables configured but I think It’s not working

Did you find this page helpful?