C
CrowdSec•2mo ago
alukas

Postoverflow Whitelist Ignored

Hi there! We have a custom postoverflow whitelist solution, but lately a customer has been complaining that their API bot is being banned despite being on our whitelist. I've been debugging the issue for hours, but can't find why this happens. We have the same whitelist system running on multiple servers, and it works flawlessly on our SMTP servers for example. The server where we first became aware of this issue is running Apache, and the reason for the alerts and bans are crowdsecurity/http-backdoors-attempts and crowdsecurity/http-probing. Unfortunately our customer is running a badly written API bot, so all their requests come from the same IP, regardless of the actual user's IP. They acknowledged this, and are not worried that a whitelist would let all this through. This is how our whitelist system is set up currently (company name obscured): /etc/crowdsec/postoverflows/s01-whitelist/custom_whitelist.yaml 
name: <company>/custom-whitelist
description: "Whitelist special IP's and ranges on customer's requests"
whitelist:
        reason: "<company> customer requested whitelist"
        expression:
            - "any(File('<company>_ipwl.txt'),  IpInRange(evt.Overflow.Alert.Source.IP ,#))"
data:
        - source_url: https://www.<company>.hu/ipWhitelist/custom_whitelist.txt
          dest_file: <company>_ipwl.txt
          type: string
name: <company>/custom-whitelist
description: "Whitelist special IP's and ranges on customer's requests"
whitelist:
        reason: "<company> customer requested whitelist"
        expression:
            - "any(File('<company>_ipwl.txt'),  IpInRange(evt.Overflow.Alert.Source.IP ,#))"
data:
        - source_url: https://www.<company>.hu/ipWhitelist/custom_whitelist.txt
          dest_file: <company>_ipwl.txt
          type: string
The <company>_ipwl.txt contains IP ranges, something like this: /var/lib/crowdsec/data/<company>_ipwl.txt 
172.66.0.120/32
212.92.11.158/32
172.66.0.120/32
212.92.11.158/32
I've checked the contents of this file, it is correct, it contains the right IP. It seems to me that the postoverflow step just doesn't run for some reason. I'm not sure if it has something to do with the type of alert that is being triggered, given that this system works on our mail servers. Any help, or any debugging steps I could take are much appreciated. Here's what I know for sure: - the contents of the ipwl.txt file are correct - the postoverflow is loaded
27 Replies
CrowdSec
CrowdSec•2mo ago
Important Information
This post has been marked as resolved. If this is a mistake please press the red button below or type /unresolve
© Created By WhyAydan for CrowdSec ❤️
alukas
alukasOP•2mo ago
The whitelist is hit, but 0 requests get whitelisted. Due to privacy reasons I can't share any actual logs or alerts here, but I'd be glad to supply anything you need to help me solve this in private messages.
blotus
blotus•2mo ago
You can set debug: true in your postoverflow to see what crowdsec is trying to match in it
blotus
blotus•2mo ago
Also, if the LAPI is dedicated to this customer, you can use centralized allowlists (https://docs.crowdsec.net/docs/next/local_api/centralized_allowlists) in the LAPI: they are a bit easier to work with
Allowlists | CrowdSec
The AllowLists feature in CrowdSec lets users manage IP-based allowlists at the LAPI level, affecting both local decisions and blocklist pulls. Paying customers can also control AllowLists directly from the console for added convenience. This ensures greater flexibility in managing trusted IPs while maintaining CrowdSec’s robust security measu...
alukas
alukasOP•2mo ago
We are running a company-wide LAPI unfortunately and allowlists lack a function that would allow us to switch to it. I'll try debug: true
blotus
blotus•2mo ago
What is lacking ?
alukas
alukasOP•2mo ago
https://github.com/crowdsecurity/crowdsec/issues/3525
GitHub
Add node-specific allowlists, extend HTTP LAPI · Issue #3525 · cr...
What would you like to be added? Add node-specific allowlists the ability to tie allowlists to specific bouncers / a list of bouncers if no bouncer is set, the allowlist can remain global, thus not...
blotus
blotus•2mo ago
just tried your postoverflow locally, it seems to work for me: 
INFO[2025-07-03T11:29:18+02:00] Ban for 1.2.3.4 whitelisted, reason [<company> customer requested whitelist]  id=polished-bush name=foo/custom-whitelist stage=s01-whitelist
DEBU[2025-07-03T11:29:18+02:00] Event leaving node : ok                       id=polished-bush name=foo/custom-whitelist stage=s01-whitelist
INFO[2025-07-03T11:29:18+02:00] Ip 1.2.3.4 performed 'crowdsecurity/http-open-proxy' (1 events over 0s) at 2025-07-03 09:29:18.891243543 +0000 UTC 
INFO[2025-07-03T11:29:18+02:00] [Ip 1.2.3.4 performed 'crowdsecurity/http-open-proxy' (1 events over 0s) at 2025-07-03 09:29:18.891243543 +0000 UTC] is whitelisted, skip. 
INFO[2025-07-03T11:29:18+02:00] Ban for 1.2.3.4 whitelisted, reason [<company> customer requested whitelist]  id=polished-bush name=foo/custom-whitelist stage=s01-whitelist
DEBU[2025-07-03T11:29:18+02:00] Event leaving node : ok                       id=polished-bush name=foo/custom-whitelist stage=s01-whitelist
INFO[2025-07-03T11:29:18+02:00] Ip 1.2.3.4 performed 'crowdsecurity/http-open-proxy' (1 events over 0s) at 2025-07-03 09:29:18.891243543 +0000 UTC 
INFO[2025-07-03T11:29:18+02:00] [Ip 1.2.3.4 performed 'crowdsecurity/http-open-proxy' (1 events over 0s) at 2025-07-03 09:29:18.891243543 +0000 UTC] is whitelisted, skip.
alukas
alukasOP•2mo ago
It works for us with postfix, e.g. it whitelists requests when the scenario is crowdsecurity/postfix-relay-denied But I just checked all our webhosting servers (running apache) and none of them seem to whitelist the IPs. These are the logs that got the user banned with crowdsecurity/http-backdoors-attempts: 
<IP> - - [03/Jul/2025:07:04:41 +0200] "GET /api/post/c99.php HTTP/1.1" 404 53 "-" "Mozilla/5.0 (compatible; NuxtSSRBot/1.0; +https://<customer_site>.hu)"
<IP> - - [03/Jul/2025:07:04:49 +0200] "GET /api/post/xx.php HTTP/1.1" 404 53 "-" "Mozilla/5.0 (compatible; NuxtSSRBot/1.0; +https://<customer_site>.hu)"
<IP> - - [03/Jul/2025:07:04:41 +0200] "GET /api/post/c99.php HTTP/1.1" 404 53 "-" "Mozilla/5.0 (compatible; NuxtSSRBot/1.0; +https://<customer_site>.hu)"
<IP> - - [03/Jul/2025:07:04:49 +0200] "GET /api/post/xx.php HTTP/1.1" 404 53 "-" "Mozilla/5.0 (compatible; NuxtSSRBot/1.0; +https://<customer_site>.hu)"
Before this, they had a user agent that got constantly banned with bad-user-agent, we told them to fix that, and that helped, there are no more bans like that.
blotus
blotus•2mo ago
are you sure those are legit requests ? Because c99.php is a very well known PHP backdoor It's extremely suspicious for a request to request this and are you sure the postoverflows is properly loaded on the apache servers ?
alukas
alukasOP•2mo ago
Yeah, as I said the customer is using some kind of self-developed nuxt js software, and they basically created a proxy that pushes through every request, but the IP is always the same. I am sure it is a malicious request, we have told the customer about this, they are aware of the problems this could cause, but still asked us to whitelist the IP. This is a shared hosting server, so we can't just turn off crowdsec altogether. As far as I am aware, they are loaded correctly, I'm not sure how else I could check: 
$ cscli postoverflows inspect <company>/custom-whitelist
type: postoverflows
stage: s01-whitelist
name: <company>/custom-whitelist
file_name: <company>_custom.yaml
dependencies: {}
local_path: /etc/crowdsec/postoverflows/s01-whitelist/<company>_custom.yaml
downloadpath: ""
up_to_date: true
tainted: false
installed: true
local: true
$ cscli postoverflows inspect <company>/custom-whitelist
type: postoverflows
stage: s01-whitelist
name: <company>/custom-whitelist
file_name: <company>_custom.yaml
dependencies: {}
local_path: /etc/crowdsec/postoverflows/s01-whitelist/<company>_custom.yaml
downloadpath: ""
up_to_date: true
tainted: false
installed: true
local: true
And it is also visible with postoverflows list, the status is enabled, local
iiamloz
iiamloz•2mo ago
if you check /var/log/crowdsec.log do you see it being loaded? like grep "custom-whitelist" /var/log/crowdsec.log
alukas
alukasOP•2mo ago
Yes 
time="2025-07-03T11:30:18+02:00" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/postoverflows/s01-whitelist/<company>_custom.yaml stage=s01-whitelist
time="2025-07-03T11:30:18+02:00" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/postoverflows/s01-whitelist/<company>_custom.yaml stage=s01-whitelist
iiamloz
iiamloz•2mo ago
and within ls -la /var/lib/crowdsec/data | grep "ipwl.txt" exists?
alukas
alukasOP•2mo ago
Yes, and the IP is in there (with a /32 range) 
~# ls -alh /var/lib/crowdsec/data/ | grep ipwl
-rw-r--r-- 1 root root  285 Jun 30 16:50 <company>_ipwl.txt
~# ls -alh /var/lib/crowdsec/data/ | grep ipwl
-rw-r--r-- 1 root root  285 Jun 30 16:50 <company>_ipwl.txt
blotus
blotus•2mo ago
what does the debug output show ?
alukas
alukasOP•2mo ago
Also, the crowdsec install and the configurations are the same across all servers, as they are put there with SaltStack. So the one that works on the SMTP servers is the same as the one that doesn't work on our webservers. I'll try the debug output, i'll just have to set up a quick test environment, because the customer has started using another provider temporarily. Ah, I think I've found an issue based on the debug output. Is reloading crowdsec after changing the file not enough? In the debug output when it loops through the whitelisted IP range array, the newly added IP ranges are not there after a 
systemctl reload crowdsec
systemctl reload crowdsec
(they are in the ipwl.txt file though) systemctl restart crowdsec did the trick, now it shows up in the debug output. Is this really necessary / designed this way though? Restarts take much longer, and we'd prefer not to restart the service every time a new IP is added to the whitelist.
blotus
blotus•2mo ago
that's weird: internally, a reload is basically a soft restart (we kill almost everything, and then start it again)
alukas
alukasOP•2mo ago
That's what I thought too. And we modeled our whitelist script after the cron.daily upgrade script, which also only does a reload.
blotus
blotus•2mo ago
i can confirm the issue
blotus
blotus•2mo ago
the bug was introduced by this PR: https://github.com/crowdsecurity/crowdsec/pull/2419
GitHub
File init improvements by LaurenceJJones · Pull Request #2419 · c...
Current implementation has two flaws: If user supplies same file X time, its loaded X times. This problem is extended with RE2 feature flag The current map check only checks strings map, so added ...
alukas
alukasOP•2mo ago
Can we expect this to be fixed in the next few releases or should we find a workaround?
blotus
blotus•2mo ago
it will most likely be fixed in the next release, the fix should be fairly easy for the time being, the only workaround I can think of is to restart crowdsec :/
alukas
alukasOP•2mo ago
Alright, thank you very much for the help. I'm glad we could get to the bottom of it. Could you please ping me with a github issue when one gets created if it isn't too much of a hassle?
blotus
blotus•2mo ago
https://github.com/crowdsecurity/crowdsec/issues/3711
GitHub
Datafiles are not properly re-read on reload · Issue #3711 · crow...
When crowdsec is reloaded, any datafile (of type file or regexp) existing before the reload will not be reread, and the content will not be available for use by the parsers/scenarios. The bug was i...
alukas
alukasOP•2mo ago
Thank you very much! Have a nice day!
CrowdSec
CrowdSec•2mo ago
Resolving Postoverflow Whitelist Ignored This has now been resolved. If you think this is a mistake please run /unresolve

Did you find this page helpful?