Parser failures with NPMplus logs
Using the collection here: https://app.crowdsec.net/hub/author/ZoeyVid/collections/npmplus
The npmplus-logs parser fails to parse log lines and I wanted to reach you before I waste too much time by learning to fix it if the issue is somewhere else. I force upgraded the parser to restore the original state before reporting about the issue. Explain parameter was used with log and file options. The setup itself consists of the npmplus container installed with proxmox helper scripts.
If I'm not mistaken, parser success is achieved only when it's whitelisted or matches a scenario. It could explain the test results in cscli_explain_pt2.txt. Basically lines with local IP work well or at least they are handled because of the whitelist. I cannot make safe or attack attempts to work when public IP is used. There are many scenarios installed, such as http-sensitive-files.
I tested the access.log itself and for some reason most if not all of it was whitelisted even with the public IPs. The whitelisted local IP, a real attack by someone and the custom whitelist itself are attached in whitelist+access-log_lines.txt.
I also tried a lot of things by modifying the parser, such as modifying timestamp and other pattern related parts a lot and modifying place of the onsuccess. None of them gave any good results to share.
This is a bit unrelated own topic but I would really like to understand it better. I expect openappsec to handle many attacks but I still see something logged with crowdsec often and mostly I'm worried of the security of the locally hosted publicly accessible services. I'm a bit confused if openappsec/crowdsec should only block whenever some scenarios fulfill it or should I take e.g. openwrt router firewall into use with it to block the IPs(or just the server/container side firewall). As far as I understand, currently no firewalls play any role with this topic. Also openappsec log checking is missing on Crowdsec side which I'm not sure about if it's necessary.
13 Replies
Important Information
This post has been marked as resolved. If this is a mistake please press the red button below or type
/unresolveĀ© Created By WhyAydan for CrowdSec ā¤ļø
it just means the line did not match any know parser that would match scenario while whitelisting just allows it to go through as safe
The public IP(the first line in whitelist+access-log_lines) shouldn't be whitelisted. The reason even refers to the same mentioned in my custom whitelist which should affect local IPs.
your whiltelist is:
and when you do a request where remote address is
192.168.100.191 then it gets whitelisted by the first entry in the whitelist CIDR range ( "192.168.100.0/24")Yes, that's understandable and the expected situation.
so it even says that it went through whitelisted check succesfully and becasue it is trustend then it will be allowed
then next check goes from
142.168.100.191, and this is parsed, but is not detected as a whitelisted address, so it goes further in processing
but because it does not match any other rules it ends as parse falure, which is expected (but not very intuitive)
compare commands where there is HTTP 200 changed to HTTP 403
and see what happens
(ok maybe not the best example XD)
(oh wait maybe because I don't have that parser and need to adjust it for nginx )I expected the existing scenarios to handle important attack attempts that get through. I'm not sure if there's enough coverage. At least your example is clear and parser shows failure with the second command as I expected.
However, checking the access.log with the cscli explain seems to give confusing results as it's shown in the already mentioned file that also contains the whitelist. There's the weird behavior of the public IP and whitelist. I just tried it manually with log parameter and in that case it's not whitelisted. š®
adjusted for nginx logs, so it got parsed but no scenarios, but notice that I use nginx logs (there is a change after HTTP/1.1 with response code, bytes set (probably?) and the other values are in my case in double quotes, not sure how npmplus logs it, maybe it needs adjustment
it was parsed but it did not match any scenarios
but if I change the response code from HTTP 200 to HTTP 400 (bad request) it looks differently:
so maybe that parser is specific, or just wrong š
here are example test logs https://github.com/crowdsecurity/hub/blob/master/.tests/npmplus-logs/ and the parser https://github.com/crowdsecurity/hub/blob/master/parsers/s01-parse/ZoeyVid/npmplus-logs.yaml so maybe it needs more tests
or you may need additional rules to detect certain requests which you thing are malicious
I got it working or at least hopefully. I was able to trigger the ban based on the openappsec actions and also directly with npmplus log parsing. The ban action was shown in cscli alerts and decisions list.
ā 1491 ā Ip:REDACTED ā crowdsecurity/http-path-traversal-probing ā SE ā 212238 Datacamp Limited ā ban:1 ā ----
ā 1490 ā Ip:REDACTED ā openappsec/openappsec-probing ā SE ā 212238 Datacamp Limited ā ban:1 ā ----
Unfortunately it's difficult to summarize every change that could affect and the root cause may still be something that I don't fully understand. Firstly, I modified docker compose.yaml which was originally based on the file shared in npmplus repo by ZoeyVid. The main changes were:
- ipc from shareable to host in openappsec-agent
- added volume "shm-volume:/dev/shm/check-point" for npmplus and openappsec
- added IP(127.0.0.1) for port definitions in crowdsec container and changed the network_mode from host to bridge
- regenerated api key for the bouncer
I commented out the following lines under acquis.d/npmplus.yaml because it looked like duplicate processing of the same file may have broken something but maybe I need to retry that part if I ever need modsecurity.
> #---
> #filenames:
> #- /opt/npmplus/nginx/*.log
> #labels:
> #type: modsecurity
I removed the custom whitelist yaml under crowdsec/conf/parsers/s01-whitelist/ because it caused some trouble. Now that I rethink it, maybe this update affects:
Ā Ā Ā Ā |Ā Ā Ā Ā ā š¢ ZoeyVid/npmplus-logs (+20 ~2)
Ā Ā Ā Ā |Ā Ā Ā Ā Ā Ā Ā Ā ā update evt.Stage : s01-parse -> s01-whitelist
I had to enable debug logging in the crowdsec/conf/config.yaml and there was more proof what could affect:
time="REDACTED" level=debug msg="Event not parsed, expected stage 's02-enrich' got 's01-whitelist', abort"
I modified trusted_ips list in config.yaml in config.yaml to avoid accidentally banning other related docker containers.
- 172.17.0.0/24
- 172.18.0.0/24`
Earlier I did take allowlist into use because I understood that it's used in various places and sounds better option than the whitelist.Yeah the stage
s01-whitelist is not an officially supported stage under parsers (crowdsec is designed to be adaptable in stages but only for very advance users that need additional stages), whitelists at IP level should go under s02-enrich
the only s01-whitelist is for postoverflows as shown https://docs.crowdsec.net/docs/next/log_processor/whitelist/introI must have made a mistake with the whitelist location at that time. So, would I be fine by using the centralized allowlist or is there some benefit to e.g. whitelist local IP addresses earlier to improve the performance? As I said, the allowlist is basically the only list in use right now.
technically yes, using parser level whitelist does improve performance as it prevents the IP from being poured into scenarios.
However, at the end of the day it would only make a massive difference if you processing millions of logs lines.
Resolving Parser failures with NPMplus logs
This has now been resolved. If you think this is a mistake please run
/unresolve